Efficient and accurate approximations of nonlinear convolutional networks

Xiangyu Zhang; Jianhua Zou; Xiang Ming; Kaiming He; Jian Sun

doi:10.1109/CVPR.2015.7298809

Efficient and accurate approximations of nonlinear convolutional networks

Xiangyu Zhang
, Jianhua Zou
, Xiang Ming
, Kaiming He
, Jian Sun

Faculty of Electronic and Information Engineering

Xi'an Jiaotong University
Microsoft USA

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

228 Scopus citations

Abstract

This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the 'AlexNet' [11], but is 4.7% more accurate.

Original language	English
Title of host publication	IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015
Publisher	IEEE Computer Society
Pages	1984-1992
Number of pages	9
ISBN (Electronic)	9781467369640
DOIs	https://doi.org/10.1109/CVPR.2015.7298809
State	Published - 14 Oct 2015
Event	IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 - Boston, United States Duration: 7 Jun 2015 → 12 Jun 2015

Publication series

Name	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume	07-12-June-2015
ISSN (Print)	1063-6919

Conference

Conference	IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015
Country/Territory	United States
City	Boston
Period	7/06/15 → 12/06/15

Access to Document

10.1109/CVPR.2015.7298809

Cite this

Zhang, X., Zou, J., Ming, X., He, K., & Sun, J. (2015). Efficient and accurate approximations of nonlinear convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 (pp. 1984-1992). Article 7298809 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 07-12-June-2015). IEEE Computer Society. https://doi.org/10.1109/CVPR.2015.7298809

@inproceedings{b600fa10a04d4146b992ab6a19e7de99,

title = "Efficient and accurate approximations of nonlinear convolutional networks",

abstract = "This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9\%. Our accelerated model has a comparably fast speed as the 'AlexNet' [11], but is 4.7\% more accurate.",

author = "Xiangyu Zhang and Jianhua Zou and Xiang Ming and Kaiming He and Jian Sun",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 ; Conference date: 07-06-2015 Through 12-06-2015",

year = "2015",

month = oct,

day = "14",

doi = "10.1109/CVPR.2015.7298809",

language = "英语",

series = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",

publisher = "IEEE Computer Society",

pages = "1984--1992",

booktitle = "IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015",

}

Zhang, X, Zou, J, Ming, X, He, K & Sun, J 2015, Efficient and accurate approximations of nonlinear convolutional networks. in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015., 7298809, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 07-12-June-2015, IEEE Computer Society, pp. 1984-1992, IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, United States, 7/06/15. https://doi.org/10.1109/CVPR.2015.7298809

Efficient and accurate approximations of nonlinear convolutional networks. / Zhang, Xiangyu; Zou, Jianhua; Ming, Xiang et al.
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015. IEEE Computer Society, 2015. p. 1984-1992 7298809 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 07-12-June-2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Efficient and accurate approximations of nonlinear convolutional networks

AU - Zhang, Xiangyu

AU - Zou, Jianhua

AU - Ming, Xiang

AU - He, Kaiming

AU - Sun, Jian

PY - 2015/10/14

Y1 - 2015/10/14

N2 - This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the 'AlexNet' [11], but is 4.7% more accurate.

AB - This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the 'AlexNet' [11], but is 4.7% more accurate.

UR - https://www.scopus.com/pages/publications/84959238721

U2 - 10.1109/CVPR.2015.7298809

DO - 10.1109/CVPR.2015.7298809

M3 - 会议稿件

AN - SCOPUS:84959238721

T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

SP - 1984

EP - 1992

BT - IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015

PB - IEEE Computer Society

T2 - IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015

Y2 - 7 June 2015 through 12 June 2015

ER -

Efficient and accurate approximations of nonlinear convolutional networks

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this