Neural Network Golf: smallest network for a certain level of performance

I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few arithmetic operations (=fast).

I am interested primarily in convolutional neural networks for vision applications, using something simple like CIFAR-10 without augmentation as the benchmark. Top-performing networks on CIFAR in recent years have had anywhere between 100 million and 0.7 million parameters (!!), so clearly small size is not (always) a bad thing.

Small networks are also in general faster to train and overfit less. Moreover, recent work on Knowledge Distillation, FitNets, etc show ways of making smaller networks from large networks while preserving most of the performance.

Another question is, what is the best performance achievable with a network no larger than a fixed size?

Examples of especially small networks that get good performance (100k parameters with 10% on CIFAR, anyone?) or systematic studies of the size vs performance tradeoff would be appreciated.

Topic convnet hyperparameter neural-network

Category Data Science

1
answered at

You might be interested in this paper that explores a few of the questions you are asking: http://arxiv.org/pdf/1312.6184.pdf. It is aptly titled: "Do Deep Networks Really Need to Be Deep?"

The crux of the matter is that deep networks allow for a LOT of non-linearity in the data that is being described. For CIFAR-10, I suspect that something similar to what works well for MNIST will work well. Yan LeCunn's page about MNIST results gives a lot of clues about models with few parameters: http://yann.lecun.com/exdb/mnist/

The double-problem of image classification is that in addition to the classification you need to extract features somehow. Often the parameters in deep networks go towards feature extraction rather than the classification itself, which is usually a softmax multi-class logistic regression classifier.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.