Abstract

Recent advances in training deep (multi-layer) architectures have inspired a renaissance in neural network use. For example, deep convolutional networks are becoming the default option for difficult tasks on large datasets, such as image and speech recognition. However, here we show that error rates below 1 % on the MNIST handwritten digit benchmark can be replicated with shallow non-convolutional neural networks. This is achieved by training such networks using the ‘Extreme Learning Machine ’ (ELM) approach, which also enables a very rapid training time ( * 10 minutes). Adding distortions, as is common practise for MNIST, reduces error rates even further. Our methods are also shown to be capable of achieving less than 5.5 % error rates on the NORB image database. To achieve these results, we intro-duce several enhancements to the standard ELM algorithm, which individually and in com-bination can significantly improve performance. The main innovation is to ensure each hidden-unit operates only on a randomly sized and positioned patch of each image. This form of random ‘receptive field ’ sampling of the input ensures the input weight matrix is sparse, with about 90 % of weights equal to zero. Furthermore, combining our methods with