Abstract

The human brain is adept at solving difficult high-level visual processing prob-lems such as image interpretation and object recognition in natural scenes. Over the past few years neuroscientists have made remarkable progress in understand-ing how the human brain represents categories of objects and actions in natural scenes [1, 2]. However, all current models of high-level human vision operate on hand annotated images in which the objects and actions have been assigned semantic tags by a human operator. No current models can account for high-level visual function directly in terms of low-level visual input (i.e., pixels). To overcome this fundamental limitation we sought to develop a new class of mod-els that can predict human brain activity directly from low-level visual input (i.e., pixels). We explored two classes of models drawn from computer vision and ma-chine learning. The first class of models was based on Fisher Vectors (FV) [3] and the second was based on Convolutional Neural Networks (ConvNets) [4, 5]. We find that both classes of models accurately predict brain activity in high-level visual areas, directly from pixels and without the need for any semantic tags or hand annotation of images. This is the first time that such a mapping has been obtained. The fit models provide a new platform for exploring the functional prin-ciples of human vision, and they show that modern methods of computer vision and machine learning provide important tools for characterizing brain function. 1