Although knowledge indexes our experiences of the world, the neural basis of this relationship remains to be determined. Previous neuroimaging research, especially involving knowledge biased to visual and functional information, suggests that semantic representations depend on modality-specific brain mechanisms. However, it is unclear whether sensory cortical regions, in general, support retrieval of perceptual knowledge. Using neuroimaging methods, we show that semantic decisions that index tactile, gustatory, auditory, and visual knowledge specifically activate brain regions associated with encoding these sensory experiences. Retrieval of tactile knowledge was specifically associated with increased activation in somatosensory, motor, and premotor cortical regions. In contrast, decisions involving flavor knowledge increased activation in an orbitofrontal region previously implicated in processing semantic comparisons among edible items. Perceptual knowledge retrieval that references visual and auditory experiences was associated with increased activity in distinct temporal brain regions involved in the respective sensory processing. These results indicate that retrieval of perceptual knowledge relies on brain regions used to mediate sensory experiences with the referenced objects.