In this paper, we present anapproach to colorimage understandingthat accountsforcolorvariationsdue to highlights and shading. We demonstrate that the reflected light from every point on a dielectric object. such as plastic, can be described asa linearcombination of the object color and the highlight color. The colors of all light rays reflected from one object then form a planar cluster in the color space.The shapeof this cluster is determined by the object and highlight colors and by the object shape and illumination geometry. We present a method that exploits the difference between object color and highlight color to separate the color of every pixel into a matte component and a highlight component.This generates two intrinsic images, one showing the scene without highlights, and the other one showing only the highlights. The intrinsic images may be a useful tool for a variety of algorithms in computer vision. such as stereo vision, motion analysis, shape from shading,and shapefrom highlights. Ourmethod combines the analysis of matte and highlight reflection with a sensor model that accounts for camera limitations. This enables us to successfully run our algorithm on real images taken in a laboratory setting. We show and discuss the results.