Abstract. The COTESYS cluster of excellence a investigates cognition for technical systems such as vehicles, robots, and factories. Cognitive technical systems (CTS) are information processing systems equipped with artificial sensors and actuators, integrated and embedded into physical systems, and acting in a physical world. They differ from other technical systems as they perform cognitive control and have cognitive capabilities. Cognitive control orchestrates reflexive and habitual behavior in accord with longterm intentions. Cognitive capabilities such as perception, reasoning, learning, and planning turn technical systems into systems that “know what they are doing”. The cognitive capabilities will result in systems of higher reliability, flexibility, adaptivity, and better performance. They will be easier to interact and cooperate with.

In this study we review the area of Knowledge-Based Computer Vision (KBCV), putting in focus its knowledge representation aspects, starting from the year 2000. We introduce the topic by reviewing what researchers during the 1990’s saw as the future path to be taken and how these influenced recent contributions. We then proceed analysing the contributions in visual knowledge representation. We divide the analysis in the what part and in the how part. In the what part, we discuss the reviewed contributions under the light of different types of knowledge necessary for KBVC. In the how part, we discuss contributions in how this knowledge can be represented and structured, regarding representation formalism, model structure and symbol grounding. We also review full approaches for KBVC concerning their application domain. We conclude by exploring some alternatives for future research. 1