Ideally, we might hope to improve the performance of our MT systems by improving the system, but it might be even more important to improve performance by looking for a more appropriate application. A survey of the literature on evaluation of MT systems seems to suggest that the success of the evaluation often depends very strongly on the selection of an appropriate application. If the application is well-chosen, then it often becomes fairly clear how the system should be evaluated. Moreover, the evaluation is likely to make the system look good. Conversely, if the application is not clearly identified (or worse, if the application is poorly chosen), then it is often very difficult to find a satisfying evaluation paradigm. We begin our discussion with a brief review of some evaluation metrics that have been tried in the past and conclude that it is difficult to identify a satisfying evaluation paradigm that will make sense over all possible applications. It is probably wise to identify the application first, and then we will be in a much better position to address evaluation questions. The discussion will then turn to the main point, an essay on how to pick a good niche application for state-of-the-art (crummy) machine translation.- 2-1.

This paper is an attempt to survey the field in this period and to provide, in effect, an updating of the historical review completed in late 1984 and early 1985 (Hutchins 1986). The aim is to give a general picture of the recent developments of systems and projects already established at that time and to document the emergence of new systems and projects worldwide. Descriptions of individual systems are given in section II. This part outlines the main issues and lines of development at the present time. There is no claim of completeness, but it is hoped that all significant activities have been noticed

Most MT systems seem to aim at simulation of the behaviour of human translators, although this is not very often stated as explicitly as in e.g. [JW87, page 136]). Like human translators they translate texts normally not written with the specific goal of translation in mind and just as in the case of human translators their translations will normally be revised prior to distribution to the end user. This predominance of what we will call the Human Translator Metaphor has – as a natural consequence – led to a view of MT evaluation where the main purpose of evaluation is to determine to what extent the makers of a system have succeeded in mimicking the human translator. Yet it has to be noted that this interpretation of the notion of MT system is the one least likely to be successful over the next 10 or 20 years. There has been no real breakthrough over the last decade, and there is little evidence that the next 10 years will present us with a revolution in MT. It is very tempting to object that progress has been made, and that