Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in

The testing process is typically systematic in test data selection and test execution. For the most part, however, the effective use of test oracles has been neglected, even though they are a critical component of the testing process. Test oracles prescribe acceptable behavior for test execution. In the absence of judging test results with oracles, testing does not achieve its goal of revealing failures or assuring correct behavior in a practical manner; manual result checking is neither reliable nor cost-effective. We argue that test oracles should be derived from specifications and in conjunction with testing criteria, represented in a common form, and their use made integral to the testing process. For complex, reactive systems, oracles must reflect the multiparadigm nature of the required behavior. Such systems are often specified using multiple languages, each selected for its utility in specifying a particular computational paradigm. Thus, we are developing an approach for derivi...

This paper presents general criteria for generating test inputs from state-based specifications. Software testing can only be formalized and quantified whena solid basis for test generation can be defined. Formal specifications of complex systems represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can easily be manipulated. These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. Empirical results from a comparative case study application of these criteria are presented.

Abstract- Our interest is in the design process that spans the gap between the requirements acquisition process and the implementation process, in which the basic architecture of a system is defined, and functions are allocated to software, hard-ware, and human agents. We call this process composite system design. Our goal is an interactive model of composite system design incorporating deficiency-driven design, formal analysis, incremental design and rationalization, and design reuse. We discuss knowledge representations and reasoning techniques for the product (composite system) that we are designing, and for the design process, which support these goals. To evaluate our model, we report on its use to rationally reconstruct the design of two existing composite systems. Index Terms-Automated analysis, composite systems, knowl-edge-based design, rational reconstruction, software specification. 0

Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated. This paper presents general criteria for generating test inputs from state-based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults.

Current software testing practices focus, almost exclusively, on the implementation, despite widely acknowledged benefits of testing based on software specifications. We propose approaches to specification-based testing by extending a wide variety of implementation-based testing techniques to be applicable to formal specification languages. We demonstrate these approaches for the Anna and Larch specification languages. 1 Introduction Specifications provide valuable information for testing. Most software testing techniques, however, rely solely on the implementation for information upon which to select test data. These implementation-based testing techniques focus on the actual behavior of the implementation but ignore intended behavior, except inasmuch as test output is manually compared against it. On the other hand, considering information from formal specifications enables testing intended behavior as well as actual functionality. Specification-based testing techniques may direct ...

A fundamental problem in software processes is how the intrinsic rigidity of a predefined (formal) model can be reconciled with the need for flexibility, change, and evolution. We therefore distinguish between software processes, as specified in a process description, and their actual performance by humans. Further, we claim that the two inevitably diverge, and thus it is necessary to provide means to reconcile them. We present a preliminary exploration into the problem. In particular, we illustrate how a temporal logic-based approach can be used to capture and tolerate some deviations from the process description during execution. We present a simple process language (LATIN), and its prototype environment (SENTINEL), in which these ideas are currently experimented. 1 Introduction The need for quality improvement in software production has led to a research focus on software processes. In particular, considerable research activity has been (and still is) devoted to process modeling l...

The problem of formally analyzing properties of real-time systems is addressed. A method is proposed that allows specifying system properties in the TRIO language (an extension of temporal logic suitable to deal explicitly with the “time ” variable and to measure it) and modeling the system as a timed Petri net. It is argued that such an approach is more general than analyzing program properties. The proof method is based on an axiomatization of timed Petri nets in terms of TRIO so that their properties can be derived as suitable theorems in much the same spirit as classical Hoare’s method allows proving properties of programs coded in a Pascal-like language. The method is then exemplified through two classical “benchmarks ” of the literature on concurrent and real-time systems, namely an elevator system and the dining philosophers problem. A thorough review of the related literature and a comparison thereof with the new method is also provided. Possible alternative methods, theoretical extensions, and practical applications are briefly discussed.

Specification exemplars are familiar to most software engineering researchers. For instance, many will have encountered the well known library and lift "problems", and will have seen one or more published solutions. Exemplars may serve several purposes: as drivers of and communication vehicles for individual research advances; to establish research agendas and to compare and contrast alternative approaches; and, ultimately, to lead to advances in software development practices. Because of their prevalence in the literature, exemplars are worth critical study. In this paper we consider the purposes that exemplars may serve, and explore the incompatibilities inherent in trying to simultaneously serve several of them at once. Researchers should therefore be clear about what successfully handling an exemplar demonstrates. We go on to examine the use of exemplars for not only specifications (an end product of requirements engineering), but also for the requirements engineering process itsel...

The paper defines formal methods (FMs) and describes economic issues involved in their application. From these considerations and the concepts implicit in “No Silver Bullet”, it becomes clear that FMs are best applied during requirements engineering. A explanation of why FMs work when they work is offered and it is suggested that FMs help the most when the applier is most ignorant about the problem domain.