Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in

Recently, many formal methods, such as the SCR (Software Cost Reduction) requirements method, have been proposed for improving the quality of software specifications. Although improved specifications are valuable, the ultimate objective of software development is to produce software that satisfies its requirements. To evaluate the correctness of a software implementation, one can apply black-box testing to determine whether the implementation, given a sequence of system inputs, produces the correct system outputs. This paper describes a specification-based method for constructing a suite of test sequences, where a test sequence is a sequence of inputs and outputs for testing a software implementation.

This paper illustrates how software can be described precisely using LD-relations, how these descriptions can be presented in a readable manner using tabular notations, and one way such descriptions can be used to test programs. We describe an algorithm that can be used to generate a test oracle from program documentation, and present the results of using a tool based on it to help test part of a commercial network management application. The results demonstrate that these methods can be effective at detecting errors and greatly increase the speed and accuracy of test evaluation when compared with manual evaluation. Such oracles can be used for unit testing, --in situ testing, constructing self-checking software and ensuring consistency between code and documentation. Index Terms---Program testing, test oracle, formal specification, finite state machine. u 1 Introduction The Software Engineering Research Group at McMaster University is studying ways to improve the quality and...

This paper describes a generic tableau algorithm, which is the basis for a general customizable method for producing oracles from temporal logic specifications. A generic argument gives semantic rules with which to build the semantic tableau for a specification. Parameterizing the tableau algorithm by semantic rules permits it to easily accommodate a variety of temporal operators and provides a clean mechanism for fine-tuning the algorithm to produce efficient oracles. The paper develops conditions to ensure that a set of rules results in a correct tableau procedure. It gives sample rules for a variety of linear-time temporal operators and shows how rules are tailored to reduce the size of an oracle. Keywords: formal specification, verification, specificationbased test oracles, tableau methods, propositional temporal logic, test validation. 1 Introduction Temporal specifications describe constraints on the order in which events can occur in executions of a concurrent software syste...

Software testing involves execution of a program under test using some fault revealing input data and examination of the output to determine success or failure. A fundamental assumption of this testing is that there is some mechanism, an oracle, that will determine whether or not the results of a test execution are correct. In practice, this is often done by comparing the output, either automatically or manually, to some pre-calculated, presumably correct, output [39]. However, if the program is formally documented it is possible to use the specification to determine the success or failure of a test execution, as in [1], for example. This thesis discusses the development of a prototype tool that automatically generates a test oracle from formal program documentation. In [25], [27] and [28] Parnas et al. advocate the use of a relational model for documenting the intended behaviour of programs. In this method, tabular expressions are used to improve readability so that formal documentati...

... This thesis develops a unified solution to the GUI testing problem with the particular goals of automation and integration of tools and techniques used in various phases of GUI testing. These goals are accomplished by developing a GUI testing framework with a GUI model as its central component. For efficiency and scalability, a GUI is represented as a hierarchy of components, each used as a basic unit of testing. The framework also includes a test coverage evaluator, test case generator, test oracle, test executor, and regression tester. The test coverage evaluator employs hierarchical, event-based coverage criteria to automatically specify what to test in a GUI and to determine whether the test suite has adequately tested the GUI. The test case generator employs plan generation techniques from artificial intelligence to automatically generate a test suite. A test executor automatically executes all the test cases on the GUI. As test cases are being executed, a test oracle automatically determines the correctness of the GUI. The test oracle employs a model of the expected state of the GUI in terms of its constituent objects and their properties. After changes are made to a GUI, a regression tester partitions the original GUI test suite into valid test cases that represent correct input/output for the modified GUI and invalid test cases that no longer represent correct input/output. The regression tester employs a new technique to reuse some of the invalid test cases by repairing them.

Formal specifications have been a focus of software engineering research for many years and have been applied in a wide variety of settings. Their industrial use is still limited but has been steadily growing. After recalling the essence, role, usage, and pitfalls of formal specification, the paper reviews the main specification paradigms to date and discuss their evaluation criteria. It then provides a brief assessment of the current strengths and weaknesses of today's formal specification technology. This provides a basis for formulating a number of requirements for formal specification to become a core software engineering activity in the future.

We present a method for black-box boundary testing from B and Z formal speci cations. The basis of the method is to test every operation of the system at every boundary state using all input boundary values of that operation. The test generation process is highly automated.

AbstractResource-constrained devices are becoming ubiquitous. Examples include cell phones, palm pilots, and digital ther-mostats. It can be difficult to fit required functionality into such a device without sacrificing the simplicity and clarityof the software. Increasingly complex embedded systems require extensive brute-force testing, making development andmaintenance costly. This is particularly true for system components that are written in assembly language. Static check-ing has the potential of alleviating these problems, but until now there has been little tool support for programming at theassembly level.

All software testing methods depend on the availability of an oracle, that is, some method for checking whether the system under test has behaved correctly on a particular execution. An ideal oracle would provide an unerring pass/fail judgment for any possible program execution, judged against a natural specification of intended behavior. Practical approaches must make compromises to balance trade-offs and provide useful capabilities. This report surveys proposed approaches to the oracle problem that are general in the sense that they require neither pre-computed input/output pairs nor a previous version of the system under test. The survey is not encyclopedic, but discusses representative examples of the main approaches and tactics for solving common problems. Partially supported by the Italian National Research Council (CNR). This work has also been supported by the Defense Advanced Research Projects Agency and Rome Laboratory, Air Force Materiel Command, USAF, under agreement number F30602-97-2-0034. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency, Rome Laboratory, or the U.S. Government. 1 Contents 1