Structural coverage criteria are often used as an indicator of the thoroughness of testing, but complete satisfaction of a criterion is seldom achieved. When a software product is released with less than 100% coverage, testers are explicitly or implicitly assuming that executions satisfying the remaining test obligations (the residue) are either infeasible or occur so rarely that they havenegligible impact on quality. Violation of this assumption indicates shortcomings in the testing process. Monitoring in the deployed environment, even in the beta test phase, is typically limited to error and sanity checks. Monitoring the residue of test coverage in actual use can provide additional useful information, but it is unlikely to be accepted by users unless its performance impact is very small. Experience with a prototype tool for residual test coverage monitoring of Java programs suggests that, at least for statementcoverage, the simple strategy of removing all probes except those corresponding to the residue of coverage testing reduces execution overhead to acceptably low levels. Keywords Testing, coverage, instrumentation. 1

Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Tests are commonly generated from program source code, graphical models of software (such as control flow graphs), and specifications/requirements. UML collaboration diagrams represent a significant opportunity for testing because they precisely describe how the functions the software provides are connected in a form that can be easily manipulated by automated means. This paper presents novel test criteria that are based on UML collaboration diagrams. The most novel aspect of this is that tests can be generated automatically from the software design, rather than the code or the specifications. Criteria are defined for both static and dynamic testing of specification-level and instance-level collaboration diagrams. These criteria allow a formal integration tests to be based on high level design notations, which can help lead to software that is significantly more reliable. 1

This paper presents general criteria for generating test inputs from state-based specifications. Software testing can only be formalized and quantified whena solid basis for test generation can be defined. Formal specifications of complex systems represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can easily be manipulated. These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. Empirical results from a comparative case study application of these criteria are presented.

Integration testing is an important part of the testing process, but few integration testing techniques have been systematically studied or defined. The goal of this research istodevelop practical, effective, formalizable, automatable techniques for testing of connections between components during software integration. This paper presents an integration testing technique that is based on couplings between software components. This technique can be used to support integration testing of software components, and satisfies part of the USA's Federal Aviation Authority's (FAA) requirements for structural coverage analysis of software. The coupling-based testing technique is described, and the coverage criteria for three types of couplings are defined. Techniques and algorithms for developing coverage analyzers to measure the extent to which a test set satisfies the criteria are presented, and results from a comparative case study are presented.

Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated. This paper presents general criteria for generating test inputs from state-based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults.

Low module coupling is considered to be a desirable quality for modular programs to have. Previously, coupling has been defined subjectively, and not quantified, making it difficult to use in practice. In this paper, we extend previous work to reflect newer programming languages, and quantify coupling by developing a general software metric system that allows us to automatically measure coupling. We have precisely defined the levels of coupling so that they can be determined algorithmically, incorporated the notion of direction into the coupling levels, and accounted for different types of non-local variables present in modern programming languages. With our system, we can measure the coupling between all pairs of modules in a system, measure the coupling of a particular module with all other modules in a system, and measure the coupling of an entire system. We have implemented our metric system so that it measures the coupling between pairs of procedures in arbitrary C progra...

This paper presents results from a project that is trying to find ways to incorporate test data generation into practical test processes. The results include a new procedure for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and dynamically "pushes" the values through the control-flow graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed. This procedure uses new analysis techniques, offers improvements over previous research results in constraint-based testing, and combines several steps into one coherent process. The dynamic nature of this procedure yields several benefits. Moving through the control flow graph dynamically allows path constraints to be resolved immediately, which is more efficient both in space and time, and more often successful than constraint-based testing. This new procedure also incorporates an intelligent search technique based on bisection. The dynamic nature of this procedure also allows certain improvements to be made in the handling of arrays, loops, and expressions; language features that are traditionally difficult to handle in test data generation systems. The paper presents the test data generation procedure, examples to explain the working of the procedure, and results from a proof-of-concept implementation.

Mutation testing is a technique for testing software units that has great potential for improving the quality of testing, and thereby increasing our ability to assure the high reliability of critical software. It will be shown that recent advances in mutation research have brought a practical mutation testing system closer to reality. One recent advance is a partial solution to the problem of automatically detecting equivalent mutant programs. Equivalent mutants are currently detected by hand, which makes it very expensive and time-consuming. The problem of detecting equivalent mutants is a specific instance of a more general problem, commonly called the feasible path problem, which says that for certain structural testing criteria some of the test requirements are infeasible in the sense that the semantics of the program imply that no test case satis es the test requirements. Equivalent mutants, unreachable statements in path testing techniques, and infeasible DU-pairs in dataflow testing are all instances of the feasible path problem. This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to automatically detect some equivalent mutants and infeasible paths.

Condition coverage testing is a family of testing techniques that are based on the logical flow of control through a program. The condition coverage techniques include a variety of requirements, including that each statement in the program is executed and that each branch is executed. Mutation testing is a fault-based testing technique that is widely considered to be very powerful, and that imposes requirements on testing that include, and go beyond, many other techniques. In this paper, we consider the six common condition coverage techniques, and formally show that these techniques are subsumed by mutation testing, in the sense that if mutation testing is satisfied, then the condition coverage techniques are also satisfied. The fact that condition coverage techniques are subsumed by mutation has immediate practical significance because the extensive research that has already been done for mutation can be used to support condition coverage techniques, including automated tool...

The all-du-paths structural testing criterion is one of the most discriminating of the data flow testing criteria. Unfortunately, in the worst case, the criterion requires an intractable number of test cases. In a case study of an industrial software system, we find that the worst case scenario is rare. Eighty percent of the subroutines require ten or fewer test cases. Only one subroutine out of 143 requires an intractable number of tests. However, the number of required test cases becomes tractable when using the all-uses criterion. This paper includes a formal specification of both the all-du-paths criterion and the software tools used to estimate a minimal number of test cases necessary to meet the criterion. Keywords: Software testing, software measures formal specifications, data flow analysis. Department of Computer Science Technical Report #CS--91--105 Colorado State University (Revision of Technical Report #CS--89--118) A version of this paper was published in IEE/BCS Software Engineering Journal, 7(1):43-51, Jan. 1992. This research was partially supported by the NATO Collaborative Research Grants Program under RG. 0343/88. 1