Structural coverage criteria are often used as an indicator of the thoroughness of testing, but complete satisfaction of a criterion is seldom achieved. When a software product is released with less than 100% coverage, testers are explicitly or implicitly assuming that executions satisfying the remaining test obligations (the residue) are either infeasible or occur so rarely that they havenegligible impact on quality. Violation of this assumption indicates shortcomings in the testing process. Monitoring in the deployed environment, even in the beta test phase, is typically limited to error and sanity checks. Monitoring the residue of test coverage in actual use can provide additional useful information, but it is unlikely to be accepted by users unless its performance impact is very small. Experience with a prototype tool for residual test coverage monitoring of Java programs suggests that, at least for statementcoverage, the simple strategy of removing all probes except those corresponding to the residue of coverage testing reduces execution overhead to acceptably low levels. Keywords Testing, coverage, instrumentation. 1

Abstract not found

Regression testing is a commonly used activity whose purpose is to determine whether the modifications made to a software system have introduced new faults. For many large, complex, software systems the retest all strategy is not practical: the resources required to reexecute and verify all available test cases (i.e., time and human effort) are prohibitive. Ad hoc methods are not desirable, as they can compromise the reliability of the regression test activity and consequently the reliability of the software system being tested. In this paper we present a new technique for selecting regression test cases based on the modifications that have been made on the program. The technique, which is based on the idea of directly comparing source files from the old and the new version of the program, has been implemented in a tool called Pythia. A novel characteristic of Pythia, which is capable of analyzing large software systems written in C, is that it has been implemented primarily through th...

Abstract not found

All software testing methods depend on the availability of an oracle, that is, some method for checking whether the system under test has behaved correctly on a particular execution. An ideal oracle would provide an unerring pass/fail judgment for any possible program execution, judged against a natural specification of intended behavior. Practical approaches must make compromises to balance trade-offs and provide useful capabilities. This report surveys proposed approaches to the oracle problem that are general in the sense that they require neither pre-computed input/output pairs nor a previous version of the system under test. The survey is not encyclopedic, but discusses representative examples of the main approaches and tactics for solving common problems. Partially supported by the Italian National Research Council (CNR). This work has also been supported by the Defense Advanced Research Projects Agency and Rome Laboratory, Air Force Materiel Command, USAF, under agreement number F30602-97-2-0034. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency, Rome Laboratory, or the U.S. Government. 1 Contents 1

This paper presents Jartege, a tool which allows random generation of unit tests for Java classes specified in JML. JML (Java Modeling Language) is a specification language for Java which allows one to write invariants for classes, and pre- and postconditions for operations. As in the JML-JUnit tool, we use JML specifications on the one hand to eliminate irrelevant test cases, and on the other hand as a test oracle. Jartege randomly generates test cases, which consist of a sequence of constructor and method calls for the classes under test. The random aspect of the tool can be parameterized by associating weights to classes and operations, and by controlling the number of instances which are created. The practical use of Jartege is illustrated by a small case study.

Requirements engineering is concerned with the elicitation of high-level goals to be achieved by the system envisioned, the refinement of such goals and their operationalization into services and constraints, and the assignment of responsibilities for the resulting requirements to agents such as humans, devices, and software. Requirements engineering processes may often result in requirements and assumptions about agent behaviour that are too ideal; some of them are likely to be violated from time to time in the running system due to unexpected agent behaviour. The lack of anticipation of exceptional behaviors results in unrealistic, unachievable and/or incomplete requirements. As a consequence, the software developed from those requirements will inevitably result in poor performance, sometimes with critical consequences on the environment. This paper proposes systematic techniques for reasoning about obstacles to the satisfaction of goals, requirements, and assumptions elaborated in t...

The thesis proposes a number of techniques for elaborating requirements constructively from high-level goals. The techniques are based on the KAOS goal-oriented method for requirements engineering. This method consists in identifying goals and refining them into subgoals until the latter can be assigned as responsibilities of single agents such as humans, devices and software. Domain properties and assumptions about the software environment are also used during the goal refinement process. The method supports the exploration of alternative goal refinements and alternative responsibility assignments of goals to agents. It also supports the identification and resolution of conflicts between goals, and the identification and resolution of exceptional agent behaviors, called obstacles, that violate goals and assumptions produced during the goal refinement process. The thesis enriches the KAOS framework through three kinds of techniques: (a) techniques for identifying agents, goal refinements, and alternative responsibility assignments, and for deriving agent interfaces from such responsibility assignments; (b) techniques for deriving operational requirements from goal specifications; (c) techniques for generating obstacles to the satisfaction of idealized goals and assumptions, and for generating alternative obstacle resolutions. The result is a coherent body of systematic techniques for requirements elaboration that are both theoretically well-founded (a formal model of agent is defined) and effective in practice (the techniques are validated on two real case studies of significant size: the London ambulance despatching system, and the Bay Area Rapid Transit train system).

The trend towards processors with more and more parallel cores is increasing the need for software that can take advantage of parallelism. The most widespread method for writing parallel software is to use explicit threads. Writing correct multithreaded programs, however, has proven to be quite challenging in practice. The key difficulty is non-determinism. The threads of a parallel application may be interleaved non-deterministically during execution. In a buggy program, non-deterministic scheduling will lead to nondeterministic results—some interleavings will produce the correct result while others will not. We propose an assertion framework for specifying that regions of a parallel program behave deterministically despite nondeterministic thread interleaving. Our framework allows programmers to write assertions involving pairs of program states arising from different parallel schedules. We describe an implementation of our deterministic assertions as a library for Java, and evaluate the utility of our specifications on a number of parallel Java benchmarks. We found specifying deterministic behavior to be quite simple using our assertions. Further, in experiments with our assertions, we were able to identify two races as true parallelism errors that lead to incorrect non-deterministic behavior. These races were distinguished from a number of benign races in the benchmarks.

For two decades, software inspections have proven e ective for detecting defects in software. We have reviewed the di erent ways software inspections are done, created a taxonomy of inspection methods, and examined claims about the cost-e ectiveness of di erent methods. We detect a disturbing pattern in the evaluation of inspection methods. Although there is universal agreement on the e ectiveness of software inspection, their economics are uncertain. Our examination of several empirical studies leads us to conclude that the bene ts of inspections are often overstated and the costs (especially for large software developments) are understated. Furthermore, some of the most in uential studies establishing these costs and bene ts are 20 years old now, which leads us to question their relevance to today's software development processes. Extensive work is needed to determine exactly how, why, and when software inspections work, and whether some defect detection techniques might be more cost-e ective than others. In this article we ask some questions about measuring e ectiveness of software inspections and determining how much they really cost when their e ect on the rest of the development process is considered. Finding answers to these questions will enable us to improve the e ciency of software development. 1