Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in

The basic unit of testing in an object-oriented program is a class. Although there has been much recent research on testing of classes, most of this work has focused on black-box approaches. However, since blackbox testing techniques may not provide sufficient code coverage, they should be augmented with code-based or white-box techniques. Dataflow testing is a codebased testing technique that uses the dataflow relations in a program to guide the selection of tests. Existing dataflow testing techniques can be applied both to individual methods in a class and to methods in a class that interact through messages, but these techniques do not consider the dataflow interactions that arise when users of a class invoke sequences of methods in an arbitrary order. We present a new approach to class testing that supports dataflow testing for dataflow interactions in a class. For individual methods in a class, and methods that send messages to other methods in the class, our technique is similar ...

Is there such a thing anymore as a software system that doesn't need to be secure? Almost every softwarecontrolled system faces threats from potential adversaries, from Internet-aware client applications running on PCs, to complex telecommunications and power systems accessible over the Internet, to commodity software with copy protection mechanisms. Software engineers must be cognizant of these threats and engineer systems with credible defenses, while still delivering value to customers. In this paper, we present our perspectives on the research issues that arise in the interactions between software engineering and security.

An experiment comparing the effectiveness of the all-uses and all-edges test data adequacy criteria was performed. The experiment was designed so as to overcome some of the deficiencies of previous software testing experiments. A large number of test sets was randomly generated for each of nine subject programs with subtle errors. For each test set, the percentages of executable edges and definition-use associations covered were measured and it was determined whether the test set exposed an error. Hypothesis testing was used to investigate whether all-uses adequate test sets are more likely to expose errors than are all-edges adequate test sets. All-uses was significantly more effective than all-edges for five of the subjects, and appeared guaranteed to detect the error in four of them. Further analysis showed that in four of these subjects, all-uses-adequate test sets were more effective than all-edges-adequate test sets of similar size. Logistic regression analysis was used to investigate whether the probability that a test set exposes an error increases as the percentage of definition-use associations or edges covered by it increases. The evidence did not strongly support this conjecture. Error exposing ability was shown to be strongly positively correlated to percentage of covered definition-use associations in only four of the nine subjects. Error exposing ability was also shown to be positively correlated to the percentage of covered edges in four (different) subjects, but the relationship was weaker. Author's address: Computer Science Dept., Polytechnic University, 6 Metrotech Center, Brooklyn, N.Y. 11201. E-mail: pfrankl@poly.edu. Supported in part by NSF Grants CCR-8810287 and CCR9206910 and by the New York State Science and Technology Founda...

After changes are made to a previously tested program, a goal of regression testing is to perform retesting based on the modifications while maintaining the same testing coverage as completely retesting the program. We present a novel approach to data flow based regression testing that uses slicing algorithms to explicitly detect definition-use associations that are affected by a program change. An important benefit of our slicing technique is that, unlike previous techniques, neither data flow history nor recomputation of data flow for the entire program is required to detect affected definition-use associations. The program changes drive the recomputation of the required partial data flow through slicing. Another advantage is that our technique achieves the same testing coverage with respect to the affected definition-use associations as a complete retest of the program without maintaining a test suite. Thus, the overhead of maintaining and updating a test suite is eliminated. 1. Int...

Although there is much interest in creating libraries of well-designed, thoroughly-tested classes that can be confidently reused for many applications, few class testing techniques have been developed. In this paper,wepresent a class testing technique that exploits the hierarchical nature of the inheritance relation to test related groups of classes by reusing the testing information for a parent class to guide the testing of a subclass. We i nitially test base classes having no parents by designing a test suite that tests each member function individually and also tests the interactions among member functions. To design a test suite for a subclass, our algorithm incrementally updates the history of its parent to reflect both the modified, inherited attributes and the subclass'snewly defined attributes. Only those new attributes or affected, inherited attributes are tested and the parent class'stest suites are reused, if possible, for the testing. Inherited attributes are retested in t...

this article, we present a general framework for developing demand-driven interprocedural data flow analyzers and report our experience in evaluating the performance of this approach. A demand for data flow information is modeled as a set of queries. The framework includes a generic demand-driven algorithm that determines the response to a query by iteratively applying a system of query propagation rules. The propagation rules yield precise responses for the class of distributive finite data flow problems. We also describe a two-phase framework variation to accurately handle nondistributive problems. A performance evaluation of our demand-driven approach is presented for two data flow problems, namely, reaching-definitions and copy constant propagation. Our experiments show that demand-driven analysis performs well in practice, reducing both time and space requirements when compared with exhaustive analysis.

Pointer aliasing analysis is crucial to compile-time analyses for languages with general-purpose pointer usage (such as C), but many aliasing methods have proven quite costly. We present a technique that partitions the statements of a program to allow separate, and therefore possibly different, pointer aliasing analysis methods to be used on independent parts of the program. This decomposition enables exploration of tradeoff between algorithm efficiency and precision. We also present a new, efficient flow-insensitive pointer aliasing algorithm, which is used together with an existing flow-sensitive aliasing algorithm in our experiments. We demonstrate our technique in the context of determining side effects and variable fetches through names containing pointer dereferences (Thru-deref MOD/REF). Initial empirical results using a combination of a flow-sensitive and a flowinsensitive aliasing analysis on the same program, demonstrate that the resulting analysis is much faster than solely ...

This paper discusses the principles of data flow testing, describes a software testing and analysis tool called ATAC, which measures the effectiveness of testing data, and presents two case studies connecting software quality with the control and data flow testing coverage measures: block, decision, c-uses and p-uses. Results from these studies indicate that these measures are useful in uniformly assessing the quality of software testing, in controlling the testing process, and in program understanding. Although programming style can affect test coverage, a quantitative relationship between high testing coverage and program quality is emerging. We suggest practical guidelines in defining high quality software in the light of testing coverage measures.

This paper presents general criteria for generating test inputs from state-based specifications. Software testing can only be formalized and quantified whena solid basis for test generation can be defined. Formal specifications of complex systems represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can easily be manipulated. These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. Empirical results from a comparative case study application of these criteria are presented.