Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in

An experiment comparing the effectiveness of the all-uses and all-edges test data adequacy criteria was performed. The experiment was designed so as to overcome some of the deficiencies of previous software testing experiments. A large number of test sets was randomly generated for each of nine subject programs with subtle errors. For each test set, the percentages of executable edges and definition-use associations covered were measured and it was determined whether the test set exposed an error. Hypothesis testing was used to investigate whether all-uses adequate test sets are more likely to expose errors than are all-edges adequate test sets. All-uses was significantly more effective than all-edges for five of the subjects, and appeared guaranteed to detect the error in four of them. Further analysis showed that in four of these subjects, all-uses-adequate test sets were more effective than all-edges-adequate test sets of similar size. Logistic regression analysis was used to investigate whether the probability that a test set exposes an error increases as the percentage of definition-use associations or edges covered by it increases. The evidence did not strongly support this conjecture. Error exposing ability was shown to be strongly positively correlated to percentage of covered definition-use associations in only four of the nine subjects. Error exposing ability was also shown to be positively correlated to the percentage of covered edges in four (different) subjects, but the relationship was weaker. Author's address: Computer Science Dept., Polytechnic University, 6 Metrotech Center, Brooklyn, N.Y. 11201. E-mail: pfrankl@poly.edu. Supported in part by NSF Grants CCR-8810287 and CCR9206910 and by the New York State Science and Technology Founda...

This article presents a testing methodology that adapts data flow adequacy criteria and coverage monitoring to the task of testing spreadsheets. To accommodate the evaluation model used with spreadsheets, and the interactive process by which they are created, our methodology is incremental. To accommodate the users of spreadsheet languages, we provide an interface to our methodology that does not require an understanding of testing theory. We have implemented our testing methodology in the context of the Forms/3 visual spreadsheet language. We report on the methodology, its time and space costs, and the mapping from the testing strategy to the user interface. In an empirical study, we found that test suites created according to our methodology detected, on average, 81% of the faults in a set of faulty spreadsheets, significantly outperforming randomly generated test suites

Size and code coverage are important attributes of a set of tests. When a program P is executed on el-ements of the test set T, we can observe the fault detecting capability of T for P. We can also observe the degree to which T induces code coverage on P according to some coverage criterion. We would like to know whether it is the size of T or the coverage of T on P which determines the fault detection effec-tiveness of T for P. To address this issue we ask the following question: While keeping coverage constant, what is the effect on fault detection of reducing the size of a test set? We report results from an empiri-cal study using the block and all-uses criteria as the coverage measures.

Security in computer systems is important so as to ensure reliable operation and to protect the integrity of stored information. Faults in the implementation of critical components can be exploited to breach security and penetrate a system. These faults must be identified, detected, and corrected to ensure reliability and safeguard against denial of service, unauthorized modification of data, or disclosure of information. We define a classification of security faults in the Unix operating system. We state the criteria used to categorize the faults and present examples of the different fault types. We present the design and implementation details of a prototype database to store vulnerability information collected from different sources. The data is organized according to our fault categories. The information in the database can be applied in static audit analysis of systems, intrusion detection, and fault detection. We also identify and describe software testing methods that should be effective in detecting different faults in our classification scheme.

Software maintainers are faced with the task of regression testing: retesting a modified program on an often large number of test cases. The cost of regression testing can be reduced if the size of the program that must be retested is reduced and if old test cases and old test results can be reused. Tw o complimentary algorithms for reducing the cost of regression testing are presented. The first produces a program called differences that captures the semantic change between certified, apreviously tested program, and modified, achanged version of certified. Itis more efficient to test differences, because it omits unchanged computations. The program differences is computed using a combination of program slices. The second algorithm identifies test cases for which certified and modified will produce the same output and existing test cases that will test components new inmodified. Not rerunning test cases that produce the same output avoids unproductive testing; testing new components with existing test cases avoids the costly construction of new test cases. The second algorithm is based on the notion of common execution patterns, which is the interprocedural extension of the notion introduced by Bates and Horwitz. Program components with common execution patterns have the same execution pattern during some call to their procedure. They are computed using a new type of interprocedural slice called a calling context slice. Whereas an interprocedural slice includes the program components necessary to

Form-based visual programming languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that form-based visual programs often contain faults. We would like to provide at least some of the benefits of formal testing methodologies to the creators of these programs. This paper presents a testing methodology for form-based visual programs. To accommodate the evaluation model used with these programs, and the interactive process by which they are created, our methodology is validationdriven and incremental. To accommodate the users of these languages, we provide an interface to the methodology that does not require an understanding of testing theory. We discuss our implementation of this methodology and empirical results achieved in its use.

This paper compares the fault-detecting ability of several software test data adequacy criteria. It has previously been shown that if C_1 properly covers C_2, then C_1 is guaranteed to be better at detecting faults than C_2, in the following sense: a test suite selected by independent random selection of one test case from each subdomain induced by C_1 is at least as likely to detect a fault as a test suite similarly selected using C_2. In contrast, if C_1 subsumes but does not properly cover C_2, this is not necessarily the case. These results are used to compare...

Current software testing practices focus, almost exclusively, on the implementation, despite widely acknowledged benefits of testing based on software specifications. We propose approaches to specification-based testing by extending a wide variety of implementation-based testing techniques to be applicable to formal specification languages. We demonstrate these approaches for the Anna and Larch specification languages. 1 Introduction Specifications provide valuable information for testing. Most software testing techniques, however, rely solely on the implementation for information upon which to select test data. These implementation-based testing techniques focus on the actual behavior of the implementation but ignore intended behavior, except inasmuch as test output is manually compared against it. On the other hand, considering information from formal specifications enables testing intended behavior as well as actual functionality. Specification-based testing techniques may direct ...

: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : xiii 1. INTRODUCTION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Problem and Motivation : : : : : : : : : : : : : : : : : : : : : : : : 2 1.2 Scope of This Research : : : : : : : : : : : : : : : : : : : : : : : : : 2 1.3 Organization of This Dissertation : : : : : : : : : : : : : : : : : : : 5 2. AN OVERVIEW OF MUTATION AND DATA FLOW TESTING : : : : 9 2.1 Mutation Testing : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 2.2 Alternate Mutation Testing : : : : : : : : : : : : : : : : : : : : : : 10 2.2.1 Randomly Selected x% Mutation Testing : : : : : : : : : : : 10 2.2.2 Constrained Mutation Testing : : : : : : : : : : : : : : : : : 11 2.2.3 Other Forms of Mutation Testing : : : : : : : : : : : : : : : 12 2.3 Data Flow Testing : : : : : : : : : : : : : : : : : : : : : : : : : : : 14 2.4 Testing Tools : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 16 3. A FORMALEVALUATIONOF M...