We report the results of three empirical studies of fault detection and stability performance of the predicate-based BOR (Boolean OperatoR) testing strategy. BOR testing is used to develop test cases based on formal software specification, or based on the implementation code. We evaluated the BOR strategy with respect to some other strategies by using Boolean expressions and actual software. We applied it to software specification cause-effect graphs of a safety-related real-time control system, and to a set of N-version programs. We found that BOR testing is very effective at detecting faults in predicates, and that BOR-based approach has consistently better fault detection performance than branch testing, thorough (but informal) functional testing, simple statebased testing, and random testing. Our results indicate that BOR test selection strategy is practical and effective for detection of faulty predicates, and is suitable for generation of safety-sensitive test-cases. 1. Introduct...

In this paper we discuss the advantages and limitations of a specification-based software testing technique we call CEG-BOR. There are two phases in this approach. First, informal software specifications are converted into cause-effect graphs (CEG). Then, the Boolean OperatoR (BOR) strategy is applied to design and select test cases. The conversion of an informal specification into a CEG helps detect ambiguities and inconsistencies in the specification and sets the stage for design of test cases. The number of test cases needed to satisfy the BOR strategy grows linearly with the number of Boolean operators in CEG, and BOR testing guarantees detection of certain classes of Boolean operator faults. But, what makes the approach especially attractive is that the BOR based test suites appear to be very effective in detecting other fault types. We have empirically evaluated this broader aspect of the CEG-BOR strategy on a simplified safety-related real-time control system, a set of N-version programs, and on elements of a commercial data-base system. In all cases, CEG-BOR testing required fewer test cases than those generated for the applications without the use of CEGBOR. Furthermore, in all cases CEG-BOR testing detected all faults that the original, and independently generated, application test-suites did. In two instances CEG-BOR testing uncovered additional faults. Our results indicate that the CEG-BOR strategy is practical, scalable, and effective across diverse applications. We believe that it is a cost-effective methodology for the development of systematic specification-based software test-suites. ii Paradkar, Tai, and Vouk, Specification-Based Testing Using Cause-Effect Graphs 1 1

Mutation analysis is a fault-based testing technique that uses mutation operators to introduce small changes into a program or specification, producing mutants, and then chooses test cases to distinguish the mutants from the original. Mutation operators differ in the coverage they get. They also differ in the number of mutants they generate. Consequently, selecting mutation operators is an important problem whose solution affects the effectiveness and cost of mutation testing. We use the automated test generation and evaluation method that combines a model checker and mutation analysis. We define a set of mutation operators and implement a mutation generator for specifications written in SMV, a popular model checker. To select the most effective mutation operators and sets of operators, we compare them using both theoretical and experimental methods. We construct mutation detection conditions and develop a technique to theoretically compare mutation operators. We apply mutation coverage and pairwise coverage metrics to empirically compare the effectiveness of mutation operators. To detect a fault in a program, a test case must cause the fault to affect the outputs, not

Cause-Effect Graphing (CEG) is used to derive test cases from a given natural language specification to validate its corresponding implementation. This paper surveys some known drawbacks to CEG analysis and shows how these can be overcome. It also shows how the CEG technique can be used to derive discriminating customer-directed scenarios (use-cases) for validating requirements.

A recent study has classified faults in Boolean expressions into ten classes and has proved that there are five key fault classes, namely CCF, CDF, ORF, ENF and ASF, such that if a test suite can kill all faulty versions of these five core fault classes, if can kill all faulty versions of all fault classes. In order to generate more effective test suites, we should prioritize these five fault classes further, such that test cases with stronger fault detection capability could be generated as early as possible. Such a process is referred to as the fault class prioritization. Based on the observation in the fault class hierarchy, we divide the five fault classes into two groups {CCF, CDF} and {ORF, ENF, ASF}. Two strategies of fault class prioritization are proposed to generate test cases efficiently. We design experiments using TCAS Boolean expressions and some randomly generated Boolean expressions. The experimental results suggest that if we generate test cases for CCF and CDF firstly, the final test suite always have a higher efficiency of killing faults.