Constraint-based testing is a novel way of generating test data to detect specific types of common programming faults. The conditions under which faults will be detected are encoded as mathematical systems of constraints in terms of program symbols. A set of tools, collectively called Godzilla, has been implemented that automatically generates constraint systems and solves them to create test cases for use by the Mothra testing system. Experimental results from using Godzilla show that the technique can produce test data that is very close in terms of mutation-adequacy to test data that is produced manually, and at substantially reduced cost. Additionally, these experiments have suggested a new procedure for unit testing, where test cases are viewed as throw-away items rather than scarce resources. 1 INTRODUCTION This paper describes experimental results that are based on a new technique for generating test data. This technique, called constraint-based testing (CBT), uses the source code to automatically generate test data that attempts to satisfy the mutation-adequacy criteria. Elsewhere, we describe the technique [9, 12], and the details and algorithms of the implementation [29]; here we describe a set of experiments that measure CBT.

The conventional classification of software fault detection techniques as static or dynamic analysis is inadequate as a basis for identifying useful relationships between techniques. A more useful distinction is between techniques that sample the space of possible executions, and techniques that fold the space. The new distinction provides better insight into the ways different techniques can interact, and is a basis for considering hybrid fault detection techniques including combinations of testing and formal verification.

ProDAG is an analysis toolset that provides an application programmatic interface for program dependence analysis. Program dependences are syntactic relationships between program statements. The ProDAG interface provides a uniform set of operations for creating and accessing several pre-defined dependence relations, which are represented as graphs, as well as a standard mechanism for developing new dependence graphs. ProDAG is one analysis toolset in Arcadia. ProDAG was developed in the Arcadia environment and has been integrated with the language processing, object management, and process definition components of the Arcadia architecture, and further integration with the user interface development system and the measurement and evaluation components are underway. This paper describes the design, implementation and integration of ProDAG within Arcadia. 1 Introduction ProDAG (Program Dependence Analysis Graph) is a toolset that performs analysis of dependences between program statemen...

Database systems are an important asset for many businesses. As such, it is important to test database systems thoroughly, as any faults that remain hidden may significantly impact critical business processes. However, these systems bring additional complexities that make them amongst the most complex and difficult kind of system to test. While software testing in general is a well-developed area, techniques specifically aimed at testing database systems are still in their infancy. In this paper, we present a family of test adequacy criteria for database systems that can be used to determine the “quality ” of a test suite. These criteria consider various aspects of database systems including the source code structure (in terms of patterns of database operations), the existence of define–use pairs between database operations and the interactions between different applications of the database system. The criteria we present differ from existing adequacy criteria as we focus on a general definition of a database test case that is based on intensional constraints. This overcomes the problems associated with adequacy being constrained to a single static database state. We also consider transactional operators that alter the behaviour of a database system and influence adequacy.

"Propagation, infection, and execution analysis" (termed PIE) is used for predicting where faults can more easily hide in software. To make such predictions, programs are dynamically executed with test cases. In this paper, we collect information concerning those test cases into a histogram. Each bin in a histogram represents a single test case. The score in a bin predicts the likelihood that the test case will reveal a fault through the production of a failure (if a fault exists in the set of program locations that the test case executes). Preliminary experiments using program mutations suggest that the histogram technique presented in this paper can rank test cases according to their fault revealing ability.

I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state ” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra [Dijkstra, 1968] 1

Conformance testing procedures for generating tests from the finite state model representation of Role Based Access Control (RBAC) policies are proposed and evaluated. A test suite generated using one of these procedures has excellent fault detection ability but is astronomically large. Two approaches to reduce the size of the generated test suite were investigated. One is based on a set of six heuristics and the other directly generates a test suite from the finite state model using random selection of paths in the policy model. A fault model specific to the implementations of RBAC systems was used to evaluate the fault detection effectiveness of the generated test suites; the model incorporates both mutation-based and malicious faults. Empirical studies revealed that adequacy assessment of test suites using faults that correspond to first-order mutations may lead to a false sense of confidence in the correctness of policy implementation. The second approach to test suite generation, combined with one or more heuristics, is most effective in the detection of both first-order mutation and malicious faults and generates a significantly smaller test suite than the one generated directly from the finite state models.

As test case automation increases, the volume of tests can become a problem. Further, it may not be immediately obvious whether the test generation tool generates effective test cases. Indeed, it might be useful to have a mechanism that is able to learn, based on past history, which test cases are likely to yield more failures versus those that are not likely to uncover any. We present experimental results on using a neural network for pruning a testcase set while preserving its effectiveness. 1 INTRODUCTION Software testing still consumes major resources in a software product's lifecycle. A myriad of testing methods exist ranging from white box testing methods for unit and integration testing to grey and black box methods for system testing [17, 2]. Methods are usually associated with testing criteria, that identify what to test and howwe know that we are done. Examples include branch testing, dataflow testing, etc. Coverage metrics measure compliance with testing criteria and identi...

This paper proposes criteria for the verification of behavioral designs for hardware written in VHDL. The criteria are analogous to testing criteria for software, but were adapted to the specific needs and constructs of hardware designs written in VHDL. We examine the potential value of these criteria with respect to desirable properties for evaluation criteria that were originally developed for software. Then we apply the VHDL criteria to several design examples with varying complexities to demonstrate their practical usefulness. Although, applying software testing techniques to hardware design at the behavioral level is not new, this work, to the best of our knowledge, is the first attempt to analyze the approach from the theoretical point of view and to lay the groundwork for achieving error-free design at the behavioral level. 1. Introduction Since the standardization of VHDL and Verilog hardware description languages and the significant progress in behavioral synthesis, the prob...

This paper presents an approach to automatic unit test data generation for branch coverage using mixed-integer linear programming, execution trees, and symbolic execution. This approach can be useful to both general testing and regression testing after software maintenance and reengineering activities. Several strategies, including original algorithms, to move towards practical test data generation have been investigated in this paper. Methods include: ffl the analysis of minimum path-length partial execution trees for unconstrained arcs, thus increasing the generation performance and reducing the difficulties originated by infeasible paths ffl the reduction of the difficulties originated by nonlinear path conditions by considering alternative linear paths ffl the reduction of the number of test cases, which are needed to achieve the desired coverage, based on the concept of unconstrained arcs in a control flow graph ffl the extension of symbolic execution to deal with dynamic memory allocation and deallocation, pointers and pointers to functions Execution trees are symbolically executed to produce Extended Path Constraints (EPC), which are then partially mapped by an original algorithm into linear problems whose solutions correspond to the test data to be used as input to cover program branches. Partially mapping this problem into a linear optimization problem avoids infeasible and non-linear path problems, if a feasible linear alternate path exists in the same execution tree. The presented approach has been implemented in C++ and tested on C-language programs on a Pentium/Linux system. Preliminary results are encouraging and show that a high percentage of the program branches can be covered by the test data automatically produced. The approach is flexible to br...