We present an evaluation of exhaustive testing of linked data structures with sophisticated structural constraints. Specifically, we use the Korat testing framework to systematically enumerate all legal inputs within a certain size. We then evaluate the quality of this test suite according to several measurements: ability to detect injected faults in the original correct implementations, code coverage, and specification coverage. Our results indicate that it is feasible to use exhaustive testing to obtain, within a reasonable amount of time, a high-quality test suite that can detect almost all faults and achieve complete code and specification coverage. Moreover, our results show that our exhaustive tests are of higher quality than randomly selected test suites that contain the same number of inputs selected from a larger potential input set. We conclude that exhaustive testing is a practical and effective testing methodology for sophisticated linked data structures. 1.

Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties of structurally complex test inputs; presents a technique that automates generation of such inputs; describes the Korat tool that implements this technique for Java; and evaluates the effectiveness of Korat in testing a set of data-structure implementations. Our approach allows the developer to describe the properties of valid test inputs using a familiar implementation language such as Java. Specifically, the user provides an imperative predicate—a piece of code that returns a truth value—that returns true if the input satisfies the required property and false otherwise. Korat implements our technique for solving imperative predicates: given a predicate and a bound on the size of the predicate’s inputs, Korat automatically generates the bounded-exhaustive

This dissertation describes a method for systematic constraint-based test generation for programs that take as inputs structurally complex data, presents an automated SAT-based framework for testing such programs, and provides evidence on the feasibility of using this approach to generate high quality test suites and find bugs in non-trivial programs.

We present novel algorithms for parallel testing of code that takes structurally complex test inputs. The algorithms build on the Korat algorithm for constraint-based generation of structurally complex test inputs. Given an imperative predicate that specifies the desired structural constraints and a finitization that bounds the desired input size, Korat performs a systematic search to generate all test inputs (within the bounds) that satisfy the constraints. We present how to generate test inputs with a parallel search in Korat and how to execute test inputs in parallel, both off-line (when the inputs are saved on disk) and on-line (when execution immediately follows generation). The inputs that Korat generates enable bounded-exhaustive testing that checks the code under test exhaustively for all inputs within the given bounds. We also describe a novel methodology for reducing the number of equivalent inputs that Korat can generate. Our development of parallel Korat and the methodology for reducing equivalent inputs are motivated by testing an application developed at Google. The experimental results on running parallel Korat across up to 1024 machines on the Google’s infrastructure show that parallel test generation and execution can achieve significant speedup, up to 543.55 times.

Structural constraint solving is being increasingly used for software reliability tasks such as systematic testing or error recovery. For example, the Korat algorithm provides constraint-based test generation: given a Java predicate that describes desired input constraints and a bound on the input size, Korat systematically searches the bounded input space of the predicate to generate all inputs that satisfy the constraints. As another example, the STARC tool uses a constraint-based search to repair broken data structures. A key issue for these approaches is the efficiency of search. This paper presents a novel approach that significantly improves the efficiency of structural constraint solvers. Specifically, most existing approaches use backtracking through code re-execution to explore their search space. In contrast, our approach performs checkpoint-based backtracking by storing partial program states and performing abstract undo operations. The heart of our approach is a light-weight search that is performed purely through code instrumentation. The experimental results on Korat and STARC for generating and repairing a set of complex data structures show an order to two orders of magnitude speed-up over the traditionally used searches.

This paper presents a case study on how Korat can be used in system testing, specifically in testing a large fault-tree analyzer developed for NASA. A fault-tree analyzer takes as input a fault tree that models how combinations of failures in the components of a system produce overall failures of the system. Testing a fault-tree analyzer requires generating fault trees. Korat is a previously developed testing tool that automates generation of structurally complex test inputs. Fault trees are structural in that they can be represented as graphs, and the nodes in the graphs need to satisfy certain complex constraints. Korat allows the user to express these constraints in widely used imperative programming languages such as Java. Previous research has shown how to test a fault-tree analyzer using another testing tool that requires the user to express constraints in a declarative language. This paper compares these two approaches. The results show that Korat generates a larger number of inputs but does not prune out non-equivalent inputs and thus can generate inputs that reveal errors in the system under test. 1

This paper presents a case study on how Korat can be used in system testing, specifically in testing a large fault-tree analyzer developed for NASA. A fault-tree analyzer takes as input a fault tree that models how combinations of failures in the components of a system produce overall failures of the system. Testing a fault-tree analyzer requires generating fault trees. Korat is a previously developed testing tool that automates generation of structurally complex test inputs. Fault trees are structural in that they can be represented as graphs, and the nodes in the graphs need to satisfy certain complex constraints. Korat allows the user to express these constraints in widely used imperative programming languages such as Java. Previous research has shown how to test a fault-tree analyzer using another testing tool that requires the user to express constraints in a declarative language. This paper compares these two approaches. The results show that Korat generates a larger number of inputs but does not prune out non-equivalent inputs and thus can generate inputs that reveal errors in the system under test.