When developing non-trivial formalizations in a theorem prover, a considerable amount of time is devoted to “debugging ” specifications and conjectures by failed proof attempts. To detect such problems early in the proof and save development time, we have extended the Isabelle theorem prover with a tool for testing specifications by evaluating propositions under an assignment of random values to free variables. Distribution of the test data is optimized via mutation testing. The technical contributions are an extension of earlier work with inductive definitions and a generic method for randomly generating elements of recursive datatypes. 1.

Abstract. Formulas are difficult to formulate and to prove, and are often invalid during specification development. Testing formulas prior to attempting any proofs could potentially save a lot of effort. Here we describe an implementation of random testing in the PVS verification system. 1

Abstract. This article presents the generation and test case execution under the framework Focal. In the programming language Focal, all properties of the program are written within the source code. These properties are considered, here, as the program specification. We are interested in testing the code against these properties. Testing a property is split in two stages. First, the property is cut out in several elementary properties. An elementary property is a tuple composed of some pre-conditions and a conclusion. Lastly, each elementary property is tested separately. The pre-conditions are used to generate and select the test cases randomly. The conclusion allows us to compute the verdict. All the testing process is done automatically. 1

We propose a method for improving confidence in the correctness of Haskell programs by combining testing and proving. Testing is used for debugging programs and specification before a costly proof attempt. During a proof development, testing also quickly eliminates wrong conjectures. Proving helps us to decompose a testing task in a way that is guaranteed to be correct. To demonstrate the method we have extended the Agda/Alfa proof assistant for dependent type theory with a tool for random testing. As an example we show how the correctness of a BDD-algorithm written in Haskell is verified by testing properties of component functions. We also discuss faithful translations from Haskell to type theory.

Abstract. Developing dependently typed functional programs can be difficult because the user may be required to write proofs and program errors are often hard to identify and fix. We describe a framework, implemented in Coq, that combines testing with user-assisted proof automation to make development easier. Testing occurs within Coq and is used to give user feedback to program errors and faulty conjectures, as well as guiding automated proof search. Dependently typed functional programming languages, such as Epigram [1] and ATS [2], offer an approach for developing verified software. These languages use dependent types to assign more accurate typing to terms, compared to simple typing, thereby enabling program properties to be verified at compile-time. However, programming with dependent types can be difficult. The user can be expected to write proofs for the proof obligations that arise and program errors can be hard to identify and fix. In this paper, we describe techniques that use a combination of testing and user-assisted proof automation for making dependently typed functional programming easier. These techniques are generic enough to support user-defined types and functions. We have implemented our ideas in the Coq theorem prover [3]. Coq can be used as a dependently typed programming language. The contributions of this paper are: – A description of how to provide useful counterexample-based program error feedback (see Section 2). – A description of the important role of testing in our user-assisted proof automation (see Section 3). – A small-scale usability study examining the utility of our counterexamplebased program error feedback (see Section 4). 1

Abstract. Program testing implies selecting test data from the program input space. In many cases, test data satisfying user-specified properties or preconditions (a.k.a. positive test data) are required. However, current automatic test data generation techniques adopt direct generate-andtest approaches for this task. In FocalTest, the testing tool of the Focal correct-by-construction environment for developing certified objectoriented functional programs, test data are generated at random and rejected when they do not satisfy preconditions. In this paper, we improve FocalTest with a test-and-generate approach, through the usage of constraint reasoning. A particular difficulty is the handling of function calls in the preconditions as they require constraint reasoning on conditionals and pattern matching which introduce disjunctions in constraint systems. Our experimental results show that a non-naive implementation of constraint reasoning on these constructions outperform traditional generation techniques when used to find test data for testing properties. 1

Abstract. Program verification systems based on automated theorem provers rely on user-provided axioms in order to verify domain-specific properties of code. However, formulating axioms correctly (that is, formalizing properties of an intended mathematical interpretation) is nontrivial in practice, and avoiding or even detecting unsoundness can sometimes be difficult to achieve. Moreover, speculating soundness of axioms based on the output of the provers themselves is not easy since they do not typically give counterexamples. We adopt the idea of model-based testing to aid axiom authors in discovering errors in axiomatizations. To test the validity of axioms, users define a computational model of the axiomatized logic by giving interpretations to the function symbols and constants in a simple declarative programming language. We have developed an axiom testing framework that helps automate model definition and test generation using off-the-shelf tools for meta-programming, property-based random testing, and constraint solving. We have experimented with our tool to test the axioms used in AutoCert, a program verification system that has been applied to verify aerospace flight code using a first-order axiomatization of navigational concepts, and were able to find counterexamples for a number of axioms. Key words: model-based testing, program verification, automated theorem proving, property-based testing, constraint solving

Abstract. Isabelle/HOL is a popular interactive theorem prover based on higherorder logic. It owes its success to its ease of use and powerful automation. Much of the automation is performed by external tools: The metaprover Sledgehammer relies on resolution provers and SMT solvers for its proof search, the counterexample generator Quickcheck uses the ML compiler as a fast evaluator for ground formulas, and its rival Nitpick is based on the model finder Kodkod, which performs a reduction to SAT. Together with the Isar structured proof format and a new asynchronous user interface, these tools have radically transformed the Isabelle user experience. This paper provides an overview of the main automatic proof and disproof tools. 1

We present a novel counterexample generator for the interactive theorem prover Isabelle based on a compiler that synthesizes test data generators for functional programming languages (e.g. Standard ML, OCaml) from specifications in Isabelle. In contrast to naive type-based test data generators, the smart generators take the preconditions into account and only generate tests that fulfill the preconditions. The smart generators are constructed by a compiler that reformulates the preconditions as logic programs and analyzes them by an enriched mode inference. From this inference, the compiler can construct the desired generators in the functional programming language. These test data generators are applied to find errors in specifications, as we show in a case study of a hotel key card system.

Abstract. The new Quickcheck is a counterexample generator for Isabelle/HOL that uncovers faulty specifications and invalid conjectures using various testing strategies. The previous Quickcheck only tested conjectures by random testing. The new Quickcheck extends the previous one and integrates two novel testing strategies: exhaustive testing with concrete values; and symbolic testing, evaluating conjectures with a narrowing strategy. Orthogonally to the strategies, we address two general issues: First, we extend the class of executable conjectures and specifications, and second, we present techniques to deal with conditional conjectures, i.e., conjectures with restrictive premises. We evaluate the testing strategies and techniques on a number of specifications, functional data structures and a hotel key card system. 1