Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in

White-box testing allows developers to determine whether or not a program is partially consistent with its specified behavior and design through the examination of intermediate values of variables during program execution. These intermediate values are often recorded as an execution trace produced by monitoring code inserted into the program. After program execution, the values in an execution trace are compared to values predicted by the specified behavior and design. Inconsistencies between predicted and actual values can lead to the discovery of errors in the specification and its implementation. This paper describes an approach to (1) verify the execution traces created by monitoring statements during white-box testing using a model checker as a semantic tableau; (2) organize multiple execution traces into distinct equivalence partitions based on requirements specifications written in linear temporal logic (LTL); and (3) use the counter-example generation mechanisms found in most model-checker tools to generate new testcases for unpopulated equivalence partitions.

We describe an methodology for testing a software system for possible security flaws. Based on the observation that most security flaws are caused by the program's inappropriate interactions with the environment, and triggered by user's malicious perturbation on the environment (which we call an environment fault), we view the security testing problem as the problem of testing for the fault-tolerance properties of a software system. We consider each environment perturbation as a fault and the resulting security compromise a failure in the toleration of such faults. Our approach is based on the well known technique of fault-injection. Environment faults are injected into the system under test and system behavior observed. The failure to tolerate faults is an indicator of a potential security flaw in the system. An Environment-Application Interaction (EAI) fault model is proposed which guides us to decide what faults to inject. Based on EAI, we have developed a security testing methodology, and apply it to several applications. We successfully identified a number of vulnerabilities include vulnerabilities in Windows NT operating system.

Random testing is a well known concept that requires that each test is selected randomly regardless of the test previously applied. This paper introduces the concept of antirandom testing. In this testing strategy each test applied is chosen such that its total distance from all previous tests is maximum. Two distance measures are defined. Procedures to construct antirandom sequences are developed. A checkpoint encoding scheme is introduced that allows automatic generation of efficient test cases. Further developments and studies needed are identified. 1 Introduction Exhaustive testing of software is infeasible except for very small programs [1, 2]. Achieving 100% test coverage using a specific measure does not assure that all defects have been found [4, 5]. Obtaining total coverage itself may be hard; 85% branch coverage is often used as a target. The testing time, often a significant fraction of the overall development time, is always limited. The testers thus have the challenging t...

Testing of database applications is crucial for ensuring high software quality as undetected faults can result in unrecoverable data corruption. The problem of database application testing can be broadly partitioned into the problems of test cases generation, test data preparation and test outcomes verification. Among the three problems, the problem of test cases generation directly affects the effectiveness of testing. Conventionally, database application testing is based upon whether or not the application can perform a set of predefined functions. While it is useful to achieve a basic degree of quality by considering the application to be a black box in the testing process, white box testing is required for more thorough testing. However, the semantics of the Structural Query Language (SQL) statements embedded in database applications are rarely considered in conventional white box testing techniques. In this paper, we propose to complement white box techniques with the inclusion of the SQL semantics. Our approach is to transform the embedded SQL statements to procedures in some general-purpose programming language and thereby generate test cases using conventional white box testing techniques. Additional test cases that are not covered in traditional white box testing are generated to improve the effectiveness of database application testing. The steps of both SQL statements transformation and test cases generation will be explained and illustrated using an example adapted from a course registration system. We successfully identify additional faults involving the internal states of databases. 1

We describe an approach for testing a software system for possible security flaws. Traditionally, security testing is done using penetration analysis and formal methods. Based on the observation that most security flaws are triggered due to a flawed interaction with the environment, we view the security testing problem as the problem of testing for the fault-tolerance properties of a software system. We consider each environment perturbation as a fault and the resulting security compromise a failure in the toleration of such faults. Our approach is based on the well known technique of fault-injection. Environment faults are injected into the system under test and system behavior observed. The failure to tolerate faults is an indicator of a potential security flaw in the system. An Environment-Application Interaction (EAI) fault model is proposed. EAI allows us to decide what faults to inject. Based on EAI, we present a security-flaw classification scheme. This scheme was used to classify 142 security flaws in a vulnerability database. This classification revealed that 91 % of the security flaws in the database are covered by the EAI model.

When migrating from conventional to object-oriented programming, developers face di cult decisions in modifying their development process to best use the new technology. In particular, ensuring that the software is highly reliable in this new environment poses different challenges and developers need to understand effective ways to test the software. Much previous work in testing OO software has focused on developing new techniques and procedures. We ask whether existing techniques can work, and present empirical data that show that the existing technique of category-partition testing can effectively find faults in object-oriented software, and new techniques may not be needed. For this study, we identified types of faults that are common to C++ software and inserted faults of these types into two C++ programs. Test cases generated using the category-partition method were used to test the programs. A fault was considered detected if it caused the program to terminate abnormally or if the output was different from the output of the original program. The results show that the combination of the category-partition method and a

The implementation of an efficient automatic test generation scheme for black-box testing environment is discussed. It uses checkpoint encoding and antirandom testing schemes. Checkpoint encoding converts test generation to a binary problem. The checkpoints are selected to probe the input space such that boundary and illegal cases are generated in addition to valid cases. Antirandom testing selects each test case such that it is as different as possible from all the previous tests. The implementation is illustrated using benchmark examples that have been used in the literature. Use of random testing both with checkpoint encoding and without is also reported. Comparison and evaluation of the effectiveness of these methods is also presented. Implications of the observations for larger software systems are noted. Overall, antirandom testing has higher code coverage than encoding random testing, encoding random testing has higher code coverage than pure random testing. Keywords: antirando...

This paper describes a method for automatically generating (and re-generating) test oracles during software development using the counter-example generation mechanism found in most model checker tools. Given a state-based specification of a system, our method helps organize test cases into a complete cover of disjoint equivalence partitions on a test space. These partitions are comprised of paths in the test space that conform to specified requirements written in linear temporal logic (LTL) formulae or quantified regular expressions (QRE). The oracles can also be used to drive test executions in cases where the test environment must generate events and conditions in order to force particular behaviors in nondeterministic systems. Keywords specification-base testing, formal methods, model checking INTRODUCTION Software developers often use models to reason about the design of their systems, but keeping the models and source code in fidelity during development is a difficult task [1]...

The past 20 years have seen the formulation of numerous analytical software reliability models for estimating the reliability growth of a software product. The predictions obtained by applying these models tend to be optimistic due to the inaccuracies in the operational profile, and saturation effect of testing. Incorporating knowledge gained about some structural attribute of the code such as test coverage, into the time-domain models can help alleviate this optimistic trend. In this paper we present an Enhanced non-homogeneous Poisson process (ENHPP) model which incorporates explicitly the time-varying test-coverage function in its analytical formulation, and provides for defective fault detection and test coverage during the testing and operational phases. It also allows for a time varying fault detection rate. The ENHPP model offers a unifying framework for all the previously reported finite failure NHPP models via test coverage. We also propose the log-logistic coverage function which can capture an increasing=decreasing failure detection rate per fault, which cannot be accounted for by the previously reported finite failure NHPP models. We present a methodology based on the ENHPP model for reliability prediction earlier in the testing phase. Expressions for predictions in the operational phase of the software, software availability, and optimal software release times subject to various constraints such as cost, reliability, and availability are developed based on the ENHPP model. We also validate the ENHPP model based on four different coverage functions using five failure data sets.