Results 1 - 10
of
27
On the danger of coverage directed test case generation
- In 15th Int’l Conf. on Fundamental Approaches to Software Engineering (FASE
, 2012
"... Abstract. In the avionics domain, the use of structural coverage criteria is legally required in determining test suite adequacy. With the success of automated test generation tools, it is tempting to use these criteria as the basis for test generation. To more firmly establish the effectiveness of ..."
Abstract
-
Cited by 19 (6 self)
- Add to MetaCart
(Show Context)
Abstract. In the avionics domain, the use of structural coverage criteria is legally required in determining test suite adequacy. With the success of automated test generation tools, it is tempting to use these criteria as the basis for test generation. To more firmly establish the effectiveness of such approaches, we have generated and evaluated test suites to satisfy two coverage criteria using counterexample-based test generation and a random generation approach, contrasted against purely random test suites of equal size. Our results yield two key conclusions. First, coverage criteria satisfaction alone is a poor indication of test suite effectiveness. Second, the use of structural coverage as a supplement—not a target—for test generation can have a positive impact. These observations points to the dangers inherent in the increase in test automation in critical systems and the need for more research in how coverage criteria, generation approach, and system structure jointly influence test effectiveness. 1
Safety and Software Intensive Systems: Challenges Old and New
, 2007
"... There is an increased use of software in safety-critical systems; a trend that is likely to continue in the future. Although traditional system safety techniques are applicable to software intensive systems, there are new challenges emerging. In this report we will address four issues we believe wil ..."
Abstract
-
Cited by 13 (0 self)
- Add to MetaCart
There is an increased use of software in safety-critical systems; a trend that is likely to continue in the future. Although traditional system safety techniques are applicable to software intensive systems, there are new challenges emerging. In this report we will address four issues we believe will pose challenges in the future. First, the nature of safety is continuing to be widely misunderstood and known system safety techniques are not applied. Second, our ability to demonstrate (certify) that safety requirements have been met is inadequate. Third, modeling and automated tools, for example, code generation and automated testing, are introduced in a hope to increase productivity; this reliance on tools rather than people, however, introduces new and poorly understood problems. Finally, safety-critical systems are increasingly relying on data (configuration data or databases), incorrect data could have catastrophic and widespread consequences.
Property relevant software testing with model-checkers
- SIGSOFT Software Engineering Notes
"... Verification is applied to software as a proof method with respect to its requirements. Software testing is necessary due to the fact that verification is often infeasible. Automation is desirable since the complexity and the effort involved are significant. However, auto-mated software testing is c ..."
Abstract
-
Cited by 9 (5 self)
- Add to MetaCart
(Show Context)
Verification is applied to software as a proof method with respect to its requirements. Software testing is necessary due to the fact that verification is often infeasible. Automation is desirable since the complexity and the effort involved are significant. However, auto-mated software testing is commonly used to ensure confidence in the conformance of an implementation to an abstract model, not to its requirement properties. In this paper, we introduce the notion of property relevance of test-cases. Property relevant test-cases can be used to determine property violations. It is shown how to detect the properties relevant to a test-case. New coverage criteria based on property relevance are introduced. Automated generation of test-suites satisfying these criteria is also presented. Finally, feasibility is illustrated with an empirical evaluation.
Reducing the costs of bounded-exhaustive testing
- In FASE
, 2009
"... Abstract. Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the c ..."
Abstract
-
Cited by 8 (3 self)
- Add to MetaCart
(Show Context)
Abstract. Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of boundedexhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability. 1
Redundancy based test-suite reduction
- of Lecture Notes in Computer Science
, 2007
"... Abstract. The size of a test-suite has a direct impact on the costs and the effort of software testing. Especially during regression testing, when software is re-tested after some modifications, the size of the test-suite is important. Common test-suite reduction techniques select subsets of test-su ..."
Abstract
-
Cited by 8 (2 self)
- Add to MetaCart
(Show Context)
Abstract. The size of a test-suite has a direct impact on the costs and the effort of software testing. Especially during regression testing, when software is re-tested after some modifications, the size of the test-suite is important. Common test-suite reduction techniques select subsets of test-suites that achieve given test requirements. Unfortunately, not only the test-suite size but also the fault detection ability is reduced as a consequence. This paper proposes a novel approach where test-cases created with model-checker based techniques are transformed such that redundancy within the test-suite is avoided, and the overall size is reduced. As test-cases are not simply discarded, the impact on the fault sensitivity is minimal. 1
Using and improving requirement properties for mutation based test-case generation. 22
- WI-MAW Rundbrief, Jahrgang
"... Automated test-case generation is expected to significantly improve software development, where a considerable part of the effort involved is devoted to testing. Many solutions have been proposed, differing on the methods and tools involved. Approaches based on the utilization of model-checkers have ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Automated test-case generation is expected to significantly improve software development, where a considerable part of the effort involved is devoted to testing. Many solutions have been proposed, differing on the methods and tools involved. Approaches based on the utilization of model-checkers have been shown to be very promising. This paper presents such an approach, incorporating ideas of mutation testing. The resulting test-cases can be directly mapped to requirement properties. We show that this approach can also be used to identify incompleteness and potential errors in the re-quirements specification. Empirical results are provided to demonstrate the feasibility of this approach.
On the Order of Test Goals in Specification-Based Testing
, 2008
"... Model-based testing techniques often select test cases according to test goals such as coverage criteria or mutation adequacy. Complex criteria and large models lead to large test suites, and a test case created for one coverage item usually covers several other items as well. This can be problemati ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
Model-based testing techniques often select test cases according to test goals such as coverage criteria or mutation adequacy. Complex criteria and large models lead to large test suites, and a test case created for one coverage item usually covers several other items as well. This can be problematic if testing is expensive and resources are limited. Therefore, test case generation can be optimized in order to avoid unnecessary test cases and minimize the test generation and execution costs. Because of this optimization the order in which test goals are selected is expected to have an impact on both the performance of the test case generation and the size of resulting test suites, although finding the optimal order is not feasible in general. In this paper we report on experiments to determine the effects of the order in which test goals are selected on performance and the size of resulting test suites, and evaluate different heuristics to select test goals such that the time required to generate test suites as well as their size are minimized. The test case generation approach used for experimentation uses model checkers, and experimentation shows that good results can be achieved with any random ordering, but some improvement is still possible with simple heuristics.
1A Uniform Representation of Hybrid Criteria for Regression Testing
"... Abstract—Regression testing tasks of test case prioritization, test suite reduction/minimization, and regression test selection are typically centered around criteria that are based on code coverage, test execution costs, and code modifications. Researchers have developed and evaluated new individua ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
Abstract—Regression testing tasks of test case prioritization, test suite reduction/minimization, and regression test selection are typically centered around criteria that are based on code coverage, test execution costs, and code modifications. Researchers have developed and evaluated new individual criteria; others have combined existing criteria in different ways to form what we—and some others—call hybrid criteria. In this paper, we formalize the notion of combining multiple criteria into a hybrid. Our goal is to create a uniform representation of such combinations so that they can be described unambiguously and shared among researchers. We envision that such sharing will allow researchers to implement, study, extend, and evaluate the hybrids using a common set of techniques and tools. We precisely formulate three hybrid combinations, Rank, Merge, and Choice, and demonstrate their usefulness in two ways. First, we recast, in terms of our formulations, others ’ previously reported work on hybrid criteria. Second, we use our previous results on test case prioritization to create and evaluate new hybrid criteria. Our findings suggest that hybrid criteria of others can be described using our Merge and Rank formulations, and that the hybrid criteria we developed most often outperformed their constituent individual criteria. Index Terms—Test case prioritization, test criteria, hybrid test criteria, web testing, GUI testing 1
Improving the effectiveness of test suite reduction for user-session-based testing of web applications
- Information and Software Technology, 54(7):724 – 738
, 2012
"... a b s t r a c t Context: Test suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a reduced test suite can be time c ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
a b s t r a c t Context: Test suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a reduced test suite can be time consuming. Objective: We propose ordering the tests in a reduced suite to increase its rate of fault detection. The ordered reduced test suite can be executed in time constrained situations, where, even if test execution is stopped early, the best test cases from the reduced suite will already be executed. Method: In this paper, we present several approaches to order reduced test suites using experimentally verified prioritization criteria for the domain of web applications. We conduct an empirical study with three subject applications and user-session-based test cases to demonstrate how ordered reduced test suites often make a practical contribution. To enable comparison between test suites of different sizes, we develop Mod_APFD_C, a modification of the traditional prioritization effectiveness measure. Results: We find that by ordering the reduced suites, we create test suites that are more effective than unordered reduced suites. In each of our subject applications, there is at least one ordered reduced suite that outperforms the best unordered reduced suite and the best prioritized original suite. Conclusions: Our results show that when a tester does not have enough time to execute the entire reduced suite, executing an ordered reduced suite often improves the rate of fault detection. By coupling the underlying system's characteristics with observations from our study on the criteria that produce the best ordered reduced suites, a tester can order their reduced test suites to obtain increased testing effectiveness.
