Results 1 - 10
of
99
Coverage and adequacy in software product line testing
- In Proc. of the ISSTA 2006 workshop on Role of
, 2006
"... ABSTRACT Software product line modeling has received a great deal of attention for its potential in fostering reuse of software artifacts across development phases. Research on the testing phase, has focused on identifying the potential for reuse of test cases across product line instances. While t ..."
Abstract
-
Cited by 47 (4 self)
- Add to MetaCart
ABSTRACT Software product line modeling has received a great deal of attention for its potential in fostering reuse of software artifacts across development phases. Research on the testing phase, has focused on identifying the potential for reuse of test cases across product line instances. While this offers potential reductions in test development effort for a given product line instance, it does not focus on and leverage the fundamental abstraction that is inherent in software product lines -variability. In this paper, we illustrate how rich software product line modeling notations can be mapped onto an underlying relational model that captures variability in the feasible product line instances. This relational model serves as the semantic basis for defining a family of coverage criteria for testing of a product line. These criteria make it possible to accumulate test coverage information for the product line itself over the course of multiple product line instance development efforts. Cumulative coverage, in turn, enables targeted testing efforts for new product line instances. We describe how combinatorial interaction testing methods can be applied to define test configurations that achieve a desired level of coverage and identify challenges to scaling such methods to large, complex software product lines.
A Survey of Combinatorial Testing
, 2011
"... Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work ..."
Abstract
-
Cited by 47 (1 self)
- Add to MetaCart
Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.
Combinatorial aspects of covering arrays
- LE MATEMATICHE (CATANIA
, 2004
"... Covering arrays generalize orthogonal arrays by requiring that t-tuples be covered, but not requiring that the appearance of t-tuples be balanced. Their uses in screening experiments has found application in software testing, hardware testing, and a variety of fields in which interactions among fact ..."
Abstract
-
Cited by 35 (9 self)
- Add to MetaCart
Covering arrays generalize orthogonal arrays by requiring that t-tuples be covered, but not requiring that the appearance of t-tuples be balanced. Their uses in screening experiments has found application in software testing, hardware testing, and a variety of fields in which interactions among factors are to be identified. Here a combinatorial view of covering arrays is adopted, encompassing basic bounds, direct constructions, recursive constructions, algorithmic methods, and applications.
Pairwise Testing in Real World Practical Extensions to Test Case Generators
"... Pairwise testing has become an indispensable tool in a software tester’s toolbox. The technique has been known for almost twenty years [22] but it is the last five years that we have seen a tremendous increase in its popularity. Information on at least 20 tools that can generate pairwise test cases, ..."
Abstract
-
Cited by 24 (0 self)
- Add to MetaCart
(Show Context)
Pairwise testing has become an indispensable tool in a software tester’s toolbox. The technique has been known for almost twenty years [22] but it is the last five years that we have seen a tremendous increase in its popularity. Information on at least 20 tools that can generate pairwise test cases, have so far been published [1]. Most tools, however, lack practical features necessary for them to be used in industry. This paper pays special attention to usability of the pairwise testing technique. In particular, it does not describe any radically new method of efficient generation of pairwise test suites, a topic that has already been researched extensively, neither does it refer to any specific case studies or results obtained through this method of test case generation. It does focus on ways in which pure pairwise testing approach needs to be modified to become practically applicable and on features tools need to offer to support a tester trying to use pairwise in practice. The paper makes frequent references to PICT, an existing and publicly available tool built on top of a flexible combinatorial test case generation engine, which implements several of the concepts described herein.
Reference-driven performance anomaly identification
- In ACM SIGMETRICS
, 2009
"... Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions—those of similar execution conditions from the target—to help identify the symptoms and causes of performance anomalies. Fir ..."
Abstract
-
Cited by 24 (4 self)
- Add to MetaCart
(Show Context)
Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions—those of similar execution conditions from the target—to help identify the symptoms and causes of performance anomalies. First, to identify anomaly symptoms, we construct change profiles that probabilistically characterize expected performance deviations between target and reference executions. By synthesizing several single-parameter change profiles, we can scalably identify anomalous reference-totarget changes in a complex system with multiple execution parameters. Second, to narrow the scope of anomaly root cause analysis, we filter anomaly-related low-level system metrics as those that manifest very differently between target and reference executions. Our anomaly identification approach requires little expert knowledge or detailed models on system internals and consequently it can be easily deployed. Using empirical case studies on the Linux I/O subsystem and a J2EE-based distributed online service, we demonstrate our approach’s effectiveness in identifying performance anomalies over a wide range of execution conditions as well as multiple system software versions. In particular, we discovered five previously unknown performance anomaly causes in the Linux 2.6.23 kernel. Additionally, our preliminary results suggest that online anomaly detection and system reconfiguration may help evade performance anomalies in complex online systems.
Developing a Single Model and Test Prioritization Strategies for Event-Driven Software
"... Abstract—Event-Driven Software (EDS) can change state based on incoming events; common examples are GUI and web applications. These EDS pose a challenge to testing because there are a large number of possible event sequences that users can invoke through a user interface. While valuable contribution ..."
Abstract
-
Cited by 17 (2 self)
- Add to MetaCart
(Show Context)
Abstract—Event-Driven Software (EDS) can change state based on incoming events; common examples are GUI and web applications. These EDS pose a challenge to testing because there are a large number of possible event sequences that users can invoke through a user interface. While valuable contributions have been made for testing these two subclasses of EDS, such efforts have been disjoint. This work provides the first single model that is generic enough to study GUI and web applications together. In this paper we use the model to define generic prioritization criteria that are applicable to both GUI and web applications. Our ultimate goal is to evolve the model and use it to develop a unified theory of how all EDS should be tested. An empirical study reveals that the GUI- and web-based applications, when recast using the new model, show similar behavior. For example, a criterion that gives priority all pairs of event interactions did well for GUI and web applications; another criterion that gives priority to the smallest number of parameter value settings did poorly for both. These results reinforce our belief that these two subclasses of applications should be modeled and studied together. Index Terms—Combinatorial interaction testing, covering arrays, event driven software (EDS), t-way interaction coverage, test suite prioritization, user-session testing, web-application testing, GUI testing 1
A density-based greedy algorithm for higher strength covering arrays
, 2008
"... Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for t-way interactions with t ≥ 3. This work extends an efficient density-based algorithm for pairwise coverage to generate t-way interaction ..."
Abstract
-
Cited by 13 (4 self)
- Add to MetaCart
(Show Context)
Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for t-way interactions with t ≥ 3. This work extends an efficient density-based algorithm for pairwise coverage to generate t-way interaction test suites, and show that it guarantees a logarithmic upper bound on the size of the test suites as a function of the number of factors. To complement this theoretical guarantee, an implementation is outlined and some practical improvements are made. Computational comparisons with other published methods are reported. Many of the results improve upon those in the literature. However, limitations on the ability of one-test-at-a-time algorithms are also identified.
An Evaluation of Combination Strategies for Test Case Selection
- Empirical Software Engineering, Available in SpringerLink electronic Online First version
, 2003
"... This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting ” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination str ..."
Abstract
-
Cited by 9 (4 self)
- Add to MetaCart
(Show Context)
This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting ” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard ” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.
On the Testing Maturity of Software Producing Organizations
"... This paper presents data from a study of the current state of practice of software testing. Test managers from twelve different software organizations were interviewed. The interviews focused on the amount of resources spent on testing, how the testing is conducted, and the knowledge of the personne ..."
Abstract
-
Cited by 9 (1 self)
- Add to MetaCart
This paper presents data from a study of the current state of practice of software testing. Test managers from twelve different software organizations were interviewed. The interviews focused on the amount of resources spent on testing, how the testing is conducted, and the knowledge of the personnel in the test organizations. The data indicate that the overall test maturity is low. Test managers are aware of this but have trouble improving. One problem is that the organizations are commercially successful, suggesting that products must already be “good enough.” Also, the current lack of structured testing in practice makes it difficult to quantify the current level of maturity and thereby articulate the potential gain from increasing testing maturity to upper management and developers.
