Results 1  10
of
25
A Survey of Combinatorial Testing
, 2011
"... Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work ..."
Abstract

Cited by 45 (1 self)
 Add to MetaCart
Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.
Combinatorial aspects of covering arrays
 LE MATEMATICHE (CATANIA
, 2004
"... Covering arrays generalize orthogonal arrays by requiring that ttuples be covered, but not requiring that the appearance of ttuples be balanced. Their uses in screening experiments has found application in software testing, hardware testing, and a variety of fields in which interactions among fact ..."
Abstract

Cited by 34 (8 self)
 Add to MetaCart
(Show Context)
Covering arrays generalize orthogonal arrays by requiring that ttuples be covered, but not requiring that the appearance of ttuples be balanced. Their uses in screening experiments has found application in software testing, hardware testing, and a variety of fields in which interactions among factors are to be identified. Here a combinatorial view of covering arrays is adopted, encompassing basic bounds, direct constructions, recursive constructions, algorithmic methods, and applications.
A Framework of Greedy Methods for Constructing Interaction Test Suites
 Intl. Conf. on Software Engineering (ICSE’05
, 2005
"... Greedy algorithms for the construction of software interaction test suites are studied. A framework is developed to evaluate a large class of greedy methods that build suites one test at a time. Within this framework are many instantiations of greedy methods generalizing those in the literature. Gre ..."
Abstract

Cited by 33 (8 self)
 Add to MetaCart
(Show Context)
Greedy algorithms for the construction of software interaction test suites are studied. A framework is developed to evaluate a large class of greedy methods that build suites one test at a time. Within this framework are many instantiations of greedy methods generalizing those in the literature. Greedy algorithms are popular when the time for test suite construction is of paramount concern. We focus on the size of the test suite produced by each instantiation. Experiments are analyzed using statistical techniques to determine the importance of the implementation decisions within the framework. This framework provides a platform for optimizing the accuracy and speed of "onetestatatime" greedy methods.
Constraint Models for the Covering Test Problem
, 2006
"... Covering arrays can be applied to the testing of software, hardware and advanced materials, and to the effects of hormone interaction on gene expression. In this paper we develop constraint programming models of the problem of finding an optimal covering array. Our models exploit global constraints ..."
Abstract

Cited by 25 (1 self)
 Add to MetaCart
(Show Context)
Covering arrays can be applied to the testing of software, hardware and advanced materials, and to the effects of hormone interaction on gene expression. In this paper we develop constraint programming models of the problem of finding an optimal covering array. Our models exploit global constraints, multiple viewpoints and symmetrybreaking constraints. We show that compound variables, representing tuples of variables in our original model, allow the constraints of this problem to be represented more easily and hence propagate better. With our best integrated model, we are able to either prove the optimality of existing bounds or find new optimal solutions, for arrays of moderate size. Local search on a SATencoding of the model is able to find improved solutions and bounds for larger problems.
IPOG: A General Strategy for TWay Software Testing
"... Most existing work on tway testing has focused on 2way (or pairwise) testing, which aims to detect faults caused by interactions between any two parameters. However, faults can also be caused by interactions involving more than two parameters. In this paper, we generalize an existing strategy, cal ..."
Abstract

Cited by 21 (2 self)
 Add to MetaCart
(Show Context)
Most existing work on tway testing has focused on 2way (or pairwise) testing, which aims to detect faults caused by interactions between any two parameters. However, faults can also be caused by interactions involving more than two parameters. In this paper, we generalize an existing strategy, called InParameterOrder (IPO), from pairwise testing to tway testing. A major challenge of our generalization effort is dealing with the combinatorial growth in the number of combinations of parameter values. We describe a tway testing tool, called FireEye, and discuss design decisions that are made to enable an efficient implementation of the generalized IPO strategy. We also report several experiments that are designed to evaluate the effectiveness of FireEye. 1.
An Orchestrated Survey on Automated Software Test Case Generation
, 2013
"... Test case generation is among the most labourintensive tasks in software testing and also one that has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has also been one of the most active topics in the research on software testing for several decades, ..."
Abstract

Cited by 20 (2 self)
 Add to MetaCart
Test case generation is among the most labourintensive tasks in software testing and also one that has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has also been one of the most active topics in the research on software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in selfstanding sections. The techniques presented include: (a) structural testing using symbolic execution, (b) modelbased testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) searchbased testing. Each section is contributed by worldrenowned active researchers on the technique, and briefly covers the basic ideas underlying the technique, the current state of art, a discussion of the open research problems, and a perspective of the future development in the approach. As a whole, the paper aims at giving an introductory, uptodate and (relatively) short overview of research in automatic test case generation, while ensuring comprehensiveness and authoritativeness.
A densitybased greedy algorithm for higher strength covering arrays
, 2008
"... Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for tway interactions with t ≥ 3. This work extends an efficient densitybased algorithm for pairwise coverage to generate tway interaction ..."
Abstract

Cited by 11 (4 self)
 Add to MetaCart
(Show Context)
Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for tway interactions with t ≥ 3. This work extends an efficient densitybased algorithm for pairwise coverage to generate tway interaction test suites, and show that it guarantees a logarithmic upper bound on the size of the test suites as a function of the number of factors. To complement this theoretical guarantee, an implementation is outlined and some practical improvements are made. Computational comparisons with other published methods are reported. Many of the results improve upon those in the literature. However, limitations on the ability of onetestatatime algorithms are also identified.
Classification of Orthogonal Arrays by Integer Programming
 JOURNAL OF STATISTICAL PLANNING AND INFERENCE, 138, 8.
, 2006
"... The problem of classifying all isomorphism classes of OA(N, k, s, t)s is shown to be equivalent to finding all isomorphism classes of nonnegative integer solutions to a system of linear equations under the symmetry group of the system of equations. A branchandcut algorithm developed by Margot (20 ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
The problem of classifying all isomorphism classes of OA(N, k, s, t)s is shown to be equivalent to finding all isomorphism classes of nonnegative integer solutions to a system of linear equations under the symmetry group of the system of equations. A branchandcut algorithm developed by Margot (2002, 2003a, 2003b, 2005) for solving integer programming problems with large symmetry groups is used to find
Backtracking algorithms and search heuristics to generate test suites for combinatorial testing
 Proceeding of the 30th Annual International Computer Software and Applications Conference. IEEE Computer Society
, 2006
"... Combinatorial covering arrays have been used in several testing approaches. This paper first discusses some existing methods for finding such arrays. Then a SATbased approach and a backtracking search algorithm are presented to solve the problem. A novel pruning strategy called SCEH is proposed ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
(Show Context)
Combinatorial covering arrays have been used in several testing approaches. This paper first discusses some existing methods for finding such arrays. Then a SATbased approach and a backtracking search algorithm are presented to solve the problem. A novel pruning strategy called SCEH is proposed to increase the efficiency of the methods. Several existing search heuristics and symmetry breaking techniques are also used in the backtracking search algorithm. Lastly, this paper introduces a tool called EXACT (EXhaustive seArch of Combinatorial Test suites) which implements all the above techniques to construct the covering arrays automatically. The experimental results show that our backtracking search method outperforms other methods in many small size cases. 1
Exploiting Constraint Solving History to Construct Interaction Test Suites
, 2007
"... Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highlyconfigurable software systems is complicated by the fact that, in many such systems, there are const ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highlyconfigurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. In recent work, automated constraint solving methods have been combined with searchbased CIT methods to address this problem with promising results. In this paper, we observe that the pattern of computation in greedy CIT algorithms leads to sequences of constraint solving problems that are closely related to one another. We propose two techniques for exploiting the history of constraint solving: (1) using incremental algorithms that are present within available constraint solvers and (2) mining constraint solver data structures to extract information that can be used to reduce the CIT search space. We evaluate the costeffectiveness of these reductions on four realworld highlyconfigurable software systems and on a population of synthetic examples that share the characteristics of those four systems. In combination our techniques reduce the cost of CIT in the presence of constraints to that of traditional unconstrained CIT methods without sacrificing the quality of solutions.