Results 1 - 10
of
24
Bridging the Gap between the Total and Additional Test-Case Prioritization Strategies
"... Abstract—In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total num ..."
Abstract
-
Cited by 19 (8 self)
- Add to MetaCart
(Show Context)
Abstract—In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total numbers of elements covered per test, and numbers of additional (not-yet-covered) elements covered per test, are two widely-adopted generic strategies used for such prioritization. This paper proposes a basic model and an extended model that unify the total strategy and the additional strategy. Our models yield a spectrum of generic strategies ranging between the total and additional strategies, depending on a parameter referred to as the p value. We also propose four heuristics to obtain differentiated p values for different methods under test. We performed an empirical study on 19 versions of four Java programs to explore our results. Our results demonstrate that wide ranges of strategies in our basic and extended models with uniform p values can significantly outperform both the total and additional strategies. In addition, our results also demonstrate that using differentiated p values for both the basic and extended models with method coverage can even outperform the additional strategy using statement coverage. I.
An Empirical Study of JUnit Test-Suite Reduction
- 22ND IEEE INTERNATIONAL SYMPOSIUM ON SOFTWARE RELIABILITY ENGINEERING
, 2011
"... As test suites grow larger during software evolution, regression testing becomes expensive. To reduce the cost of regression testing, test-suite reduction aims to select a minimal subset of the original test suite that can still satisfy all the test requirements. While traditional test-suite reduct ..."
Abstract
-
Cited by 8 (6 self)
- Add to MetaCart
As test suites grow larger during software evolution, regression testing becomes expensive. To reduce the cost of regression testing, test-suite reduction aims to select a minimal subset of the original test suite that can still satisfy all the test requirements. While traditional test-suite reduction techniques were intensively studied on C programs with specially generated test suites, there are limited studies for test-suite reduction on programs with real-world test suites. In this paper, we investigate test-suite reduction techniques on Java programs with real-world JUnit test suites. We implemented four representative test-suite reduction techniques for JUnit test suites. We performed an empirical study on 19 versions of four real-world Java programs, ranging from 1.89 KLoC to 80.44 KLoC. Our study investigates both the benefits and the costs of test-suite reduction. The results show that the four traditional test-suite reduction techniques can effectively reduce these JUnit test suites without substantially reducing their fault-detection capability. Based on the results, we provide a guideline for achieving cost-effective JUnit testsuite reduction.
Unit test virtualization with vmvm.
, 2013
"... ABSTRACT Testing large software packages can become very time intensive. To address this problem, researchers have investigated techniques such as Test Suite Minimization. Test Suite Minimization reduces the number of tests in a suite by removing tests that appear redundant, at the risk of a reduct ..."
Abstract
-
Cited by 5 (2 self)
- Add to MetaCart
(Show Context)
ABSTRACT Testing large software packages can become very time intensive. To address this problem, researchers have investigated techniques such as Test Suite Minimization. Test Suite Minimization reduces the number of tests in a suite by removing tests that appear redundant, at the risk of a reduction in fault-finding ability since it can be difficult to identify which tests are truly redundant. We take a completely different approach to solving the same problem of long running test suites by instead reducing the time needed to execute each test, an approach that we call Unit Test Virtualization. With Unit Test Virtualization, we reduce the overhead of isolating each unit test with a lightweight virtualization container. We describe the empirical analysis that grounds our approach and provide an implementation of Unit Test Virtualization targeting Java applications. We evaluated our implementation, VmVm, using 20 real-world Java applications and found that it reduces test suite execution time by up to 97% (on average, 62%) when compared to traditional unit test execution. We also compared VmVm to a well known Test Suite Minimization technique, finding the reduction provided by VmVm to be four times greater, while still executing every test with no loss of fault-finding ability.
Test Case Prioritization for Regression Testing based on Severity of Fault
"... Abstract — Regression testing is one of the most critical activities of software development and maintenance. Whenever software is modified, a set of test cases are run and the comparison of new outputs is done with the older one to avoid unwanted changes. If the new output and old output match it i ..."
Abstract
-
Cited by 3 (0 self)
- Add to MetaCart
(Show Context)
Abstract — Regression testing is one of the most critical activities of software development and maintenance. Whenever software is modified, a set of test cases are run and the comparison of new outputs is done with the older one to avoid unwanted changes. If the new output and old output match it implies that the modifications made in one part of the software don’t affect the remaining software. It is impractical to re-execute every test case for a program if changes occur. The problem of regression test case selection can be solved by prioritizing the test cases. Regression test prioritization techniques reorder the execution of a test suit in an attempt to ensure that faults are revealed at the earlier stage of the testing process. Test case prioritization techniques schedule test cases for execution so that those with higher priority, according to some criterion are executed earlier than those with lower priority to meet some performance goal. In this paper an algorithm is proposed to prioritize test cases based on rate of fault detection and fault impact. The proposed algorithm identifies the severe fault at earlier stage of the testing process and the effectiveness of prioritized test case and comparison of it with unprioritized ones with the help of APFD.
Balancing Trade-Offs in Test-Suite Reduction
"... Regression testing is an important activity but can get ex-pensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redun-dant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commo ..."
Abstract
-
Cited by 3 (1 self)
- Add to MetaCart
(Show Context)
Regression testing is an important activity but can get ex-pensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redun-dant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commonly shares three properties: (1) requirements are de-fined by a coverage criterion such as statement coverage; (2) the reduced test suite has to satisfy all the requirements as the original test suite; and (3) the quality of the reduced test suites is measured on the software version on which the reduction is performed. These properties make it hard for test engineers to decide how to use reduced test suites. We address all three properties of traditional test-suite reduction: (1) we evaluate test-suite reduction with require-ments defined by killed mutants; (2) we evaluate inadequate reduction that does not require reduced test suites to sat-isfy all the requirements; and (3) we propose evolution-aware metrics that evaluate the quality of the reduced test suites across multiple software versions. Our evaluations allow a more thorough exploration of trade-offs in test-suite reduc-tion, and our evolution-aware metrics show how the quality of reduced test suites can change after the version where the reduction is performed. We compare the trade-offs among various reductions on 18 projects with a total of 261,235 tests over 3,590 commits and a cumulative history spanning 35 years of development. Our results help test engineers make a more informed decision about balancing size, cover-age, and fault-detection loss of reduced test suites.
Reducing the cost of regression testing by identifying irreplaceable test cases
- in: Proceedings of the 6th International Conference on Genetic and Evolutionary Computing, IEEE Computer Society
, 2012
"... Abstract—Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to ..."
Abstract
-
Cited by 3 (0 self)
- Add to MetaCart
(Show Context)
Abstract—Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to decrease the size of the test suite. Yet, the difference in the execution cost of the tests is often significant and it may be costly to use a test suite consisting of a few long-running test cases. Thus, this paper proposes an algorithm, based on the concept of test irreplaceability, which creates a reduced test suite with a decreased execution cost. Leveraging widely used benchmark programs, the empirical study shows that, in comparison to existing techniques, the presented algorithm is the most effective at reducing the cost of running a test suite. Keywords-regression testing; test suite reduction; code coverage I.
Flower: Optimal test suite reduction as a network maximum flow.
- In ISSTA,
, 2014
"... ABSTRACT A trend in software testing is reducing the size of a test suite while preserving its overall quality. Given a test suite and a set of requirements covered by the suite, test suite reduction aims at selecting a subset of test cases that cover the same set of requirements. Even though this ..."
Abstract
-
Cited by 2 (0 self)
- Add to MetaCart
(Show Context)
ABSTRACT A trend in software testing is reducing the size of a test suite while preserving its overall quality. Given a test suite and a set of requirements covered by the suite, test suite reduction aims at selecting a subset of test cases that cover the same set of requirements. Even though this problem has received considerable attention, finding the smallest subset of test cases is still challenging and commonly-used approaches address this problem only with approximated solutions. When executing a single test case requires much manual effort (e.g., hours of preparation), finding the minimal subset is needed to reduce the testing costs. In this paper, we introduce a radically new approach to test suite reduction, called FLOWER, based on a search among network maximum flows. From a given test suite and the requirements covered by the suite, FLOWER forms a flow network (with specific constraints) that is then traversed to find its maximum flows. FLOWER leverages the Ford-Fulkerson method to compute maximum flows and Constraint Programming techniques to search among optimal flows. FLOWER is an exact method that computes a minimum-sized test suite, preserving the coverage of requirements. The experimental results show that FLOWER outperforms a non-optimized implementation of the Integer Linear Programming approach by 15 − 3000 times in terms of the time needed to find an optimal solution, and a simple greedy approach by 5 − 15% in terms of the size of reduced test suite.
An Optimal Technique for Reducing the Effort of Regression Test
"... AbstractRegression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset o ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
AbstractRegression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset of the test case from the existing test suite is an important problem in re-gression testing and is addressed in the regression test selection technique. Safe regression test selection technique selects and identifies the program parts that are affected by the change. The test selection is performed by matching the identified change information with the coverage information. A tool is implemented that reduces the testing effort efficiently and the result shows that it can achieve considerable savings in the regression testing time.
Regression Optimizer A Multi Coverage Criteria Test Suite Minimization Technique Saran Prasad
"... Regression test suites are developed and maintained throughout the lifetime of the software product. For testers, it is common practice to add new testcases to the existing regression test suite, with intent to test new features in the software product or to capture any newly discovered fault. Many ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Regression test suites are developed and maintained throughout the lifetime of the software product. For testers, it is common practice to add new testcases to the existing regression test suite, with intent to test new features in the software product or to capture any newly discovered fault. Many a times the intention is to check whether the program is sufficiently tested or not. This is done by measuring code coverage. In case if not, then additional tests are added until the test suite has achieved a specified coverage level according to a specific criterion. Due to this continuous addition of testcases, regression test suites tend to grow in size. As a result, multiple testcases may exist which may test the same feature or same set of requirements. Test Suite minimization techniques identify redundant test cases from a test suite based on some criterion. In this paper we propose a novel test suite minimization technique which identifies redundancy in a given test suite based on multiple coverage criteria for example function, function call stack, line and branch coverage of given test cases. Paper also talks about the benefits of our approach over other existing test suite minimization techniques. General Terms Regression test suite minimization technique.
