## An Empirical Evaluation (and Specification) of the All-du-paths Testing Criterion (Extended Version) (1992)

Venue: | Software Engineering Journal |

Citations: | 17 - 2 self |

### BibTeX

@ARTICLE{Bieman92anempirical,

author = {James M. Bieman and Janet L. Schultz},

title = {An Empirical Evaluation (and Specification) of the All-du-paths Testing Criterion (Extended Version)},

journal = {Software Engineering Journal},

year = {1992},

volume = {7},

pages = {43--51}

}

### OpenURL

### Abstract

The all-du-paths structural testing criterion is one of the most discriminating of the data flow testing criteria. Unfortunately, in the worst case, the criterion requires an intractable number of test cases. In a case study of an industrial software system, we find that the worst case scenario is rare. Eighty percent of the subroutines require ten or fewer test cases. Only one subroutine out of 143 requires an intractable number of tests. However, the number of required test cases becomes tractable when using the all-uses criterion. This paper includes a formal specification of both the all-du-paths criterion and the software tools used to estimate a minimal number of test cases necessary to meet the criterion. Keywords: Software testing, software measures formal specifications, data flow analysis. Department of Computer Science Technical Report #CS--91--105 Colorado State University (Revision of Technical Report #CS--89--118) A version of this paper was published in IEE/BCS Software Engineering Journal, 7(1):43-51, Jan. 1992. This research was partially supported by the NATO Collaborative Research Grants Program under RG. 0343/88. 1

### Citations

842 |
A Complexity Measure
- McCabe
- 1976
(Show Context)
Citation Context ...can be calculated from the flowgraph dimensions using the following formula: d = e \Gamma n where d is the number of decisions, e is the number of edges, and n is the number of nodes in the flowgraph =-=[30]-=-. The most striking finding is that in 115 of the 143 subroutines (80%) the all-du-paths criterion can be met with ten or fewer complete paths. And, in 91% of the subroutines, the all-du-paths criteri... |

618 |
The Art of Software Testing
- Myers
- 1979
(Show Context)
Citation Context ...at the test data cause all statements to be executed at least once. Myers describes statement coverage as a "weak criterion" and demonstrates the advantages of branch coverage over statement=-= coverage [1]. The "all branches"-=- criterion can uncover more errors than the "all statements" criterion and usually requires more tests. The most discriminating path-based criterion is the "all paths" criterion wh... |

284 | Selecting software test data using data flow information
- Rapps, Weyuker
- 1985
(Show Context)
Citation Context ...ventually, any practical and useful criterion must be met with a finite and acceptably small number of test cases. A number of published analytical studies compare various structural testing criteria =-=[2, 3, 4, 5, 6]-=-. These studies examine the inclusion orderings and worst case complexity of the criteria. Criterion A includes criterion B if and only if any test data and program that satisfies criterion A also sat... |

277 |
Flow Analysis of Computer Programs
- Hecht
- 1977
(Show Context)
Citation Context ...r starting with the first token and ending with the last token. Program unit data flow is represented as the sequence of definitions and uses in the nodes that represent basic blocks. Following Hecht =-=[27]-=-, a variable definition is source program code that, when executed, can (potentially) modify the value stored by a program variable; a variable use or reference is code that, when executed, references... |

182 |
Toward a theory of test data selection
- Goodenough, Gerhart
- 1975
(Show Context)
Citation Context ...infeasible. Thus, it might be impossible to select test data that satisfies the particular criterion. As are many problems in testing, determining whether a particular path is feasible is undecidable =-=[15, 16, 17]-=-. Frankl and Weyuker suggest the use of heuristics to identify infeasible paths [18]. Since some complete paths and du-paths may be infeasible, our measure of the necessary number of complete paths wi... |

179 |
An Applicable Family of Data Flow Testing Criteria
- Frankl, Weyuker
- 1988
(Show Context)
Citation Context ...riterion. As are many problems in testing, determining whether a particular path is feasible is undecidable [15, 16, 17]. Frankl and Weyuker suggest the use of heuristics to identify infeasible paths =-=[18]-=-. Since some complete paths and du-paths may be infeasible, our measure of the necessary number of complete paths will tend to be higher than the number of feasible paths. In developing our research t... |

130 |
Specification Case Studies
- Hayes
- 1987
(Show Context)
Citation Context ...hs Criterion The software tools used in this study are specified in the SPECS specification language [19]. SPECS is an abstract model specification language and is similar to VDM's meta-iv [25] and Z =-=[26]-=-. SPECS specifications are defined as abstract data types in terms of mathematical sets, sequences, labelled tuples, integers, booleans, and operations on these primitives. A SPECS specification consi... |

124 | A data flow oriented program testing strategy
- LASKI, KOREL
- 1983
(Show Context)
Citation Context ...,3,4i hs,1,2,3,5i hs,1,2,6i h1,2,3i h1,2,6i h3,4i h3,5i h4,2,3i h4,2,6i h5,2,3i h5,2,6i Table 2: Binary Search Definition/Use Paths testing criteria of Ntafos [8], and the criteria of Laski and Korel =-=[9] strictly -=-subsume the all-du-paths criterion with the modifications suggested by Clarke et al. [10]. Thus, these "stronger" criteria may require even more test cases than the all-du-paths criterion. A... |

102 |
Reliability of the Path Analysis Testing Strategy
- Howden
- 1976
(Show Context)
Citation Context ...infeasible. Thus, it might be impossible to select test data that satisfies the particular criterion. As are many problems in testing, determining whether a particular path is feasible is undecidable =-=[15, 16, 17]-=-. Frankl and Weyuker suggest the use of heuristics to identify infeasible paths [18]. Since some complete paths and du-paths may be infeasible, our measure of the necessary number of complete paths wi... |

80 |
AFormal Evaluation of Data Flow Path Selection Criteria
- Clarke, Podgruski, et al.
- 1989
(Show Context)
Citation Context ...inary Search Definition/Use Paths testing criteria of Ntafos [8], and the criteria of Laski and Korel [9] strictly subsume the all-du-paths criterion with the modifications suggested by Clarke et al. =-=[10]. Thus, th-=-ese "stronger" criteria may require even more test cases than the all-du-paths criterion. A testing strategy that requires an intractable number of test cases is not realistic. However, the ... |

76 |
A comparison of some structural testing strategies
- Ntafos
- 1988
(Show Context)
Citation Context ...,4,5], [4,5,8] and [5,8,9,10] are all included on [1,2,4,5,8,9,10], but [8,9,10,11,12] is not. When the du-paths for the binary search routine are condensed we get: conlist([[s,1,2,3,4], [s,1,2,3,5], =-=[s,1,2,6]-=-, [4,2,3], [4,2,6], [5,2,6], [5,2,3]]). 3.2.4 Counting complete paths The final step of the Prolog algorithm estimates the fewest number of complete paths that include all of the du-paths. The intuiti... |

73 |
Pascal User Manual and Report
- Jensen, Wirth
- 1974
(Show Context)
Citation Context ...e repeated by others. Note that the StandardRep can be used to specify a number of useful software structural measures. We utilize a tool [21] that generates a StandardRep from an ISO Standard Pascal =-=[22]-=- program. The output routines of the generator are modified to produce a Prolog data base for each program unit. Each data base represents an augmented flowgraph in which the nodes and edges are inclu... |

67 |
On required element testing
- NTAFOS
- 1984
(Show Context)
Citation Context ...Definitions, & Uses DU Paths hs,1i hs,1,2,3,4i hs,1,2,3,5i hs,1,2,6i h1,2,3i h1,2,6i h3,4i h3,5i h4,2,3i h4,2,6i h5,2,3i h5,2,6i Table 2: Binary Search Definition/Use Paths testing criteria of Ntafos =-=[8], and the -=-criteria of Laski and Korel [9] strictly subsume the all-du-paths criterion with the modifications suggested by Clarke et al. [10]. Thus, these "stronger" criteria may require even more test... |

33 |
A comparison of data flow path selection criteria
- Clarke, Podgurski, et al.
- 1985
(Show Context)
Citation Context ...utput file is accessed for the list of condensed du-paths found in Step 3 and the successor nodes found in Step 2. For example, we begin by looking for a du-path that starts with the start node s. If =-=[s,1,2,3,4,5]-=- is initially selected, the next du-path we look for in the list should begin with the initial sequence of [1,2,3,4,5,...]. (There cannot be another du-path that begins with [s,1,2,3,4,5,...] due to t... |

33 |
The cost of data flow testing: An empirical study
- Weyuker
- 1990
(Show Context)
Citation Context ...ti, hs,1,2,3,5,2,3,5,2,6,ti, and hs,1,2,6,ti. Only empirical studies can determine whether the worst case scenario is common. One empirical study of the du family of criteria was conducted by Weyuker =-=[11, 12]-=-. This study determined how many test cases were required to meet criteria using a suite of programs collected by Kernighan and Plauger [13]. The study was performed using the ASSET system, described ... |

22 | Selecting software test data using data ¯ow information - Rapps, Weyuker - 1985 |

18 |
An empirical study of the complexity of data flow testing
- Weyuker
- 1988
(Show Context)
Citation Context ...ti, hs,1,2,3,5,2,3,5,2,6,ti, and hs,1,2,6,ti. Only empirical studies can determine whether the worst case scenario is common. One empirical study of the du family of criteria was conducted by Weyuker =-=[11, 12]-=-. This study determined how many test cases were required to meet criteria using a suite of programs collected by Kernighan and Plauger [13]. The study was performed using the ASSET system, described ... |

18 | An applicable family of data ow testing criteria - Frankl, Weyuker - 1988 |

16 | Speci cation Case Studies - Hayes, editor - 1993 |

15 |
ASSET: A system to select and evaluate tests
- Frankl, Weiss, et al.
- 1985
(Show Context)
Citation Context ...and Weyuker, ASSET, determines whether a given test set is adequate with respect to the criterion, and produces a list of any node pairs required by 5 the criterion but not exercised by the test data =-=[23, 24]-=-. This list can then be used to strengthen the test data set. In contrast, our research tools are designed to estimate the number test cases required by a criterion. ASSET is designed for use on a spe... |

14 |
A data flow testing tool
- Frankl, Weyuker
- 1987
(Show Context)
Citation Context ...and Weyuker, ASSET, determines whether a given test set is adequate with respect to the criterion, and produces a list of any node pairs required by 5 the criterion but not exercised by the test data =-=[23, 24]-=-. This list can then be used to strengthen the test data set. In contrast, our research tools are designed to estimate the number test cases required by a criterion. ASSET is designed for use on a spe... |

12 |
The Complexity of Data Flow Criteria for Test Data Selection
- Weyuker
- 1984
(Show Context)
Citation Context ...utput file is accessed for the list of condensed du-paths found in Step 3 and the successor nodes found in Step 2. For example, we begin by looking for a du-path that starts with the start node s. If =-=[s,1,2,3,4,5]-=- is initially selected, the next du-path we look for in the list should begin with the initial sequence of [1,2,3,4,5,...]. (There cannot be another du-path that begins with [s,1,2,3,4,5,...] due to t... |

11 |
Comparison of structural test coverage metrics
- Weiser, Gannon, et al.
- 1985
(Show Context)
Citation Context ...ventually, any practical and useful criterion must be met with a finite and acceptably small number of test cases. A number of published analytical studies compare various structural testing criteria =-=[2, 3, 4, 5, 6]-=-. These studies examine the inclusion orderings and worst case complexity of the criteria. Criterion A includes criterion B if and only if any test data and program that satisfies criterion A also sat... |

10 |
How To Solve it by Computer
- Dromey
- 1982
(Show Context)
Citation Context ...based on data flow relationships [5]. These criteria focus on the program paths that connect the definitions and uses of variables (du-paths). Consider the Pascal binary search procedure from Dromley =-=[7]-=- shown in Figure 1. The binary search procedure has the flowgraph shown in Figure 2. For each basic block represented by flowgraph nodes, the variables defined and referenced and the program code are ... |

9 | A comparison of data ow path selection criteria - Clarke, Podgurski, et al. - 1985 |

5 | Estimating the number of test cases required to satisfy the alldu-paths testing criterion
- Bieman, Schultz
- 1989
(Show Context)
Citation Context ...tisfy the all-du-paths criterion. Our estimate is based on finding a minimal sized set of complete paths (paths from the start to terminal node of a program flowgraph) that covers all of the du-paths =-=[14]. Each com-=-plete path can be "exercised" by one test case if appropriate input data can be found. We find that for most of the subroutines, the all-du-paths criterion can be satisfied with fewer than t... |

5 | A standard representation of imperative language programs for data collection and software measures speci cation
- Bieman, Baker, et al.
- 1988
(Show Context)
Citation Context ...rously define the all-du-paths criterion using the SPECS specification language [19]. The criterion is specified in terms of a language-independent representations(StandardRep) of imperative programs =-=[20]-=-. Our formal definition of the all-du-paths criterion is applicable to any imperative language that can be mapped to the representation. Because our empirical study is based on a formal specification,... |

5 |
Criteria for finite set of paths that characterize control flow
- Baker, Howatt, et al.
- 1986
(Show Context)
Citation Context ...called procedure are used at the point of a procedure call and all variables in the actual parameters of a function call are used. Our specification makes use of the notion of "path subset criter=-=ion" [28]-=-. Consider a flowgraph G = (N; E; s; t), where E ae N \Theta N; s 2 N; t 2 N , and all nodes x 2 N lie on a path from s to t. Any path P from s to t is a complete path. A path subset criterion is a bo... |

5 | A formal evaluation of data ow path selection criteria - Clarke, Podgurski, et al. |

4 |
Basic mathematical definitions and results in testing
- White
- 1981
(Show Context)
Citation Context ...infeasible. Thus, it might be impossible to select test data that satisfies the particular criterion. As are many problems in testing, determining whether a particular path is feasible is undecidable =-=[15, 16, 17]-=-. Frankl and Weyuker suggest the use of heuristics to identify infeasible paths [18]. Since some complete paths and du-paths may be infeasible, our measure of the necessary number of complete paths wi... |

4 |
Implications for formal specifications -- results of specifying a software engineering tool
- Baker, Bieman, et al.
- 1987
(Show Context)
Citation Context ...er of complete paths will tend to be higher than the number of feasible paths. In developing our research tools, we rigorously define the all-du-paths criterion using the SPECS specification language =-=[19]-=-. The criterion is specified in terms of a language-independent representations(StandardRep) of imperative programs [20]. Our formal definition of the all-du-paths criterion is applicable to any imper... |

4 | Criteria for nite sets of paths that characterize control ow - Baker, Howatt, et al. - 1986 |

3 | Generating a Standard Representation from Pascal Programs - Doh, Bieman, et al. - 1986 |

3 |
Measuring the Cardinality of Execution Path Subsets Meeting the All-DU-Paths Testing Criterion
- Schultz
- 1988
(Show Context)
Citation Context ...he corresponding StandardRep as output [21]. The output routines of the original generator were modified to produce a PDB for each program unit. A listing of the modified C output routines appears in =-=[29]-=- and the original StandardRep generator is in [21]. A PDB consists of an annotated flowgraph for one program unit, or UnitRepType. A PDB contains the information needed for our analysis. The structure... |

3 | The cost of data ow testing: An empirical study - Weyuker - 1990 |

3 | Implications for formal speci cations { results of specifying a software engineering tool - Baker, Bieman, et al. - 1987 |

2 | The complexity of data ow criteria for test data selection - Weyuker - 1984 |

2 | A data ow oriented program testing strategy - Laski, Korel - 1983 |

2 | Basic mathematical de nitions and results in testing - White - 1981 |

2 | A data ow testing tool - Frankl, Weyuker - 1985 |

1 | An empirical study of the complexity of data ow testing - Weyuker - 1988 |