## Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis (1992)

Venue: | IEEE Transactions on Computers |

Citations: | 19 - 2 self |

### BibTeX

@ARTICLE{Geist92estimationand,

author = {Robert M. Geist and A. Jefferson Offutt and Frederick C. Harris and Jr.},

title = {Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis},

journal = {IEEE Transactions on Computers},

year = {1992},

volume = {41},

pages = {550--558}

}

### Years of Citing Articles

### OpenURL

### Abstract

A simulation-based method for obtaining numerical estimates of the reliability of N-version, real-time software is proposed. An extended stochastic Petri net is used to represent the synchronization structure of N versions of the software, where dependencies among versions are modeled through correlated sampling of module execution times. The distributions of execution times are derived from automatically generated test cases that are based on mutation testing. Since these test cases are designed to reveal software faults, the associated execution times and reliability estimates are likely to be conservative. Experimental results using specifications for NASA's planetary lander control software suggest that mutation-based testing could hold greater potential for enhancing reliability than the desirable but perhaps unachievable goal of independence among N versions. Nevertheless, some support for N-version enhancement of high quality, mutation-tested code is also offered. Experimental ...

### Citations

414 |
Hints on test data selection: Help for the practicing programmer
- DeMillo, Lipton, et al.
- 1978
(Show Context)
Citation Context ...n Testing Generating test cases that are effective at finding faults is a technically difficult task. One important criterion for generating test data is relative adequacy as defined by DeMillo et al =-=[7]-=-: Definition. If P is a program to implement function F and \Pi is a collection of programs, then test set T is adequate for P relative to \Pi if P(t)=F(t) 8 t2T, and 8 Q 2 \Pi, Q 6= F ) 9 t2T such th... |

271 | A Class of Generalized Stochastic Petri Nets for the Performance Evaluation of Multiprocessor Systems
- Marsan, Balbo, et al.
- 1984
(Show Context)
Citation Context ... token to each output place. ffl If firing an enabled transition would disable a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions =-=[20, 21, 22]-=- have been used by many authors in systems performance modeling [3, 10, 14] and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing... |

262 | An experimental evaluation of the assumption of independence in multiversion programming
- Knight, Leveson
- 1986
(Show Context)
Citation Context ... partially supported by NASA Langley Research Center under grant NAG-1-1024. 1 Introduction The use of multi-version software to improve computer system reliability remains a topic of vigorous debate =-=[2, 15, 17, 18]. One caus-=-e for concern is easily seen in considering a simple model of majority voting: if each of three voters independently votes "yes" (meaning a correct vote) with probability p, then the probabi... |

257 | Constraint-based automatic test data generation
- DeMillo, Offutt
- 1991
(Show Context)
Citation Context ...nd domain perturbation; the mutation operators also directly model many types of faults. Unfortunately, generating mutation-adequate tests can be a labor-intensive task. To solve this problem, Offutt =-=[9]-=- devised an adequacy-based scheme for automatically generating test data through a Constraint-Based Testing (CBT) system. In constraint-based testing, we represent the conditions under which each muta... |

252 | The n-version approach to faulttolerant software - Avizienis - 1985 |

204 |
Performance Analysis Using Stochastic Petri Nets
- Molloy
- 1982
(Show Context)
Citation Context ... token to each output place. ffl If firing an enabled transition would disable a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions =-=[20, 21, 22]-=- have been used by many authors in systems performance modeling [3, 10, 14] and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing... |

169 |
Probability & Statistics with Reliability, Queueing, and Computer Science Applications
- TRIVEDI
- 1982
(Show Context)
Citation Context ...e dependence on R j is as described, then the conditional density of R i is f R i jR j (r i jr j ) = K ffi (r i \Gamma r j ) + (1 \Gamma K)fR j (r i ) where ffi denotes the unit impulse function (see =-=[24]-=-). We then have E[R i jR j = r j ] = Kr j + (1 \Gamma K)=2; E[R i R j ] = K=3 + (1 \Gamma K)=4; V [R i ] = 1=12; V [R j ] = 1=12; COV [R i ; R j ] = K=12; and thus the correlation of R i and R j is ae... |

166 |
Net Theory and the Modeling of Systems
- Petri
- 1981
(Show Context)
Citation Context ... token to each output place. ffl If firing an enabled transition would disable a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions =-=[20, 21, 22]-=- have been used by many authors in systems performance modeling [3, 10, 14] and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing... |

103 | A Fortran Language System for Mutation-Based Software Testing
- King, Offutt
- 1991
(Show Context)
Citation Context ...to be high quality tests. The most recent mutation system is Mothra [6], which allows a tester to examine remaining live mutants and design tests that kill them. The mutation operators used by Mothra =-=[16]-=- represent more than 10 years of refinement through several mutation systems. These operators explicitly require that the test data meet statement and (extended) branch coverage criteria, extremal val... |

95 |
Data diversity: an approach to software fault tolerance
- Ammann, Knight
- 1988
(Show Context)
Citation Context ...iability. One such technique, providing programmers with mutationgenerated I/O pairs, is examined in section 5. A second technique, a mutation-directed variation on Ammann and Knight's data diversity =-=[1]-=-, is discussed in section 6. Conclusions follow in section 7. 2 The Synchronization Model We consider a program module to be a self-contained piece of software, typically a subroutine, function, or sm... |

92 |
A theoretical basis for the analysis of multiversion software subject to coincident errors
- Eckhardt
- 1985
(Show Context)
Citation Context ...sue is readily identified as "version correlation," but the meaning of this phrase in the software development environment can be elusive. A substantial clarification was provided by Eckhard=-=t and Lee [12]-=- and by Littlewood and Miller [19]. Using Littlewood's notation [19], we let random variable X represent an input to any of a collection P of programs designed to perform the same task, and let \Theta... |

80 |
An extended overview of the mothra software testing environment
- DeMillo, Guindi, et al.
- 1988
(Show Context)
Citation Context ...led by a test case that also reveals the fault. Mutation-adequate tests have been shown experimentally [4, 13] and analytically [5] to be high quality tests. The most recent mutation system is Mothra =-=[6]-=-, which allows a tester to examine remaining live mutants and design tests that kill them. The mutation operators used by Mothra [16] represent more than 10 years of refinement through several mutatio... |

74 |
Mutation Analysis of Program Test Data
- Budd
(Show Context)
Citation Context ...emise: if the software contains a fault, it is likely that there is a mutant that can only be killed by a test case that also reveals the fault. Mutation-adequate tests have been shown experimentally =-=[4, 13]-=- and analytically [5] to be high quality tests. The most recent mutation system is Mothra [6], which allows a tester to examine remaining live mutants and design tests that kill them. The mutation ope... |

56 |
Extended stochastic Petri nets: applications and analysis
- Dugan, Trivedi, et al.
- 1984
(Show Context)
Citation Context ... for estimating the reliability of multi-version, real-time software that incorporates non-zero correlation as an independent model parameter. In section 2 we develop an Extended Stochastic Petri Net =-=[11]-=- as a representation of the synchronization structure of N software versions, where dependencies among version performances are captured through correlated sampling of module execution times. The exec... |

52 |
Two notions of correctness and their relation to testing
- Budd, Angluin
- 1982
(Show Context)
Citation Context ...ntains a fault, it is likely that there is a mutant that can only be killed by a test case that also reveals the fault. Mutation-adequate tests have been shown experimentally [4, 13] and analytically =-=[5]-=- to be high quality tests. The most recent mutation system is Mothra [6], which allows a tester to examine remaining live mutants and design tests that kill them. The mutation operators used by Mothra... |

50 |
A generalized timed petri net model for performance analysis
- Holliday, Vernon
- 1987
(Show Context)
Citation Context ...le a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions [20, 21, 22] have been used by many authors in systems performance modeling =-=[3, 10, 14]-=- and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing time distributions. Specifically, we attach to each transition a distribut... |

24 |
Combining Queueing Network and Generalized Stochastic Petri Net Models for the Analysis of A Software Blocking Phenomenon", International Workshop on Timed Petri Nets
- Balbo, Bruell, et al.
- 1985
(Show Context)
Citation Context ...le a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions [20, 21, 22] have been used by many authors in systems performance modeling =-=[3, 10, 14]-=- and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing time distributions. Specifically, we attach to each transition a distribut... |

24 |
R An experimental comparison of the error exposing ability of program testing criteria
- R, WOODWARD
(Show Context)
Citation Context ...emise: if the software contains a fault, it is likely that there is a mutant that can only be killed by a test case that also reveals the fault. Mutation-adequate tests have been shown experimentally =-=[4, 13]-=- and analytically [5] to be high quality tests. The most recent mutation system is Mothra [6], which allows a tester to examine remaining live mutants and design tests that kill them. The mutation ope... |

24 |
An empirical study of failure probabilities in multi-version software
- Leveson
- 1986
(Show Context)
Citation Context ... partially supported by NASA Langley Research Center under grant NAG-1-1024. 1 Introduction The use of multi-version software to improve computer system reliability remains a topic of vigorous debate =-=[2, 15, 17, 18]. One caus-=-e for concern is easily seen in considering a simple model of majority voting: if each of three voters independently votes "yes" (meaning a correct vote) with probability p, then the probabi... |

13 |
A conceptual model of multi-version software
- Littlewood, Miller
- 1987
(Show Context)
Citation Context ...ion correlation," but the meaning of this phrase in the software development environment can be elusive. A substantial clarification was provided by Eckhardt and Lee [12] and by Littlewood and Mi=-=ller [19]-=-. Using Littlewood's notation [19], we let random variable X represent an input to any of a collection P of programs designed to perform the same task, and let \Theta(x) be the probability that a rand... |

10 |
Experimental results of automatically generated adequate test sets, in
- DeMillo, Offutt
- 1988
(Show Context)
Citation Context ...cessity constraint to create a test case consisting of values for the input variables that will make the constraints true. Godzilla consistently generates test cases that kill over 95% of the mutants =-=[8]-=-. 4 A Correlated Sampling Experiment NASA's planetary lander control software is designed as an N-version voting system. Five implementationss(M 1 ; M 2 ; :::; M 5 ) of the accelerometer sensor proces... |

8 |
The design of a unified package for the solution of stochastic Petri net models
- Dugan, Bobbio, et al.
- 1985
(Show Context)
Citation Context ...le a concurrently enabled transition (conflict), the firing transition is chosen at random. Petri nets and their extensions [20, 21, 22] have been used by many authors in systems performance modeling =-=[3, 10, 14]-=- and reliability modeling [23, 26]. We augment Petri nets with two extensions. The first extension we need is non-zero firing time distributions. Specifically, we attach to each transition a distribut... |

7 |
A specification-oriented multi-version software experiment
- Kelly, Avizienis
- 1983
(Show Context)
Citation Context ... partially supported by NASA Langley Research Center under grant NAG-1-1024. 1 Introduction The use of multi-version software to improve computer system reliability remains a topic of vigorous debate =-=[2, 15, 17, 18]. One caus-=-e for concern is easily seen in considering a simple model of majority voting: if each of three voters independently votes "yes" (meaning a correct vote) with probability p, then the probabi... |

3 |
Software requirements: Guidance and control software development specification
- Withers, Rich, et al.
- 1990
(Show Context)
Citation Context ...bel some correct answers incorrect, this limitation contributes to the conservativeness of the reliability estimate. The software used for this experiment was based on specifications provided by NASA =-=[25]-=-. These specifications are quite thorough with regard to what must be accomplished in each module and what the parameters must be. The module selected has tight timing requirements and reasonable comp... |

3 |
A fast timing verification method based on the independence of units
- Yoneda, Nakade, et al.
- 1989
(Show Context)
Citation Context ...(conflict), the firing transition is chosen at random. Petri nets and their extensions [20, 21, 22] have been used by many authors in systems performance modeling [3, 10, 14] and reliability modeling =-=[23, 26]-=-. We augment Petri nets with two extensions. The first extension we need is non-zero firing time distributions. Specifically, we attach to each transition a distribution of firing times. When a transi... |

1 |
Modeling of fault-tolerant techniques in hierarchical systems
- Shieh, Ghosal, et al.
- 1989
(Show Context)
Citation Context ...(conflict), the firing transition is chosen at random. Petri nets and their extensions [20, 21, 22] have been used by many authors in systems performance modeling [3, 10, 14] and reliability modeling =-=[23, 26]-=-. We augment Petri nets with two extensions. The first extension we need is non-zero firing time distributions. Specifically, we attach to each transition a distribution of firing times. When a transi... |