## Formalism in Safety Cases Appears in Making Systems Safer: Proceedings of the Eighteenth Safety-Critical

### BibTeX

@MISC{Rushby_formalismin,

author = {John Rushby},

title = {Formalism in Safety Cases Appears in Making Systems Safer: Proceedings of the Eighteenth Safety-Critical},

year = {}

}

### OpenURL

### Abstract

Abstract Suitable formalisms could allow the arguments of a safety case to be checked mechanically. We examine some of the issues in doing so. 1

### Citations

586 |
The uses of argument
- Toulmin
- 1958
(Show Context)
Citation Context ...over hazards. The work at Adelard has identified eight different tactics and it remains to be seen whether each of these can be formalized effectively. Some proponents of safety cases look to Toulmin =-=[26]-=- rather than classical logic in framing cases [3]; Toulmin stresses justification rather than inference. My opinion is that Toulmin’s approach has merit in arguing topics such as aesthetics or moralit... |

216 |
Mathematical Logic
- Shoenfield
- 1967
(Show Context)
Citation Context ...concerned. We can informally attach interpretations to the symbols (e.g., system means “the system under consideration”), or we can do so formally by supplying axioms or formal theory interpretations =-=[24]-=-. If the formal elaborations are done correctly (and part of what a theorem prover does is check that we do do it correctly), then anything we can prove about the uninterpreted constants remains true ... |

58 | The Infeasibility of Experimental Quantification of Life-Critical Software Reliability
- Butler, Finelli
- 1991
(Show Context)
Citation Context ...at activate its faults. For modest values, say down to about 1 × 10 −4 probability of failure on demand, it is feasible to measure software failure probabilities by statistically valid random testing =-=[5]-=-, where “statistically valid” means that the test case selection probabilities are exactly the same as those that are encountered in real operation. When the required probabilities are smaller than ca... |

49 | Byzantine agreement with authentication: observations and applications in tolerating hybrid and link faults. Dependable Computing for CT~tical Applications 5
- Gong, Lincoln, et al.
- 1995
(Show Context)
Citation Context ...f between a design that makes fewer assumptions vs. one that tolerates more faults at the cost of more assumptions—or they can be motivated to explore algorithms that combine the best of both choices =-=[7]-=-. Analysis of fault-tolerant systems is one example where appropriate formalization allows the case for correctness to be separated from the case for reliability: formal verification provides assuranc... |

35 | Generating Efficient Test Sets with a Model Checker
- Hamon, Moura, et al.
- 2004
(Show Context)
Citation Context ...rue properties, but also provide explicit counterexamples to false ones (cf. a cut set in FTA). The counterexample capability can be exploited for other purposes, such as the generation of test cases =-=[10]-=-. Holloway’s example states that hazard H1 is eliminated by formal verification, and that the probabilities of hazards H2 and H3 are established by FTA. The formalized top-level safety case simply mak... |

19 |
The Use of Multilegged Arguments to Increase Confidence in Safety Claims for Software-Based Systems: A Study Based on a BBN Analysis of an Idealized Example
- Littlewood, Wright
(Show Context)
Citation Context ...hese kinds. For example, we may have evidence for software based on testing and on its integrity level, and we will wish to combine these two “legs” to yield a “multi-legged” case, perhaps using BBNs =-=[17]-=-. A question is whether these probabilistic calculations should be opaque to the formalization, in the way that hazard analysis is, or at least partially represented in the formalization—e.g., by atta... |

14 |
Harnessing disruptive innovation in formal verification
- Rushby
- 2006
(Show Context)
Citation Context ...ly for the unquantified case. These capabilities are a (subset) of the capabilities of formalisms built on, or employing, SMT solvers (i.e., solvers for the problem of Satisfiability Modulo Theories) =-=[20]-=-. Modern SMT solvers are very effective, often able to solve problems with hundreds of variables and thousands of constraints in seconds. They are the subject of an annual competition, and this has dr... |

13 | Evaluating, testing, and animating PVS specifications
- Crow, Owre, et al.
- 2001
(Show Context)
Citation Context ... approp claim doc to represent existence of this documentation, and the documentation itself can be attached to the constant. Attachments are used quite widely in AI and in formal verification (e.g., =-=[6]-=-), usually to provide a computational interpretation to some term, in which case they are called “semantic attachments.” Here, we have “documentation attachments” and a theorem prover could easily be ... |

13 | The use of proofs in diversity arguments
- Littlewood
- 1999
(Show Context)
Citation Context ...ensive reviews and testing, are really about ensuring correctness, and there is no clear justification for determining a correspondence between SILs and failure probabilities. In contrast, Littlewood =-=[15]-=- introduced the idea that software may be possibly perfect and that we can contemplate its probability of perfection. This is attractive because probability of perfection can be interpreted as a subje... |

11 | S.: The future of goal-based assurance cases
- Bishop, Bloomfield, et al.
- 2004
(Show Context)
Citation Context ...ith a summary and suggestions for further research. 2 The Top-Level Argument The concepts, notations, and tools that have been developed for representing, managing, and inspecting safety cases (e.g., =-=[3, 13]-=-) provide strong support for structuring the argument of a safety case. Nonetheless, the safety case for a real system is a very large object and one wonders how reliably a human reviewer can evaluate... |

10 | A unified faulttolerance protocol
- Miner, Geser, et al.
- 2004
(Show Context)
Citation Context ...lysis, or other methods of automated deduction to show that these have desired properties. Verification systems such as PVS have been used to verify important properties of significant designs (e.g., =-=[18]-=-). However, PVS and its like are general purpose—that is why they can model abstract safety cases—and greater automation in verification of software systems and their designs can be achieved using not... |

9 |
R.: The goal structuring notation–a safety argument notation
- Kelly, Weaver
- 2004
(Show Context)
Citation Context ...ith a summary and suggestions for further research. 2 The Top-Level Argument The concepts, notations, and tools that have been developed for representing, managing, and inspecting safety cases (e.g., =-=[3, 13]-=-) provide strong support for structuring the argument of a safety case. Nonetheless, the safety case for a real system is a very large object and one wonders how reliably a human reviewer can evaluate... |

9 | Runtime Certification
- Rushby
- 2008
(Show Context)
Citation Context ...s can be assured by checking or monitoring them at runtime. If the assumptions are formalized, then construction of monitors can be automated by methods developed in the field of runtime verification =-=[21]-=-. Reliability of monitored architectures with formal (and possibly perfect) monitors is an interesting topic [16]. Yet another benefit of formalization is that it could allow development of canonical ... |

8 | A taxonomy of fallacies in system safety arguments
- Greenwell, Knight, et al.
(Show Context)
Citation Context ...ident can we be that a human reviewer would detect the flaws in the perturbed case? These concerns are not merely speculative: Greenwell and colleagues found flaws in several cases that they examined =-=[8]-=-. Although a safety case is an argument, it will generally contain elements that are not simple logical deductions: some elements of the argument will be probabilistic, some will enumerate over a set ... |

8 | Software verification and system assurance
- Rushby
- 2009
(Show Context)
Citation Context ...parts of the case, particularly any verifications and analyses, are formalized and subject to mechanical checking. I suggest considerations for the assessment of these probabilities in a recent paper =-=[23]-=-. Another area where formalization intersects with probability is in assurance for fault-tolerant systems. Many system failures are due to flaws in fault tolerance: the very mechanisms that are intend... |

7 | Safety Case Notations: Alternatives for the Non-Graphically Inclined
- Holloway
(Show Context)
Citation Context ...n automation, and on the other, we need a choice that is able to express the kinds of arguments used in a safety case. To make this concrete, here is the top level of an argument examined by Holloway =-=[11]-=-: “The control system is acceptably safe, given a definition of acceptably safe, because all identified hazards have been eliminated or sufficiently mitigated and the software has been developed to th... |

7 | Why System Safety Professionals Should Read Accident Reports
- Holloway, Johnson
- 2006
(Show Context)
Citation Context ...esign of the mechanism for fault tolerance. When the fault-tolerance aspects of the safety case are informal, the failure assumptions may be imprecise, and their probabilities assessed optimistically =-=[12]-=-. Formal verification forces precision in the statement of failure mode assumptions and, thereby, explicit recognition of the cases not tolerated—and realistic assessment of their probability. The lat... |

7 |
Arguing Safety—A Systematic Approach to Safety Case Management
- Kelly
- 1998
(Show Context)
Citation Context ...ety case. On the other hand, the motivation for introducing safety cases in the first place came from investigations into a number of disasters where traditional approaches were deemed to have failed =-=[14]-=-. Perusal of recent aircraft accident and incident reports (e.g., [1, 27]) certainly erodes complacency about the standards-based approach employed for airborne software [19]. 1 We may conclude that s... |

6 |
The Nimrod Review: An Independent review into the broader issues surrounding the loss
- Haddon-Cave
- 2009
(Show Context)
Citation Context ...m my personal perspective—which is as a practitioner of formal methods—and may not coincide with the views of those with more experience 1 A recent report finds massive fault with a major safety case =-=[9]-=-.Formalism in Safety Cases 3 in safety cases. My hope is that it will help develop a dialog between these two bodies of knowledge and experience. The next section considers the top-level argument of ... |

5 | B.: Deriving safety cases from automatically constructed proofs
- Basir, Denney, et al.
- 2009
(Show Context)
Citation Context ...se topics. Using a simple example [11], I illustrated one way to formalize the toplevel argument of a simple case in classical logic (I actually used the higher order logic PVS). Basir and colleagues =-=[2]-=- have undertaken a similar exercise using pure first order logic. The example illustrates only one tactic for safety argumentation: namely, enumeration over hazards. The work at Adelard has identified... |

4 | A safety-case approach for certifying adaptive systems
- Rushby
- 2009
(Show Context)
Citation Context ...ties can be attached to the uninterpreted functions by means of axioms supplied directly to the SMT solver or, indirectly, by synchronous observers attached to the model supplied to the model checker =-=[22]-=-. The value in applying formal verification to very abstract designs is that this can be used to automate, or provide automated assistance for, some kinds of safety analyses traditionally performed in... |

3 | System Safety as an Emergent Property in Composite Systems
- Koopman
(Show Context)
Citation Context ...y case argumentation. Techniques for developing safety cases in a modular or compositional manner would be a breakthrough; the topic of emergent properties is particularly interesting in this context =-=[4]-=-. The most important tasks for the future, however, are experiments to determine whether formalization does deliver benefit in the development and assessment of safety cases. Acknowledgements My resea... |

2 |
Reasoning about the Reliability of FaultTolerant Systems
- Littlewood, Rushby
(Show Context)
Citation Context ...ble perfection of one “channel” can be shown to be conditionally independent of the reliability of the other; hence, the probability of system failure is the product of these individual probabilities =-=[16]-=-. Using the idea of possible perfection has two ramifications on a safety case. One is that the upper level assessment of the probability of system failure will employ probabilities of software perfec... |