Testing is an important part of any software development project, and can typically surpass more than half of the development cost. For safety-critical computer based systems, testing is even more important due to stringent reliability and safety requirements. However, most safety-critical computer based systems are real-time systems, and the majority of current testing and debugging techniques have been developed for sequential (non real-time) programs. These techniques are not directly applicable to real-time systems, since they disregard issues of timing and concurrency. This means that existing techniques for reproducible testing and debugging cannot be used. Reproducibility is essential for regression testing and cyclic debugging, where the same test cases are run repeatedly with the intention of verifying modified program code or to track down errors. The current trend of consumer and industrial applications goes from single microcontrollers to sets of distributed micro-controllers, which are even more challenging than handling real-time per-see, since multiple loci of observation and control additionally must be considered. In this thesis we try to remedy these problems by presenting an integrated approach to monitoring, testing, and debugging of distributed real-time systems. For monitoring

This article focuses on test automation for safety-critical reactive systems. In the first part of the paper we introduce a methodology for specification, design and verification of fault-tolerant systems allowing to combine different methods in a systematic and consistent way, provided that these methods are compositional. The methodology indicates how to "switch" between formal verification and testing during the construction of (possibly large) reactive systems. We introduce the basic notions of testing as far as relevant in the context of reactive systems and relate them to the verification methodology. Part II formally describes our test automation method which is based on Hoare's CSP and takes Hennessy's testing theory as a starting point. It is indicated how this specific method fits into the general approach described in Part I. We introduce CSP test drivers which are trustworthy in the sense that they "approximate" refinement proofs, converging to a full proof with the increas...

For testing of sequential software it is usually sufficient to provide the same input (and program state) in order to reproduce the output. For real-time systems, on the other hand, we need also to control, or observe, the timing and order of the inputs. If the system additionally is multitasking, we also need to take timing and the concurrency of the executing tasks into account.

In this State of the art Report (SotA), we will give an introduction to work presented in the area of debugging large software systems with modern hardware architectures. We will discuss techniques used for single- multi- and distributed systems. In addition we will provide pointers to work by large players in the field, and major conferences of importance.

In this paper we present a new approach to deterministic replay using standard components. Our method facilitates cyclic debugging of real-time systems with industry standard real-time operating systems using industry standard debuggers. The method is based on a number of new techniques: A new marker for deterministic differentiation between e.g., loop iterations for deterministic reproduction of interrupts and task preemptions, an algorithm for finding well-defined starting points of replay sessions, as well as a technique for using conditional breakpoints in standard debuggers to replay the target system. We also propose and discuss different methods for deterministic monitoring, and provide benchmarking results from an industrial strength case study demonstrating the feasibility of our method. Previously published solutions to the problem of debugging real-time systems have been based on the concept of deterministic replay: where significant system events like task-switches of multitasking software and external inputs are recorded during run-time, and later replayed (re-executed) off-line. Previous works have been based on either non-standard hardware, specially designed compilers or modified real-time operating systems. The reliance on non-standard components has limited the success of the approach. Even though this idea has been around for 20 years, no industrial application for debugging of real-time systems of the method has been presented. Keywords Real-time systems, distributed real-time systems,

This paper describes an approach to the feature location problem for distributed systems, that is, to the problem of locating which code components are important in providing a particular feature for an end user. A feature is located by observing system execution and noting time intervals in which it active. Traces of execution in intervals with and without the feature are compared. Earlier experience has shown that this analysis is di#cult because distributed systems often exhibit stochastic behavior and because time intervals are hard to identify with precision.

In order to test, or debug, a system we must observe its run-time behavior and deem how well the observations comply with the system requirements. There are two significant differences between debugging and testing of software for desktop computers and embedded real-time systems: (1) It is more difficult to observe embedded computer systems, simply because they are embedded, and that they thus have very few interfaces to the outside world, and (2) the actual act of observing a real-time systems or distributed real-time system can change their behavior. Monitoring of sequential software is straightforward, but for distributed realtime systems it is more complicated, since race conditions with respect to order of access to shared resources occur naturally. Any intrusive observation, or probing, of the distributed real-time system affects the timing and consequently the outcome of the races. In this paper we present a method for deterministic observations of single tasking, multi-taskin...

In the DeeDS prototype, active database functionality and critical timing constraints are combined with integrated monitoring techniques. In the scope of DeeDS, this paper presents a mathematical model which is used to derive two important design constraints; worst-case minimum delay and maximum frequency of events. This model is based on a dual-processor hybrid-monitoring solution. Furthermore, different interaction styles between the scheduler and the event monitor are evaluated. I. Introduction Distributed processing and sharing of large amounts of data under critical timing constraints are needed by many complex real-time systems. Real-time database systems are thus an important component in such complex systems, since they provide efficient storage and access to shared data as well as predictable transaction execution, thereby eliminating the need for ad hoc methods in these respects. Real-time systems have two fundamental characteristics. First, the time at which a result is pre...

This paper reports on an interview survey of embedded systems software engineers working in several different companies. The study has attempted to identify instrumentation techniques that the engineers have used and found to be effective, and that seem to be applicable in other similar systems. A case study is presented to illustrate each technique. The study's goal is to accumulate experience in instrumenting embedded systems for program comprehension, as a step towards more systematic use of such techniques in new systems. The techniques presented are, of course, a preliminary list, and we would welcome a chance to extend our interviews to other systems known to the reader.

Real-time systems are becoming increasingly widespread, especially for safety-critical applications. It is therefore crucial that these systems be correct; however, there are few automated tools for analyzing concurrency and timing properties of these systems. The PARTS toolset uses a Petri-net-based reachability analysis to analyze programs written in an Ada subset. Our simple time Petri nets or STP nets are specifically aimed at facilitating real-time analysis. In order to control the state-explosion problem, PARTS employs several optimization techniques aimed at state-space reduction. In this paper we discuss our approach and we report on extensive experiments with several examples of real-time Ada programs. When this is possible, we also compare our experimental results with results obtained by other approaches to real-time analysis. 1 Introduction Real-time software systems are becoming more and more widespread. By real-time software systems, we mean systems with timing constra...