... This paper discusses the experience of the authors, based upon a collection of experiments, in terms of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. To support the framework, this paper discusses the experiences of the authors in carrying out empirical studies, with specific emphasis on persistent problems encountered in experimental design, threats to validity, criteria for evaluation, and execution of experiments in the domain of software engineering.

propose a preliminary set of research guidelines aimed at stimulating discussion among software researchers. They are based on a review of research guidelines developed for medical researchers and on our own experience in doing and reviewing software engineering research. The guidelines are intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and evaluating empirical studies. Editorial boards of software engineering journals may wish to use our recommendations as a basis for developing guidelines for reviewers and for framing policies for dealing with the design, data collection, and analysis and reporting of empirical studies. Index TermsÐEmpirical software research, research guidelines, statistical mistakes. 1

This paper contributes an integrated survey of the work in the area of software inspection. It consists of two main sections. The first introduces a detailed description of the core concepts and relationships that together define the field of software inspection. The second elaborates a taxonomy that uses a generic development life-cycle to contextualize software inspection in detail. After Fagan's seminal work presented in 1976, the body of work in software inspection has greatly increased and reached measured maturity. Yet, there is still no encompassing and systematic view of this research body driven from a life-cycle perspective. This perspective is important since inspection methods and refinements are most often aligned to particular life-cycle artifacts. It also provides practitioners with a road-map available in their terms. To provide a systematic and encompassing view of the research and practice body in software inspection, the contribution of this survey is, in a first s...

Abstract—Test-Driven Development (TDD) is based on formalizing a piece of functionality as a test, implementing the functionality such that the test passes, and iterating the process. This paper describes a controlled experiment for evaluating an important aspect of TDD: In TDD, programmers write functional tests before the corresponding implementation code. The experiment was conducted with undergraduate students. While the experiment group applied a test-first strategy, the control group applied a more conventional development technique, writing tests after the implementation. Both groups followed an incremental process, adding new features one at a time and regression testing them. We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed. Index Terms—General programming techniques, coding tools and techniques, testing and debugging, testing strategies, productivity,

Software review is a fundamental tool for software quality assurance. Nevertheless, there are significant controversies as to the most efficient and effective review method. One of the most important questions currently being debated is the utility of meetings. Although almost all industrial review methods are centered around the inspection meeting, recent findings call their value into question. To gain insight into these issues, the two authors of this paper separately and independently conducted controlled experimental studies. This paper discusses a joint effort to understand the broader implications of these two studies. To do this, we designed and carried out a process of "reconciliation" in which we established a common framework for the comparison of the two experimental studies, re-analyzed the experimental data with respect to this common framework, and compared the results. Through this process we found many striking similarities between the results of the two studies, stren...

This paper presents results of a software engineering experiment in which a new technology for constructing program generators from domain-specific specification languages has been compared with a reuse technology that employs sets of reusable Ada program templates. Both technologies were applied to a common problem domain, constructing message translation and validation modules for military command, control, communications and information systems (C 3 I). The experiment employed four subjects to conduct trials of use of the two technologies on a common set of test examples. The experiment was conducted with personnel supplied and supervised by an independent contractor. Test cases consisted of message specifications taken from Air Force C 3 I systems. The main results are that greater productivity was achieved and fewer error were introduced when subjects used the program generator than when they used Ada templates to implement software modules from sets of specifications. The d...

Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. 1 Introduction Software inspection is a method for statically verifying documents. It was first described by Michael Fagan in 1976 [6]. Since then there have been many variations and experiences described, but a typical inspection involves a team of three to five people reviewing and understanding a document to find defects. The bene...

In a previous experiment, we determined how various changes in three structural elements of the software inspection process (team size and the number and sequencing of sessions) altered effectiveness and interval. Our results showed that such changes did not significantly influence the defect detection rate, but that certain combinations of changes dramatically increased the inspection interval. We also observed a large amount of unexplained variance in the data, indicating that other factors must be affecting inspection performance. The nature and extent of these other factors now have to be determined to ensure that they had not biased our earlier results. Also, identifying these other factors might suggest additional ways to improve the efficiency of inspections. Acting on the hypothesis that the “inputs ” into the inspection process (reviewers, authors, and code units) were significant sources of variation, we modeled their effects on inspection performance. We found that they were responsible for much more variation in defect detection than was process structure. This leads us to conclude that better defect detection techniques, not better process structures, are the key to improving inspection effectiveness. The combined effects of process inputs and process structure on the inspection interval accounted for only a small percentage of the variance in inspection interval. Therefore, there must be other factors which need to be identified.

Techniques for detecting defects in source code are fundamental to the success of any software development approach. A software development organization therefore needs to understand the utility of techniques such as reading or testing in its own environment. Controlled experiments have proven to be an effective means for evaluating software engineering techniques and gaining the necessary understanding about their utility. This paper presents a characterization scheme for controlled experiments that evaluate defect-detection techniques. The characterization scheme permits the comparison of results from similar experiments and establishes a context for crossexperiment analysis of those results. The characterization scheme is used to structure a detailed survey of four experiments that compared reading and testing techniques for detecting defects in source code. We encourage educators, researchers, and practition- Also with the Department of Computer Science, University of Kaiserslaut...

There is a growing interest in empirical study in software engineering, both for validating mature technologies and for guiding improvements of less-mature technologies. This paper introduces an empirical methodology, based on experiences garnered over more than two decades of work by the Empirical Software Engineering Group at the University of Maryland and related organizations, for taking a newly proposed improvement to development processes from the conceptual phase through transfer to industry. The methodology presents a series of questions that should be addressed, as well as the types of studies that best address those questions. The methodology is illustrated by a specific research program on inspection processes for ObjectOriented designs. Specific examples of the studies that were performed and how the methodology impacted the development of the inspection process are also described. Categories and Subject Descriptors K.6.3 [Management of Computing and Information Systems]: Software Management --Software process. General Terms Measurement, Design, Experimentation, Verification. Keywords Empirical studies, OO design inspections, software process, experimental process, software quality 1 CHALLENGES OF SOFTWARE TECHNOLOGY