... This paper discusses the experience of the authors, based upon a collection of experiments, in terms of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. To support the framework, this paper discusses the experiences of the authors in carrying out empirical studies, with specific emphasis on persistent problems encountered in experimental design, threats to validity, criteria for evaluation, and execution of experiments in the domain of software engineering.

Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have.

Computer scientists and practitioners defend the lack of experimentation with a wide range of arguments. Some arguments suggest that experimentation may be inappropriate, too difficult, useless, and even harmful. This article discusses several such arguments to illustrate the importance of experimentation for computer science. This is a preprint of an article with the same title that appeared in IEEE Computer, 31(5), May 1998, 32--40. Keywords: Empiricism, experiments, laboratory, scientific method. 1 Is computer science an experimental science? Do computer scientists need to experiment at all? Only if we answer "yes" does it make sense to ask whether there is enough of it. In his Allen Newell Award Lecture, Fred Brooks suggests that computer science is "not a science, but a synthetic, an engineering discipline"[2]. In an engineering field, testing theories by experiments would be misplaced. Brooks and others seem troubled by the fact that the phenomena studied by computer scientists ...

This paper reports the results of an empirical comparison of several static analysis tools for evaluating properties of concurrent software and also reports the results of our attempts to build predictive models for each of the tools based on program and property characteristics. Although this area seems well suited to empirical investigation, we encountered a number of significant issues that make designing a sound and unbiased study surprisingly difficult. These experiment design issues are also discussed in this paper.

The SPICE project aims to deliver an international standard for software process assessment by the end of 1996. As part of this project there is an empirical trials phase whose purpose is to ascertain the effectiveness of the prospective SPICE standard. Two of the objectives of the trials phase are: (a) to determine the extent to which SPICEconformant assessments are repeatable (i.e., reliability), and (b) to determine the extent to which SPICE-conformant assessments are really measuring best software process practices (i.e., validity). This paper introduces the theoretical foundations for evaluating the reliability and validity of measurement, suggests some empirical research methods for investigating them in SPICE, and discusses the constraints and limitations of these methods within the context of the SPICE project.

In this article we summarize the strengths and weaknesses of empirical research in software engineering. We argue that in order to improve the current situation we must create better studies and draw more credible interpretations from them. We finally present a roadmap for this improvement, which includes a general structure for software empirical studies and concrete steps for achieving these goals: designing better studies, collecting data more effectively, and involving others in our empirical enterprises.

Although experimentation is an accepted approach toward scientific validation in most scientific disciplines, it only recently has gained acceptance within the software development community. In this paper we discuss a 12 model classification scheme for performing experimentation within the software development domain. We evaluate over 600 published papers in the computer science literature and over one hundred papers from other scientific disciplines in order to determine: (1) how well the computer science community is succeeding at validating its theories, and (2) how computer science compares to other scientific disciplines. Keywords: Data collection; Experimentation; Evaluation; Measurement 1 Experimentation Experimentation and data collection are the tools of science for validating theories. The classical scientific method depends upon theory formation followed by experimentation and observation in order to provide a feedback loop to validate, modify and improve on the th...

Where the development, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support rigorous controlled experimentation with testing techniques, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of testing techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with software testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have on controlled experimentation with testing techniques.

Experimentation is important within science for determining the effectiveness of proposed theories and methods. However, computer science has not developed a concise taxonomy of methods applicable for demonstrating the validity of a new technique. In this paper we discuss the methods generally employed to validate an experiment and propose a taxonomy consisting of 12 techniques that can be used to show that a new technology achieves its hypothesized goals. An evaluation of over 600 papers published from 1985 through 1995 shows that the 12 methods can be effectively applied to research papers, and we provide some observations of how well the research community validates its claims in these papers. Keywords: Classification; Experimentation; Measurement; Qualitative studies; Quantitative studies; Taxonomy; Validation 1. Experimentation Experimentation and data collection are needed to evaluate practices within the software engineering community as a means to understand both software and...

Spatial joins are join operations that involve spatial data types and operators. Spatial access methods are often used to speed up the computation of spatial joins. This paper addresses the issue of benchmarking spatial join operations. For this purpose, we first present a WWW-based tool to produce sets of rectangles. Experimentators can use a standard Web browser to specify the number of rectangles, as well as the statistical distributions of their sizes, shapes, and locations. Second, using the rectangle generator and a well-defined set of statistical models we defined several test suites to compare the performance of three spatial join algorithms: nested loop, scan-and-index, and synchronized tree traversal. We also added a reallife data set from the Sequoia 2000 storage benchmark. Our results confirm that the use of spatial indices leads to performance gains of several orders of magnitude. The tests also show that highly selective join predicates enjoy greater performanc...