Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments.

In Software Engineering, researchers are continually developing new tools, techniques and methods. The problem is that very often these new technologies never make it out of the research laboratory into real-world practice, and when they do, there is often little empirical data capable of showing their likely effect

Software Engineering studies run in classroom environments can and have made important contributions to empirical software engineering. Because the goal of such studies is to improve the state of the practice in industry, researchers must understand and account for the differences between university students and industrial professionals. One major difference identified is the amount of training and practice that students and professional may have when learning a new technique. We propose and test a method of allowing university subjects to cost-effectively gain experience to compensate for this difference. The results show that the proposed method for gaining experience provided subjects with enough experience to improve their effectiveness in some but not all cases. There was also an indication from the results that the proposed method allowed the subjects to become more comfortable with a new technique.

For a growing population of researchers in software engineering, empirical studies have become a key approach of research. Empirical studies may be used, for example, to evaluate technologies and help to direct further research by revealing what problems and difficulties people have in practice. Without empirical studies, we have to rely only on intuition or educated opinion. Individual empirical studies often yield interesting results for their particular context, but typically this context is not described in sufficient detail to decide whether another context is similar enough to apply the conclusions of the study also there. We argue instead that families of experiments with a common framework for collecting context data are necessary in order to abstract conclusions at a useful level of detail. This paper describes a method to plan, conduct, and analyze coordinated, or concerted, families of experiments. The goal of the method is to maximize the quality and benefit of the individual empirical studies as part of the family and to minimize the effort for researchers by reusing experiment know-how. This is achieved by providing, for all studies of the experiment family, a common framework for context measurement, study preparation, material, and analysis. We apply the method to describe the planning steps for an experiment family on the influence of context on the effectiveness of defect reduction techniques. We focus on a particular technology, reading techniques for inspections, to instantiate this work. The first step of this experiment family is a broad survey of software companies on the state of the practice of inspection process and inspection techniques. The second step is to benchmark state-of-the-art inspection techniques with the participating organization's own documents and inspection techniques.

Complex software development is a risky job. The number of unsuccessful projects surpasses the number of successful developments, particularly when large projects are analyzed. This paper describes an approach to develop, retrieve, and reuse management knowledge and experience concerned with software development risks. Scenarios are used to model risk impact and resolution strategies efficacy within risk archetypes. A risk archetype is an information structure that holds knowledge about software development risks. A risk management process organizes the use of risk archetypes within an application development effort. The process resembles a reuse process framework, where two sub-processes are respectively responsible for identifying and reusing risk information. Simulating the impact of the expected risks can support some of the decisions throughout the software development process. The contribution of this paper is to show how risk archetypes and scenario models can represent reusable project management knowledge. An observational analysis of applying such an approach in an industrial environment and a feasibility study are also described.

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments. In our work, we have reasoned that the availability of laboratory packages for experiments can encourage better replications and complementary studies. However, even with effectively specified laboratory packages, transfer of experimental know-how can still be difficult. In this paper, we discuss the collaboration structures we have been using in the Readers ’ Project, a bilateral project supported by the Brazilian and American national science agencies that is investigating replications and transfer of experimental know-how issues. In particular, we discuss how these structures map to the Nonaka-Takeuchi knowledge sharing model, a well-known paradigm used in the knowledge management literature. We describe an instantiation of the Nonaka-Takeuchi Model for software engineering experimentation, establishing a framework for discussing knowledge sharing issues related to experimental software engineering. We use two replications to illustrate some of the knowledge sharing issues we have faced and discuss the mechanisms we are using

When learning a new software engineering technique, there is a learning curve that must be overcome. In particular, when studies are conducted in a classroom setting, researchers need a method for quickly accelerating the experience of novice subjects to allow the results to be more applicable in industrial settings. In this paper, we propose and test a method to enable novices to gain process experience to allow them to more quickly overcome the learning curve. The method we evaluate allows an inspector to gain experience with the inspection process by observing an inspection performed by someone else. The results of the study show that the proposed method for gaining experience appears to be useful in some limited cases, that is, for low experienced subjects who were inspecting a requirements document from a domain in which they had low knowledge. Based on the results of this study, we are able to propose some new related hypotheses to be tested in future studies.

Requirements inspections are a process for improving the quality of software by allowing sofare developers to detect defects early in the lifecycle when they are cheaper to fix. One issue that arises is the experience level with a particular inspection technique an inspector needs before he or she is effective and efficient in using that technique. This technical report describes a study run in CMSC735 in Fall 2001. The goal of this study was to begin to understand the impact of process experience on a sofare inspection. Some of the subjects were given a chance to observe an inspection using the Perspective Based Reading (PBR) techniques before they had to use these techniques themselves. This report discusses how the particular experience with process was evaluated and how the efficiency and effectiveness of these subjects compared with that of the subjects who did not get the opportunity to observe someone else using PBR prior to their own use of it.

While the inspection of various software artifacts increases the quality of the end product, the effectiveness of an inspection depends largely on the individual inspectors involved. To address that issue, a large-scale controlled inspection experiment with over 70 professionals was conducted at Microsoft Corporation that focused on the relationship between an inspector’s background and their effectiveness during a requirements inspection. The results of the study showed that inspectors with university degrees in majors not related to computer science found significantly more defects than those with degrees in computer science majors. We also observed that level of education (Masters, PhD), prior industrial experience or other job related experiences did not significantly impact the effectiveness of an inspector. The only other type of experience that had a significant impact on effectiveness was experience in writing requirements, i.e. professionals with prior experience writing requirements found statistically significant more defects than their counterparts.

Abstract. The growing interest in experimental studies in software engineering and the difficulties found in their execution had led software engineering researchers to look for ways to (semi) automate the experimental process. This paper introduces the concept of experimental Software Engineering Environment (eSEE) – an infrastructure capable of instantiating software engineering environments to manage knowledge about the definition, planning, execution and packaging of experimental studies in software engineering. 1