Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have.

In software development, bug reports provide crucial information to developers. However, these reports widely differ in their quality. We conducted a survey among developers and users of APACHE, ECLIPSE, and MOZILLA to find out what makes a good bug report. The analysis of the 466 responses revealed an information mismatch between what developers need and what users supply. Most developers consider steps to reproduce, stack traces, and test cases as helpful, which are at the same time most difficult to provide for users. Such insight is helpful to design new bug tracking tools that guide users at collecting and providing more helpful information. Our CUEZILLA prototype is such a tool and measures the quality of new bug reports; it also recommends which elements should be added to improve the quality. We trained CUEZILLA on a sample of 289 bug reports, rated by developers as part of the survey. In our experiments, CUEZILLA was able to predict the quality of 31–48% of bug reports accurately. Categories and Subject Descriptors:

Test-driven development is a software development practice that has been used sporadically for decades. With this practice, test cases (preferably automated) are incrementally written before production code is implemented. Test-driven development has recently reemerged as a critical enabling practice of the Extreme Programming software development methodology. We ran a case study of this practice at IBM. In the process, a thorough suite of automated test cases was produced. In this case study, we found that the code developed using a test-driven development practice showed, during functional verification and regression tests, approximately 40 % fewer defects than a baseline prior product developed in a more traditional fashion. The productivity of the team was not impacted by the additional focus on producing automated test cases. This test suite will aid in future enhancements and maintenance of this code. The experiment and the results are discussed in detail. 1.

Where the development, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support rigorous controlled experimentation with testing techniques, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of testing techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with software testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have on controlled experimentation with testing techniques.

There is a growing interest in empirical study in software engineering, both for validating mature technologies and for guiding improvements of less-mature technologies. This paper introduces an empirical methodology, based on experiences garnered over more than two decades of work by the Empirical Software Engineering Group at the University of Maryland and related organizations, for taking a newly proposed improvement to development processes from the conceptual phase through transfer to industry. The methodology presents a series of questions that should be addressed, as well as the types of studies that best address those questions. The methodology is illustrated by a specific research program on inspection processes for ObjectOriented designs. Specific examples of the studies that were performed and how the methodology impacted the development of the inspection process are also described. Categories and Subject Descriptors K.6.3 [Management of Computing and Information Systems]: Software Management --Software process. General Terms Measurement, Design, Experimentation, Verification. Keywords Empirical studies, OO design inspections, software process, experimental process, software quality 1 CHALLENGES OF SOFTWARE TECHNOLOGY

Software development remains mentally challenging despite the continual advancement of training, techniques, and tools. Because completely automating software development is currently impossible, it makes sense to seriously consider how tools can improve the mental activities of developers apart from automating them away. Such mental assistance can be called “cognitive support”. Understanding and developing cognitive support in software engineering tools is an important research issue but, unfortunately, at the moment our theoretical foundations for it are inadequately developed. Furthermore, much of the relevant research has occurred outside of the software engineering community, and is therefore not easily available to the researchers who typically develop software engineering tools. Tool evaluation, comparison, and development are consequently impaired. The present work introduces a theoretical framework intended to seed further systematic study of cognitive support in the field of software engineering tools. This theoretical framework, called RODS, imports ideas and methods from a field of cognitive science called “distributed cognition”. The crucial concept in RODS is that cognitive support can be understood and explained in terms of the computational advantages that are conferred when cognition is redistributed between software developer and their tools and environment. The name RODS, in fact, comes from the

Abstract—Analysis of anomalies that occur during operations is an important means of improving the quality of current and future software. Although the benefits of anomaly analysis of operational software are widely recognized, there has been relatively little research on anomaly analysis of safety-critical systems. In particular, patterns of software anomaly data for operational, safety-critical systems are not well understood. This paper presents the results of a pilot study using Orthogonal Defect Classification (ODC) to analyze nearly two hundred such anomalies on seven spacecraft systems. These data show several unexpected classification patterns such as the causal role of difficulties accessing or delivering data, of hardware degradation, and of rare events. The anomalies often revealed latent software requirements that were essential for robust, correct operation of the system. The anomalies also caused changes to documentation and to operational procedures to prevent the same anomalous situations from recurring. Feedback from operational anomaly reports helped measure the accuracy of assumptions about operational profiles, identified unexpected dependencies among embedded software and their systems and environment, and indicated needed improvements to the software, the development process, and the operational procedures. The results indicate that, for long-lived, critical systems, analysis of the most severe anomalies can be a useful mechanism both for maintaining safer, deployed systems and for building safer, similar systems in the future. Index Terms—Software and system safety, diagnostics, maintenance process, product metrics. 1

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments.

In software development, resources for quality assurance are limited by time and by cost. In order to allocate resources effectively, managers need to rely on their experience backed by code complexity metrics. But often dependencies exist between various pieces of code over which managers may have little knowledge. These dependencies can be construed as a low level graph of the entire system. In this paper, we propose to use network analysis on these dependency graphs. This allows managers to identify central program units that are more likely to face defects. In our evaluation on Windows Server 2003, we found that the recall for models built from network measures is by 10 % points higher than for models built from complexity metrics. In addition, network measures could identify 60 % of the binaries that the Windows developers considered as critical—twice as many as identified by complexity metrics. Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics—Performance measures, Process metrics, Product metrics. D.2.9 [Software Engineering]: Management—Software quality assurance (SQA)

A longitudinal case study evaluating the effects of adopting the Extreme Programming (XP) methodology was performed at Sabre Airline Solutions™. The Sabre team was a characteristically agile team in that they had no need to scale or re-scope XP for their project parameters and organizational environment. The case study compares two releases of the same product. One release was completed just prior to the team’s adoption of the XP methodology, and the other was completed after approximately two years of XP use. Comparisons of the new release project results to the old release project results show a 50 % increase in productivity, a 65 % improvement in pre-release quality, and a 35 % improvement in post-release quality. These findings suggest that, over time, adopting the XP process can result in increased productivity and quality. 1.