Software products are often released with missing functionality or errors that result in failures in the eld. In previous work, we presented the Gamma technology, which facilitates remote monitoring of deployed software and allows for a prompt reaction to failures. In this paper, we investigate one of the principal technologies on which Gamma is based: software tomography. Software tomography splits monitoring tasks across many instances of the software, so that partial information can be (1) collected from users by means of light-weight instrumentation and (2) merged to gather the overall monitoring information. After describing the technology, we illustrate an instance of software tomography for a speci c monitoring task. We also present two case studies that we performed to evaluate the presented technique on a real program. The results of the studies show that software tomography can be successfully applied to collect accurate monitoring information using only minimal instrumentation on each deployed program instance.

Where the development, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support rigorous controlled experimentation with testing techniques, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of testing techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with software testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have on controlled experimentation with testing techniques.

Recent advances in networking and embedded device technology have made the vision of ubiquitous computing a reality; users can access the Internet's vast offerings anytime and anywhere. Moreover, battery-powered devices such as personal digital assistants and web-enabled mobile phones have successfully emerged as new access points to the world's digital infrastructure. This ubiquity offers a new opportunity for software developers: users can now participate in the software development, optimization, and evolution process while they use their software. Such participation

Producing reliable and robust software has become one of the most important software development concerns in recent years. Testing is a process by which software quality can be assured through the collection of information. While testing can improve software reliability, current tools typically are inflexible and have high overheads, making it challenging to test large software projects. In this paper, we describe a new scalable and flexible framework for testing programs with a novel demand-driven approach based on execution paths to implement test coverage. This technique uses dynamic instrumentation on the binary code that can be inserted and removed on-the-fly to keep performance and memory overheads low. We describe and evaluate implementations of the framework for branch, node and def-use testing of Java programs. Experimental results for branch testing show that our approach has, on average, a 1.6 speed up over static instrumentation and also uses less memory.

In program profiling to assess test set adequacy, a challenge is to select code to be included in the assessment. Current mechanisms are coarse-grained; biased to dominant modularizations; require tedious, error-prone manual selection; and leave tester intent implicit in inputs to testing tools. Aspect-oriented constructs promise to improving testing in two ways: by improving our ability to select the code to include in adequacy criteria, and by documenting selection intentions in declarative form in the code itself. One problem is that current join point models do not reveal program behavior in enough detail to support white-box coverage analysis. Our contribution is the formulation, prototyping, and evaluation of a language-and-tool-based approach to white-box coverage adequacy analysis that we call concern coverage. We develop and evaluate one instance of the general idea in which branches, in particular, are exposed as join points to support branch coverage analysis of crosscutting concerns. Our results are consistent with the claim that the idea has the potential to improve test coverage analysis. Categories and Subject Descriptors D.2.5 [Testing and Debugging]: Testing tools – code coverage analysis, testing. D.3.3 [Programming Languages]: Language constructs and features – generalized join point model, generalized advice.

Analyzing a program run can provide important insights about its correctness. Dynamic analysis of complex correctness properties, however, usually results in significant run-time overhead and, consequently, it is rarely used in practice. In this paper, we present an approach for exploiting properties of stateful program specifications to reduce the cost of their dynamic analysis. With our approach, analysis results are guaranteed to be identical to those of a traditional expensive dynamic analyses, while analysis cost is very low – between 23 % and 33 % more than the un-instrumented program for the analyses we studied. We describe the principles behind our adaptive online program analysis technique, extensions to our Java run-time analysis framework that support such analyses, and report on the performance and capabilities of two different families of adaptive online program analyses. 1.

In profiling program execution to assess the adequacy of a test set, a challenge is to select the code to be included in the coverage assessment. Current mechanisms for doing this are coarse-grained, assume traditional concepts of modularity, require tedious and error-prone manual selection, and leave the tester’s intent implicit in the input provided to the testing tools. The aspect-oriented constructs of languages such as AspectJ promise to help ease and extend our ability to select the code to be included in a test adequacy criterion and to document the tester’s intent within the source code itself. Our contribution is a language-centric approach to automated test adequacy analysis that we call concern coverage. We claim that our approach enables explicit, precise, abstract, and machine-readable representation of the tester’s intent and that it can ease testing by eliminating the need for manual selection and explicit maintenance of test adequacy criteria. Categories and Subject Descriptors D.2.5 [Testing and Debugging]: Testing tools – code coverage analysis, testing. D.3.3 [Programming Languages]: Language constructs and features – generalized join point model, generalized advice.

Key to understanding and optimizing complex applications, is our ability to dynamically monitor executing programs with low overhead and high accuracy. Toward this end, we present HPS, a Hybrid Profiling Support system. HPS employs a hardware/software approach to program sampling that transparently, efficiently, and dynamically samples an executing instruction stream. Our system is an extension and application of Dynamic Instruction Stream Editing (DISE), a hardware technique that macro-expands instructions in the pipeline decode stage at runtime. HPS toggles profiling to sample the executing program as required by the profile consumer, e.g. a dynamic optimizer. Our system requires few hardware resources and introduces no “basic ” overhead – the execution of instructions that triggers profiling. We use HPS to investigate the tradeoffs between overhead and accuracy of different profile types as well as different profiling schemes. In particular, we empirically evaluate hot data stream, hot call pair, and hot method identification using a number of parameterizations of bursty tracing, a popular sampling scheme used in dynamic optimization systems. 1

The widespread use of ubiquitous, mobile, and continuously-connected computing agents has inspired software developers to change the way they test, debug, and optimize software. Users now play an active role in the software evolution cycle by dynamically providing valuable feedback about the execution of a program to developers. Software developers can use this information to isolate bugs in, maintain, and improve the performance of a wide-range of diverse and complex embedded device applications. The collection of such feedback poses a major challenge to systems researchers since it must be performed without degrading a user’s experience with, or consuming the severely restricted resources of the mobile device. At the same time, the resource constraints of embedded devices prohibit the use of extant software profiling solutions. To achieve efficient remote profiling of embedded devices, we couple two efficient hardware/software program monitoring techniques: Hybrid Profiling Support(HPS) and Phase-Aware Sampling. HPS efficiently inserts profiling instructions into an executing program using a novel extension to Dynamic Instruction Stream Editing(DISE). Phase-aware sampling exploits the recurring behavior of programs to identify key opportunities during execution in order to collect profile information (i.e. sample). Our prior work on phase-aware sampling required code duplication to toggle

Software dynamic translators (SDT) typically monitor, profile, and affect the execution of a program. Such systems have been used to build many useful applications, including dynamic code optimizers for binary machine code and Java bytecode, software security checkers, binary translators, code profilers and program introspection tools. While all of these systems use program instrumentation, the instrumentation is usually tailored to a specific application and infrastructure. What is missing is a single scalable and flexible instrumentation framework that can be used in different SDT infrastructures. In this paper, we describe such a new framework, called "FIST," that can be used and targeted by different algorithms and tools to enable instrumentation applications that are portable across SDTs and machine platforms. Our interface supports multiple levels of granularity from source level constructs to the instruction and machine level. We demonstrate FIST's flexibility and evaluate it in the Strata SDT system for the SPARC and the Jikes Research Virtual Machine for Java on the Intel x86. To show the framework's scalability, we describe new instrumentation applications to prototype dynamic optimization techniques and apply software tests to Java programs.