Java applications represent a broad class of programs, ranging from programs running on embedded products to highperformance server applications. Standard Java benchmarks ignore this fact and assume a fixed workload. When an actual application’s beha vior differs from that included in a standard benchmark, the benchmark results are useless, if not misleading. In this paper, we present HBench:Java, an application-specific benchmarking framework, based on the concept that a system's performance must be measured in the context of the application of interest. HBench:Java employs a methodology that uses vectors to characterize the application and the underlying JVM and carefully combines the two vectors to form a single metric that reflects a specific application ’ s performance on a particular JVM such that the performance of multiple JVMs can be realistically compared. Our performance results demonstrate HBench:Java ’ s superiority over traditional benchmarking approaches in predicting real application performance and its ability to pinpoint performance problems.

Virtual execution environments, such as the Java Virtual Machine, promote platform-independent software development. However, when it comes to analyzing algorithm complexity and performance bottlenecks, available tools focus on platform-specific metrics, such as e.g. the CPU time consumption on a particular system. Other drawbacks of many prevailing profiling tools are high overhead, significant measurement perturbation, as well as reduced portability of profiling tools, which are often implemented in platformdependent native code. This article presents a novel profiling approach, which is entirely based on program transformation techniques, in order to build a profiling data structure that provides calling-contextsensitive program execution statistics. We explore the use of platform-independent profiling metrics, in order to make the instrumentation entirely portable and to generate reproducible profiles. We implemented these ideas within a Java-based profiling tool called JP. A significant novelty is that this tool achieves complete bytecode coverage, by statically instrumenting the core runtime libraries, and dynamically instrumenting the rest of the code. JP provides a small and flexible API to write customized profiling agents in pure Java, which are periodically activated to process the collected profiling information. Performance measurements point out that, despite the presence of dynamic instrumentation, JP causes significantly less overhead than a prevailing tool for the profiling of Java code.

This thesis introduces a novel approach to performance evaluation, called application-specific benchmarking, and presents techniques for designing and constructing meaningful benchmarks. A traditional benchmark usually includes a fixed set of programs that are run on different systems to produce a single figure of merit, which is then used to rank system performance. This approach often overlooks the relevance between the benchmark programs and the real applications they are supposed to represent. When the behaviors of the benchmark programs do not match those of the intended application, the benchmark scores are uninformative, and sometimes can be misleading. Furthermore, with the rapid pace of application development, it is impractical to create a new standard benchmark whenever a new “killer ” application emerges. The application-specific benchmarking approach incorporates characteristics of the application of interest into the benchmarking process, yielding performance metrics that reflect the expected behavior of a particular application across a range of different

Profiling is essential for finding execution time hot spots in applications. However, in embedded systems resources are usually scarce and profiling is not an option, although the detection and optimization of hot spots is particularly important in such resource-constrained systems. In this paper we propose cross-profiling for embedded systems equipped with a Java processor; the cross-profiles are collected in any standard Java environment, but represent the execution time metrics of the embedded target platform. We present a novel cross-profiler that relies on Java bytecode instrumentation and generates calling-contextsensitive cross-profiles with CPU cycle estimations for each calling context. Our cross-profiler reconciles platformindependence, portability, compatibility with standard Java runtime systems, complete bytecode coverage, moderate profiling overhead, and high accuracy of the generated cross-profiles. Keywords: Cross-profiling, embedded Java processors, bytecode instrumentation, platform-independent dynamic metrics 1.

Abstract — Many performance analysis tools for Java focus on tracking executed bytecodes, but provide little support in determining the specific contribution of native code libraries. This paper introduces and assesses a portable approach for characterizing the amount of native code executed by Java applications. A profiling agent based on the JVM Tool Interface (JVMTI) accurately keeps track of all runtime transitions between bytecode and native code. It relies on a combination of JVMTI events, Java Native Interface (JNI) function interception, bytecode instrumentation, and hardware performance counters. 1 I.

Abstract. Java provides a specification for a user-defined general purpose equivalence operator for objects, but collections such as Set have more stringent requirements. This inconsistency breaks polymorphism: programmers must take care to follow Set’s contract rather than the more general Object contract if their object could enter a Set. We have dynamically profiled 30 Java applications to better understand the way programmers design their objects, to determine whether they program with collections in mind. Our results indicate that objects which enter collections behave very differently to objects which do not. Our findings should help developers understand the impact of design choices they make, and guide future language designers when adding support for collections and/or equality. 1

Java requires objects ’ hashcodes to be consistent with object equality, and to change only when an object’s equality changes. Equality dependent Java collections impose a stricter contextual contract on an objects ’ equality, comparability, and hashcode methods: that they cannot change while those objects are in collections. By dynamically profiling 30 Java applications, we have determined that programmers use quite sophisticated techniques to ensure their objects meet these contracts. Some objects are immutable; some objects are initially mutable, but become immutable when inserted into equality- dependent collections; some objects continue to change when inserted into equality dependent collections — but those changes do not affect equality or hashcodes. On the other hand, some objects inserted into collections that do not depend on object equality, hashcodes, or comparisons will contine to change all their state, including equality and hashcode, while they are in those collections. Programmers and educators can use these results to ensure they (or their students) are aware of these subtle conditional contracts underlying Java’s collection design. These results can also guide language designers adding support for collections, relations, or equality: in particular; while immutablility suffices for many cases, it does not cover all of them. 1.

Calling-context profiles and dynamic metrics at the bytecode level are important for profiling, workload characterization, program comprehension, and reverse engineering. Prevailing tools for collecting calling-context profiles or dynamic bytecode metrics often provide only incomplete information or suffer from limited compatibility with standard JVMs. However, completeness and accuracy of the profiles is essential for tasks such as workload characterization, and compatibility with standard JVMs is important to ensure that complex workloads can be executed. In this paper, we present the design and implementation of JP2, a new tool that profiles both the inter- and intra-procedural control flow of workloads on standard JVMs. JP2 produces calling-context profiles preserving callsite information, as well as execution statistics at the level of individual basic blocks of code. JP2 is complemented with scripts that compute various dynamic bytecode metrics from the profiles. As a case-study and tutorial on the use of JP2, we use it for crossprofiling for an embedded Java processor.