lmbench: Portable tools for performance analysis lmbench is a micro-benchmark suite designed to focus attention on the basic building blocks of many common system applications, such as databases, simulations, software development, and networking. In almost all cases, the individual tests are the result of analysis and isolation of a customer’s actual performance problem. These tools can be, and currently are, used to compare different system implementations from different vendors. In several cases, the benchmarks have uncovered previously unknown bugs and design flaws. The results have shown a strong correlation between memory system performance and overall performance. lmbench includes an extensible database of results from systems current as of late 1995. 1.

Most performance analysis today uses either microbenchmarks or standard macrobenchmarks (e.g., SPEC, LADDIS, the Andrew benchmark). However, the results of such benchmarks provide little information to indicate how well a particular system will handle a particular application. Such results are, at best, useless and, at worst, misleading. In this paper, we argue for an application-directed approach to benchmarking, using performance metrics that reflect the expected behavior of a particular application across a range of hardware or software platforms. We present three different approaches to application-specific measurement, one using vectors that characterize both the underlying system and an application, one using trace-driven techniques, and a hybrid approach. We argue that such techniques should become the new standard. 1

. Current I/O benchmarks suffer from several chronic problems: they quickly become obsolete, they do not stress the I/O system, and they do not help in understanding I/O system performance. We propose a new approach to I/O performance analysis. First, we propose a self-scaling benchmark that dynamically adjusts aspects of its workload according to the performance characteristic of the system being measured. By doing so, the benchmark automatically scales across current and future systems. The evaluation aids in understanding system performance by reporting how performance varies according to each of five workload parameters. Second, we propose predicted performance, a technique for using the results from the self-scaling evaluation to quickly estimate the performance for workloads that have not been measured. We show that this technique yields reasonably accurate performance estimates and argue that this method gives a far more accurate comparative performance evaluation than tradition...

. An ever-widening mismatch between storage and processor performance is causing storage performance evaluation to become increasingly more important. In this paper, we discuss the metrics and benchmarks used in storage performance evaluation. We first highlight the technology trends taking place in storage systems, such as disk and tape evolution, disk arrays, and solid state disks. We then describe, review, and run today's popular I/O benchmarks on three systems: a DECstation 5000/200 running the Sprite Operating System, a SPARCstation 1+ running SunOS, and an HP Series 700 (Model 730) running HP_UX. We also describe two new approaches to storage benchmarks---LADDIS and A Self-Scaling Benchmark with Predicted Performance. Keywords. I/O, storage, benchmark, workload, self-scaling benchmark, predicted performance, disk, performance evaluation. 1. Introduction In the last decade, innovations in technology have led to extraordinary advances in computer processing speed. These advances h...

To Maddie, who didn’t understand why Daddy had to work late And to Jackie, who did A fundamental problem with the current generation of file system benchmarks is that they fail to take into account the fact that a file system’s performance can vary depending on the workload running on it. Many benchmarks attempt to reduce file system perfor-mance to a single number, producing a simplistic one-dimensional ordering of the sys-tems being tested. Although this may be useful for marketing literature, the performance of file systems in the real world is more complicated. Different workloads place different demands on the file system, and can result in different behavior from the underlying sys-tem. A file system that provides superior performance for a web server may have inferior performance when running a software development workload. In this dissertation I demonstrate that the “one size fits all ” approach of current file system benchmarks does not accurately predict the performance of different workloads on different file systems. I then present a new benchmarking methodology

Performance is often a key factor in determining the success of a parallel software system. Performance evaluation...

An ever-widening mismatch between storage and processor performance is causing storage pe$ormance evaluation to become increasingly more important. In this paper, we discuss the metrics and benchmarks used in storage performance evaluation. We first high fight the technology trends taking place in storage system, such as disk and tape evolution, disk arrays, and solid-state disks. We then describe, review, and run today’s popular I/O benchmarks on three system: a DECstation 5000/200 running the Sprite Operating System, a SPARCstation I + running SunOS, and an HP Series 700 (Model 730) running HP-UX. We also describe two new appmaches to storage benchmarks-UDDIS and A Self-scaling Benchmark with Predicted Pegormance. I.

The performance scalability of the Hurricane File System (HFS) is studied under the context of the K42 Operating System. Both systems were designed for scalability on large-scale, shared-memory, non-uniform memory access multiprocessors. However, scalability of HFS was never studied extensively. Microbenchmarks for reading, writing, creating, obtaining file attributes, and name lookup were used to measure scalability. As well, a macrobenchmark in the form of a simulated Web server was used. The unoptimized version of HFS scaled poorly. Optimizations to the meta-data cache in the form of (1) finer grain locks, (2) larger hash tables, (3) modified hash functions, (4) padded hash list headers and cache entries, and (5) a modified block cache free list, resulted in significant scalability improvements.

lmbench3 extends the lmbench2 system to measure a system'sperformance under scalable load to make it possible to assess parallel and distributed computer performance with the same power and flexibility that lmbench2 brought to uni-processor performance analysis. There is a new timing harness, benchmp, designed to measure performance at specific levels of parallel (simultaneous) load, and most existing benchmarks have been converted to use the new harness. lmbench is a micro-benchmark suite designed to focus attention on the basic building blocks of many common system applications, such as databases, simulations, software development, and networking. It is also designed to make it easy for users to create additional micro-benchmarks that can measure features, algorithms, or subsystems of particular interest to the user.

Abstract — This paper introduces a new benchmark tool,