We have developed a technology, fine-grained dynamic instrumentation of commodity kernels, which can splice (insert) dynamically generated code before almost any machine code instruction of a completely unmodified running commodity operating system kernel. This technology is well-suited to performance profiling, debugging, code coverage, security auditing, runtime code optimizations, and kernel extensions. We have designed and implemented a tool called KernInst that performs dynamic instrumentation on a stock production Solaris kernel running on an UltraSPARC. On top of KernInst, we have implemented a kernel performance profiling tool, and used it to understand kernel and application performance under a Web proxy server workload. We used this information to make two changes (one to the kernel, one to the proxy) that cumulatively reduce the percentage of elapsed time that the proxy spends opening disk cache files from 40 % to 7%. 1

We have developed a facility for run-time optimization of a commodity operating system kernel. This facility is a first step towards an evolving operating system, one that adapts and changes over time without need for rebooting. Our infrastructure, currently implemented on UltraSPARC Solaris 7, includes the ability to do a detailed analysis of the running kernel's binary code, dynamically insert and remove code patches, and dynamically install new versions of kernel functions. As a first use of this technology, we have implemented a run-time kernel version of the code positioning I-cache optimizations, and obtained noticeable speedups in kernel performance. As a first case study, we performed run-time code positioning on the kernel's TCP read-side processing routine while running a Web client benchmark. We found that the code positioning optimizations reduced this function's execution time by 17.6%, resulting in an end-to-end benchmark speedup of 7%. The primary contributions of this paper are the first run-time kernel implementation of code positioning, and an infrastructure for turning an unmodified commodity kernel into an evolving one. Two further contributions are made in kernel performance measurement. First, we provide a simple and effective algorithm for deriving control flow edge execution counts from basic block execution counts, which contradicts the widely held belief that edge counts cannot be derived from block counts. Second, we describe a means for converting wall time instrumentation-based kernel measurements into virtual (i.e., CPU) time measurements via instrumentation of the kernel's context switch handlers. 1