One of the most common forms of security attacks involves exploiting a vulnerability to inject malicious code into an executing application and then cause the injected code to be executed. A theoretically strong approach to defending against any type of code-injection attack is to create and use a process-specific instruction set that is created by a randomization algorithm. Code injected by an attacker who does not know the randomization key will be invalid for the randomized processor effectively thwarting the attack. This paper describes a secure and efficient implementation of instruction-set randomization (ISR) using software dynamic translation. The paper makes three contributions beyond previous work on ISR. First, we describe an implementation that uses a strong cipher algorithm—the Advanced Encryption Standard (AES), to perform randomization. AES is generally believed to be impervious to known attack methodologies. Second, we demonstrate that ISR using AES can be implemented practically and efficiently (considering both execution time and code size overheads) without requiring special hardware support. The third contribution is that our approach detects malicious code before it is executed. Previous approaches relied on probabilistic arguments that execution of non-randomized foreign code would eventually cause a fault or runtime exception.

Software Dynamic Translation (SDT) systems have been used for program instrumentation, dynamic optimization, security policy enforcement, intrusion detection, and many other uses. To be widely applicable, the overhead (runtime, memory usage, and power consumption) should be as low as possible. For instance, if an SDT system is protecting a web server against possible attacks, but causes 30 % slowdown, a company may need 30 % more machines to handle the web traffic they expect. Consequently, the causes of SDT overhead should be studied rigorously. This work evaluates many alternative policies for the creation of fragments within the Strata SDT framework. In particular, we examine the effects of ending translation at conditional branches; ending translation at unconditional branches; whether to use partial inlining for call instructions; whether to build the target of calls immediately or lazily; whether to align branch targets; and how to place code to transition back to the dynamic translator. We find that effective translation strategies are vital to program performance, improving performance from as much as 28 % overhead, to as little as 3 % overhead on average for the SPEC CPU2000 benchmark suite. We further demonstrate that these translation strategies are effective across several platforms, including Sun SPARC UltraSparc

Software dynamic translation (SDT) is a technology that allows programs to be modified as they are running. The overhead of monitoring and modifying a running program’s instructions is often substantial in SDT systems. As a result, SDT can be impractically slow, especially in SDT systems that do not or can not employ dynamic optimization to offset overhead. This is unfortunate since SDT has obvious advantages in modern computing environments and interesting applications of SDT continue to emerge. In this paper, we investigate several overhead reduction techniques, including indirect branch translation caching, fast returns, and static trace formation, that can improve SDT performances significantly. 1.

Translation-based virtual execution environments (VEEs) are becoming increasingly popular because of their usefulness. With dynamic translation, a program in a VEE has two binaries: an input source binary and a dynamically generated target binary. Program analysis is important for these binaries, and both the developers and users of VEEs need an instrumentation system to customize program analysis tools. However, existing instrumentation systems for use in VEEs have two drawbacks. First, they are tightly bound with a specific VEE and thus are difficult to reuse without a lot of effort. Second, most of them can not support instrumentation on both the source and target binaries. This paper presents Dimension, a flexible tool that provides instrumentation services for a variety of VEEs. To our knowledge, it is the first stand-alone instrumentation tool that is specially designed for use in VEEs. Given an instrumentation specification, Dimension can be used by a VEE to provide customized instrumentation, enabling analyses on both the source and target binaries. We present two case studies demonstrating that Dimension can be reused easily by different VEEs. We experiment with the two cases and show that the same instrumentation provided by Dimension does not lose efficiency compared to its manual implementation for that particular VEE (the average performance difference is within 2%). We also illustrate that by interfacing with a special VEE that has the same source and target binary formats, Dimension can be used to build an efficient dynamic instrumentation system for traditional execution environments.

Traditional code optimizers have produced significant performance improvements over the past forty years. While promising avenues of research still exist, traditional static and profiling techniques have reached the point of diminishing returns. The main problem is that these approaches have only a limited view of the program and have difficulty taking advantage of the actual run-time behavior of a program. We are addressing this problem through the development of a dynamic optimization system suited for aggressive optimization—using the full power of the most beneficial optimizations. We have designed our optimizer to operate using a software dynamic translation (SDT) execution system. Difficult challenges in this research include reducing SDT overhead and determining what optimizations to apply and where in the code to apply them. Another challenge is having the necessary tools to ensure the reliability of software that is dynamically optimized. In this paper, we describe our efforts in reducing overhead in SDT and efficient techniques for instrumenting the application code. We also describe our approach to determine what and where an optimization should be applied. We discuss other fundamental issues in developing a dynamic optimizer and finally present a basic debugger for SDT systems. 1.

Being able to directly affect code compilation with code transformations allows the seamless addition of custom optimizations and specialized functionality to code at compile time. Traditionally, this has only been possible by directly modifying compiler source code: a very difficult task. Using GCC plug-ins, developers can directly affect code compilation, without actually modifying the source code of GCC. While this makes applying a completed plug-in easy, plug-in development is transformation development nonetheless: an arduous task. The plug-in developer is required to have the same thorough understanding of compiler internals, complex compiler internal representations, and non-trivial source to internal representation mappings as any other transformation developer. Recently, simplified representations, such as CIL, have been developed to help developers overcome some transformation design challenges. Although useful in their own respect, representations like CIL are often language specific by design. This requires the developer to make the unfortunate choice between relative ease of development on a simplified representation or language generality on a more complex representation. We have developed a visual approach to transformation development consisting of a two components: a plugin to extract GCC’s intermediate representation and a Java-based tool to visualize it. This thesis demonstrates how our visual technique significantly reduces many of the problems facing transformation development without sacrificing the inherent benefits of a more generalized intermediate representation. 1

This dissertation was presented by