Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein.

Heap spraying is a security attack that increases the exploitability of memory corruption errors in type-unsafe applications. In a heap-spraying attack, an attacker coerces an application to allocate many objects containing malicious code in the heap, increasing the success rate of an exploit that jumps to a location within the heap. Because heap layout randomization necessitates new forms of attack, spraying has been used in many recent security exploits. Spraying is especially effective in web browsers, where the attacker can easily allocate the malicious objects using JavaScript embedded in a web page. In this paper, we describe NOZZLE, a runtime heap-spraying detector. NOZZLE examines individual objects in the heap, interpreting them as code and performing a static analysis on that code to detect malicious intent. To reduce false positives, we aggregate measurements across all heap objects and define a global heap health metric. We measure the effectiveness of NOZZLE by demonstrating that it successfully detects 12 published and 2,000 synthetically generated heap-spraying exploits. We also show that even with a detection threshold set six times lower than is required to detect published malicious attacks, NOZZLE reports no false positives when run over 150 popular Internet sites. Using sampling and concurrent scanning to reduce overhead, we show that the performance overhead of NOZZLE is less than 7 % on average. While NOZZLE currently targets heap-based spraying attacks, its techniques can be applied to any attack that attempts to fill the address space with malicious code objects (e.g., stack spraying [42]). 1

Abstract. We introduce a notion, behavioral distance, for evaluating the extent to which processes—potentially running different programs and executing on different platforms—behave similarly in response to a common input. We explore behavioral distance as a means to detect an attack on one process that causes its behavior to deviate from that of another. We propose a measure of behavioral distance and a realization of this measure using the system calls emitted by processes. Through an empirical evaluation of this measure using three web servers on two different platforms (Linux and Windows), we demonstrate that this approach holds promise for better intrusion detection with moderate overhead.

Abstract. As state-of-the-art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to evade detection. Although recent results have been promising, most existing proposals can be defeated using only minor enhancements to the attack vector. We present a heuristic detection method that scans network traffic streams for the presence of polymorphic shellcode. Our approach relies on a NIDS-embedded CPU emulator that executes every potential instruction sequence, aiming to identify the execution behavior of polymorphic shellcodes. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem. 1

Fine-grained binary instrumentations, such as those for tainttracking, have become very popular in computer security due to their applications in exploit detection, sandboxing, malware analysis, etc. However, practical application of taint-tracking has been limited by high performance overheads. For instance, previous software based techniques for taint-tracking on binary code have typically slowed down programs by a factor of 3 or more. In contrast, source-code based techniques have achieved better performance using high level optimizations. Unfortunately, these optimizations are difficult to perform on binaries since much of the high level program structure required by such static analyses is lost during the compilation process. In this paper, we address this challenge by developing static techniques that can recover some of the higher level structure from x86 binaries. Our new static analysis enables effective optimizations, which are applied in the context of taint tracking. As a result, we achieve a substantial reduction in performance overheads as compared to previous works.

Transactional memory dramatically reduces the complexity of writing concurrent code. Yet, seamless integration of transactional constructs in application code typically comes with a significant performance penalty. Recent studies have shown that compiler support allows producing highly efficient STM-based applications without putting the hassle on the programmer. So far, STM integration has been partially implemented in custom, proprietary compiler infrastructures. In this paper, we propose and evaluate the use of the LLVM open compiler framework to generate efficient concurrent applications using word-based STM libraries. Since LLVM uses the GCC compiler suite as front-end, it can process code written in C or C++ (with partial support for other languages). We also present a tool that allows “transactifying” assembly code and can complement LLVM for legacy code and libraries. Experiments using a lightweight C wordbased STM library show that LLVM integration performs as well as hand-optimized calls to the STM library and better than assembly code instrumentation of the application code. 1.

This document describes our investigation into software obfuscation for building Self-Protecting Mobile Agents (SPMA). The original goal of the SPMA project was to develop automated tools to protect mobile agents from attacks by malicious hosts. In development of those tools, we realized obfuscation could not be relied upon to give a reasonable amount of security. Because of this, we redirected the SPMA project to studying obfuscation. Our conclusions include theoretical results about obfuscation and evidence that supports those results. Our most important conclusion is that there is no general obfuscation problem (i.e. a definition and theory of obfuscation that will always apply). We believe that all automated obfuscation is merely emulation; this will certainly be an area of future research. We conclude that if software obfuscation is to be useful, it must be employed for a specific purpose (not “obfuscate any program protecting all information”), and use fundamentally new ideas. Future theoretical work on obfuscation will have to define it clearly, and use a restricted set of programs, so that the result of Barak et al. [BGI+01] does not apply. In the course of developing obfuscation tools, we evaluated the properties of programming languages under several obfuscating transforms, concluding that strict typesafe programming languages were the best for obfuscation. In addition, programs specifically designed to be obfuscated will give better results, as the programmers will avoid implementing unobfuscatable constructs.

Reasoning about Java bytecode (JBC) is complicated due to its unstructured control-flow, the use of three-address code combined with the use of an operand stack, etc. Therefore, many static analyzers and model checkers for JBC first convert the code into a higher-level representation. In contrast to traditional decompilation, such representation is often not Java source, but rather some intermediate language which is a good input for the subsequent phases of the tool. Interpretive decompilation consists in partially evaluating an interpreter for the compiled language (in this case JBC) written in a high-level language w.r.t. the code to be decompiled. There have been proofs-of-concept that interpretive decompilation is feasible, but there remain important open issues when it comes to decompile a real language such as JBC. This paper presents, to the best of our knowledge, the first modular scheme to enable interpretive decompilation of a realistic programming language to a high-level representation, namely of JBC to Prolog. We introduce two notions of optimality which together require that decompilation does not generate code more than once for each program point. We demonstrate the impact of our modular approach and optimality issues on a series of realistic benchmarks. Decompilation times and decompiled program sizes are linear with the size of the input bytecode program. This demonstrates empirically the scalability of modular decompilation of JBC by partial evaluation.

Advances in reverse engineering and program analyses have made software extremely vulnerable to malicious host attacks. These attacks typically take the form of intellectual property violations, against which the software needs to be protected. The intellectual property that needs to be protected can take on di#erent forms. The software might, e.g., consist itself of proprietary algorithms and datastructures or it could provide controlled access to copyrighted material. Therefore, in recent years, a number of techniques have been explored to protect software. Many of these techniques provide a reasonable level of security against static-only attacks. Many of them however fail to address the problem of dynamic or hybrid static-dynamic attacks. While this type of attack is already commonly used by black-hats, this is one of the first scientific papers to discuss the potential of these attacks through which an attacker can analyze, control and modify a program extensively. The concepts are illustrated through a case study of a recently proposed algorithm for software watermarking [6].

The majority of security vulnerabilities published in the literature are due to software bugs. Many researchers have developed program transformation and analysis techniques to automatically detect or eliminate such vulnerabilities. So far, most of them cannot be applied to commercially distributed applications on the Windows/x86 platform, because it is almost impossible to disassemble a binary file with 100 % accuracy and coverage on that platform. This paper presents the design, implementation, and evaluation of a binary analysis and instrumentation infrastructure for the Windows/x86 platform called BIRD (Binary Interpretation using Runtime Disassembly), which provides two services to developers of security-enhancing program transformation tools: converting binary code into assembly language instructions for further analysis, and inserting instrumentation code at specific places of a given binary without affecting its execution semantics. Instead of requiring a highfidelity instruction set architectural emulator, BIRD combines static disassembly with an on-demand dynamic disassembly approach to guarantee that each instruction in a binary file is analyzed or transformed before it is executed. It takes 12 student months to develop the first BIRD prototype, which can successfully work for all applications in Microsoft Office suite as well as Internet Explorer and IIS web server, including all DLLs that they use. Moreover, the additional throughput penalty of the BIRD prototype on production server applications such as Apache, IIS, and BIND is uniformly below 4%. 1.