A precise characterization of those security policies enforceable by program rewriting is given. This characterization exposes and rectifies problems in prior work on execution monitoring, yielding a more precise characterization of those security policies enforceable by execution monitors and a taxonomy of enforceable security policies. Some but not all classes can be identified with known classes from computational complexity theory.

Proof-carrying code (PCC) is a framework for mechanically verifying the safety of machine language programs. A program that is successfully verified by a PCC system is guaranteed to be safe to execute, but this safety guarantee is contingent upon the correctness of various trusted components. For instance, in traditional PCC systems the trusted computing base includes a large set of low-level typing rules. Foundational PCC systems seek to minimize the size of the trusted computing base. In particular, they eliminate the need to trust complex, low-level type systems by providing machine-checkable proofs of type soundness for real machine languages. In this thesis, I demonstrate the use of logical relations for proving the soundness of type systems for mutable state. Specifically, I focus on type systems that ensure the safe allocation, update, and reuse of memory. For each type in the language, I define logical relations that explain the meaning of the type in terms of the oper-ational semantics of the language. Using this model of types, I prove each typing rule as a lemma. The major contribution is a model of System F with general references — that is, mutable cells that can hold values of any closed type including other references, functions, recursive types, and impredicative quantified types. The model is based on ideas from both possible worlds and the indexed model of Appel and McAllester. I show how the model of mutable references is encoded in higher-order logic. I also show how to construct an indexed possible-worlds model for a von Neumann machine. The latter is used in the Princeton Foundational PCC system to prove type safety for a full-fledged low-level typed assembly language. Finally, I present a semantic model for a region calculus that supports type-invariant references as well as memory reuse. iii

Malware detectors require a specification of malicious behavior. Typically, these specifications are manually constructed by investigating known malware. We present an automatic technique to overcome this laborious manual process. Our technique derives such a specification by comparing the execution behavior of a known malware against the execution behaviors of a set of benign programs. In other words, we mine the malicious behavior present in a known malware that is not present in a set of benign programs. The output of our algorithm can be used by malware detectors to detect malware variants. Since our algorithm provides a succinct description of malicious behavior present in a malware, it can also be used by security analysts for understanding the malware. We have implemented a prototype based on our algorithm and tested it on several malware programs. Experimental results obtained from our prototype indicate that our algorithm is effective in extracting malicious behaviors that can be used to detect malware variants.

Most of the recent work on Web security focuses on preventing attacks that directly harm the browser’s host machine and user. In this paper we attempt to quantify the threat of browsers being indirectly misused for attacking third parties. Specifically, we look at how the existing Web infrastructure (e.g., the languages, protocols, and security policies) can be exploited by malicious Web sites to remotely instruct browsers to orchestrate actions including denial of service attacks, worm propagation and reconnaissance scans. We show that, depending mostly on the popularity of a malicious Web site and user browsing patterns, attackers are able to create powerful botnet-like infrastructures that can cause significant damage. We explore the effectiveness of countermeasures including anomaly detection and more fine-grained browser security policies.

Mobile users of computation and communication services have been rapidly adopting battery-powered mobile handhelds, such as PocketPCs and SmartPhones, for their work. However, the limited battery-lifetime of these devices restricts their portability and applicability, and this weakness can be exacerbated by mobile malware targeting depletion of battery energy. Such malware are usually difficult to detect and prevent, and frequent outbreaks of new malware variants also reduce the effectiveness of commonlyseen signature-based detection. To alleviate these problems, we propose a power-aware malware-detection framework that monitors, detects, and analyzes previously unknown energy-depletion threats. The framework is composed of (1) a power monitor which collects power samples and builds a power consumption history from the collected samples, and (2) a data analyzer which generates a power signature from the constructed history. To generate a power signature, simple and effective noise-filtering and data-compression are applied, thus reducing the detection overhead. Similarities between power signatures are measured by the χ 2-distance, reducing both false-positive and false-negative detection rates. According to our experimental results on an HP iPAQ running a Windows Mobile OS, the proposed framework achieves significant (up to 95%) storage-savings without losing the detection accuracy, and a 99 % true-positive rate in classifying mobile malware.

Malware detection is a crucial aspect of software security. Current malware detectors work by checking for “signatures,” which attempt to capture (syntactic) characteristics of the machine-level byte sequence of the malware. This reliance on a syntactic approach makes such detectors vulnerable to code obfuscations, increasingly used by malware writers, that alter syntactic properties of the malware byte sequence without significantly affecting their execution behavior. This paper takes the position that the key to malware identification lies in their semantics. It proposes a semantics-based framework for reasoning about malware detectors and proving properties such as soundness and completeness of these detectors. Our approach uses a trace semantics to characterize the behaviors of malware as well as the program being checked for infection, and uses abstract interpretation to “hide” irrelevant aspects of these behaviors. As a concrete application of our approach, we show that the semantics-aware malware detector proposed by Christodorescu et al. is complete with respect to a number of common obfuscations used by malware writers.

this paper will show that viruses pose theoretical problems also. Indeed, this paper, by laying out some of these problems, begs many questions that raise many further research questions. Some of these research questions will be pointed out explicitly throughout the paper.

Metamorphic viruses transform their code as they propagate, thus evading detection by static signature-based virus scanners, while keeping their functionality. They use code obfuscation techniques to challenge deeper static analysis and can also beat dynamic analyzers, such as emulators, by altering their behavior. To achieve this, metamorphic viruses use several metamorphic transformations, including register renaming, code permutation, code expansion, code shrinking, and garbage code insertion. In this thesis, an in-depth analysis of metamorphic viruses is presented, along with the techniques they use to transform their code to new generations. In order to give a better understanding of metamorphic viruses, a general discussion on malicious code and detection techniques is given first. Then, the description of several techniques to detect metamorphic viruses is given. A fair number of papers on metamorphic viruses exists in the literature, but no one is a complete discussion of all metamorphic techniques and detection methods. This thesis aims at a complete discussion of all metamorphic techniques used by

To defeat current commercial antivirus software, the virus developers are employing obfuscation techniques to create mutating viruses. The current antivirus software cannot handle the obfuscated viruses well since its detection methods that are based upon static signatures are not resilient to even slight variations in the code that forms the virus. In this paper, we propose a new type of virus signature, called dynamic signature, and an algorithm for matching dynamic signatures. Our dynamic signature is created based on the runtime behavior of a virus. Therefore, an obfuscated virus can also be detected using a dynamic signature as long as it dynamically behaves like the original virus. We also discuss issues related to deploying our virus detection approach. Our experiments based upon several known mutating viruses show that our method is effective in identifying obfuscated viruses. 1.

Traditional approaches to protecting computer systems from malicious or other misbehaved code typically involve (1) monitoring code for unacceptable behavior as it runs, or (2) detecting potentially misbehaved code and preventing it from running at all. These approaches are effective when unacceptable behavior can be detected in time to take remedial action, but in many settings and for many im-portant security policies this is computationally expensive or provably impossible. A third approach, termed in this dissertation program-rewriting, involves auto-matically rewriting code prior to running it in such a way that acceptable behavior is preserved but unacceptable behavior is not. Rewritten code can be run without further analysis or monitoring because it is guaranteed to exhibit only acceptable behavior. Program-rewriting has received recent attention in the literature in the form of in-lined reference monitors, which implement approach 1 above by in-lining security checks directly into the code being monitored. Program-rewriting generalizes in-lined reference monitoring, encompassing many other strategies for automatically rewriting programs as well.