Operating systems should provide system transactions to user applications, in which user-level processes execute a series of system calls atomically and in isolation from other processes on the system. System transactions provide a simple tool for programmers to express safety conditions during concurrent execution. This paper describes TxOS, a variant of Linux 2.6.22, which is the first operating system to implement system transactions on commodity hardware with strong isolation and fairness between transactional and non-transactional system calls. System transactions provide a simple and expressive interface for user programs to avoid race conditions on system resources. For instance, system transactions eliminate time-of-check-to-time-of-use (TOCTTOU) race conditions in the file system which are a class of security vulnerability that are difficult to eliminate with other techniques. System transactions also provide transactional semantics for user-level transactions that require system resources, allowing applications using hardware or software transactional memory system to safely make system calls. While system transactions may reduce single-thread performance, they can yield more scalable performance. For example, enclosing link and unlink within a system transaction outperforms rename on Linux by 14 % at 8 CPUs.

We defeat two proposed Unix file-system race condition defense mechanisms. First, we attack the probabilistic defense mechanism of Tsafrir, et al., published at USENIX FAST 2008[26]. We then show that the same attack breaks the kernel-based dynamic race detector of Tsyrklevich and Yee, published at USENIX Security 2003[28]. We then argue that all kernel-based dynamic race detectors must have a model of the programs they protect or provide imperfect protection. The techniques we develop for performing these attacks work on multiple Unix operating systems, on uni- and multi-processors, and are useful for exploiting most Unix file-system races. We conclude that programmers should use provably-secure methods for avoiding race conditions when accessing the file-system. 1.

Operating systems can efficiently provide system transactions to user applications, in which user-level processes can execute a series of system calls atomically and in isolation from other processes on the system. The effects of system calls performed during a system transaction are not visible to the rest of the system (other threads or hardware devices) until the transaction commits. This paper describes TxOS, a variant of Linux 2.6.22, which is the first operating system to implement system transactions on commodity hardware with recent techniques from the transactional memory literature. The paper demonstrates that system transactions can solve problems in a wide range of domains, including security, isolating extensions, and user-level transactional memory. We also show that combining semantically lightweight system calls to perform heavyweight operations can yield better performance scalability: for example, enclosing link and unlink within a system transaction outperforms rename on Linux by 14 % at 8 CPUs. 1

Process races occur when multiple processes access shared operating system resources, such as files, without proper synchronization. We present the first study of real process races and the first system designed to detect them. Our study of hundreds of applications shows that process races are numerous, difficult to debug, and a real threat to reliability. To address this problem, we created RACEPRO, a system for automatically detecting these races. RACEPRO checks deployed systems in-vivo by recording live executions then deterministically replaying and checking them later. This approach increases checking coverage beyond the configurations or executions covered by software vendors or beta testing sites. RACEPRO records multiple processes, detects races in the recording among system calls that may concurrently access shared kernel objects, then tries different execution orderings of such system calls to determine which races are harmful and result in failures. To simplify race detection, RACEPRO models under-specified system calls based on load and store micro-operations. To reduce false positives and negatives, RACEPRO uses a replay and go-live mechanism to distill harmful races from benign ones. We have implemented RACEPRO in Linux, shown that it imposes only modest recording overhead, and used it to detect a number of previously unknown bugs in real applications caused by process races. Categories and Subject Descriptors: D.2.4 [Software Engineering]: Software/Program Verification;

Time Of Check To Time Of Use (TOCTTOU) race conditions for file accesses in user-space applications are a common problem in Unix-like systems. The mapping between filename and inode and device is volatile and can provide the necessary preconditions for an exploit. Applications use filenames as the primary attribute to identify files but the mapping between filenames and inode and device can be changed by an attacker. DynaRace is an approach that protects unmodified applications from file-based TOCTTOU race conditions. DynaRace uses a transparent mapping cache that keeps additional state and metadata for each accessed file in the application. The combination of file state and the current system call type are used to decide if (i) the metadata is updated or (ii) the correctness of the metadata is enforced between consecutive system calls. DynaRace uses user-mode path resolution internally to resolve individual file atoms. Each file atom is verified or updated according to the associated state in the mapping cache. More specifically, DynaRace protects against race conditions for all file-based system calls, by replacing the unsafe system calls with a set of safe system calls that utilize the mapping cache. The system call is executed only if the state transition is allowed and the information in the mapping cache matches. DynaRace deterministically solves the problem of file-based race conditions for unmodified applications and removes an attacker’s ability to exploit the TOCTTOU race condition. DynaRace detects injected alternate inode and device pairs and terminates the application.