Abstract. Eight sites participated in the second DARPA off-line intrusion detection evaluation in 1999. Three weeks of training and two weeks of test data were generated on a test bed that emulates a small government site. More than 200 instances of 58 attack types were launched against victim UNIX and Windows NT hosts. False alarm rates were low (less than 10 per day). Best detection was provided by networkbased systems for old probe and old denial-of-service (DoS) attacks and by host-based systems for Solaris user-to-root (U2R) attacks. Best overall performance would have been provided by a combined system that used both host- and network-based intrusion detection. Detection accuracy was poor for previously unseen new, stealthy, and Windows NT attacks. Ten of the 58 attack types were completely missed by all systems. Systems missed attacks because protocols and TCP services were not analyzed at all or to the depth required, because signatures for old attacks did not generalize to new attacks, and because auditing was not available on all hosts. 1

Abstract — The ad-hoc methodology that is prevalent in today’s testing and evaluation of network intrusion detection algorithms and systems makes it difficult to compare different algorithms and approaches. After conducting a survey of the literature on the methods and techniques being used, it can be seen that a new approach that incorporates an open source testing methodology and environment would benefit the information assurance community. After summarizing the literature and presenting several example test and evaluation environments that have been used in the past, we propose a new open source evaluation environment and methodology for use by researchers and developers of new intrusion detection and denial of service detection and prevention algorithms and methodologies.

Network intrusion detection systems have become a standard component in security infrastructures. Unfortunately, current systems are poor at detecting novel attacks without an unacceptable level of false alarms. We propose that the solution to this problem is the application of an ensemble of data mining techniques which can be applied to network connection data in an offline environment, augmenting existing real-time sensors. In this paper, we expand on our motivation, particularly with regard to running in an offline environment, and our interest in multisensor and multimethod correlation. We then review existing systems, from commercial systems, to research based intrusion detection systems. Next we survey the state of the art in the area. Standard datasets and feature extraction turned out to be more important than we had initially anticipated, so each can be found under its own heading. Next, we review the actual data mining methods that have been proposed or implemented. We conclude by summarizing the open problems in this area, along with some questions of a broader scope. We hope that by providing the motivation and summarizing the work in this area that we can stimulate further research.

and electronic copies of this document in whole or in part, and to grants others the right to do so. Author......................................................................................................................

The Lincoln Adaptable Real-time Information Assurance Testbed, LARIAT, is an extension of the testbed created for DARPA 1998 and 1999 intrusion detection (ID) evaluations. LARIAT supports real-time, automated and quantitative evaluations of ID systems and other information assurance (IA) technologies. Components of LARIAT generate realistic background user traffic and real network attacks, verify attack success or failure, score ID system performance, and provide a graphical user interface for control and monitoring. Emphasis was placed on making LARIAT easy to adapt, configure and run without requiring a detailed understanding of the underlying complexity. LARIAT is currently being exercised at four sites and is undergoing continued development and refinement.

The current Intrusion Detection System (IDS) technology is a major investment for a firm and its evaluation is desired prior to a commitment. A testbed compares different IDSs on a common platform. A major challenge in evaluating IDSs stems from the fact that they are generally tested in specific environments. A real-world environment could be different from the environment designed for a testbed. The results obtained, from such testbeds, may not be accurate and reliable. Hence, a quantitative and metrics based evaluation of IDSs is desired. We propose Testbed for evaluating Intrusion Detection Systems (TIDeS), that allows a user to select the best IDS for a specific customized environment. A quantitative analysis is provided by TIDeS, using fuzzy logic, under varying network loads. We also propose robust metrics to evaluate an IDS. We follow up with recommendations, based on our experience, on the general practices in the field of IDSs.

detection evaluations assessed the performance of intrusion detection systems using realistic background traffic and many examples of realistic attacks. This paper discusses three extensions to these evaluations. First, the Lincoln Adaptable Real-time Information Assurance Testbed (LARIAT) has been developed to simplify intrusion detection development and evaluation. LARIAT allows researchers and operational users to rapidly configure and run real-time intrusion detection and correlation tests with robust background traffic and attacks in their laboratories. Second, "Scenario Datasets" have been crafted to provide examples of multiple component attack scenarios instead of the atomic attacks as found in past evaluations. Third, extensive analysis of the 1999 evaluation data and results has provided understanding of many attacks, their manifestations, and the features used to detect them. This analysis will be used to develop models of attacks, intrusion detection systems, and intrusion detection system alerts. Successful models could reduce the need for expensive experimentation, allow proof-of-concept analysis and simulations, and form the foundation of a theory of intrusion detection.