Malicious attacks on systems are a threat to business, government, and defense. Many attacks exploit system behavior unknown to the developers who created it. In today’s state of art, software engineers have no practical means to determine how a sizable program will behave in all circumstances of use. This sobering reality lies at the heart of many problems in security and survivability. If full behavior is unknown, so too are embedded errors, vulnerabilities, and malicious code. This paper describes function-theoretic foundations for automated calculation of full program behavior. These foundations treat program control structures as mathematical functions or relations. The function, or behavior, of control structures can be abstracted in a stepwise process into procedurefree expressions that specify their net functional effects. Problems of computability and complexities of language semantics appear to have engineering solutions. Automated behavior calculation will add rigor to security and survivability engineering. 1. Understanding Program Behavior Traditional engineering disciplines depend on rigorous methods to evaluate the expressions (equations, for example) that represent and manipulate their subject matter. Yet the discipline of software engineering has no practical means to fully evaluate the expressions it produces. In this case, the expressions are computer programs, and evaluation means understanding their full behavior, right or wrong, intended or malicious. Short of substantial time and effort, no software engineer can say for sure what a sizable program does in all circumstances of use. Yet modern society is dependent on the correct functioning of countless large-scale systems composed of programs whose full behavior and security properties are

v 1 The Problem of Understanding Program Behavior ....................................1 2 Background: Function-Theoretic Foundations of FSQ Flow Structures ..3 3 Function-Theoretic Calculation of Program Behavior ...............................6 4 The Architecture of an Abstraction Engine ..............................................19 5 Using Abstraction in Automated Verifiers, Integrators, and Certifiers ...21 5.1 Automated Program Verifiers...............................................................21 5.2 Automated Program Integrators...........................................................21 5.3 Automated Program Certifiers .............................................................23 6 Acknowledgements....................................................................................24 References .........................................................................................................25 ii CMU/SEI-2003-TN-003 CMU/SEI-2003-TN-003 iii List of Figures Figure 1: A Miniature Program for Abstraction ......................................................13 Figure 2: The First Abstraction Step .....................................................................13 Figure 3: The Second Abstraction Step ................................................................14 Figure 4: The Final Abstraction Step ....................................................................14 Figure 5: The Abstracted Behavior Catalog of a Java Program ............................16 Figure 6: Architectural Structure of an Automatic Abstraction Engine ...................19 iv CMU/SEI-2003-TN-003 CMU/SEI-2003-TN-003 v Abstract No software engineer can say wi...

Survivability is the capability of an information system to support critical enterprise missions in adverse environments of attacks, failures, and accidents [Ellison et al 1999]. A research program in survivability must therefore address both systems and the environments within which they operate. Survivability is a combination of quality attributes, including security, reliability, safety, fault tolerance, dependability, and others [Mead et al 2000]. The SEI CERT Coordination Center, in cooperation with other researchers, has embarked on a multi-year, dual-thread research program, one thread to create engineering practices for survivable system design and development, the other to create engineering practices for analysis and definition of adverse environments. We believe that lack of theoretical foundations in both areas has been a serious impediment to survivable system development. In essence, we seek to move beyond natural language descriptions of survivability to a computational capability for engineering analysis of survivability properties. Accordingly, our agenda is to progress from theoretical foundations, to formal language representations, to engineering practices. We take it as an article of faith that to be effective, engineering practices must be based on rigorous foundations. At the same time, it is important to target foundations and engineering practices to the present reality and future evolution of information system architectures and technologies. Today’s large-scale infrastructure