Malicious attacks on systems are a threat to business, government, and defense. Many attacks exploit system behavior unknown to the developers who created it. In today’s state of art, software engineers have no practical means to determine how a sizable program will behave in all circumstances of use. This sobering reality lies at the heart of many problems in security and survivability. If full behavior is unknown, so too are embedded errors, vulnerabilities, and malicious code. This paper describes function-theoretic foundations for automated calculation of full program behavior. These foundations treat program control structures as mathematical functions or relations. The function, or behavior, of control structures can be abstracted in a stepwise process into procedurefree expressions that specify their net functional effects. Problems of computability and complexities of language semantics appear to have engineering solutions. Automated behavior calculation will add rigor to security and survivability engineering. 1. Understanding Program Behavior Traditional engineering disciplines depend on rigorous methods to evaluate the expressions (equations, for example) that represent and manipulate their subject matter. Yet the discipline of software engineering has no practical means to fully evaluate the expressions it produces. In this case, the expressions are computer programs, and evaluation means understanding their full behavior, right or wrong, intended or malicious. Short of substantial time and effort, no software engineer can say for sure what a sizable program does in all circumstances of use. Yet modern society is dependent on the correct functioning of countless large-scale systems composed of programs whose full behavior and security properties are

Technology on Next-Generation

v 1 The Problem of Understanding Program Behavior ....................................1 2 Background: Function-Theoretic Foundations of FSQ Flow Structures ..3 3 Function-Theoretic Calculation of Program Behavior ...............................6 4 The Architecture of an Abstraction Engine ..............................................19 5 Using Abstraction in Automated Verifiers, Integrators, and Certifiers ...21 5.1 Automated Program Verifiers...............................................................21 5.2 Automated Program Integrators...........................................................21 5.3 Automated Program Certifiers .............................................................23 6 Acknowledgements....................................................................................24 References .........................................................................................................25 ii CMU/SEI-2003-TN-003 CMU/SEI-2003-TN-003 iii List of Figures Figure 1: A Miniature Program for Abstraction ......................................................13 Figure 2: The First Abstraction Step .....................................................................13 Figure 3: The Second Abstraction Step ................................................................14 Figure 4: The Final Abstraction Step ....................................................................14 Figure 5: The Abstracted Behavior Catalog of a Java Program ............................16 Figure 6: Architectural Structure of an Automatic Abstraction Engine ...................19 iv CMU/SEI-2003-TN-003 CMU/SEI-2003-TN-003 v Abstract No software engineer can say wi...