## THEORY OF EFFECTIVENESS MEASUREMENT (2006)

### BibTeX

@MISC{Bullock06theoryof,

author = {Richard K. Bullock and Richard K. Bullock and Major Usaf},

title = {THEORY OF EFFECTIVENESS MEASUREMENT},

year = {2006}

}

### OpenURL

### Abstract

Effectiveness measures provide decision makers feedback on the impact of deliberate actions and affect critical issues such as allocation of scarce resources, as well as whether to maintain or change existing strategy. Currently, however, there is no formal foundation for formulating effectiveness measures. This research presents a new framework for effectiveness measurement from both a theoretical and practical view. First, accepted effects-based principles, as well as fundamental measurement concepts are combined into a general, domain independent, effectiveness measurement methodology. This is accomplished by defining effectiveness measurement as the difference, or conceptual distance from a given system state to some reference system state (e.g. desired

### Citations

2314 | Game Theory - Fudenberg, Tirole - 1991 |

2130 |
A new approach to linear filtering and prediction problems
- Kalman
- 1960
(Show Context)
Citation Context ...f the system. A popular filter for trying to estimate the state of a system based on uncertain and error prone measurements is known as the Kalman filter, first presented by Rudolf E. Kalman in 1960 (=-=Kalman, 1960-=-). The Kalman filter can take ‘noisy’ measurements and estimate the state of any system (Maybeck, 1979:4). A key assumption of the Kalman filter is the current system state estimate, xˆ k, is a linear... |

1110 |
The visual display of quantitative information
- Tufte
- 1983
(Show Context)
Citation Context ... for an information 32sdisplay should be to present the maximum amount of information possible while ensuring unambiguous understanding of the insights and their implications for the target audience (=-=Tufte, 1983-=-:105; Jordan, 2001:43). The key point is regardless of how impeccable the measurement plan and implementation, and regardless of how rigorous the assessment, if the insights cannot be effectively comm... |

945 | An Introduction to Support Vector Machines - Cristianini, Shawe-Taylor - 2000 |

679 | G.: An introduction to the Kalman filter - WELCH, BISHOP - 1995 |

605 |
Game Theory: Analysis of Conflict
- Myerson
- 1991
(Show Context)
Citation Context ... pursues their own interests (Luce, 1957:89). In cooperative games, however, decision 120smakers are free to form coalitions and make agreements, essentially combining their decision making problems (=-=Myerson, 1991-=-:244). Table 13. Attributes of Games Yet another way to classify Game Theory problems is by time. In general, Game Theory does not put a time constraint on the decision maker to make a decision at a d... |

498 | The balanced scorecard: Measures that drive performance - Kaplan, Norton - 1992 |

251 |
WS: Theory & methods of scaling
- Torgerson
- 1958
(Show Context)
Citation Context ... of the abstract concept of numbers (Campbell, 1957:295). The assignment of numerals then allows the formal language of mathematics to be applied, enabling further insight into the system’s behavior (=-=Torgerson, 1958-=-:1) or more specifically the system’s change in behavior, which is central to effectiveness measurement. All measurement is carried out within a context (Morse, 2003:2). This context is shaped by a pu... |

235 |
Stochastic models, estimation, and control, volume 1
- Maybeck
- 1979
(Show Context)
Citation Context ...stem state representation in the form of vectors and is often used where the internal behavior of a system cannot be observed or is not known and must be inferred from the system’s external behavior (=-=Maybeck, 1979-=-:4; Welch, 2004:1). In practice, filtering is the task of computing the current state of a system in the face of uncertainty as well as partial and noisy measurements (Zarchan, 2005:91). Mathematicall... |

203 |
Making Hard Decisions: An Introduction to Decision Analysis, 2 nd Edition
- Clemen
- 1996
(Show Context)
Citation Context ...surements, or subject matter experts on the system of interest, a series of lottery or certainty equivalent questions (Luce, 1957:21; Keeney, 1992:6) to identify indifference curves (Keeney, 1992:79; =-=Clemen, 1996-=-:540). This topic of achieving equal intervals is related to the concept of differentially value equivalence (Keeney, 1993:94) and is just an extension of the substitutability axiom of expected utilit... |

176 |
Visual Explanations: Images and Quantities, Evidence and Narrative
- Tufte
- 1997
(Show Context)
Citation Context ...alse-positive, occurs when a hypothesis is not rejected when it is false. Finally, an important, but often underemphasized aspect of system measurement is communication, or the design of information (=-=Tufte, 1997-=-:9). As noted, measurement is carried out within a context. This context could be exploratory or for a resource commitment decision. Modern word-processing, spreadsheet, and database software provide ... |

154 |
Software measurement: A necessary scientific basis
- Fenton
- 1994
(Show Context)
Citation Context ...rting its essential characteristics (Bueno de Mesquita, 1981:10; Gartner, 1997:9). Clearly, there are no definitive measures which can be prescribed for every objective across every application area (=-=Fenton, 1994-=-:200; Park, 1996:1). Because of this, effectiveness measurement concepts need to be defined in general along with the mathematical properties that characterize these concepts, regardless of the specif... |

151 | Using the balanced scorecard as a strategic management system - Kaplan, Norton - 1996 |

126 | Filtering: Theory and Practice - Grewal, Andrew, et al. - 1993 |

124 | Towards a Framework for Software Measurement Validation - Kitchenham, Pfleeger, et al. - 1995 |

85 | Foundations of measurement - Krantz, Luce, et al. - 1971 |

84 |
Differential Games: A mathematical theory with applications to warfare and pursuit, control and optimization
- Isaacs
- 1965
(Show Context)
Citation Context ...ctives. Typically, however, the objectives will have a common ordering among all the decision makers which has the effect of polarizing the decision context and essentially making it a 2-person game (=-=Isaacs, 1965-=-:306). Examples of this phenomenon are numerous. A historical example is the unlikely alliance of the USA, Soviet Union, and China during World War II. Although these ‘players’ had fundamental differe... |

75 |
Measure Theory
- Cohn
- 1997
(Show Context)
Citation Context ... all ordered pairs of vectors in S (Marlow, 1978:2). Additionally, if the second condition, δ(Si, Sj) = 0 ⇔ Si = Sj, is replaced with δ(Si, Sj) = 0 ⇒ Si = Sj, then δ is a semimetric or psudeo-metric (=-=Cohn, 1980-=-:8). It follows, (S, δ) is a metric space. The above demonstrates how mathematics, and specifically Measure Theory, defines a measure via rules or axioms. Through the use of an axiomatic approach, mea... |

75 | Foundational aspects of theories of measurement - Scott, Suppes - 1958 |

73 | Simultaneous conjoint measurement: A new type of fundamental measurement - Luce, Tukey - 1964 |

65 |
The Performance Measurement Manifesto
- Eccles
- 1991
(Show Context)
Citation Context ...em will be measured to include how measures will be determined and how measurements will be collected, as well as the allocation of resources for measurement activities to include training and tools (=-=Eccles, 1991-=-:133). The plan contains all information required to conduct system measurement within a specific context (Neely, 1997:1138) and is sometimes referred to as the measurement protocol (Kitchenham, 1995:... |

54 | The War Trap - Mesquita, Bruce - 1981 |

52 | Multivariate Analysis - Dillon, Goldstein - 1984 |

45 | Introduction to Measurement Theory - Allen, Yen - 1979 |

45 |
Abstract measurement theory
- Narens
- 1985
(Show Context)
Citation Context ...he invariance of a specific symbolic (numerical) statement across admissible transformations. Many attributes can be measured directly. These are termed extensive attributes, or fundamental measures (=-=Narens, 1985-=-:78). Other measurements may be based on assumed relations or by arbitrary definition (Torgerson, 1958:22). However, as already noted, not all attributes are easily measured. For these intensive attri... |

43 |
Logisitics Engineering and Management
- Blanchard
- 1998
(Show Context)
Citation Context ...formation (Leonard, 2004:14). Measurement, although a crucial element, is only a part of the process of system assessment (Wilbur, 1995:16). Assessment is a systematic process of monitoring a system (=-=Blanchard, 1991-=-:14). Assessment converts raw measurement data into information and knowledge yielding insight (Artley, 2001a:41). Assessment can be categorized either as enumerative or analytical (Evans, 2004:219). ... |

41 | Innovations in performance measurement: Trends and research implications - Ittner, Larcker - 1998 |

39 | Irreducible Semi-Autonomous Adaptive Combat (ISAAC): An Artificial-Life Approach to Land Warfare
- Ilachinski
- 1997
(Show Context)
Citation Context ...d Chaos Theory, which is typically implemented as a complex adaptive system using agent-based modeling, because existing models did not capture the way Marines fight with respect to maneuver warfare (=-=Ilachinski, 1997-=-). The US Air Force also explored using agent-based models to capture airpower strategic effects (Bullock, 2000). Although the efforts produced promising results (Hill, 2003:17), the non-linear scienc... |

38 |
Towards a system of systems concepts
- Ackoff
- 1971
(Show Context)
Citation Context ...construct is visualizing a network of linked input-output systems, where outputs of one system are the inputs of others (Figure 6). In fact, every system can be seen as part of another larger system (=-=Ackoff, 1971-=-:663). Thus, the combining of systems yields a larger system with its own inputs, outputs, effects, outcomes, purpose/behavior, and boundaries. However, within this larger system, each sub-system stil... |

35 |
Effects based operations: Applying network centric warfare in peace, crisis, and war
- Smith
- 2002
(Show Context)
Citation Context ...sures are key to the approach since they tie the three pillars together and are used to determine if the intended effects are being achieved and if the strategy and course of action needs adjustment (=-=Smith, 2002-=-:355). Since the planning process starts with the end goal and does not apply weapon system or target solutions during COA development, inherent to EBO is the application of operational art, allowing ... |

35 | 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and - No |

34 |
Theory of measurement
- Pfanzagl, Baumann, et al.
- 1971
(Show Context)
Citation Context ...deas about measurement within a particular domain. Measurement is an abstraction because measurement does not directly represent the system but only addresses the attributes selected to represent it (=-=Pfanzagl, 1971-=-:16). In this light, measurement can be thought of as the process of assigning symbols to the attributes of a system such that the assigned symbols reflect the underlying nature of the attributes (Caw... |

32 |
Nonlinear Systems,” 3rd Edition
- Khalil
(Show Context)
Citation Context ...rom fields such as Complexity Theory and Chaos Theory. These techniques are especially well suited for exploring the non-linear dynamics of systems which arise from repeated interaction and feedback (=-=Khalil, 2002-=-). Warfare is often characterized in such a manner (Schmitt, 1999:5). Thus, it is no surprise there have been many efforts to use the non-linear sciences to model military actions and their resulting ... |

30 | Fundamentals of Kalman filtering: a practical approach. Second edition - Zarchan, Musoff - 2005 |

25 | Influence Net Modeling With Causal Strengths: An Evolutionary Approach - Rosen, Smith - 1996 |

22 |
Value focused thinking: a path to creative decision making
- Keeney
- 1992
(Show Context)
Citation Context ...s the overall desired or expected system end-state. Alternatively, the horizontal structure is usually linked with means objectives, where a means objective is an enabler for a fundamental objective (=-=Keeney, 1992-=-:66). Typically, measures in the vertical construct are associated with system effectiveness and measures in the horizontal construct concern system efficiency. However, these structures are not exclu... |

20 |
The Mathematics of Games of Strategy: Theory and Applications
- Dresher
- 1981
(Show Context)
Citation Context ...o influence the decision at hand. Game Theory provides a framework for thinking about strategic interaction and helps formulate an optimal strategy by forecasting the outcome of strategic situations (=-=Dresher, 1961-=-:1). Thus, Game Theory concerns games of strategy versus games of pure chance such as slot machines or non-interactive games like solitaire. The idea of a general theory of games was introduced by Joh... |

20 | Dedene G.: Distance-based Software Measurement: Necessary and Sufficient Properties for Software Measures - Poels - 2000 |

19 | Designing Performance Measures: A Structured Approach - Neely, Richards, et al. - 1997 |

16 |
Keeping Score: Using the Right Metrics to Drive World-Class Performance, Quality Resources
- Brown
- 1996
(Show Context)
Citation Context ..., there are numerous challenges confounding the measurement process. The key to successful measurement is ensuring the right measures are being used to gauge the system purpose or normative behavior (=-=Brown, 1996-=-:3; Leonard, 2004:2). The goal is to understand which inputs or environmental conditions lead to which outcomes (Morse, 2003:38). The key challenge, however, is what we would like to 23smeasure and wh... |

14 | Nations at War: A - Geller, Singer - 1998 |

14 |
How the right measures help teams excel
- Meyer
- 1994
(Show Context)
Citation Context ...nge can be attributed to a system input (Neely, 28s1997:1137). However, other measures, such as process measures, are important for determining why a system is behaving the way it is (Brown, 1996:44; =-=Meyer, 1994-=-:97). • Timely – Measures should be collected and processed in a timeframe that is needed to be relevant within the context (Kaplan, 1991:73; Harbour, 1997:8). This property is at the heart of the tra... |

13 | Effects-based operations: A grand challenge for the analytical community
- Davis
- 2002
(Show Context)
Citation Context ...Influence (1966), despite his emphasis on strategic bombing of the populous as the means of influence. Value-based models have shown promise in forming the foundation of cognitive models of an enemy (=-=Davis, 2001-=-:76; Whittemore, 1999). Typically, however, value-based approaches focus on what is ‘valuable’ from a military capability standpoint (Doyle, 1997). Technically, these approaches specify tasks, objecti... |

13 |
Strategic Decision Making
- Kirkwood
- 1997
(Show Context)
Citation Context ...ncreased Sampling....................... 134 vii PagesTABLES Table 1. Scale Types (Narens, 1986:168)........................................................................ 13 Table 2. Measure Types (=-=Kirkwood, 1997-=-:24)................................................................ 26 Table 3. Effect Attributes................................................................................................. 37 T... |

12 |
Strategic Assessment in War
- Gartner
- 1997
(Show Context)
Citation Context ...etween strategy and execution, essentially translating strategy into reality (Melnyk, 2004:209). Effectiveness measures amount to ‘cognitive shortcuts’ in the face of an overwhelming complex reality (=-=Gartner, 1997-=-:43). Effectiveness measures influence how decision 2smakers assess the impact of deliberate actions and affect critical issues such as resource allocation as well as whether to maintain or change exi... |

11 |
A Framework of Software Measurement. de Gruyter
- Zuse
- 1998
(Show Context)
Citation Context ...ve to the desired end-state (Glenn, 2002; Murray, 2001; Bowman, 2002:24). History has shown theory is of limited value if not supported by an empirically feasible measurement method (Scott, 1958:113; =-=Zuse, 1998-=-:84). The challenge in military operations is the system of interest is often ill-defined, exhibiting dynamic, non-deterministic relationships. Although EBO is typically used in a military context, th... |

10 |
Effects-Based Operations: change in the nature of warfare
- Deptula
(Show Context)
Citation Context ...n military operations is called Effects-based Operations (EBO).sEffects-based Operations are activities designed to achieve specific outcomes versus activities focused on particular targets or tasks (=-=Deptula, 2001-=-a:53; Lazarus, 2005:23). EBO offers the potential to effectively and efficiently attain objectives across a wide spectrum of complex environments (Henningsen, 2003:3). Based on its potential and suppo... |

10 |
Performance measurement and performance management
- Lebas
- 1995
(Show Context)
Citation Context ...problematic since past data may not necessarily be a predictor of future system behavior (Meyer, 1997:33). Thus, analysis insights based on historical measurements assume system relations are stable (=-=Lebas, 1995-=-:26). Further, to objectively state an input had a significant system impact requires use of statistical techniques (Evans, 2004:219) yielding a confidence statement. Assessment, as well as identifica... |

10 |
Mathematics for Operations Research
- Marlow
- 1978
(Show Context)
Citation Context ...) = 0 ⇔ Si = Sj (identity), δ(Si, Sj) = δ(Sj, Si) (symmetry), and δ(Si, Sj) ≤ δ(Si, Sk) + δ(Sk, Sj) (triangle inequality), where × denotes the Cartesian product, or all ordered pairs of vectors in S (=-=Marlow, 1978-=-:2). Additionally, if the second condition, δ(Si, Sj) = 0 ⇔ Si = Sj, is replaced with δ(Si, Sj) = 0 ⇒ Si = Sj, then δ is a semimetric or psudeo-metric (Cohn, 1980:8). It follows, (S, δ) is a metric sp... |

10 |
Measurement Theory: Frequently Asked Questions. ftp://ftp.sas.com/ pub/neural/measurement.html. (Accessed 10
- Sarle
- 1997
(Show Context)
Citation Context ...erlying relationships as the system attribute being measured (Mitchell, 2003: 304). A scale (Figure 2) is a predefined mapping from one domain to another, representing empirical system relationships (=-=Sarle, 1995-=-). Because of this, measurement is closely tied to definition (Caws, 1959:3) and the family of mappings for attributes of a system can be considered a mathematical model of the system, since the embed... |