Abstract

Some important health policy topics, such as those related to the delivery, organization, and financing of health care, present substantial challenges to established methods for evidence synthesis. For example, such reviews may ask: What is the effect of forprofit versus not-for-profit delivery of care on patient outcomes? Or, which strategies are the most effective for promoting preventive care? This paper describes innovative methods for synthesizing evidence related to the delivery, organization, and financing of health care. We found 13 systematic reviews on these topics that described novel methodologic approaches. Several of these syntheses used 3 approaches: conceptual frameworks to inform problem formulation, systematic searches that included nontraditional literature sources, and hybrid synthesis methods that included simulations to address key gaps in the literature. As the primary literature on these topics expands, so will opportunities to develop additional novel methods for performing high-quality comprehensive syntheses. Ann Intern Med. 2005;142:1056-1065 www.annals.org For author affiliations, see end of text. Q uantitative systematic reviews have traditionally focused on estimating the effectiveness of specific therapeutic interventions, assessing the accuracy of diagnostic tests, and, to a lesser extent, quantifying epidemiologic relationships (1-3). Such meta-analyses lend themselves to clear formulation of problems with research questions such as the following: Does a particular intervention increase survival in patients with a particular diagnosis? What are the sensitivity and specificity of a given diagnostic test? What is the probability that patients with a particular risk factor are likely to develop a given disease? The literature searches, development of inclusion and exclusion criteria, and selection of quantitative methods for combining evidence follow readily from such research questions. By contrast, systematic reviews of some important policy issues, such as those related to the delivery, organization, and financing of health care, present 3 key challenges to established methods of systematic reviews (4). First, such reviews may require considerable effort at the problem formulation stage to narrow the scope of the question to one that can feasibly be addressed and to define appropriate inclusion and exclusion criteria that facilitate literature searches. They may address research questions such as the following: What is the relative effect of private forprofit versus private not-for-profit delivery of care on patient outcomes? Which strategies are the most effective for promoting preventive care? Second, they may not lend themselves to straightforward literature searches, as some of the literature addressing these complex questions falls outside the traditional medical literature and may be difficult to retrieve (5). For example, the relevant literature may consist principally of institutional reports, white papers, and other "grey literature," which are often not well cataloged in electronic databases. Third, the included articles may be highly heterogeneous with respect to study design (including the key reported outcomes) and quality. This can lead to difficulties appraising and interpreting the evidence and can severely limit opportunities to synthesize the data quantitatively, as well as qualitatively. There is increasing interest in using high-quality systematic reviews of topics related to the delivery, organization, and financing of health care. For example, economic evaluations of technologies are increasingly being used to inform coverage decisions by organizations such as the National Institute for Clinical Excellence (NICE) in the United Kingdom and the Australian Government's Pharmaceutical Benefits Advisory Committee (PBAC) (6). Several research groups are developing expertise in performing high-quality systematic reviews on these topics. Notably, within the Cochrane Collaboration, the Cochrane Effective Practice and Organisation of Care Group has performed numerous reviews on related topics The Agency for Healthcare Research and Quality (AHRQ) has increasingly asked its Evidence-based Practice Centers to synthesize evidence on complex, policy-related topics that do not lend themselves to traditional methods of evidence synthesis. In this review, we discuss innovative methods used by Evidence-based Practice Centers and other researchers to synthesize evidence related to the study of the organization, delivery, and financing of health care. We reviewed 94 Evidence-based Practice Center reviews and other published systematic reviews to identify reviews that specified their method for problem formulation, discussed the challenges of using nontraditional or medical literatures, and synthesized broad-based literature relevant to health care delivery, organization, or financing. In this supplement article, we review approaches used in these systematic reviews for problem formulation, data collection, and analysis and interpretation. METHODOLOGIC INNOVATIONS DESCRIBED IN PUBLISHED REVIEWS We found 13 published reviews of broad topics related to the delivery, organization, and financing of health care that specified their methods for problem formulation, data collection, and data analysis and interpretation. The topics meeting these criteria broadly included patient safety (9 -11), quality improvement (12, 13), bioterrorism preparedness (14, 15), health promotion (16), health care financing (17-19), telemedicine (20), and health literacy (21). We present the results of our review in the Appendix In the sections that follow, we summarize innovative approaches to solving the unusual methodologic challenges characteristic of these types of evidence synthesis topics. We have organized these sections according to 3 of the stages of evidence synthesis from the Cooper model of research synthesis: problem formulation, data collection, and data analysis and interpretation (22). CHALLENGE: FORMULATING RESEARCH QUESTIONS AND INCLUSION CRITERIA The problem formulation stage has 3 research objectives: development of the key research questions, construction of the definitions of the key concepts that distinguish relevant from irrelevant studies, and the establishment of inclusion and exclusion criteria for the review (22). In general, given the complexity and breadth of the topics addressed in the included reviews, the authors often described the use of conceptual frameworks for problem formulation. Specifically, they often adapted theoretical frameworks from related fields to develop conceptual models that they then used to specify key research questions and direct their literature searches. Systematic Methods for Defining Concepts and Inclusion Criteria The evidence related to health care delivery, organization, and financing often describes complex systems processes. Without clearly stated definitions of the processes of interest, it can be difficult to identify the research questions, develop appropriate literature searches, and formulate inclusion and exclusion criteria. Thus, for some projects, the essential task of the problem formulation stage is the defining of relevant concepts. For example, in the review "Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators," the authors evaluated patient safety indicators developed from administrative data (9). They began by developing a set of standardized definitions of 9 key concepts relevant to their topic, including quality of health care, quality indicators, patient safety, patient safety indicators, medical error, and adverse events (9). They used these definitions to develop a list of potential indicators of medical errors, patient safety, or potentially preventable complications (9). The authors then invited 21 professional clinical organizations and 16 surgical subspecialty organizations to submit nominations of experts to evaluate potential patient safety indicators. An expanded expert advisory group of 76 clinicians (divided into 11 panels) reviewed and evaluated indicators according to a method adapted from the RAND/UCLA Appropriateness Method (23). This consisted of an initial independent assessment of each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists, followed by a final independent assessment by clinician panelists using the same questionnaire (9). The panel process served to refine definitions of some indicators, add new measures, and dismiss indicators with major concerns from further consideration (9). Finally, a subset of 34 of the most promising indicators was developed and additional literature was sought for studies describing these indicators. This unique use of a series of expert advisory panels produced inclusion criteria with considerable face validity given the breath and depth of the clinical specialists involved in the process. Use of a Decision Analytic Framework To Define Key Research Questions The authors of the review "Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems" used an approach based on decision theory for defining the key research questions of their study Adaptation of Established Theoretical Frameworks Among the included articles, we found that many reviews adapted established theoretical frameworks to develop their conceptual models. For example, each of the 3 reviews on patient safety relied on previously existing models of patient safety, human work performance, and processes associated with human error (9 -11). Adaptation of theoretical frameworks from relevant allied literatures for problem formulation is of particular relevance for topics with literature in its infancy, such as that related to the delivery, organization, and financing of health care. The structure of the Evidence-based Practice Centers, which requires the establishment and active solicitation of input from expert advisors, is particularly effective for developing such conceptual frameworks for problem formulation. These expert panels can direct the research team to theoretical resources in related fields, advise in the adaptation of these established frameworks, and provide feedback on iterative improvements in the development and application of the conceptual model (18). CHALLENGE: IDENTIFYING PERTINENT LITERATURE The key research objectives of the data collection stage are the selection of data sources, development of search strategies for each data source, and abstraction of relevant data from retrieved articles (22). For topics related to the delivery, organization, and financing of health care, the relevant articles will almost always require searches beyond the traditional medical literature. We found 3 commonalities in the searches of the included reviews. First, the conceptual models developed during problem formulation directed the literature searches. Second, as we will describe in greater detail in the following section, several reviews used iterative search strategies. A recent systematic review of factors associated with effective diffusion of innovations emphasizes the importance of a structured search process that combines electronic searches directed by a conceptual model supplemented by input from experts with iterative refinements to the search (25). Third, most of the included reviews used a variety of types of data sources: traditional medical databases such as MEDLINE, alternative databases of relevant associated literatures (for example, psychology, economics, and engineering), structured Internet searches, and input from expert advisors. Iterative Literature Searches Directed by Problem Formulation Most of the included reviews stated that their search strategies were directly informed by their conceptual frameworks and were iterative. For example, the decision analytic framework developed in the review on information technologies for bioterrorism preparedness and response also informed the authors' literature searches (14). In this case, the authors first sought articles describing bioterrorism-relevant events such as the 2001 anthrax cases, disasters, acts of terrorism, and infectious disease outbreaks to determine the key decisions and task of clinicians and public health officials (14). They then performed additional searches to identify articles about information technologies that could assist each of these decisions (14). For systematic reviewers faced with broad topics, an initial search may help to define the key elements of heterogeneity. For example, an initial search for quality improvement strategies for diabetes may find potentially relevant articles that include evaluations directed at both inpatient and outpatient management, of type 1 and type 2 diabetes, and of pediatric and adult populations. Such a search could be used to establish inclusion criteria that minimize heterogeneity (for example, limiting the included studies to those that include an outpatient component) and direct subsequent searches. Using a Variety of Literature Sources Given the nascent nature of the quality-based purchasing literature, the authors of the review "Strategies to Sup- This figure uses influence diagram notation to depict the key decisions of clinicians and public health officials responding to bioterrorism (rectangular decision nodes), to identify the uncertain events affecting these decisions (elliptic chance nodes), and to evaluate the information that is observable by the decision makers at the time they make their decisions. The schematic depicts 3 critical time periods as follows: Time period 1 refers to the interval in which decisions are made by clinicians regarding the events associated with the initial cases; time period 2 refers to the interval in which decisions are made by public health officials regarding the events associated with the initial cases; and time period 3 refers to the interval in which decisions are made by clinicians regarding the events associated with the subsequent cases. The decisions and processes depicted in this figure could be supported by information technologies and decision support systems designed to facilitate bioterrorism preparedness and response. port Quality-based Purchasing: A Review of the Evidence" performed searches to identify ongoing research in addition to published or completed research (18). The objective of the search strategy for this project was to identify all published and ongoing randomized trials of quality-based purchasing strategies. In addition to traditional medical literatures referenced in MEDLINE and the Cochrane databases, the authors searched Web sites of relevant organizations (for example, employer organizations pursuing quality-based purchasing) and reference lists provided by their expert advisors (18). To perform this search, they relied primarily on Web-based searches of organizations Top. An analytic framework with the linkages between antecedent factors (generally not modifiable), work environment conditions (which can either impede or improve work quality, and are therefore denoted as "demands" or "resources"), and the resultant influence on patient safety and outcomes. Bottom. Conceptual model of patient safety derived from several sources. The research team used these representations to develop a series of research questions focusing on the effects of the 5 categories of working conditions on patient outcomes related to patient safety, rate of medical error, rate of error recognition, and probability of adverse events. For example, one key question of this review was the following: Does the complexity of the plan of care influence whether working conditions affect patient outcomes that are related to patient safety? An asterisk denotes where this key question fits in the conceptual framework. ten not yet been referenced in the electronic databases), and can direct systematic reviewers to unpublished data. Several specialized databases may facilitate directed literature searches on topics related to the delivery, organization, and financing of health care. A review of these is outside the scope of this paper; however, we briefly highlight 4 databases that may be of particular interest to the systematic reviewer of related topics. First, the Cochrane Collaboration's Specialized Register of Effective Practice and Organization of Care (EPOC) maintains a database of "educational, behavioral, financial, organizational, and regulatory interventions designed to improve health professional practice and the organization of health care services" (7). Interested systematic reviewers can submit a search request to the EPOC trials search coordinator. Second, the Information Service Centre for Reviews and Dissemination at the University of York has recently prepared a review of resources of the health economics literature, including electronic sources of international health economics evaluations and repositories of research papers and working papers in health economics (26). This is in addition to the 3 other databases that can be searched directly from their Web site, including registries of health technology assessments and economic evaluations. Third, the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) of the University of London performs systematic reviews on topics related to health promotion interventions (27). For systematic reviewers interested in topics related to health promotion, the EPPI-Center has contributed health promotion trials to the searchable Trial Registry (CENTRAL) of the Cochrane Library. Additionally, they have developed a publicly available database of these trials (Trials Register of Promoting Health Interventions) (TROPHI) (28). Finally, reviewers addressing policy-oriented questions related to implementation research from fields such as education, psychology, and criminology may be interested in the 2 databases of intervention trials and policy evaluations maintained by the Campbell Collaboration (29). CHALLENGE: SYNTHESIZING AND INTERPRETING DATA The key research objective of the data analysis and interpretation stage is to evaluate the included articles for heterogeneity and select appropriate methods for combining homogeneous studies (22). The published literature on the delivery, organization, and financing of health care often comprises heterogeneous interventions described in small studies that often differ significantly from each other with respect to the study design and outcomes evaluated. This heterogeneity often prevents quantitative combination of studies. Thus, most of the included reviews present their data descriptively. For example, in the review of quality-based purchasing, given the very small number of included articles, the authors briefly summarized each individual article. In addition, organizing the included studies according to the key criteria developed in the conceptual framework can contribute significantly to an appreciation of the relevant literature. For example, the review on information technologies for bioterrorism preparedness organized the included articles according to the key types of decisions the technologies were designed to facilitate (for example, detection, diagnosis, management, surveillance, and communication). The Government Accounting Office recently used this classification system to evaluate federally funded information technologies for biodefense (30). Two ongoing projects may provide additional insight and recommendations for the systematic reviewer interested in descriptive methods or narrative synthesis: One project sponsored by the Information Service Centre for Reviews and Dissemination at the University of York is attempting to developing methods for the narrative synthesis of quantitative and qualitative data in systematic reviews of effects (31). Second, the Cochrane/Campbell Qualitative Methods Network is actively working to develop methodologic standards for searching, appraising, and combining data from qualitative research for systematic reviews (32). We also direct interested readers to a recent review by Dixon-Woods and colleagues that discusses the advantages and disadvantages of various methods for synthesizing qualitative data, including thematic analysis, qualitative comparative analysis, and Bayesian meta-analysis (33). Study Assessment Users of systematic reviews such NICE and PBAC have identified an increasing need for guidance in judging the quality of technology assessments and economic evaluations (6). Among the included studies, we found that the authors often used selected questions from published scales (34) to describe the quality of the literature in general or to stratify included studies before qualitative synthesis. This approach is consistent with the general movement away from the use of global quality scores as a weight in metaanalyses (35). However, the heterogeneity of studies used to evaluate the organization, delivery, and financing of health care can make it difficult to identify a short list of quality features that apply equally well to all relevant studies. Pooling Heterogeneous Interventions and Outcomes Because of the heterogeneity of the study designs and outcome measures encountered by syntheses of topics re- lated to the organization, delivery, and financing of health care, systematic reviewers typically rely on methods that use the direction of the effect rather than the magnitude of the effect to combine studies quantitatively. Such analyses amount to "vote counting," the weakest and potentially most misleading form of quantitative synthesis (36). The authors of "The Quality Gap: A Critical Analysis of Quality Improvement Strategies," a recent synthesis of the evidence supporting quality improvement strategies for diabetes and hypertension, faced this issue as the multiple studies of changes in patient and provider behavior targeted differing processes of care The focus on the incremental benefit of a particular quality improvement strategy represents an advance over analyses that focus on single quality improvement strategies despite their coming from complex interventions (38-40), but it relies on 2 assumptions: that the combinations of strategies are close to random (for example, that it is not the case that 2 particular strategies always appear together) and that the effects of the strategies are additive. From an analytic perspective, this approach of summarizing effect using a median effect size is not as susceptible to the effects of outliers; however, it does not incorporate information about study size. To address this limitation, the authors of the quality improvement review stratified the results by study size and other design characteristics to evaluate whether the median effects of large studies differed from those of small studies. Simulations Fill Gaps in Available Evidence Several included reviews developed simulation models to address gaps in the primary literature (15, 18). For example, in the review "Regionalization of Bioterrorism Preparedness and Response," the authors found minimal published evidence that specifically addressed key policy questions regarding the development of local inventories of medical and pharmaceutical supplies for bioterrorism responses (15). Therefore, they used the data collected from the systematic review on the expected progression of disease for a single bioterrorism agent (anthrax) to develop a dynamic compartmental model (41, 42) that evaluates anthrax disease progression in a population of exposed and potentially exposed individuals. The model calculates expected anthrax-associated mortality and costs given assumptions about stockpiling of local antibiotics, event detection, and strategies for dispensing antibiotics. Assumptions were informed by the available published evidence and expert opinion. This model enabled the authors to investigate several additional policy questions of national relevance for which there is currently very limited evaluative evidence but for which policy decisions are being formulated. These included questions of the effect of early detection of a covert bioterrorism attack on expected mortality (for example, the role of bioterrorism surveillance systems) and the effect of poor adherence to prophylactic antibiotics on expected mortality. The results of the model in combination with the results of the systematic review highlighted key gaps in the literature and the key targets of future research by identifying those assumptions to which the model results were most sensitive, and for which there was little or no published literature to inform the assumptions. The use of simulation models may be helpful for topics related to the delivery, organization, and financing of health care because the literature in these areas is somewhat less developed than that in other medical fields. Models can serve to complement the often limited primary evidence by highlighting data that may affect the conclusions of the systematic review, and thus identify key targets for future research. Although model-based analyses can identify and help prioritize targets for research, when data for key model inputs are lacking, other conclusions drawn from the model will usually be limited. CONCLUSION AND RECOMMENDATIONS We identified several key approaches used in rigorous syntheses of topics related to the delivery, organization, and financing of health care. We did not attempt to evaluate whether the methodologic approaches used in the included reviews are better or worse than alternative approaches. We described the methods used and, on the basis of our findings, we make the following recommendations about potentially useful methodologic approaches for the syntheses of broad-based health care topics First, start with a conceptual model. Seek established theoretical frameworks from relevant fields to provide structure to the problem formulation and data analysis stages of the research. For example, the use of a decision analytic framework may facilitate problem formulation of topics that include multiple decision makers and multiple decisions whose outcomes may be related, or that require the explicit description of information required for decision making. Second, adopt an iterative approach to problem formulation and literature searches. Initial (somewhat broad) literature searches can assist in problem formulation. Subsequent, tailored searches can then be performed for relevant studies. The use of expert advisors is particularly helpful in terms of receiving guidance after iterations of the problem formulation and literature searches. Third, identify relevant literature sources outside those referenced in MEDLINE and other traditional medical bibliographic databases. The Appendix Table (available at www.annals.org) provides example data sources. Fourth, use the conceptual model to organize narrative descriptions of the included studies. Fifth, consider using selected, validated evaluation criteria to help organize groups of studies and select them for either qualitative or quantitative synthesis. Sixth, if quantitative synthesis is possible, consider collapsing related outcomes to facilitate multivariate assessments of the effects of interventions on complex processes. If this type of quantitative analysis is not possible, model-based analyses may be useful. Seventh, consider simulation models that use data from the review to explore key gaps in the literature and offer insights to policymakers and people who fund research. Topics related to health care delivery, organization, and financing pose inherent challenges for systematic reviews. Problem formulation may be difficult, evaluative studies may be scarce, and the common quantitative approaches for meta-analysis may not be applicable. Nonetheless, we found systematic reviews that made important contributions and helped provide a coherent overview of the studies they assessed. As the literature in these areas grows, so will the opportunities to refine conceptual frameworks for problem formulation, to develop rigorous methods for searching nontraditional literature sources, and to explore multivariate methods for data synthesis.