Contents
A user’s guide
Alejandro R Jadad
6 From individual trials to groups of trials: reviews, meta-analyses, and guidelines
- Use bibliographic databases, prepared summaries, and common interest groups to access and use RCTs.
- Narrative and systematic reviews are the two major types of review articles used as tools to guide health care decisions.
- Meta-analysis increases the precision and conclusions of a review but can be prone to bias.
- The quality of reviews should be assessed.
- The Cochrane Library is the most comprehensive source of evidence, especially from RCTs, needed to make informed health care decisions.
- Clinical practice guidelines based on sound evidence help practitioners improve the care they offer patients.
Often you will find more than one trial that addresses your question or a very similar research question. As these trials are conducted in different groups of people, in different settings, and use the interventions differently, it is unlikely that they will provide identical results. Sometimes, different trials on the same topic have totally opposite results. The corollary is that it may be risky for you and for your patients to make decisions based on the information from a single trial. If you want to make decisions based on the best available knowledge, I am sure that you would like to consider as many relevant trials as possible. You will also want to take into account other types of information (see Chapter 7). Identifying and synthesising the information from all relevant trials to guide a particular decision are not, however, easy tasks. In this chapter, I introduce you to the identification of trials and the role of reviews of multiple trials to guide health care decisions. As in previous chapters, most of the issues that I discuss deserve a chapter and have been addressed more extensively elsewhere. My intention here, as in the rest of the book, is to highlight the most relevant information, directing you to more comprehensive sources that you could consult at your own convenience. Let's start.
What are the main impediments to identifying all relevant trials on a given topic?The main problem derives from the speed with which the literature is growing. It has been estimated, for example, that over 2 million articles and more than 17,000 biomedical books are published annually.1,2 It is difficult to estimate the total number of trials that have been completed to date, but it is thought to be in the hundreds of thousands.3 In some areas, the time it takes for the number of published trials to double is less than 10 years.4
This information ‘explosion’ is compounded by the fact that there is not a single source of information that could provide easy and reliable access to all randomised controlled trials (RCTs) on a given topic. All existing databases are incomplete (one of the main problems being the poor access to unpublished trials) and use coding systems that cannot cope with the diversity of topics in health care.
What is the best source of RCTs?Perhaps the most advanced and comprehensive source of RCTs in health care is the Cochrane Controlled Trials Database, which contains citations for more than 150,000 controlled trials identified through the collective effort of members of the Cochrane Collaboration (see below) to improve the identification of primary studies. This has been achieved not only through the development of high yield strategies to search bibliographic databases, but also through extensive hand searching of journals to identify studies that cannot be identified efficiently by electronic searches or that are not indexed in bibliographic databases. The complete database is available on CD-ROM and is updated four times a year. More information on this database and other products of the Cochrane Collaboration can be obtained from the Internet at: http://hiru.mcmaster.ca/cochrane.
Other sources of information that you could use to identify citations of RCTs are traditional bibliographic databases such as MEDLINE or EMBASE. You can now search MEDLINE, free of charge, on the Internet. There are many providers of access to MEDLINE, but you may want to access it directly through the US National Library of Medicine at http://ncbi.nlm.nih.gov/PubMed. At this site, you could access using search strategies specifically designed to optimise the yield of clinically useful studies. To access these strategies, go to http://ncbi.nlm.nih.gov/PubMed/clinical.html. You can also access MEDLINE through other vendors that provide more comprehensive and expensive services, including access to full text articles.5
Even if you identify citations for all the studies you require to inform a particular decision, an additional problem is the time required to obtain hard copies of the articles and to read them. Sometimes, key articles are published in languages that you may not understand. You would have to ignore them or invest resources to have them translated. You will have to face all these problems if you decide to identify the trials on your own. On the other hand, you may choose to explore other options that could facilitate your efforts to identify and synthesise multiple trials.
Are there ways to make it easier to find and use multiple RCTs?Fortunately, there are several options that can give you access to information from many trials with less effort and hassle. One of these options is to read a summary prepared by others, relying on those who have already spent time, money, and energy to summarise information from multiple trials on the topic. Alternatively, you could join a group of people with common interests and review a particular topic or group of topics with them, sharing the workload and resources. In the following sections, I concentrate on review articles in general, highlighting the different types of reviews and their strengths and weaknesses.
What are the different types of reviews?As a result of the rapid expansion of the literature and the progressive compression of our time and resources, review articles are becoming increasingly attractive tools to guide health care decisions. As with any other tools, however, reviews can be well built or defective, and can be well used or abused.
In theory, there are two major types of reviews: narrative and systematic. In practice, most reviews share elements of these two major types. These reviews may consider only RCTs, or RCTs and other study designs. In this chapter, I discuss some general principles of reviews, but focus on reviews of RCTs. In Chapter 7, I introduce you to the relationship between RCTs and other study designs, as well as other sources of information.
What is a narrative review?
A narrative review is the typical review article that you find in most journals. These reviews are produced by individuals who are often considered to be experts in a given field, using idiosyncratic, often informal and subjective, methods to collect and interpret information. They appeal to reviewers because they are relatively easy and quick to produce. They are also attractive to readers because they distill the views of an authority in a field in a short piece, saving the readers time and effort. The main problem with these reviews, however, is that the reader must take them at face value, because they are impossible to replicate. This could be a real problem in an era of information overload in which we make increasingly complex decisions and need to be increasingly accountable. In fact, it has been shown recently that narrative reviews are not only incomplete,6 but that they can delay the identification of effective or harmful interventions by 10-15 years when compared with more systematic approaches to reviewing the literature.7,8
What is a systematic review?
A systematic review, in its ideal form, is a review that includes an explicit and detailed description of how it was conducted so that any interested reader would be able to replicate it. The ideal systematic review should incorporate strategies to minimise bias and to maximise precision. In its purest form, the report of a systematic review should include a clear research question, criteria for inclusion or exclusion of primary studies, the process used to identify primary studies, the methods used to assess the methodological quality of the selected studies, and the methods used to extract and summarise the results of primary trials on which the conclusions are based.9 Entire series of articles devoted to systematic reviews have been published recently in major peer-reviewed journals (worth mentioning are series published by the British Medical Journal in 1994 and by the Annals of Internal Medicine in 1997).
Systematic reviews clearly overcome the major limitations of narrative reviews. Their main disadvantage is that they require more time and resources to prepare than do their narrative counterparts.
What is meta-analysis?Meta-analysis is a name that is given to any review article in which the results of several independent studies are combined statistically to produce a single estimate of the effect of a particular intervention or health care situation. In other words, the results of independent studies are lumped together to produce a number or a graph (usually with confidence intervals) that summarises the effect of the intervention. Other names given to meta-analysis include overview, quantitative overview, pooling, pooled analysis, integrative research review, research integration, research consolidation, data synthesis, quantitative synthesis, and combining studies.10
The main purpose of meta-analysis is to increase the precision of the conclusions of a review. This increased precision could do the following:
- Make comparisons of interventions more objective and accurate.
- Help resolve controversies arising from studies with conflicting results.
- Enable clinicians and patients to make better decisions.
- Guide clinical research by generating new hypotheses.
- Identify areas in which insufficient research has been performed or in which additional research may not be necessary.
Meta-analyses also have limitations. Some of these limitations result from the limitations of the individual RCTs that they combine, because the trials could have insufficient sample sizes, biased designs, be reported incompletely, or try to answer irrelevant questions. Other limitations arise from the way in which meta-analyses are designed and conducted. Meta-analyses, for example, could be prone to bias of many types and from many sources. A catalogue of the biases that can affect meta-analyses was compiled recently and is presented elsewhere,11 and empirical evidence is emerging to support their existence and importance.12-14 Another limitation of meta-analyses is that they usually require the use of statistical techniques that are still poorly understood by reviewers and readers. A useful description of procedures used in meta-analyses was published recently and I would encourage you to read it.15 The BMJ also published a series of articles on meta-analysis in 1997 and 1998.
Are systematic reviews and meta-analysis the same thing?Even though most people use the terms interchangeably, systematic reviews and meta-analyses are not synonyms. Let's elaborate on this a little.
How can a systematic review not be a meta-analysis?
A review may incorporate state-of-the-art strategies to minimise bias and to maximise precision, but, at the end, the reviewer may decide that the results of the RCTs included should not be combined. Data combination may be inappropriate for many reasons. For example, the trials may be too different in terms of their eligibility criteria, interventions, outcome time-points, the volume of data available, or methodological quality. Assessing how much heterogeneity exists among trials included in a review and assessing whether they should be combined make up one of the crucial steps in a systematic review. If, after evaluating the characteristics of the included trials, the reviewer decides that the trials can be combined, the systematic review includes meta-analysis and could also be called a systematic quantitative review. If a decision is made against combining the trials, the review is still systematic and should perhaps be called a systematic qualitative review. These two terms are important for two reasons: first, they define the distinction between meta-analyses that do and do not result from a systematic and scientific review process; and, second, they highlight the fact that not all systematic processes for reviewing a particular body of scientific evidence should necessarily lead to the statistical combination of data across studies.
Is it possible to use meta-analysis in a review that is not systematic?
The minimum requirement to produce a meta-analysis is the availability of data from two or more studies, irrespective of whether they are being reviewed narratively or systematically. For instance, a reviewer may decide to combine data from two studies that he found on top of his desk on the morning he was to submit the review for publication. In this case, the resulting review could be called a meta-analysis, but it would be far from a systematic review.
Multiple studies during the past decade have shown consistently that most meta-analyses published in peer-reviewed journals lack several of the components of a rigorous systematic review.13,16-18 In response to this evidence, multiple efforts are being made to improve the methodological quality of meta-analyses. Perhaps the most important effort is that of an initiative known as QUOROM (Quality of Reporting Meta-analyses), which is similar to the CONSORT efforts to improve the quality of randomised trials. For more information on QUOROM, you can contact the coordinating office at the University of Ottawa (David Moher, Children's Hospital of Eastern Ontario Research Institute, 401 Smyth Road, Ottawa, Ontario, K1H 8L1, Canada).
Needless to say, inappropriate meta-analyses may result in more harm than good. Reviewers should understand that a systematic qualitative review of the literature, in its own right, is a more effective way to summarise the evidence than an inappropriate or misleading meta-analysis.
How can you evaluate the quality of a review?The assessment of the quality of reviews, whether narrative or systematic, has the same challenges as the assessment of the quality of individual trials (see Chapter 4). There are several published instruments for assessing the quality of reviews, but only one has been extensively validated,19 and published elsewhere.13 This instrument, with slightly modified instructions, is shown in Fig 6.1. Modifications in the instructions were made during the course of several empirical methodological studies to maximise interrater agreement. Another version of the instrument, in this case with major modifications, is included as part of the ‘Users' guides to the medical literature’.20
Do reviews on the same topic always agree?As the number of published systematic reviews increases, you will often find more than one systematic review addressing the same or a very similar therapeutic question. Despite the promise for systematic reviews to resolve conflicting results generated by primary studies, conflicts among reviews are now emerging. These conflicts produce difficulties for decision makers who rely on these reviews to help them make choices among alternative health interventions where experts and individual trials disagree. In a recent article, I proposed a tool in the form of a decision algorithm that I developed with two of my colleagues at McMaster, which could help in the interpretation of discordant reviews (Fig 6.2).21
Can rigorous reviews eliminate the need for further trials?This is one of the most controversial and complex issues in health care research today. The controversy goes on throughout a continuum. At one end are those who insist that if a rigorous meta-analysis shows evidence of effectiveness or harm for an intervention, it would be unethical to conduct another study.7 At the other end are those who regard meta-analysis, particularly of small trials, as untrustworthy and advocate for rigorous mega-trials to determine the effectiveness or harm of health care interventions.22 This controversy has been fuelled recently by the publication of several studies showing frequent discrepancies in the results of meta-analyses of small trials and large randomised controlled trials.14,23-25 Possible reasons for the discordance between meta-analyses and large trials include major differences in the protocols and research questions, publication bias, and the inclusion of patient populations with different levels of risk in small and large trials.23
What is the role of the Cochrane Collaboration in all this?The Cochrane Collaboration is an international organisation that aims to help people make informed decisions about health, by preparing, maintaining, and ensuring the accessibility of rigorous, systematic, and up to date reviews (and, where possible, meta-analyses) of the benefits and risks of health care interventions.26-29 It was founded at a meeting of about 80 people from several countries who gathered in Oxford, England, in the fall of 1993.28 The Collaboration is named after a physician-epidemiologist, Archie Cochrane, who more than 15 years ago criticised the medical profession for not having organised ‘a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials’ to guide their clinical decisions.30
Since its creation, the Cochrane Collaboration has undergone an unprecedented growth. The Collaboration holds such promise to facilitate health care decision making that it has been described as ‘an enterprise that rivals the Human Genome Project in its potential implications for modern medicine’.31 Nevertheless, this rapidly developing organisation is also experiencing ‘growing pains’ and facing important challenges.29 The structure of the Collaboration, and its recent achievements and challenges, are described elsewhere.32
The main product of the Cochrane Collaboration is the Cochrane Library. As mentioned above, this is a regularly updated electronic library designed to give decision makers the evidence they need to make informed health care decisions, with special emphasis on data from RCTs. Launched in April 1995 as the Cochrane Database of Systematic Reviews (CDSR), it was renamed to reflect the inclusion of additional, important, related databases, making it perhaps the most comprehensive source of evidence for all those interested in evidence based health care (see Chapter 7). The Cochrane Library is issued quarterly and contains general information on the Cochrane Collaboration, a handbook on how to conduct systematic reviews, and the following four databases.32
The Cochrane Database of Systematic Reviews (CDSR) This is a rapidly growing collection of regularly updated systematic reviews of the effects of health care prepared by members of Collaborative Review Groups. It also includes protocols of reviews in progress.
The Database of Abstracts of Reviews of Effectiveness (DARE) This is a database of structured abstracts of thousands of systematic reviews from around the world, all of which have been completed outside of the Cochrane Collaboration. These reviews have been approved by reviewers at the National Health Service Centre for Reviews and Dissemination at the University of York, England. DARE also includes brief records of reviews that may be useful for background information, abstracts of reports of health technology agencies world wide, and abstracts of reviews in the journals ACP Journal Club and Evidence-Based Medicine (see Chapter 7).
The Cochrane Controlled Trials Register See above for information.
The Cochrane Review Methodology Database (CRMD) This is a database that contains hundreds of citations of articles on the science of research synthesis and on practical aspects of preparing systematic reviews.The Cochrane Library is available on CD-ROM for Windows and Macintosh and should be regarded as an evolving product. More details on the Cochrane Library and on distributors can be found on the Internet: http://hiru.mcmaster.ca/cochrane.
Should systematic reviews include only RCTs?Even though most systematic reviews in health care focus on therapy and RCTs, the same approach could and should be used to summarise information produced in any other area of health care and by any other type of study design. It is still unclear, however, whether and how to use meta-analysis to complement systematic reviews in areas other than therapy (that is, diagnostics, screening, natural history, health economics) and for studies other than RCTs.
What is the difference between systematic reviews and clinical practice guidelines?Clinical practice guidelines have been defined as ‘systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances’.33 These statements are usually presented as ‘recommendations’ for clinical practice and policy. Clinical practice guidelines represent a step beyond systematic reviews when their recommendations integrate research evidence from systematic reviews with the values and preferences of the developers of the guidelines and the context in which the guidelines will be applied. Systematic reviews should not include recommendations for practice or policy, but should be limited to providing a summary of the data available in the studies included and perhaps only suggestions for practice and research. Instead, they should be used as a component of clinical practice guidelines, providing the ‘evidence base’ from which recommendations are derived. Despite this, do not be surprised if you find systematic reviews with very strong recommendations for clinical practice and policy.
Apart from an explicit process for deriving guidelines from the evidence, guidelines should consider all important management options. Guidelines on attention deficit/hyperactivity disorder, for example, should not simply look at the role of stimulants, given that there are other treatment options, including other pharmacological (for example, antidepressants) and non-pharmacological (for example, dietary, behavioural, or cognitive) therapies. Guidelines should also provide information on the strength of the evidence in support of a recommendation and the extent of the potential impact on clinical practice and policy if the recommendation is implemented.34
Developing guidelines necessarily involves relatively small numbers of people with a limited range of views and skills. This is why it is important that the recommendations from such groups be evaluated and modulated by external review and comment from others who are interested in the problems addressed by the guidelines (for example, a wide range of practitioners, managers, policy makers, and patients or their representatives) and tested in the field in which they are to be implemented.
Many practitioners distrust practice guidelines, and it is true that some guidelines should be distrusted. As with any other tool, including RCTs and systematic reviews, guidelines can be misused, especially when they are not based on a rigorous and objective analysis of current best evidence or when they are used in settings very different from the one in which they were developed. An increasing number of guidelines, however, provide practical recommendations based on sound evidence. Some, quite appropriately, even incorporate feedback from practitioners, and provide separate clinical and policy recommendations.35 These sound evidence based guidelines can help practitioners to improve the care that they offer their patients.
References
1. Mulrow CD. Rationale for systematic reviews. In: Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publishing Group, 1995;1-8.
2. Lowe HJ, Barnett O. Understanding and using the medical subject subheadings (MeSH) vocabulary to perform literature searches. JAMA 1994;271:1103-8.
3. Jadad AR, Rennie D. The randomized controlled trial gets a middle-aged checkup. JAMA 1998;279:319-20.
4. Jadad AR, Carroll D, Moore RA, McQuay HJ. Developing a database of published reports of randomised clinical trials in pain research. Pain 1996;66:239-46.
5. Haynes RB, Jadad AR, Hunt DL. What's up in medical informatics? Can Med Assoc J 1997;157:1718-19.
6. Hillier TLB, Jadad AR. The development of a database on the measurement of pain. Supportive Care in Cancer 1996;3:246.
7. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med 1992;327:248-54.
8. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA 1992;268:240-8.
9. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Assoc J 1988;138:697-703.
10. Jenicek M. Meta-analysis in medicine: where we are and where we want to go. J Clin Epidemiol 1989;42:35-44.
11. Felson DT. Bias in meta-analytic research. J Clin Epidemiol 1992;45:885-92.
12. Simes RJ. Confronting publication bias: a cohort design for meta-analysis. Statist Med 1987;6:11-29.
13. Jadad AR, McQuay HJ. Meta-analysis to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol 1996;49:235-43.
14. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34.
15. Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Inter Med 1997;127:820-6.
16. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280.
17. Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM. The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA 1995;274:1942-8.
18. Sacks HS, Berrier J, Reitman D, Pagano D, Chalmers T. Meta-analyses of randomized control trials: An update of the quality and methodology. In: Bailar JC III, Mosteller F, eds. Medical uses of statistics. Boston, MA: New England Medical Journal Publications, 1992;427-42.
19. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44:1271-8.
20. Oxman AD, Cook DJ, Guyatt GH for the Evidence-Based Medicine Working Group. Users' guides to the medical literature. VI. How to use an overview. JAMA 1994;272:1367-71.
21. Jadad AR, Cook D, Browman GP. When arbitrators disagree: a guide to interpreting discordant systematic reviews of health care interventions. Can Med Assoc J 1997;156:1411-16.
22. Bailar JC III. The promise and problems of meta-analysis. N Engl J Med 1997;337:559-60.
23. Cappelleri JC, Ioannidis JPA, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, Lau J. Large trials vs meta-analysis of smaller trials: How do their results compare? JAMA 1996;276:1332-8.
24. LeLorier J, Gregoire, G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-42.
25. Villar J, Carroli G, Pelizan JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet 1995;345:772-6.
26. Godlee F. The Cochrane Collaboration. Br Med J 1994;309:969-70.
27. Sackett DL. The Cochrane Collaboration. ACP Journal Club 1994;May/June:A-11.
28. Bero L, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA 1995;274:1935-8.
29. Huston P. Cochrane Collaboration helping unravel tangled web woven by international research. Can Med Assoc J 1996;154:1389-92.
30. Cochrane AL. 1931-1971: a critical review, with particular reference to the medical profession. London: Office of Health Economics, 1979.
31. Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet 1995;345:840-2.
32. Jadad AR, Haynes RB. The Cochrane Collaboration—Advances and challenges in improving evidence-based decision making. Medical Decision Making 1998;18:2-9.
33. Field MJ, Lohr KN (eds). Clinical practice guidelines: Directions for a new program. Institute of Medicine. Washington, DC: National Academy Press, 1990.
34. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ for the Evidence-based Medicine Working Group. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA 1995;274:1800-4.
35. Browman GP, Levine MN, Mohide EA, Hayward RSA, Pritchard KI, Gafni A, Laupacis A. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J Clin Oncol 1995;13:502-12.
|
Home | Contents | Foreword | Introduction | Acknowledgments | How to order
© BMJ Books 1998. BMJ Books is an imprint of the BMJ Publishing Group. First published in 1998 by BMJ Books, BMA House, Tavistock Square, London WC1H 9JR. A catalogue record for this book is available from the British Library. ISBN 0-7279-1208-9