Contents
A user’s guide
Alejandro R Jadad
7 From trials to decisions: the basis of evidence based health care
- Quantitative and qualitative approaches should be combined to answer research questions more effectively.
- Anecdotal information has a role to play in health care decisions.
- Evidence based health care (EBHC) uses the best available research evidence to guide health care decisions.
- Dangers arise from the misuse of EBHC.
- There are many interrelated barriers to the successful practice of EBHC.
- Ways exist to overcome the limitations to the practice of EBHC.
Health care decisions are usually the result of the interaction of the information available to the decision makers, their values and preferences, and the circumstances or context in which the decisions are made.1 Randomised controlled trials (RCTs), even if assembled into a perfect systematic review, are just one of the many different types of information that can inform decisions. In this chapter, I describe the different types of information that can influence the role of RCTs in decision making, and how such information can be processed, modulated, and integrated into decisions, according to different values, preferences, and circumstances.
What types of information can be used to make health care decisions?Decisions can be influenced by formal research studies or by anecdotal information. By anecdotal information I mean any type of information informally gained, from either personal or clinical experiences, one's own or that of others, without the use of formal research methodology.2
If you want to understand the role of RCTs in decision making, it is crucial that you recognise that RCTs are just one of many different types of research studies, and that research studies are just one type of information. You should also realise that the interaction between different types of information in health care decision making is perhaps as complex and poorly understood as the interaction of information, values, preferences, and circumstances. The obvious first step to exploring this interaction is a basic understanding of the different types of information that you can use to make decisions. Let's start with formal research.
What are the different types of research studies?
Research studies can be either quantitative or qualitative.
What is a quantitative study?
A quantitative study is one that presents its results using numbers. Studies are called experimental if investigators influence the way in which the interventions are administered. RCTs are obviously the most important type of experimental study. This group also includes controlled clinical trials in which participants are allocated to the study groups using methods other than randomisation (see Chapter 1).
When the investigators do not influence the course of the events, the studies are called observational. These studies can be controlled or non-controlled and, depending on how the data are gathered in time, they can be prospective, retrospective, or cross sectional. The controlled observational studies can be classified further into those with contemporaneous controls (studies in which data from the different groups are obtained during the same period of time) and those with historical controls (data from one or more groups gathered at different points in time). It has been shown that, other things being equal, observational controlled studies with historical controls tend to favour the new intervention more frequently than those with contemporaneous controls.3
Specific examples of observational studies include cohort studies (usually prospective with contemporaneous or historical controls), case-control studies (retrospective and usually with contemporaneous controls), and surveys (cross sectional and usually non-controlled). A detailed description of the different types of observational studies can be found elsewhere.4,5
We still know much more about RCTs than we do about other study designs. Therefore, we also need to improve our understanding of these designs, and learn more about how to weight the evidence provided by studies with various designs and different degrees of methodological rigour. We tend to place RCTs at the top of the evidence hierarchy, assuming that RCTs are always better than other study designs. This is reflected in most of the levels of evidence produced as part of guideline development processes or to teach evidence based decision making.6,7 These levels of evidence, which may have didactic value, may not be appropriate in practice, where we are confronted with studies of different types and with different levels of methodological rigour. For instance, it might be incorrect to give more weight to a flawed RCT than to a rigorous cohort study.
What is a qualitative study?A qualitative study is a study that does not attempt to provide quantitative answers to research questions. Instead, the objective is to try to interpret events or things in their natural settings and to develop conceptual frameworks about social phenomena.8 In qualitative studies, the following happens:
- Emphasis is placed on the meanings, experiences, and views of all participants.9
- Major data collection methods include interviews, participant observation, examination of documents.
- Hypotheses are usually developed during or after the research, rather than a priori.10
Are there different types of qualitative studies?
Yes. Some examples include in-depth interviews, focus groups, ethnographic studies, and case studies. A detailed description of qualitative studies, and their strengths and limitations, was presented in a series of articles published in the British Medical Journal.9 Frequently, three or more types are used to address the same issue. This strategy, which is known as triangulation, is used to compare the results obtained from different data collection techniques and study designs, and judge their robustness or validity.
Are quantitative and qualitative research approaches mutually exclusive?
Often a research question in health care is approached using either a quantitative or a qualitative approach as if the approaches were mutually exclusive or even antagonistic. This is explained, at least in part, by the lack of experience or familiarity of most researchers with both approaches. Recently, however, it has been proposed that a more coherent and efficient strategy to answer research questions would be to combine quantitative and qualitative approaches.8,9 This could be achieved: by using a quantitative study to guide the design of a subsequent qualitative study or vice versa; by conducting quantitative and qualitative studies simultaneously to answer a particular question; and by introducing elements of the one approach into the other (that is, a quantitative study that includes some qualitative components). All these possibilities should be explored more frequently in health care research. In the end, the key is to match the right type of design to the right type of question or the right aspect of the same question (see Chapter 8).
At this point, it is important for you to remember that information can be obtained not only from research studies but also outside of formal research activities. This information can have a profound influence on health care decisions. Anecdotes, for instance, can be used to convey ideas and influence behaviour, and make causal inferences. The first two (convey ideas and influence behaviour) are well established, but the last (causal inferences) is very controversial.
A large body of experimental research has shown that anecdotes are very convenient and efficient vehicles for conveying information and modifying behaviour.11,12 A number of independent but related factors contribute to the impact of anecdotal information. One of the most important factors is that anecdotal information has emotional interest. Events that happen to us personally are more interesting than those that happen to others; those that happen to people we know, respect, or care about are more persuasive than those that occur to strangers or people about whom we have neutral feelings.2 Another important factor is the vividness of the anecdotal information. Face to face recommendations have been shown experimentally to be more influential than informationally superior data presented impersonally.13,14 Health care recommendations by a respected local peer were shown to be a more powerful force for change in clinical practice than were evidence based consensus guidelines published nationally.15
The role of informal observation as a source of information to guide health care decisions is much more controversial. As human beings, we tend to make inferences from anecdotal information using simple rules of thumb, also known as heuristics, which allow us to define and interpret the data of physical and social life, and to reduce complex tasks to much simpler operations. These rules of thumb are essential for everyday life, and will often lead to correct and useful conclusions.
Data can, however, be misperceived and misinterpreted. To the extent that motivation influences behaviour, inferences can be distorted by needs and wishes, and the rules of thumb can lead to incorrect and potentially harmful conclusions. Remarkably little attention has been paid to the value of anecdotal information in health care decision making at all levels. We need more research on the role of anecdotes in health care decisions, because most of the research on this type of information and its role in human inference and decision making has been conducted by social and cognitive psychologists. In the meantime, we should acknowledge the role of anecdotal information in health care decisions. Ignoring its powerful influence is likely to hinder communication among decision makers, and to retard their uptake of research evidence. Anecdotal information should be considered a complement to, rather than a replacement for, formal research evidence. They could also be used as vehicles to deliver the results of formal research to people involved in decisions, regardless of their background.
How can information be integrated into decisions?At this point, I hope that you are convinced that information is an essential component of decisions. At the same time, I hope that you do not think that information is sufficient to make decisions. It does not matter how much information you have or how valid and relevant it is, it should always be modulated by the values and preferences of the decision makers and those who will be affected by the decisions, and the circumstances in which the decisions are made. You will even see that, in some cases, your values (both as consumer or provider of health care), your knowledge of other individuals involved in a particular decision, the particular characteristics of the setting in which the decision is being made, and your own previous experiences will be more important than the more generalised evidence from even the best formal research studies. In other cases, the research evidence will be so compelling that it will be very difficult for you to ignore it or to justify decisions that depart from it. In most cases, however, it will be unclear how much your anecdotal information, values, and preferences (and those of others involved in the same decision) should modulate the research evidence available. As a clinician, you will find yourself constantly walking a fine line parallel to that walked by patients and other lay members of the public involved in health care decisions, including your own. The need to walk this fine line in safe and responsible ways is what motivates the supporters of evidence based health care.
What is evidence based health care?Evidence based health care (EBHC) is a term used to describe the explicit, conscientious, and judicious use of the currently best available evidence from research to guide health care decisions.16 There are other related terms, most of which are used interchangeably. The first term used, evidence based medicine,17 focused primarily on physicians and medicine. Other terms proposed since include evidence based decision making, which refers to the process itself, independently of the particular area or even outside health care; and evidence based practice, used by the US Agency for Health Care Policy and Research to designate a series of centres in North America, which have been charged with producing evidence reports and technology assessments to support guideline development by other groups.18. More recently, more specific terms have been emerging. These include evidence based management,19 evidence based nursing,20 evidence based mental health,21 evidence based chaplaincy,22 and so on.23
What are the elements of EBHC?From the above definition, any activity related to EBHC should be:
- Explicit, as it should be clearly described and replicable.
- Conscientious, involving careful systematic steps to use research evidence as part of the decisions (see below).
- Judicious, rather than blind adherence to research evidence.
- Centred on the use of the current best available evidence from research, but not fixated on randomised trials or meta-analyses as some may think, but on the best available studies that exist at a given point.
- Aimed at guiding decisions, not making decisions on the basis of information alone.
Even though many decision makers (particularly clinicians with a long trajectory) could claim that EBHC has been practised for many decades, others purport that EBHC is a new process of decision making which includes a new blend of elements, and even constitutes a paradigm shift in health care.17,24
What steps should be followed during the practice of EBHC?The steps of the process have been described in relation to evidence based medicine and the care of individual patients.25 I have adapted these steps to go beyond individual patient care and applied them to health care decisions in general. The steps are as follows:
- First, the decision maker must formulate answerable questions in relation to the decision being made (the questions should take into account the elements discussed in Chapter 1).
- Once the question has been formulated, the decision maker should make systematic efforts to locate research evidence that could be used to answer the question.
- After identifying the research evidence, the decision maker should appraise its validity, relevance, and applicability.
- After appraising the evidence, the decision maker should be able to use it, integrating it with other types of information, his or her own values and preferences, and the circumstances in which the decision is being made.
- Once the research evidence is integrated with the other ‘modulating factors’ and used to guide a decision, the decision maker should make efforts to evaluate the outcomes of the decision and his or her own performance.
The understanding and application of the basic principles of EBHC may help decision makers do the following:
- Overcome the barriers that hinder adequate utilisation of information as part of health care decisions.
- Provide a common ground on which all decision makers could interact.
- Enhance the appropriateness of practice and lead to better patient outcomes (as interventions of established efficacy become more widely used).
- Spend resources more appropriately, as resources spent on interventions shown to be ineffective are transferred to more effective ones.
- Identify knowledge gaps, leading to highly relevant new research efforts to fill these gaps.
The dangers associated with the practice of EBHC are likely to emerge from its misuse.
Evidence can be abused by both patients and clinicians who may pay attention only to research evidence that supports their previously held views, overriding all contradicting evidence or other sources of information. Evidence can also be readily abused by politicians, policy makers, and third party payers who are more interested in financial stringency than improving health. If they ignore the fact that a lack of evidence of effectiveness is not the same as evidence of a lack of effectiveness, these politicians and third party payers may decide to pay only for those treatments supported by strong evidence in order to save money or increase profits. Evidence can also be abused by sensation-seeking journalists, who are interested in ‘breakthroughs’ to make headlines, and therefore report only positive trials portraying them as the best available knowledge.
The practice of EBHC can also become cult-like. Some decision makers may be willing to adhere to research evidence blindly, applying it in circumstances where it may not be appropriate, while ignoring its limitations, the role of other types of information, and the values and preferences of other decision makers. These dangers can easily be prevented by a proper understanding of the principles of EBHC outlined above.
What barriers exist to the practice of EBHC?
There are many interrelated barriers to the successful practice of EBHC. Some of these are specific to decision makers, some to health care evidence, and others to the health care system.
What are the most important barriers to EBHC from a decision maker's perspective?
Despite the impressive efforts of the past 20 years to improve the use of evidence in decisions, most clinicians still face many barriers to the practice of EBHC. These barriers are shared by decision makers who lack clinical training (for example, consumers, journalists, planners) and are therefore more difficult to overcome. The existing barriers may not affect all decision makers in the same way or order. Among others, important barriers to the practice of EBHC from a decision maker's perspective include the following.
Lack of awareness Some decision makers may not know about EBHC, and if they do they may have a poor understanding of it, and therefore do not realise its full benefits.
Lack of time As a result of the existing reward systems and the increasing workloads of clinicians, the amount of time available to study and to keep up to date is likely to decrease rather than increase.
Lack of motivation Some decision makers, even if aware of EBHC, may not feel motivated to practise it. This could be because they feel they do not have the skills or resources required to do it properly, or because they may be sceptical about the added value of the whole approach. Instead, these decision makers may decide to rely more on informal methods for decision making.
Poor skills for question formulation Even if aware and motivated, some decision makers would fail to practise EBHC simply because they do not know how to formulate answerable questions that include all the elements described in Chapter 1.
Inadequate literature searching skills and resources to access the literature Most training programmes in health care do not include formal courses and practical sessions to teach decision makers how to search the literature. Even if they do, decision makers are unlikely to have the time or motivation to maintain these skills and keep up with the developments in bibliographic databases. Many other decision makers may have the skills, but no access to sources of research evidence, namely computers, bibliographic databases, or even journals.
Limited critical appraisal skills Few decision makers have received formal courses on critical appraisal and, even if they had, the effects of such courses are still unclear.24
Limited knowledge There might be limited knowledge about how to integrate research evidence with other types of information, values, preferences, and circumstances (see above).
What are the most important barriers to EBHC emanating from the existing evidence?Even if decision makers have optimal skills to practise EBHC, they would face many barriers emanating from the current status of research evidence. Some of the most prominent evidence related barriers include the following.
Abundance The amount of research evidence that is being produced in the world makes it impossible for any decision maker to keep up to date in isolation or, even more importantly, to find valid and clinically relevant material in the overwhelming body of literature.
Poor internal validity (great risk for bias) Many studies, including RCTs, lack the methodological rigour required to produce unbiased results. The main sources of bias are described in detail in Chapters 3 and 4.
Limited relevance Few studies are designed with the day to day needs of clinicians and patients in mind. Most RCTs, for instance, have placebo-controlled designs, which are good for meeting regulatory and academic needs, but inadequate for helping clinicians and patients to select treatments among many options that have never been compared directly. Most studies also lack clinician and patient input during the selection of outcome measures. For example, most RCTs lack measurements of quality of life, patient preferences, or the resource implications of interventions, and few last long enough to produce meaningful answers.
Limited precision Most studies are too small to provide precise answers.
What are the most important barriers to EBHC from the health care system's perspective?
Typically, the most important barriers imposed by almost any health care system on earth include the following.
Lack of financial incentives Clinicians' performance tends to be judged by the number of patients they see and the resources they generate, rather than by how much they study or strive to make decisions based on the best available evidence.
Lack of training opportunities and effective training programmes There are few opportunities for clinicians to gain the skills required to practise EBHC. Even if more opportunities for training were available, the feasibility and effectiveness of the available teaching strategies have recently been questioned. A systematic review of 17 comparative studies, which examined the effects of different strategies on teaching critical appraisal to undergraduate medical students or residents, showed consistent gains in knowledge at the undergraduate level, but fairly small improvements in knowledge at the residency level.24 Surprisingly, these studies had methodological deficiencies that could have been easily prevented and that are often the focus of the criticisms of critical appraisers (for example, short duration, inadequate outcome measures, small sample sizes). The studies had other problems that were expected and more difficult to overcome, such as the lack of a culture for proper experimental studies of interventions targeting curricula, mismatch between the courses taught and the environments where the learners had to apply their skills, and a high risk of contamination.26
What can you do to overcome the limitations to the practice of EBHC?
You will face your own special blend of barriers and opportunities to the practice of EBHC. To overcome them, you should first identify those skills that you need to improve. At the same time, you should explore ways in which your own working environment could be more conducive to the practice of EBHC. Given the huge number of combinations of different barriers related to decision makers and health care systems, I terminate my discussion of them here. Instead, I concentrate on a series of information resources that have developed during the past decade that you will find useful, regardless of your barriers and circumstances. These developments focus on getting the evidence straight, on delivering the evidence to decision makers, or both. Examples of initiatives to improve the design, execution, and synthesis of research studies include the CONSORT and QUOROM initiatives (see Chapters 4 and 5), and the Cochrane Collaboration (see Chapter 6). The following are examples of efforts that involve a combination of the best available evidence and the best available information technology.
Free access to MEDLINE on the Internet
MEDLINE became available to clinical users in the 1980s and in 1997 it became free of charge on the Internet (http:igm.nlm.nih.gov/; http://www4.ncbi.nlm.nih.gov/PubMed/clinical.html). PubMed includes refined search strategies for optimising searches for clinically useful studies. Other suppliers of MEDLINE, such as Ovid (http://www.ovid.com) and Silver Platter (http://php.silverplatter.com/physicians/php/answer.htm), provide more comprehensive and expensive services, including access to full text articles.27
Evidence based publications
This group of secondary or ‘new breed’ publications do the initial sorting of evidence using explicit methods to identify valid and clinically useful articles from a large number of journals, and then provide concise informative titles (see Chapter 5), abstracts, and commentaries that help readers discern whether the information applies to their own decisions. Surprisingly, these publications are typically thin and relatively infrequent in publication (for example, bimonthly or quarterly), reflecting the fact that a very small proportion of the literature is really worth reading. ACP Journal Club (ACPJC), the first of these ‘evidence based’ publications, appeared in 1991. It was followed by Evidence-Based Medicine (EBM) in 1996, and Evidence-Based Nursing and Evidence-Based Mental Health in February of 1998.27 Another example, Bandolier, is produced monthly by the Oxford Anglia NHS Region in the UK (http://www.jr2.ox.ac.uk.Bandolier/index.html). Similar journals are likely to emerge in many other areas in health care. The main challenge will be to keep them full of valid and clinically useful content.
Specialised compendia of evidence for clinical practice
Two superb examples are the Cochrane Library (see Chapter 6) and Best Evidence. The latter is an easy-to-search database that includes the contents of all the issues of the journals ACP Journal Club and Evidence-based Medicine (see above).
Evidence based textbooks
This even more recent development includes a group of textbooks that are kept up to date. These are in their early stages, but the basic principle, that declarations of what practitioners should do should be based on current best evidence, is present in their evolution. Scientific American Medicine and UpToDate are heading in this direction, as are others.27 Clinicians who lack access to CD-ROM computer drives will find themselves progressively out of touch.
Specialised web sites
These are Internet-based ‘look up’ services that provide access to evidence based information and to sites that support EBHC. Examples include the Evidence Based Medicine Project at http://hiru.mcmaster.ca/ebm; Netting the Evidence at http://www.shef.ac.uk/uni/academic/R-Z/scharr/ir/netting.html. Most major journals are starting to make substantial amounts of information available on the Internet, free of charge.
The bottom line
Health care decisions are the result of a complex and poorly understood interaction among many factors, of which research information (and RCTs in particular) is just one component. Other types of study designs and even anecdotes can provide valuable information on aspects of a decision that trials do not address. In addition, information alone is insufficient to make decisions. The values and preferences of the decision makers, and the circumstances in which decisions are made, often act as modulators of the information available to the decision makers and can sometimes even override it. There are major challenges to optimising the way in which decision makers use information and integrate it with their values and circumstances. Evidence based health care has evolved as a strong movement to accelerate this process. To succeed, however, we will require massive efforts and strong commitment by everyone involved in health care decisions to ensure that major barriers are overcome. The rapid development of information technology and international initiatives are providing unprecedented opportunities to overcome these barriers.
References
1. Haynes RB, Sackett DL, Gray JRM, Cook DL, Guyatt GH. Transferring evidence from research into practice: 1. The role of clinical care research evidence in clinical decisions [editorial]. ACP J Club 1996;125:A-14; Evidence-based Medicine 1996;11:196-8.
2. Enkin M, Jadad AR. Using anecdotal information in evidence-based decision- making: heresy or necessity? In review
3. Sacks H, Chalmers TC, Smith H. Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233-40.
4. Altman, DG. Practical statistics for medical research, 1st edn. London. Chapman & Hall, 1991.
5. Streiner DL, Norman GR. PDQ epidemiology, 2nd edn. St Louis, MI: CV Mosby, 1996.
6. Guyatt GH, Sackett DL, Sinclair JC, Hayward RSA, Cook DJ, Cook RJ for the Evidence-Based Medicine Working Group. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA 1995;274:1800-4.
7. Browman GP, Levine MN, Mohide EA, Hayward RSA, Pritchard KI, Gafni A, Laupacis A. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J Clin Oncol 1995;13:502-12.
8. Jones R. Who does qualitative research? BMJ 1995;311:2.
9. Pope C, Mays N. Reaching the parts that other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ 1995;311:42-5.
10. Kuckelman-Cobb A, Nelson-Hagemaster J. Ten criteria for evaluating qualitative research proposals. J Nursing Educ 1987;26:138-43.
11. Nisbett RE, Ross L. Human inference: strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice-Hall, 1980.
12. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974;185:1124-31.
13. Redelmeier DA, Rozin P, Kahneman D. Understanding patients' decisions: cognitive and emotional perspectives. JAMA 1993;270:72-6.
14. Borgida E, Nisbett RE. The differential impact of abstract vs. concrete information on decisions. J Appl Soc Psychol 1977;7:258-71.
15. Lomas J, Enkin M, Anderson GM, Hannah WJ, Vayda E, Singer J. Opinion leaders vs audit and feedback to implement practice guidelines. JAMA 1991;265:2202-7.
16. Jadad AR, Haynes RB. The Cochrane Collaboration—Advances and challenges in improving evidence-based decision making. Med Decision Making 1998;18:2-9.
17. Guyatt G, Rennie D. Users' guides to the medical literature. JAMA 1993;270:2096-7.
18. Atkins D, Kamerow D, Eisenberg GM. Evidence-based medicine at the Agency for Health Care Policy and Research. ACP Journal Club 1998;128:A-14-16.
19. Smith K. Evidence-based management in health care. In: Peckham M, Smith R, eds. Scientific basis of health service. London: BMJ Publishing Group, 1996;92-98.
20. Mulhall A. Nursing research and the evidence. Evidence-Based Nursing 1998;1:4-6.
21. Geddes J, Reynolds S, Streiner D, Szatmari P, Haynes B. Evidence-based practice in mental health. Evidence-Based Mental Health 1998;1:4-5.
22. O'Connor T, Meakes E. Hope in the midst of challenge: an evidence-based approach to pastoral care. J Pastoral Care 1998:52(4).
23. Gray JAM. Evidence-based healthcare: how to make health policy and management decisions. London: Churchill-Livingstone; 1997.
24. Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. Can Med Assoc J 158:177-81.
25. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone, 1997.
26. Sackett DL, Parkes J. Teaching critical appraisal: no quick fixes. Can Med Assoc J 1998;158:203-4.
27. Haynes RB, Jadad AR, Hunt DL. What's up in medical informatics? Can Med Assoc J 1997;175:1718-19.
|
Home | Contents | Foreword | Introduction | Acknowledgments | How to order
BMJ Books 1998. BMJ Books is an imprint of the BMJ Publishing Group. First published in 1998 by BMJ Books, BMA House, Tavistock Square, London WC1H 9JR. A catalogue record for this book is available from the British Library. ISBN 0-7279-1208-9