Contents
A user’s guide
Alejandro R Jadad
8 My wish list: thinking it all over
In this chapter I describe what I think are the most important barriers to the optimal use of randomised controlled trials (RCTs) in health care. I also propose some strategies that could be used to overcome them. This is not intended to be a comprehensive list. I just want to share my thoughts with you and, hopefully, motivate you to create your own list of barriers and opportunities, and to think about possible solutions that could be implemented in your own setting.
Each section in this chapter starts with a wish and continues with a description of strategies that could be used to overcome barriers to the optimal use of RCTs in health care decision making. You will notice that most of the wishes and barriers described also apply to other types of research information. Some of the strategies to overcome existing barriers may prove difficult to implement in today's world. However, I hope that they (or other effective alternatives) may become feasible in the near future.
The following is my wish list.
I wish for better RCTsDuring the past ten years, important research efforts have looked at RCTs as the subject rather than the tool of research.1,2 These studies, which have generated valuable empirical evidence about the deficiencies in the design, reporting, dissemination, and use of RCTs in health care, have been consistently ignored by researchers, peer reviewers, and journal editors.2 Given the differences between the‘ideal RCT’ and the RCTs that are actually conducted, there is a wide gap between methodological research and methodological practice. This gap is analogous to the gap that exists between clinical research and clinical practice (see Chapter 7). Bridging this methodological gap will be as challenging as bridging the clinical gap, and will require important efforts by researchers, funders, journal editors, and peer reviewers. The following are some examples of efforts that are needed if we are to bridge the gap.
We need to reduce the likelihood of bias in RCTs
This is one of our greatest challenges. As human beings, there is no doubt that we are prone to bias. The empirical methodological research accumulated to date shows, consistently, that the results of most RCTs are likely to exaggerate the benefits of treatments.2 This could be explained by the fact that: many studies are designed, conducted, and reported by researchers whose careers are closely linked to the interventions that they evaluate; most patients want interventions to be effective; and funding is often provided by organisations that thrive on breakthroughs and positive results. The last is particularly true for studies funded by pharmaceutical companies.3 Herculean efforts will be required to minimise or eliminate the influence of all these secondary gains on trial findings. Some of these efforts could include, for instance, mechanisms to prevent direct funding of RCTs by the developers of the interventions, and strategies to minimise the participation of individuals with clear conflicts of interest in the design, execution, and reporting of RCTs.
We need more trials that address clinically relevant questions
Most trials are designed to meet the needs of academics and funding organisations, but may fail to meet the needs of clinicians, policy makers, and patients. For example, few trials include head-to-head comparisons of interventions, assess the impact of interventions or quality of life or resource utilisation, or express their results in ways that clinicians and patients could easily apply to their decisions. In addition, cuts in the budgets of funding agencies are making it more difficult for RCTs to address complex questions. We cannot afford to perpetuate this situation, either on financial or on ethical grounds. The clinical relevance of RCTs could be increased, easily and substantially, if researchers and funding agencies were willing to include consumers and providers of health care as active members of research teams. This approach is proving successful in the development of tools to promote shared decision making, and in the design and dissemination of systematic reviews.4-6
Although interest in efforts to include consumers in discussions about the design of RCTs seems to be increasing,7 almost nothing seems to be occurring to increase the role of health planners, managers, and policy makers in the design, execution, and dissemination of RCTs. We need to find effective strategies to include these stakeholders in research decisions and to evaluate the effect of this involvement on the quality and impact of RCTs. Researchers should also understand the limitations of RCTs, feel comfortable with the fact that there are many situations in which RCTs are not feasible or appropriate, and recognise that the question being asked is what should determine the study design that can best answer it.8
We need trials to provide more precise results
The biomedical literature is plagued with small studies that provide imprecise answers to complex questions. Increasing the sample size of RCTs to achieve the precision needed to answer important questions often requires efficient collaboration among different groups of researchers, often working in different countries. Although there seems to be an increasing number of‘mega-trials’, successful examples of collaboration remain isolated and concentrated in a few areas such as oncology and cardiovascular medicine. Achieving similar degrees of successful collaboration will not be easy in other areas of health care and will require an important departure from current mechanisms to fund research and to reward researchers. Most funding agencies lack the administrative structure or the financial strength required to support international mega-trials. In addition, academic systems continue to reward researchers by judging their ability to‘engage in independent research’ and the number of publications in which they are the lead authors. We need new reward systems that value collaboration, not individualism, and that recognise quality, and not just quantity, of research.
We need to improve the quality of the reports|
As discussed in Chapters 3-5, most published trials are reported incompletely. Most reporting problems could be easily eliminated if all clinical journals, funders, and researchers endorsed and embraced the CONSORT statement1 (see Chapter 5). At the time of writing, just over 70 journals had endorsed the CONSORT statement. Given that there are tens of thousands of biomedical journals and that most editors are probably unaware of the CONSORT statement, it is reasonable to expect that its endorsement by all relevant journals will take lots of time and effort. The World Association of Medical Editors, the largest organisation of its kind, could play an important role in accelerating this process. The success of these efforts will also depend on the commitment of researchers to produce good reports of good RCTs. If the researchers don't take the responsibility when they do the study, they will probably lie about it (or shade the truth), producing good reports of bad RCTs just to comply with the editors' requests and get their manuscripts published.
We need better ways to present the results of trials to users
Most biomedical articles are written by researchers for other researchers. This is entirely appropriate for trials with preliminary results. This also occurs, however, when a trial has been done well with adequate size and clinically important outcome measures. As a result of this, most articles are unpalatable to clinicians, patients, journalists, policy makers, and other audiences. As a consequence, we should not be surprised that trials have little influence in health care decision making. We need better modes of communication and dissemination of findings. The separation between research oriented journals and clinician oriented journals could be a good start.9 Perhaps we also need journals oriented to consumers and policy makers. Regardless of the strategy, if trials are to have the greatest impact, we need to depart from the traditional paper-based articles, full of jargon and numbers, boring and unintelligible to most, and move to more engaging and intellectually appealing ways to present information to users. To succeed, we will need input from individuals with expertise in marketing, early education, graphic design, and advertisement. Similar efforts could improve the way in which other types of research (that is, systematic reviews, clinical practice guidelines, and observational studies) are presented to users.
I wish that information on all ongoing and completed trials was readily available
Trials have little use, even if they are perfectly designed, conducted, reported, and written, if they are not available to users. For trials to be readily available, however, the following items are necessary.
All trials are registered at inception
If trials were registered at inception (from the time they are designed and approved by ethics committees), it would be easy to create databases or registers, accessible to people anywhere in the world through the Internet, with information on all trials that are ongoing or completed, but that have not been published. These databases will ensure that duplication of research efforts is avoided and will contribute to the elimination of publication bias (see Chapter 3). Despite many potential benefits to users, most efforts to create such databases and registers have failed.10 We need public debates about the reasons for these failures, with full disclosure from parties opposed to compulsory registration of trials. Representatives from mass media and consumer groups should be involved in these discussions.
All trials are published, soon after completion, regardless of the direction of their results
Delays in releasing research findings can have harmful effects, especially if trials with positive results are published years before those with negative results.2 Ensuring that all trials are published soon after completion, regardless of the direction of their results, will eliminate publication bias and time lag bias (see Chapter 3). As with compulsory registration of trials, public debates that include the media and consumers may prove effective for overcoming existing barriers. We should also educate patients to demand publication of study results as a requirement of their participation in research studies. Ethics committees can also encourage researchers and funders to make the results of their studies available, through either biomedical journals or other media such as the Internet. Several years ago, serious ethical concerns were expressed about the failure to publish research findings. Individuals who consent to participate in research and the agencies that provide funding support do so with the understanding that the work will make a contribution to knowledge.11 As a result of potential effects of study findings on health care decisions, it could be argued that trials with flawed designs, or those that ask clinically irrelevant questions and provide incomplete reports or delayed publication, are also unethical.2
Individual trials are incorporated in rigorous systematic reviews
As discussed in Chapter 6, it is risky to make clinical decisions using information from a single trial. It would be ideal if new trials were incorporated into a rigorous systematic review designed to summarise the existing evidence in a particular topic. I hope that the Cochrane Collaboration (or any other effort to‘get the evidence straight’) succeeds in producing rigorous, up-to-date, systematic reviews of trials in all areas of health care.
I wish that users could access RCTs more efficiently
It would be ideal if decision makers could access RCTs when they need them and where they need them. For this to occur, we will need the best available information technology and decision makers who are willing and able to use it.
Information technology is evolving rapidly. At no other time in history have there been so many powerful information tools for providers and consumers to access information. We already have resources such as the Cochrane Library and Best Evidence which provide fast and easy access to RCTs and other types of evidence.12 The rapid development of the Internet promises even better opportunities. Keeping up with technological developments, however, is not an easy task for most decision makers. Most people still lack formal training in health informatics and have few opportunities to become familiar with and adopt new technologies. This is compounded by the fact that we know little about the preferences, patterns of utilisation, and barriers to the adoption of different information technologies by different users. If users' skills are to match the techological possibilities, we will need research efforts to address existing knowledge gaps, and strategies to speed up the adoption of new technological developments by people with different backgrounds, expectations, motivations, and skills. Both goals could be achieved through the creation of research‘environments’ that simulate real life settings (that is, consulting offices, classrooms, boardrooms, waiting rooms, homes) and serve as laboratories in which learners, educators, researchers, policy makers, providers, and consumers can use state-of-the-art technology to access the best available evidence to guide their decisions. These laboratories could support the study of new informatics tools, human-computer interaction, evidence based decision making, and computer assisted learning. They would also provide ideal conditions for research on barriers that hinder the use of information technology for accessing high quality research evidence, and would support the development of strategies to overcome them in the real world.
People involved in efforts to improve the skills of decision makers to access information should be aware that differences in information access will occur within groups of users (for example, rich and poor; old and young; urban and rural; English speaking and non-English speaking) and across groups of users (for example, physicians who lack access to information resources being upstaged by their patients). Although these differences are unavoidable, strategies should be put in place to monitor and minimise them, and to avoid the deleterious effects that may arise from such differences.
I wish that all decision makers could understand RCTsEven if RCTs were perfectly designed and readily available to users, they could not influence health care decisions and outcomes if users could not understand them.
During the past 15 years, I have noticed that most people do not understand the concept of randomisation and its strengths, the different sources of bias in RCTs, and the role of RCTs in health care decision making. This could be explained, at least in part, by the fact that most efforts to promote a better understanding of RCTs (and research in general) have focused on researchers in training, particularly graduate students. Little has been done to promote an understanding of research among other users of research. Even though some training programmes for health professionals, policy makers, health planners, and managers include courses on research methodology, they often lack formal activities to promote a better understanding of RCTs. The situation for patients, their family members, and other lay members of the public is even worse. These groups, without whom most RCTs would not exist or would not be needed, have been left unaided to handle research information, let alone RCTs. Journalists who have a profound influence on the dissemination and impact of research information are in a similar situation.
It would be reasonable to assume that those who understand research will be more likely to use it to their advantage than those who do not understand it. People who do not understand research are more likely to ignore it or misuse it. They also have a higher risk of becoming confused, anxious, and frustrated when trying to use research, which could result in worse health outcomes if irrelevant or biased information is used to guide their decisions. The potentially harmful effects of the lack of understanding of users is now compounded by the amount of information available. Until recently, only health care providers had to deal with information overload. Now, with the growth of the Internet, an increasing number of patients, family members, and other lay members of the public are gaining unprecedented access to information and experiencing the effects of information overload.
Against this background, I feel confident in saying that the development and implementation of effective strategies to increase users' understanding of research, and RCTs in particular, is a top priority. Part of the success of these strategies will depend on the way in which the results from research are presented to users (see above). Part also depends on our ability to recognize, understand, and overcome specific barriers to the adequate use of RCTs by different groups of decision makers. Success will also depend on how well these strategies target all groups of potential and actual users of research.
Timing will be crucial. To date, efforts to promote a better understanding of RCTs and research have focused mainly on adults. Perhaps the effectiveness of such efforts could be enhanced if they were targeted to younger learners. For instance, the basic principles of decision making, research, and critical appraisal in health care could be incorporated in school curricula7,13 and taught using interactive video games and other innovative computer based methods. If children can understand these principles, they will not only be in a better position to participate in health care decisions, but will require little additional education and reinforcement once they become adults fulfilling the roles of health professionals, policy makers, planners, managers, journalists, patients, family members of patients, and other healthy adult members of society.
Efforts to increase our understanding of RCTs should take into account the tendency of human beings to rely excessively on intuition and rules of thumb, to follow inadequately built theories, and to be strongly influenced by vivid experiences and anecdotes.13-17 Efforts also need to account for the fact that research information will be modulated, not only by other types of information, but also by the values and preferences of the decision makers and the unique circumstances in which most decisions are made.18 If trials are to be used efficiently, and if evidence based decision making is to reach its full potential, we will need to couple our efforts to increase the understanding of RCTs with efforts to promote a better understanding of the relationship between RCTs and other study designs, between research information of all kinds and other types of information, and between information available to decision makers, their values and preferences, and the circumstances in which they are making the decisions. We will need a better understanding of the interaction among different groups of decision makers (that is, nurses and physicians, physicians and hospital managers, nurses and patients) in terms of their own information, values, and preferences in different contexts. Gaining this understanding will require different research approaches, input from multiple disciplines, and an enormous commitment at all levels.
Closing remarksThe RCT is one of the simplest, most powerful, and revolutionary tools of research.19 Despite their extensive use as research tools over the past 50 years, most trials are biased, too small, or too trivial. It is essential that we make more efforts to protect ourselves against ourselves during the design, analysis, dissemination, and use of RCTs. Such efforts will hopefully benefit patients, scientists, governments, industry, research institutions, funding and regulatory agencies, ethics committees, journalists, and other consumers of information. Overcoming the existing barriers will, however, require innovative research strategies, and unprecedented levels of commitment, participation, and contribution by us all. Only by meeting the current challenges will we ensure that the RCT occupies its righteous place and provides the information that study participants and users of research expect and deserve.
References
1. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup D. Improving the quality of reporting of randomized controlled trials - The CONSORT Statement. JAMA 1996;276:37-9.
2. Jadad AR, Rennie D. The randomized controlled trial gets a middle-aged checkup. Editorial. JAMA 1998;279:319-20.
3. Bero LA, Rennie D. Influences on the quality of published drug trials. International J Technol Assess Health Care 1996;12:209-37.
4. Jadad AR, Haynes RB. The Cochrane Collaboration - Advances and challenges in improving evidence-based decision making. Med Decision Making 1998;18:2-9.
5. Jadad AR, Whelan T, Rayno L, Pirocchi J, Farrell S, Neimanis H, Montesanto B, Sobeirajski T, Browman GP.‘A team approach to pain relief’: a guide developed with patients and family members. Supportive Care in Cancer 1996;3:245.
6. Bero L, Jadad AR. How consumers and policy makers can use systematic reviews for decision making. Ann Intern Med 1997;127:37-42.
7. McNamee D. Public's perception of RCTs. Lancet 1998;351:772.
8. Sackett DL, Wennberg JE. Choosing the best research design for each question. Br Med J 1997;315:1636.
9. Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med 1990;113:724-8.
10. Dickersin K. How important is publication bias? A synthesis of available data. AIDS Education Prevention 1997;9(suppl A):15-21.
11. Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263:1405-8.
12. Haynes RB, Jadad AR, Hunt DL. What's up in medical informatics? Can Med Assoc J 1997;157:1718-19.
13. Nisbett RE, Ross L. Human inference: strategies and shortcomings of social judgement. Englewood Clifs, N.J. Prentice-Hall Inc., 1980.
14. Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974;185:1124-31.
15. Redelmeier DA, Rozin P, Kahneman D. Understanding patients' decisions: cognitive and emotional perspectives. JAMA 1993;270:72-6.
16. Enkin MW, Jadad AR. Using anecdotal information in evidence-based health care: heresy or necessity? In review.
17. McDonald CJ. Medical heuristics: the silent adjudicators of clinical practice. Ann Intern Med 1996;124:56-62.
18. Haynes RB, Sackett DL, Gray JRM, Cook DL, Guyatt GH. Transferring evidence from research into practice: 1. The role of clinical care research evidence in clinical decisions [editorial]. ACP J Club 1996;125:A-14; Evidence-based Medicine 1996;1:196-8,
19. Silverman WA, Chalmers I. Sir Austin Bradford Hill: an appreciation. Control Clin Trials 1992;13:100-5.
|
Home | Contents | Foreword | Introduction | Acknowledgments | How to order
© BMJ Books 1998. BMJ Books is an imprint of the BMJ Publishing Group. First published in 1998 by BMJ Books, BMA House, Tavistock Square, London WC1H 9JR. A catalogue record for this book is available from the British Library. ISBN 0-7279-1208-9