Statistics from Altmetric.com
Introduction to critical appraisal
Evidence-based nursing (EBN) means using the best available evidence from research, along with patient preferences and clinical experience, when making nursing decisions.1 Nurses are increasingly concerned about ensuring that care is research based, and EBN offers a strategy to help nurses achieve this goal by using 5 steps:
Step 1: reflecting on practice and identifying areas of uncertainty
Step 2: translating these areas of uncertainty into focused, searchable questions2
Step 4: critically appraising the research
Step 5: changing practice if the research suggests this is necessary.
Previous Notebooks in Evidence-Based Nursing that have described the process of EBN 2–6 were written with busy clinical nurses in mind. The next series of Notebooks will explore step 4 in detail—how to assess whether research findings are valid and appropriate for clinical application.
The volume of healthcare literature has been described in graphic terms. Did you know, for example, that Medline has indexed more than 9 million citations in more than 4,000 journals since 1966?7 Clearly, no individual practitioner can read this volume of research; nor should they try, as only a small proportion of it is of good quality. The trick to keeping abreast of the latest research is to quickly filter the good from the bad. For example, staff of the evidence-based journals office recently calculated that of 136 journals read for 4 evidence-based journals (Evidence-Based Nursing, Evidence-Based Mental Health, Evidence-Based Medicine, and ACP Journal Club), an estimated 8% of 21 100 articles passed evidence-based methodologic filters (personal communication, A McKibbon, 20 Mar 2000).
Anyone who has designed or implemented a research study knows that it can be difficult to do well. Researchers, like the rest of us, like to spread “news” and may exaggerate the importance and strength of the findings of their own research. The clinician is faced with the challenge of filtering the thousands of articles published each year into manageable information. This challenge is increased because researchers are notoriously bad at communicating in a language that is accessible to anybody other than researchers. It is not surprising that time and again, nurses tell researchers that they find it extremely difficult, if not impossible to use research findings in their practice.
But don't despair! EBN is possible because (1) there are now a number of clinical evidence sources which present appraised and summarised research for clinicians; (2) good search strategies can identify research that has used an appropriate design to answer your particular clinical question; and (3) critical appraisal is easy to learn, becomes easier with practice, and may be made easier with the Users' Guides series, which will appear in upcoming issues of Evidence-Based Nursing.
Sources of pre-appraised clinical evidence
Clinicians are no longer expected to fend for themselves when trying to deal with the huge volume of research literature. Systematic reviews, for example, are summaries of all the research on a particular topic. An international collaboration, the Cochrane Collaboration, produces and disseminates high quality systematic reviews, and even more importantly, keeps them up to date as new studies become available. The Cochrane Library (available on CD-ROM and via the Internet at http://hiru.mcmaster.ca/cochrane/cochrane/cdsr.htm) is probably the most important source of reliable information about health care anywhere in the world. The Library contains systematic reviews prepared by Cochrane review groups and others, and a large database of clinical trials.3
Journals such as Evidence-Based Nursing, Evidence-Based Mental Health, and Evidence-Based Medicine summarise the most valid and clinically applicable research selected from the international research literature. High quality studies and systematic reviews are identified using explicit methodologic criteria and then summarised in the form of structured abstracts. This approach differs from “current awareness” services, which serve to merely announce potentially interesting new research; evidence-based journals use an explicit, quality filtering process to identify only those studies of high methodologic quality. The structured abstracts provide a brief summary of the research question, study design, methods, and main results and the accompanying commentaries help us to interpret the study findings and to suggest how they might be applied in clinical practice.
Clinical Evidence is a new publication that produces and updates rigorous summaries of the research on many common conditions, including, for example, child health and wound care.8
Sensitive and specific search strategies
Databases such as Medline and CINAHL provide electronic access to the healthcare literature, but they index letters, editorials, and general articles as well as primary studies and systematic reviews. Furthermore, high quality studies and reviews comprise an extremely small proportion of the indexed literature. The trick to searching these databases to find studies to answer your clinical questions lies in using search strategies that have maximum sensitivity (ie, identify a high proportion of the valid and relevant studies) and specificity (ie, do not identify many irrelevant studies). For example, the best single term Medline search to find studies that are likely to have used an appropriate design to evaluate a therapy or intervention is “clinical trial (publication type)”.4 Of course, any primary studies identified through searches of Medline, CINAHL, or other databases must be appraised for validity and applicability using the critical appraisal techniques that will be addressed in the remainder of this users' guide and the series that follows.
Imagine that you have recently been promoted to a new role in your hospital and that you have responsibility for practice development on the medical wards. For the first time in your career you are able to set aside time in your week for reading and keeping abreast of new research—you think you have about 3 hours each week and are keen to use those 3 hours as efficiently has possible. The library is full of medical and nursing journals, but how do you start?
One suggestion is that you start by gathering a list of frequently asked questions (FAQs); that is, those questions you often ask yourself during your clinical work, and those often asked by your colleagues. For example:
Is it safe for patients with diabetes to inject insulin directly through their clothing without skin cleansing?
Is it useful to give dietary supplements to critically ill patients on your ward?
Should you be educating and counselling patients who have had a myocardial infarction, while they are still in hospital?
If you apply what you have learned about asking answerable questions to the first question in your list, it becomes2:
Is it effective and safe for patients with diabetes to inject insulin through clothing compared with the standard technique of skin cleansing and injection?
The most important lesson to learn about filtering your reading is to read only studies or reviews that have used an appropriate design to answer your particular question. The question about insulin injection is a question of whether a particular treatment (ie, injection of insulin through clothing) is effective and whether it is safe. In an earlier EBN notebook, we learned that randomised controlled trials (RCTs) are the strongest research design for answering this type of question. Systematic reviews of RCTs are better still as they summarise all the RCTs done on a particular topic.5 Searching Medline using PubMed on the Internet (a free service that can be found at http://www.ncbi.nlm.nih.gov/PubMed/) with the terms insulin AND inject* AND cloth* AND clinical trial identifies 2 papers, one of which is a report of a study that appears to directly address our question.
The next step is to look at the study or review and decide whether it is valid and clinically applicable. Some of our medical colleagues have developed an excellent series of Users' Guides to the Medical Literature, and we intend to use these guides as a basis for this series.9 As you might expect, separate guides exist for different types of clinical questions, as each is best answered using a specific study design. Regardless of the type of study, the clinician should always ask whether the results of the study are valid—that is, is the way the study was done likely to give a true result? The criteria to look for under this broad heading of validity vary, depending on whether the question is about treatment, prognosis, harm, or about understanding complex phenomena, such as feelings or perceptions. The validity checks can be applied quickly, and with a little practice, you will be able to decide efficiently whether to read a paper at all (on the principle that if the methods are not valid, there is no point in reading the paper). For example, the filtering question for a clinical question about whether a treatment works or not is was the assignment of patients to treatments randomised? If a study fails the basic filtering question, you will need to consider carefully whether or not to read it as it is not likely to direct your practice safely.
This approach to filtering your reading helps to keep the amount you do have to read to a manageable amount. If a study passes the validity filtering question and warrants a more detailed review, then there are further questions to help you determine the study validity with greater confidence, and others to help you tease out what the results of the study really mean, and whether they apply to your patients.
Returning to your question of injecting insulin through clothing, a search of Medline on PubMed would have identified a paper by Fleming and colleagues, which compared insulin injection through clothing with conventional techniques.10 Better still, a search through back copies of Evidence-Based Nursing would have identified that the Fleming study had been summarised in a structured abstract, accompanied by a commentary that addressed the question of clinical applicability. The answer seems to be that injecting insulin through clothing appears to be as safe as conventional techniques and more convenient.11
To make the most efficient use of the limited time available to keep abreast of clinical evidence, we should filter our reading so as to read only studies or reviews that are likely to provide valid results to our focused clinical questions. We should always look for high quality systematic reviews on a topic for which the author has already identified, appraised, and summarised the relevant studies. Other sources of pre-appraised research, such as the evidence-based journals and Clinical Evidence, can also save us work. Where pre-appraised research is not available or up to date, simple critical appraisal questions can help us to rapidly filter the useful studies from those that contain useless or even harmful information, and help us make decisions about clinical applicability. The figure⇓ summarises this approach to reading. Critical appraisal checklists for use in appraising different study designs will form the basis of future users' guides.