Evaluation and adaptation of clinical practice guidelines
- 1School of Nursing, University of Ottawa Ottawa, Ontario, Canada
- 2School of Nursing, Queen’s University Kingston, Ontario, Canada
Clinical practice guidelines are “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”1 They are intended to offer concise instructions on how to provide healthcare services.2 The most important benefit of clinical practice guidelines is their potential to improve both the quality or process of care and patient outcomes.3 Increasingly, clinicians and clinical managers must choose from numerous, sometimes differing, and occasionally contradictory, guidelines.4 This situation is further complicated by concerns about the quality of available guidelines.5,6,7,8,9,10,11 Indeed, adoption of guidelines of questionable validity can lead to the use of ineffective interventions, inefficient use of scarce resources, and perhaps most importantly, harm to patients.12,13
Determining which guidelines are quality products worthy of adoption can be daunting. Every effort should be made to identify existing guidelines that have been rigorously developed and to adopt or adapt them for local use.12 However, organisations and clinicians should scrutinise the methods by which the guidelines were developed, as well as the content and utility of the recommendations. Even guidelines developed by prominent professional groups or government bodies should not be exempt from this scrutiny as it has been shown that these guidelines may be of substandard quality.10
The Practice Guidelines Evaluation and Adaptation Cycle14,15 is a framework for organising and making decisions about which high quality guidelines to adopt (figure). Although the cycle was originally intended for use by organisations and groups wanting to implement best practice, most steps of the process are also helpful in guiding evaluation of guidelines by individual clinicians. This Users’ guide will describe strategies for identifying, critically appraising, and adopting or adapting guidelines for local use.
1. IDENTIFY A CLINICAL AREA TO PROMOTE BEST PRACTICE
The first step is to select an area in which to promote best practice. Reasons for selecting a particular area can include the prevalence of the condition or its associated burden, concerns about large variations in practice or care gaps, costs associated with different practice options, the likelihood that a guideline will be effective in influencing practice, a desire to keep practice up to date or evidence-based, or awareness of the existence of relevant evidence-based guidelines.
2. ESTABLISH AN INTERDISCIPLINARY GUIDELINE EVALUATION GROUP AND 3. ESTABLISH A GUIDELINE APPRAISAL PROCESS
When an organisation or group is interested in providing best practice, a local interdisciplinary guideline evaluation group should be established comprising key stakeholders who will be affected by the selection of guideline recommendations, including patients or individuals from the community.14 The advantages of using a group to evaluate guidelines include sharing of work among group members, reduced potential for bias in the evaluation process, and increased awareness of guidelines and opportunities for group members to develop ownership of the resulting decisions.
It is important to select an appraisal process. Guideline appraisal instruments are intended to be used to systematically assess and compare guidelines using the same criteria. They typically consist of a several quality criteria or items that assess the extent to which each guideline meets these criteria. To date, many appraisal instruments have been developed.16 The Appraisal of Guidelines Research and Evaluation (AGREE) Instrument (http://www.agreecollaboration.org)17 is rapidly becoming accepted as the gold standard for guideline appraisal. The AGREE instrument has been tested in 11 countries on >100 guidelines and >200 appraisers.17 It is endorsed by the World Health Organization, the Council of Europe, and the Guidelines International Network (http://www.g-i-n.net).
The AGREE instrument was designed to assess the process of guideline development and the extent to which the process is reported. It consists of 23 Likert scale items organised into 6 domains. Each domain is intended to capture a separate dimension of guideline quality (see box). Each guideline assessed is assigned standardised dimension scores ranging from 0–100. It also includes a question to provide a global assessment of the overall quality of the guideline, ie, whether one would “strongly recommend this guideline for use in practice without modifications,” “recommend this guideline for use in practice on condition of some alterations or with provisos,” or “not recommend this guideline (not suitable for use in practice).” Complete information about the instrument can be found at www.agreecollaboration.org.
Although some might want to view the results from this quantitative evaluation as an objective measure of guideline quality, it is important to remember that scores are influenced by the extent to which the guideline developers described the methods used to develop the guideline and reach consensus on the recommendations. A rigorously developed guideline may score poorly if the process was not well described.
Nurses caring for patients who had had a stroke wanted to provide best practice across the continuum of care in their community. A working group was formed, with nurses representing acute care, long term care, and rehabilitation settings. The group identified the first clinical priority for nursing care as the development of a consistent, evidence-based approach for risk assessment. Key areas were risk of falling, skin breakdown, and swallowing problems. In reviewing practices, they found that all settings had existing evidence-based policies for assessment of falls and pressure ulcers. Assessment of dysphagia, however, was an area of concern. The group invited a local speech and language pathologist and representatives from rehabilitation therapy and medicine to join the working group.
The working group decided to use a transparent and rigorous process to engage each of their settings and develop a solid foundation for future implementation of the risk assessment recommendations they would be developing. Given its reputation, they decided to use the AGREE instrument to evaluate guidelines that included risk assessment recommendations for patients with stroke.
AGREE quality criteria
Scope and purpose
1. The overall objective(s) of the guideline is (are) specifically described.
2. The clinical question(s) covered by the guideline is (are) specifically described.
3. The patients to whom the guideline is meant to apply are specifically described.
4. The guideline development group includes individuals from all the relevant professional groups.
5. The patients’ views and preferences have been sought.
6. The target users of the guideline are clearly defined.
7. The guideline has been piloted among target users.
Rigour of development
8. Systematic methods were used to search for evidence.
9. The criteria for selecting the evidence are clearly described.
10. The methods used for formulating the recommendations are clearly described.
11. The health benefits, side effects, and risks have been considered in formulating the recommendations.
12. There is an explicit link between the recommendations and the supporting evidence.
13. The guideline has been externally reviewed by an expert panel prior to publication.
14. A procedure for updating the guideline is provided.
Clarity and presentation
15. The recommendations are specific and unambiguous.
16. The different options for management of the condition are clearly presented.
17. Key recommendations are easily identifiable.
18. The guideline is supported with tools for application.
19. The potential organisational barriers in applying the guideline have been discussed.
20. The potential cost implications of applying the recommendations have been considered.
21. The guideline presents key review criteria for monitoring and/nor audit purposes.
22. The guideline is editorially independent from the funding body.
23. Conflicts of interest of guideline development members have been recorded.
4. SEARCH FOR AND RETRIEVE GUIDELINES
The next step is to clarify the issues of particular interest. The PICO approach involves considering the Population, Intervention, Control or context, and Outcomes of interest.18,19 Based on the identified areas of interest, criteria for searching for and selecting guidelines for review are identified. Such criteria may include language of publication (eg, English only) or date of publication (eg, within the past 5 y). One study suggests that some recommendations included in well developed guidelines (eg, those produced by the US Agency for Healthcare Research and Quality) may become outdated within 3–4 years of release.20 The same study, however, noted that wound care guidelines were still current up to 7 years after their release. Although the language and year of publication can be used to limit the search for guidelines, other criteria can only be applied once potentially relevant guidelines have been retrieved. For example, a group may only be interested in guidelines based on high quality scientific literature and therefore exclude consensus documents; or they may include only guidelines developed by credible professional organisations and exclude those developed by one person. Regardless, the criteria should be determined before starting the search.
To ensure that high quality guidelines are not inadvertently missed, a systematic search for all relevant guidelines on the topic should be done. Guidelines can be identified using a few simple strategies. Check the US National Guideline Clearinghouse (http://www.guideline.gov), sponsored by the US Agency for Health Care Research and Quality. Another guideline repository is the Guidelines International Network (G-I-N) (http://www.g-i-n.net). G-I-N is an international not-for-profit association of organisations and individuals involved in clinical practice guidelines. Although G-I-N membership is required to access guidelines compiled by the network, non-members can access the websites of some of the guideline developers. Another efficient strategy is to search the websites of known guideline developers (eg, Scottish Intercollegiate Guidelines Network or Royal College of Nurses). To be thorough, it is also important to search the National Library of Medicine, which can be done for free using PubMed (http://www.ncbi.nlm.nih.gov/pubmed). Search terms known as MeSH (Medical Subject Headings) and text words that can be used are practice guideline, practice guidelines, clinical practice guideline, clinical practice guidelines, standards, consensus statement, and consensus.10 “Practice guidelines” can also be used as a publication type (pt) in searching.12
Increasingly, guideline developers are posting their guidelines directly on the web. This avoids delays in publication of guidelines by journals, permits rapid updating of guidelines, and reduces dissemination costs. When guidelines are posted directly to the web, there is a greater chance that they may not be indexed in commonly consulted bibliographic databases such as Medline. For this reason, it is prudent to also search the internet using a search engine such as Google (http://www.google.com). One should not assume that guidelines found on the internet are poor quality or that those indexed in Medline are necessarily high quality. All guidelines that meet the inclusion criteria should be retrieved. Because the appraisal process is based on information reported by guideline developers, all relevant documents related to the development process should be retrieved. In some cases, the published guideline will have minimal information about the development process because this information is presented elsewhere, perhaps in a technical report. Efforts should be made to obtain such supplemental documents.
The working group began by identifying guidelines they were familiar with and those currently used in their settings and scanned the bibliographies to identify additional guidelines. They decided to restrict the search to guidelines published in English or French as the group did not have the capability of reviewing documents in other languages and to restrict the search to documents published since 1997. The group then enlisted the help of a local hospital librarian, who worked with them to develop a search strategy to identify general guidelines about care of patients with stroke, with the expectation that some of these guidelines might include recommendations about dysphagia. They also decided to search specifically for guidelines about dysphagia. The databases searched were Medline, EMBASE/Excerpta Medica, CINAHL, and the US National Guideline Clearinghouse database. Websites of known guideline developers were accessed, and a Google internet search was done. The following topic related key words were used alone and in combination to identify general stroke guidelines: cerebrovascular accident, cerebrovascular disorders, stroke, rehabilitation, spasticity, electromyography, gait, assistive devices and equilibrium. The search for guidelines on assessment of dysphagia included all guidelines for the elderly and was not limited to the stroke population. Key words included risk, dysphagia, swallow disorders, and deglutition disorder.
All guidelines identified by the search strategy were retrieved and assessed based on the following predefined criteria : (1) produced by a group or organization (ie, not authored by one person); (2) included a bibliography; and (3) made recommendations for bedside swallowing assessment/screening targeted at clinicians such as nurses, speech-language therapists, general practitioners, physiotherapists, or occupational therapists.
5. ASSESS THE GUIDELINES
Determining whether a guideline is valid involves 3 separate but related steps: appraising the quality of the guideline as a whole, determining the currency of the guideline (ie, are the recommendations up to date?), and assessing the content of the recommendations.
a. Assess the quality of the guideline as a whole
Ideally, the AGREE instrument should be applied to all guidelines meeting the minimum inclusion criteria. However, this may not be practical or possible depending on the number of guidelines identified, the number of individuals who can participate in the appraisal, and time constraints. One strategy for quickly identifying the higher quality, evidence-based guidelines is to first screen the guidelines using the AGREE’s “rigour of development” domain. The 7 items comprising this domain specifically focus on the degree to which the guideline development process was evidence-based and how evidence/research was incorporated into the recommendations. No universal agreement exists about specific cut off scores to identify high quality guidelines. Some domains (eg, rigour of development) may be considered more important than others and thus have a higher benchmark. The group should identify the range of acceptable quality scores (eg, ⩾70/100) and whether different domains should have different cut offs. Guidelines meeting the benchmark for the “rigour of development” domain can then be assessed using the other AGREE domains. Those scoring below the cut off can be excluded at this point. Regardless, the developers of the AGREE instrument suggest that ⩾4 appraisers should be used to ensure adequate inter-rater reliability.
b. Determine the currency of the guideline
Guidelines that meet minimum quality criteria must then be assessed to determine whether they are still current. Methods of checking the currency of guidelines include reviewing the date of release/publication; scanning the bibliography for the dates of the original studies cited; and checking with developers about whether they still consider the guideline to be current or have plans to update it. A quick Medline search for systematic reviews published since the release of the guideline may also be useful. Other sources of high quality systematic reviews include the Cochrane Database of Systematic Reviews and the Cochrane Controlled Trials Register (http://www.cochrane.org), the NHS Database of Abstracts of Reviews of Effectiveness (DARE) (http:/nhscrd.york.ac.ek/darehp.htm), and the York Centre for Reviews and Dissemination (www.york.ac.uk/inst/crd).27 “Netting the Evidence” (www.shef.ac.uk/uni/academic/R-Z/scharr/ir/netting.html) and the Joanna Briggs Institute (www.joannabriggs.edu) are useful for locating databases of evidence.
Six group members appraised each guideline using the AGREE instrument. The results of the quality appraisal process revealed some unexpected findings. Quality scores for rigour of development varied considerably, ranging from 16–82%. The 2 guidelines with the lowest quality scores on this dimension were from regional organisations well known to the working group members. For these reasons, the group decided to not exclude these 2 guidelines at this stage despite their poor quality scores. Scores on the domain of scope and purpose were more consistent and exceeded 60% for all guidelines. Scores on stakeholder involvement ranged from 33–90%. The lower scores on this dimension largely reflected the lack of inclusion of patient views and not piloting the guidelines. Scores on clarity and presentation, applicability, and editorial dependence were also variable. Examination of the global assessment by the appraisers revealed that no guideline was rejected by a majority of working group members. Three guidelines were strongly recommended “as is” by two thirds of the appraisers. More than half rated the other 3 guidelines as being “in need of modifications” or were unsure about whether to recommend them for use. All guidelines were produced since December 2000, suggesting that their recommendations were fairly current. The librarian also did a literature search for meta-analyses, reviews, and primary studies on the assessment of dysphagia that could be used as supplementary material.
c. Systematically assess the clinical content of guideline recommendations
Guideline appraisal instruments provide little detailed information on the actual recommendations being advanced in specific guidelines. Thus, if ⩾1 guideline is being considered, the next step is to conduct a “content analysis” of the recommendations in each guideline. It is useful to have 1 or 2 clinicians experienced in the content area produce a table comparing each guideline in terms of the specific recommendations made and the level of evidence supporting each recommendation (if such information is provided). Such a table, or recommendations matrix,14 can be the focus of the group’s discussion or an individual clinician’s deliberations about the content of the recommendations from each guideline. The recommendations matrix facilitates identification of similar recommendations in different guidelines, as well as differences in recommendations between guidelines. The matrix also facilitates easy identification of recommendations supported by strong evidence. Often, guidelines include several recommendations supported by evidence of differing strengths. When this happens, a group may decide to select recommendations supported by the best evidence from the guidelines under consideration. The recommendations matrix also provides a basis for considering the clinical usefulness of the recommendations and provides useful information to address the question of whether the recommendations will help in caring for patients in the relevant settings.
To facilitate comparison of the recommendations among the 6 guidelines, a nurse and the speech-language pathologist created a recommendations matrix that included the level of evidence supporting each recommendation.
6. ADOPT OR ADAPT GUIDELINES FOR LOCAL USE
At this point, the group must decide whether it will adopt one of the guidelines “as is” or adapt ⩾1 of the guidelines (ie, select some, but not all, recommendations from different guidelines). The recommendations must be considered in terms of whether they will be helpful for caring for patients and whether they are appropriate and feasible to implement in the specific practice setting(s) (eg, resources available to purchase special equipment needed to comply with guideline recommendation).
The choices at this step are to adopt or adapt existing guidelines. Adopting a guideline involves choosing the best guideline and accepting all recommendations as written. This may not be practical or feasible for many reasons, and the group may need to adapt or tailor ⩾1 guideline to their needs. Selection of this option may be appropriate if the recommendations are not a good fit with the practice setting (eg, some recommendations may not apply to the types of patients seen in the setting, or a practice group may be only willing to accept recommendations supported by strong evidence). As well, implementation issues, such as contextual factors or logistical or resource considerations, can make implementation of the recommendations impractical. Guideline adaptation essentially involves taking the best or most appropriate recommendations and repackaging them into a new local guideline.
Guideline developers are always concerned that local adaptation of guidelines will result in modifications to recommendations that ignore the evidence. Local adaptation of existing guidelines should never involve changing evidence-based recommendations unless the supporting evidence has changed since release of the guideline. If recommendations are modified in any way, the rationale for changes should be explicitly stated in the resulting local guideline document. The group may also want to reconsider evidence that was located when conducting the search for systematic reviews as it could influence which recommendations are adapted.
After reviewing the recommendations and supporting evidence, the working group decided it would produce its own set of recommendations by adapting recommendations from existing guidelines. In some cases, the wording of existing recommendations was modified slightly to make them clearer. Caution was used when rewording recommendations to ensure that the intent of the original recommendation was not altered. The local guideline recommendations for dysphagia risk assessment were as follows:
1. All patients should be kept NPO until their swallowing has been screened using a simple valid bedside testing protocol (adapted from Department of Veterans Affairs/Department of Defense,21 Level B; Royal College of Physicians,22Level B).
2. The gag reflex alone is a poor predictor of swallowing function and should not be used for screening for dysphagia in stroke patients (Scottish Intercollegiate Guidelines Network–Dysphagia, 24 Level B).
3. Patients should be reviewed for dysphagia at least once a week after the initial assessment (adapted from Scottish Intercollegiate Guidelines Network–Dysphagia, 24 Level B).
The working group also decided to review the literature to identify a suitable valid bedside dysphagia testing protocol.
7. SEEK EXTERNAL REVIEW OF THE PROPOSED LOCAL GUIDELINE
When the guideline evaluation process is undertaken on behalf of a group, the resulting draft of local recommendations should be sent to local practitioners, other stakeholders, and organisational policy makers for review and comment. This step should be done even if a single guideline is adopted in its entirety. Seeking feedback on the proposed guideline ensures that those intended to use the guideline have an opportunity to review the document and identify potential difficulties for implementation before the guideline is finalised. This step allows policy makers to consider the organisational effects of implementing the recommendations and to begin preparing for its future adoption. It also serves as the first wave of dissemination of the guideline and provides the group with an opportunity to address the issues raised by reviewers before finalising the local guideline.
Depending on the extensiveness of the adaptation process, it may also be reasonable to send the local guideline to external experts for review of its content validity, clarity, and applicability. This can help to ensure that recommendations from existing guidelines have not been taken out of context or adapted inappropriately.
8. FINALISE THE LOCAL GUIDELINE AND 9. OBTAIN OFFICIAL ENDORSEMENT AND ADOPTION OF THE GUIDELINE BY THE ORGANISATION
The group should consider all feedback and, if necessary, modify the local guideline recommendations to address the concerns. All changes made should be documented in the local guideline as well as reasons for not making suggested changes. Being explicit and transparent about the process should increase the credibility of the process among potential guideline users. Once finalised, official endorsement of the guideline should be sought from policy makers in settings where the guideline is intended to be implemented. This step involves review of the proposed guideline (which may have been modified based on feedback) by the organisation and formal adoption, with official status. This is done, for example, when an organisation endorses a guideline as policy. This administrative step provides the organisation with a final opportunity to consider the effects of the proposed guideline on its functioning. The formal decision making and procedural process required to endorse a guideline needs to be explicit and documented by the organisation. Once the organisation provides its “seal of approval,” the guideline is ready for dissemination and implementation. If plans for dissemination and implementation of recommendations have not been considered, they should be at this point.
10. SCHEDULE REVIEW AND REVISION OF THE LOCAL GUIDELINE
The group should develop a plan for when and how the local guideline will be reviewed and updated (which will obviously depend on when the original guidelines expire) or provide a guideline expiry date. Other criteria for determining when a guideline needs updating include changes in evidence on existing benefits or harms associated with recommendations, important outcomes, available interventions, evidence that current practice is optimal, values placed on outcomes, and resources available for health care.27
Depending on the extent of the changes to recommendations required by new evidence, the guideline evaluation group may want to simply seek practitioner or policy maker feedback on the changes, or begin the entire guideline evaluation cycle over again. In any case, plans for reviewing and revising the guideline should be documented. Individual clinicians who make decisions about which guidelines or specific recommendations they will personally follow must also be aware of the currency of the guidelines and any new guidelines or evidence which may be more current.
RESOLUTION OF CLINICAL SCENARIO
The local guideline document was sent to the practice committees or councils in each of the involved settings for review and feedback. The working group reviewed the feedback from all sources. The feedback revealed that front line nurses were positive about assessing patients for dysphagia but needed training and time to develop confidence in their assessment skills. The local guideline was finalised and sent to the regional stroke team as well as the heads of programmes in each setting for official endorsement and adoption. Working with clinical managers, the working group developed a strategy to plan for staff education, phase-in the recommendations, and enhance current documentation processes to capture dysphagia assessment. The working group, managers, and librarian also developed a strategy for periodically reviewing the literature for new guidelines on the topic or updates of existing guidelines in order to keep the local guideline current.
Practice guidelines have the potential to improve process of care as well as patient outcomes. However, their beneficial effects are contingent on successful implementation. Clinical settings can move towards explicit use of evidence in practice by adopting existing guidelines or by local adaptation of existing guidelines. Careful consideration of available guidelines using the process described above can inform clinical and programme level decision making about which guidelines or recommendations are most suitable for their setting. Use of a rigorous and transparent process for identifying, appraising, and adopting/adapting guidelines is crucial as practice guidelines are essentially multiple interventions and the decisions made affect both patients and providers.