IS THIS A SYSTEMATIC REVIEW OF RANDOMISED TRIALS?

Good systematic reviews determine at the outset to include those studies that used the most appropriate design to answer the clinical question. Questions about the effectiveness of treatment or prevention are best answered by randomised controlled trials, whereas questions about harm or prognosis are best answered by cohort studies.⁵ The review by Glazener and Evans includes 22 randomised trials: 5 compared alarms with no intervention and 17 compared different alarms, alarms with drugs or behavioural interventions, or combination interventions.⁴

DOES THE SYSTEMATIC REVIEW INCLUDE A DESCRIPTION OF THE STRATEGIES USED TO FIND ALL RELEVANT TRIALS?

It is usually necessary to search several electronic bibliographic databases and to use other strategies, such as journal handsearching and consultation with experts, to ensure that every primary study is identified. A search confined only to the Medline and CINAHL databases will be biased towards studies published in English, and those that found significant differences between interventions (if a reviewer only finds the 2 studies that found a difference and not the 10 that did not, she is likely to draw misleading conclusions).⁶ Therefore, a search strategy is considered thorough if it includes several databases, if the reference lists of relevant papers were searched, if key journals were searched by hand, and if key informants were contacted. In the Glazener review, 11 different electronic databases were searched.⁴ As another step to ensure that all possible studies were included and to minimise publication bias,⁶ organisations, manufacturers, researchers, and health professionals concerned with enuresis were contacted to identify unpublished studies. References were also checked for additional studies.⁴ Handsearching of key journals was not reported for this review. Handsearching contributes to the completeness of retrieval, as occasionally those who index articles for databases such as Medline may use inappropriate keywords or may miss articles or even whole journal issues. This particular review could be considered to have an extensive and thorough search strategy, with the exception of the lack of handsearching.

Every systematic review should grow from a focused question, which itself leads to the development of inclusion criteria and a sensitive and specific search strategy. Once the articles are retrieved, it is necessary to be pedantic about the application of inclusion and exclusion criteria. The review by Glazener and Evans identifies clear, focused questions: do alarm interventions reduce nocturnal enuresis? Are alarm interventions more effective than other interventions?⁴ Inclusion criteria were that children had to be randomised to alarm treatment or to a control condition, which could involve no intervention or other behavioural methods or drugs for nocturnal enuresis. Trials of children with organic causes for bed wetting were excluded, as were trials of daytime wetting.⁴

DOES THE SYSTEMATIC REVIEW INCLUDE A DESCRIPTION OF HOW THE VALIDITY OF THE INDIVIDUAL STUDIES WAS ASSESSED?

An unsystematic narrative review often reports on study findings without considering the methodological strength of the studies. Differences in study quality might explain differences in results because studies of poorer quality tend to overestimate the effectiveness of interventions.⁷ Quality rating scales are sometimes used in the analysis to make comparisons of outcome by study strength. If there are many studies to consider, the authors may decide to apply a quality rating threshold for inclusion of studies in the review, or to give greater weight to stronger studies.

A prespecified quality checklist helps to ensure that reviewers appraise each study consistently and thoroughly—another means of minimising bias. Having ≥2 raters, with some mechanism described for dealing with divergence of opinion, helps to reduce both mistakes and bias and increases the confidence of the reader in the systematic review. The quality rating tool usually includes criteria such as those presented in previous users' guides in this series.^1,2

The review by Glazener and Evans applied explicit, prespecified validity criteria including the level of concealment of allocation at randomisation, the comparability of the groups at baseline, blinding of outcome assessment, use of intention to treat analysis, and the extent of follow up.⁴ The reviewers were explicit about the quality criteria used in this review.

WERE THE RESULTS CONSISTENT FROM STUDY TO STUDY?

Although we would not expect to find the same magnitude of effect in all studies, we would be more confident in using the results of a review if the results of individual studies were qualitatively similar—that is, all showing a positive effect or all showing no effect. But what if treatment effect differs across studies? Many systematic reviews identify studies with important differences between them in terms of the types of patients included; the timing, duration and intensity of the intervention; or the outcome measures. The reviewers make decisions about whether meta-analysis is appropriate by using a combination of judgment and a statistical test for heterogeneity—a test of the extent to which differences between the results of individual studies are greater than you would expect if all studies were measuring the same underlying effect and the observed differences were only because of chance. The more significant the test of heterogeneity, the less likely that the observed differences are because of chance alone, indicating that some other factor (eg, study design, patients, intervention, or outcomes) is responsible for differences in treatment effects across studies.³

The value of judgment in deciding whether statistical synthesis is appropriate cannot be overemphasised, however, because the statistical test for heterogeneity has low power and may fail to identify important differences in study results. Readers of reviews must make judgments, using their clinical expertise, about whether meta-analysis makes clinical and methodological sense. In the review by Glazener and Evans, the authors chose to statistically combine 3 RCTs that compared alarms with no intervention for the outcome of “number not achieving 14 consecutive dry nights or relapsing” (this outcome will now be referred to as “treatment failure or relapse”).⁴ All 3 trials had similar populations, interventions, and outcome measures, and the results were similar in direction and size of differences between the treatment and control groups. The test for heterogeneity was not significant.⁴

Different statistical approaches can be used in meta-analysis. Fixed effects models are most commonly used when no significant heterogeneity exists between studies; this model assumes that the true effect of the treatment is the same in each study, with differences in results arising because of chance. A random effects model assumes that the study results vary around some overall average treatment effect, and the calculation of the summary statistic incorporates an estimate of between study variation.⁸ The Glazener review used a fixed effects model.⁴

WERE INDIVIDUAL PATIENT DATA OR AGGREGATE DATA USED IN THE ANALYSIS?

Less commonly, authors may request individual patient data from the investigators of individual studies. In this case, rather than using the study results (eg, relative risks or odds ratios), individual patient data may be combined across studies to allow comparisons of outcomes for subgroups, defined, for example, by age or severity of illness. The Glazener and Evans review did not include individual patient data in their results.⁴

HOW LARGE WAS THE TREATMENT EFFECT?

Comparing a simple count of the number of studies that found a positive effect of an intervention with the number that found no effect or a harmful effect, would be misleading. This would assume that all studies had equal validity, power, duration, dosage, etc, to have the potential for the same effect. Meta-analysis, when appropriate, can assign different weights to individual studies so that those with greater precision, or higher quality, will make a greater contribution to the summary estimate. Glazener and Evans used meta-analysis to determine the relative risk (RR) of treatment failure or relapse in the intervention versus the control group and used the graphical display common to Cochrane Reviews to summarise their findings (fig⇓). It is useful to learn how to interpret these figures as they can portray much information at a quick glance.

• Download figure
• Open in new tab
• Download powerpoint

Graphical display of results of meta-analysis. Reprinted from Glazener CMA, Evans JHC. Alarm interventions for nocturnal enuresis in children. Cochrane Database Syst Rev 2001;(1):CD002911.

Firstly, focus on the far left column. Looking down you will see a row for each of the 3 studies included in this comparison, referenced by the name of the first author and year the study was done. The list begins with the study by Bollard (1981 B) and ends with the study by Wagner (1985). Looking along each study (row), there is a box in the middle of the figure⇑, with a horizontal line through it. It is possible to interpret whether an intervention is more or less effective than its comparator without reading the actual numbers. The box indicates the result of that particular study, in this case a RR, and the horizontal line indicates the 95% confidence interval (CI) around that RR. A RR of 1 would mean there is no difference in the event rate (ie, treatment failure or relapse) between the treatment and control groups; the risks are the same. A RR on the right side of the vertical line representing 1 (ie, a RR >1) would favour the control condition (ie, more children with treatment failures or relapses in the alarm group), and a RR on the left side of the line would favour the treatment (more children with treatment failures or relapses in the control group).

The reader can get an idea of the heterogeneity of the study results simply by looking at how the lines are scattered. In this case, the results of each of the 3 studies are on the left side of 1—that is, all estimates of RR were <1, and the CIs overlapped considerably. If the boxes for different studies were on both sides of 1 and had non-overlapping CIs, you would have less confidence in using the results of this review in a clinical decision. Applying this information, the study in the first row by Bollard (1981 B) has an associated RR on the left side of 1, indicating that fewer children in the alarm group had treatment failures or relapses than children in the control group. Because the CI does not cross the vertical line (ie, does not include 1), this difference is statistically significant. Looking down the list, all 3 studies reported a statistically significant reduction in treatment failures when alarms were used.

The result of combining all 3 studies is found at the bottom of the figure⇑. The overall summary statistic, in this case the combined RR, is depicted as a diamond which also encompasses the 95% CI. The edges of the diamond do not cross or touch 1, indicating a statistically significant difference in favour of alarms. When outcomes are dichotomous (eg, alive or dead, dry or wet), meta-analyses generally use RRs or odds ratios for reporting the summary statistic; when the outcomes are continuous (eg, blood pressure, blood glucose, or weight), the calculation is the mean effect size or mean difference.⁹ Each of these statistics may be weighted or unweighted. When mean effect size or mean difference are reported, the vertical line of no difference is at 0 rather than 1.

Although the summary statistic is the most important “bottom line,” more can be learnt from the figure⇑. The second column from the left (Expt n/N) gives the number in the treatment group who experienced the outcome of interest (treatment failure or relapse) (n) out of the total number in the treatment (alarm) group (N). The third column (Ctrl n/N) gives the same information for the control group in each study. The fifth column of numbers (Weight %) tells you how much a particular study contributed to the overall summary statistic, with more weight given to studies of greater precision. The column on the far right [RR (95% CI fixed)] provides the RRs and accompanying 95% CIs for each individual study corresponding to the box and horizontal line for that study.

Rather than reporting RRs, Evidence-Based Nursing usually reports the relative risk reduction (RRR) and the number needed to treat (NNT), each with accompanying 95% CI's. Previous EBN notebooks explained these measures of effect in detail.^8–,¹⁰

From the table in the Glazener and Evans abstract, the RRR is 42%. In other words, the alarm intervention reduced the risk of treatment failure or relapse (a bad outcome) by 42% in the intervention compared with the control group. You can calculate this from the figure⇑ by subtracting the RR from 1 (ie, 1−0.58 = 0.42 or 42%). The 95% CI around the RRR of 42% is 26% to 54%, indicating that the true RRR may be only 26% or may be as large as 54%. You can calculate this from the figure ⇑by subtracting each end of the CI from 1 (ie, 1−0.46 = 0.54 or 54% and 1−0.74 = 0.26 or 26%). The NNT to prevent one additional treatment failure or relapse for alarms versus usual care was 3 (95% CI 2 to 4). In other words, to prevent 1 additional treatment failure or relapse (ie, not achieving 14 consecutive dry nights or relapsing after treatment completion), you would need to treat 3 children with alarms, and we are 95% certain that the true NNT may be as low as 2 and as high as 4 children.¹⁰ One of the limitations of using NNTs derived from meta-analyses is that the patients entered into individual trials may vary considerably—particularly in terms of how susceptible they were to the outcome of interest.¹¹ In many reviews, the length of follow up varies across the primary studies, making the NNT difficult to interpret. Furthermore, a decision about whether or not to use enuresis alarms would also involve consideration of risks, costs, and patient preference and acceptability.

HOW PRECISE IS THE ESTIMATE OF TREATMENT EFFECT?

As described in a previous users' guide,² CIs around the RR and the RRR indicate the precision around the estimate of the true treatment effect, which can never be really known. Wide CIs indicate less precision in the estimate. The convention is to use the 95% CI, which represents the range within which we are 95% certain that the true value lies.¹⁰ Precision increases with larger sample sizes, although this is difficult to see in our example of the Glazener and Evans review because the 3 studies had similar sample sizes.

The CI for the final summary RR is fairly narrow (0.46 to 0.74).⁴ The CI is useful for decision making because we can look at the limit closest to 1 (no effect) and ask ourselves “if the effect was as small as this, would it be worthwhile?” In this review, the risk, or probability, of treatment failure or relapse in the alarm group is 58% of the risk of treatment failure or relapse in the control group. The 95% CI indicates that the true risk of treatment failure or relapse in the alarm group may be only 46% of the risk in the control group, or the risk may be as high as 74%. To determine your confidence in adopting the intervention, you should consider the boundary of the CI closest to 1. If the risk of treatment failure really is approximately 75% of the risk in the control condition, given the cost, inconvenience, and side effects of the intervention, would you still want to implement the intervention?

ARE MY PATIENTS SO DIFFERENT FROM THOSE IN THE STUDY THAT THE RESULTS DON'T APPLY?

As in the users' guide for evaluating a treatment,² you need to consider the characteristics of the patients within the individual studies included in a review, and how similar they are to your own. Are there reasons why the results would not be applicable to your patients? Data from the review by Glazener and Evans showed that participants were somewhat more likely to be boys and to have a mean age of about 9 years,⁴ and so are similar to some of the patients you see in your clinic. The review also needs to provide enough information about the intervention to enable implementation.

IS THE TREATMENT FEASIBLE IN MY SETTING?

Use of nocturnal alarms seems a clinically feasible intervention in terms of care provider ability to recommend alarm systems. Feasibility for this scenario, however, would relate to the ability of parents to either buy the alarm system or have it provided to them.

WERE ALL CLINICALLY IMPORTANT OUTCOMES (HARMS AS WELL AS BENEFITS) CONSIDERED?

Researchers try to look at all outcomes, both positive and negative, that are a result of treatment and are important to the patient and the healthcare system. These might include mortality, morbidity, quality of life, cost effectiveness and patient satisfaction. Aspects of harm may not be systematically collected and fully reported in primary studies and few systematic reviews of treatments undertake thorough searches for relevant harm data likely to be found in cohort studies.⁵ Glazener and Evans simply described different outcomes that were reported in a few primary studies, such as alarm malfunction, false alarms, fright, failure to awaken the child, and awakening others.⁴ Costs were not reported. This review did not incorporate an economic evaluation, and such information would contribute to decision making.