Changes in beliefs identify unblinding in randomized controlled trials: a method to meet CONSORT guidelines

https://doi.org/10.1016/j.cct.2004.11.020Get rights and content

Abstract

Double-blinded trials are often considered the gold standard for research, but significant bias may result from unblinding of participants and investigators. Although the CONSORT guidelines discuss the importance of reporting “evidence that blinding was successful”, it is unclear what constitutes appropriate evidence. Among studies reporting methods to evaluate blinding effectiveness, many have compared groups with respect to the proportions correctly identifying their intervention at the end of the trial. Instead, we reasoned that participants' beliefs, and not their correctness, are more directly associated with potential bias, especially in relation to self-reported health outcomes.

During the Water Evaluation Trial performed in northern California in 1999, we investigated blinding effectiveness by sequential interrogation of participants about their “blinded” intervention assignment (active or placebo). Irrespective of group, participants showed a strong tendency to believe they had been assigned to the active intervention; this translated into a statistically significant intergroup difference in the correctness of participants' beliefs, even at the start of the trial before unblinding had a chance to occur. In addition, many participants (31%) changed their belief during the trial, suggesting that assessment of belief at a single time does not capture unblinding. Sequential measures based on either two or all eight questionnaires identified significant group-related differences in belief patterns that were not identified by the single, cross-sectional measure.

In view of the relative insensitivity of cross-sectional measures, the minimal additional information in more than two assessments of beliefs and the risk of modifying participants' beliefs by repeated questioning, we conclude that the optimal means of assessing unblinding is an intergroup comparison of the change in beliefs (and not their correctness) between the start and end of a randomized controlled trial.

Section snippets

Introduction and background

The randomized, placebo-controlled trial is often described as the gold standard for research involving human participants. An important aim of the study design is to allow comparisons between groups of participants whose characteristics are, on average, as similar as possible with the exception of the intervention being studied. This is achieved primarily through randomization, which aims to prevent selection bias and balance the distribution of measured and unmeasured confounding variables

Methods

We previously investigated participant blinding in a community-based pilot study in northern California to investigate the effect on gastrointestinal infectious illness of water treatment units installed at the kitchen sink [25], [26]. To promote successful blinding, the active and placebo units were designed to be externally identical and to produce water with similar characteristics such as temperature, taste and odor. The active unit consisted of a filter and ultraviolet light system in

Results

Water treatment units were installed in 80 households, 3 of which were subsequently excluded from the study [25]; the remaining 77 households comprised 236 individuals. Participants aged ≥12 years (N=179) were asked to report their beliefs on 8 occasions and did so on average 6.6 times (median 8, interquartile range 6–8). Valid responses were provided at week 2 by 172 (96%) respondents, at week 16 by 145 (81%) and on all 8 occasions by 123 (69%); 132 (74%) reported their beliefs on at least 6

Discussion

In most evaluations of blinding, participants are only asked to try to identify their intervention group at the end of the trial. For a simple comparison of beliefs at the end of the trial to identify significant intergroup differences caused by unblinding depends on the assumption that participants are undecided at baseline with respect to beliefs. By actually measuring beliefs at baseline in the sequential analysis, we increased our sensitivity to identify changes in belief indicative of

Conclusion

In summary, assessment of blinding effectiveness via changes in beliefs identified group-related differences in participants' beliefs that were not identified by cross-sectional analyses of beliefs at the end of the trial. The longitudinal evidence for unblinding was statistically significant and was supported by evidence that beliefs were associated with—and hence might bias—the primary study outcome, suggesting that the intervention effect should be reported within belief strata.

Our

Acknowledgement

Funding for this work was provided through Cooperative Agreement U50/CCU915546-02-1 from the Centers for Disease Control and Prevention. For their contributions to this work, we thank Anne Benker, Sue Binder, Cliff Bowen, Susan Burns, Rebecca Calderon, Joshua Ergas, Kim Fox, Allen Hightower, Ron Hoffer, Alan Hubbard, Dennis Juranek, Asheena Khalakdina, Catherine Ma, Daniel Mills, Gretchen Rothrock, Art Reingold, Sona Saha, Rick Sakaji, Suhminder Sandhu, Susan Shaw, Kate Steiner, Duc Vugia.

References (33)

  • H.-C. Diener et al.

    Unbalanced randomization influences placebo response: scientific versus ethical issues around the use of placebo in migraine trials

    Cephalalgia

    (1999)
  • R.I. Horwitz et al.

    Adherence to treatment and health outcomes

    Arch. Int. Med.

    (1993)
  • K. McPherson et al.

    Are randomized controlled trials controlled? Patient preferences and unblind trials

    J. R. Soc. Med.

    (1997)
  • R.P. Byington et al.

    Assessment of double-blindness at the conclusion of the beta-blocker heart attack trial

    JAMA

    (1985)
  • J. Howard et al.

    How blind was the patient blind in AMIS?

    Clin. Pharmacol. Ther.

    (1982)
  • T.R. Karlowski et al.

    Ascorbic acid for the common cold. A prophylactic and therapeutic trial

    JAMA

    (1975)
  • Cited by (29)

    • A novel blinding protocol to test participant and therapist blinding during dry needling: a randomised controlled experiment

      2021, Physiotherapy (United Kingdom)
      Citation Excerpt :

      Participants were naïve to the sham during testing, which may have introduced recall bias because the blinding questionnaire had to be completed retrospectively after testing. Alternatively however, naïve participants may reduce biases associated with knowledge that they might be allocated a sham, for example, biases that arise from wishful thinking and the potential for beliefs to change over time [35,42,43]. Mathieu et al. [42] suggested that establishing a state of complete ambivalence about allocation might minimise errors in trial outcomes associated with beliefs, although this comes with a compromise to ecological validity.

    • Random guess and wishful thinking are the best blinding scenarios

      2016, Contemporary Clinical Trials Communications
      Citation Excerpt :

      However, we are cautious to recommend statistical correction with hard-to-satisfy-or-verify assumptions partly because of non-statistical reason; more data collection about blinding may do more harm than good [20,21]. A more desirable approach could be rigorous pilot testing of blinding before actual trial [26], together with better adherence to blinding protocol [5,20], and stratified analyses based on guess status as exploratory or secondary analysis, similarly to stratified analyses based on compliance status [23,27,28]. Third, we used blinding scenario classification with previously used names based on BI for convenience and evaluation purpose.

    • A theoretical analysis showed that blinding cannot eliminate potential for bias associated with beliefs about allocation in randomized clinical trials

      2014, Journal of Clinical Epidemiology
      Citation Excerpt :

      Any person involved in the trial could be blinded, but it is usually considered particularly important to blind trial participants, the clinicians who treat trial participants, and the assessors who ascertain and document trial outcomes. The aim of blinding is to prevent bias caused by beliefs about participants' allocations [1,2]. Beliefs about allocation could bias estimates of the effects of intervention in several ways: trial participants who believe they were allocated to the treated group might experience placebo effects; in trials with self-reported outcomes, participants' beliefs about allocation could influence reports of outcomes; clinicians could provide different cointerventions to trial participants who they believe have been allocated to treated and control groups; and assessors could measure or report trial outcomes differently in trial participants who they believe have been allocated to treated and control groups.

    • Listening to Placebo in Clinical Trials for Female Sexual Dysfunction

      2013, Journal of Sexual Medicine
      Citation Excerpt :

      Nevertheless, until the mechanisms of placebo‐mediated change are better understood, it is worthwhile to examine blinding and detection of treatment assignment in trial participants. Pending a more sophisticated strategy to detect unblinding, Rees et al. suggest that investigators focus on measuring subjects' beliefs about their treatments, especially if these beliefs change across the course of the trial, rather than accuracy of detection per se [56]. Measurement should allow for some degree of ambiguity, as qualitative research suggests that many trial participants maintain “hope” if not necessarily strong expectations for improvement [57].

    View all citing articles on Scopus
    View full text