Evaluation of studies of health economics
- Patricia W Stone, RN, PhD1,
- Suzanne Bakken, RN, DNSc, FAAN1,
- Christine R Curran, RN, PhD, CNA2,
- Patricia H Walker, RN, PhD, FAAN3
- 1Columbia University New York, New York, USA
- 2The Ohio State University and Medical Center Columbus, Ohio, USA
- 3Graduate School of Nursing Uniformed Services University of the Health Sciences Bethesda, Maryland, USA
A rural hospital hires a new infection control professional, EL, who has just received her master’s degree and trained in infection control departments that were up to date on the best evidence. EL reviews the infection control policies and learns that traditional central line catheters are used. Where she trained, a new, more expensive, antiseptic catheter was used. The new catheter is impregnated with chlorhexidine and silver sulfadiazine and costs $25 more than the traditional catheter.
Because the new catheter is designed to reduce the incidence of hospital associated infections, particularly blood stream infections that are catheter related, EL recommends that the hospital begin using it in place of the traditional central line catheter. However, hospital administration and the purchasing department question the value of this new, more expensive, antiseptic impregnated catheter. Issues related to both cost and the possibility of improved outcomes are discussed at a meeting with intensive care nurses, physicians, and representatives from hospital administration and the purchasing department. Specifically, the questions are: (1) is the antiseptic impregnated catheter economically efficient? In other words, are the additional costs of the catheter offset by potential cost savings in reduced number of infections, which will affect length of stay? (2) Is the antiseptic impregnated catheter more cost effective when used in certain types of patients (eg, patients at high risk of catheter related infections, such as immunosuppressed patients)? An interdisciplinary team is asked to conduct a literature search and make an evidence-based recommendation.
As healthcare technology continues to expand, the cost of using all effective clinical services exceeds available resources. Because of the scarcity of resources, decisions regarding the implementation of new services frequently need to be based on economic analysis. Economic analysis is a set of formal, quantitative methods used to compare 2 or more treatments, programmes, or strategies with respect to their use of resources and their expected outcomes.1 Economic evidence seeks to inform resource allocation decisions at different levels, including institutions (eg, hospitals) and regional or national governments. Unfortunately, not all economic evidence is of high quality. To identify scientifically sound evidence that can inform nursing related resource allocation decisions, nurses need to evaluate the research by answering the questions outlined in Table 1. These questions will be discussed and illustrated with examples of interest to nurses; then, the clinical scenario will be addressed and a published economic evaluation will be critiqued to further illustrate these points.
Are the results of this economic evaluation valid?
The number of published economic evaluations is growing.2 However, many experts in the field have questioned the methodological rigour of these evaluations.3,4 Therefore, it is important for nurses to be able to assess the validity of the methods used in these economic studies in order to determine which findings they will apply to their setting.
All rigorous economic evaluations consider the opportunity cost; that is, the value of the benefits foregone in alternative uses when a resource is used for a given purpose.5 For example, the opportunity cost to a municipality’s health department of funding a community health screening programme for the elderly may be the foregone benefits of not being able to fund a prenatal programme for at risk pregnant women. In addition, economic evaluations should consider the extra (ie, incremental) benefit that would be gained for the extra (ie, incremental) cost. A classic example illustrating incremental cost and effectiveness is the sixth stool guaiac test used to screen for colorectal cancer in people over 40 years of age. In the US, the average cost per case of cancer detected using ≤5 tests has been calculated to be US $2541, but the incremental cost of the sixth test was estimated to range between $47 and $127 million.6 This is because the probability of detecting an additional case of cancer (the incremental effectiveness) is so low that millions of dollars (the incremental costs) would be spent before identifying one more person with cancer.
IS THE APPROPRIATE ECONOMIC EVALUATION METHOD USED?
5 different analytic tools are commonly used to assess the economic effects of new and established healthcare interventions (Table 2). Each method will be discussed below. The costs should be measured the same in all analyses, but the methods differ in how effects or outcomes are measured.
Cost-minimisation analysis (CMA)
In a true CMA, only costs are evaluated. The central assumption is that the outcomes of the alternative interventions or strategies are considered equivalent, and so the goal is to find the least expensive way of achieving the outcomes.1 For example, a cost minimisation study was used to evaluate the savings from early hospital discharge of patients with osteomyelitis followed up by outpatient antibiotic treatment when compared with conventional inpatient treatment. When total costs per patient were compared, the early discharge programme resulted in a savings of US $510 per patient.7 Because CMA assumes that clinical outcomes are the same, the evaluation is essentially a search for the least costly alternative — the minimal cost strategy (in this case, early hospital discharge). However, the circumstances under which CMA is an appropriate method of analysis are limited because the technique is only appropriate when outcomes have been rigorously shown to be equivalent.
Cost consequence analysis (CCA)
In a CCA, the consequences of 2 or more alternatives are measured in addition to the costs. In a CCA, the consequences of each alternative are listed, and decision makers form their own opinions about the relative importance of the findings. For example, researchers evaluated the outcomes and costs of a modified therapeutic community intervention for homeless mentally ill people who abused chemicals versus treatment as usual.8 The incremental costs of the therapeutic community compared with treatment as usual were computed, and the outcomes of each model of care were listed. Readers were left to draw conclusions about the choice of intervention based on costs and outcomes.
Cost effectiveness analysis (CEA)
In CEA, the health outcomes of each alternative must be reported using the same units, such as life years gained or migraine headaches prevented or mm Hg of diastolic blood pressure lowered. Additionally, costs and effects are summarised in a cost effectiveness ratio, which is calculated using the following formula:where C1 = the cost of the new intervention, C2 = the cost of the comparator, E1 = the effect of the new intervention, and E2 = the effect of the comparator. With CEA, analysts often use a decision analytic approach (ie, a complex mathematical modelling technique) that captures the long term costs and effectiveness.
A few examples of CEA can be found in the nursing literature, such as the cost effectiveness of neonatal nurse practitioners and the cost effectiveness of a free standing birth centre.9,10 In CEAs that use a standard unit for analysis, such as life years gained, the advantage is that comparisons can be made across groups or settings by using league tables. Results of separate analyses from various healthcare settings can be compared, at least in theory, across patient populations.11
Cost utility analysis (CUA)
CUA is a special type of CEA that uses a standardised measurement and includes measures for both the quantity and quality of life. Outcome measures that consider both quality and quantity of life include disability adjusted life years (DALYs), healthy year equivalents (HYEs), and quality adjusted life years (QALYs), which is the most common. The QALY combines patient longevity and individuals’ preferences about different levels of health related quality of life into a single measure, which, in principle, has meaning across all areas of health service activity.12 It is a unit of measure for survival that accounts for the effects of suboptimal health status and the resulting limitations in quality of life. It is usually measured on a scale of 0 (representing death) to 1 (representing full health). Quality adjustment involves placing a lower value on time spent with impaired physical and emotional function than time spent in full health: the greater the impairment, the lower the value of a particular health state. It is not realistic to expect an individual to sustain the same quality of life over time; rather, people move in and out of health states (eg, a person has a stroke and then recovers). Individual QALYs are calculated by multiplying the time spent in each health state by the preference for that health state. These are then summed to arrive at an overall QALY. In a classic study, Boyle et al used a utility measure to calculate that treating critically ill infants weighing 1000–1499 g at birth cost CN $3200 per QALY gained, whereas treating infants with birth weights of 500–999 g cost $22 400 per QALY gained.13
Cost benefit analysis (CBA)
In CBA, outcomes are measured according to a monetary unit. A single currency figure, representing benefits minus costs, is calculated. The challenge of this approach is that many healthcare situations are difficult to quantify financially and require a value judgment. Furthermore, there are ethical concerns surrounding the assignment of a monetary amount to the value of human life.14 Because of these issues, the use of CEA has been more prevalent in the healthcare literature than CBA.3 An advantage of using CBA in health care is that the results may be compared to other areas of government investment (eg, school education or transportation safety).
ARE THE ALTERNATIVE COURSES OF ACTION APPROPRIATE AND WELL DEFINED?
In determining the validity of any economic evaluation, you must decide if the alternative ways of providing care being evaluated in the study are appropriate and well defined. For example, researchers did a study to determine if a transprofessional model of home health care could produce savings in service delivery costs compared with usual home care for patients with AIDS who were terminally ill.15 The researchers found no improvements in clinical outcomes, but did find an 8% reduction in delivery costs for patients who received transprofessional care management. Of key importance was that the new intervention (ie, transprofessional care management) was clearly articulated and appropriately compared with current usual care.
In an economic analysis, it is not necessary to view an intervention as “all or nothing”. The costs and consequences of treatment are likely to vary depending on patients’ levels of risk. The greater a patient’s risk, the lower the cost per unit of benefit. Therefore, subgroups with different levels of risk need to be identified, and costs and consequences need to be provided separately for each subgroup. Certain interventions may only be economically efficient in high risk individuals.
IS THE PERSPECTIVE OF THE ANALYSIS STATED, AND ARE APPROPRIATE COSTS CONSIDERED?
Table 3 lists some of the resources frequently measured in economic evaluations. When assessing costs, 2 components drive the measurement: (1) the perspective of the analysis and whether the appropriate resources were considered for that perspective; and (2) how the value (or cost) of the resources was determined.
The perspective of the analysis
Central to any economic evaluation is the issue of whose costs and benefits are to be considered. Costs and benefits might be seen differently from the points of view of society, the patient, the payer, or the provider. The appropriate resources to consider vary depending on the perspective from which the analysis is done and the question being asked.16,17 For example, an intensive care unit (ICU) manager may only be interested in costs incurred while patients are in the ICU. Therefore, only resources consumed by the intervention or the alternatives and the length of ICU stay may be of interest. However, a hospital administrator would be interested in the resources used during the entire hospital stay. Often, resources related to the intervention itself and those associated with net downstream healthcare utilisation are of interest. For example, an insurer or national healthcare service will be interested in the direct costs associated with the initial hospital stays, including resources related to supplies and staffing as well as downstream resource utilisation, such as hospital readmissions, outpatient visits, and other treatments associated with the diagnosis of bloodstream infection and other related diseases (eg, sepsis). Furthermore, when conducting an analysis from a societal perspective, costs and effects are included regardless of who accrues them. Therefore, non-healthcare resources are included.
Ideally, all economic evaluations should have a societal perspective. However, because of the difficulty of assessing societal perspectives as well as the specific concerns of decision makers, analyses are often done using a more narrow perspective. The danger in using a narrow perspective is that costs may simply be shifted. For example, the cost savings related to a shorter hospital stay may be shifted to outpatient visits and to the family in terms of extra caregiver burden.
Valuing resources or costs
Analysts vary in how they value resources.18 In healthcare environments, charges generally do not equal true costs. Third party payers negotiate payment for services rendered based on the cost of the service and an agreed upon profit margin. This occurs in both for profit and not for profit institutions. In order for healthcare institutions to generate more revenue, fee for service customers are often asked to pay full charges (ie, a higher rate of pay). This is called institutional “cost shifting.” Therefore, in the US, many analysts use cost to charge ratios (CCRs), which are calculated by dividing the total costs in a given cost centre by the total charges for the same resource. CCRs are recognised as a gross adjustment to charges. They are better than using charges alone, but not as accurate as cost accounting systems.
Additionally, because 1 dollar (or 1 pound) in 1980 does not have the same purchasing power as a current dollar (or pound), the costs from different years must be calculated and placed into a standard year format. Standardisation of all costs to the same currency and year is essential.
IF CUA METHODS ARE USED, ARE THE UTILITIES (PATIENT PREFERENCES) REASONABLE?
Methods for eliciting utilities (patient preferences) vary and are the focus of a unique body of science and research.19–22 However, a systematic review found that many similar health states have been valued quite differently (ie, mild angina had a low value of 0.7 and a high value of 0.95) and these assigned values were often based on the opinions of the authors rather than rigorous assessment of patient or community based preferences.23 Therefore, careful attention is warranted when assessing preference weight scores and how they were derived. Increasingly, utilities are being collected for various health states.24,25
ARE THE EVENTS USED COMPREHENSIVE, AND ARE THE PROBABILITIES OF EVENTS DERIVED FROM CREDIBLE SOURCES?
For each alternative (eg, intervention) considered, there are probabilities of events occurring (eg, desired outcomes or side effects). Decision makers need to assess if the events included are comprehensive, drawing on their own clinical expertise or collaborating with clinical experts. Additionally, the reliability of the evidence should be considered. Some economic evaluations are done in conjunction with randomised controlled trials (RCTs), whereas others are based on existing evidence from published RCTs and meta-analyses. Because the economic evaluation may include modelling of long term outcomes, analysts often identify probabilities of these events from various sources.
ARE DATA ON COSTS AND OUTCOMES APPROPRIATELY DISCOUNTED?
After all costs and benefits have been calculated, future costs and benefits are discounted to present value. Discounting reflects the principles of time preference and opportunity costs. Time preference suggests that people place greater value on something they have today than on something they will have in the future. Interest rates reflect these same principles — if you forego the opportunity of buying something today and invest the money, you will enjoy a higher rate of return in the future. Therefore, future costs and benefits are discounted to present value using the following formula:where F = the future value, r = the discount rate, and n = the number of years.25 Currently, most experts recommend using a 3% discount rate to discount both costs and effects.26
IS UNCERTAINTY IN THE DATA ADEQUATELY ADDRESSED?
Decision making in health care is inevitably done in a context of uncertainty. Sensitivity analysis and cost effectiveness acceptability curves determine the degree to which this uncertainty could influence conclusions about the economic impact of clinical decisions. Through the use of these techniques, results are calculated separately, varying uncertain estimates of risks, benefits, and values over a reasonable set of parameters.26,27 This allows analysts to determine how different the results of the analysis might be given the different possible estimates. For example, in a univariate sensitivity analysis, a parameter such as a utility (preference) weight is varied and indicates the degree of influence the particular value has on the outcome of the entire analysis. Although univariate sensitivity analyses are insightful, looking at only one variable is usually inadequate. Multivariate sensitivity analysis and cost effectiveness acceptability curves examine multiple sources of uncertainty at one time and may generate a more accurate estimate of cost effectiveness under varying conditions and better inform decision makers. Assessing uncertainty is an important element of a sound economic evaluation.
What are the results of this economic analysis?
Once it has been determined that the economic analysis is valid, the results of the analysis can be examined. To do this, we must consider whether the intervention will provide a benefit at an acceptable cost. For example, if a CMA was done, we should consider if the difference in cost is large enough to warrant switching to the new intervention. If CUA was done, we should consider how the costs/QALY compare with other interventions.
HOW DO THE RESULTING COSTS OR COSTS/UNIT OF HEALTH GAINED COMPARE WITH OTHER INTERVENTIONS?
Given finite resources, it is often necessary to make choices among interventions. League tables list cost effectiveness results from several different studies and, in theory, allow readers to compare the cost effectiveness of different interventions. Several such tables are available in the published literature, but, because of the wide variation in methods used in the original studies, these tables may be misleading if taken at face value.28 In one league table, all results were standardised to a common currency and attempts were made to ensure correct calculation of cost effectiveness ratios to improve comparability of the results across analyses.29 However, the assumptions made in the individual analyses, such as the choice of comparator, were not judged.
ARE THE CONCLUSIONS LIKELY TO CHANGE WITH SENSIBLE CHANGES IN COSTS AND OUTCOMES?
Readers of economic evaluations often wish to judge the robustness of the results given different assumptions or scenarios. For example, Meenan et al found that an in-hospital smoking cessation counselling and education programme was more effective than current practice for helping patients to stop smoking at 1 year after their hospital stay (13.5% in the smoking cessation programme group v 9.2% in the control group). Incremental cost of the intervention was US $3680 per incremental discounted life year saved.30 However, only 52% of participants provided samples for biochemical confirmation of abstinence. Because of this low rate, it is possible that some participants misreported their smoking status because of a social desirability factor. However, the authors used sensitivity analysis and found that even when the cessation rate was varied from 8.0% to 0.6%, the intervention was still cost effective, with an incremental cost per incremental discounted life year saved of US $1978 to $26373.
Will the results help me in caring for patients?
The applicability or generalisability of the results to your own setting needs to be considered. In an economic evaluation, the applicability of both the costs and effectiveness are assessed.
DO THE COSTS IN THE REPORT APPLY IN MY OWN SETTING?
Costs may vary because of local prices and different practice patterns. The sensitivity analysis should cover a range of costs that may account for such differences.
WILL THE INTERVENTION (OR NEW MODEL OF CARE) BE EFFECTIVE IN MY SETTING?
As always, you need to consider how your patients are similar to or different from those in the study. You also need to assess your patients’ preferences for the outcomes and utilities used in the analysis.
The interdisciplinary team charged with reviewing the literature on the new antiseptic impregnated catheter, searched Medline, the NHS Economic Evaluation Database, and the Health Technology Assessment Database (the latter 2 are part of the Cochrane Library) using the terms costs, central lines, and catheter related infection. The study thought best to answer the question is summarised in Table 4.
Veenstra et al assessed the cost effectiveness of short term use (2–10 d) of a standard multilumen central venous catheter compared with a catheter impregnated with chlorhexidine silver sulfadiazine for a hypothetical cohort of patients at high risk for catheter related infections.31 The economic evaluation method chosen, CEA, was appropriate for the question. The 2 alternative interventions were well defined and appropriate costs for each of the alternatives were considered in the analysis. Direct medical costs were measured in 1998 dollars and included costs related to incremental length of stay in the ICU and ward for management of catheter related bloodstream infections, hypersensitivity reactions, and locally infected catheter insertion sites.
The probabilities of clinical events were well documented. Probabilities for catheter related bloodstream infections and catheter colonisation were based on summary risk ratios calculated from a meta-analysis of 13 RCTs that compared outcomes for antiseptic impregnated catheters with standard central venous catheters. This meta-analysis included results from clinical trials in which most patients were at high risk (such as patients in ICUs or with immunosuppression). Probability of death attributable to catheter related bloodstream infections was based on reports in the literature. The authors found no reported cases of hypersensitivity to chlorhexidine silver sulfadiazine impregnated central venous catheters in the US, so they used rates from Japan. The analysis showed that antiseptic impregnated central venous catheters had greater efficacy and lower costs than standard catheters. These results remained consistent through an appropriate series of multivariate and univariate sensitivity analyses.
Although this study does not allow us to answer our first question about the total population of hospital patients requiring catheters, it does inform clinical decision making related to patients at high risk of catheter related infection. We have learned through appraisal of a high quality study that antiseptic impregnated central venous catheters are cost effective in high risk patients. Based on this economic evidence, the interdisciplinary team recommends use of the new catheters in their ICUs but not throughout the hospital until further studies are done.