Original Contribution

risk tolerance for the exclusion of potentially life-threatening diseases in the ED

Jesse M. Pines MD, MBA^a,b,*, Demian Szyld MD^a

^aDepartment of Emergency Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA 19104, USA

^bCenter for Clinical Epidemiology and Biostatistics, Philadelphia, PA 19104, USA

Received 1 September 2006; revised 4 October 2006; accepted 13 October 2006

Abstract

Objective: Given the same pretest probability (10%) for subarachnoid hemorrhage , pulmonary embolism (PE), and acute coronary syndrome (ACS), we determined if differences exist in the risk tolerance for disease exclusion according to published guidelines given a negative test result.

Methods: Published guidelines that make practice recommendations on the evaluation of ACS, PE, and SAH were sought using the National Guideline Clearinghouse in low-risk settings. Second-order Monte Carlo simulation was performed to determine point estimates and confidence intervals (CIs) for posttest probabilities assuming a pretest probability of 10%.

Results: Guidelines recommend that patients with low-risk suspected ACS should undergo Stress testing. For SAH, computed tomography (CT) followed by lumbar puncture is recommended without mention of pretest probability; and D-dimer testing is recommended to exclude PE in low-risk patients. Test sensitivity for thallium-201 single photon emission computed tomography (SPECT) was 89%, exercise echocardiogram was 85%, D-dimer testing was 95%, and CT/LP for SAH was 100% (as a gold standard) and CT only was 97.5%. Given a negative test result, for PE, posttest probability was 0.5% (95% CI 0.1%-0.9%); for SPECT, 1.1% (SD 0.5%-1.6%); and for exercise echocardiogram, 1.5% (95% CI 0.5%-2.5%) compared with a posttest probability of 0% for CT followed by LP for SAH. Using a CT-only approach gives a posttest probability of 0.2% (95% CI 0.2%-0.4%).

Conclusions: Guidelines for suspected PE and ACS allow small but nonzero calculated risk end points in low-risk settings, whereas SAH guidelines afford no misses. Because many gold standard tests are more invasive and can have adverse effects, guideline authors should consider adopting a standard acceptable miss rate as an end point for workups with low clinical suspicion to avoid the overuse of invasive testing.

D 2007

* Corresponding author. Department of Emergency Medicine, Hospital of the University of Pennsylvania, 3400 Spruce Street, Ground Ravdin, Philadelphia, PA 19104. Tel.: +1 215 662 4050.

E-mail address: [email protected] (J.M. Pines).

Introduction

Diagnostic testing in the emergency department (ED) is an integral part of practice and a considerable proportion of the cost of medical care. Of the 113 million US ED visits in 2003, 43% of patients had imaging studies, 36% had one or more blood tests, 18% had urinalysis, and 16% had

0735-6757/$ - see front matter D 2007 doi:10.1016/j.ajem.2006.10.011

electrocardiograms [1]. Many clinical practice guidelines recommend tailored Diagnostic strategies based on pretest probability of disease, where different tests are recommen- ded in high-risk versus low-risk situations. In High-risk situations, often, a gold standard test is recommended to afford no misses. A gold standard test is defined as one that has 100% sensitivity; that is, all people with the disease will have a positive test. However, when a non-gold standard test is negative, patients still have a small chance of having the disease as determined by the false-negative rate.

We define this small nonzero posttest probability as the risk tolerance for the exclusion of disease. In order words, risk tolerance is the remaining probability of disease given a negative test result; risk tolerance is the level of uncertainty after diagnostic testing at which the physician can still reassure patients that their symptoms are not caused by the disease in question. Risk tolerances are particularly impor- tant in emergency medicine practice because often the focus of the diagnostic evaluations is to exclude any life- threatening or debilitating disease in low-risk situations.

Gold standard tests are pulmonary angiography for pulmonary embolism (PE), cardiac catheterization for acute coronary syndrome (ACS) and coronary artery disease, and noncontrast head computed tomography followed by lumbar puncture for atraumatic subarachnoid hemor- rhage (SAH). All of these tests are invasive and carry potential risks to patients [2-4]. For patients at low risk for suspected PE and ACS, current guidelines do not recom- mend gold standard testing. [5,6] By contrast, the 1996 American College of Emergency Physicians headache guideline recommends gold standard testing when SAH is suspected and does not mention of pretest probability [7].

We sought to determine whether the risk tolerance for disease exclusion allowed by published guidelines is different given the same pretest probability. We hypothe- sized that because gold standard testing is not recommen- ded for the evaluation of low-risk PE or ACS, risk tolerance for the exclusion of these diseases would be higher than that for SAH.

Methods

We performed second-order Monte Carlo simulation to estimate posttest probabilities for ACS, PE, and SAH

given a negative testing using a pretest probability of 10%. Monte Carlo methods are algorithms used for Simulation modeling of physical and mathematical systems. Monte Carlo models are stochastic (ie, nondeterministic), using random numbers chosen from known or estimated prob- ability distribution functions. In the case of this study, all probability distributions for diagnostic testing were esti- mated from the literature. In Monte Carlo modeling, a large number of simulations can be run using this random sampling technique from estimated probability functions to predict statistical error in the average result. Fig. 1 shows the model in TreeAge Pro version 6 (Williamstown, MA; 2005). Guidelines on diagnostic testing were determined by searching the National Guideline Clearinghouse (http:// www.guideline.gov) specifically for guidelines on evalua- tion of low-risk cases in the ED. We then searched Pubmed (www.pubmed.gov), which includes MEDLINE, for published meta-analyses on test sensitivity. Where meta-analyses were unavailable, we pooled data together from the best available evidence to determine a point estimate and distribution for test sensitivity. The b distribution was used to model sensitivity as is common practice in Monte Carlo simulation.

For each diagnostic test, given a set pretest probability of

10%, the model was run 1000 times, where for each run test sensitivity was chosen at random from the b distribution. This sampling strategy functioned as the sensitivity analysis (ie, probabilistic sensitivity analysis) inherent to second- order Monte Carlo sampling. Posttest probabilities were considered significantly different from zero if the lower bound 95% confidence interval (CI) did not cross zero.

Results

Two guidelines were found that addressed the evaluation of the 3 conditions with Low pretest probability (for PE and ACS) and one without regard to pretest probability (SAH). The American College of Cardiology recommends that patients with low-risk suspected ACS should undergo risk stratification with stress testing [6]. The American College of Emergency Physicians published recommendations on aTraumatic SAH and PE. For SAH, a CT followed by LP is recommended without mention of pretest probability [7]. D-Dimer testing is recommended to exclude low-risk PE [5].

Fig. 1 Monte Carlo model of diagnostic testing. A b1Q is shown for a misdiagnosis; a b0Q is shown either for a correct diagnosis; 0 is for a correct diagnosis (whether it is present or absent). Pretest indicates pretest probability; sensitivity, test sensitivity.

Table 1 Expected posttest probabilities given negative

testing at a pretest probability of 10%

Suspected disease/test

SAH

CT followed by LP CT only

ACS

Exercise echocardiogram Thallium-201 SPECT

PE

D-Dimer

Posttest probability (95% CI)

0%^a

0.2% (0.0%-0.4%)

1.5% (0.5%-2.5%)

1.1% (0.5%-1.6%)

0.5% (0.1%-0.9%)

^a Estimated probability is equal to zero because this is the gold standard.

Sensitivity of stress thallium-201 SPECT was 89% [8], and that of stress echocardiogram was 85% [9]. The sensitivity of D-dimer testing was 95% [10]. The sensitivity of CT/LP is 100% because it is by definition the gold standard. There were no meta-analyses of diagnostic testing for SAH. The reported sensitivity for Head CT. performed in the first 12 to 24 hours with fifth-generation CT scanners is 100% [11]; reported sensitivities with third-generation

scanners are 93% [12], 100% [13], 98% [14], and 93% [15]. We pooled all these data for a combined sensitivity of 97.5% (522 of 535) for this analysis.

Given a pretest probability of 10%, posttest probabilities given a negative test were significantly different from the posttest probability for SAH, which was zero. Excluding the required LP (ie, noncontrast head CT only) gives a posttest probability of 0.2% (95% CI 0.0%-0.4%) for SAH. For PE, the posttest probability is 0.5% (95% CI 0.1%-0.9%). For evaluation of ACS, exercise echocardiogram leaves a posttest probability of 1.5% (95% CI 0.5%-2.5%); and thallium-201 SPECT testing leaves a posttest probability of 1.1% (95% CI 0.5%-1.6%) (Table 1).

Discussion

We found that the posttest probabilities allowed by published practice guidelines were considerably different given the same pretest probability. These theoretical posttest probabilities are dependent on specific test recommendations and inherent test sensitivity. Specifically, the recommended risk tolerance for the exclusion of SAH was zero because the gold standard evaluation is always recommended, whereas small but theoretical nonzero risk tolerances are allowed for ACS and PE in low-risk settings. Interestingly, the posttest probability for SAH given a negative head CT alone (a strategy that is not endorsed by any professional society or guideline) is considerably lower than the posttest probability allowed for the evaluation of low-risk ACS with either exercise echocardiography or pharmacologic stress testing.

In the context of clinical practice, physicians come away from the patient encounter with a differential diagnosis,

where the probability of a given diagnosis is assigned a pretest probability either explicitly or implicitly. The practice and training in emergency medicine stress the consideration of the most likely diagnosis but never lose focus on excluding potentially lethal or disabling diagnoses, even in those with minor complaints [16]. Afterward, either through the history and physical examination or through diagnostic testing, practitioners must either rule in or exclude the most lethal diseases. When there is a low chance of the disease in question, non-gold standard tests are often ordered to further reduce the probability of the presence of a disease and confirm the clinical suspicion of the absence of the dangerous disease at the end of the ED evaluation.

The ED evaluations of PE, ACS, and SAH are similar. All can present subtly and atypically, and each disease has evidence-basED treatments that can improve outcomes. In addition, a missed diagnosis in the ED of any of the 3 conditions can be life threatening; and all are high risk with regard to medical litigation [17]. In the context of a busy ED, the patient presenting with symptoms suggesting that any of these conditions may be in the differential diagnosis represents a low-probability but high-risk situation. low probability and high risk refer to the experience of both the patient and the evaluating physician because failure to make the diagnosis has grave implications for both (potential for death or disability and potential for litigation and loss of income).

For patients who are low risk for PE and ACS, guidelines have recognized that given the risks of gold standard testing, the use of pulmonary angiogram and/or cardiac catheteri- zation on every patient with chest pain is impractical. By contrast, the 1996 headache guideline recommends invasive gold standard testing for every patient with suspected SAH, regardless of pretest probability. Although a guideline certainly does not define the standard of care in all cases, the fact that this guideline exists may lower the threshold of physicians to recommend invasive LP even in cases of headache who are low risk for SAH. In addition, physicians who decide to perform a head CT on patients with headaches may feel obligated to recommend LP despite the fact that there is only a remote possibility that this will add important clinical information to the case given that the posttest probability only decreases from 0.2% (1 in 500 patients) to zero. Although no cases are missed in the extreme (LP everyone with headache), this may lead to a tendency to overtest. Given that the rate of postdural headache after LP has been estimated to be from 11% to 41% [18,19], it may not be in the best interest of the patient or the physician performing the test, not to mention the additional Economic burden to the medical system repre- sented by repeated ED visits and to the economy repre- sented by missed days of work. Comparing the 0.2% reduction (from 1 in 500 to 0 in 500) in posttest probability with the considerable risk of postdural headache in low-risk settings may be a way to present information to patients regarding the potential risks and potential benefits of LPs.

One potential reason for this discrepancy in testing recommendations and practice may be that for atraumatic SAH, emergency physicians can perform diagnostic LP without the involvement of consulting physicians such as cardiologists and radiologists. There is also a spectrum effect in the evaluation of acute SAH, where minor cases and cases where days have passed because the initial sentinel bleed has occurred can change diagnostic sensi- tivity of head CT. In addition, there are no validated decision rules to date that can reliably either risk stratify or exclude patients with SAH, although these risk stratifica- tion tools exist for both PE and ACS [20,21]. Although headache (3 million ED visits per year) and chest pain (5 million ED visits per year) are common, the number of cases of suspected SAH (ie, worst headache ever) requiring head CT and LP may not be as common as the patient with chest pain that requires simple routine diagnostic testing (electrocardiogram, cardiac markers, and/or D-dimer test- ing) to evaluate for PE and/or ACS. There are also differences in prevalence: ACS and PE are much more common diseases in the population than atraumatic SAH, making invasive testing in the latter disease a true case of needle in a haystack.

So how can emergency physicians best determine the optimal risk tolerance for disease exclusion? In everyday clinical practice, this threshold is determined by the individual risk behavior of the emergency physician. This risk threshold may be determined by the physician’s experiences both in and out of the hospital, Clinical training, or malpractice fear. Whereas some physicians may be satisfied with a negative head CT for SAH, others may be more risk averse and recommend diagnostic LP to all patients even in the lowest-risk cases. It has been shown that physician risk aversion is predictive of admission in low- risk patients with chest pain [22].

One approach to answering the question of the most appropriate risk tolerance for disease exclusion may be economic, whereby diagnostic strategies are compared based on costs, outcomes, and quality of life. In these models, approaches that are below the threshold of $50000 per quality-adjusted-life-year (QALY) are considered cost effective. A criticism of this approach is that it assumes a constant willingness of pay; that is, although $50000/QALY is a common threshold, Bill Gates may be willing to pay more per QALY. Inasmuch as the risk in diagnostic testing is largely assumed by the patient (risk of death, pain, missed diagnosis, complication, cost of the test, cost of adverse event, etc), a practical approach used by many physicians is to include patients in the decision-making process. By considering patient preferences in determining the most appropriate diagnostic strategy, patients and their physicians may be able to weigh the risks and benefits of invasive testing to determine risk tolerance for each individual patient encounter.

Although these strategies are reasonable, there still exists no commonly accepted cutoff below which a life-threaten-

ing disease should be excluded. Thus, there are differences in guidelines for allowable risk tolerance among life- threatening diseases. Authors of guidelines should consider developing a standard acceptable miss rate (such as 1%, 0.5%, or 0.25%) or consider patient preferences to provide a framework for recommending testing in low-risk cases where there are risks (such as pain, adverse effects, or otherwise) to patients who undergo gold standard testing.

Limitations

The primary limitation of this study is the use of a Mathematical model to describe clinical decision making and the assumption that the model reflects real-life clinical decision making. There was no direct comparison of actual patients in this study. Another limitation is the noninclusion of test specificity and the potential need for additional testing (with its associated cost and risk) due to false- positive tests. An additional limitation may be the accuracy point estimates for test sensitivity. Specifically with regard to the point estimate for the sensitivity of CT for SAH, because there is a spectrum effect and a time effect, test sensitivity may actually be lower than 97.5% for patients with minor symptoms or for patients who do not present early in the course of the disease.

Conclusion

Guidelines for suspected PE and ACS allow small but nonzero diagnostic end points in low-risk settings, whereas SAH guidelines allow no misses. Because gold standard tests carry considerable risks, authors of guidelines should consider developing and adopting a standard acceptable miss rate as a diagnostic end point for workups with low clinical suspicion to avoid the overuse of invasive testing.

References

1. Advance data from vital health statistics. The National Hospital Ambulatory Medical Care Survey 2003. Centers for Disease Control, 2005;358:1 - 38.
2. Shah KH, Richard KM, Nicholas S, Edlow JA. Incidence of traumatic lumbar puncture. Acad Emerg Med 2003;10:151 - 4.
3. James J, O’Meara MD, Gregory J, Dehmer MD. Care of the patient and management of complications after percutaneous coronary artery interventions. Ann Intern Med 1997;127(6):458 - 71.
4. Stein PD, Athanasoulis C, Alavi A, Greenspan RH, Hales CA, Saltzman HA, et al. Complications and validity of pulmonary angiography in acute pulmonary embolism. Circulation 1992;85: 462 - 8.
5. Clinical policy: critical issues in the evaluation and management of adult patients presenting with suspected pulmonary embolism. Ann Emerg Med 2003;41(2):257 - 70.
6. American College of Cardiology Foundation, American Heart Association. ACC/AHA guidelines for the management of patients

with unstable angina and Non-ST-segment elevation myocardial infarction. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Bethesda (MD)7 American College of Cardiology Foundation (ACCF); 2002. Accessed May 30, 2006 at http://www.guideline.gov/summary/

summary.aspx?doc_id=3190.

1. Clinical policy for the initial approach to adolescents and adults presenting to the emergency department with a chief complaint of headache. Ann Emerg Med 1996;27:821 - 44.
2. Kim C, Kwok YS, Heagerty P, et al. Pharmacologic stress testing for coronary disease diagnosis: a meta-analysis. Am Heart J 2001;142: 934 - 44.
3. Fleishmann KE, Hunink MGM, Kuntz KM, et al. Exercise echocar- diography or exercise SPECT imaging? A meta-analysis of diagnostic test performance. JAMA 1998;280:913 - 20.
4. Brown MD, et al. The accuracy of the enzyme-linked immunosorbent assay d-dimer test in the diagnosis of pulmonary embolism: a meta- analysis. Ann Emerg Med 2002;40:133 - 44.
5. Boesiger BM, Shiber JR. Subarachnoid hemorrhage diagnosis by computed tomography and lumbar puncture: are fifth generation CT scanners better at identifying subarachnoid hemorrhage? J Emerg Med 2005;29:23 - 7.
6. Sames TA, Storrow AB, Finklestein JA, et al. Sensitivity of new generation computed tomography in subarachnoid hemorrhage. Acad Emerg Med 1996;3:16 - 20.
7. Sidman R, Connelly E, Lemke T, et al. Subarachnoid hemorrhage diagnosis: lumbar puncture is still needed when the computer tomography scan is normal. Acad Emerg Med 1996;3:827 - 31.
8. van der Wee N, Rinkel GJ, Hasan D, van Gijn J. Detection of subarachnoid haemorrhage on early CT: is lumbar puncture still

needed after a negative scan? J Neurol Neurosurg Psychiatry 1995; 58(3):357 - 9.

1. Morganstern LB, Luna-Gonzales H, Huber JC, et al. Worst headache and subarachnoid hemorrhage: prospective, modern computed tomog- raphy and spinal fluid analysis. Ann Emerg Med 1998;32:297 - 304.
2. Marx JA, Hockberger RS, Walls RM, editors. Rosen’s emergency medicine: concepts and clinical practice. 5th ed. St Louis (Mo)7 Mosby; 2004. p. 110.
3. Karcz A, Holbrook J, Auerbach BS, et al. Preventability of malpractice claims in emergency medicine: a closed claims study. Ann Emerg Med 1990;19:865 - 73.
4. Thomas SR, Jamieson DRS, Muir KW. Randomized controlled trial of atraumatic versus standard needles for Diagnostic lumbar puncture. BMJ 2000;321:986 - 90.
5. Strupp M, Brandt T, Muller A. Incidence of post-lumbar puncture syndrome reduced by reinserting the stylet: a randomized prospective study of 600 patients. J Neurol 1998;245(9):589 - 92.
6. Antman EM, Cohen M, Bernink PJ, McCabe CH, Horacek T, Papuchis G, et al. The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 2000;284(7):835 - 42.
7. Wells PS, Anderson DR, Rodger M, Stiell I, Dreyer JF, Barnes D, et al. Excluding pulmonary embolism at the bedside without diagnostic imaging: management of patients with suspected pulmonary embo- lism presenting to the emergency department by using a simple clinical model and d-dimer. Ann Intern Med 2001;135(2):98 - 107.
8. Pearson SD, Goldman L, Orav EJ, Guadagnoli E, Garcia TB, Johnson PA, et al. Triage decisions for emergency department patients with chest pain: do physicians’ risk attitudes make the difference? J Gen Intern Med 1995;10(10):557 - 64.