a b s t r a c t

Background: Infectious endocarditis (IE) in febrile Injection drug users (IDUs) is a critical diagnosis to identify in the emergency department (ED). A decision tool that identifies patients at Very low risk for endocarditis using readily available clinical data could reduce admissions and cost.

Objective: To evaluate the diagnostic performance of a previously derived decision instrument to rule out endocarditis in febrile IDUs (Prediction Rule for Endocarditis in Injection Drug users [PRE-IDU]) and to develop a prediction model for likelihood of endocarditis for those who are not ruled out by PRE-IDU. Methods: Febrile IDUs admitted to rule out endocarditis were prospectively enrolled from 2 urban EDs in June 2007 to March 2011. Clinical data from ED presentation (first 6 hours) and outcome data from inpatient records were recorded and reviewed by 2 independent investigators. Diagnosis of IE was based on modified Duke criteria and discharge summaries. The diagnostic performance of PRE-IDU, which combines tachycardia, cardiac murmur, and absence of skin infection, was determined using recursive partitioning and logistic regression modeling.

Results: Of the 249 subjects, 18 (7%) had IE. Recursive partitioning yielded an instrument with 100% sensitivity (95% confidence interval [CI], 84%-100%) and 100% negative predictive value (95% CI, 91%-100%), but low specificity (13%; 95% CI, 12%-13%). Multiple logistic regression modeling with the 3 Clinical predictors allowed risk stratification with posttest probabilities ranging from 3% to 20%.

Conclusion: The PRE-IDU instrument predicted IE with high sensitivity and ruled out IE with high negative predictive value. Our logistic regression model provided posttest probabilities ranging from 3% to 20%. The PRE-IDU instrument and the associated model may help guide hospital admission and diagnostic testing in evaluation of febrile IDUs in the ED.

(C) 2014

Introduction

Infective endocarditis (IE) has an incidence of 1 to 20 cases per 10 000 injection drug users (IDUs) every year and accounts for 5% to 20% of hospital admissions in this population [1-3]. Given its associated complications as well as a 5% to 10% mortality rate, accurate diagnosis of IE is imperative [1,3]. However, diagnosis of IE in the emergency department (ED) remains challenging. Although previous studies have suggested an association between IE and clinical criteria such as urine sediment, higher median temp, recent intravenous drug use

? Grants: This study was supported by the Resident Research Grant provided by the Clinical and Translational Science Institute at the University of California San Francisco.

?? The contents of this manuscript were presented at: SAEM Annual Meeting,

Chicago, IL, on May 10, 2012; ACEP Scientific Assembly, San Francisco, CA, on October 16, 2011; and SAEM Western Regional Research Forum, Las Vegas, NV, on March 17,

2012.

? The authors have no financial disclosures.

* Corresponding author. San Francisco General Hospital, Box 1377, San Francisco, CA 94143-0208.

E-mail address: [email protected] (R.M. Rodriguez).

within the last 5 days, and mild hyponatremia [1,4-6], attempts to risk stratify febrile IDUs based on these criteria and clinical assessment in the ED were unsuccessful [2-5,7]. Therefore, current standard of practice mandates admission for IDUs with fever of unclear etiology for blood cultures and echocardiography [3-5]. A decision instrument (DI) that could reliably identify IDUs at low risk for IE using clinical and laboratory criteria available in the ED could spare admissions and guide further testing for more efficient resource use.

To this end, we previously developed a DI based on ED clinical data using recursive partitioning modeling techniques: Prediction Rule for Endocarditis in Injection Drug Users (PRE-IDU). We identified tachycardia, lack of skin infection, and cardiac murmur as 3 criteria whose combination into a DI yielded 100% sensitivity (95% confidence interval [CI], 84%-100%) and 100% negative predictive value (95% CI, 88%-100%) for IE [8]. Our initial model was designed to generate a “directive” yes/no algorithm to identify very-low-risk patients who may be safe for discharge, maximizing sensitivity at the price of low specificity (14%, 95% CI, 10%-20%). Alternatively, an “assistive” modeling approach using logistic modeling to obtain posttest likelihood ratios to guide medical decision making may be useful to

0735-6757/$ - see front matter (C) 2014 http://dx.doi.org/10.1016/j.ajem.2014.01.008

provide risk stratification for patients who do not meet low-risk criteria [9]. The objectives of our study were (1) to validate PRE-IDU for the prediction of IE in an independent, prospectively enrolled cohort of IDUs admitted to rule out endocarditis, and (2) to generate a logistic model with likelihood of IE ratios for those patients explicitly not meeting PRE-IDU low-risk criteria.

Methods

Study design and patient enrollment

From June 2007 to March 2011, we prospectively enrolled patients from 2 urban, county EDs, each with an approximate annual census of 60 000 patients. We used the following inclusion criteria: (1) history of injection drug use, (2) age N 17 years, (3) fever (temperature >= 38.0?C), (4) admission to the hospital, and (5) ED diagnosis of “rule out endocarditis,” “shooter with a fever,” “fever without source,” or “IDU with fever.” Patients were excluded from analysis if they left against medical advice prior to diagnostic workup (blood cultures and echocardiograms). Study protocols were approved by the respective institutional review boards. We calculated a target sample size of 588 patients to generate a narrow CI for our sensitivity point estimate, but due to funding constraints and loss of one study site, we were unable to meet our target enrollment and terminated the study at the half- way point of enrollment.

Data collection, criteria, and outcome

Study personnel reviewed medical records and abstracted data according to the guidelines proposed by Gilbert et al [10]. Multiple quality assurance measures, including standardized data abstraction forms and protocols, double data entry checking, regular meetings of abstractors, and interrater assessments of abstractors, were imple- mented. With blinding to subjects’ criteria data, we determined the outcome classification of subjects. Discrepancies in data after abstraction, which were less than 2% of data elements, were resolved by consensus of the authors.

The following ED clinical, laboratory, and radiography data were collected using template medical records and confirmed through review of computer records: (1) history of IE, (2) HIV status,

(3) presence or absence of tachycardia (heart rate N 100 beats/min at any time during the first 6 hours of ED stay), (4) cardiac murmur (as assessed by the ED provider), (5) skin infection (abscess or cellulitis) on physical examination, (6) leukocytosis (N 11.7 x 1000 cells/mm³),

(7) hyponatremia (b 136 miliequivalents/L), (8) thrombocytopenia (b 150 x 1000 platelets/mm³), and (9) presence of infiltrates or cavitations on ED chest x-ray (based on blinded final radiologist reading). These criteria for the logistic model were chosen on the basis of our pilot study and review of past studies [8]. Missing and unrecorded data elements, which were less than 1% of the total, were excluded from analysis.

Outcome data pertaining to the final diagnosis of IE including microbiology, echocardiogram, and discharge diagnoses were abstract- ed from discharge summaries and final signed reports. Diagnosis of IE was confirmed if a patient received a diagnosis of endocarditis in their discharge summaries and if they met the modified Duke criteria [11]. Discrepancies in outcomes were resolved by consensus of investigators.

Data analysis and statistical methodology

All data were entered into Microsoft Access (Microsoft Corp, Seattle, WA) and analyzed using Stata 12 (Stata Corporation, College Station, TX) and SAS v. 9.2 (SAS institute Inc, Cary, NC). We evaluated the screening performance (sensitivity, specificity, and odds ratio) of PRE-IDU and individual clinical criteria using standard formulae and Clopper-Pearson binomial method for 95% CIs.

To generate an alternative predictive model using logistic regression, we analyzed the original derivation data from the pilot study to identify independent variables with adequate Predictive power [8]. We then derived odds ratios and the logistic coefficients for each of these variables using the logistic regression formula.

Model selection was performed in a stepwise fashion with potential predictor variables added and retained if the associated P value was less than .2. We found no significant 2- and 3-way interactions between variables. We generated a receiver operating characteristic (ROC) curve from the selected optimal model using both derivation (Fig. 1A) and validation data (Fig. 1B) by generating sensitivity and specificity values for a prespecified range of probabil- ities. The area under the curve for the ROC curve was generated using the trapezoidal method, and the SE was generated using the equations set forth by Hanley and McNeil [12].

Results

Of the 296 patients initially identified for possible enrollment, 43 subjects were excluded because they did not have any of the inclusion diagnoses on their list of ED diagnoses. Four subjects left the hospital against medical advice prior to receiving their endocarditis workup and were also excluded. SubjeCT characteristics are summarized in Table 1.

Of the 249 subjects included in the analysis, 18 (7.2%) were diagnosed as having endocarditis. Among the 18 subjects with IE, 16 had positive blood cultures, half of which grew methicillin-resistant Staphylococcus aureus. Sixteen patients had abnormal echocardiograms consistent with endocarditis. Echocardiograms in 2 subjects were initially reported to be normal, but both subjects had evidence of septic emboli to multiple organs and methicillin-resistant S aureus bacteremia.

Fig. 1. Receiver operator characteristic curves for the logistic regression model. A, ROC curve for derivation data. B, ROC curve for validation data.

Table 1

Subject characteristics

	Endocarditis	No endocarditis	Total
	(% or SD)	(% or SD)	(% or SD)
n	18 (7%)	231 (93%)	249
Sex, male	11 (61%)	148 (64%)	159 (64%)
Mean age (y)	42 (11)	42 (10)	42 (10)
Ethnicity
White	8 (44%)	101 (44%)	109 (44%)
African American	5 (28%)	51 (22%)	56 (23%)
Hispanic	0	21 (9%)	21 (8%)
Asian/Pacific	1 (6%)	2 (1%)	3 (1%)
Islander Other	0	3 (1%)	3 (1%)
Unknown	4 (22%)	53 (23%)	57 (23%)
Hospital length of stay	19.2 (14.3)	6.6 (7.1)	7.6 (8.5)
Hospital survival	18 (100%)	221 (96%)	239 (96.0%)
HIV	1 (6%)	66 (30%) 6 unknown	67 (28%)

Validation of the recursive model

The Predictive performance of the recursive model is described in Table 2. In this validation cohort, PRE-IDU had similar screening performance as it did in the derivation cohort: 100% sensitivity (95% CI, 82%-100%), 100% negative predictive value (95% CI, 88%-100%), and 13% specificity (95% CI, 9%-18%). Of the 249 patients, it identified 29 patients (12%) as having low risk for endocarditis.

Logistic regression model

We used our original derivation data to determine the independent variables that held the greatest predictive power for IE. Using this approach, we identified tachycardia, cardiac murmur, and absence of skin infection as variables with most diagnostic power. The odds ratios generated from these variables using both the derivation and prospec- tive data are depicted in Tables 3. The optimal logistic model generated from these variables is represented by the following equation:

P(endocarditis) = 1/ 1 + e^-z ,

z = 0.63[tachycardial] + 0.61[cardiac murmur]-1.0[skin infection]-2.61.

Applying this equation, we created a scoring system and generated posttest probabilities ranging from 3% to 20%, depending on the total number of criteria observed (Table 3). Receiver operating character- istic curves from the optimal model in the prospective data demonstrate an area under the curve of 0.8 +- 0.1 (Fig. 1B).

Discussion

Given the difficulty in identifying endocarditis in the ED and its grave morbidity, the current cautious practice of admitting all IDUs with fever of unclear etiology seems appropriate. However, as in prior studies [1,2,4], we found that this is a low-yield practice-7% of admitted IDUs were ultimately diagnosed with IE. A rule that reliably predicts those with (and more importantly those without) endocarditis could be useful in directing diagnostic testing and limiting hospital admissions.

Table 2

Predictive performance of the recursive partitioning model DI

	Decision instrument
Sensitivity (95% CI)	1.0 (0.8-1.0)
Specificity (95% CI)	0.1 (0.1-0.2)
Positive predictive value (95% CI)	0.1 (0.0-0.1)
Negative predictive value (95% CI)	1.0 (0.9-1.0)
+Likelihood ratio (95% CI)	1.1 (1.1-1.2)
-Likelihood ratio (95% CI)	0
Diagnostic odds ratio (95% CI)	Infinite

Table 3

Derivation and performance of the logistical regression model. Odds ratios for each of the variables were generated from the derivation (a) and validation (b) data. The model was represented as a scoring system which could generate a post-test probability based on the number of factors observed (c)

	(a)
	Variable	Odds ratio	95% CI	P
	Tachycardia	1.9	0.6-6.0	.3
	Cardiac murmur	1.8	0.7-5.0	.2
	Skin infection	0.4	0.1-1.1	.07
	(b)
	Variable	Odds ratio	95% CI	P
	Tachycardia	6.3	1.4-29	.01
	Cardiac murmur	5.6	1.9-17	.002
	Skin infection	0.1	0.0-0.9	.04
	(c)
Total points		Endocarditis Risk	Criteria	Points
-1		0.03	Tachycardia	1
0		0.05-0.07	Cardiac murmur	1
1		0.09-0.1	Skin infection	-1
2		0.2

In this retrospective collection of data in an independent, prospectively enrolled cohort of patients, we have internally validated our previously derived DI (PRE-IDU) for predicting IE among febrile IDUs. This is the first clinical tool to successfully undergo validation in a prospective cohort. Prior attempts failed to identify clinical criteria with significant predictive power for IE and noted poor correlation between IE and white blood cell count, hematocrit, age, sex, and clinical suspicion [4,5,7]. Only one other study attempted to derive and validate a Prediction tool for IE in febrile IDUs: Young et al [3] derived an “endocarditis score” using history of endocarditis, total white blood cell, percentage neutro- phils and bands on differential white blood cell count, infiltrate on chest x-ray, and Arterial oxygenation, but this score failed validation in a separate cohort. Interestingly, cardiac murmur was not a predictive factor for endocarditis in their study, although this may be due to reporting bias. Of note, the high sensitivity obtained for our tool is limited by its large CIs due to our small sample size. Further external validation of PRE-IDU with a larger sample size will be required prior to its clinical implementation.

Should our work be externally validated, we envision implemen- tation in 2 general ways. First, PRE-IDU may be used to rule out endocarditis in a manner similar to decision rules like the Ottawa Ankle [13] and NEXUS Cervical Spine rules [14]. The rule should be applied to patients in the ED with history of injection drug use presenting with fever. If patients lack all 3 criteria (no murmur, no tachycardia, but presenting with a skin infection), they have a low likelihood of endocarditis and may be considered “ruled out” for this disease. On the other hand, in febrile IDUs who have any of these 3 criteria (murmur, tachycardia, without identifiable skin infection), we suggest they undergo further evaluation for IE given the low specificity of our tool. An important consideration in all rules used in this manner is the determination of an “acceptable miss rate.” Although practitioners may agree that IE is a grave illness with high morbidity, and therefore, the miss rate should be low, the absolute acceptable miss rate will vary between individual practitioners and would require consensus opinion to establish a standard threshold. That being said, our lowest posttest probability of 3% is similar to miss rates described for other High-risk conditions such as 1.8% in pulmonary embolism [15] and 2.1% in myocardial infarction [16]. However, given the low sample size of patients used in our study, our prediction rule requires external validation in a larger, prospectively enrolled cohort prior to clinical use.

The second use of our work pertains to the logistic model, which cannot strictly rule out disease, but assists in risk stratification. We recommend physicians to use the pretest probability of IE from the literature (5%-20% [1-3]) then apply the model and likelihood ratios to a nomogram as in Fig. 2. For example, the presence of tachycardia and skin infection would yield a score of 0, which would correspond to a probability of 5% to 7%, or at baseline risk. These data could potentially be combined with further diagnostics to guide Disposition decisions. Of note, although evaluation of valves for vegetation is not part of the current Ultrasound imaging criteria for point-of-care cardiac ultra- sound performed by emergency physicians [17], and sensitivity of TTE for IE may be as low as 55% [18], diagnosis of IE by ED cardiac ultrasound has been recently described [19,20]. Eventually, with further improvements in technology and technique to evaluate valvular vegetations, future workup could incorporate the use of point-of-care echocardiography with our logistic model in the ED. In addition, advances in laboratory technique such as polymerase chain reaction assays using 16S rRNA primers could augment our detection of bacteremia in the ED [21]. Refinement of these tests into microassays may allow for their broad use as additional screens for

a

endocarditis or incorporation into algorithms, in which our logistic model could be used as an initial clinical risk stratification tool.

Limitations

Although we enrolled a separate cohort, this study was conducted at the same derivation study institutions. External validation at other sites with larger sample size is needed prior to clinical implementa- tion. Injection drug user status was self-declared, and details such as duration, frequency, and last use were not recorded. Owing to the retrospective collection of data despite prospective enrollment of patients, presence of cardiac murmur and skin infection were documented solely by the treating provider, limiting the reliability of these findings in our study. Also, even if our instrument could safely exclude endocarditis, patients may develop other undetected serious illness or require admission for other reasons. The efficacy of PRE-IDU on resource use will require further evaluation in a prospective study, but based on our model performance, 12% (29) patients were identified as being safe for discharge.

The most important limitation is our failure to meet our target sample size, which resulted in wide CIs for sensitivity and negative predictive value. The true sensitivity for PRE-IDU may be as low as 82%. Owing to the small sample size of our study, our prediction rule needs further external validation in a larger cohort prior to clinical implementation.

Conclusions

We internally validated our previously derived DI to predict IE in a prospective cohort with high sensitivity similar to our derivation study. Given the limitations of a rule-out strategy in a high-risk, low- frequency condition, we also developed and validated a logistic regression model, which allows risk stratification of febrile IDUs with posttest probabilities ranging from 3% to 20%. Although the small sample size used in our study and subsequent large CIs around the sensitivity limit the tool’s clinical application at this time, our results demonstrate the feasibility of risk stratifying febrile IDUs based on readily available clinical data in the ED.

Acknowledgment

b 1

0.75

Sensitivity

0.5

0.25

0

0 0.25 0.5 0.75 1

AUC = 0.8

1-Specificity

Fig. 2. Likelihood nomogram.

This study was supported by the Resident Research Grant provided by the Clinical and Translational Science Institute at the University of California San Francisco. The authors would also like to thank Jonathan Fortman for his assistance in data collection, and John Kornak, David Compeau, and Chengshi Jin for their assistance in statistical analysis and logistic regression modeling.

References

1. Brown PD, Levine DP. Infective endocarditis in the injection drug user. Infect Dis Clin North Am Sep 2002;16(3):645-65, viii-ix.
2. Delaney KA. Endocarditis in the emergency department. Ann Emerg Med 1991;20(4):405-14.
3. Young GP, Hedges JR, Dixon L, et al. Inability to validate a predictive score for infective endocarditis in intravenous drug users. J Emerg Med 1993;11(1):1-7.
4. Marantz PR, Linzer M, Feiner CJ, et al. Inability to predict diagnosis in febrile intravenous drug abusers. Ann Intern Med 1987;106(6):823-8.
5. Samet JH, Shevitz A, Fowle J, et al. hospitalization decision in febrile intravenous drug users. Am J Med 1990;89(1):53-7.
6. Palepu A, Cheung SS, Montessori V, et al. Factors other than the Duke criteria associated with infective endocarditis among injection drug users. Clin Invest Med 2002;25(4):118-25.
7. Weisse AB, Heller DR, Schimenti RJ, et al. The febrile parenteral drug user: a prospective study in 121 patients. Am J Med 1993;94(3):274-80.
8. Rodriguez R, Alter H, Romero KL, et al. A pilot study to develop a prediction instrument for endocarditis in injection drug users admitted with fever. Am J Emerg Med 2011;29:894-8.
9. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144(3):201-9.
10. Gilbert EH, Lowenstein SR, Koziol-McLain J, et al. Chart reviews in emergency medicine research: where are the methods? Ann Emerg Med 1996;27(3):305-8.
11. Li JS, Sexton DJ, Mick N, et al. Proposed modifications to the Duke criteria for the diagnosis of infective endocarditis. Clin Infect Dis 2000;30(4):633-8.
12. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143(1):29-36.
13. Stiell IG, Greenberg GH, McKnight RD, et al. Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation. JAMA 1993;269(9):1127-32.
14. Hoffman JR, Mower WR, Wolfson AB, et al. Validity of a set of clinical criteria to rule out injury to the cervical spine in patients with blunt trauma. National Emergency X-Radiography Utilization Study Group. N Engl J Med 2000;343(2): 94-9.
15. Kline JA, Mitchell AM, Kabrhel C, et al. Clinical criteria to prevent unnecessary diagnostic testing in emergency department patients with suspected pulmonary embolism. J Thromb Haemost 2004;2(8):1247-55.
16. Schull MJ, Vermeulen MJ, Stukel TA. The risk of missed diagnosis of acute myocardial infarction associated with emergency department volume. Ann Emerg Med 2006;48(6):647-55.
17. Physicians ACoE. Emergency Ultrasound Imaging Criteria Compendium; 2006.
18. Reynolds HR, Jagen MA, Tunick PA, Kronzon I. Sensitivity of transthoracic versus transesophageal echocardiography for the detection of native valve vegetations in the modern era. J Am Soc Echocardiogr 2003;16(1):60-7.
19. Cheng AB, Levine DA, Tsung JW, et al. Emergency physician diagnosis of pediatric infective endocarditis by point-of-care echocardiography. Am J Emerg Med. 2012;30(2):386 e381-383.
20. Walsh B, Bomann JS, Moore C. Diagnosing infective endocarditis by emergency department echocardiogram. Acad Emerg Med Jun 2009;16(6):572-3.
21. Rothman RE, Majmudar MD, Kelen GD, et al. Detection of bacteremia in emergency department patients at risk for infective endocarditis using universal 16S rRNA primers in a decontaminated polymerase chain reaction assay. J Infect Dis 2002;186(11):1677-81.