American Journal of Emergency Medicine 38 (2020) 1995-1999

Contents lists available at ScienceDirect

American Journal of Emergency Medicine

journal homepage:

Comparison of an ED triage sepsis screening tool and qSOFA in identifying CMS SEP-1 patients

Ethan Sterk, DO ^a,b,?, Byung Hun Hyun ^b, Megan A. Rech, PharmD, MS ^a,c

^a Loyola University Chicago, Stritch School of Medicine, Department of Emergency Medicine, 2160 S 1st Ave, Maywood, IL 60153, United States of America

^b Loyola University Chicago, Stritch School of Medicine, 2160 S 1st Ave, Maywood, IL 60153, United States of America

^c Department of Pharmacy, Loyola University Medical Center, 2160 S 1st Ave, Maywood, IL 60153, United States of America

Introduction

Sepsis is of great interest and concern to the medical community given its high risk of mortality, incidence and cost. Each year, at least

1.7 million adults in the United States develop sepsis, resulting in nearly 270,000 deaths [1]. The emergency department (ED) serves as a pri- mary site of initial identification and treatment of most sepsis patients [2]. The cost of sepsis management in U.S. hospitals ranks highest among admissions for all Disease states. For example, in 2013, sepsis accounted for more than $24 billion in hospital expenses, far surpassing the next most costly conditions [3]. The timing of sepsis diagnosis is crit- ical in terms of outcomes given the acute and significant impact of the condition. Poor sepsis outcomes are observed when diagnosis and treat- ment are delayed.

The Surviving sepsis campaign guidelines of 2016, as well as those of 2012, emphasize routine screening of potentially infected patients who are likely to be septic, encouraging a performance improvement pro- gram that involves early recognition and management of sepsis [4]. The Systemic Inflammatory Response Syndrome criteria have historically been used to screen for and diagnose sepsis, however they have long been criticized for their lack of specificity [5-7]. In 2016, the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) proposed the Quick Sequential Organ Failure Assessment score as an alternative strategy to efficiently identify patients with suspected infection at risk for poor outcomes [8]. qSOFA is com- prised of a systolic blood pressure <= 100 mmHg, altered mental status, and a respiratory rate >= 22 breaths/min, with a “positive” score being 2 or more of these variables. Although it was not developed as an ED sepsis screening tool, it nevertheless has been proposed to be used as such. Several concerns arise when attempting to use qSOFA in this man- ner. First, given that qSOFA was developed in a heterogeneous group of patients (including many outside of the ED), it may not accurately re- flect the ED population [9]. Second, it was designed to predict mortality, not screen for the presence of sepsis. Finally, it has not been prospec- tively validated and sacrifices sensitivity for specificity [10]. Being that it is preferable to have a screening test with a high sensitivity in order

* Corresponding author at: Department of Emergency Medicine, Loyola University Chicago, Stritch School of Medicine, Loyola University Medical Center, 2160 South First Avenue, Bldg 110 – Emergency Medicine, Maywood, IL 60153, United States of America.

E-mail address: [email protected] (E. Sterk).

to rule out disease, this could lead to a higher False negative rate and incorrectly identify septic patients as non-septic.

The Centers for Medicare and Medicaid Services established SEP-1, the nation’s first quality measure on early sepsis intervention [11]. Starting October 1, 2015, CMS began requiring hospitals to keep track of how many septic patients they effectively treated. It is impor- tant to note that CMS defines sepsis using traditional SIRS criteria, not the Sepsis-3 definition (Table 1). CMS quality measures are federal reg- ulations; as such, hospitals are required to report their compliance with these measures. In order to ensure that sepsis is being properly docu- mented and treated in compliance with the CMS SEP-1 core measure, it is important to use the same definitions as the CMS SEP-1 and attempt to have a screening tool that is more accurate for this measure. To date, there have not been any studies looking at the performance characteris- tics of ED sepsis Screening tools in identifying septic patients according to the CMS SEP-1 definitions. The Loyola University Medical Center (LUMC) ED developed a sepsis screening tool adapted from the Greater New York Hospital Association STOP Sepsis Collaborative Severe Sepsis Triage Screening Tool [12]. The purpose of this study was to determine the sensitivity, specificity and accuracy of qSOFA against the current sepsis screening tool used in the LUMC ED in identifying CMS SEP-1 core measure patients.

Methods

This was a retrospective cohort of adult patients who presented to the LUMC ED in May 2017. LUMC is a suburban quaternary-care aca- demic center with approximately 50,000 ED visits per year. All patients visiting within a 30-day period were included with the exception of those presenting as a trauma, ST elevation myocardial infarct, acute stroke, Burn injury, age < 18 years, and pregnancy. Those with incom- plete or missing data were also excluded.

The LUMC screening tool was developed based off the Greater New York Hospital Association STOP Sepsis Collaborative Severe Sepsis Tri- age Screening Tool [12]. This group focused on the early identification and treatment of sepsis called the STOP Sepsis Collaborative [13]. At LUMC, this tool was slightly Modified SInce to include a mean arterial pressure (MAP) < 65 mmHg. A paper version was implemented in 2013 and then built into the Electronic Medical Record (Epic Systems, Verona, WI) in 2015. The tool was comprised of ten variables: three questions that were answered by the triaging nurse and seven vital

https://doi.org/10.1016/j.ajem.2020.06.030

0735-6757/(C) 2020

Table 1

Study definitions.

CMS SEP-1 definitions qSOFA LUMC ED triage sepsis screening tool

SIRS criteria:

Temperature > 38.3 ?C or < 36 ?C Heart Rate > 90 beats per minute

Respiratory Rate > 20 breaths per minute

WBC > 12 or < 4 (thousand per mm³) or > 10% bands

Sepsis:

At least 2 SIRS criteria with suspected or confirmed infection

Severe Sepsis:

Sepsis plus one or more variable of organ dysfunction*

Septic Shock**:

Severe sepsis plus SBP < 90 mmHg, or a MAP <65 mmHg, or a reduction in SBP by more than 40 mmHg from a previously recorded normal SPB

*Organ dysfunction is defined as follows:

• Lactate >2 mmol/L
• INR > 1.5 or aPTT >60 s
• Platelet count <100, x10 [3]/uL
• Bilirubin >2 mg/dL
• Creatinine >2 mg/dL
• Urine output <0.5 mL/kg/h x 2 h

Altered mental status Respiratory rate >= 22 Systolic blood pressure <= 100 mmHg

• A positive score is >=2 of these components
  

  Suspected infection?

• Rigors present?
• Altered mental status?
• Temp >38.3 ?C
• Temp <36 ?C
• Heart rate > 90
• Respiratory rate > 20
• Systolic blood pressure < 90
• Mean arterial pressure < 65
• Pulse oximetry <90%
  • A positive sepsis screen = “yes” to suspected infection PLUS at least 2 other criteria
  • Acute respiratory failure: need for new invasive or noninvasive ventilation or increased need from intermittent to continuous mechanical ventilation
  • SBP < 90 mmHg or MAP <65 mmHg or decreased in SBP more than 40 mmHg from previously recorded patient normal

  **Valid only after the patient has received a 30 mL/kg crystalloid fluid bolus or if the initial lactate level greater than or equal to 4 mmol/L.

  SIRS = Systemic Inflammatory Response Syndrome, SBP = systolic blood pressure, MAP = mean arterial pressure, WBC = white blood cell count, INR = International Normalized Ratio, aPTT = activated partial thromboplastin time, mmol/L = millimole per liter.

  signs that were automatically pulled into the tool each time a new set is documented (Table 1). A positive sepsis screen occurred when the nurse answers “yes” to “suspected infection” plus at least two other positive variables. This populated a Best Practice Alert for the nurse to notify an ED attending physician to review the case for further or- ders. If the patient’s clinical scenario was concerning for sepsis, then the sepsis order set was utilized to initiate an infectious workup. The sepsis order set contained all laboratory parameters, intravenous

  fluids, antimicrobials based on suspected source of infection, and vasopressors.

  The primary objective of this study was to compare the test charac- teristics of the two screening tools against the reference standard and provide comparisons in discrimination between the screening tools. This was accomplished by determining the sensitivity, specificity and overall accuracy of the two tools against the reference standard established through manual chart review of the medical record and

  796 Patients Included

  Non-septic group (n = 754)

  6 patients excluded

  3 – Screening tool not completed 1 – STEMI

  1 – Records incomplete

  1 – Cardiac arrest within minutes of presentation

  802 Adult ED Patients May 2017

  Septic Group (n = 42)

  Fig. 1. Study flowchart.

  determining the presence of sepsis according to the CMS SEP-1 defini- tions (Table 1). One reviewer (BH) performed this blinded review ini- tially, and then two other reviewers (ES, MR) evaluated patient cases independently and in a blinded manner. Only the first encounter for sepsis was recorded.

  Data was collected on demographic information, including age, sex, race and the presence of chronic comorbid conditions such as hypertension, diabetes, coronary artery disease. Triage vital signs consisted of temperature, heart rate, respiratory rate, systolic

  Table 2

  Baseline characteristics.

  Non-septic group

  (n = 754)

  Age (years), median (IQR) 55 (37-70)	57.5 (43-74)	0.16
  Male, n (%) 314 (41.6)	19 (45.2)	0.65
  Race, n (%)		0.78
  African-American 262 (34.8)	13 (31)
  Caucasian 359 (47.6)	21 (50)

  Septic group (n = 42)

  P value

  blood pressure, mean arterial blood pressure and oxygen satura-	Hispanic	103 (13.7)	6 (14.3)
  tion. Other triage scoring systems variables incorporated the pres-	Asian	15 (2)	2 (4.8)

  ence of altered mental status, rigors and suspected infection. Laboratory parameters included baseline white blood cell count, bands, platelets, creatinine, total bilirubin, INR (International Nor- malized Ratio), aPTT (activated partial thromboplastin time) and lactate level. We used the initial labs that were resulted in the ED for this. We also collected the LUMC triage screening tool scores and used the above data to calculate the patients’ qSOFA score upon triage.

  Other	15 (2)	0 (0)
  Comorbidities, n (%) COPD	28 (3.7)	4 (9.5)	0.06
  ESRD	67 (8.9)	7 (16.7)	0.10
  CAD	193 (25.6)	15 (35.7)	0.15
  Diabetes	180 (23.6)	10 (23.8)	0.99
  ESLD	12 (1.6)	1 (2.4)	0.69
  Stroke	55 (7.3)	2 (4.8)	0.54
  Immunocompromised	29 (3.9)	9 (21.4)	<0.001
  Hypertension	379 (50.3)	27 (64.3)	0.08
  Atrial fibrillation	41 (5.4)	0 (0)	0.12
  Vital Signs, median (IQR) MAP (mmHg)	95 (84-106)	88.5 (71-101)	0.02
  SBP (mmHg)	135 (120-152)	128 (102-151)	0.07
  Temperature (?C)	36.8 (36.6-37)	37.7	<0.001
  		(36.8-38.9)
  HR (beats/min)	82 (72-95)	108 (98-114)	<0.001
  RR (breaths/min)	18 (17-20)	22 (20-26)	<0.001
  Oxygen saturation (%)	98 (97-100)	96.5 (94-99)	<0.001
  Other Scoring systems variables,
  n (%)
  Altered mental status	10 (1.3)	7 (16.7)	<0.001
  Rigors	9 (1.2)	7 (16.7)	<0.001
  Suspected infection	78 (10.3)	42 (100)	<0.001
  Laboratory Parameters, median (IQR) WBC, thousand per mm3	8.2 (6.3-10.6)	13.1	<0.001
  Bands, %	3 (1-6)	(7.9-17.2) 8 (3-21)	0.09
  Platelets, x 103/uL	228 (179-284)	227 (162-315)	0.60
  Creatinine, mg/dL	0.97	0.97 (0.8-1.9)	0.42
  	(0.77-1.22)
  Bilirubin, mg/dL	0.7 (0.6-1)	0.9 (0.7-1.2)	0.02
  INR	1.1 (1-1.5)	1.25 (1.1-1.6)	0.17
  aPTT, seconds	31.1 (28.1-35)	32.5 (27-37.3)	0.83
  Lactate, mmol/L	1.3 (0.9-1.7)	1.8 (1.2-2.6)	0.01
  qSOFA score, n (%)			<0.001
  0	577 (76.5)	10 (23.8)

  Statistical analysis

  Descriptive statistics were used to characterize baseline demo- graphics between septic and non-septic patients. Continuous variables were expressed as medians and interquartile ranges (IRQ) and analyzed using the Mann-Whitney U test. Categorical data were expressed as pro- portions and analyzed using the chi-squared of Fischer’s exact test, as appropriate.

  The LUMC screening tool performance was evaluated by measur- ing its sensitivity, specificity, and diagnostic accuracy against qSOFA. Discrimination of each method was evaluated in logistic re- gression using the area under the receiver operating characteristic (ROC) curve. All the models showed adequate performance with the likelihood ratio having a P < .0001. The nonparametric approach of DeLong et al. was used to compare the ROC curves from each al- gorithm against the appropriate reference standard with the best area under the ROC [14]. Analysis was performed with STATA ver- sion 15 (College Station, TX).

  Results

  Of the 802 patients reviewed, 6 patients were excluded, leaving a	1	164 (21.7)	23 (54.8)
  total of 796 patients that were included in this study (Fig. 1). Of these	2	12 (1.6)	7 (16.7)

  patients, 42 (5.3%) met the CMS SEP-1 definition of sepsis. Baseline characteristics were similar between the groups except that the sepsis group had significant higher Immunocompromised patients (3.9% non-septic group vs. 21.4% septic group, p < .01, Table 2). The septic group had a higher incidence of chronic obstructive pulmonary disease, end stage renal disease, hypertension and coronary artery disease, while the non-septic group had a higher incidence of atrial fibrillation, al- though not significant. The following differences in vital signs were noted: temperature (36.8 ?C non-septic group vs. 37.7 septic group, p < .01), heart rate (82 beats per minute non-septic group vs. 108 septic group, p < .01), respiratory rate (18 breaths per minute non-septic group vs. 22 septic group, p < .01) and WBC (8.2 thousand/mm³ non- septic group vs. 13.1 septic group, p < .01). The sepsis group also had a higher percentage of altered mental status (1.3% non-septic group vs. 16.7% septic group, p < .01).

  The LUMC screening tool correctly identified 38 of the 42 cases of sepsis (90.5%), while qSOFA correctly identified 9 (21.4%) cases. The sensitivity and specificity of the LUMC screening tool was found to be 90.5% and 98.9%, respectively, and the sensitivity and specificity of the qSOFA was found to be 21.4% and 98.1%, re- spectively. LUMC screening tool had a positive predictive value of 82.6%, and negative predictive value of 99.5%. qSOFA had a posi- tive predictive value of 39.1%, and negative predictive value of

  3 1 (0.1) 2 (4.8)

  LUMC screening tool score, median 1 (0-1) 3.5 (3-4) <0.001 (IQR)

  Source of infection, n (%) <0.001

  N/A 666 (88.3) 0 (0)

  Respiratory 23 (3) 18 (42.9)

  GI 8 (1.1) 2 (4.8)

  GU 21 (2.8) 8 (19)

  Skin/Soft tissue 22 (2.9) 8 (19)

  Musculoskeletal 4 (0.5) 1 (2.4)

  Viral 2 (0.3) 0 (0)

  Other 7 (0.9) 1 (2.4)

  FUO 1 (0.1) 4 (9.5)

  Compares the baseline characteristics of patients meeting CMS SEP-1 criteria for sepsis vs not. IQR = interquartile range, COPD = chronic obstructive pulmonary disease, ESRD = end stage renal disease, CAD = coronary artery disease, ESLD = end stage liver disease, MAP = mean arterial pressure, SBP = systolic blood pressure, HR = heart rate, RR = re- spiratory rate, WBC = white blood cell count, INR = International Normalized Ratio, aPTT = activated partial thromboplastin time, mmol/L = millimole per liter, GI = gastro- intestinal, GU = genitourinary, FUO = fever of unknown origin.

  95.7%. The LUMC screening tool had an accuracy of 98.5% and qSOFA’s was 94.1% (Table 3). The ROCs for identifying sepsis were LUMC screening tool 95% (95% confidence interval [CI] 90.2-99.2%) vs qSOFA 60% (95% CI 53.5-66.1%, Fig. 2).

  Table 3

  Performance characteristics of qSOFA and the LUMC screening tool.

  qSOFA LUMC

  Patients with sepsis 42 42

  True Positive 9 38

  False Negative 33 4

  Non-septic Patients 754 754

  True Negative 740 746

  False Positive 14 8

  Sensitivity 0.214 0.905

  Specificity	0.981	0.989	[17]. So, in order to ensure that sepsis is being properly treated and doc-
  PPV	0.391	0.826	umented in compliance with the CMS SEP-1 core measure, it is impor-

  and lack of consensus regarding them. However, as CMS quality mea- sures are federal regulations, hospitals are required to report their com- pliance with these measures and thus use their definitions for this. There are several potential consequences for poor scores. If a Joint Com- mission survey of a hospital exposes poor compliance to CMS measures, that hospital risks losing its accreditation. Furthermore, after a CMS measure has been in use for some time, it may also be used for value- based purchasing in which Medicare and/or Medicaid reimbursement for sepsis cases might be directly tied to rates of measure adherence

  NPV	0.957	0.995
  Accuracy	0.941	0.985

  

  Fig. 2. ROC curves for qSOFA and the LUMC screening tool.

  Discussion

  This study demonstrated that the LUMC screening tool outperformed qSOFA in the overall accuracy of identifying septic pa- tients compared to the reference standard of diagnosis of sepsis, accord- ing to the CMS SEP-1 metric. This difference in accuracy was largely driven by improved sensitivity; both tools had similar rates of specific- ity. The LUMC screening tool is easy to use, just requiring the triaging nurse to answer three questions. The vital signs component is built into the electronic medical record, making it a tool that could be used in a wide variety of settings and EDs across the country. Given its im- proved test characteristics compared to qSOFA, institutions currently using this method to identify septic patients should consider adapting a more robust tool.

  Being that qSOFA has been proposed to be used as a screening tool to detect sepsis, the low sensitivity demonstrated in this study raises ques- tions regarding patient safety associated with adopting it as a primary sepsis screening tool [9]. A recent study comparing SIRS, qSOFA, and Na- tional Early Warning Score (NEWS) for the early identification of sepsis in the ED demonstrated that qSOFA had the lowest sensitivity and is a poor tool for ED sepsis screening [15]. Last year, the American Academy of Emergency Medicine (AAEM) published a Clinical Practice Statement and concluded that based on multiple retrospective and a few prospec- tive studies, it appears that qSOFA performs poorly in comparison to SIRS as a diagnostic tool for ED patients who may have sepsis or septic shock [16].

  Since CMS began requiring hospitals to keep track of how many sep- tic patients they effectively treated, this has proved to be somewhat problematic because of the newer, different proposed sepsis definitions

  tant to use the same definitions as the CMS SEP-1 and thus attempt to have a screening tool that is more accurate for this measure.

  This study has several limitations. First, though all patients were col- lected over the course of an entire month, there were a relatively small number of septic patients. This was a single center investigation in an academic U.S. hospital, so the results may not be generalizable to other settings. Furthermore, it was a retrospective design, which may increase the risk for misclassification biases. Additionally, the one- month time period chosen cannot account for seasonal variation. Fi- nally, the LUMC ED screening tool can only flag positive if the triage nurse answers “yes” to the question “suspected infection?”, which can be subjective and variable among the staff. However, this data is moni- tored monthly and feedback provided when components are missed, through our Sepsis Quality Improvement Committee.

  Conclusion

  Our results show that the LUMC screening tool performed signifi- cantly better than qSOFA when challenged with real patient data in identifying CMS SEP-1 patients. Given that early detection is paramount in the management of sepsis and in order to ensure that sepsis is being properly treated and documented in compliance with the CMS SEP-1 core measure, it is important to use its same definitions and attempt to have a screening tool that is more accurate for this measure. We hope this small study will lead to future prospective evaluation with larger sets of patients to further understand their performance charac- teristics in this group of patients. The goal is to find a screening tool that is readily accessible at triage, highly sensitive, and has a user- friendly interface with the electronic medical record. Accordingly, this may increase the emergency provider’s clinical acumen for diagnosing sepsis and ultimately improve outcomes for this common, but deadly condition.

  Declaration of Competing Interest

  No financial support was used for this study. The authors have no conflicts of interest to disclose.

  References

  1. Centers for Disease Control and Prevention. Data and Reports. [online] Available at: https://www.cdc.gov/sepsis/datareports/index.html; 2020. [Accessed 16 Feb. 2020].
  2. Venkatesh Arjun K, et al. Preliminary performance on the new CMS sepsis-1 national quality measure: early insights from the emergency quality network (E-QUAL). Ann Emerg Med. 2020;71(1). https://doi.org/10.1016/j.annemergmed.2017.06.032 10 – 15.e1.
  3. Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and costs of sepsis in the United States-an analysis based on timing of diagnosis and severity level. Crit Care Med. 2018;46(12):1889-97. https://doi.org/10.1097/CCM.0000000000003342.
  4. Kim HI, Park S. Sepsis: early recognition and optimized treatment. Tuberc Respir Dis.

     2019;82(1):6-14. https://doi.org/10.4046/trd.2018.0041.

     Bone Roger C, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. CHEST. 2020;101(6):1644-55. https://doi.org/10. 1378/chest.101.6.1644.

  5. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference. Intensive Care Med. 2003 Apr;29(4):530-8. https://doi.org/10.1097/01.CCM.0000050454. 01978.3B.
  6. Anand V, et al. Epidemiology of quick sequential organ failure assessment criteria in undifferentiated patients and association with suspected infection and sepsis. CHEST. 2020;156(2):289-97. https://doi.org/10.1016/j.chest.2019.03.032.
  7. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus def- initions for sepsis and septic shock (sepsis-3). JAMA. 2016;315(8):801-10. https:// doi.org/10.1001/jama.2016.0287.
  8. Rodriguez RM, Greenwood JC, Nuckton TJ, et al. Comparison of qSOFA with current emergency department tools for screening of patients with sepsis for critical illness. Emerg Med J. 2018;35:350-6. https://doi.org/10.1136/emermed-2017-207383.
  9. Farkas J. Top ten problems with the new sepsis definition. [online] Emcrit.org. Avail- able from: https://emcrit.org/pulmcrit/problems-sepsis-3-definition/; 2016. [Accessed 16 Feb. 2020].
  10. Centers for Disease Control and Prevention. Severe Sepsis And Septic Shock: Man- agement Bundle (Composite Measure). [online] Available at: https://www.cdc.gov/ sepsis/datareports/index.html>; 2015. [Accessed 11 March 2020].
  11. Weingart S. Lessons From the STOP Sepsis Collaborative. [online] Emcrit.org. Avail- able from: https://emcrit.org/emcrit/lessons-sepsis-collaborative/; 2012. [Accessed 16 Feb. 2020].
  12. GNYHA. Sepsis Archives – GNYHA. [online] Available from: https://www.gnyha.org/ topic/sepsis/; 2013. [Accessed 16 Feb. 2020].
  13. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Bio- metrics. 1988;44:837-45.
  14. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department. Am J Emerg Med. 2018;37(Aug):1490-7. https://doi.org/10.1016/j.ajem.2018.10.058.
  15. Waligora G, Mills L, Church A. American Academy of emergency medicine clinical practice statement – the use of qSOFA in the emergency department. [online] Aaem.org. Available from: https://www.aaem.org/UserFiles/UseofqSOFAintheED. pdf; 2019.
  16. Faust J, Weingart S. The past, present, and future of the centers for medicare and medicaid services quality measure SEP-1: the Early Management Bundle for severe sepsis/septic shock. Emerg Med Clin North Am. 2017;35(1):219-31. https://doi. org/10.1016/j.emc.2016.09.006.