a b s t r a c t

Objectives: The Quick Sequential [Sepsis-related] Organ Failure Assessment (qSOFA) score has been shown to ac- curately predict mortality in septic patients and is part of recently proposed diagnostic criteria for sepsis. We sought to ascertain the sensitive of the score in diagnosing sepsis, as well as the diagnostic timeliness of the score when compared to traditional Systemic Inflammatory Response Syndrome criteria in a population of emergency department (ED) patients treated in the ED, admitted, and subsequently discharged with a diagno- sis of sepsis.

Methods: electronic health records of 200 patients who were treated for suspected sepsis in our ED and ultimately discharged from our hospital with a diagnosis of sepsis were randomly selected for review from a population of adult ED patients (N = 1880). Data extracted included the presence of SIRS criteria and the qSOFA score as well as time required to meet said criteria.

Results: In this cohort, 94.5% met SIRS criteria while in the ED whereas only 58.3% met qSOFA. The mean time from arrival to SIRS documentation was 47.1 min (95% CI: 36.5-57.8) compared to 84.0 min (95% CI: 62.2- 105.8) for qSOFA. The median ED “door” to positive SIRS criteria was 12 min and 29 min for qSOFA. Conclusions: Although qSOFA may be valuable in predicting sepsis-related mortality, it performed poorly as a screening tool for identifying sepsis in the ED. As the time to meet qSOFA criteria was significantly longer than for SIRS, relying on qSOFA alone may delay initiation of evidence-based interventions known to improve sep- sis-related outcomes.

(C) 2017

Introduction

Recently published consensus definitions for diagnosing sepsis sug- gest replacing routine use of Systemic Inflammatory Response Syn- drome (SIRS) criteria with the newly developed Sequential [Sepsis- related] Organ Failure Assessment (SOFA) and quick SOFA (qSOFA) model introduced in the Sepsis-3 manuscript series [1-3]. Describing improved specificity of qSOFA over SIRS for predicting mortality in the setting of infection, Singer and colleagues outlined use of qSOFA as a part of an algorithm to identify sepsis and septic shock in ED patients with suspected infection ^{2 (Figure pg.808)}.

While subsequent research in emergency department patients has validated the enhanced specificity of the qSOFA score in predicting in- hospital mortality [4], little attention has been paid to how the qSOFA score and SIRS criteria may vary temporally in the diagnosis of sepsis. With the 2016 Surviving Sepsis guidelines adoption of the Sepsis-3

* Corresponding author at: Maine Medical Center, Department of Emergency Medicine, 22 Bramhall Street, Portland, ME 04102.

E-mail address: [email protected] (T. Strout).

definition, ascertaining the timeliness of the qSOFA score in diagnosing sepsis is paramount to ensure continued early resuscitative therapies and antibiotic administration.

Goals of this investigation

In this pilot study, we sought to ascertain the sensitivity of the qSOFA score in diagnosing sepsis, as well as the diagnostic timeliness of the score when compared to traditional SIRS criteria in a population of emergency department (ED) patients treated, admitted and subse- quently discharged from our hospital with a diagnosis of sepsis.

Methods

Study design and patient population

This retrospective study was conducted at a single academic tertiary care hospital. The Institutional Review Board exempted the study.

We used de-identified data drawn from the electronic health record to conduct our analyses (EPIC Systems Corporation, Verona, WI).

http://dx.doi.org/10.1016/j.ajem.2017.07.001

0735-6757/(C) 2017

Using random number generation, a sample of 200 adult septic pa- tients was drawn from a base population of Medicare and Medicaid pa- tients (N = 1880) treated with antibiotics in the ED for suspected infection, admitted to the hospital, and subsequently discharged with a Center for Medicare Services diagnosis related Grouping (DRG) for sepsis (DRG 870,871,872) over a one-year period between September 2014 and September 2015. Patients who were transferred to the study ED from outside facilities were excluded. All patients in this study were treated in the emergency department with antibiotics for suspected infection, therefore patients either meeting 2 or more SIRS criteria or those with a qSOFA score >= 2 were defined as meeting diag- nostic criteria for sepsis while in the ED.

Data extraction protocol

Demographic data and disposition was electronically pulled from the EHR. Chart reviews were completed by a trained physician research- er (following predetermined guidelines defining abstraction criteria) to determine the presence and timing of the various components of both SIRS criteria and the qSOFA score. The qSOFA criteria were defined as al- tered mental status (AMS), respiratory rate (RR) N 22/min, and systolic blood pressure (SBP) b 100 mmHg. SIRS criteria were defined as heart rate (HR) N 90 bpm, white blood cell (WBC) N 12 k/mcL b 4 k/mcL, RR N 20/min, and temperature N 38.5 ?C b 36 ?C. Only vital signs obtained and documented in the ED were used, those recorded in the pre-hospi- tal setting were excluded. Laboratory values were considered to be present at the time of the Blood draw rather than at the time results were available. The timing of AMS was governed by the abstractor’s best estimate of when Mental status changed through review of the re- cord; the earliest time stamp allowed was the ED arrival time. Both phy- sician and registered nurse clinical documentation was reviewed.

Interrater reliability between data abstractors was assessed using a

random sample of 10% of the encounters that were reviewed indepen- dently by both reviewers. A low initial kappa value (k = 0.40) for the timing of mental status changes prompted a formal review of all charts by two trained abstractors to obtain consensus regarding the initial time of AMS for each encounter.

Statistical analyses

Data were analyzed using SPSS for Windows v. 24 (SPSS, Inc., Chica- go, IL) statistical software. Continuous data are presented as means and 95% confidence intervals or medians and interquartile ranges. Categori- cal data are summarized using counts and percentages. Test character- istics including sensitivity, specificity, and the positive and negative predictive values were calculated using standard methods while the discriminative ability of the SIRS and qSOFA criteria were assessed using the area under the Receiver operating characteristic curve . We accepted significance as p b 0.05. Cohen’s kappa statistic was used to estimate interrater reliability.

Results

Characteristics of the study subjects

In our study population of 200 ED patients, one patient accepted in transfer was excluded, leaving 199 subjects for analysis. The median age of subjects was 71 years (range 18-102); 45% (n = 90) were female. Hospital mortality for the cohort was 11.1% (n = 22), with a mean initial lactate level of 2.7 mmol/L (range 0.8-17 mmol/L). Additional demo- graphic characteristics are presented in Table 1.

Main results

188 subjects (94.5%) met SIRS criteria in the ED while only 116 (58.3%) met qSOFA criteria during the same period. For qSOFA

Table 1

Characteristics of the study subjects.

Characteristic
Sex (%)
Male	109 (54.5)
Female	91 (45.5)
Median Age in Years (range)	71 (18-102)
Race (%)
White	194 (97.0)
Asian	4 (2.0)
Black/African-American	1 (0.5)
Unknown	1 (0.5)
Ethnicity (%)
Non-Hispanic	200 (100)
Hispanic	0 (0)
triage acuity (%) ESI-1a	7 (3.5)
ESI-2	175 (87.5)
ESI-3	18 (9.0)
Hospital Disposition (%)
Discharge home	100 (50.0)
Expired	22 (11.0)
Hospice	15 (7.5)
Rehabilitation Facility	14 (7.0)
Skilled Nursing Facility	45 (22.5)
Transfer Outside Hospital	2 (1.0)
Left Against Medical Advice	2 (1.0)
Sepsis DRG (%) 870b	7 (3.5)
871c	144 (74.5)
872d	44 (22.0)
Serum Lactate (range)
Initial (mmol/L), n = 187	2.7 (0.8-17)

Abbreviations: ESI, Emergency Severity Index; DRG, diagnosis related group; CI, confidence interval.

^a Acuities range from 1 to 5, with lower scores indicating greater acuity.

^b Severe sepsis and septic shock requiring mechanical ventilation.

^c Severe sepsis and septic shock not requiring mechanical ventilation.

^d Meets SIRS criteria with suspected source infection.

assessment parameters, 78.4% of subjects (n = 156) met the respiratory rate requirements, 52.8% (n = 105) met the blood pressure require- ments, while 36.7% (n = 73) met the AMS requirements. The mean time from ED arrival to documented presence of SIRS criteria was

47.1 min (+- 73.884, 95% CI: 36.5-57.8 min) while the mean time from arrival to documentation of qSOFA criteria was 84.0 min (+- 118.565, 95% CI: 62.2-105.8 min), t = - 4.267, df = 187, p b 0.001.

The median ED “door” to positive SIRS criteria was 12 min and 29 min for qSOFA. See Fig. 1 for more details.

Twenty-two subjects (11.1%) experienced in-hospital mortality. Of the 22 hospital deaths, both SIRS and qSOFA criteria were met in the ED in 19 cases (86.4%). Sensitivity for correctly identifying those experiencing in-hospital mortality was 95.5% (95% CI: 75.1-99.7%) for SIRS and was 90.1% (95% CI: 69.4-98.4%) for qSOFA criteria. Specificity for SIRS was 5.6% (95% CI: 2.9-10.4%) and for qSOFA was 45.7% (95%

CI: 38.3-53.4%). With regards to sepsis related mortality, the area under the receiver operating characteristic curves (AUROCs) for the Predictive ability of the SIRS and qSOFA criteria did not differ signifi- cantly, z = -1.7416, p = 0.082. For the SIRS criteria, the AUROC was

0.51 (95% CI: 0.38-0.63) while for qSOFA it was 0.68 (0.58-0.78). Per-

formance characteristics are presented in Table 2.

The interrater reliability for all variables obtained through chart re- view were excellent (k = 0.81-1.0), except for the timing of AMS onset (k = 0.40).

Fig. 1. Box and whisker plots for time to SIRS, qSOFA, and individual qSOFA criteria. Abbreviations: SIRS, systemic inflammatory response system; qSOFA, quick Sequential [Sepsis-related] Organ Failure Assessment; RR, respiratory rate in breaths per minute; SBP, systolic blood pressure in mmHg; AMS, altered mental status. This figure presents box and whisker plots for door to documentation of SIRS criteria (n = 188), qSOFA criteria (n = 116), and individual elements of the qSOFA criteria (respiratory rate N 22 breaths/min [n = 156], systolic blood pressure b 100 mmHg [n = 105], altered mental status [n = 73]). Box plots represent the median, 75th, and 25th percentile times with the upper and lower adjacent values representing the most extreme data points within 1.5 times the interquartile range. The red dotted lines represent the mean value for each time.

Discussion

Rivers and colleagues established the foundation for the evidential basis for current day sepsis protocols [5-9]. With early detection and ag- gressive management as the accepted core to best practice for this pa- tient population, the use of SIRS plus a known or suspected source of infection as the operational definition of sepsis has led to well docu- mented reductions in both morbidity and mortality [5-9]. Early admin- istration of resuscitative therapies and antibiotics has been become standard practice and is tied to Hospital quality measures. When the qSOFA score diagnostic criteria were met our study population, the mean time required to meet said criteria was 37 min longer than that of the traditional SIRS criteria. Additionally, if the qSOFA score were to be employed as a sepsis screening tool, we found a concerning low sen- sitively compared to SIRS criteria (58.3% vs 94.5%, respectively).

Study limitations

This study has limitations that are inherent to retrospective chart re- views. The study population was limited to primarily white Medicare/ Medicaid adult patients, which may limit generalizability to other populations.

We found that retrospectively determining the onset of a patient’s altered mental status to be challenging, primarily due to the lack of stan- dardized, objective documentation practices regarding a patient’s men- tal status by clinicians. In our Geriatric population, assessing for acute change in mental status in the ED was also challenging due to underly- ing disease processes such dementia. This is a limitation also document- ed by the developers of the Sepsis-3 definitions and the qSOFA score [1], calling into question whether the AMS aspect of the qSOFA criteria should be considered as a reliable early marker of sepsis without further guidance from prospective studies.

It should also be noted that our institution utilizes an aggressive bundled approach to sepsis management Our protocol is geared to- wards identifying patients as early as possible, instituting interven- tions (including antibiotics) as soon as a patient meets SIRS based Sepsis criteria with a suspected source. This evidence-based protocol was designed to prevent the development of end organ damage, he- modynamic instability and the morbidity and mortality associated with severe sepsis and septic shock. As such, our institution’s ability to identify patients with potential sepsis and quickly implement in- terventions may be partially responsible for the differences noted in the study.

Table 2

Comparison of test characteristics and discriminatory performance for SIRS and qSOFA.

	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	AUROC (95% CI)
SIRS qSOFA	95.5 (75.1-99.8) 90.9 (69.4-98.4)	5.6 (2.9-10.4) 45.8 (38.3-53.4)	11.2 (7.2-16.8) 17.2 (11.1-25.6)	90.9 (57.1-99.5) 97.6 (90.8-99.6)	0.51 (0.38-0.63) 0.68 (0.58-0.78)

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the receiver operating characteristic curve; SIRS, systemic inflammatory response syn- drome criteria; qSOFA, quick Sequential [Sepsis-related] Organ Failure Assessment Score.

Conclusions

While the qSOFA score may serve as a valuable tool for the iden- tification of patients with sepsis at increased risk of mortality, it per- forms poorly as a screening tool for early identification of sepsis in the emergency department. The presence of any of the qSOFA com- ponents portents a severity of illness that often is not present in the early phases of this disease process, and may be useful for deter- mining patients who require critical care services. However, relying on qSOFA alone as screening tool in the ED may delay sepsis diagno- sis and initiation of evidence-based interventions known to improve sepsis-related outcomes. We hope this small study will provoke more investigation into the appropriateness of fully adopting the qSOFA score as a sepsis screening tool by emergency medicine clinicians.

Prior Presentations:

Presented as a plenary presentation at the 21st Annual New England Society for Academic Emergency Medicine Regional Meeting. March 29th, 2017. Worchester, MA.

Presented as an oral abstract at the 2017 Society for Academic Emer- gency Medicine Annual Meeting. May 18th, 2017. Orlando, FL.

Funding Sources:

This research did not receive any specific grant from funding agen- cies in the public, commercial, or not-for-profit sectors.

References

1. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis. JAMA 2016;315(8):762-74.
2. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus def- initions for sepsis and septic shock (sepsis-3). JAMA 2016;315(8):801-10.
3. Shankar-Hari M, Phillips GS, Seymour CW, et al. Developing a new definition and assessing new clinical criteria for septic shock. JAMA 2016;315(8):775-87.
4. Freund Y, Lemachatti N, Krastinova E, et al. Prognostic accuracy of sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA 2017;317(3):301-8.
5. Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001;345:1368-77.
6. The ARISE Investigators & ANZICS Clinical Trials Group. Goal-directed resuscitation for patients with early septic shock. N Engl J Med 2014;371(16):1496-506.
7. Mouncey PR, Osborn TM, Power S, et al. Trial of early, goal-directed resuscitation for septic shock. N Engl J Med 2015;372:1301-11.
8. Shapiro N, Howell MD, Talmor D, et al. Implementation and outcomes of the multiple urgent sepsis therapies (MUST) protocol. Crit Care Med 2006;34(4):1025-32.
9. Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med 2013;41(2):580-637.