a b s t r a c t

Background: Limited data are available on the discriminatory capacity of quick sequential [sepsis-related] organ failure assessment (qSOFA) versus IDSA/ATS minor criteria for predicting mortality in patients with community- acquired pneumonia (CAP).

Methods: An observational prospective cohort study of 2116 patients with CAP was performed. construct validity was determined using Cronbach ?. Discrimination was assessed using the area under the Receiver operating characteristic curve and Net reclassification improvement .

Results: Overall in-hospital mortality was 6.43%. Mortality was 25.96% for patients with a qSOFA score of 2 or higher versus 3.05% for those with a qSOFA score less than 2 (odds ratio for mortality 6.57, P < 0.0001), and 13.85% for patients with at least 3 minor criteria versus 2.03% for those with 2 or fewer minor criteria (odds ratio for mortality 2.27, P < 0.0001). qSOFA had a higher correlation with mortality than minor criteria, as well as higher internal consistency (Cronbach alpha 0.43 versus 0.14) and diagnostic values of individual elements (larger AUROCs and higher Youden’s indices). qSOFA >=2 was less sensitive but more specific for predicting mor- tality than >=3 minor criteria (qSOFA sensitivity 59.6%, specificity 88.3% and positive likelihood ratio 5.11 versus >=3 minor criteria sensitivity 80.1%, specificity 65.8% and positive likelihood ratio 2.34). The Predictive validity of qSOFA was good for mortality (AUROC = 0.868), was statistically greater than minor criteria, was equal to pneu- monia severity index, and was inferior compared with CURB-65 (AUROC, 0.824, 0.902, 0.919; NRI, 0.088, -0.068,

-0.103; respectively). Conclusions: The qSOFA predicted mortality in CAP better than IDSA/ATS minor criteria and worse than CURB-65 with robust elements and higher convergence. qSOFA as a bedside prompt might be positioned as a proxy for minor criteria and increase the recognition and thus merit more appropriate management of CAP patients likely to fare poorly, which might have implications for more accurate clinical triage decisions.

(C) 2021

Abbreviations: qSOFA, quick sequential [sepsis-related] organ failure assessment; CAP, Community-acquired pneumonia; IDSA/ATS, The Infectious Disease Society of America/ American Thoracic Society; ICU, Intensive care unit; ROC, The receiver operating charac- teristic; AUROC, The area under the receiver operating characteristic curve; SOFA, Sequential [sepsis-related] organ failure assessment; PSI, pneumonia severity index; CURB-65, Confusion, urea >7 mmol.L – 1, respiratory rate >= 30 breaths.min – 1, low blood pressure, and age >= 65 years; SD, Standard deviation; OR, Odds ratio; RR, Relative risk; CI, Confidence interval; NRI, Net reclassification improvements; PLRs, Positive likeli- hood ratios; NLRs, Negative likelihood ratios; PPVs, positive predictive values; NPVs, Negative predictive values; SBP, Systolic blood pressure; LCL, Lower confidence limit.

* Corresponding author at: Department of Pulmonary and Critical Care Medicine,

Shenzhen Hospital, Peking University, Shenzhen, 518036, Guangdong, China.

E-mail address: [email protected] (Q. Guo).

1. Introduction

An international task force of experts recommend a new Clinical score termed quick sequential [sepsis-related] organ failure assessment (qSOFA) in the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3), which incorporates respiratory rate of 22/min or greater, altered mentation, and systolic blood pressure of 100 mmHg or less (range, 0-3; 1 point for each of the criteria), and pro- vides simple bedside criteria to identify adult patients with suspected infection who are likely to have poor outcomes [1,2]. Community- acquired pneumonia (CAP) causes great mortality and morbidity and high costs worldwide, and is often complicated by sepsis [3,4]. The

https://doi.org/10.1016/j.ajem.2021.11.029

0735-6757/(C) 2021

Infectious Disease Society of America/American Thoracic Society (IDSA/ ATS) designed minor criteria for severe CAP with the aim to guide inten- sive care unit (ICU) admission, not to predict mortality [5], but we [6], Liapikou et al. [7], and Sibila et al. [8] have reported that some of these criteria might be predictors of mortality, while others not. IDSA/ATS minor criteria is used extensively since the guidelines were published, especially in China. However, owing to emerging insights into its cum- bersome application and less accurate Mortality prediction, a proxy for IDSA/ATS minor criteria is warranted to improve Predictive ability and facilitate clinical care [9].

It is important to assess the outcome prediction ability of qSOFA in patients with pneumonia and is also necessary to compare qSOFA to other pneumonia-specific scores. Ranzini et al. [10] corroborated that qSOFA is clinically useful in emergency department patients with CAP, although Ahnert et al. [11] found that IDSA/ATS minor criteria had a su- perior area under the receiver operating characteristic (ROC) curve (AUROC) for primary endpoint (mortality and transfer to the ICU) com- pared to qSOFA. qSOFA >=2 is strongly associated with mortality in pa- tients with pneumonia, but the poor sensitivity may have limitations in the early identification of mortality [12]. For Sepsis-3 criteria to be globally endorsed as a predictor of mortality in CAP, more external val- idations in different settings are essential. As qSOFA is easier to practi- cally implement at bedside, it seems worthwhile to further explore the discriminatory capacity of qSOFA versus IDSA/ATS minor criteria for predicting mortality in patients with CAP, which could have impor- tant triage implications.

Therefore, an observational prospective cohort study of patients with CAP was conducted to determine the predictive differences be- tween the two scoring systems.

1. Methods
   1. Design and setting

This is an observational prospective cohort study of patients with CAP conducted in the Departments of Pulmonary and Critical Care Med- icine in two Chinese tertiary hospitals of two universities between Jan- uary 1, 2016 and December 31, 2019. The database used for the article published partly overlapped the current database [9].

• 1. Criteria for enrollment

CAP was defined as an Acute infection of the pulmonary parenchyma associated with an acute infiltrate on the chest radiograph with two or more symptoms including fever (>38 ?C), hypothermia (<36 ?C), rigors, sweats, new cough or change in color of respiratory secretions, chest discomfort or dyspnoea [13]. Patients who were younger than 18 years, who had been hospitalized during the 28 days preceding the study, who had severe immunosuppression, active tuberculosis, or end-stage diseases, who had a written “do not resuscitate” order, who had COVID-19, or whose baseline status was unconscious before suffer- ing from pneumonia, were excluded.

• 1. Clinical management

The study was conducted in accordance with the principles de- scribed in human experimentation guidelines of the United States De- partment of Health and Human Services. We based our report on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines. Patients with CAP were attended by respiratory physicians using the IDSA/ATS guidelines [5] and the Surviving sepsis campaign guidelines [14,15]. qSOFA scores of 2 or higher justified a transfer to respiratory ICU. The empirical antibiotic regimens were ad- herence to the guidelines, and then adjusted based on subsequently cul- tured pathogens. All patients were discharged home when they reached clinical stability and became afebrile.

• 1. Approval of study design

The study was approved by our Institutional Review Boards (Review Board of Sun Yat-sen University and Review Board of Peking University, No. 20152958 and No. 20153043, respectively). All procedures per- formed in the study involving human participants were in accordance with the 1964 Helsinki Declaration and its later amendments. Ethical approval from the regulation committee (Ethical Committee of Shenzhen) was granted for the study protocol. Written informed con- sent (except that from the patients with confusion who were asked af- terwards and before enrollment a first degree relative gave assumed consent) was obtained from the patient prior to enrollment. They were informed of the content of the study on admission and then signed the documents if they agreed.

• 1. Sample size calculation

Unit-level design prevalence, cluster-level design prevalence, test sensitivity, target cluster sensitivity, and target system sensitivity were 12%, 1%, 0.9, 0.5, and 0.95, respectively. The total numbers of clus- ters to be sampled were 598, and the maximum number of samples was 4186.

• 1. Outcomes

In accordance with Seymour et al.[2], the primary outcome was in- hospital mortality. Secondary outcomes were the performances of scores to predict mortality in CAP, including qSOFA, SOFA (sequential [sepsis-related] organ failure assessment), minor criteria, PSI (pneumo- nia severity index), and CURB-65 (confusion, urea >7 mmol.L^-1, respi- ratory rate >= 30 breaths.min^-1, low blood pressure, and age >= 65 years).

• 1. Data collection

A total of 2147 patients were enrolled consecutively and 31 cases were excluded due to exclusion criteria. All the patients had chest radio- graphs and/or computer tomography scans. The frontal and lateral chest radiographic findings and computer tomography scan images were classified independently by two senior radiologists (LH Liang and QZ Zhao). Clinical and diagnostic data, and Radiological features were col- lected. qSOFA, SOFA, PSI and CURB-65 scores on admission were calcu- lated. laboratory variables were measured by the hospital clinical laboratories. The statistician was blinded to the study.

• 1. Statistical analysis

All statistical analyses were performed with Statistical Package for the Social Science for Windows version 16.0 (SPSS, Chicago, IL, USA) and MedCalc version 19.6.1 (Mariakerke, Belgium). Categorical vari- ables and continuous variables with normal distribution were reported as the percentages and the mean +- standard deviation (SD), respec- tively. Chi-Square test, Spearman rank correlation, and univariate logis- tic regression were employed. Odds ratio (OR) and Relative risk for mortality were calculated. Construct validity was determined by exam- ining the agreement between different measures analogous to the multitrait-multimethod matrix approach of Campbell and Fiske, using the Cronbach ? to measure agreement or commonality [16,17]. We con- structed ROC curves and calculated the corresponding AUROCs with the 95% confidence interval (CI) to assess the performances of scores to pre- dict mortality. We considered AUROCs to be poor at 0.6 to 0.7, adequate at 0.7 to 0.8, good at 0.8 to 0.9, and excellent at 0.9 or higher [18]. Predic- tion performance of scores was also compared by calculating net reclas- sification improvement (NRI) [19-21]. The sensitivities, specificities, positive likelihood ratios (PLRs), negative likelihood ratios (NLRs), pos- itive predictive values (PPVs), Negative predictive values , and Youden’s indices were also calculated to assess robustness of the

variables. All testing was two-sided. P-values less than 0.05 were con- sidered statistically significant.

1. Results
   1. Patient characteristics

Baseline characteristics were summarized in Table 1. Overall in- hospital mortality was 6.43%: 25.96% for patients with a qSOFA score of 2 or higher versus 3.05% for those with a qSOFA score less than 2, and 13.85% for patients with at least 3 minor criteria versus 2.03% for those with 2 or fewer minor criteria (P < 0.0001 for both).

• 1. Associations with in-hospital mortality

The mortality rates were positively associated with qSOFA scores and the numbers of minor criteria present (x², p. 792.123, < 0.0001; 277.998, < 0.0001; respectively. Table 2). qSOFA demonstrated signifi- cant higher increased OR for mortality than minor criteria (6.57 versus 2.27, P < 0.0001 for both). The correlation of qSOFA scores with the mortality rates was higher than that of the numbers of minor criteria present (Rank correlation coefficient value, p. 0.349, < 0.0001; 0.286,

< 0.0001; respectively). qSOFA >=2 and minor criteria >=3 confirmed a

similar pattern with the mortality rates (x², p. 232.197, < 0.0001; 114.798, < 0.0001. OR, p. 11.15, <0.0001; 7.75, <0.0001. Rank correla-

tion coefficient value, p. 0.331, <0.0001; 0.233, <0.0001. Respectively. Table 1).

Table 1

Baseline characteristics of study cohort (Mean +- SD, n = 2116)

Characteristic Value

Age (years) 51.2 +- 23.4

Sex, No. (%)

Men 943 (45)

Women 1173 (55)

Comorbidities, No. (%)

Hypertension 641 (30)

Coronary heart disease 205 (10)

Heart failure 80 (4)

NYHA class IV 31 (2)

COPD 145 (7)

GOLD 3 and 4 82 (4)

Diabetes mellitus 162 (8)

Chronic renal insufficiency 87 (4)

Dialysis 49 (2)

Liver disease 106 (5)

Nervous system disease 78 (4)

Tumour 150 (7)

Alcohol abuse, No. (%) 74 (4)

Smoking, No. (%) 411 (19)

qSOFA >=2, No. (%)^{a, b}

No/Died 1804 (85)/55 (3.05)

Yes/Died 312 (15)/81 (25.96)

Minor criteria >=3, No. (%)^{c, b}

No/Died 1329 (63)/27 (2.03)

Yes/Died 787 (37)/109 (13.85)

Outcomes

Ventilated patients 147 (7)

Patients received catecholamines 203 (10)

Hospital Length of stay (days) 12.0 +- 8.7

In-hospital mortality 136 (6.43)

NYHA: New York heart association. COPD: Chronic obstructive pulmonary disease. GOLD: Global initiative for chronic obstructive lung disease. qSOFA: quick sequential [sepsis-re- lated] organ failure assessment.

^a Score ranges from 0 to 3, with higher scores indicating greater likelihood of having severe CAP.

^b P < 0.0001.

^c More numbers represent greater severity of CAP.

Table 2

Relationship between number of adverse features and risk of mortality (n = 2116)

Features	No. Present or score	Total (%)	Died (%)
Minor criteria	0	776 (37)	2 (0.26)
	1	428 (20)	6 (1.40)
	2	124 (6)	20 (16.13)
	3	477 (23)	38 (7.97)
	4	190 (9)	27 (14.21)
	5	121 (6)	43 (35.54)
qSOFA	0	1167 (55)	4 (0.34)
	1	637 (30)	51 (8.01)
	2	251 (12)	26 (10.36)
	3	61 (3)	55 (90.16)

IDSA/ATS: The Infectious Disease Society of America and the American Thoracic Society. qSOFA: quick sequential [sepsis-related] organ failure assessment.

• 1. Performances of the individual criteria for the prediction of mortality among patients with CAP

Individual qSOFA elements demonstrated greater RRs and ORs com- pared with variables of minor criteria (Table 3). Prognostic perfor- mances of these criteria were reported in Table 4. Individual qSOFA elements showed superior diagnostic values to the variables of minor criteria, indicated by larger AUROCs and higher Youden’s indices.

• 1. The discriminatory capacities of scoring systems for predicting in-hospital mortality in patients with CAP

Cronbach ? with variables of qSOFA (? = 0.43; 95% lower confi- dence limit [LCL], 0.40) was greater than that of minor criteria (? = 0.14; 95% LCL, 0.10) or CURB-65 (? = 0.36; 95% LCL, 0.32). qSOFA

agreed well with minor criteria (? = 0.86; 95% LCL, 0.85), CURB-65 (? = 0.86; 95% LCL, 0.85), and PSI (? = 0.79; 95% LCL, 0.78).

qSOFA had the lowest sensitivity among all analyzed tools (59.6%; 95% CI, 50.8%-67.9%), a reasonable specificity of 88.3% (95% CI, 86.8%-

89.7%) and a higher PLR of 5.105 (95% CI, 4.247-6.137) for predicting mortality. However, minor criteria demonstrated a reasonable sensitiv- ity of 80.1% (72.5%-86.5%), a lower specificity of 65.8% (95% CI, 63.6%-

67.8%) and a lower PLR of 2.341 (95% CI, 2.110-2.596). The RR of

qSOFA for death was 8.515 (95% CI, 6.179-11.735) versus 6.817 (95% CI, 4.515-10.293) for minor criteria (Table 5).

The predictive validity of qSOFA was good for mortality (AUROC = 0.868; 95% CI, 0.853-0.882), was statistically greater than SOFA

(AUROC = 0.831; 95% CI, 0.815-0.847; P = 0.0002. NRI, 0.073) and

minor criteria (AUROC = 0.824; 95% CI, 0.807-0.840; P < 0.0001. NRI,

0.088), was equal to PSI (AUROC = 0.902; 95% CI, 0.889-0.915; P =

0.0700. NRI, -0.068), and was inferior compared with CURB-65 (AUROC = 0.919; 95% CI, 0.907-0.931; P < 0.0001. NRI, -0.103)

(Table 6 and Fig. 1).

1. Discussion

The main findings of the current study involving 2116 patients with CAP comprise the following: qSOFA demonstrated significant higher in- creased OR for mortality than minor criteria, a higher correlation with mortality, as well as higher internal consistency (Cronbach alpha 0.43 versus 0.14) and superior diagnostic values of individual elements (larger AUROCs and higher Youden’s indices). qSOFA >=2 was less sensi- tive but more specific for predicting mortality than >=3 minor criteria (qSOFA sensitivity 59.6%, specificity 88.3% and positive likelihood ratio

5.11 versus >=3 minor criteria sensitivity 80.1%, specificity 65.8% and pos-

itive likelihood ratio 2.34). The predictive validity of qSOFA with robust elements and higher convergence was good for mortality, was statisti- cally greater than minor criteria, was equal to PSI, and was inferior com- pared with CURB-65.

Association of the predictive rules of severe CAP with mortality (n = 2116)

Respiratory rate >= 22/min 602 (82)	132 (18)	249.444 < 0.0001	62.13 < 0.0001 75.54 < 0.0001 (23.07-167.33) (27.81-205.20)
Respiratory rate >= 30/min 252	24	2.7140	1.43 0.0979 1.47 0.1012
(91)	(9)	.0994	(0.94-2.18) (0.93-2.33)
PaO /FiO <= 250 mmHg 351	102	247.982	11.01 < 0.0001 13.92 < 0.0001
2 2	(23)	< 0.0001	(7.58-16.01) (9.29-20.87)
Multilobar infiltrates 892	75	5.2250	1.46 0.0231 1.50 0.0230
(92)	(8)	.0223	(1.05-2.03) (1.06-2.13)
Altered mentation (Confusion/disorientation) 160	70	247.184	8.70 < 0.0001 12.06 < 0.0001
(70)	(30)	< 0.0001	(6.40-11.82) (8.31-17.52)
Uremia 386	108	255.173	12.67 < 0.0001 15.93 < 0.0001
(78)	(22)	<0.0001	(8.46-18.96) (10.36-24.49)
Leukopenia 226	21	2.0010	1.38 0.1551 1.42 0.1590
(92)	(9)	.1572	(0.89-2.16) (0.87-2.30)
Thrombocytopenia 120	55	198.188	7.53 < 0.0001 10.53 < 0.0001
(69)	(31)	< 0.0001	(5.55-10.22) (7.13-15.53)
Hypothermia 142	12	0.5140	1.23 0.4710 1.25 0.4741
(92)	(8)	.4733	(0.70-2.180) (0.68-2.32)
Hypotensiona 471	34	0.1030	1.06 0.7481 1.07 0.7484
(93)	(7)	.7484	(0.73-1.55) (0.72-1.60)
SBP <=100 mmHg 297	66	100.618	4.55 < 0.0001 5.34 < 0 0.0001
(82)	(18)	< 0.0001	(3.32-6.25) (3.73-7.64)
Age >= 65 years 631	132	234.453	58.52 < 0.0001 70.55 < 0.0001
(83)	(17)	< 0.0001	(21.73-157.61) (25.97-191.63)

Variable Patients alive (%) x² and P value RR (95% CI) P value OR (95% CI) P value Yes No

(78)

NOTE: CAP: Community-acquired pneumonia. RR: Relative risk. CI: Confidence interval. OR: Odds ratio. PaO₂/FiO₂: Arterial oxygen pressure/fraction inspired oxygen. SBP: Systolic blood pressure.

^a Hypotension was defined as a systolic blood pressure < 90 mmHg or diastolic blood pressure <= 60 mmHg.

The Sepsis-3 task force estimated that patients with sepsis would have an in-hospital mortality rate greater than 10% [1]. In the present study, patients with qSOFA >=2 had an in-hospital mortality rate of 25.96% compared with 3.05% for patients with qSOFA <2. This result is consistent with prior studies of qSOFA in the non-ICU [2,22]. It implies that those with a high qSOFA score (>=2) may merit immediate deploy- ment of scarce critical care resources. All individual elements of qSOFA made contributions to mortality prediction, respiratory rate >= 22/min predicted best regarding discrimination for mortality, whereas 5

variables of minor criteria, i.e. respiratory rate >= 30/min, multilobar infil- trates, leukopenia, hypothermia and hypotension showed AUROCs less than 0.6 (that is similar to our previous findings [6]), which might be one of the foundations leading to the predictive difference between qSOFA and minor criteria.

The criteria thresholds of hypotension and tachpnea were stricter for minor criteria [5] and CURB-65 [23] than for qSOFA [1,2]. Which thresh- old is more accurate? The current study shed light on the questions. Re- spiratory rate >= 22/min predicted mortality better, as did systolic

Table 4

Performance of the individual criteria for the prediction of mortality among patients with CAP (n = 2116)

Variable	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PLR (95% CI)	NLR (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)	Youden’s index	AUROC (95% CI)
Respiratory rate >= 22/min	97.1	69.6	3.192	0.042	18.0	99.7	0.67	0.833
	(92.6-99.2)	(67.5-71.6)	(2.968-3.433)	(0.016-0.111)	(16.9-19.1)	(99.2-99.9)		(0.817-0.847)
Respiratory rate >= 30/min	17.6	87.3	1.387	0.944	8.7	93.9	0.05	0.525
	(11.6-25.1)	(85.7-88.7)	(0.947-2.029)	(0.871-1.022)	(6.1-12.2)	(93.4-94.4)		(0.503-0.546)
PaO2/FiO2 <= 250 mmHg	75.0	82.3	4.231	0.304	22.5	98.0	0.57	0.786
	(66.9-82.0)	(80.5-83.9)	(3.694-4.846)	(0.227-0.407)	(20.2-25.0)	(97.3-98.5)		(0.768-0.804)
Multilobar infiltrates	55.1	54.9	1.224	0.816	7.8	94.7	0.1	0.550
	(46.4-63.7)	(52.7-57.2)	(1.004-1.435)	(0.675-0.988)	(6.7-9.0)	(93.6-95.6)		(0.529-0.572)
Altered mentation	51.5	91.9	6.369	0.528	30.4	96.5	0.43	0.717
(Confusion/disorientation)	(42.8-60.1)	(90.6-93.1)	(5.108-7.942)	(0.444-0.628)	(26.0-35.3)	(95.9-97.0)		(0.697-0.736)
Uremia	79.4	80.5	4.073	0.256	21.9	98.3	0.60	0.800
	(71.6-85.9)	(78.7-82.2)	(3.599-4.610)	(0.184-0.356)	(19.8-24.1)	(97.6-98.8)		(0.782-0.816)
Leukopenia	15.4	88.6	1.353	0.955	8.5	93.8	0.04	0.520
	(9.8-22.6)	(87.1-90.0)	(0.896-2.043)	(0.887-1.027)	(5.8-12.3)	(93.4-94.3)		(0.499-0.542)
Thrombocytopenia	40.4	93.9	6.673	0.634	31.4	95.8	0.34	0.672
	(32.1-49.2)	(92.8-95.0)	(5.106-8.721)	(0.552-0.729)	(26.0-37.5)	(95.2-96.3)		(0.651-0.692)
Hypothermia	8.8	92.8	1.230	0.982	7.8	93.7	0.02	0.508
	(4.6-14.9)	(91.6-93.9)	(0.701-2.160)	(0.931-1.036)	(4.6-12.9)	(93.4-94.0)		(0.487-0.530)
Hypotensiona	25.0	76.2	1.051	0.984	6.7	93.7	0.01	0.506
	(18.0-33.1)	(74.3-78.1)	(0.777-1.421)	(0.890-1.088)	(5.1-8.9)	(93.0-94.2)		(0.485-0.528)
SBP <=100 mmHg	48.5	85.0	3.235	0.606	18.2	96.0	0.34	0.668
	(39.9-57.2)	(83.4-86.5)	(2.264-3.961)	(0.514-0.714)	(15.4-21.4)	(95.3-96.6)		(0.647-0.688)
Age >= 65 years	97.1	68.1	3.046	0.043	17.3	99.7	0.65	0.826
	(92.6-99.2)	(66.0-70.2)	(2.838-3.269)	(0.016-0.113)	(16.3-18.3)	(99.2-99.9)		(0.809-0.842)

CAP: Community-acquired pneumonia. CI: Confidence interval. PLR: Positive likelihood ratio. NLR: Negative likelihood ratio. PPV: Positive predictive value. NPV: Negative predictive value. AUROC: The area under the receiver operating characteristic curve. PaO₂/FiO₂: Arterial oxygen pressure/fraction inspired oxygen. SBP: Systolic blood pressure.

^a Hypotension was defined as a systolic blood pressure <90 mmHg or diastolic blood pressure <=60 mmHg.

Association of the scores for severe CAP with mortality and the performance for the prediction of mortality among patients with CAP (n = 2116)

Variable	qSOFA >=2	Minor criteria >=3	CURB-65 >= 3	PSI rank >=IV
Patients Alive (%)
Yes	231 (74)	678 (86)	184 (66)	723 (84)
No	81 (26)	109 (14)	94 (34)	134 (16)
x2	232.088	114.743	398.920	202.988
p value	<0.0001	<0.0001	<0.0001	<0.0001
RR (95% CI)	8.52 (6.18-11.74)	6.82 (4.52-10.29)	14.80 (10.52-20.82)	98.43 (24.43-396.57)
P value	< 0.0001	< 0.0001	< 0.0001	< 0.0001
OR (95% CI)	11.15 (7.71-16.13)	7.75 (5.04-11.94)	21.85 (14.73-32.40)	116.49 (28.75-472.01)
P value	<0.0001	<0.0001	<0.0001	<0.0001
Sensitivity, % (95% CI)	59.6 (50.8-67.9)	80.1 (72.5-86.5)	69.1 (60.6-76.8)	98.5 (94.8-99.8)
Specificity, % (95% CI)	88.3 (86.8-89.7)	65.8 (63.6-67.8)	90.7 (89.3-92.0)	63.5 (61.3-65.6)
PLR (95% CI)	5.105 (4.247-6.137)	2.341 (2.110-2.596)	7.438 (6.227-8.884)	2.698 (2.537-2.870)
NLR (95% CI)	0.458 (0.373-0.562)	0.302 (0.215-0.424)	0.340 (0.265-0.438)	0.023 (0.006-0.092)
PPV, % (95% CI)	26.0 (22.6-29.7)	13.9 (12.7-15.1)	33.8 (30.0-37.9)	15.6 (14.8-16.5)
NPV, % (95% CI)	97.0 (96.3-97.5)	98.0 (97.2-98.5)	97.7 (97.1-98.2)	99.8 (99.4-100.0)
Youden’s index	0.48	0.46	0.60	0.62
AUROC (95% CI)	0.739 (0.720-0.758)	0.730 (0.710-0.748)	0.799 (0.781-0.816)	0.810 (0.793-0.827)

CAP: Community-acquired pneumonia. qSOFA: quick sequential [sepsis-related] organ failure assessment. CURB-65: Confusion, urea >7 mmol.L^-1, respiratory rate >= 30 breaths.min^-1, low blood pressure, and age >= 65 years. PSI: pneumonia severity index. RR: Relative risk. CI: Confidence interval. OR: Odds ratio. PLR: Positive likelihood ratio. NLR: Negative likelihood ratio. PPV: Positive predictive value. NPV: Negative predictive value. AUROC: The area under the receiver operating characteristic curve.

arterial pressure <=100mmHg, indicated by orchestrating significant in- cremental improvements of AUROC and NRI, which might also be one of the foundations leading to the discrepancy between the two scoring systems. qSOFA was performed worse than CURB-65, as indicated by de- creases of AUROC and NRI. prognostic performance of age >=65 years was good for mortality. This might be envisaged to interpret the reason why CURB-65 was performed better than qSOFA. These above-mentioned findings require external validation. CURB-65 predicted mortality equally to PSI, which is in consonance with what Chalmers et al. [24] concluded. The convergent validity (one of the 2 domains of construct validity) between qSOFA and the other scores were good, indicated by high Cronbach ?. Furthermore, the convergent validity among variables of qSOFA was greater than that of minor criteria despite Cronbach ? less than 0.7, which might be another significant foundation leading to the predictive difference of the two scoring systems, although minor criteria responsible for clinical triage were well validated [7,13,25]. Discrimina- tive power of the scores was assessed by ROC analysis and NRI regarding the presence of our primary outcome. We revealed that qSOFA per- formed well, which is in line with the Sepsis 3 task force study that re- ported an AUROC of 0.81 for qSOFA for non-ICU encounters [2],

Table 6

AUROC values for different rules to predict mortality and their comparisons

Feature	AUROC value	Standard error	95% CI
qSOFA	0.868	0.0139	0.853-0.882
SOFA	0.831	0.0151	0.815-0.847
Minor criteria	0.824	0.0149	0.807-0.840
CURB-65	0.919	0.0092	0.907-0.931
PSI	0.902	0.0117	0.889-0.915
	Difference	z statistic	P value
qSOFA ~ SOFA	0.0365	3.699	0.0002
qSOFA ~ Minor criteria	0.0441	4.307	< 0.0001
qSOFA ~ CURB-65	0.0514	5.121	< 0.0001
qSOFA ~ PSI	0.0343	1.812	0.0700
SOFA ~ Minor criteria	0.0076	0.871	0.3839
SOFA ~ CURB-65	0.0879	6.920	< 0 0.0001
SOFA ~ PSI	0.0708	3.449	0.0006
Minor criteria ~ CURB-65	0.0955	9.083	< 0.0001
Minor criteria ~ PSI	0.0784	3.828	0.0001
CURB-65 ~ PSI	0.0171	1.102	0.2704

AUROC: The area under the receiver operating characteristic curve. CI: Confidence inter- val. qSOFA: quick sequential [sepsis-related] organ failure assessment. SOFA: Sequential [sepsis-related] organ failure assessment. CURB-65: Confusion, urea >7 mmol.L^-1, respi- ratory rate >= 30 breaths.min^-1, low blood pressure, and age >= 65 yrs. PSI: Pneumonia se- verity index.

Freund’s study (an AUROC of 0.80 for patients with suspected infection) [22], Ranzani’s study (an AUROC of 0.70 for patients with CAP) [10] and Jiang’s meta-analysis (a summary AUROC of 0.70 for patients with pneumonia) [12], clearly outperformed minor criteria, which is contra- dictory to what Ahnert et al. [11] reported, and performed worse than CURB-65, which is similar to Ranzani’s study [10], and was different from those reported by George et al. [26], Kesselmeier et al. [27], Muller et al. [28], and Chen et al. [29]. The outperformance of qSOFA against minor criteria for predicting mortality might have implications for more convenient and accurate clinical triage decisions. However, Ahnert et al. [11] deduced that SOFA-like scores (such as minor criteria), covering the current state of the lung and extrapulmonary organs prone

Fig. 1. ROC curves for mortality prediction by qSOFA, SOFA, minor criteria, CURB-65, and PSI.

ROC: The receiver operating characteristic. qSOFA: quick sequential [sepsis-related] organ failure assessment. SOFA: Sequential [sepsis-related] organ failure assessment. CURB-65: Confusion, urea >7 mmol.L^-1, respiratory rate >= 30 breaths.min^-1, low blood pressure, and age >= 65 years. PSI: Pneumonia severity index.

Table 7

The heterogeneity of qSOFA’s sensitivities and specificities for mortality in different studies

Study	Design	Participant	Sensitivity	Specificity
Guo (the current)	Prospective cohort	2116 patients with CAP	59.6%	88.3%
Machado [32]	Prospective cohort	10,171 patients with suspected infection or sepsis	53.9%	83.6%
Ranzani [10]	Retrospective cohort	6874 patients with CAP	50%	81%
Jiang [12]	Systematic review and meta-analysis	6 studies with 17,868 pneumonia patients	43%	86%
Tan [33]	Systematic review and meta-analysis	36 studies with 272,478 suspected infection patients	48%	86%
Tian [34]	Population-based cohort	1716 patients with infection	50.2%	78.1%
Ahnert [11]	Prospective cohort	1532 patients with CAP	85%	47%
Kesselmeier [27]	Retrospective population-based cohort	1,262,250 patients with CAP	25.8%	92.7%
Freund [22]	Prospective cohort	879 patients with suspected infection	70%	79%
Finkelsztein [35]	Prospective cohort	152 patients with suspicion of sepsis	90%	42%

qSOFA: quick sequential [sepsis-related] organ failure assessment. CAP: Community-acquired pneumonia.

to deterioration during CAP, perform better in identifying a severe state of CAP, which might be the causation leading to better performance of minor criteria compared with qSOFA. The current study demonstrated that SOFA and minor criteria were equal in their predictive ability of mortality in patients with CAP, but revealed that qSOFA was superior to SOFA. 28-day mortality was low in Ahnert’s cohort. Therefore, most primary endpoint cases were transferred to the ICU. The patients were enrolled at general wards (75%), ICUs (13%), and emergency depart- ments (10%) in Ahnert’s study [11]. Therefore, the characteristics were different from those in our cohort, which might be envisaged to inter- pret the discrepancy between the two studies. Hence, further research is warranted to better understand potential generalizability.

A high sensitivity should be a prerequisite for any screening tool. If the sensitivity is too low, some patients may be missed, but if too high, too many patients may be “flagged” as possibly having sepsis, making practi- cal implementation impossible [30]. However, thoughtful criticisms have also been articulated. One of the consequences of using predictive validity for mortality is an excessive weight on specificity at the expense of sensi- tivity, which may lead to delays in initiation of treatment [31]. qSOFA had a low sensitivity of 59.6%, although with a reasonable specificity of 88.3%, which is similar to Machado’s study (53.9% and 83.6% for patients with suspected infection) [32], is higher than those reported by Ranzani et al.

[10] (50% and 81% for patients with CAP), Jiang et al. [12] (43% and 86% for patients with pneumonia), Tan et al. [33] (48% and 86% for patients with suspected infection) and Tian et al. [34] (50.2% and 78.1% for pa- tients with infection), and yet differs from what Ahnert et al. [11] found (85% and 47% for patients with CAP), Kesselmeier et al. [27] showed (25.8% and 92.7% for patients with CAP), Freund et al. [22] discovered (70% and 79% for patients with suspected infection) and Finkelsztein et al. [35] reported (90% and 42% for patients with suspicion of sepsis). The heterogeneity of qSOFA’s sensitivities and specificities for mortality in different patient populations and different studies was arranged in Table 7. A low sensitivity of qSOFA suggests that it would miss about half of the severe CAP patients, leading to a deferral of these patients who are at higher risk of death until development of overt organ failure [32,34]. This should alert the healthcare team that many patients are at risk of dying even with a qSOFA <=1. This is in consonance with the concept that qSOFA is a rule-in and not a rule-out tool [1].

Taken together, these findings demonstrated the robustness and su- periority of qSOFA for predicting mortality in patients with CAP. An ob- vious advantage of qSOFA is that it does not require any biological tests, and therefore may be particularly valuable in resource-limited settings. The specificity of qSOFA and the brevity of this tool make it a good op- tion to rapidly identify those patients with CAP at higher risk of death and thus those who should be prioritized. Furthermore, operationalizing qSOFA with 3 easily measured variables should in- crease both generalizability and clinical utility.

• 1. Limitations

Our findings must be interpreted in light of several limitations. First, the prospective cohort was derived from two centers in a city, but not

multicenter settings located in different cities in different countries. This may limit the generalizability of the results. Second, only 136 pa- tients met the primary outcome, which may be considered relatively low. Third, this study focused on adult patients only and did not evalu- ate children at risk for severe CAP.

1. Conclusions

The qSOFA predicted mortality in CAP better than IDSA/ATS minor criteria and worse than CURB-65 with robust elements and higher con- vergence. qSOFA as a bedside prompt might be positioned as a proxy for minor criteria and facilitate earlier and better recognition and thus merit more timely and appropriate management of CAP patients likely to fare poorly or at risk of developing severe CAP, which might have im- plications for more accurate clinical triage decisions.

Authors’ contributions

Q.G was in charge of funding acquisition and project administration. Q.G, H-Y.L and W-D.S made substantial contributions to conception and design, and were in charge of data collection and curation, and the writ- ing of the manuscript. L-H.L and Q-Z.Z read the chest radiographs and computer tomography scans. H.L, H-Q.Y, Y-H.L, and Z-D.L made substan- tial contributions to acquisition of data. M.J was in charge of statistical analysis. Each author has participated in the writing of the manuscript, been involved in the analysis of the data, and seen and approved the submitted version.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Funding

This work was supported by the medical science and technology foundation of Guangdong province [grant number A2010553]; the planned science and technology project of Shenzhen municipality [grant number 201102078]; and the non-profit scientific research pro- ject of Futian district [grant number FTWS201120].

Role of the Funder/Sponsor

The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Declaration of Competing Interest

None.

Acknowledgements

We are indebted to the nurses, further education physicians, and postgraduates of the Departments of Pulmonary and Critical Care Med- icine for making contributions to this study.

References

1. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis- 3). JAMA. 2016;315:801-10. https://doi.org/10.1001/jama.2016.0287.
2. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assess- ment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315:762-74. https://doi.org/10. 1001/jama.2016.0288.
3. Dremsizov T, Clermont G, Kellum JA, Kalassian KG, Fine MJ, Angus DC. Severe sepsis in community-acquired pneumonia: when does it happen, and do systemic inflam- matory response syndrome criteria help predict course? Chest. 2006;129:968-78. https://doi.org/10.1378/chest.129.4.968.
4. Prina E, Ranzani OT, Torres A. Community-acquired pneumonia. Lancet. 2015;386: 1097-108. https://doi.org/10.1016/S0140-6736(15)60733-4.
5. Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC, et al. In- fectious Diseases Society of America/American Thoracic Society consensus guide- lines on the management of community-acquired pneumonia in adults. Clin Infect Dis. 2007;44:S27-72. https://doi.org/10.1086/511159.
6. Guo Q, Li HY, Zhou YP, Li M, Chen XK, Liu H, et al. Weight of the IDSA/ATS minor criteria for severe community-acquired pneumonia. Respir Med. 2011;105: 1543-9. https://doi.org/10.1016/j.rmed.2011.06.010.
7. Liapikou A, Ferrer M, Polverino E, Balasso V, Esperatti M, Piner R, et al. Severe community-acquired pneumonia: validation of the Infectious Diseases Society of America/American Thoracic Society guidelines to predict an intensive care unit ad- mission. Clin Infect Dis. 2009;48:377-85. https://doi.org/10.1086/596307.
8. Sibila O, Mortensen EM, Redrow G, Lugo E, Laserna E, Anzueto A, et al. Evaluation of the IDSA/ATS minor criteria for severe community-acquired pneumonia. Hosp Pract (1995). 2012;40:158-64. https://doi.org/10.3810/hp.2012.04.981.
9. Guo Q, Song WD, Li HY, Li M, Chen XK, Liu H, et al. Cold-inducible RNA-binding pro- tein might determine the severity and the presences of major/minor criteria for se- vere community-acquired pneumonia and best predicted mortality. Respir Res. 2020;21:192. https://doi.org/10.1186/s12931-020-01457-2.
10. Ranzani OT, Prina E, Menendez R, Ceccato A, Cilloniz C, Mendez R, et al. New Sepsis Definition (Sepsis-3) and Community-acquired Pneumonia Mortality. A Validation and clinical decision-making Study. Am J Respir Crit Care Med. 2017;196: 1287-97. https://doi.org/10.1164/rccm.201611-2262OC.
11. Ahnert P, Creutz P, Horn K, Schwarzenberger F, Kiehntopf M, Hossain H, et al. Se- quential organ failure assessment score is an excellent operationalization of disease severity of adult patients with hospitalized community acquired pneumonia – re- sults from the prospective observational PROGRESS study. Crit Care. 2019;23:110. https://doi.org/10.1186/s13054-019-2316-x.
12. Jiang J, Yang J, Jin Y, Cao J, Lu Y. Role of qSOFA in predicting mortality of pneumonia: A systematic review and meta-analysis. Medicine (Baltimore). 2018;97:e12634. https://doi.org/10.1097/MD.0000000000012634.
13. Phua J, See KC, Chan YH, Widjaja LS, Aung NW, Ngerng WJ, et al. Validation and clin- ical implications of the IDSA/ATS minor criteria for severe community-acquired pneumonia. Thorax. 2009;64:598-603. https://doi.org/10.1136/thx.2009.113795.
14. Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and sep- tic shock, 2012. Crit Care Med. 2013;41:580-637. https://doi.org/10.1097/CCM. 0b013e31827e83af.
15. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43:304-77. https://doi.org/10.1007/ s00134-017-4683-6.
16. Bland JM, Altman DG. Cronbach’s alpha. BMJ. 1997;314:572. https://doi.org/10. 1136/bmj.314.7080.572.
17. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait- multimethod matrix. Psychol Bull. 1959;56:81-105.
18. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29-36. https://doi.org/10.1148/ra- diology.143.1.7063747.
19. Pencina MJ, D’Agostino Sr RB, D’Agostino Jr RB, Vasan RS. Evaluating the added pre- dictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157-72. https://doi.org/10.1002/sim.2929.
20. Pencina MJ, D’Agostino Sr RB, Steyerberg EW. Extensions of net reclassification im- provement calculations to measure usefulness of new biomarkers. Stat Med. 2011; 30:11-21. https://doi.org/10.1002/sim.4085.
21. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassifica- tion improvement: computation, interpretation, and controversies: a literature re- view and clinician’s guide. Ann Intern Med. 2014;160:122-31. https://doi.org/10. 7326/M13-1522.
22. Freund Y, Lemachatti N, Krastinova E, Van Laer M, Claessens YE, Avondo A, et al. Prognostic accuracy of Sepsis-3 Criteria for In-Hospital Mortality Among Patients With Suspected Infection Presenting to the Emergency Department. JAMA. 2017; 317:301-8. https://doi.org/10.1001/jama.2016.20329.
23. Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, Town GI, et al. Defin- ing community acquired pneumonia severity on presentation to hospital: an inter- national derivation and validation study. Thorax. 2003;58:377-82. https://doi.org/ 10.1136/thorax.58.5.377.
24. Chalmers JD, Singanayagam A, Akram AR, Mandal P, Short PM, Choudhury G, et al. Severity assessment tools for predicting mortality in hospitalised patients with community-acquired pneumonia. Systematic review and meta-analysis. Thorax. 2010;65:878-83. https://doi.org/10.1136/thx.2009.133280.
25. Brown SM, Jones BE, Jephson AR, Dean NC, Infectious Disease Society of America/ American Thoracic Society 2007.. Validation of the Infectious Disease Society of America/American Thoracic Society 2007 guidelines for severe community- acquired pneumonia. Crit Care Med. 2009;37:3010-6. https://doi.org/10.1097/ CCM.0b013e3181b030d9.
26. George N, Elie-Turenne MC, Seethala RR, Baslanti TO, Bozorgmehri S, Mark K, et al. External Validation of the qSOFA Score in Emergency Department Patients With Pneumonia. J Emerg Med. 2019;57:755-64. https://doi.org/10.1016/j.jemermed. 2019.08.043.
27. Kesselmeier M, Pletz MW, Blankenstein AL, Scherag A, Bauer T, Ewig S, et al. Valida- tion of the qSOFA score compared to the CRB-65 score for risk prediction in community-acquired pneumonia. Clin Microbiol Infect. 2021;27:1345.e1-6. https://doi.org/10.1016/j.cmi.2020.10.008.
28. Muller M, Guignard V, Schefold JC, Leichtle AB, Exadaktylos AK, Pfortmueller CA. Utility of quick Sepsis-related Organ Failure Assessment to predict outcome in patients with pneumonia. PLoS One. 2017;12:e0188913. https://doi.org/10.1371/ journal.pone.0188913.
29. Chen YX, Wang JY, Guo SB. Use of CRB-65 and quick Sepsis-related Organ Failure As- sessment to predict site of care and mortality in pneumonia patients in the emer- gency department: a retrospective study. Crit Care. 2016;20:167. https://doi.org/ 10.1186/s13054-016-1351-0.
30. Vincent JL, Mira JP, Antonelli M. Sepsis: older and newer concepts. Lancet Respir Med. 2016;4:237-40. https://doi.org/10.1016/S2213-2600(15)00522-6.
31. Simpson SQ. New Sepsis criteria: a change we should not make. Chest. 2016;149: 1117-8. https://doi.org/10.1016/j.chest.2016.02.653.
32. Machado FR, Cavalcanti AB, Monteiro MB, Sousa JL, Bossa A, Bafi AT, et al. predictive accuracy of the Quick Sepsis-related Organ Failure Assessment score in Brazil. A Pro- spective Multicenter Study. Am J Respir Crit Care Med. 2020;201:789-98. https:// doi.org/10.1164/rccm.201905-0917OC.
33. Tan TL, Tang YJ, Ching LJ, Abdullah N, Neoh HM. Comparison of Prognostic Accuracy of the quick Sepsis-Related Organ Failure Assessment between Short- & Long-term Mortality in Patients Presenting Outside of the Intensive Care Unit – A Systematic Review & Meta-analysis. Sci Rep. 2018;8:16698. https://doi.org/10.1038/s41598- 018-35144-6.
34. Tian H, Zhou J, Weng L, Hu X, Peng J, Wang C, et al. Accuracy of qSOFA for the diag- nosis of sepsis-3: a secondary analysis of a population-based cohort study. J Thorac Dis. 2019;11:2034-42. https://doi.org/10.21037/jtd.2019.04.90.
35. Finkelsztein EJ, Jones DS, Ma KC, Pabon MA, Delgado T, Nakahira K, et al. Comparison of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sep- sis outside the intensive care unit. Crit Care. 2017;21:73. https://doi.org/10.1186/ s13054-017-1658-5.