a b s t r a c t

Background: Boarding of ICU patients in the ED is increasing. Illness severity scores may help emergency physi- cians stratify risk to guide earlier transfer to the ICU and assess pre-ICU interventions by adjusting for baseline mortality risk. Most existing illness severity scores are based on data that is not available at the time of the hos- pital admission decision or cannot be extracted from the electronic health record . We adapted the SOFA score to create a new illness severity score (eccSOFA) that can be calculated at the time of ICU admission Order entry in the ED using EHR data. We evaluated this score in a cohort of emergency critical care (ECC) patients at a single academic center over a period of 3 years.

Methods: This was a retrospective cohort study using EHR data to assess predictive accuracy of eccSOFA for esti- mating in-hospital mortality risk. The patient population included all adult patients who had a critical care admis- sion order entered while in the ED of an academic medical center between 10/24/2013 and 9/30/2016. eccSOFA’s discriminatory ability for in-hospital mortality was assessed using ROC curves.

Results: Of the 3912 patients whose in-hospital mortality risk was estimated, 2260 (57.8%) were in the low-risk group (scores 0-3), 1203 (30.8%) in the intermediate-risk group (scores 4-7), and 449 (11.5%) in the high-risk group (scores 8+). In-hospital mortality for the low-, intermediate, and high-risk groups was 4.2% (95%CI: 3.4-5.1), 15.5% (95% CI 13.5-17.6), and 37.9% (95% CI 33.4-42.3) respectively. The AUROC was 0.78 (95%CI:

0.75-0.80) for the integer score and 0.75 (95% CI: 0.72-0.77) for the categorical eccSOFA.

Conclusions: As a predictor of in-hospital mortality, eccSOFA can be calculated based on variables that are com- monly available at the time of critical care admission order entry in the ED and has discriminatory ability that is comparable to other commonly used illness severity scores. Future studies should assess the calibration of our absolute risk predictions.

(C) 2020

1. Introduction

In caring for acute critical illness, the ED’s role is to provide initial evaluation and resuscitation, not ongoing Longitudinal care after ICU ad- mission [1]. However, when the demand for ICU beds exceeds the sup- ply, the number of ICU patients boarding in the ED increases. We refer to patients with ICU admission orders who remain in the ED as emergency critical care (ECC) patients. Stratification of ECC patients based on in- hospital mortality risk can be a useful triage tool to help identify those who may benefit most from immediate transfer to the ICU and those least likely to be harmed by prolongED boarding in the ED. [2] In addi- tion to use as a triage tool, risk stratification for ECC patients is

* Corresponding author.

E-mail address: [email protected] (K. Niknam).

important to analyze and compare the efficacy of pre-ICU interventions, such as implementation of ED-basED intensive care units [3,4] or, as in our case, specialized physician and nursing teams to care for ECC pa- tients [5,6].

Illness severity scores have been used widely to risk-stratify patients who are already in the ICU. However, these scores are calculated from 1 to 24 h after arrival in the ICU and cannot be applied to ECC patients or they cannot be calculated automatically from the electronic health re- cord (EHR). We set out to adapt and test the Sequential Organ Failure Assessment score for the use in the ED based on information available at the time of the critical care admission order and calculated automatically from the electronic health record (EHR). We also modi- fied the score to reflect changes in clinical care of the critically ill since the score was developed in 1996 [7]. This new emergency critical care SOFA (eccSOFA) score is a one-time measurement - not a measurement

https://doi.org/10.1016/j.ajem.2020.12.018

0735-6757/(C) 2020

that is taken sequentially; it is based on the worst values of physiological measurements from ED arrival to the time of the critical care admission order.

This retrospective cohort study evaluated the discrimination of the eccSOFA score as a predictor of in-hospital mortality in the ECC patients at our hospital over a 3-year period. We also compared the eccSOFA to a much simpler adaptation of the SOFA score, the qSOFA [8], to assess whether eccSOFA added prognostic value.

1. Methods
   1. population and setting

This was a retrospective cohort study that included all patients

>=18 years old who had a critical care admission order entered while in the ED of Stanford University Hospital, a suburban university teaching hospital and Level I trauma center, between 10/24/2013 and 9/30/ 2016. This included medical, surgical (including trauma), neurological/ neurosurgical, cardiac, and cardiothoracic ICU patients. Patients who were transferred to an outside acute care hospital from our ED were ex- cluded. Patients who were admitted to our hospital and subsequently transferred to another acute care hospital were included as surviving to discharge. This study was approved by our institution’s review board (IRB).

• 1. Measurements

Demographics and variables required to calculate the eccSOFA score were obtained through queries of the EHR’s (EPIC) data warehouse (Clarity). The eccSOFA has the same six components as the original SOFA score [9], each rated 0 to 4 and summed into a final score from 0 to 24 (Table 1). The six components are 1) cardiovascular (mean arterial pressure and vasopressor support); 2) central nervous system (Glasgow Coma Score); 3) coagulation (platelet count); 4) hepatic (total biliru- bin); 5) renal (creatinine); and 6) respiratory (PaO2/FiO2 ratio and re- spiratory support). Instead of using the worst measurements over the previous 24 h, eccSOFA uses the worst measurements up to the time of the critical care admission order. In this study, we used data that

was entered into the record up to one hour after the admission order to allow for lags in documentation. If the measurement needed to calcu- late a component score was unavailable, the score was set to 0.

The eccSOFA score also includes modifications to the cardiovascular, central nervous system, renal, and respiratory components. The cardio- vascular component is still based on mean arterial pressure (MAP) mea- surements and the dose of epinephrine, norepinephrine, dobutamine and/or dopamine, but it is modified to add one extra point (up to a max- imum of four) if the patient was on vasopressin or phenylephrine, vaso- pressors commonly used today but not included in the original SOFA calculation. The CNS component uses the worst Glasgow Coma Score up to the time a paralytic was administered. The renal component uses the worst creatinine measurement, but not urine output, which is often unavailable or unreliable in the ED.

For the respiratory component, in addition to calculating PaO2/FiO2 ratios (P/F ratios) from recorded PaO2 and FiO2 measurements, we used oxygen saturation (SpO2) and oxygen flow rates in liters per minute and converted them to PaO2 and FiO2 using standard calculations as described in Appendix I. In determining the P/F ratio, the PaO2 (or converted SpO2) was paired with an FiO2 (or convertED flow rate) that was recorded at the same time or prior. We included high-flow nasal cannula in addition to non-invasive and invasive mechanical ventilation as respiratory support. These modifications reflect contem- porary treatment of hypoxic respiratory failure and the reduced use of arterial blood gases in the ED. We did not use P/F ratios after ECMO was initiated. Appendix I contains additional details on calculation of the eccSOFA score.

The above modifications were not made by “tuning” to a derivation dataset. They were necessary to allow calculation of the score based on EHR data available at the time of admission order entry or to reflect changes in critical care that have occurred since the original SOFA score was developed.

The qSOFA score [8] is a much simpler adaptation of the original SOFA score than eccSOFA. The score assigns one point each for respira- tory rate >= 22, GCS < 15, and systolic blood pressure <= 100. We calcu- lated qSOFA using the same data that went into the eccSOFA score. Although respiratory rate is not part of the eccSOFA score, it was in- cluded with blood pressure and other vital signs in the EHR data.

Table 1

Emergency critical care sequential organ failure assessment (eccSOFA) scoring

eccSOFA score 0 1 2 3 4

Cardiovascular

Hypotension, ug/kg/min MAP >=70 mmHg

and

No vasopressor

MAP <70 mmHg Dopamine <5 or

Dobutamine (any dose)

Dopamine 5 to <15 or

Epinephrine <=0.1 or Norepinephrine <=0.1

Dopamine >=15 or

Epinephrine >0.1 or

Norepinephrine >0.1

Vasopressin or phenylephrine: Add 1, up to a maximum of 4

Central nervous system

Glasgow Coma Score 15 13 to 14 10 to 12 6 to 9 < 6

GCS not used in score after paralytics administered to avoid falsely low values.

Coagulation

Platelets, 103/mm3 Liver	>= 150	100 to <150	50 to <100	20 to <50	< 20
Bilirubin, mg/dl Renal	< 1.2	1.2 to <2.0	2.0 to <6.0	6.0 to <12.0	>= 12.0
Creatine, mg/dl	< 1.2	1.2 to <2.0	2.0 to <3.5	3.5 to <5.0	>= 5.0

Urine output not used as often unavailable or unreliable during the early phase of ED care

Respiration

PaO2/FiO2 >= 400 300 to <400 200 to <300 100 to <200 and

respiratory support

PaO2 paired with FiO2 recorded at the same time or prior (including high-flow nasal cannula > 15 L/min)

< 100 and respiratory support

(including high-flow nasal cannula > 15 L/min)

Note.

MAP = mean arterial pressure, ECMO = extracorporeal membrane oxygenation. Glasgow coma scores were excluded if the patient is receiving paralytics.

Respiratory support included invasive and non-invasive modalities including high-flow nasal cannula. P/F ratios after initiation of ECMO were excluded.

eccSOFA component scores set to zero if measurement is missing.

• 1. Statistical analysis

We assessed the performance of eccSOFA in the above-described 3- year cohort of ED patients with critical care admission orders. In addi- tion to assessing the integer eccSOFA scores, patients were combined into low (eccSOFA 0-3), intermediate (4-7), and high (8+) risk catego- ries. The in-hospital mortality proportions with 95% confidence interval by integer score and by risk category were calculated. The discrimina- tion of the integer scores and risk categories was assessed using ROC curves with calculated area under the ROC curve (AUROC) [10]. The AUROCs for the integer eccSOFA, the categorical eccSOFA, and the qSOFA were compared using the DeLong method [11]. Statistical analy- ses were conducted using STATA/SE 15.1.

1. Results

A total of 3912 patients were included in the final cohort covering 3 years. The average age was 61.8; 57% were male (Table 2). The mean eccSOFA score was 3.5. Of the 3912, 453 patients died during the hospitalization (11.6%, 95%CI: 10.7-12.5%). The total of 3912 ex- cludes 9 patients who were transferred from the ED to be admitted to another acute care hospital but includes 213 (5.4%) transfers after reaching the ICU. These were coded as survival to discharge.

By risk group, in-hospital mortality was as follows: low-risk - 4.2%, intermediate risk- 15.5%, and high-risk 37.9% (Table 3). Mortality risk by integer score is shown in Fig. 1. Of the 21 patients (0.5%) who had eccSOFA scores >=14, 15 patients (85.7%) died. The ROC curve depicts how well the eccSOFA score discriminates between ECC patients who subsequently died in the hospital and those who were discharged alive. (Fig. 2). The AUROC was 0.775 (95% CI: 0.753-0.797) for the inte- ger score and 0.745 (95% CI: 0.722-0.769) for the categorical score. For comparison, the qSOFA score had an AUROC of 0.68 (95% CI 0.65-0.70), significantly lower than for the categorical eccSOFA (p < 0.0001).

1. Discussion

The need to provide care for ICU patients boarding in the ED (ECC pa- tients) brought our research team together. In our institution, the aver- age boarding time between entry of the ICU admission order and transfer out of the ED increased by 2 h from 2014 to 2016 and the pro- portion of ECC patients boarding for >6 h doubled from 14% to 27% dur- ing the same period [6]. Similar increases are reported widely [2] ina trend that was apparent 10 years ago [12]. We are studying nursing and physician interventions to improve care of ECC patients and find it necessary to adjust for illness severity. We also thought an illness sever- ity score that can be calculated in real time from the EHR might help EDs identify the highest risk ECC patients for immediate ICU transfer. In our experience, when one ICU bed is available for two patients boarding in

Table 2

In-hospital mortality and emergency critical care patient characteristics.

N 3912

Male sex^a 2249 (57%)

Age, mean (SD) 61.8 (20.0)

Death 453 (11.6%)

eccSOFA, mean (SD) 3.5 (3.0)

eccSOFA score strata

Low risk (0 to 3)		2260 (57.8%)
Medium risk (4 to 7)		1203 (30.8%)
High risk (>= 8)		449 (11.5%)
Trauma	1016 (26.0%)
Time to ICU admission order, mean (SD)	3.4 (3.1)
Boarding time hours, mean (SD)	4.6 (5.6)

Note. Data is presented as n (%) unless otherwise specified.

eccSOFA = Emergency critical care sequential organ failure assessment.

^a Two patients with unknown sex.

Table 3

Mortality stratified by emergency critical care sequential organ failure assessment (eccSOFA)

	N	Died	% (95% CI)
Low risk (0 to 3)	2260	96	4.2% (3.4%, 5.1%)
Medium risk (4 to 7)	1203	187	15.5% (13.5%, 17.6%)
High risk (>= 8)	449	170	37.9% (33.4%, 42.3%)
Total	3912	453	11.6% (10.7%, 12.5%)

the ED, the higher acuity patient gets the bed. This is assumed to benefit both the patient and the ED. Our study does not test this assumption. It is possible that an illness severity score could be used as a tool to divide ECC patients into 3 groups: Yellow - less sick and can stay in ED; Red - sicker and should be transferred immediately; Black - too sick to benefit from immediate transfer.

Rather than modifying the SOFA score, we originally considered using the Mortality prediction Model at time 0 (MPM₀), which Higgins et al. found to have an AUROC of 0.823 when calculated based on data extracted up to one hour after arrival in the ICU [13]. However, while the MPM₀ is calculated at the time of ICU admission, it is used primarily to adjust for severity of illness as part of a benchmarking effort, not for prospective risk stratification [13]. Moreover, the MPM₀ requires chronic and acute diagnoses, a determination of whether intracranial mass effect is present on neuroimaging, and whether the patient re- ceived CPR within 24 h prior to admission, all of which are difficult to re- liably extract from the EHR. Finally, the MPM₀ includes “code status” (i.e., whether the patient has a do-not-resuscitate order). While patient and family preferences about resuscitation are important in emergency critical care, we seek an understanding of the patient’s mortality risk based on physiological data alone. Indeed, in our experience, ECC pa- tients and families often request a mortality risk estimate during goals of Care discussions and may alter code status based on that assessment. Besides MPM₀, we also considered using the Acute Physiology and Chronic Health Evaluation (APACHE II) and Simplified Acute Physiology Score (SAPS-3) to assess the illness severity of ECC patients. Both have favorable AUROCs (>0.8) for in-hospital mortality prediction in criti- cally ill patients [14,15], but neither was designed for use in ECC pa- tients. APACHE requires data from the first 24 h of ICU admission and a subjective judgement about severe organ failure/immune compromise. In addition, it has a “high abstraction burden” [16]. While SAPS-3 can be calculated within 1 h of ICU admission, it includes several variables (length of stay before ICU admission, location before ICU admission, planned vs. unplanned admission, etc.) that do not apply to ECC patients. It also requires information about comorbidities, which may not be available in ED patients and can be difficult to extract

from the EHR.

Jones et al. previously assessed SAPS-2, MPM₀-II, and LODS for pa- tients presenting to the ED with shock using only variables available in the ED, and found only moderate Predictive ability for in-hospital death (SAPS-II AUROC 0.72, 95% CI: 0.57-0.87; MPM₀-II AUROC 0.69,

95%CI 0.54-0.84; and LODS AUROC 0.60, 95%CI: 0.45-0.76) [17]. On

the other hand, the same authors found that the Sequential Organ Fail- ure Assessment (SOFA) score discriminated fairly well in the evaluation of ED patients with sepsis [9]. The AUROC was 0.75 (95%CI: 0.68-0.83). Garcia-Gigorro et al. assessed the SOFA as a predictor of mortality in ED patients and found an AUROC of 0.75 (CI 0.66-0.83) [18].

We adapted SOFA so it could be calculated 1) at the time of admis- sion order entry; 2) from the EHR; and 3) accommodating changes in the clinical practice for the critically ill. The adaptations were not based on a derivation dataset. Since our scoring variables were not se- lected based on our cohort, this is effectively a validation dataset. When applied to 3912 ECC patients with 11.6% in-hospital mortality, the eccSOFA score had an AUROC of 0.775 (95% CI: 0.753-0.797). For comparison, the standard SOFA score, when applied to 248 severe sepsis

Fig. 1. In-hospital mortality proportion and number of patients with each integer eccSOFA score.

Fig. 2. receiver operating characteristic curves predicting in-hospital mortality using both integer and categorical emergency critical care sequential organ failure assessment (eccSOFA) scores and qSOFA. AUROC = Area Under the ROC curve.

ED patients with 20.6% in-hospital mortality, had an AUROC of 0.75 (95% CI: 0.68-0.83) [9]. When applied to 184,875 sepsis patients in the ICU for 24 h (in-hospital mortality of 18.7%) the AUROC was 0.753 (0.750

to 0.757) [19].

The main distinction between eccSOFA and standard SOFA is the time at which it is calculated. eccSOFA is intended to be calculated at the time of ICU admission order entry. Standard SOFA is calculated after 24 h in the ICU. We debated just referring to this score as “SOFA in ED” and then adding what we had to in order to calculate it - exclude urine output, add phenylephrine/vasopressin, and accommodate high flow nasal cannula and ECMO. These modifications will result in higher scores for patients on phenylephrine or vasopressin but otherwise sim- ply make the score easier to calculate in the ED. We expect the eccSOFA score’s discrimination to be reproducible in other populations. The AUROC should be maintained, although the calibration may be less re- producible. (See Limitations.)

Although qSOFA was originally intended for sepsis screening, several authors have considered using it in the general critical care population as a predictor of in-hospital mortality. Lo et al. reviewed 12 studies eval- uating qSOFA as a predictor of in-hospital mortality in critical care pa- tients with or without infection [20]. The AUROC ranged from 0.58-0.81 (median 0.65), and the authors concluded that qSOFA is not a clinically useful Prognostic tool for in-hospital mortality. We found a similar AUROC for qSOFA of 0.676 in ECC patients.

In addition to qSOFA, another adaptation of the original SOFA score called eSOFA has been utilized to assess illness severity [21]. Much like eccSOFA, eSOFA was developed as a simpler way to calculate a SOFA score by utilizing readily available EHR data. However, it is the change from baseline in eSOFA that is used to predict in-hospital mortality. We desired a score that can be calculated at a single time point, rather than evaluating changes from a baseline.

• 1. Limitations

This was a single-center retrospective study. In analyzing this retro- spective data set, we found that important measurements, especially re- lated to calculating the P/F Ratio, were time-stamped shortly after the admission order entry. For this reason, we used measurements entered up to one hour after the time of the admission order.

While typically an illness severity score’s stratum-specific mortality estimates are less reproducible in other populations, the absolute mor- tality risks associated with integer scores <14 in our cohort were quite similar to those of Raith et al. in their study of the standard SOFA score as a predictor of risk in septic ICU patients [19]. Nevertheless, the mortal- ities associated with the low-, intermediate-, and high-risk eccSOFA cat- egories may not be generalizable. For example, the overall mortality of our patients was 11.6%. In an ECC population with a substantially differ- ent overall mortality rate, the category-specific risk predictions may re- quire re-calibration. The original SOFA score does not contain an indicator of patient type, e.g. surgical vs. medical, and we did not add

one to eccSOFA. We would expect another ED with a higher proportion of medical ICU patients and higher in-hospital mortality to also have higher eccSOFA scores, but this requires separate validation.

During the data collection period for this study, we saw a large in- crease in ED boarding of ICU patients and initiated a nursing interven- tion described elsewhere [6].

This study primarily shows the discriminatory ability of eccSOFA and does not show if the eccSOFA score would change management or im- prove outcomes if presented to the treating emergency physician at the time of admission decision. We are planning an EHR-based clinical trial of presenting the eccSOFA score to all physicians when they write admission orders and choose a level of care. This will cover all admission orders, not just critical care admission orders. The main outcome will be in-hospital mortality, but we will also look at upgrades within 24 h of the initial admission order and ICU utilization, which this study does not currently assess.

1. Conclusion

As a predictor for in-hospital mortality, eccSOFA can be calculated based on variables that are available in the EHR at the time of the critical care admission order in the ED. It has discriminatory ability that is com- parable to other commonly used illness severity scores, and it can be used to adjust for severity of illness when assessing interventions that take place while patients are still boarding in the ED. Future studies should assess the calibration of our absolute risk predictions and vali- date our findings in other populations. Further study is needed to eval- uate its usefulness in improving ICU triage decisions when presented to admitting physicians.

Funding

This study was supported financially by the Stanford Nurse Alumnae Legacy Project Grant and the Stanford Emergency Critical care section.

Author contributions

Study concept and design (KN, JN, JGW, TM, MAK), acquisition of the data (JN, KN, MAK, MN), data analysis and interpretation (JN, TM, JGW, KN, MN, MAK), drafting of the manuscript (KN, JN, TM, JGW, MAK, MN), statistical expertise (KN, MN, JGW, MAK), acquisition of funding (JN, TM). All authors have critically reviewed the manuscript.

Declaration of Competing Interest

The authors of this manuscript have no conflicts of interest to dis- close (KN, MAK, JN, JGW, TM, MN, AJG).

Appendix I. Additional details on calculation of the eccSOFA score

PaO₂:

1. A PaO2 of <40 mmHg was excluded as they likely represented a venous specimen.
2. An SpO₂ of <75 was excluded from conversion into PaO₂ since this corresponded to a PaO2 of <40.
3. SpO₂‘s greater than 96 were excluded because PaO2 cannot be accurately converted [22]
4. The Severinghaus method was used to calculate an estimated PaO₂ from SpO₂:

0 0 1 11

1

(SpO2)⁶

Tsum = @0.385? log @ 1 -1AA + 3.32-

SpO2

72?SpO2 - 6

PaO₂ = e^Tsum. FiO₂:

1. “Room air” was given an FiO2 value of 0.21
2. Nasal canula readings of <0.5 LPM were not converted to FiO2.
3. O₂ LPM was converted to FiO2 [23] FiO₂ = 0.21 + (0.03 * O₂ L/min)
4. If the flow rate was <15 L/min, the maximum converted FiO₂ was 0.6.
5. If the flow rate was >=15 L/min then the maximum converted FiO₂ was 1.0 and this was classified as respiratory support (along with BiPAP and mechanical ventilation).
6. Systolic BP < 45 was ignored as a presumed data entry error.

Appendix II

Table A1

Counts of patients and deaths by eccSOFA score (used for ROC curve in Fig. 2)

							deaths
	eccSOFA Category		eccSOFA		Patients		n	%		Sensitivity		1 - Specificity
	High		14+		21		18	85.7%		4.0%		0.1%
			13		17		13	76.5%		6.8%		0.2%
			12		30		12	40.0%		9.5%		0.7%
			11		42		17	40.5%		13.2%		1.4%
			10		73		30	41.1%		19.9%		2.7%
			9		116		36	31.0%		27.8%		5.0%
			8		150		44	29.3%		37.5%		8.1%
	Intermediate		7		177		38	21.5%		45.9%		12.1%
			6		259		37	14.3%		54.1%		18.5%
			5		329		52	15.8%		65.6%		26.5%
			4		438		60	13.7%		78.8%		37.4%
	Low		3		478		39	8.2%		87.4%		50.1%
			2		544		30	5.5%		94.0%		65.0%
			1		608		15	2.5%		97.4%		82.1%
			0		630		12	1.9%		100.0%		100.0%
					3912		453

Table A2

Mortality Stratified by qSOFA

qSOFA	N	Died	%	95% Confidence Interval
0	861	26	3.0%	1.98% - 4.39%
1	1567	144	9.2%	7.80% - 10.73%
2	1178	202	17.1%	15.04% - 19.42%
3	306	81	26.5%	21.61% - 31.79%
Total	3912	453	11.6%	10.59% - 12.62%

References

1. ACEP. ACEP Policy Statement: Boarding of Admitted and Intensive Care Patients in the Emergency Department [updated June 2017]. Available from: https://www. acep.org/patient-care/policy-statements/boarding-of-admitted-and-intensive-care- patients-in-the-emergency-department/.
2. Mathews KS, Rodriguez SM, Nelson JE, Richardson LD. Triage and ongoing care for critically ill patients in the emergency department: results from a national survey of emergency physicians. West J Emerg Med. 2020;21(2):330-5.
3. Gunnerson KJ, Bassin BS, Havey RA, Haas NL, Sozener CB, Medlin Jr RP, et al. Associ- ation of an emergency department-based intensive care unit with survival and inpa- tient intensive care unit admissions. JAMA Netw Open. 2019;2(7):e197584.
4. Leibner E, Spiegel R, Hsu CH, Wright B, Bassin BS, Gunnerson K, et al. Anatomy of re- suscitative care unit: expanding the borders of traditional intensive care units. Emerg Med J. 2019;36(6):364-8.
5. Mitarai TWJ, Nudelman M, Nesbitt JL, Urdaneta AE, Gordon AJ, Niknam KR, et al. 114: impact of a novel emergency critical care program on in-hospital mortality and emergency department downgrades. Acad Emerg Med. 2020.
6. Nesbitt JLCG, Niknam K, Mitarai T, Wilson JG, Cinkowski C, Kohn MA. 92: emergency critical care nurses and management of critically ill patients boarded in the emer- gency department. Acad Emerg Med. 2020.
7. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related organ failure assessment) score to describe organ dysfunction/fail- ure. On behalf of the working group on Sepsis-related problems of the European So- ciety of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707-10.
8. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assess- ment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA. 2016;315(8):762-74.
9. Jones AE, Trzeciak S, Kline JA. The sequential organ failure assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med. 2009;37(5): 1649-54.
10. Newman TBKM. Evidence-based diagnosis : an introduction to clinical epidemiol- ogy. Second edition. Press CNYCU, editor; 2019.
11. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Bio- metrics. 1988;44(3):837-45.
12. Herring AA, Ginde AA, Fahimi J, Alter HJ, Maselli JH, Espinola JA, et al. Increasing crit- ical care admissions from U.S. emergency departments, 2001-2009. Crit Care Med. 2013;41(5):1197-204.
13. Higgins TL, Kramer AA, Nathanson BH, Copes W, Stark M, Teres D. Prospective vali- dation of the intensive care unit admission mortality probability model (MPM0-III). Crit Care Med. 2009;37(5):1619-23.
14. Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3-from evaluation of the patient to evaluation of the intensive care unit. Part 1: objectives, methods and cohort description. Intensive Care Med. 2005;31(10):1336-44.
15. Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP. Intensive Care Society’s acute physiology and chronic health evaluation (APACHE II) study in Brit- ain and Ireland: a prospective, multicenter, cohort study comparing two methods for predicting outcome for adult intensive care patients. Crit Care Med. 1994;22 (9):1392-401.
16. Salluh JI, Soares M. ICU severity of illness scores: APACHE, SAPS and MPM. Curr Opin

Crit Care. 2014;20(5):557-65.

1. Jones AE, Fitch MT, Kline JA. Operational performance of validated physiologic scor- ing systems for predicting in-hospital mortality among critically ill emergency de- partment patients. Crit Care Med. 2005;33(5):974-8.
2. Garcia-Gigorro R, Saez-de la Fuente I, Marin Mateos H, Andres-Esteban EM, Sanchez- Izquierdo JA, Montejo-Gonzalez JC. Utility of SOFA and Delta-SOFA scores for predicting outcome in critically ill patients from the emergency department. Eur J Emerg Med. 2018;25(6):387-93.
3. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA. 2017;317(3):290-300.
4. Lo RSL, Leung LY, Brabrand M, Yeung CY, Chan SY, Lam CCY, et al. qSOFA is a Poor Predictor of Short-Term Mortality in All Patients: A Systematic Review of 410,000 Patients. J Clin Med. 2019;8(1).
5. Rhee C, Zhang Z, Kadri SS, Murphy DJ, Martin GS, Overton E, et al. Sepsis surveillance using adult Sepsis events simplified eSOFA criteria versus Sepsis-3 sequential organ failure assessment criteria. Crit Care Med. 2019;47(3):307-14.
6. Severinghaus J. Simple, accurate equations for human blood O2 dissociation compu- tations. Journal of Applied Physiology: Respiratory, Environmental and Exercise Physiology. 1979;46(3):599-602 [0161-7567].
7. Vincent JL. Le manuel de re animation, soins intensifs et me decined’urgence. France Quatrie’me e dition Paris, France: Springer; 2013 67. 2013.