Development and validation of a knowledge-driven risk calculator for critical illness in COVID-19 patients

Journal logoUnlabelled imageAmerican Journal of Emergency Medicine 39 (2021) 143-145

Contents lists available at ScienceDirect

American Journal of Emergency Medicine

journal homepage:

Development and validation of a knowledge-driven risk calculator for critical illness in COVID-19 patients

Amos Cahan, MD a,b,1,?, Tamar Gottesman, MD c,d,1, Michal Tzuchman Katz, MD a, Roee Masad, BSc a, Gal Azulay, BSc a, Dror Dicker, MD d,e, Aliza Zeidman, MD d,f, Evgeny Berkov, MD d,g,

Boaz Tadmor, MD i, Shaul Lev, MD d,h

a Kahun Medical Ltd, Tel Aviv, Israel

b Infectious Diseases Unit, Samson Assuta Ashdod University Hospital, Ashdod, Israel

c Department of Infectious Diseases and Infection Control Unit, Hasharon Campus, Rabin Medical Center, Petach Tikva, Israel

d The Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel

e Internal Medicine D, Hasharon Campus, Rabin Medical Center, Petach Tikva, Israel

f Internal Medicine B, Hasharon Campus, Rabin Medical Center, Petach Tikva, Israel

g Internal Medicine C, Hasharon Campus, Rabin Medical Center, Petach Tikva, Israel.

h General Intensive Care Unit, Hasharon Hospital, Rabin Medical Center, Petach Tikva, Israel

i Research Authority, Rabin Medical Center, Petach Tikva, Israel

a r t i c l e i n f o

Article history:

Received 8 July 2020

Received in revised form 31 August 2020 Accepted 20 September 2020

a b s t r a c t

Facing the novel coronavirus disease (COVID-19) pandemic, evidence to inform decision-making at all care levels is essential. Based on the results of a study by Petrilli et al., we have developed a calculator using patient data at admission to predict critical illness (intensive care, mechanical ventilation, hospice care, or death).

We report a retrospective validation of the calculator on 145 consecutive patients admitted with COVID-19 to a single hospital in Israel. Despite considerable differences between the original and validation study populations, of 18 patients with critical illness, 17 were correctly identified (sensitivity: 94.4%, 95% CI, 72.7%-99.9%; specific- ity: 81.9%, 95% CI, 74.1%-88.2%). Of 127 patients with non-critical illness, 104 were correctly identified.

Our results indicate that published knowledge can be reliably applied to assess patient risk, potentially reducing the cognitive burden on physicians, and helping policymakers better prepare for future needs.

(C) 2020


Facing the rapidly spreading novel coronavirus disease (COVID-19), evidence to inform decision-making at both the clinical and policy- making level is highly needed [1]. In an impressive work, Petrilli and coworkers [2] have recently reported a multivariable analysis of data collected on 2729 hospitalized patients with COVID-19 at an academic health system in New York City (NY), to predict critical illness (defined as a composite of care in the intensive care unit, use of mechanical ven- tilation, discharge to hospice, or death). Based on this analysis, we have developed a computed calculator for risk stratification of hospitalized COVID-19 patients. Since rates of severe disease and mortality vary widely by country [3], we aimed to validate the risk calculator on a pop- ulation of COVID-19 patients in Israel (IL).

* Corresponding author: Kahun Medical Ltd., Yigal Alon St 114, Israel.

E-mail address: (A. Cahan).

1 Equal contribution.


Risk calculator development

We used the odds ratios (OR’s) obtained by Petrilli et al. through multivariable regression to develop a risk calculator. This was done by directly applying OR’s of predictors as observed in the NY population to a multivariate linear model. Predictors are listed on Tables 1 and 2. These included demographics, past medical history, temperature and oxygen saturation on presentation,and selected labs (the first result of c-reactive protein, d-dimer, ferritin, procalcitonin, and troponin). In their analysis, Petrilli et al. found that the risk of developing critical ill- ness was considerably lower for patients admitted later during the study period. This observation was limited to the 6 weeks of the study period and may be the result of unique and local circumstances. Thus, we did not include the week of presentation as a feature in the risk cal- culator. Petrilli et al. provided OR’s for missing data on predictors, and these were used in our risk calculator. 0735-6757/(C) 2020

Table 1

Characteristics of hospitalized patients.

Characteristic NY population IL population p-Value

tool is freely available on the Web at Calculate/7.

2.3. Statistical analysis

Hospitalized, N = 2729

N (%) or median (IQR)

Hospitalized, N = 145

N (%) or median (IQR)

Analysis was done using R (version 3.6.3) [4]. Differences between the NY and IL populations were assessed using the one sample t-test and one sample Wilcoxon signed rank test. Confidence intervals for the calculator performance measures were computed using the “exact” Clopper-Pearson method.

The study was approved and informed consent waived by the Rabin Medical Center Institutional Review Board (ref. 0339-20-RMC).


63 (51-74)

62(46, 71)


Male sex

1672 (61.3)



White race

1089 (39.9)



Current smoker

141 (5.2)




1081 (39.6)



Any chronic condition?

2176 (79.7)




1157 (42.4)




1693 (62.0)



Coronary artery disease

602 (22.1)




Heart failure

349 (12.8)



A total of 145 patients were admitted to HaSharon Hospital between


950 (34.7)



March 9, 2020 and May 13, 2020. At the time of analysis,137 had been

Asthma or chronic obstructive

453 (16.5)



discharged, none were still hospitalized, and 8 had deceased. Table 1

pulmonary disorder

Chronic kidney disease

580 (21.3)




292 (10.7)



Temperature at presentation,

37.4 (36.9-38.2)

38.5(38.1, 39.0)


degrees Celsius

Temperature >=38 at presentation

846 (31.0)



Oxygen saturation <88%

422 (15.5)



Oxygen saturation at presentation

94 (90-96)

97(95, 99)



* Not used in the model.

?? Ambient air.

Table 2

Values of first laboratory tests used in the risk calculator.?






1.0 (0.8, 1.5)



0.88 (0.75, 1.05)

C-reactive protein


0.37 (0.09, 1.10)



602.0 (336, 1125.7)



388.1 (161.7, 689.5)



8.0 (6.0, 14.0)

* Procalcitonin testing is not performed at HaSharon hospital.

2.2. Validation process

We studied patients admitted to the Rabin Medical Center, HaSharon Campus, a teaching medical center at Petach Tikva, Israel. HaSharon Hospital was designated a coronavirus care center, to which COVID-19 patients residing in Central Israel were referred. Included were patients hospitalized starting from March 9, 2020 with confirmed Covid-19, defined as a positive result on real-time reverse transcriptase- polymerase-chain-reaction (RT-PCR) assay of nasopharyngeal or oro- pharyngeal swab specimens. This was done using a kit by Seegene (Songpa-gu, South Korea). At the time of data extraction, there were no patients still hospitalized who had not reached the composite out- come. Patients’ electronic health records were reviewed by two clini- cians (MTK and RM) and relevant demographic data, clinical findings and results of the first laboratory and imaging studies done during the admission were extracted. Based on their state at the time of data ex- traction, patients were determined to have critical illness (at least one of ICU admission, mechanical ventilation or death; no COVID-19 pa- tients were transferred to hospice care) or non-critical disease. Ex- tracted data was then loaded to the risk calculator and analysis of the relationship between predicted risk and actual outcome was performed. A threshold score for predicting critical illness was empirically selected based on actual patient scores such that optimized discrimination be- tween patients with critical and non-critical illness. The risk calculator

shows a comparison of the IL and NY populations baseline characteris- tics. The median age in the IL and NY populations was similar (62 and 63, respectively), and in both, the majority of patients were males (62.1% vs 61.3%). There were no patients of race other than white in the IL population. Fever and oxygen saturation at presentation were sig- nificantly higher, on average, in the IL population. Of note, procalcitonin levels were not available for any of the patients, as this test is not rou- tinely performed at HaSharon hospital. Values of other predictors used in the calculator are shown in Table 2. Results of serum troponin, ferritin and d-dimer from the first 48 h of admission were missing in 31 (21.3%), 30 (20.7%), and 40 (27.6%) of the patients, respectively. Missing data was observed more frequently in patients admitted early in the course of the study period, before protocols for routine testing were adopted, and in patients with milder disease.

Of the 18 patients with critical illness, 17 were correctly identified by the model when a threshold score of 2.2 points was used (sensitivity: 94.4%, 95% CI, 72.7% to 99.9%; specificity: 81.9%, 95% CI, 74.1% to

88.2%). Of the 127 patients with non-critical illness, 104 were correctly identified. The accuracy was 0.83 (95% CI: 0.76-0.89) and the C-statistic, or the area under the ROC (receiver operator characteristic) curve was 0.943.


We used the results of an analysis performed by Petrilli on a popula- tion of COVID-19 in NY to develop a risk calculator for critical illness. We report the application of this calculator to make predictions on the out- come of COVID-19 hospitalized patients in Israel. Of note, predictions were fairly accurate, despite considerable differences in baseline charac- teristics between the two populations.

Thousands of papers have been published to date on COVID-19. For clinicians, keeping current on medical literature is challenging in nor- mal times but even more demanding during the COVID-19 pandemic, when their abilities are stretched to the limit. Moreover, given the lim- ited capacity of humans (including physicians) to apply probabilistic reasoning [5], integration of published evidence probably remains mostly at the intuitive level in the minds of clinicians. This state of affairs calls for equipping clinicians with reliable tools to properly evaluate the abundant empirical knowledge, and properly weigh it against their own patients’ data. Many current EHR systems document the information re- quired by the calculator in a machine-readable format, allowing for risk assessment to be done automatically (without active physician involve- ment). Automatic extraction of information and risk calculation per- formed in the background as patients present to the hospital can reduce the time and effort required from physicians for risk assessment. It can also assure that policy makers are provided a complete view of predicted disease burden for continuous monitoring.

Public health measures, such as quarantine and shelter-in-place, are guided by the capacity of the healthcare system, with ICU beds and ven- tilator availability being the “rate limiting factor”. As severe disease often develops during the second week of illness, there is a reported 12 day average lag between illness onset and ICU admission or the de- velopment of acute respiratory distress syndrome (ARDS) [6]. In an ef- fort to avoid overwhelming the healthcare system capacity, further lifting of restrictions on social interactions is thus delayed until the ef- fects of policy changes can be measured. The turnaround time for policy-makers to get feedback on policy changes is therefore around two weeks. Our results provide reason to believe that future critical ill- ness can be reliably predicted at the time of admission.Such predictions may be used in triage to make sure that high risk patients remain where medical care is rapidly available, or to select patients for (investiga- tional) interventions. At the institutional and healthcare system level, predicting the future burden of critical illness could improve allocation of resources (e.g., personnel and supplies) to preempt shortages. At the State and National levels, such predictions could shorten the feed- back turnaround time, allowing policy makers to effectively flatten the epidemic curve and avoid breakdown of medical care, while minimizing restrictions on the workforce to curb the financial crisis.

Our work has several limitations. It is based on a single study, which, albeit large in scale, includes patients from a single metropolitan area, presenting during a relatively short period of time, in which the local healthcare system was heavily burdened by scores of severely ill pa- tients. The validation population was relatively small and taken from a single hospital. The paucity of critically ill patients in our cohort may contribute to the high area under the ROC curve. As the threshold score for predicting critical illness was empirically determined from the validation population, results may reflect over-fitting. Finally, our dataset lacked information on some laboratory tests which were in- cluded in the original analysis, however this could have weakened our results. Further research will be needed to validate our findings in other patient populations.


We believe that computer-aided risk assessment is a means to put research-derived knowledge to work in the clinical setting. As new re- search is published, other calculators could be developed. Based on local circumstances, a calculator using predictors from the most rele- vant study could be used. Moreover, several calculators could be applied in parallel, and calculator voting used to derive potentially more robust predictions. Integration with EHR systems can facilitate automatic data extraction, which could make the use of such tools more user friendly and practical.

Declaration of Competing Interest

AC, MZK, RM and GA are employed by Kahun Medical Ltd. All other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


  1. Lipsitch M, Swerdlow DL, Finelli L. Defining the epidemiology of covid-19 – studies needed. N Engl J Med. 2020;382:1194-6.
  2. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O’Donnell L, Chernyak Y, et al. Factors as- sociated with hospital admission and critical illness among 5279 people with corona- virus disease 2019 in New York City: prospective cohort study. BMJ. 2020;369: m1966.
  3. World Health Organization COVID-19 situation reports. emergencies/diseases/novel-coronavirus-2019/situation-reports.
  4. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2020 URL https://www.R-project. org/.
  5. Dolan JG, Bordley DR, Mushlin AI. An evaluation of clinicians’ subjective prior proba- bility estimates. Med Decis Making. 1986;6:216-23.
  6. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study – Lancet. journals/lancet/article/PIIS0140-6736(20)30566-3/fulltext.