a b s t r a c t

Introduction: Chest pain is one of the most common reasons for contacting the emergency medical services (EMS). About 15% of these chest pain patients have a high-risk condition, while many of them have a low-risk condition with no need for acute hospital care. It is challenging to at an early stage distinguish whether patients have a low- or high-risk condition. The objective of this study has been to develop prediction models for optimising the identification of patients with low- respectively high-risk conditions in acute chest pain early in the EMS work flow.

Methods: This prospective observational cohort study included 2578 EMS missions concerning patients who contacted the EMS in a Swedish region due to chest pain in 2018. All the patients were assessed as having a low-, intermediate- or high-risk condition, i.e. occurrence of a time-sensitive diagnosis at discharge from hospital. Multivariate regression analyses using data on symptoms and symptom onset, clinical findings including ECG, previous medical history and Troponin T were carried out to develop models for identification of patients with low- respectively high-risk conditions. Developed models where then tested hold-out data set for internal vali- dation and assessing their accuracy.

Results: Prediction models for risk-stratification based on variables mutual for both low- and high-risk prediction were developed. The variables included were: age, sex, previous medical history of kidney disease, atrial fibrilla- tion or heart failure, Troponin T, ST-depression on ECG, paleness, pain debut during activity, constant pain, pain in right arm and pressuring pain quality. The high-risk model had an area under the receiving operating character- istic curve of 0.85 and the corresponding figure for the low-risk model was 0.78.

Conclusions: Models based on readily available information in the EMS setting can identify high- and low-risk conditions with acceptable accuracy. A Clinical decision support tool based on developed models may provide valuable clinical guidance and facilitate referral to less resource-intensive venues.

(C) 2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://

creativecommons.org/licenses/by/4.0/).

Abbreviations: ACS, Acute Coronary Syndrome; AMI, Acute Myocardial Infarction; AUROC, Area Under the Receiver Operating Curve; ALS, Advanced Life Support; BLS, Basic Life Support; CCU, Cardiac Coronary Unit; ED, Emergency Department; EMS, Emergency Medical Services; MACE, Major adverse cardiac events; NSTEMI, Non-ST- Elevation Myocardial Infarction; PCI, Percutaneous Coronary Intervention; STEMI, ST- Elevation Myocardial Infarction; Tnt, Troponin T.

? All authors take responsibility for all aspects of the reliability and freedom from bias in

the data presented and the discussion of their interpretation.

* Corresponding author at: Ambulanssjukvarden Kungsbacka Varlabergsvagen, 29 434 39 Kungsbacka, Sweden.

E-mail addresses: [email protected] (K. Wibring), [email protected] (M. Lingman), [email protected] (J. Herlitz), [email protected] (A. Ashfaq), [email protected] (A. Bang).

1. Introduction

Chest pain is one of the most common reasons for contacting the emergency medical services (EMS) and includes 10-15% of all EMS mis- sions [1,2]. About 15% of these chest pain patients have a high-risk con- dition, while many of them have no need for acute hospital care [3,4].

Safe Prehospital identification of patients with suspected ST- elevation myocardial infarction (STEMI) is well defined and the use of percutaneous coronary intervention (PCI) fast-tracks is well established [5]. In patients with suspected non-ST-elevation myocardial infarction , prehospital identification is more problematic. This is also true for other high-risk conditions such as pulmonary embolism and aortic dissection presenting with chest pain. All these conditions are

https://doi.org/10.1016/j.ajem.2021.09.079

0735-6757/(C) 2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

associated with high mortality rates. Early identification, already in the prehospital setting, with prompt and adequate care being rendered already in the prehospital setting, may be one way to improve the prognosis for patients suffering from such high-risk conditions.

Prehospital identification of patients with chest pain caused by a low-risk condition would make it possible for the EMS personnel to refer these patients to less resource-intensive venues such as a primary healthcare centre or to let them stay at home. In this way, patients could avoid unnecessary Hospital visits. At the same time this would reduce emergency department (ED) crowding and lower Healthcare costs [6]. Alternatively, if these patients were referred to alternative modes of transportation, scarce EMS resources could be made available for new assignments.

Previous research on prehospital risk stratification of patients with chest pain is sparse [7]. However, there are ongoing research projects investigating the use of hospital developed Prediction tools in the prehospital setting [8-12]. Compared with ED patients, prehospital emergency patients are older, have more comorbidity, differ in symptomology [13-16] and more commonly have a high-risk condition [15,17]. These aspects may affect pre-test probabilities hence diminishing the validity of the ED models when applied in the prehospital setting. This is also endorsed by previous research which states that several variables in these prediction tools have little or no prognostic value in the prehospital setting but that several variables that are not included actually are of prognostic importance when applied before arrival in hospital [4,18].

It is reasonable to assume that accuracy can be improved by using a tool entirely developed based on prehospital data. Furthermore, these ongoing research projects are based on cohorts of patients in whom an acute coronary syndrome (ACS) specifically is suspected by the EMS personnel [8-11]. Also, these tools predict Major adverse cardiac events and acute myocardial infarction (AMI) respectively and do not take other equally important high-risk conditions into ac- count. Not being able to rule out these other conditions limits the tools’ clinical value in the real-world prehospital setting, also in terms of safe low-risk prediction.

The objective of this study is to develop prediction models for optimising identification of patients with low- and high-risk conditions respectively in acute chest pain early in the EMS workflow.

1. Methods

This report is part of a larger research project, the BRIAN research pro- gramme (BRostsmarta I AmbulaNs (Swedish), EMS Chest pain (English)), on improved prehospital risk assessment of patients with chest pain. The study design and the study cohort have been described in detail else- where [18]. In this report, focus is on the development of prehospital pre- diction models identifying patients with low- and high-risk conditions when the reason for calling for an ambulance was acute chest pain.

• 1. Study population

In all, 3121 EMS missions were carried out in 2018 including pa- tients, >=18 years old, with a chief complaint of chest pain in the county catchment area with 329,000 inhabitants localised in the southwest of Sweden. All these missions were eligible for inclusion. After excluding patients declining to participate and patients who were lost to follow- up, 2917 EMS missions remained and were prospectively and consecu- tively included in the study.

EMS missions regarding patients with ST-elevation on ECG or red (strongly deviating) vital signs according to National Early Warning Score 2 (NEWS 2) [19] were thereafter excluded (since clear and valid pathways already exist for these groups). When excluding these mis- sions from the 2917 EMS missions originally included, 2578 remained for prediction model development. No other criteria for study exclusion were applied.

• 1. Data collection

Each patient was tracked throughout the entire acute healthcare chain, from EMS mission to hospital discharge. Data on symptoms were retrieved using a questionnaire filled in by the EMS personnel dur- ing the EMS mission along with the patients’ EMS medical report (the questionnaire is found in previous report) [18]. Data on vital signs and ECG were retrieved from the EMS digital medical report. The ECGs were interpreted using a pre-set template [18] and the investigators were blinded to all patient and outcome data except age and sex. Prehospital Tnt, previous medical history and diagnosis on hospital discharge was re- trieved from the hospital medical record. A detailed overview on the ad- junctions made for previous medical history classification is provided in Supplementary material 1 The questionnaire used for symptom data col- lection can be seen in previous report along with the ECG interpretation template [18]. Data collection did not affect patient care.

During the EMS mission, a blood sample was obtained and brought to the ED. This blood sample was analysed in hospital for high-sensitive Tnt using Roche Cobas e 601, detecting values >=5 ng/l. Retrieved high- sensitive Tnt-values were converted into semi-quantitate data with the following intervals <40, 40-100, 101-1000 and >1000 ng/l. In this way we could stipulate semi-quantitative data on Tnt as it would have been presented if Roche’s device for bedside Tnt analysis, Cobas h 232, had been used by the EMS [4]. Cobas h 232 has been validated previously and showed excellent concordance with reference high sensitivity tropo- nin laboratory method [20].

Data on time since pain debut was collected as a continuous variable. This was dichotomised into more or less than three hours since pain debut. The three hours cut-off was determined based on the release pat- tern of Tnt and since three hours is used as cut off to determine if appro- priate to apply the 0 h/1 h hour rule-out and rule-in algorithm for patients with suspected NSTEMI in the ED as advocated by the European Society of Cardiology [21].

Pain intensity was measured using the Numeric rating scale ranging from 0 to 10. This was then divided into two different dichotomous var- iables. One representing those with no pain on EMS arrival, NRS 0, and one representing those with extremely Intense pain, NRS 9-10 [22]. Data collection and variable categorisation was performed without knowledge of outcome data.

• 1. Endpoint

The primary endpoint was a risk classification group, in terms of a low- or high-risk condition. All patients were classified as having either a low-, intermediate- or high-risk condition as the cause of their chest pain. The adjudication was based on the final diagnosis on discharge from hospital according to the physician in charge. A high-risk condition was defined as a time-sensitive condition with high risk of death and in need of immediate hospital care, for example AMI, pulmonary embo- lism or aortic dissection. An intermediate-risk condition was defined as a final diagnosis probably in need of hospital care, but for which time was not a critical factor, for example cancer and atrial fibrillation. A low-risk condition was defined as a final diagnosis with no medical need for prompt hospital care, thus suitable for non-conveyance to hos- pital, for example unspecified chest pain, gastritis and anxiety disorders. A full overview of how the different diagnoses were classified along with a more detailed description of how this risk classification was car- ried out can be found in a previous publication [18]. Data on included variables and all other patient data than diagnosis on hospital discharge were blinded when the risk classification group was ascertained.

• 1. Data imputation

Missing data, predominantly regarding certain symptoms and Tnt, were handled by imputation to improve the prerequisites for multivar- iate analysis. In this study, we leveraged the MissForest algorithm that

iteratively builds Random forest models to impute missing instances in the variables of interest [23].

• 1. Statistical analysis

After imputation, the complete data set of 2578 EMS missions were randomly split into a training and a hold-out test set (80/20 ratio). The training set of approximately 2000 EMS missions would enable a detec- tion of a relative difference of 42% and an absolute difference of 6% for a factor present at 20% of all observations. For these calculations, 80% power, significance level of 5% (two sided) and a 15% incidence rate of high-risk conditions were applied.

Prediction model development analyses were executed on the train- ing set. This was done in two steps. First, two multivariate logistic re- gression analyses were carried out with all available variables as candidates for model entry. One analysis using high- and one using low-risk condition as endpoint, generating a full high- respectively a full low-risk model. In the second step, those variables common for both the full high- and low-risk models were included in new analyses generating limited high- and low-risk models. This second step was car- ried out to generate models with fewer and joint variables. Thereby, en- abling both low- and high-risk prediction using the same variables. The inclusion of fewer variables also ease clinical utilisation [24].

All analyses were carried out using forward logistic regression with p-value threshold for model entry set to <0.05 and removal to <0.1. Variables, with fewer than 50 observations in the training set were ex- cluded to ensure the robustness of the analyses.

Generated models were then tested on the hold-out test data to as- sess their predictive accuracy in terms of sensitivity, specificity, negative and positive predictive value along with area under the receiver operat- ing curve (AUROC). High-risk models were tested with 50% predicted endpoint probability as cut-off for assigning the risk classification group. The corresponding figure for low-risk models were 90% in order to reduce the risk of wrongly classify patients with high-risk conditions as low-risk. Imputation was done using Python (Scikit- learn package). All other analyses were carried out using IBM SPSS Statistics 27.

The TRIPOD-checklist (Supplementary material 2) for multivariable prediction models development studies [25] has been applied when writing this report.

1. Results

• Age
• Sex
• Previous history of:
  • Heart failure
  • Kidney disease
  • Atrial fibrillation/flutter
• Tnt
• ST-depression on ECG
• Paleness
• Pain debut during activity
• Constant pain
• Pain in right arm’
• Pressuring pain quality

Tnt and ST-depression on ECG were the strongest predictors for both low- and high-risk condition prediction (Table 2). Previous medical his- tory of atrial fibrillation/flutter was not predictive of having a low-risk condition when including only those variables mutual for the full high- and low-risk models and was not in the limited low-risk model (Table 2).

The full high- and low risk models shows that numerous variables are of predictive value for prehospital risk stratification of patients with chest pain (Supplementary material 3). Models predicting high- risk conditions have a larger AUROC compared with counterparts predicting low-risk conditions (Table 3).

The AUROC for the limited high-risk model was 0.85 (CI 0.76-0.91) and corresponding figure for the limited low-risk model was 0.78 (CI

Table 2

Limited high- and low-risk prediction models.

p-Value? Odds Confidence

ratio interval 95%

			Lower	Upper
Limited high-risk model
Age	<0.001	1.03	1.02	1.04
Male sex	0.003	1.58	1.12	2.13
Previous history of heart failure	0.007	0.54	0.34	0.84
Previous history of kidney disease	0.001	0.42	0.25	0.71
Previous history of atrial fibrillation/flutter Prehospital Tnt <40 ng/l (reference) Prehospital Tnt 40-100 ng/l	<0.001 <0.001 <0.001	0.31 5.68	0.21 3.82	0.48 8.44
Prehospital Tnt 101-1000 ng/l Prehospital Tnt >1000 ng/l	<0.001 <0.001	18.13 41.21	10.80 5.63	30.46 301.39

Of the 2578 EMS missions included, 12.5% concerned patients with a	ECG - ST-depression	<0.001	3.81	2.24	6.49
high-risk condition (8.4% with acute myocardial infarction (AMI)). The	Pale	<0.001	2.02	1.36	3.00
corresponding figures for intermediate-risk and low-risk conditions	Pain debut during activity	<0.001	2.31	1.63	3.27

were 13.5% and 74% respectively. Median age was 72 years old (Q25- Q75, 58-82), and 52% were women. In the hold-out test set of 505 EMS missions the median age was 71 years old (Q25-Q75 57-82), and 46% were women. The rate of high-risk conditions were somewhat lower while low-risk conditions were slightly more common (Table 1).

Constant pain	<0.001	1.85	1.35	2.53
Pain in right arm	<0.001	3.02	1.70	5.36
Pressuring pain	0.010	1.55	1.11	2.16
Constant	<0.001	0.01
Limited low-risk model Age	<0.001	0.98	0.97	0.98

The full high- and low-risk models shared 12 different variables	Male sex	0.005	0.73	0.58	0.91
(Supplementary material 3):	Previous history of heart failure	0.004	1.57	1.16	2.13
	Previous history of kidney disease	0.001	1.91	1.31	2.78

Table 1

Description of data sets.

Complete set, n (%)

Training set, n (%)

Test set, n (%)

Prehospital Tnt <40 ng/l (reference) <0.001

Prehospital Tnt 40-100 ng/l <0.001 0.25 0.18 0.35

Prehospital Tnt 101-1000 ng/l <0.001 0.09 0.05 0.15

Prehospital Tnt >1000 ng/l	0.029	0.11	0.02	0.80
ECG - ST-depression	<0.001	0.24	0.15	0.38

Paleness <0.001 0.41 0.30 0.56

All 2578 (100) 2073 (80) 505 (20)

Male sex 1243 (48) 1011 (49) 232 (46)

Median age (Q25-Q75) 72 (58-82) 72 (58-82) 71 (57-82)

High-risk condition 323 (13) 272 (13) 51 (10)

Intermediate-risk condition 348 (14) 280 (14) 68 (14)

Low-risk condition 1907 (74) 1521 (73) 386 (76)

Pain debut during activity 0.010 0.69 0.52 0.92

Constant pain 0.001 0.67 0.54 0.84

Pain in right arm 0.001 0.44 0.28 0.71

Pressuring pain <0.001 0.64 0.51 0.82

Constant <0.001 52.56

* Stepwise forward logistic regression, entry <0.05, removal <0.1.

Table 3

Overview of prediction model accuracy.

Name of prediction model	Number of variables in model	Cut-offa	Sensitivityb, %	Specificityc, %	Positive predictive valued,%	Negative predictive valuee,%	AUROC (CI 95%)	High-risk patients classified as low-risk, n (%)
Full high-risk model	19	0.5	35	98	64	93	0.84 (0.78-0.91)	\\
Limited high-risk model	12	0.5	35	99	75	93	0.85 (0.76-0.91)	\\
Full low-risk model	20	0.9	29	95	95	29	0.79 (0.75-0.84)	3 (3)
Limited low-risk model	11	0.9	13	98	94	26	0.78 (0.72-0.83)	1 (2)

^a The endpoint probability that must be reached if the prediction model should assign the risk classification group.

^b The proportion of patients with a condition that is correctly identified as having the condition.

^c The proportion of patients without a condition that is correctly identified as not having the condition.

^d The proportion of true positives among all patients with a positive prediction.

^e The proportion of true negatives among all patients with a negative prediction.

0.72-0.83) (Table 2). The limited high-risk model had a sensitivity of 35% and specificity of 99%. For the limited low-risk model the sensitivity was 13% and the specificity 98%. The positive predictive value was 94%. The 6% with a false positive value when applying the limited low-risk model comprised of one patient (2%) with a high-risk condition and two (4%) patients with an intermediate condition (Table 3). Thereby, 6% of the patients classified as low-risk by the limited low-risk model actually had an intermediate- or high-risk condition. In all, 0.6% of the patients were incorrectly predicted as low-risk when applying the lim- ited low-risk model.

1. Discussion

In this report, using population-wide, prospectively collected data, we present new prediction models with excellent respectively accept- able accuracy [26] when identifying patients with high- or low-risk con- ditions respectively, to be used on unselected prehospital patients with chest pain. To the best of our knowledge this approach to risk assess- ment has not been reported before. Including only 12 readily available variables, predicting both low- and high-risk condition is still feasible. If entering data on these 12 variables into a digital application these models could provide the EMS personnel with information on the prob- ability of the patient having a low- or high-risk condition.

The models may be used to provide clinical guidance on whether a patient should be considered as low-risk and suitable for less resource-intensive alternatives such as stay at home, referral to a pri- mary healthcare centre, or self-transportation to the ED. Or, if assessed as high-risk, whether the patient is suitable for rapid transport and alerting the ED, by-passing hospitals without PCI or vascular surgery ca- pabilities, or direct admission to a cardiac care unit (CCU).

Depending on the kind of clinical decision-making the models are intended to support, they can be adjusted even further. For example, re- ferring patients to stay at home requires a higher positive predictive value for low-risk prediction than if instead advocating transport to the ED with an alternative Mode of transport than by a double staffed ambulance. If aiming at referring high-risk patients directly to the CCU or a hospital with PCI capabilities a higher positive predictive value is needed compared to if using the model to identify patients to assign a high-triage level on ED arrival. To validate the models’ clinical potential fully, it needs to be tested on new out-of-sample data, and preferably in a different clinical setting.

In this study, Tnt, along with ST-depression on ECG, is the most predic- tive variable for both low- and high-risk conditions. This highlights the importance of using Tnt already in the prehospital phase by implementing POC devices for prehospital Tnt testing. Or, at least, obtain Tnt prehospitally for later in-hospital analysis to speed up the diagnostic process at ED [27]. More sensitive prehospital Tnt analyses may be of value in order to improve prediction accuracy further. Such high- sensitive Tnt analysis is not available in the prehospital setting at present.

The accuracy of the limited low-risk model is deficient. This is shown by a AUROC of <0.8 and the 6% false positives, including one high-risk

patient assessed as low-risk. Reducing the number of high-risk patients wrongly classified as low-risk to less than 1% would require an increase of the probability cut-off from 90% to 92%, resulting in a reduced sensi- tivity by 3%. Thereby reducing the number of patients referred to less re- source intensive venues. This highlight how difficult it is to correctly assess patients with chest pain in the prehospital setting. Upcoming studies within this research programme will investigate whether using machine learning for model development will result in more ac- curate predictions.

However, perfect prediction models are not achievable. Instead, predic- tion models aim to improve clinical decision-making beyond the human level where zero tolerance is not practiced. Clinical decision-making is al- ways about weighing different risks against each other. In this case, transporting all patients to the ED by ambulance seems to be the safest al- ternative. However, this approach will inevitably result in unnecessary prehospital and hospital resource utilisation, resulting in ED and Hospital crowding and shortage of available EMS resources for those with greater needs. This is supported by previous research showing that ED physicians assessing patients with chest pain have good sensitivity, usually ruling in patients with an ACS correctly even though not perfectly. However, this comes at the cost of low rule-out rates and further unnecessary hospital in- vestigation is therefore common [28,29]. Another approach to increase pa- tient safety could be to refer patients predicted as having a low-risk condition to an alternative mode of ED transportation than ambulance. Thereby ensuring hospital follow up but reducing EMS workload.

The AUROC of the limited models is slightly better compared to HEART-score when it comes to Prehospital use [4,30,31]. In addition, our models have the advantage over HEART-score in that they can pre- dict both low- and high-risk conditions, whereas HEART-score only pre- dict the occurrence of MACE within 45 days. Since MACE are influenced by several confounding factors that depend on how patients are treated later on in the care process, even after being discharged from hospital, HEART-score is impractical for prehospital assessment. Most impor- tantly, several high-risk conditions are not included in MACE.

Previous medical history of atrial fibrillation/flutter, heart failure and kidney disease turned out as predictors lowering the risk of a high-risk condition. This finding is surprising. That atrial fibrillation/flutter lowers the risk may be explained by the patients’ chest pain/discomfort experi- enced being due to their previously diagnosed atrial fibrillation/flutter rather than a new high-risk event. It is also possible that their chronic anticoagulant medication in part protects them from the most common high-risk conditions myocardial infarction or pulmonary embolism. However, this contradicts previous research that states that atrial fibril- lation/flutter increases the long-term risk of myocardial infarction [32,33]. Maybe this can be explained by the short-term follow-up that was used in this study along with studying a specific cohort, i.e. prehospital patients with chest pain.

Regarding patients with previous medical history of kidney disease, this group includes patients with renal insufficiency (Supplementary material 1). Renal insufficiency may result in elevated Tnt levels due to impaired clearance. Heart failure may also be associated with Tnt

levels. [34]. Therefore, elevated Tnt levels in these patients does not nec- essarily imply myocardial damage [21]. This may explain why a previ- ous history of kidney disease or heart failure reduces the odds of a high-risk condition. This is strengthened by secondary analysis showing that a previous history of kidney disease or heart failure do not have any predictive value if excluding Tnt from the models.

• 1. Strengths and limitations

This study is strengthened by prospective data from an almost com- plete and Unselected population of prehospital patients with chest pain. Data available regarding demographics, previous medical history, symptoms, vital signs, ECG and Tnt enabled the development of predic- tion models using both previously known and previously unknown risk factors.

This study was conducted in a Swedish setting with a one-tiered EMS system where ambulances are staffed with nurses and the health care system is mainly tax funded through a single payer. Ethnicity is not as diverse as in some other parts of the world, with the majority being of northern European origin. These prerequisites may affect the external validity due to differences in History taking, who interprets the ECG, Tnt availability, Primary care accessibility, etc. compared to a non-Swedish or non-European setting. This should be accounted for when interpreting the results. However, Tnt point of care equipment is easy to use, also for non-nurses. Telemedicine could be used for ECG interpretation and history taking can be standardised using structured questions and forms. Thereby, it may be possible to manage developed prediction models also in other settings with non-nurse EMS personnel by creating the right conditions. Differences in healthcare models in terms of primary care availability, EMS organisation etc. will determine how to use the results of the prediction models. For example, prediction models could be used for a more refined triage, but still transporting the patients to the ED, if access to primary health care centres is limited. Or patients predicted as low-risk could be referred to a basic life support unit (BLS) instead of being transported by an advanced life support unit (ALS) in a two-tiered EMS organisation. Nevertheless, the accuracy of developed models needs to be tested further on other and differing cohorts before clinical implementation.

To improve clinical relevance, all EMS missions including patients

with ST-elevation on ECG or strongly deviating vital signs were ex- cluded. There are well-established, validated PCI fast-tracks for patients with ST-elevations, and strongly deviating vital signs are in themselves a strong indication for prompt hospital care. We therefore deemed little clinical value in further assessing risk in these patients in the prehospital setting. This should be considered when interpreting the results. Other considerations may also be of value depending on the context in which the prediction models are applied.

Basing analysis on imputed data is a limitation compared to using observed data. However, missing data are a common (and often) only partly avoidable problem in clinical research. Since most statistical tech- niques are susceptible to missing data, a standard approach is to limit the analysis to samples with complete variables. To overcome this lim- itation and reap the full benefit of the collected data, we used data im- putation. We used the MissForest random forest algorithm which successfully handle missing values and outperforms other imputation techniques particularly in datasets including different types of variables [23]. We can therefore assume that the use of imputed data has little negative impact on the results of the study as a better alternative to data selection and bias challenges.

Observer bias and variability refers to different observers making different assessments concerning how to classify data [35]. In this case, that various EMS personnel assess Patient symptoms, such as for example paleness, differently. Since the data of the study was collected by close to 200 EMS personnel, with different experiences and educa- tion, it is quite certain that observation bias is at hand. This reflects a problem existing also in real-life EMS care that will be present also if

applying developed prediction models in clinical care. This is difficult to compensate for, and maybe not appropriate if wanting to provide re- sults valid in real-world clinical care.

Tnt, along with ST-depression on ECG, were the strongest predic- tors for both low- and high-conditions. This was expected since AMI account for most of the high-risk conditions and both Tnt and ST-depression on ECG are part of the definition of AMI [21]. It is problematic to use predictors that also partially define the endpoint as this will likely result in an overestimation of the association be- tween predictor and outcome [36]. However, excluding Tnt and ECG from a prediction model to be applied on patients with acute chest pain was not deemed appropriate, but one should be aware that this may result in an optimistic model accuracy. One should also consider that models developed in this study are not to be used for diagnostics but to improve prehospital risk stratification.

Optimism and overfitting resulting in prediction model performing

better in the cohort it was developed from, than in another cohort, are a known issues in the development of prediction models [24] In this study, the internal validation, using and randomised hold-out sample for model testing, is applied to offset this. However, this approach has methodological weaknesses since the training and validation set do not differ in time or setting but only by chance [24]. Therefore, further validation on another independent cohort is needed to be able to evalu- ate the general accuracy of developed models.

1. Conclusions

Based on readily available information in the EMS setting, models identifying high- and low-risk conditions can provide valuable clinical guidance on referral to less resource-intensive venues for example. However, it cannot yet serve as an automated triage system. Therefore, clinical judgement by the EMS personnel remains the golden prediction standard but could benefit from being informed by risk evaluation tools. These tools can probably also be improved, specifically by using ma- chine learning models.

The use of Tnt in the prehospital setting could be the cornerstone for the risk classification of patients with acute chest pain. Prehospital Tnt could also speed up hospital diagnostics by providing a reference value obtained early on, in comparison with for example Tnt obtained on ED arrival.

Ethics approval and consent to participate

The study was approved by the Regional Ethical Review Board in Lund (Dno 2017/212). An opt-out procedure where all patients were given the opportunity to withdraw their participation was applied.

Consent for publication

Not required.

Funding

This study has been funded by the Department of Ambulance and Prehospital Care, Region Halland, and the Scientific Council of Region Halland (HALLAND-209901). The funding bodies had no role in the design, conduct, interpretation or writing of the report on this research.

Authors’ contributions

KW, ML, JH, and AB designed the study and planned the data collec- tion. Data analysis was carried out by KW and AA. KW, ML, JH, AA and AB contributed to writing the manuscript. All authors read and approved the final manuscript.

Declaration of Competing Interest

The authors declare that they have no competing interests.

Acknowledgements

We want to thank the following persons:

• Anders Holmen and Ulf Stromberg for statistical support.
• Margaret Myers for language editing.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi. org/10.1016/j.ajem.2021.09.079.

References

1. Clawson J, Olola C, Heward A, Patterson B, Scott G. The medical priority dispatch Sys- tem’s ability to predict Cardiac arrest outcomes and high acuity pre-hospital alerts in chest pain patients presenting to 9-9-9. Resuscitation. 2008;78(3):298-306.
2. Pedersen CK, Stengaard C, Friesgaard K, Dodt KK, Sondergaard HM, Terkelsen CJ, et al. Chest pain in the ambulance; prevalence, causes and outcome - a retrospective cohort study. Scand J Trauma Resusc Emerg Med. 2019;27(1):84.
3. Rawshani A, Larsson A, Gelang C, Lindqvist J, Gellerstedt M, Bang A, et al. Character- istics and outcome among patients who dial for the EMS due to chest pain. Int J Cardiol. 2014;176(3):859-65.
4. van Dongen DN, Tolsma RT, Fokkert MJ, Badings EA, van der Sluis A, Slingerland RJ, et al. Pre-hospital risk assessment in suspected non-ST-elevation acute coronary syndrome: A prospective observational study. Eur Heart J Acute Cardiovasc Care. 2020;9(1_suppl):5-12.
5. Hagiwara MA, Bremer A, Claesson A, Axelsson C, Norberg G, Herlitz J. The impact of direct admission to a catheterisation lab/CCU in patients with ST-elevation myocar- dial infarction on the delay to reperfusion and early risk of death: results of a sys- tematic review including meta-analysis. Scandinavian J Trauma Resuscit Emerg Med. 2014;22.
6. van Dongen DN, Ottervanger JP, Tolsma R, Fokkert M, Van Der Sluis A, Van’t Hof AWJ, et al. In-hospital healthcare utilization, outcomes, and costs in pre-hospital- adjudicated low-risk chest-pain patients. Appl Health Econ Health Policy. 2019;17 (6):875-82.
7. Wibring K, Herlitz J, Christensson L, Lingman M, Bang A. Prehospital factors associ- ated with an acute life-threatening condition in non-traumatic chest pain patients

- a systematic review. Int J Cardiol. 2016;219:373-9.

1. Stopyra JP, Harper WS, Higgins TJ, Prokesova JV, Winslow JE, Nelson RD, et al. Prehospital modified HEART score predictive of 30-day adverse cardiac events. Prehosp Disaster Med. 2018;33(1):58-62.
2. Ishak M, Ali D, Fokkert MJ, Slingerland RJ, Dikkeschei B, Tolsma RT, et al. Fast assess- ment and management of chest pain without ST-elevation in the pre-hospital gate- way: rationale and design. Eur Heart J Acute Cardiovasc Care. 2015;4(2):129-36.
3. Aarts GWA, Camaro C, van Geuns RJ, Cramer E, van Kimmenade RRJ, Damman P, et al. Acute rule-out of non-ST-segment elevation acute coronary syndrome in the (pre)hospital setting by HEART score assessment and a single point-of-care tropo- nin: rationale and design of the ARTICA randomised trial. BMJ Open. 2020;10(2): e034403.
4. Alghamdi A, Cook E, Carlton E, Siriwardena A, Hann M, Thompson A, et al. PRe- hospital evaluation of sensitive TrOponin (PRESTO) study: multicentre prospective diagnostic accuracy study protocol. BMJ Open. 2019;9(10):e032834.
5. Backus BE, Tolsma RT, Boogers MJ. The new era of chest pain evaluation in the Netherlands. Eur J Emerg Med. 2020;27(4):243-4.
6. Lavery T, Greenslade JH, Parsonage WA, Hawkins T, Dalton E, Hammett C, et al. Fac- tors influencing choice of pre-hospital transportation of patients with potential acute coronary syndrome: an observational study. Emerg Med Australas. 2017;29 (2):210-6.
7. Boothroyd LJ, Lambert LJ, Segal E, Ross D, Kouz S, Maire S, et al. Comparison of out- comes of ambulance users and nonusers in ST elevation myocardial infarction. Am J Cardiol. 2014;114(9):1289-94.
8. Thang ND, Karlson BW, Bergman B, Santos M, Karlsson T, Bengtson A, et al. Charac- teristics of and outcome for patients with chest pain in relation to transport by the emergency medical services in a 20-year perspective. Am J Emerg Med. 2012;30 (9):1788-95.
9. Thuresson M, Jarlov MB, Lindahl B, Svensson L, Zedigh C, Herlitz J. Factors that influ- ence the use of ambulance in acute coronary syndrome. Am Heart J. 2008;156(1): 170-6.
10. Charpentier S, Beaune S, Joly LM, Khoury A, Duchateau FX, Briot R, et al. Manage- ment of chest pain in the French emergency healthcare system: the prospective ob- servational EPIDOULTHO study. Eur J Emerg Med. 2018;25(6):404-10.
11. Wibring K, Lingman M, Herlitz J, Amin S, Bang A. Prehospital stratification in acute chest pain patient into high risk and low risk by emergency medical service: a pro- spective cohort study. BMJ Open. 2021;11(4):e044938.
12. Royal_College_of_Physicians: National Early Warning Score 2. Standardising the assessment of acute-illness severity in the NHS. In. London; 2017.
13. van Dongen DN, Tolsma RT, Fokkert MJ, Badings EA, van der Sluis A, Slingerland RJ, et al. Pre-hospital risk assessment in suspected non-ST-elevation acute coronary syndrome: a prospective observational study. Eur Heart J Acute Cardiovasc Care. 2020;9(1_suppl):5-12. https://doi.org/10.1177/2048872618813846 Epub 2018 Nov 23.
14. Collet JP, Thiele H, Barbato E, Barthelemy O, Bauersachs J, Bhatt DL, et al. ESC guide- lines for the management of acute coronary syndromes in patients presenting with- out persistent ST-segment elevation. Eur Heart J. 2020;2020.
15. Hawker GA, Mian S, Kendzerska T, French M. Measures of adult pain: visual analog scale for pain (VAS pain), numeric rating scale for pain (NRS pain), McGill pain ques- tionnaire (MPQ), short-form McGill pain questionnaire (SF-MPQ), chronic pain grade scale (CPGS), short Form-36 bodily pain scale (SF-36 BPS), and measure of in- termittent and constant osteoarthritis pain (ICOAP). Arthritis Care Res. 2011;63 (Suppl. 11):S240-52.
16. Stekhoven DJ, Buhlmann P. MissForest-non-parametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112-8.
17. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.
18. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivari- able prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.
19. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315-6.
20. Stopyra JP, Snavely AC, Ashburn NP, Nelson RD, McMurray EL, Hunt MR, et al. EMS blood collection from patients with acute chest pain reduces emergency department length of stay. Am J Emerg Med. 2021;47:248-52.
21. Oliver G, Reynard C, Morris N, Body R. Can emergency physician gestalt “rule in” or “rule out” acute coronary syndrome: validation in a multicenter prospective diag- nostic cohort study. Acad Emerg Med. 2020;27(1):24-30.
22. Pope JH, Aufderheide TP, Ruthazer R, Woolard RH, Feldman JA, Beshansky JR, et al. missed diagnoses of Acute cardiac ischemia in the emergency department. N Engl J Med. 2000;342(16):1163-70.
23. Cooper JG, Ferguson J, Donaldson LA, Black KMM, Livock KJ, Horrill JL, et al. The am- bulance cardiac chest pain evaluation in Scotland study (ACCESS): a prospective co- hort study. Ann Emerg Med. 2021;77(6):575-88. https://doi.org/10.1016/j. annemergmed.2021.01.012 Epub 2021 Apr 27.
24. van Dongen DN, Fokkert MJ, Tolsma RT, van der Sluis A, Slingerland RJ, Badings EA, et al. Accuracy of pre-hospital HEART score risk classification using point of care ver- sus high sensitive troponin in suspected NSTE-ACS. Am J Emerg Med. 2020;38(8): 1616-20.
25. Soliman EZ, Lopez F, O’Neal WT, Chen LY, Bengtson L, Zhang ZM, et al. Atrial fibrilla- tion and risk of ST-segment-elevation versus non-ST-segment-elevation myocardial infarction: the atherosclerosis risk in communities (ARIC) study. Circulation. 2015; 131(21):1843-50.
26. Violi F, Soliman EZ, Pignatelli P, Pastori D. Atrial fibrillation and myocardial infarc- tion: a systematic review and appraisal of Pathophysiologic mechanisms. J Am Heart Assoc. 2016:5(5).
27. Tanindi A, Cemri M. Troponin elevation in conditions other than acute coronary syn- dromes. Vasc Health Risk Manag. 2011;7:597-603.
28. Nunan D, Bankhead C, Aronsson J. Catalogue of Bias. 2017 http://www.catalogofbias. org/biases/selection - bias/.
29. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess risk of Bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170(1):W1-W33.