Article, Urology

Internal validation of a scoring system to evaluate the probability of ureteral stones: The CHOKAI score

a b s t r a c t

Objective: The CHOKAI and STONE scores are prediction models for Ureteral stones. The aims of the present study were to evaluate the diagnostic performance, to examine the optimal cut-off value, and to compare the diagnostic performance of each model.

Methods: Patients who presented to our emergency department with renal colic were considered for this pro- spective study. We analyzed the Predictive performance of both STONE and CHOKAI scores at their optimal cut-off values, using receiver operating characteristic (ROC) curve and area under the curve (AUC), as well as sen- sitivity, specificity, positive likelihood ratio (LR+), and negative likelihood ratio (LR-) at the optimal cut-off value.

Results: Of the 96 patients who met the inclusion criteria, 79 were definitively diagnosed with ureteral stones. All patients were of Japanese descent. The AUC of the CHOKAI score was 0.971 at an optimal cut-off value of 6, show- ing a sensitivity of 0.911, specificity of 0.941, LR+ of 15.49, and LR- of 0.094. The AUC of the STONE score was 0.873 at an optimal cut-off value of 8, showing a sensitivity of 0.823, specificity of 0.824, LR+ of 4.662, and LR- of

0.215. The AUC of the CHOKAI score was significantly higher than that of the STONE score (p = 0.010). Of the 73 patients with a CHOKAI score of >=6, 98.6% had ureteral stones, and of the 68 patients with a STONE score of >=8, 95.6% had ureteral stones. Conclusions: The simplified CHOKAI score is a useful tool to screen for ureteral stones in patients with renal colic.

creativecommons.org/licenses/by-nc-nd/4.0/).

Introduction

Ureteral stones are one of the most frequently diagnosed Urologic diseases in the world [1]. The prevalence and the incidence of ureteral stones are increasing in many countries, including Japan, where the an- nual incidence increased three-fold from 1965 to 2005 [1,2]. We often used non-contrast helical computed tomography (CT) to diagnose ure- teral stones because of its high sensitivity and specificity [3]. The usage of diagnostic CT for patients with suspected ureteral stones increased about 10 times from 1996 to 2007 in the United States without an asso- ciated change in the proportion of diagnosis of ureteral stones,

* Corresponding author at: Department of Urology, Yamagata University Faculty of Medicine, 2-2-2 Iida-nishi, Yamagata, Yamagata 990-9585, Japan.

E-mail addresses: [email protected] (H. Fukuhara), [email protected] (O. Ichiyanagi), [email protected] (S. Midorikawa), [email protected] (H. Kakizaki), [email protected] (H. Kaneko).

identification of significant Alternative diagnoses, or hospital admission [4]. In addition, ureteral stones often recur; therefore, serial and cumu- lative radiation exposure and a consequent risk of cancer from diagnos- tic CT are of concern, especially in young patients who are more radiosensitive [5,6].

Moore et al. [7] retrospectively investigated patients with and with- out ureterolithiasis and developed the STONE score to reduce the need for performing diagnostic CT scans for ureteral stones. They described the STONE score as a clinical Prediction tool with which to assess the risk of ureteral stones and identify important alternative diagnoses in patients suspected of having of ureteral stones. The STONE score is cal- culated as a weighted sum of five categories (sex, duration of pain, race, nausea/vomiting, and occult blood in urine), yielding 0-13 points. In a prospective validation cohort, a high score group of 10-13 points was associated with ureteral stones in 88.6% of cases and with major alterna- tive diseases in 1.6% in a prospective validation cohort, respectively [7]. Schoenfeld et al. [8] reported that the STONE score was validated in a retrospective cohort of patients aged 15-50 years who were younger

http://dx.doi.org/10.1016/j.ajem.2017.06.023

than those in the original article [7]. In contrast, Wang et al. pointed out that while the STONE score can successfully predict ureteral stones with a modest specificity, it lacks sufficient specificity to allow clinicians to defer a CT scan for a suspected ureteral stone [9]. It is unclear whether the STONE score would function properly in a different cohort of pa- tients with different medical, racial, climatic, or geographical back- grounds, especially in Japan where most inhabitants are Asian.

We recently created a prediction model for Japanese patients with ureteral stones, using a combination of seven clinical factors (age of b 60 years, male sex, duration of pain of b 6 h, nausea or vomiting, hydronephrosis, hematuria, and history of ureteral stones) [10,11]. This multivariable diagnostic model using each intercept and coefficient is a useful screening tool; however, it is slightly more complicated than the STONE score.

The aims of the present study were to 1) evaluate the diagnostic per- formance of a modified model that is a simplified model of the original one, 2) to examine the diagnostic performance at the optimal cut-off value of the STONE score in Japan and 3) to test our hypothesis that the diagnostic performance of our simplified model would outperform the STONE score by comparing the receiver operating characteristic curves and the area under the curve (AUC) of each model.

Methods
1. Study design, ethics statement and constructing scoring system

Our institutional review board approved the present study, and all enrolled patients provided written informed consent (approval no. 27-1-2).

We planned this internal prospective observational study to validate the practical applicability of the seven predictive factors (age of b 60 years, male sex, duration of pain of b 6 h, nausea or vomiting, hydronephrosis, hematuria, and prior history of ureteral stones) which were found in the former papers [10,11].

To establish a practical scoring model, each coefficient reported in the previous study [11] was rounded to the nearest integer, and each predictor’s reference was assigned a score of 0 as follows: age (b 60 years = 1 point, >= 60 years = 0 points), sex (male = 1 point, fe- male = 0 points), nausea or vomiting (presence = 1 point, absence = 0 points), medical history of ureteral stones (yes = 1 point, no = 0 points), short pain duration of b 6 h (yes = 2 points, no = 0 points), oc- cult blood in urine (presence = 3 points, absence = 0 points), and hydronephrosis on ultrasound examination (US) (presence = 4 points, absence = 0 points). The total score was 0 to 13; this score was termed the CHOKAI score (Table 1).

Table 1

CHOKAI score, factors, and categories.

Factor	Points
Distension of kidney capsule (nausea or vomiting)	Yes	1
	No	0
Hydronephrosis	Yes	4
	No	0
Occult blood in urine	Yes	3
	No	0
Kidney stone history	Yes	1
	No	0
Adult	Male	1
	Female	0
Age	b60 years	1
	>= 60 years	0
Diminution of pain within 6 h	Yes	2
Total points	No	0 0-13

Abbreviations: OR, odds ratio; CI, confidence interval.

Study population

We enrolled patients >= 15 years who visited our emergency depart- ment (ED) from 1 May 2015 to 30 April 2016, chiefly complaining of back, flank, or lower abdominal pain. All patients had to have US, urinal- ysis, and CT to be included. However, the US, urinalysis, and CT were all ordered at the discretion of the treating physician. The exclusion criteria in the present study were as follows: 1) refusal to provide informed consent, 2) Abnormal vital signs (subjective or objective fever or hypo- tension), 3) insufficient medical interview, 4) insufficient examination (lack of US, CT scan, or urinalysis), 5) C-reactive protein concentration of >= 6 mg/L when leukocytes were detected using a Urine dipstick or standard urinalysis, and 6) active malignancy.

Methods of measurement

Before this study started, we provided information and instruction regarding the study protocol, and instructions on filling-out forms and data collection techniques to all of the emergency physicians, emergen- cy medicine residents, and emergency nurses involved in the present study.

Emergency staff interviewed ED patients and/or their attending guardians, and selected the candidate patients. They then distributed the specific forms about duration of pain, concomitant nausea or vomiting, prior history of ureteral stones, and informed consent to ED patients and/or their attending guardians (Appendix 1). After medical examination by emergency physicians or emergency medicine resi- dents, the study patients underwent a urine dipstick or standard urinal- ysis, and/or point-of-care US examinations using a curviLinear probe from a compact cart-based US machine (GE Logic P5 Scanner; GE Healthcare Inc., Milwaukee, WI) if necessary for detection of hematuria and hydronephrosis, respectively. The presence or absence of hydronephrosis was judged by the primary examiner. Blood cell count, biochemical analysis, determination of serum markers of inflam- mation, coagulation, arterial blood gas and acid-base balance, or helical CT scans were performed if necessary to differentially diagnose the dis- ease conditions at the discretion of the emergency physicians.

We performed this study during Research staff working hours (08:30 to 00:00).

Methods of diagnosis

Final diagnosis was determined as follows: for ureteral stones, emer- gency physicians made the final diagnosis based on the CT findings interpreted by radiologists. As for alternative diagnoses, medical spe- cialists provided consultation or the treating physician determined the final diagnosis based on the physical, laboratory, or CT findings. Emer- gency physicians made the final diagnosis in cases where consultation was not necessary. After confirming diagnosis, emergency physicians or emergency medicine residents filled out a checklist to calculate the CHOKAI and STONE scores (Appendix 2).

Statistical analysis

We calculated the STONE and CHOKAI scores and analyzed the pre- dictive performance of each score at the optimal cut-off value using a re- ceiver operating characteristic curve and AUC. We also calculated the sensitivity, specificity, positive predictive value (PPV), negative predic- tive value (NPV), positive likelihood ratio (LR+), and negative likeli- hood ratio (LR-) at the optimal cut-off value. In addition, we calculated all of these measures of diagnostic accuracy for all potential cut-off values. All patients were classified into two risk groups according to the optimal cut-off value. A p value of b 0.05 was considered statisti- cally significant. All analyses were performed using R statistical soft- ware version 3.3.1 (http://cran.r-project.org/).

Image of Fig. 1

Fig. 1. Selection of patients with back, flank, or lower abdominal pain. Abbreviations: CT, computed tomography; CRP, C-reactive protein; US, ultrasonography.

Results
1. internal validation of CHOKAI and S”>Internal validation of CHOKAI and STONE scores

Between May 1, 2015 and April 30, 2016, 25,147 patients visited our ED. Of these patients, 823 presented with complaints of back, flank, or lower abdominal pain. Of the 823 patients, 259 patients could not be en- rolled in our study because of no research staff working or were missed while the research staff was working, and 13 were excluded because they declined to participate.

Of the 551 patients enrolled, 455 were excluded for the following reasons: abnormal vital signs (n = 129), pyuria and an elevated C-reac- tive protein concentration (n = 2), active malignancy (n = 21), lack of US (n = 201), lack of CT (n = 27), and lack of urine examination data (n

= 75).

Finally, 96 patients (79 with and 17 without ureteral stones) were included in this study (Fig. 1). All patients were Japanese. The age of the patients finally diagnosed with and without ureteral stones were

53.1 +- 14.6 years (range, 23-89 years) and 55.8 +- 17.1 years (range,

32-86 years), respectively. Fig. 2 shows the diagnostic performance of each scoring model. Table 2 shows the sensitivity, specificity, PPV, NPV, LR+, and LR- at the Optimal cutoff values of the CHOKAI and STONE scores. The AUC of the CHOKAI score was 0.971 (95% confidence interval [CI], 0.928-1.000), showing a sensitivity of 0.911, specificity of 0.941, PPV of 0.986, NPV of 0.696, LR+ of 15.49, and LR- of 0.094 at

an optimal cut-off score of 6. The AUC of the STONE score was 0.873 (95% CI, 0.797-0.948), showing a sensitivity of 0.823, specificity of 0.824, PPV of 0.956, NPV of 0.500, LR+ of 4.662, and LR- of 0.215 at

Fig. 2. Receiver operating characteristic curve indicating the performance of the CHOKAI and STONE scores at the optimal cut-off value. The CHOKAI score is composed of 7 factors yielding 0 to 13 points: age, sex, nausea or vomiting, prior stones, pain duration, microscopic hematuria, and hydronephrosis. The STONE score is composed of 5 factors yielding 0 to 13 points: sex, nausea or vomiting, pain duration, microscopic hematuria, and race. Abbreviations: AUC, area under the curve; CI, confidence interval.

Table 2

Sensitivity, specificity, PPV, NPV, LR+, and LR- at the optimal cut off value of 6 for the CHOKA score and 8 for the STONE score.

Score	Sensitivity	Specificity	PPV	NPV	LR+	LR-
CHOKAI	0.911	0.941	0.986	0.696	15.49	0.094
95% CI	0.826-0.964	0.713-0.999	0.926-1.000	0.471-0.868	2.311-103.9	0.046-0.193
STONE	0.823	0.824	0.956	0.500	4.662	0.215
95% CI	0.721-0.900	0.566-0.962	0.876-0.991	0.306-0.694	1.661-13.09	0.127-0.363

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; LR-, negative likelihood ratio; CI, confidence interval.

an optimal cut-off score of 8. The AUC of the CHOKAI score was signifi- cantly higher than that of the STONE score (p = 0.010).

Risk stratification and cut-off values in the two prediction models

The patients were classified into two risk groups according to each optimal cut-off value: high probability (score of 6-13) and low proba- bility (score of 0-5) for the CHOKAI score, and high probability (score, 8-13) and low probability (score, 0-7) for the STONE score. Of the 73 patients with a CHOKAI score of >= 6, 72 (98.6%) had ureteral stones, and of the 68 patients with a STONE score of >= 8, 65 (95.6%) had ureteral stones (Fig. 3).

Appendices 3 and 4 shows the diagnostic performance of other cut- off values of the CHOKAI and STONE scores. One patient with a CHOKAI score of >= 6 was diagnosed with herniation of a lumbar intervertebral disc as an important alternative cause of renal colic. Three patients with a CHOKAI score of >= 8 were diagnosed with diverticulitis, hernia- tion of a lumbar intervertebral disc, and dissection of the superior mes- enteric artery, respectively.

Image of Fig. 3

Fig. 3. Prevalence of ureteral stones and alternative diagnoses according to the (A) CHOKAI and (B) STONE scores at each optimal cut-off value.

Other outcomes

Of the 79 patients diagnosed with ureteral stones, 68 (86.1%) passed their stones spontaneously. Of the 64 patients with ureteral stones of b 6 mm in short diameter, 62 (96.9%) passed their stones spontaneously, and 2 patients needed ureteral stent placement because of a concomi- tant fever. Of the 15 patients with ureteral stones of N 6 mm in short di- ameter, 6 (40%) passed their stones spontaneously and 9 patients needed surgical intervention such as extracorporeal shock wave litho- tripsy (n = 3), transurethral lithotripsy (n = 5), or both (n = 1). In this study, 3 patients had a concomitant fever within 3 days after visit- ing the ED; 1 was treated with an oral antibacterial agent only, and 2 needed ureteral stone placement. None of the patients diagnosed with ureteral stones suffered septic shock.

Of the 17 patients diagnosed with no ureteral stones, 1 patient with a score of 1 in the CHOKAI and of 3 in the STONE were diagnosed with ab- dominal aortic dissection.

Discussion

In the present study, the following valuable findings were observed. First, the CHOKAI score simplified with integer coefficients proved to be a useful screening tool for predicting ureteral stones and excluding al- ternative causes of renal colic. Second, the optimal cut-off value for Jap- anese patients in the STONE scoring system would be 8 points. Third, the diagnostic performance of the simplified CHOKAI score was superior to that of the STONE score with the optimal cut-off in our cohort.

To date, four papers (one original study and three external validation studies) have reported the diagnostic performance of the STONE score; however, none of them achieved both an LR+ of N 10 and an LR- of b 0.1 [7-9,12]. In our study, the STONE score also did not achieve an LR+ of N 10 and LR- of b 0.1, and its diagnostic performance at the op- timal cut-off value of 8 was LR+ of 4.662 (95% CI, 1.661-13.09) and LR- of 0.215 (95% CI, 0.127-0.363). Generally, an LR+ of N 10 and LR- of b 0.1 are considered to provide strong evidence with which to rule out a diagnosis or not to rule out [13]. In contrast, the CHOKAI score fulfilled this condition, achieving an LR+ of 15.49 (95% CI, 2.31- 103.9) and an LR- of 0.049 (95% CI, 0.046-0.193) at the optimal cut- off value of 6, though both 95% CIs in the CHOKAI score crossed these thresholds of LR+ N 10 and LR- b 0.1. These results strongly support that the simplified CHOKAI score should be a screening model for Japa- nese patients with suspected ureteral stones. The optimal cut-off of the STONE score for our cohort was 8 in the present study. In the original study by Moore et al. [7], patients are classified into three risk groups based on the STONE score: low probability (score of 0-5), moderate probability (score of 6-9), and high probability (score of 10-13), with the cut-off point at which ureteral stones were ruled in and out being a score of 6 [14]. The optimal cut-off value may need to be changed in countries or Geographical areas in which the racial factor is minimal when using the STONE score. In a study by Schoenfeld et al. [8] in which patients of African ancestry constituted only 6.9% of all the pa- tients, the optimal cut-off value of the STONE score was set to be 8.

It is generally well known that ureterolithiasis is one of the most common diseases in urology and has been growing in number

worldwide. Ureterolithiasis has multifactorial and heterogeneous etiol- ogies. Race, sex, body weight, body mass index, diet, volume of fluid in- take, geographic localization, and climate changes are considered to contribute collaboratively to stone pathogenesis [10]. However, each of the causative factors might well contribute variably from country to country. In fact, the odds ratio of each of the four predictive factors (du- ration of pain, occult blood in urine, sex, and gastrointestinal symp- toms), which are common between the CHOKAI and STONE scores, was different [7,11]. We must emphasize that our study sample may be more homogenous than the patient populations encountered in other settings. For example, none of the patients in our study sample were of African ancestry. In this context, the CHOKAI score, whether original or simplified, might not always hold true outside of Japan. Strictly speaking, such criticism could be applicable to any predictive models that have been developed based on a certain group of patients. Though the diagnostic performance of the CHOKAI score outperformed that of the STONE score in this study, US findings contrib- uted to this result. Daniels et al. suggested that adding US findings to the STONE score improved sensitivity and specificity in low- and moderate- risk patients [15]. In addition, some papers have pointed out the useful- ness of US findings in patients with suspected renal colic [16,17]. The finding directly indicates post-renal obstruction of the upper urinary tract, producing the highest odds ratio favoring the presence of ureteral stones in our study. In fact, we gave a score of 4 out of 13 points for the

presence of hydronephrosis in the CHOKAI score.

In the present study, as many as 303 patients were excluded due to lack of certain studies, although not all will necessarily be ordered by emergency physicians on a routine basis in patients with suspected ureterolithiasis. The major reasons are as follows: first, we did not ex- clude patients whom the emergency or the treating physician consid- ered to be at high risk for serious alternative diseases such as colonic diverticulitis, pancreatitis, cholecystitis, and ileus, meaning that they did not require every clinical finding (US, urinalysis, and CT) for definite diagnosis. Second, diagnostic CT was often performed for patients suspected to have ureterolithiasis without initial ultrasonography in our ED. Since the CT evaluated the presence of hydronephrosis, as many as 201 of 303 patients were excluded for lack of US findings. For a small proportion of patients (n = 27), CT examination was unneces- sary for the emergency physicians to diagnose ureteral stones (n = 26) and acute gastroenteritis (n = 1).

Smith-Bindman et al. suggested that initial ultrasonography was as- sociated with lower cumulative radiation exposure than initial CT, with- out significant differences in high-risk diagnoses with complications, serious adverse events pain scores, return ED visits or, hospitalization [16]. However, CT-based diagnosis without initial ultrasonography was still performed in our ED. Though Japan has the highest numbers of CT and diagnostic X-rays performed [18,19], this tendency (CT- based diagnosis without initial ultrasonography) was reported outside of Japan [20]. Many reports point out that cumulative radiation expo- sure increases cancer risk, and young patients are at a higher risk from diagnostic X-rays than adults [5,21]. Schoenfeld et al. indicated that in younger patients with uncomplicated renal colic, the benefit of immedi- ate CT for suspected renal colic is questionable [22]. Nevertheless, nearly six out of seven patients with a diagnosis of renal colic in the emergency department still underwent a CT scan, and in patients with previous ED

visits for renal colic, the CT rate was still 59% [20]. Namely, patients with a history of ureteral stones and renal colic are still at an increased risk of serial CT scanning with potentially high cumulative radiation doses as previously reported [23]. The CHOKAI score, similar to the STONE score, may be helpful in reducing radiation exposure to diagnostic X- rays in the differential diagnosis of renal colic though this will require future studies in which patients with high CHOKAI scores do not under- go CT imaging to ensure the safety and clinical utility of this Diagnostic approach.

Limitations

Our study has three major limitations. First, the size of the study sample was small. Therefore, the diagnostic performance of the CHOKAI score must be verified in a larger external validation study. Of 823 pa- tients assessed for eligibility, only 96 patients were included in this study because of the choice of inclusion and exclusion criteria. The ne- cessity of all three modalities (US, urinalysis, and CT) in particular lead to the small sample in this study. In addition, we only included patients undergoing both US and CT at the discretion of the provider. This would be unusual in other EDs. Therefore, whether our study patients and our outcomes are generalizable to other settings is unclear. Second, this study has a highly homogenous population. Therefore, the diagnostic performance of CHOKAI score would change outside of the present study setting. Third, the skill of the provider performing the point-of- care US is variable. Herbst et al. [24] reported that performance of US for detection of hydronephrosis varied among clinicians. In the routine clinical setting, however, point-of-care US is feasible for the initial work- up of acute abdomen [16,17].

Conclusion

In this internal validation study, we found that the CHOKAI score achieved a better performance in predicting ureteral stones than the STONE score. The CHOKAI score thus shows promise as a first-line mo- dality for the Initial diagnosis of ureteral stones under emergent condi- tions, especially in countries with ethnic distributions similar to that reported in our patient cohort.

Source of support

No funding was provided for this study.

Author disclosure statement

No competing financial interests exist.

Acknowledgements

The authors thank Ikuko Shirahata, RN, Kazuko Kumagai, RN, Takuya Yamanobe, MD, Masaki Ushijima, MD, and Yuya Kuboki, MD for the crit- ical review of this manuscript, and are grateful to clinical residents and nurses working in the emergency department for data collection. We would like to thank Editage (www.editage.jp) for English language editing.

Appendix 1. Medical Questionnaire about pain duration, nausea or vomiting, past history of ureteral stones, and informed consent in this study

Appendix 2. Checklist to calculate CHOKAI and STONE scores. Abbreviations: CT, computed tomography

Appendix 3. CHOKAI score cut-offs and respective sensitivity, specificity, PPV, NPV, LR+, and LR-

Cut off	Sensitivity	Specificity	PPV	NPV	LR+	LR-
	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)
1 2	1.000 (0.932-1.000) 1.000	0.000 (0.000-0.273) 0.294	0.823 (0.732-0.893) 0.868	NA 1.000	1.000 (1.000-1.000) 1.417	NA NA
	(0.932-1.000)	(0.103-0.560)	(0.781-0.930)	(0.359-1.000)	(1.042-1.925)
3 4	1.000 (0.932-1.000) 0.987	0.529 (0.278-0.770) 0.824	0.908 (0.827-0.959) 0.963	1.000 (0.555-1.000) 0.933	2.125 (1.283-3.518) 5.595	NA 0.015
	(0.931-1.000)	(0.566-0.962)	(0.896-0.992)	(0.681-0.998)	(2.003-15.63)	(0.002-0.109)
5	0.962	0.882	0.974	0.833	8.177	0.043
	(0.893-0.992)	(0.636-0.985)	(0.910-0.997)	(0.586-0.964)	(2.223-30.08)	(0.014-0.132)
6 (optimal)	0.911	0.941	0.986	0.696	15.49	0.094
	(0.826-0.964)	(0.713-0.999)	(0.926-1.000)	(0.471-0.868)	(2.311-103.9)	(0.046-0.193)
7	0.848	0.941	0.985	0.571	14.42	0.161
	(0.750-0.919)	(0.713-0.999)	(0.921-1.000)	(0.372-0.755)	(2.148-96.76)	(0.095-0.275)
8	0.684	0.941	0.982	0.390	11.62	0.336
	(0.569-0.784)	(0.713-0.999)	(0.903-1.000)	(0.242-0.555)	(1.725-78.27)	(0.238-0.475)
9 10	0.582 (0.466-0.692) 0.354	1.000 (0.727-1.000) 1.000	1.000 (0.887-1.000) 1.000	0.340 (0.212-0.488) 0.250	NA NA	0.418 (0.322-0.542) 0.646
	(0.250-0.470)	(0.727-1.000)	(0.822-1.000)	(0.153-0.370)		(0.548-0.760)
11 12	0.215 (0.131-0.322) 0.101	1.000 (0.727-1.000) 1.000	1.000 (0.727-1.000) 1.000	0.215 (0.131-0.322) 0.260	NA NA	0.785 (0.699-0.881) 0.899
	(0.045-0.190)	(0.727-1.000)	(0.117-0.291)	(0.176-0.360)		(0.835-0.968)
13	0.101 (0.045-0.190)	1.000 (0.727-1.000)	1.000 (0.518-1.000)	0.260 (0.176-0.360)	NA	0.899 (0.835-0.968)

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; LR-, negative likelihood ratio; CI, confidence interval; NA, not available.

Appendix 4. STONE score cut-offs and respective sensitivity, specificity, PPV, NPV, LR+, and LR-

Cut off	Sensitivity	Specificity	PPV	NPV	LR+	LR-
	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)
1, 2, 3 4	1.000 (0.932-1.000) 0.962	0.000 (0.000-0.273) 0.059	0.823 (0.732-0.893) 0.826	NA 0.250	1.000 (1.000-1.000) 1.022	NA 0.646
	(0.893-0.992)	(0.001-0.287)	(0.733-0.897)	(0.006-0.806)	(0.901-1.160)	(0.071-5.836)
5	0.949	0.118	0.833	0.333	1.076	0.430
	(0.875-0.986)	(0.015-0.364)	(0.740-0.904)	(0.043-0.777)	(0.898-1.289)	(0.086-2.163)
6	0.924	0.471	0.890	0.571	1.745	0.161
	(0.842-0.972)	(0.230-0.722)	(0.802-0.949)	(0.289-0.823)	(1.110-2.745)	(0.064-0.405)
7	0.861	0.765	0.944	0.542	3.658	0.182
	(0.765-0.928)	(0.501-0.932)	(0.864-0.985)	(0.328-0.744)	(1.546-8.658)	(0.099-0.335)
8 (optimal)	0.823	0.824	0.956	0.500	4.662	0.215
	(0.721-0.900)	(0.566-0.962)	(0.876-0.991)	(0.306-0.694)	(1.661-13.09)	(0.127-0.363)
9	0.620	0.941	0.980	0.348	10.54	0.403
	(0.504-0.727)	(0.713-0.999)	(0.894-0.999)	(0.214-0.502)	(1.563-71.15)	(0.297-0.548)
10 11	0.456 (0.343-0.572) 0.342	1.000 (0.727-1.000) 1.000	1.000 (0.858-1.000) 1.000	0.283 (0.175-0.414) 0.246	NA NA	0.544 (0.445-0.666) 0.658
	(0.239-0.457)	(0.727-1.000)	(0.817-1.000)	(0.151-0.365)		(0.562-0.772)
12 13	0.101 (0.045-0.190) 0.063	1.000 (0.727-1.000) 1.000	1.000 (0.518-1.000) 1.000	0.193 (0.117-0.291) 0.187	NA NA	0.899 (0.835-0.968) 0.937
	(0.021-0.142)	(0.727-1.000)	(0.359-1.000)	(0.113-0.282)		(0.885-0.992)