a b s t r a c t

Purpose: The aim of this study was to construct a bacteremia prediction model using commonly available clinical variables in hospitalized patients with community-acquired pneumonia .

Basic procedures: A prospective database including patients who were diagnosed with CAP in the emergency department was analyzed. Independent risk factors were investigated by using multivariable analysis in 60% of the cohort. We assigned a weighted value to predictive factor and made a prediction rule. This model was validated both internally and externally with the remaining 40% of the cohort and a cohort from an independent hospital. The low-risk group for bacteremia was defined as patients who have a risk of bacteremia less than 3%.

Main findings: A total of 2422 patients were included in this study. The overall rate of bacteremia was 5.7% in the cohort. The significant factors for predicting bacteremia were the following 7 variables: systolic blood pressure less than 90 mm Hg, heart rate greater than 125 beats per minute, body temperature less than 35 ?C or greater than 40 ?C, white blood cell less than 4000 or 12,000 cells per microliter, platelets less than 130,000 cells per microliter, albumin less than 3.3 g/dL, and C-reactive protein greater than 17 mg/dL. After using our prediction rule for the validation cohorts, 78.7% and 74.8% of the internal and external validation cohorts were classified as low-risk bacteremia groups. The areas under the receiver operating characteristic curves were

0.75 and 0.79 for the internal and external validation cohorts.

Principal conclusions: This model could provide guidelines for whether to perform blood cultures for hospitalized CAP patients with the goal of reducing the number of blood cultures.

(C) 2014

Introduction

The identification of a causative pathogen and the antibiotic susceptibility of this pathogen are important steps in managing patients with infectious diseases. A blood culture is commonly used in research to determine the etiology of Community-acquired pneumonia . However, the yield of blood cultures for CAP patients is known to be relatively low; and several studies proved that the impact of positive blood culture is minor. Only about 5% to 14% of blood cultures from hospitalized patients with CAP were found to be positive in previous studies [1-5]. False-positive blood culture results could be associated with prolonged hospital stay or high cost [3,6]. In the Infectious Diseases

internal validation “>? Funding sources: none.

* Corresponding author. Department of Emergency Medicine, Seoul National University, Bundang Hospital, 300 Gumi-dong, Bundang-gu, Sungnam-si, Gyeonggi-do, 463-707, Republic of Korea. Tel.: +82 31 787 7572; fax: +82 31 787 4055.

E-mail address: [email protected] (K. Kim).

¹ Equally contributed as a first author.

Society of America/American Thoracic Society Consensus guidelines on the management of CAP in adults, it is recommended that blood cultures be performed selectively for CAP patients; and it is suggested that they are warranted when multiple risk factors for bacteremia are present [7]. Previous studies demonstrated an association between bacteremia in CAP patients and several risk factors, which is useful in determining which patients need blood cultures. However, with these previous guidelines, the potential to reduce the number of blood cultures was not high [6,8]. In this study, we tried to construct a more useful model to predict bacteremia in CAP patients in terms of reducing the usage of blood cultures.

Methods

Derivation and internal validation step

Study design and selection of patients

A prospective database that includes patients who were diagnosed with CAP in the emergency department (ED) of an academic urban hospital from June 2008 to January 2013 was investigated

http://dx.doi.org/10.1016/j.ajem.2014.04.010

0735-6757/(C) 2014

retrospectively. Eligible patients were older than 15 years and hospitalized with CAP. Patients who were discharged from the ED, transferred to other hospitals, or diagnosed with Hospital-acquired pneumonia or health care-associated pneumonia were excluded. Hospital-acquired pneumonia is defined as pneumonia that occurs 48 hours or more after hospitalization; and patients who were hospitalized in hospital, nursing home, and long-term care facilities for 2 or more within 90 days of the infection were classified as health care-associated pneumonia [9-11]. In the ED, the treatment method and admission are decided at the discretion of the attending physician based on the pneumonia severity index scale [12]. This study was performed in a 950-bed tertiary academic hospital with an annual ED census of 80,000 and approved by the institutional review board of the studied hospital.

Data collection and follow-up

We established a protocol for data collection and collected a clinical data set that contained more than 130 factors, which comprised epidemiological data, therapeutic procedure, and details including administered antibiotics, time, laboratory tests at the ED, organisms isolated from blood cultures, and antibiotic sensitivity test results. We recorded vital signs of patients at a stage of triage in ED [12-14]. If a patient’s data were missing or insufficient, our research team reinvestigated the electronic medical records or obtained additional information from patients by phone call. Of these 130 data points, we used epidemiological data and clinical information that has been demonstrated to be useful to predict bacteremia in CAP patients as well as common blood tests that have not been evaluated in previous studies. Specifically, the epidemiologic data included age, sex, heart failure, renal failure, liver disease, chronic obstructive pulmonary disease (COPD), diabetes mellitus , and isolation of organisms from blood cultures, whereas vital signs included systolic blood pressure (SBP), body temperature (BT), heart rate (HR), and respiratory rate (RR), and laboratory tests included white blood cell (WBC), hematocrit (Hct), platelets, albumin, C-reactive protein (CRP), Blood urea nitrogen , creatinine (Cr), sodium (Na), and glucose.

Blood collection and culture

Blood cultures were performed by the following method. First, 10 mL of blood was obtained after skin disinfection and inoculated into aerobic and anaerobic bottles. An additional 10 mL of blood was obtained from a different site and inoculated using the same method. An automated device was used to detect isolated organisms. The following isolates were considered to be contaminants: coagulase-negative Staphylococcus, Bacillus species, Micrococcus species, Corynebacterium species, Propionibacterium species, and diphtheroids.

Statistical analysis and score derivation

Continuous variables are presented as the mean with standard deviations. Binomial variables are presented as the frequency of occurrence and were compared with the ?² test. Continuous variables were dichotomized by clinical relevancy or reference range as described in previous studies [6,15]. The cutoffs selected were as follows: age greater than 65, SBP less than 90 mm Hg, HR greater than 125/min, RR greater than 30 cycles per minute, BT less than 35 ?C or greater than 40 ?C, WBC less than 4000 or 12,000 cells per microliter, Hct less than 30%, platelets less than 130,000 cells per microliter, glucose greater than 250 mg/dL, albumin less than 3.3 mg/dL, BUN greater than 30 mg/dL, Cr greater than 1.5 mg/dL, and Na less than 130 mg/dL. Because most eligible patients had a value higher than the reference range for CRP (0.5 mg/dL), we assigned a cutoff value for CRP of 17 mg/dL. We could not find any bacteremia prediction model in CAP that used a specific cutoff value of CRP. Therefore, we constructed a receiver operating characteristic (ROC) of CRP in derivation cohort and found the highest value of sensitivity and specificity (Youden index).

The enrolled patients were randomized into 2 groups to create a derivation cohort consisting of 60% of the patients and an internal validation cohort with the remaining 40% of patients. After dichoto- mizing the variables, we investigated epidemiological factors, vital signs, and laboratory results using univariable analysis in the derivation cohort. Subsequently, multivariable analysis was per- formed to find independent risk factors using the candidate variables with a factor significance of P b .05 in the univariable analysis. Variables were removed from the multivariable model in a stepwise manner, and the final model with best fit was determined using a Bayesian information criterion, which is a criterion for model selection that introduces a penalty term to the number of parameters in a model. One model is better than another if it has a smaller Bayesian information criterion value [16]. The prediction model for bacteremia was developed using a regression coefficient-based scoring method [17,18]. We found predictive factors in multivariable analysis and assigned a weighted value to each factor using ? coefficients that reflect Predictive power. The ? coefficients in multivariable logistic regression model were divided by 0.3 and rounded to the nearest whole number. We used the number as bacteremia risk points. The points were assigned to the predictive factors to estimate the risk of bacteremia in CAP. The overall risk score was regarded as the sum of the calculated point. To stratify the risk of bacteremia as low, intermediate, and high, we selected the specific cutoff values of 3% and 20%. The low-risk group for bacteremia was defined as patients who have a risk of bacteremia less than 3%. Finally, we analyzed an ROC curve and the area under the ROC curve (AUC) to investigate the performance of the prediction model.

The analyses were performed using SPSS version 17.0 (SPSS Inc, Chicago, IL). To select a maximal predictive point of CRP, MedCalc version 12.4 (MedCalc Software, Mariakerke, Belgium) was used.

Internal validation

The remaining cohort of 40% of patients was used to validate the prediction model internally. We computed the number of patients and the proportion according to the risk score that is assigned to each predictive factor. We also analyzed the ROC curve and computed AUC to quantify the performance of our prediction model.

External validation step

We retrospectively collected patients with CAP who were older than 15 years and hospitalized in the ED of an independent tertiary academic hospital between January 2011 and December 2012. Inclusion and exclusion criteria were set up in the same manner as for the derivation and internal validation cohorts. We tried to discriminate the patients at low risk for bacteremia within the external validation cohort using our prediction model and computed the number of patients according to the score. The AUC was calculated to evaluate the discrimination capability of the prediction model. This study was also approved by the institutional review board of the hospital.

Results

Derivation and internal validation

Study population

A prospective database that includes 3877 patients who were diagnosed with CAP in the ED of an academic urban hospital was analyzed retrospectively. Discharged and Transferred patients num- bered 1117 and 204, respectively. Blood cultures were not conducted for 134 hospitalized patients; the remaining 2422 patients were analyzed. Independent risk factors were evaluated using multivari- able logistic regression for 1475 patients (60% of enrolled patients). This model was validated internally with the remaining 947 (40%) patients and was externally validated with 1429 patients from an

independent academic hospital (Fig.). Table 1 shows baseline demographic findings, vital signs, and laboratory tests for each cohort.

Table 1

Characteristics of derivation, internal validation, and external validation cohorts

Among enrolled patients, true pathogens were detected in 140 (5.7%); and contaminants were detected in 114 (4.4%). Table 2 shows isolated organisms from the derivation, internal validation, and external validation cohorts. Among the true pathogens, Streptococcus pneumonia was the most frequently isolated organism in each cohort;

and Escherichia coli, Klebsiella pneumonia, Staphylococcus aureus, Pseudomonas aeruginosa, and Viridans streptococci followed (Table 2).	Male sex Heart failure	1,017 (68.9) 37 (2.5)	656 (69.2) 20 (2.1)	858 (60) 17 (1.2)	b.001 .075
	Renal failure	154 (10.4)	103 (10.9)	85 (5.9)	b.001
	Liver disease	76 (5.1)	43 (4.5)	46 (3.2)	.097
3.1.2. Score derivation	COPD	249 (16.9)	149 (15.7)	196 (13.7)	.172
In the univariable analysis, the following 12 variables were significant factors: liver disease, COPD, SBP less than 90 mm Hg, HR	DM Vital signs, mean +- SD SBP, mmHg	427 (28.9) 126.6 +-	269 (28.4) 127.5 +- 27.8	330 (23.7) 131.7 +- 28.5	.004 b.001
greater than 125 beats per minute, BT less than 35 ?C or greater than		28.4
40 ?C, WBC less than 4000 or 12,000 cells per microliter, Hct less than	HR, beats/min	103.1 +-	103.3 +- 22.4	101.3 +- 19.1	.026
30%, platelets less than 130,000 cells per microliter, albumin less than 3.3 g/dL, BUN greater than 30 mg/dL, Cr greater than 1.5 mg/dL, and	RR, cycles/min BT, ?C	22.9 23.3 +- 6.2 37.4 +- 1	23.3 +- 6.1 37.4 +- 1	21.7 +- 4.6 37.3 +- 1	b.001 .452
CRP greater than 17 mg/dL (Table 3). We subsequently used these 12	Laboratory
variables for multivariable analysis. Consequently, the following 7	findings, mean +- SD
variables were demonstrated as useful to predict bacteremia in CAP: SBP less than 90 mm Hg, HR greater than 125 beats per minute, BT less	WBC count, x103 mm3 Hct, %	12.4 +- 6.9 35.5 +- 6.2	12.4 +- 6.6 35.5 +- 6.2	10.6 +- 5.6 36 +- 5.8	b.001 .031
than 35 ?C or greater than 40 ?C, WBC less than 4000 or 12,000 cells	Platelet count,	237.3 +-	234.3 +- 118.1	227.2 +- 109.1	.131
per microliter, platelets less than 130,000 cells per microliter, albumin	x103vmm3	119.4
less than 3.3 g/dL, and CRP greater than 17 mg/dL. The ? coefficients of	Glucose, mg/dL	158.9 +-	158.2 +- 84.6	145 +- 69.2	b.001
this model were divided by 0.3 and rounded to the nearest number. The computed numbers were assigned to the independent variables	Albumin, g/dL BUN, mg/dL	88.6 3.3 +- 0.5 24.8 +-	3.3 +- 0.5 24.9 +- 18.1	3.3 +- 0.5 18.9 +- 13.4	.12 b.001
(Table 4). The overall risk score was the sum of the calculated point.		17.7
Table 5 shows the number of patients and rate of bacteremia	Cr, mg/dL	1.3 +- 1.2	1.3 +- 1.3	1.1 +- 1.2	.151
according to the assigned scores. The cutoff points were 5 and 10.	Na	135.4 +-	135.4 +- 7.8	137.2 +- 7.1	.128

Category, parameter Derivation

(n = 1475)

Epidemiological data

Mean age +- SD, y 71.4 +-

14.6

Internal validation External validation P^a

(n = 947) (n = 1429)

71.7 +- 14.5 64.6 +- 16.3 b.001

Patients who had a score lower than 5 were classified into the low-risk group, which was defined as having a bacteremia risk less than 3%.

In the derivation cohort, the number of patients with scores less than or equal to5 was 1114 (77.5%), whereas 302 (20.4%) had 6 to 10 points and 37 (2.5%) had greater than or equal to 11 points. The incidence of bacteremia in these groups was 2.7%, 15.8%, and 29.7%, respectively (Table 5). The AUC of the prediction rule was calculated as 0.78 (95% confidence interval [CI], 0.73-0.83).

7.4

CRP, mg/dL 12.9 +- 9.2 12.9 +- 9.3 12.3 +- 9.7 .106

Bacteremia 90(6.1) 50(5.3) 83(5.8) .583

Categorical data are presented as number (percentage) of patients.

^a Internal validation group and external validation group were compared.

Internal validation

We adjusted the prediction rule to the internal validation cohort and evaluated bacteremia risk according to the assigned scores. A total of 937 patients were enrolled and analyzed for internal validation. The number of patients who were classified in the bacteremia low-risk group was 746 (78.7%), and the AUC was 0.75 (95% CI, 0.68-0.83) (Table 5).

Table 2

Microorganisms isolated from blood cultures in the derivation, internal validation, and external validation cohorts

Microorganism	Derivation	Internal validation	External validation
	(n = 1475)	(n = 947)	(n = 1429)
S pneumoniae	17 (19)	10 (20)	29 (35)
E coli	14 (16)	8 (16)	9 (11)
K pneumoniae	14 (16)	7 (14)	14 (17)
S aureus	12 (13)	8 (16)	7 (8)
P aeruginosa	9 (10)	4 (8)	7 (8)
V streptococci	9 (10)	3 (6)	5 (6)
S agalactiae	3 (3)	1 (2)	3 (4)
Haemophilus influenzae	3 (3)	1 (2)	2 (2)
Other gram-positive cocci	4 (4)	4 (8)	3 (4)
O t her gra m- negative	5 (6)	4 (8)	4 (5)
bacilli Total	90	50	83

Fig. Study population. Data are presented as number (percentage) of organisms.

Table 3

Univariate analysis of clinical factors in patients with or without bacteremia in the derivation cohort

Table 5

Number of patients with or without bacteremia according to Clinical score in the derivation, internal validation, and external validation cohorts

Category, parameter Patients without

bacteremia (n = 1375)

Patients with P

bacteremia (n = 90)

Group, score Patients with bacteremia, n (%) Total patients Derivation cohort

Epidemiological data, n				Low-risk group, <=5	31 (2.7)	1144
(%)				Moderate-risk group, 6-10	48 (15.8)	302
Age N 65 y	1046 (75.5)	69 (76.7)	.807	High-risk group, >=11	11 (29.7)	37
Male sex	957 (69.1)	60 (66.7)	.629	Internal validation cohort
Heart failure	32 (2.5)	2 (2.2)	.858	Low-risk group, <=5	20 (2.6)	746
Renal failure	141 (10.2)	13 (14.4)	.2	Moderate-risk group, 6-10	25 (13.6)	183
Liver disease	67 (4.8)	9 (10)	.032	High-risk group, >=11	5 (31.2)	16
COPD	243 (17.5)	6 (6.7)	.008	External validation cohort
DM	407 (29.4)	20 (22.2)	.146	Low-risk group, <=5	25 (2.3)	1070
Vital signs, n (%)				Moderate-risk group, 6-10	47 (15.1)	311
SBP b 90 mmHg	104 (7.5)	24 (26.7)	b.001	High-risk group, >=11	10 (22.7)	44

Data are presented as number (percentage) of patients.

HR N 125 beats/min	185 (13.4)	25 (27.8)	b.001
RR N 30 cycles/min	164 (11.8)	13 (14.4)	.461
BT b35 ?C or N 40 ?C	14 (1)	3 (3.3)	.045
Laboratory findings,
mean +- SD

WBC count b 4000 or	698 (50.4)	59 (65.6)	.005	In this study, we demonstrated that 7 variables were useful to
12,000 cells/uL				discriminate patients with bacteremia in CAP: SBP less than 90 mm
Hct b30%	251 (18.1)	25 (27.8)	.023	Hg, HR greater than 125 beats per minute, BT less than 35 ?C or greater

Platelet count b 130,000 cells/uL	190 (13.7)	32 (35.6)	b.001
Glucose N 250 mg/dL	155 (11.2)	11 (12.5)	.71
Albumin b3.3 mg/dL	600 (43.3)	63 (70)	b.001
BUN N 30 mg/dL	317 (22.9)	43 (47.8)	b.001
Cr N 1.5 mg/dL	1117 (80.6)	56 (62.2)	b.001
Na b130 mmol/L	225 (16.2)	19 (21.1)	.229
CRP N 17 mg/dL	398 (28.8)	54 (60)	b.001

Categorical data are presented as number (percentage) of patients.

External validation

We retrospectively enrolled 1429 patients in an independent tertiary academic hospital from January 2011 to December 2012. External validation group was different from internal validation group in term of age, sex, renal failure, DM, SBP, HR, RR, WBC, Hct, glucose, and BUN (Table 1). After using our prediction model, 1070 (74.8%) patients in the external validation cohort were classified as part of the low-risk group. The AUC of the external validation cohort was calculated to be 0.79 (95%CI, 0.75-0.84) (Table 5). We showed that our model can be used in different group of patients.

Discussion

Blood cultures are not considered for all patients with CAP because of the low yield and the lack of definitive impact on outcome [7]. However, blood cultures are still regarded as an important tool to manage CAP because they are technically easy to obtain and susceptibility data can permit the narrowing of antibiotic therapy.

Table 4

Multivariate analysis of predictive factors in the derivation cohort

than 40 ?C, WBC less than 4000 or 12,000 cells per microliter, platelets less than 130,000 cells per microliter, albumin less than 3.3 g/dL, and CRP greater than 17 mg/dL. Among these 7 risk factors, albumin, CRP, and platelets were demonstrated to be the most useful factors to predict bacteremia in CAP with this study (Table 4).

In previous studies, 8% to 17% of cohorts were classified into the low-risk group for bacteremia, which is defined as a bacteremia rate less than 3% [6,8]. In terms of patient selection, 26.6% and 41.6% of enrolled patients did not undergo blood cultures or were excluded from analysis because of missing data in previous studies. Our prediction model discriminated 78.7% and 74.8% of the internal validation and external validation cohorts as low risk for bacteremia, and only 5% did not undergo blood cultures and were excluded from the enrolled patients. Taken together, our model may be an improvement on previous studies in terms of Predictive performance and selection bias. This is because we still perform blood cultures for most hospitalized patients with CAP.

We can recommend that it is not necessary to obtain blood cultures from patients in the low-risk group but that blood cultures should be performed for high-risk patients. In the case of the moderate-risk group, the decision could be at the discretion of managing physician. If our prediction rule is applied, the number of blood cultures performed for patients with CAP could be reduced more than with previous prediction rules. Although Medical cost is different from other countries, low-risk patients can save about US $50 without blood cultures in South Korea.

Two previous studies demonstrated that liver disease is useful to discriminate patients with bacteremia in CAP [6,8]. In our study, liver disease was not significant after the final multivariable analysis. Patients who have liver disease tend to have hypoalbuminemia and low platelet counts. We think that hypoalbuminemia and low platelet count eliminate liver disease in the multivariable analysis. Patients or treating physicians may not acknowledge whether liver disease is present. Furthermore, history of liver disease is not sufficient to

Variables	OR (95% CI)	P	Coefficient	No. of points	evaluate the severity of the disease. However, albumin and platelet levels are objective laboratory tests and could be more precise than a
SBP b90 mmHg	2.54 (1.43-4.51)	.001	0.93	3	history of liver disease.
HR N 125 beats/min	2.16 (1.26-3.72)	.005	0.77	3	This study has several limitations. First, history of prior antibiotic
BT b35 ?C or N 40 ?C	4.57 (1.04-19.9)	.043	1.52	5	therapy was not investigated in our study. For some patients, the
WBC count b4000 or 12,000 1.63 (1.01-2.64) .044 0.49 2 previous use of antibiotics could be recorded. However, patient cells/uL medical histories depend on statements from patients or their Platelet count b130,000 cells/ 2.46 (1.46-4.12) .001 0.9 3
uL	guardians, both of whom are not health care providers. Therefore,

OR, odds ratio.

we decided that we would not include the history of antibiotics.

Second, potential biases might remain because the external validation cohort was not from a prospective registry. Third, we did not include

Albumin b3.3 mg/dL	1.83 (1.08-3.09)	.025	0.6	2
CRP N 17 mg/dL	2.82 (1.75-4.55)	b.001	1.03	3

other biochemical markers that are known to be associated with the outcomes of infectious diseases, such as procalcitonin and lactate. However, we do not think these laboratory variables are available worldwide. In conclusion, this model could provide guidelines for whether to perform blood cultures for hospitalized CAP patients with the potential goal of reducing the number of blood cultures. We believe that a further prospective multicenter study is needed to improve the usefulness of this model in predicting bacteremia in CAP.

Acknowledgment

None.

References

1. Campbell SG, Marrie TJ, Anstey R, et al. The contribution of blood cultures to the clinical management of adult patients admitted to the hospital with community- acquired pneumonia: a prospective observational study. Chest 2003;123 (4):1142-50.
2. Waterer GW, Wunderink RG. The influence of the severity of community-acquired pneumonia on the usefulness of blood cultures. Respir Med 2001;95(1):78-82.
3. Houck PM, Bratzler DW, Nsa W, et al. Timing of antibiotic administration and outcomes for Medicare patients hospitalized with community-acquired pneumo- nia. Arch Intern Med 2004;164(6):637-44.
4. Fine MJ, Smith MA, Carson CA, et al. Prognosis and outcomes of patients with community-acquired pneumonia. A meta-analysis. JAMA 1996;275(2):134-41.
5. Metersky ML. Is the lateral decubitus radiograph necessary for the management of a parapneumonic pleural effusion? Chest 2003;124(3):1129-32.
6. Metersky ML, Ma A, Bratzler DW, et al. Predicting bacteremia in patients with community-acquired pneumonia. Am J Respir Crit Care Med 2004;169(3):342-7.
7. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis 2007;44(Suppl. 2): S27-72.
8. Falguera M, Trujillano J, Caro S, et al. A prediction rule for estimating the risk of bacteremia in patients with community-acquired pneumonia. Clin Infect Dis 2009;49(3):409-16.
9. Niederman MS. Guidelines for the management of respiratory infection: why do we need them, how should they be developed, and can they be useful? Curr Opin Pulm Med 1996;2(3):161-5.
10. Hutt E, Kramer AM. Evidence-based guidelines for management of nursing home- acquired pneumonia. J Fam Pract 2002;51(8):709-16.
11. Mylotte JM. Nursing home-acquired pneumonia. Clin Infect Dis 2002;35 (10):1205-11.
12. Jo S, Kim K, Jung K, et al. The effects of incorporating a pneumonia severity index into the admission protocol for community-acquired pneumonia. J Emerg Med 2012;42(2):133-8.
13. Lee JH, Kim J, Kim K, et al. Albumin and C-reactive protein have prognostic significance in patients with community-acquired pneumonia. J Crit Care 2011;26 (3):287-94.
14. Lee JH, Chung HJ, Kim K, et al. red cell distribution width as a prognostic marker in patients with community-acquired pneumonia. Am J Emerg Med 2013;31 (1):72-9.
15. Fine MJ, Auble TE, et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 1997;336(4):243-50.
16. Schwarz G. Estimating the dimension of a model. Ann Stat 1978;6:461-4.
17. Tu JV, Naylor CD. clinical prediction rules. J Clin Epidemiol 1997;50(6):743-4.
18. Moons KG, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? J Clin Epidemiol 2002;55(10):1054-5.