a b s t r a c t

Purpose: The aim of this present study was to compare the diagnostic accuracy and reproducibility of 2 clinical decision rules (the Ottawa Knee Rules [OKR] and Pittsburgh Decision Rules [PDR]) developed for selective use of x-rays in the evaluation of isolated knee trauma. Application of a decision rule leads to a more efficient evaluation of knee injuries and a reduction in Health care costs. The diagnostic accuracy and reproducibility are compared in this study.

Methods: A cross-sectional interobserver study was conducted in the emergency department of an urban teaching hospital from October 2008 to July 2009. Two observer groups collected data on standardized case- report forms: emergency medicine residents and surgical residents. Standard knee radiographs were performed in each patient. Participants were patients 18 years and older with isolated knee injuries. Pooled sensitivity and specificity were compared using ?² statistics, and Interobserver agreement was calculated by using ? statistics.

Results: Ninety injuries were assessed. Seven injuries concerned fractures (7.8%). For the OKR, the pooled

sensitivity and specificity were 0.86 (95% confidence interval [CI], 0.57-0.96) and 0.27 (95% CI, 0.21-0.35), respectively. The PDR had a pooled sensitivity and specificity of 0.86 (95% CI, 0.57-0.96) and 0.51 (95% CI, 0.44-0.59). The PDR was significantly (P = .002) more specific. The ? values for the OKR and PDR were 0.51 (95% CI, 0.32-0.71) and 0.71 (95% CI, 0.57-0.86), respectively.

Conclusion: The PDR was found to be more specific than the OKR, with equal sensitivity. Interobserver agreement was moderate for the OKR and substantial for the PDR.

(C) 2013

Introduction

Acute knee trauma is very common. About 40 000 patients with a knee injury are seen every year in emergency departments (EDs) in the Netherlands [1]. Of these patients, 9% actually have a fracture [1]. Radiography is commonly performed as a diagnostic tool. Previous studies in the United States suggest that 66% to 85% of the patients with knee injury undergo radiography [2,3], and a fracture percentage of only 6% to 12% is seen [3-5]. Thus, most patients undergo unnecessary radiography, leading to unnecessary radiation exposure, longer Waiting times, and increased Medical costs [4,6].

? Presentation: 13th European Congress of Trauma and Emergency Medicine; 15 May 2012; Basel, Switzerland. Orthopaedic Trauma Association annual meeting, Minneap- olis, United States, October 3-6, 2012 (poster).

* Corresponding author. Department of Surgery/Traumatology, VU University Medical Center, PO Box 7057, 1007 MB Amsterdam, the Netherlands. Tel.: +31-20- 4444554; fax: +31-20-4440274.

E-mail address: [email protected] (Y. Tank).

Several clinical decision rules have been developed for the evaluation of knee injuries [3-5,7], of which the Pittsburgh Decision Rule (PDR) and the Ottawa Knee Rule (OKR) are the best known guidelines with the largest validation cohorts for the selective use of radiographs in knee trauma [3,4]. Application of the rules has proved to increase the efficacy with which knee injuries are evaluated and to reduce health care costs without leading to an increase in Missed fractures [4,8].

The PDR was developed in 1994 and has been reported to reduce the number of patients who require x-rays by 78% without missing a fracture [4]. This decision rule found that the combination of a fall or blunt trauma (Fig. 1) with either the inability to ambulate or an age younger than 12 or older than 50 years was 100% sensitive and 79% specific for the detection of fractures [4]. Although only 3 studies have been performed, the PDR has been validated with good results, with sensitivities of 77% to 100% and specificities of 57% to 79% [4,6,9].

The OKR was developed in 1995 [3]. The OKR states that if any of the following clinical findings (Fig. 2) are positive, radiographs are

0735-6757/$ – see front matter (C) 2013 http://dx.doi.org/10.1016/j.ajem.2012.11.003

Fig. 1. Pittsburgh Decision Rule.

needed: (1) age 55 years or older, (2) isolated tenderness of the patella, (3) tenderness at the head of the fibula, (4) inability to flex to 90?, or (5) inability to bear weight both immediately and in the ED (4 steps) [3]. Prospective validation (n = 1096) of this rule demon- strated a sensitivity of 100% with a specificity of 49% [10]. Other validation studies showed sensitivities of 95% to 100% and specificities of 17% to 52% [11-14] and a reduction in radiographs ranging from 25% [15] to 49% [12].

There are 2 important differences between the OKR and the PDR. The OKR applies only to patients older than 18 years, whereas the PDR applies to patients of any age. Second, the Ottawa definition of ability to walk 4 steps refers to any weight transfer during walking, whereas the Pittsburgh definition requires that the patient walk 4 full-weight- bearing steps [6].

Study objectives“>Study objectives

Much research has been done on the OKR, in contrast to the PDR. Only one study has been performed comparing the OKR and the PDR [6]. However, this study was performed by the developers of the PDR, making it susceptible to bias. Therefore, the primary aim in this prospective validation study is to assess the diagnostic accuracy of the OKR vs the PDR by an independent research group. Furthermore, reproducibility is another important clinimetric property of a diagnostic test. Reproducibility has only been assessed in small extent

for both decision rules [9,10], and there are no available data that have compared the reproducibility between the 2 rules. Therefore, the second aim in this study is to assess the reproducibility of the PDR in comparison with the OKR.

Methods

Study design

A cross-sectional validation study was performed between October 2008 and July 2009 and included all consecutive patients 18 to 79 years old visiting the ED of a general hospital with a knee injury that had occurred within the prior 7 days. Each patient aged 18 to 79 years with an isolated knee injury visiting the ED was approached to participate in the study and was handed an information letter and an eligibility form. This form comprised 5 questions to be answered by the patient and 3 questions by the nurse concerning other (prior) injuries, pregnancy, past knee surgery, open wound at the knee, Altered level of consciousness, and paraplegia (exclusion criteria). In case all questions were answered with “no,” the patient was considered eligible. Written informed consent was obtained from all subjects before inclusion. After inclusion, the study subject was assessed by 2 separate observers, an emergency medicine resident (EMR) and a surgical resident (SR), using the PDR and the OKR (Fig. 3). Randomization for first observer was done. Each observer took,

Fig. 2. Ottawa Knee Rule.

Knee injury

• • • First observer randomized

Definitive diagnosis + treatment

Standard X-ray (reference standard)

OKR/PDR

Assessment SR

Assessment EMR

OKR/PDR

• • • Blinded for each others scoring results
    • CRF
    • First rule randomized

Fig. 3. Schematic representation of the study protocol. After inclusion, the study subjects were either first assessed by the EMR or the SR in a randomized manner, using the PDR and OKR (second randomization of the first rule was applied). Both observers reported their findings on standardized CRFs. All patients underwent radiography (reference standard), after which the EMR treated the injury according to his/her own findings.

independent of the other observer, the patient history and performed a clinical assessment using both the PDR and the OKR, after a second randomization had taken place for the first rule to be applied. Each observer reported his/her findings independent of one another on standardized case-report forms (CRFs) without being informed of each others’ results (blinded). The completed forms were inserted in envelopes. This concluded the observers’ participation in the trial. Pain medication was administered if necessary after evaluation.

After assessment by both observers, the reference standard was applied by having each patient undergo a standard knee x-ray (anteroposterior and lateral) and patella x-ray, which was evaluated by one radiologist. The EMR consequently decided which treatment the patient should receive based on his/her own assessment and evaluation of the radiographs (lege artis).

Approximately 1 week after the ED visit, each patient was reevaluated by a physician at the surgical outpatient clinic. The

Enrollment

evaluation of the radiographs by the radiologist was used as the reference standard.

Sample size

A 15% difference in specificity was considered clinically significant. Power analysis was aimed at enabling the detection of a 15% difference in specificity between the observer groups using a power of 80% (2 sided, ? = .05). This resulted in a minimum of 87 participants to be included in the study.

Observer groups

The EMR group consisted of 13 residents. The mean age for the EMR group was 31 years (range, 26-37 years), and the mean clinical experience in emergency medicine or in surgery was 2 years. In this

Assessed for eligibility (n = 206)

Allocation

Randomized (n = 90) for first observer and first decision rule

Excluded (n = 116 )

Not meeting inclusion criteria (n = 105)

Refused to participate (n = 11)

Fig. 4. Flow diagram of Pittsburgh Vs. Ottawa Knee Rule trial showing the enrollment and allocation of patients in both groups (EMR and SR groups).

Analysis

Analysed (n = 90)

Excluded from analysis (n = 0)

First rule assessed OKR (n = 26)

Allocated to first assessment of EMR group (n = 45)

First rule assessed PDR (n = 19)

First rule assessed OKR (n = 19)

Allocated to first assessment of SR group (n = 45)

Analysed (n = 90)

Excluded from analysis (n = 0)

First rule assessed PDR (n = 26)

Table 1

Characteristics of the 90 patients with knee injury in the study

Average age (y) 41

Range (y) Sex, n (%)	18-79
Male	50 (56)
Female	40 (44)
No. of fractures, n (%) Patella	4 (4)
Tibial plateau	2 (2)
Femoral condyle fracture	1 (1)
No. of radiographs, n (%) Knee	90 (100)
Patella	90 (100)

observer group, 6 male and 7 female observers were represented. The SR group consisted of 17 residents. The mean age was 32 years (range, 27-37 years), and the mean clinical experience in emergency medicine or in surgery was 3 years. Twelve male and 5 female observers were represented in this group.

Study subjects

Inclusion criteria for the study were patients (1) with a knee injury and (2) aged between 18 and 79 years.

Exclusion criteria were as follows: (1) patients with a knee injury as part of multiple injuries, (2) knee injury more than 7 days old, (3) recent knee injuries being reevaluated, (4) pregnancy, (5) past knee surgery, (6) open wound at the knee larger than 1 cm, (7) patients with altered levels of consciousness, and (8) paraplegia.

Outcome parameters and statistical analysis

The diagnostic accuracy for the indication of a radiograph was determined, as measured by the sensitivity, specificity, positive predictive value, and negative predictive value. The accuracy parameters were consequently compared between PDR and OKR by ?² statistics. P b .05 was considered statistically significant. Inter- observer agreement (reproducibility) was calculated by using ? statistics.

Results

During the 10-month study (October 2008 through July 2009), 206 patients were assessed for eligibility. Of these patients, 105 were excluded and 11 patients were eligible but refused participation. Consequently, 90 (44%) of 206 eligible patients with knee injury were enrolled in the study (Fig. 4). Radiography was performed for all study subjects. The average age of the study subjects (Table 1) was 41 years, with a range from 18 to 79 years. Slightly more men (56%) were represented. There were 7 (7.8%) fractures, namely, 4 patellar fractures, 2 tibia plateau fractures, and 1 femoral condyle fracture.

Diagnostic accuracy

Sensitivity of the OKR was 0.86 (95% confidence interval [CI], 0.49- 0.97) for both observer groups. Pooled sensitivity of both observer

Table 2

Diagnostic accuracy of the OKR and PDR

	OKR (95% CI)	PDR (95% CI)	P
Pooled sensitivity	0.86 (0.57-0.96)	0.86 (0.57-0.96)	.831
Pooled specificity	0.27 (0.21-0.35)	0.51 (0.44-0.59)	.002

?² Statistics were used to compare the accuracy parameters between the PDR and the OKR. P values were calculated using z statistics.

Table 3

Performance of the OKR and PDR for the EMR group

Rule	Positive radiographic findings	Negative radiographic findings	Total
OKR Positive	6	64	70
Negative	1	19	20
Total PDR	7	83	90
Positive	6	40	46
Negative	1	43	44
Total	7	83	90

OKR: sensitivity, 0.86 (95%CI, 0.49-0.97); specificity, 0.23 (95% CI, 0.15-0.33); negative

predictive value, 0.95 (95% CI, 0.76-0.99); positive predictive value 0.09 (95% CI, 0.04-

0.18). PDR: sensitivity, 0.86 (95% CI, 0.49-0.97); specificity 0.52 (95% CI, 0.41-0.62);

negative predictive value, 0.98 (95% CI, 0.88-1.00); positive predictive value 0.13 (95%

CI, 0.06-0.26).

groups was 0.86 (95% CI, 0.57-0.96). Specificity for the EMR group was

0.23 (95% CI, 0.15-0.33) compared with 0.31 (95% CI, 0.22-0.42) for

the SR group. Pooled specificity was 0.27 (95%CI, 0.21-0.35). The positive predictive value for the EMR group was 0.09 (95% CI, 0.04- 0.18) compared with 0.10 (95% CI, 0.04-0.19) for the SR group. The negative predictive value for the EMR group was 0.95 (95% CI, 0.76- 0.99) compared with 0.96 (95% CI, 0.82-0.99) for the SR group.

For the use of the PDR, sensitivity was 0.86 (95% CI, 0.49-0.97) for both observer groups. Pooled sensitivity was 0.86 (95% CI, 0.57-0.96). Specificity for the EMR group was 0.52 (95% CI, 0.41-0.62) compared with 0.51 (95% CI, 0.40-0.61) for the SR group. Pooled specificity was

0.51 (95% CI, 0.44-0.59). The positive predictive value for the EMR group was 0.13 (95% CI, 0.06-0.26) and 0.13 (95% CI, 0.06-0.25) for the

SR group. The negative predictive value for the EMR group was 0.98 (95% CI, 0.88-1.00) and 0.98 (95% CI, 0.88-1.00) for the SR group.

The 2 rules had similar sensitivities (P = .831). The PDR had a statistically significant (P = .002) higher specificity (Table 2).

Tables 3 and 4 show the performance of both rules.

Reproducibility

Interobserver agreements calculated as ? coefficient of all clinical findings of the OKR were as follows: age older than 55 years (? = 1.0; 95% CI, 1.00-1.00), isolated patellar tenderness (? = 0.64; 95% CI,

0.45-0.83), fibular head tenderness (? = 0.42; 95% CI, 0.19-0.65),

inability to bend knee to 90? (? = 0.43; 95% 0.22-0.63), and inability

to bear weight (? = 0.53; 95% CI, 0.34-0.73) (Table 5). Outlines the interpretation of the ? coefficient.

The interobserver agreement for the clinical findings of the PDR were as follows: age older than 50 years (? = 1.0; 95% CI, 1.00-1.00) and full weight bearing (? = 0.54; 95% CI, 0.36-0.72).

Table 4

Performance of the OKR and PDR for the SR group

Rule	Positive radiographic findings	Negative radiographic findings	Total
OKR Positive	6	57	63
Negative	1	26	27
Total	7	83	90
PDR Positive	6	41	47
Negative	1	42	43
Total	7	83	90

OKR: sensitivity, 0.86 (95%CI, 0.49-0.97); specificity, 0.31 (95% CI, 0.22-0.42); negative

predictive value, 0.96 (95% CI, 0.82-0.99); positive predictive value 0.10 (95% CI, 0.04-

0.19). PDR: sensitivity, 0.86 (95% CI 0.49-0.97); specificity, 0.51 (95%, 0.40-0.61);

negative predictive value, 0.98 (95% CI, 0.88-1.00); positive predictive value, 0.13

(0.06-0.25).

Table 5

Interobserver agreement for the OKR

Table 6

Interobserver agreement results displayed as ? values for the overall PDR and the subsets

Clinical finding ? (95% CI)

Age N 55 y 1.0 (1.00-1.00)

Isolated patellar tenderness 0.64 (0.45-0.83)

Fibular head tenderness 0.42 (0.19-0.65)

Inability to bend knee to 90? 0.43 (0.22-0.63)

Inability to bear weight 0.53 (0.34-0.73)

Overall 0.51 (0.32-0.71)

Interobserver agreement results displayed as ? values for the overall OKR and the subsets.

The overall interobserver agreement for the OKR was ? = 0.51 (95% CI, 0.32-0.71) and that for the PDR was ? = 0.71 (95% CI, 0.57-

0.86; Table 6).

Discussion

Clinical decision rules estimate the probability of disease (eg, fracture) and aid physicians in determining the need for a radiograph in the evaluation of injuries. Two clinical decision rules developed for knee injuries (OKR and PDR) were compared in this study. The developers of both rules presume a sensitivity of 100% in detecting knee fractures [3,4]. However, the authors who developed and assessed the PDR compared the diagnostic accuracy between the OKR and PDR and found a higher specificity for the PDR, with equal sensitivities [4,6]. Use of this rule would thus result in a larger reduction in unnecessary radiographs. No validation study has previously been performed by an independent research group. In addition to being the first independent study comparing the diagnostic accuracy and reproducibility of both rules, this study is also the first to assess the reproducibility of the OKR and PDR. In this cohort, the OKR and PDR showed an equal pooled sensitivity of 0.86 (95% CI, 0.57-0.96) and a pooled specificity of 0.27 (95% CI, 0.21-0.35) for the OKR and 0.51 (95% CI, 0.44-0.59) for the PDR. One fracture was detected in the absence of a positive finding for both rules. Interobserver agreement was moderate (? = 0.51) for the OKR and substantial (? = 0.71) for the PDR. The reason for the PDR to yield a higher ? coefficient can partly be explained by the fact that it contains only 2 findings, of which 1 (age) has an obvious perfect agreement, lifting the overall ? to a higher coefficient than the overall ? for the PDR. The OKR comprises 5 clinical findings, of which 1 (again age) has a perfect agreement, and as for the other 4, they constitute ? values in the moderate category. When compiling the overall ?, these 4 moderate findings influence the coefficient more than does the 1 moderate finding in the PDR.

This study has several clinical implications. The use of an equally

sensitive clinical decision rule with a higher specificity and a higher interobserver agreement, such as the PDR, could better aid physicians in the decision to order radiographs and to the ultimate detection of knee fractures. This could lead to a substantial reduction in unnecessary radiographs and thus a probable reduction in health care costs. In our opinion, the application of the PDR in acute knee injuries could lead to a greater efficiency in patient management, less unnecessary referrals to the ED by general practitioners, and a reduction in ED waiting times.

The trial may have had some limitations. Although the authors of the OKR and PDR suggest a sensitivity of 100% [4,6], we found a sensitivity of 86% for both decision rules. It is of great importance to yield a high sensitivity and not to miss fractures (false-negative results). In this cohort, one fracture was detected in the absence of a positive finding using both rules for the assessment of knee injuries. The detected fracture in the absence of a positive finding in both rules was a hairline fracture of the proximal tibial plateau, which was not visible (also by the radiologist) on the radiographs performed in the

Clinical finding ? (95% CI)

Age N 50 y	1.0 (1.00-1.00)
Full weight bearing	0.54 (0.36-0.72)
Overall	0.71 (0.57-0.86)

ED. The patient complained of persistent knee pain upon visiting the outpatient clinic 1 week later. A magnetic resonance imaging scan of the knee was performed, showing an intra-articular hairline fracture. Two months later, the patient experienced no complaints without any additional treatment. However, the reference standard was to perform a radiograph of the knee and patella, judged by a radiologist. In this case of a “missed fracture,” both rules were negative and the radiologist found no fracture on the x-ray. By maintaining the reference standard, the sensitivity would rise to 100%, as in previous studies. However, the rules are developed to detect fractures, and therefore, it seemed unjust to discard the magnetic resonance imaging-detected fracture, although the reference standard did not show it.

More research on the assessment of patients with knee injury with use of the PDR is recommended and should address cost-effectiveness of the PDR.

Conclusion

In this study, the PDR was found to be significantly more specific than the OKR, with equal sensitivity. Interobserver agreement was moderate for the OKR and substantial for the PDR.

References

1. Letsel Informatiesysteem 2006-2010, Consument en Veiligheid.
2. Stiell IG, Wells GA, McDowell I, Greenberg GH, McKnight RD, Cwinn AA, et al. Use of radiography in acute knee injuries: need for clinical decision rules. Acad Emerg Med 1995;2:966-73.
3. Stiell IG, Greenberg GH, Wells GA, McKnight RD, Cwinn AA, Cacciotti T, et al. Derivation of a decision rule for the use of radiography in acute knee injuries. Ann Emerg Med 1995;26(4):405-13.
4. Seaberg DC, Jackson R. Clinical decision rule for knee radiographs. Am J Emerg Med

   1994;12:541-3.

   Weber JE, Jackson RE, Peacock WF, Swor RA, Carley R, Larkin GL. Clinical decision rules discriminate between fractures and nonfractures in acute isolated knee trauma. Ann Emerg Med 1995;26(4):429-33.

5. Seaberg DC, Yealy DM, Lukens T, Auble T, Mathias S. Multicenter comparison of two clinical decision rules for the use of radiography in acute, high-risk knee injuries. Ann Emerg Med 1998;32(1):8-13.
6. Bauer SJ, Hollander JE, Fuchs SH, Thode Jr HC. A clinical decision rule in the evaluation of acute knee injuries. J Emerg Med 1995;13(5):611-5.
7. Stiell IG, Wells GA, Hoag RH, Sivilotti ML, Cacciotti TF, Verbeek PR, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA 1997;278(23):2075-9.
8. Simon LV, Matteucci MJ, Tanen DA, Roos JA, Riffenburgh RH. The Pittsburgh Decision Rule: triage nurse versus physician utilization in the emergency department. J Emerg Med 2006;31(3):247-50.
9. Stiell IG, Greenberg GH, Wells GA, McDowell I, Cwinn AA, Smith NA, et al. Prospective validation of a decision rule for the use of radiography in acute knee injuries. JAMA 1996;275(8):611-5.
10. Tigges S, Pitts S, Mukundan Jr S, Morrison D, Olson M, Shahriara A. External validation of the Ottawa Knee Rules in an urban trauma center in the United States. AJR Am J Roentgenol 1999;172(4):1069-71.
11. Emparanza JI, Aginaga JR. EstudioMulticentro en Urgencias de Osakidetza: Reglas de Ottawa (EMUORO) Group. Validation of the Ottawa Knee Rules. Ann Emerg Med 2001;38(4):364-8.
12. Jalili M, Gharebaghi H. Validation of the Ottawa Knee Rule in Iran: a prospective study. Emerg Med J 2010;27:849-85.
13. Bulloch B, Neto G, Plint A, Lim R, Lidman P, Reed M, et al. Pediatric emergency researchers of Canada. Validation of the Ottawa Knee Rule in children: a multicenter study. Ann Emerg Med 2003;42(1):48-55 Erratum in: Ann Emerg Med. 2004 Jan;43(1):142.
14. Bachmann LM, Haberzeth S, Steurer J, terRiet G. The accuracy of the Ottawa Knee Rule to rule out knee fractures. Ann Intern Med 2004;140:121-4.