a b s t r a c t

Objective: The objective of the study is to determine impact of a Clinical decision support (CDS) tool on document- ed adherence to the Ottawa Ankle Rules (OAR) and utilization and yield of ankle/foot radiography, for emergency department patients with acute ankle injury.

Methods: This is a before-and-after intervention study conducted at a 793-bed, quaternary care, academic hospi- tal from August 2012 to October 2013. Emergency department visits from adults with acute ankle injury 6 months before and 8 months after the intervention were included. The intervention embedded the OAR into a CDS tool integrated with a computerized physician Order entry system, which had data capture capability and provided feedback at the time of ankle/foot radiography order. Primary outcome was rate of documented adher- ence to OAR. Secondary outcomes were utilization and yield (clinically significant fracture rates among patients with acute ankle injuries) of ankle/foot radiography.

Results: The study population included 460 visits; 205 (44.6%) occurred preintervention. After intervention, doc- umented OAR adherence increased from 55.9% (229/410) to 95.7% (488/510; P b .001). Utilization remained sta- ble for ankle (77.5%; P = .800) and foot (48.6%; P = .514) radiography. Yield remained stable for ankle (17.8%; P = .891) and foot (19.8%; P = .889) radiography.

Discussion: Lack of documentation of key clinical data may hamper provider communication, delay care coordi- nation, and result in legal liability. By embedding the OAR into a CDS tool, we achieved the same rate of docu- mented adherence as previous onerous educational implementations while automating data collection/ retrieval. In summary, implementation of the OAR into a CDS tool was associated with an increase in documented adherence to the OAR.

(C) 2015

Introduction

A substantial gap persists between scientific knowledge and clinical practice [1], as it takes–on average–17 years for only 14% of new scien- tific evidence to be incorporated into practice [2]. In an effort to bridge

? Funding source: None.

?? Conflict of interest: Dr Khorasani is a consultant to Medicalis Corporation. Dr Khorasani is named on US Patent 6,029,138 held by Brigham and Women’s Hospital on clinical decision support-related software licensed to Medicalis Corporation in year 2000. As the result of licensing, Brigham and Women’s Hospital and its parent organiza- tion, Partners Healthcare, Inc, have equity and royalty interests in Medicalis.

* Corresponding author at: Center for Evidence Based Imaging, Department of Radiolo- gy, Brigham and Women’s Hospital, Harvard Medical School, 20 Kent Street, 2nd Floor, Boston, MA 02445. Tel.: +1 617 525 9705; fax: +1 617 525 7575.

E-mail addresses: [email protected], [email protected] (P.C. Silveira), [email protected] (I.K. Ip), [email protected] (S. Sumption), [email protected] (A.S. Raja), [email protected] (S. Tajmir), [email protected] (R. Khorasani).

this gap and decrease unnecessary health care costs, the US government has mandated the use of information technology solutions to promote the practice of evidence-based medicine [3]. The Health information technology for Economic and Clinical Health Act of 2009 required the implementation of clinical decision support (CDS) as part of certified electronic health records [4]. More recently, the Protecting Access to Medicare Act of 2014 mandates that, to be reimbursed after January 2017, clinicians ordering certain imaging procedures must consult ap- proved CDS for evidence-based appropriate use criteria [5]. At the same time, the Medicare Access and CHIP Reauthorization Act of 2015 will shift physician compensation from fee-for-service to pay-for- performance [6].

However, current methods to assess physicians’ performance in ad- hering to evidence or Guideline recommendations require labor- intensive and time-consuming medical chart review to collect key clin- ical data that may or may not have been documented. Integrating CDS and computerized physician order entry systems can automate

http://dx.doi.org/10.1016/j.ajem.2015.11.028

0735-6757/(C) 2015

the data capture and facilitate the retrieval of key data for clinical deci- sion making and quality measurement [7,8].

Acute ankle injury is a common presentation in the emergency de- partment (ED) [9]. The Ottawa Ankle Rules are validated, high- quality, and evidence-based clinical decision rules for imaging patients with suspected ankle fracture [10,11]. Before OAR implementation, Stiell et al [10,12] found that 83% to 93% of patients were referred for ankle radiography, but the prevalence of fracture was only 15%. If ap- plied, OAR can potentially save $3 million per 100000 patients annually in the United States without increasing the rate of Missed fractures [13]. The OAR is well known internationally, with more than 90% of emergen- cy physicians aware of the rules in the United States, Canada, and the United Kingdom [14]. However, less than one-third of US emergency physicians use the rules more than “most of the time” [14].

We aimed to determine whether physician-documented adherence to the OAR might increase after their inclusion into a CDS tool integrated with a CPOE system. We also reviewed the impact of this tool on the use and yield of ankle and foot radiography for patients presenting to the ED with acute ankle injury.

Methods

Study design and setting

Institutional review board approval was obtained for this Health Insurance Portability and Accountability Act-compliant before-and- after study performed at a 793-bed, quaternary care, urban academic hospital with an annual ED volume of approximately 60000 patients. The institutional review board waived the requirement for informed patient consent.

Population

The study population included all adult patients (age >=18 years old) who presented to the ED with acute (within 10 days) blunt ankle injury (eg, twisting injuries, falls from a height, direct blows, and motor vehicle accidents) during the period 6 months before and 8 months after the in- tervention. Ankle was broadly defined to include the malleolar (distal 6 cm of tibia, distal 6 cm of fibula, and talus) and midfoot (navicular, cu- boid, cuneiforms, anterior process of calcaneus, and base of fifth meta- tarsal) zones, matching the definition in the OAR derivation study [15]. We excluded patients who presented with atraumatic ankle pain, pain in other areas of the ankle/foot (eg, toes, calcaneous, or Achilles tendon), were pregnant, had isolated injuries of the skin (superficial lac- erations, abrasions, or burns), had been referred from outside the hospi- tal with radiographs, suffered their injury more than 10 days earlier, returned for reassessment of the same ankle injury, suffered multitrauma, had an open fracture, deformity, paraplegia, or change in mental status.

Intervention

The intervention, implemented on February 11, 2013, consisted of CDS based on the OAR integrated with a CPOE system, with Real-time feedback to providers at the time of an ankle or foot radiography order. The CDS was designed with data capture and accountability capa- bilities in addition to its educational purposes. Data from the CDS tool were stored in our institutional data warehouse. Three successive data capture screens determined the utility of the Imaging study according to the OAR (Fig. 1). If the utility of the study was low, an educational screen was displayed informing the provider of such, and they could choose to either cancel or proceed with the order (Fig. 2). If the advice was ignored and the provider decided to proceed with the order, a jus- tification screen was displayed, asking for the reason for the decision (Fig. 2).

Outcome measures

The primary outcome measure was the rate of documented adher- ence to the OAR, based on physician note documentation in free text format and structured data captured in the CDS. A decision to order (or not to order) a radiograph of the ankle or foot was considered adher- ent if it agreed with the OAR. For each patient presenting with acute ankle injury, the physician had to make 2 Imaging decisions (Image the ankle? Image the foot?), and therefore, the denominator for adher- ence was 2 times the number of visits. Secondary outcome measures were utilization (calculated as the no. of radiographies per total no. of visits for acute ankle injury) and clinically significant fracture rates (yield) of ankle radiography among patients with acute ankle injuries. Clinically significant fractures were defined as those greater than 3 mm in breadth, as previously defined by Stiell et al [9,10].

Data collection

All physician notes for ED visits occurring during the study period August 11, 2012, to October 11, 2013, were mapped to a list of ankle- related Systematized Nomenclature of Medicine Clinical Terms Concept Unique Identifiers (CUIs) (Appendix) using the clinical Text Analysis and Knowledge Extraction System [16], an open-source Natural Lan- guage Processing (NLP) system. The list of ankle-related CUIs was opti- mized for sensitivity. All patient visits with physician notes mapped to at least 1 ankle-related CUI were reviewed to assess for inclusion and exclusion criteria. To validate the NLP algorithm, all physician notes for ED visits from January 2014 (n = 1161) were also reviewed.

Explicit medical chart review was performed by a physician member of the study team (PCS [a research fellow]) using a structured form. If the patient visit matched the inclusion criteria, the following variables were collected: patient age, Patient sex, malleolar zone pain, midfoot zone pain, tenderness to palpation on the posterior or tip of lateral or medial malleollus, tenderness to palpation on the Fifth metatarsal base or navic- ular bone, inability to bear weight immediately after the injury and at the ED, ankle or foot radiography performed, ordering provider type (MD or PA), clinically significant fracture detected on ankle or foot radiography, and type of fracture. All ordering providers in our ED are authorized pro- viders (final signers of imaging orders) and either PAs or MDs.

The CDS tool captured the following data: study exclusion criteria,

onset of injury (whether within 10 days or not), tenderness to palpation on the posterior or tip of lateral or medial malleollus, tenderness to pal- pation on the fifth metatarsal base or navicular bone, inability to bear weight immediately after the injury and at the ED, and reason for ignor- ing the advice if ignored.

Statistical analysis and sample size

Statistical analysis was performed using commercially available soft- ware (JMP Pro 10; SAS Institute). To compare differences between the before and after intervention groups, we used ?² analysis for categorical variables and the Student t test for continuous variables. A 2-tailed P value of less than .05 was considered statistically significant. Changes in adherence to OAR over time were analyzed using a p-type statistical process control chart. Assuming a baseline adherence rate to the OAR be- fore intervention of 50% [7,17], a sample size of 334 patient visits was cal- culated based on 80% power, .05 ?, and a 0.15 proportion difference.

Results

Study cohort and NLP algorithm validation

A total of 67420 ED visits occurred during the study period, and 4016 (6.0%) included at least 1 ankle-related CUI. The NLP system for co- hort identification had a sensitivity of 100% and specificity of 97%. We excluded 3556 (5.3%) of visits identified by NLP as ankle related–2957

Fig. 1. Clinical decision support data capture screens for the OAR integrated in the CPOE system.

visits were not associated with an ankle injury, 182 were multitrauma, 116 were isolated injuries to the skin, 88 were referred with radio- graphs, 72 were injuries that happened more than 10 days previously, 49 were reassessment of the same injury, 27 were open fracture, 26 visits were associated with patients who had change in mental status, 20 were associated with patients who had deformity, 17 were associat- ed with pregnant patients, and 2 were associated with patients with paraplegia. The final study cohort included 460 ED visits due to acute blunt ankle injury (representing 0.7% of ED visits), from 457 unique pa- tients. Except for sex, there were no significant differences in patient- related characteristics between the before (n = 205) and after (n = 255) cohorts (Table 1). The prevalence of female patients was higher in the before cohort (142/205 [69.3%]) compared with the after (154/ 255 [60.4%]) cohort (P = .048). Twist was the most common mecha- nism of injury, accounting for 80.9% of ankle injuries.

Adherence to the OAR

Documented adherence to the OAR increased from 55.9% (229/410) before the intervention to 66.7% (340/510; P b .001) after the intervention when evaluating only information documented on physician notes in free

text format and to 95.7% (488/510; P b .001) after intervention when adding structured information retrieved from the CDS tool data capture. The statistical process control chart (Fig. 3) demonstrated that the impact of the intervention was immediate (the first data point after the interven- tion was above the upper control limit) and continued for the remaining 8 months after implementation for which data were analyzed (8 data points after the intervention were above the center line) [18].

Nonadherent imaging workup decisions represented only 4.3% (22/ 510) of the cohort after the intervention; all were decisions to proceed with an ankle (11/22 [50%]) or foot (11/22 [50%]) radiograph order. Clinical decision support screens did not fire appropriately due to a system error in 31.8% (7/22) of nonadherent imaging decisions. Documented adherence to the OAR after the intervention did not change significantly when excluding the7 orders where CDS did not fire appropriately (from 95.7% [488/510] to 97.0% [488/503]; P N .05). The ordering provider checked a box for one of the OAR exclusion criteria (inconsistent with information documented on the patient medical chart) at the time of order entry in 31.8% (7/22) of nonadherent imaging decisions. The recommendation on the education- al screen was ignored in 36.6% (8/22) of nonadherent imaging decisions; reasons cited were clinical judgment (8/22 [36.4%]), specialist recommen- dation (1/22 [4.5%]), or other (1/22 [4.5%]).

Fig. 2. Clinical decision support education and justification screens for low utility orders according to the OAR presented at the time of order entry.

Table 1

Characteristics of patients with acute ankle injury seen at the ED before and after the im- plementation of the OAR into the CDS system

	Before		After
Characteristics	n = 205		n = 255	P
Age, mean (SD), y	38.5 (15.9)		37.6 (15.9)	.539
Female ex	142 (69.3)		154 (60.4)	.048
Mechanism of injury Twist	174 (84.9)		198 (77.7)	.067
Fall from height	17 (8.3)		23 (9.0)
Direct blow	11 (5.4)		19 (7.5)
Motor vehicle accident	2 (1.0)	3 (1.2)
Other	1 (0.5)	12 (4.7)

Utilization

The utilization rate remained stable for ankle radiography (77.6% [357/460]; P = .839) and foot radiography (47.8% [220/460]; P =

.352) (Table 2) before and after intervention. The type of provider (MD or PA) ordering radiographs remained stable at 60% MDs and 40% PAs before and after the intervention (P = .911). Utilization rate after the intervention did not change significantly when excluding the 7 orders where CDS did not fire appropriately (from 77.6% [357/460] to 77.5% [354/457] for ankle radiographs; P N .05; and from 47.8% [220/460] to 47.6% [217/456] for foot radiographs; P N .05).

Yield

Malleolar zone pain	152 (74.2)	196 (76.9)	.500
Midfoot zone pain	88 (42.9)	111 (43.5)	.897	Yield remained stable for detecting clinically significant fractures on

Data are presented as number (percentage) of visits unless otherwise specified. Values in boldface are statistically significant.

ankle radiographs (12.3% [44/357]; P = .679) and foot radiographs (10.9% [24/220]; P = .332) (Table 2). Yield after the intervention did

Fig. 3. Control chart for ankle radiography and foot radiography adherence to the OAR before and after the implementation of a CDS system.

not change significantly when excluding the 7 orders where CDS did not fire appropriately (from 12.3% [44/357] to 12.1% [43/354] for ankle ra- diographs; P N .05; and from 10.9% [24/220] to 11.1% [24/217] for foot radiographs; P N .05).

Discussion

This is the first study of which we are aware to evaluate the impact on documented adherence, use, and yield of ankle radiography of em- bedding the OAR into a CDS tool. Lack of documentation of key clinical data may hamper provider communication, delay Care coordination, and even result in legal liability [19]. We found that documented adher- ence to the OAR increased by an absolute effect size of 39.8% and a rel- ative effect size of 71.2%, while not adversely affecting the use and yield of ankle radiography. This confirms findings by Bessen et al [20] that the use of an educational intervention increased documented OAR adherence from 57.5% to 94.7%. However, their intervention strat- egy involved a nonautomated process (educational tutorials, paper- based request form incorporating the OAR, reminders via posters and lanyard cards, Audit and feedback, and empowering radiographers to re- ject the old request forms or any incomplete new request forms). The type of intervention described by Bessen et al needs frequent active re- inforcement, which can be resource intensive in a large academic hospi- tal with constant change in trainee housestaff. It has also been shown that the effect of a traditional educational intervention decreases if only a minimal strategy (eg, posters alone) is maintained [21], whereas the effect of our information technology intervention continued for an

Table 2 Utilization and yield of ankle and foot radiography before and after the implementation of the OAR into the CDS system

	Before		After
	n = 205		n = 255	P
Utilization Ankle or foot	198/205 (96.6)		242/255 (94.9)	.379
Ankle	160/205 (78.1)		197/255 (77.3)	.839
Foot Yield	103/205 (50.2)		117/255 (45.9)	.352

additional 8 months after the intervention was implemented (data after 8 months were not evaluated). Moreover, paper-based data collec- tion and retrieval is onerous and time consuming. By embedding the OAR into a CDS tool, we were able to achieve the same rate of docu- mented adherence as Bessen et al while automating data collection and retrieval.

Implementing evidence-based CDS at the time of order entry has previously shown to improve appropriate use, decrease rate of utiliza- tion, and increase yield of imaging procedures [17,22]. However, our in- tervention had no significant impact on the utilization rate and yield of ankle radiography. A potential explanation for this finding is that order- ing providers were already using the OAR in their decision-making pro- cess before the intervention, but not documenting it appropriately. It is likely that, since the OAR was derived [15], validated [10], and imple- mented [12] more than 20 years ago, it is currently widely disseminated and adopted in clinical practice. No reduction in the use of ankle radiog- raphy from a baseline utilization rate of 73% was also found when the OAR was implemented in 10 Canadian hospitals in 1996 [23]. A survey study performed the following year found that 89% of Canadian physi- cians were already using the OAR frequently in their clinical practice [14]. Significant impact on the utilization rate of ankle radiographs was found in studies where the baseline utilization rate was more than 90% and postimplementation rates were higher than our baseline rate of 75% [20,21].

A previous study has shown that the use of health information tech- nology including a CPOE system, CDS, and natural language processing can be used to validate a clinical decision rule [24]. Our study demon- strates that these tools can also be used to implement evidence-based decision rules and guidelines, improve documentation of key clinical data, and facilitate measurement of physician performance. We have also demonstrated that adherence to the OAR improved in physician note documentation in free text form after the implementation of CDS, but it did not improve as much as when adding data captured by CDS.

Limitations

Our study has a number of limitations. First, it was performed in a single academic institution and may not be representative of the general

Ankle fracture	21/160 (13.1)	23/197 (11.7)	.679	population. Second, the data entered on the CDS capture screen at the
Foot fracture	9/103 (8.7)	15/117 (12.8)	.332	time of radiograph order entry might be incomplete or discordant

Data are presented as number (percentage) of visits.

with the patient actual presentation. However, it has been previously

shown that more than 90% of data entered by ordering providers into our imaging CDS are accurate [25]. Ordering providers checked a box for one of the OAR exclusion criteria in the CDS tool that was discordant with documentation in the free text physician note in only 2.7% of or- ders after our intervention. Third, because of a programming “bug,” the CDS did not fire appropriately in all eligible ankle and foot radio- graph orders. However, this only occurred in 2.7% of orders in the after intervention group. Recalculation of adherence, use, and yield rates excluding the 7 imaging orders where CDS did not fire appropri- ately demonstrated no significant difference in results, and therefore, the programming “bug” did not affect the conclusions of our study. In addition, because of our Observational study design, it is possible that part of the improvement that we found was due to other concurrent changes that are beyond our CDS; however, the immediate increased adherence observed in the quarter after CDS implementation suggests that CDS played a significant role in the change process.

Conclusion

Implementation of the OAR into a CDS tool automated the data cap- ture and retrieval of key clinical data and was associated with an in- crease in documented adherence to OAR. Use and yield of ankle and foot radiography were not adversely affected.

Acknowledgments

The authors would like to thank Laura E. Peterson, BSN, SM, for her assistance in editing this manuscript.

Appendix

eTable

Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT) Concept Unique Identifiers (CUIs) used to identify patients with acute foot or ankle complaints

CUI description	CUI
Ankle pain	C0238656
Foot pain	C0016512
Ankle joint–painful on movement	C0576209
Ankle joint finding	C0576176
Arthralgia of the ankle and/or foot	C0423658
Closed bimalleolar fracture	C0392611
Closed fracture ankle, bimalleolar, high fibular fracture	C0435910
Closed fracture ankle, bimalleolar, high fibular fracture	C0435910
Closed fracture ankle, bimalleolar, low fibular fracture	C0435909
Closed fracture ankle, bimalleolar, low fibular fracture	C0435909
Closed fracture ankle, trimalleolar, high fibular fracture	C0435914
Closed fracture ankle, trimalleolar, low fibular fracture	C0435913
CUI description	CUI
Closed fracture dislocation, ankle joint	C0434937
Closed fracture dislocation, ankle joint	C0434937
Closed fracture dislocation, metacarpophalangeal joint	C0434910
Closed fracture distal tibia	C0435884
Closed fracture distal tibia, extra-articular	C0435885
Closed fracture distal tibia, intra-articular	C0435886
Closed fracture of ankle	C0272769
Closed fracture of distal tibia and Distal fibula	C2919470
Closed fracture of medial malleolus	C0435890
Closed fracture subluxation, ankle joint	C0438595
Closed fracture subluxation, ankle joint	C0438595
Closed trimalleolar fracture	C0159883
Contusion, ankle and foot, excluding toe(s)	C0274237
Crush injury, ankle and foot, excluding toe(s)	C0433116
Dislocation of ankle	C0434691
Dislocation of foot	C0434694
Finding of ankle or foot	C1631064
Finding of ankle region	C0576175
First metatarsophalangeal joint pain	C0458235
Foot joint–painful on movement	C0576274
Foot joint pain	C0458239
Fracture of ankle	C0159877
Fracture of medial malleolus	C0555345

(continued on next page)

eTable (continued)

CUI description	CUI
Injury of ankle	C0085111
Injury of foot	C0149697
Lesser metatarsophalangeal joint pain	C0458236
Metatarsophalangeal joint pain	C0458240
Sprain of ankle	C0160087
Sprain of ankle and/or foot	C0434427
Sprain of ankle grade I	C3662211
Sprain of ankle grade II	C3662209
Sprain of ankle grade III	C3662210
Sprain of deltoid ligament of ankle	C0160089
Sprain of foot	C0160093
Sprain of lateral ligament of ankle joint	C0435052
Sprain of ligament of tarsometatarsal joint	C0272905
Sprain of midtarsal joint	C0434458
Sprain, ankle joint, lateral	C0434477
Sprain, ankle joint, medial	C0160089
Sprain, metatarsophalangeal joint	C0160096
Sprain, tarsometatarsal joint	C0272905
Talonavicular joint pain	C0458238
Traumatic arthropathy of first metatarsophalangeal joint	C0409734
Traumatic arthropathy of lesser metatarsophalangeal joint	C0409733
Traumatic arthropathy of talonavicular joint	C0409736
Traumatic arthropathy of the ankle and/or foot	C0409754
Traumatic arthropathy-ankle	C0409738
Trimalleolar fracture	C0159883
Foot joint finding	C0576240
Finding of foot region	C0576221
Finding of movement of foot	C0576232
Finding of mobility of foot	C0576234
Finding of ankle or foot	C1631064
Contusion of foot	C0160955
Ankle region structure	C0003086
Ankle region structure	C1261192
Crushing injury of ankle	C0160994
Closed crush injury, ankle	C0433118
Crushing injury of foot	C0160993
Closed crush injury, foot	C0433117
Contusion of ankle	C0160956
Structure of ankle and/or foot	C1690938
Foot structure	C0016504
Fracture of navicular	C0435939
Closed fracture of navicular bone of foot	C0435940
Closed fracture of talus	C0159892
Fracture of talus	C0347813
Swollen ankle	C0235439
Swollen feet	C0574002
Foot joint swelling	C0576241
Swelling of ankle joint	C0576177
Symptom of ankle	C2127215
Bony swelling of ankle joint	C0576178
Closed fracture dislocation of metatarsophalangeal joint (disorder)	C0578708
Closed fracture dislocation, midtarsal joint (disorder)	C0434942
Closed fracture dislocation, tarsometatarsal joint (disorder)	C0434943
Closed fracture of cuboid bone of foot (disorder)	C0347815
Closed fracture of cuneiform bone of foot (disorder)	C0435924
Closed fracture of foot (disorder)	C0272775
Closed fracture of Lateral malleolus (disorder)	C0435892
Closed fracture of metatarsal bone (disorder)	C0435944
Closed fracture of phalanx of foot (disorder)	C0578705
Closed fracture subluxation, metatarsophalangeal joint, single (disorder)	C0438601
Closed fracture subluxation, subtalar joint (disorder)	C0438598
Closed fracture subluxation, tarsometatarsal joint (disorder)	C0438600
Closed fractures of tarsal AND metatarsal bones (disorder)	C0272791
Closed tarsal fractures, multiple (disorder)	C0435918
Closed Traumatic dislocation ankle joint (disorder)	C0434692
Closed traumatic dislocation of joint of foot (disorder)	C0434696
Closed traumatic dislocation of metatarsal joint (disorder)	C0272856
Closed traumatic dislocation of metatarsophalangeal joint (disorder)	C0159997
Closed traumatic dislocation of tarsal joint (disorder)	C0272855
Closed traumatic dislocation of tarsometatarsal joint (disorder)	C0159995
Closed traumatic dislocation, midtarsal joint (disorder)	C0159994
Closed traumatic subluxation, subtalar joint (disorder)	C0434819
Complete tear of midtarsal joint ligament (disorder)	C0435097
Complete tear, ankle and/or foot ligament (disorder)	C0435095
Complete tear, foot ligament (disorder)	C0435096
Fracture of one or more tarsal and metatarsal bones (disorder)	C1963546

(continued on next page)

eTable (continued)

CUI description CUI

multiple fractures of foot (disorder) C0452095

Partial tear, ankle, lateral ligament (disorder) C0435140

Peroneus longus rupture (disorder) C0410100

Rupture of tendon of foot region (disorder) C0343220

Rupture of tendon of lower leg and ankle (disorder) C0410098

Ruptured Achilles tendon–traumatic (disorder) C0555311

Sprain of calcaneofibular ligament (disorder) C0272894

Sprain of distal tibiofibular ligament (disorder) C0160091

Sprain, extensor tendon, foot (disorder) C0434305

Sprain, flexor tendon, foot (disorder) C0434306

Sprain, plantaris tendon (disorder) C0434307

Strain of Achilles tendon (disorder) C0272895

Strain of peroneal tendon (disorder) C0434333

References

1. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635-45. http://dx.doi.org/10.1056/NEJMsa022615.
2. Westfall JM, Mold J, Fagnan L. Practice-based research–“blue highways” on the NIH roadmap. JAMA 2007;297:403-6. http://dx.doi.org/10.1001/jama.297.4.403.
3. Khorasani R, Hentel K, Darer J, Langlotz C, Ip IK, Manaker S, et al. Ten command- ments for effective clinical decision support for imaging: enabling evidence-based practice to improve quality and reduce waste. Am J Roentgenol 2014;203:945-51. http://dx.doi.org/10.2214/AJR.14.13134.
4. HITECH Act Enforcement Interim Final Rule. http://www.hhs.gov/ocr/privacy/hipaa/ administrative/enforcementrule/hitechenforcementifr.html. [accessed April 21, 2015].
5. Text of H.R. 4302: Protecting Access to Medicare Act of 2014 (Passed Congress/En- rolled Bill version). GovTrack.us https://www.govtrack.us/congress/bills/113/ hr4302/text. [accessed April 21, 2015].
6. Burgess M. Text–H.R.2–114th Congress (2015-2016): Medicare Access and CHIP Reauthorization Act of 2015. https://www.congress.gov/bill/114th-congress/ house-bill/2/text. [accessed April 22, 2015].
7. Gupta A, Ip IK, Raja AS, Andruchow JE, Sodickson A, Khorasani R. Effect of clinical de- cision support on documented guideline adherence for head CT in emergency de- partment patients with mild traumatic brain injury. J Am Med Inform Assoc 2014. http://dx.doi.org/10.1136/amiajnl-2013-002536.
8. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to mea- sure documented adherence to a national imaging quality measure. Acad Radiol 2014;21:378-83. http://dx.doi.org/10.1016/j.acra.2013.10.017.
9. Stiell I, Wells G, Laupacis A, Brison R, Verbeek R, Vandemheen K, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle inju- ries. BMJ 1995;311:594-7. http://dx.doi.org/10.1136/bmj.311.7005.594.
10. Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Reardon M, et al. Deci- sion rules for the use of radiography in acute ankle injuries. Refinement and pro- spective validation. JAMA 1993;269:1127-32.
11. Bachmann LM, Kolb E, Koller MT, Steurer J, ter Riet G. Accuracy of Ottawa ankle rules to exclude fractures of the ankle and mid-foot: systematic review. BMJ 2003;326:

    417. http://dx.doi.org/10.1136/bmj.326.7386.417.

    Stiell IG, McKnight RD, Greenberg GH, McDowell I, Nair RC, Wells GA, et al. Imple- mentation of the Ottawa ankle rules. JAMA 1994;271:827-32.

12. Anis AH, Stiell IG, Stewart DG, Laupacis A. cost-effectiveness analysis of the Ottawa Ankle Rules. Ann Emerg Med 1995;26:422-8.
13. Graham ID, Stiell IG, Laupacis A, McAuley L, Howell M, Clancy M, et al. Awareness and use of the Ottawa ankle and knee rules in 5 countries: can publication alone be enough to change practice? Ann Emerg Med 2001;37:259-66. http://dx.doi. org/10.1067/mem.2001.113506.
14. Stiell IG, Greenberg GH, McKnight RD, Nair RC, McDowell I, Worthington JR. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med 1992;21:384-90.
15. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clin- ical Text Analysis and Knowledge Extraction System (cTAKES): architecture, compo- nent evaluation and applications. J Am Med Inform Assoc 2010;17:507-13. http:// dx.doi.org/10.1136/jamia.2009.001560.
16. Raja AS, Ip IK, Prevedello LM, Sodickson AD, Farkas C, Zane RD, et al. Effect of com- puterized clinical decision support on the use and yield of CT pulmonary angiogra- phy in the emergency department. Radiology 2012;262:468-74. http://dx.doi.org/ 10.1148/radiol.11110951.
17. Cheung YY, Jung B, Sohn JH, Ogrinc G. quality initiatives: statistical control charts: simplifying the analysis of data for quality improvement. Radiographics 2012;32: 2113-26. http://dx.doi.org/10.1148/rg.327125713.
18. Salazar L, Best TM, Hiestand B. Incomplete documentation of elements of Ottawa Ankle Rules despite an electronic medical record. Am J Emerg Med 2011;29: 999-1002. http://dx.doi.org/10.1016/j.ajem.2010.05.012.
19. Bessen T, Clark R, Shakib S, Hughes G. A multifaceted strategy for implementation of the Ottawa ankle rules in two emergency departments. BMJ 2009;339. http://dx.doi. org/10.1136/bmj.b3056.
20. Auleley G, Ravaud P, Giraudeau B, Kerboull L, Nizard R, Massin P, et al. Implementa- tion of the Ottawa ankle rules in France: a multicenter randomized controlled trial. JAMA 1997;277:1935-9. http://dx.doi.org/10.1001/jama.1997.03540480035035.
21. Goldzweig CL, Orshansky G, Paige NM, Miake-Lye IM, Beroes JM, Ewing BA, et al. Electronic health record-based interventions for improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med 2015;162: 557-65. http://dx.doi.org/10.7326/M14-2600.
22. Cameron C, Naylor CD. No impact from active dissemination of the Ottawa Ankle Rules: further evidence of the need for local implementation of practice guidelines. CMAJ 1999;160:1165-8.
23. Silveira PC, Ip IK, Goldhaber SZ, Piazza G, Benson CB, Khorasani R. Performance of Wells score for deep vein thrombosis in the inpatient setting. JAMA Intern Med 2015. http://dx.doi.org/10.1001/jamainternmed.2015.1687.
24. Gupta A, Raja AS, Khorasani R. Examining clinical decision support integrity: is clini- cian self-reported data entry accurate? J Am Med Inform Assoc 2014;21:23-6. http://dx.doi.org/10.1136/amiajnl-2013-001617.