a b s t r a c t

Objective: Operational data are often used to make systems changes in real time. Inaccurate data, however, transiently, can result in inappropriate operational decision making. Implementing electronic health records (EHRs) is fraught with the possibility of data errors, but the frequency and magnitude of transient errors during this fast-evolving systems upheaval are unknown. This study was done to assess operational data quality in an emergency department (ED) immediately before and after an EHR implementation.

Methods: Direct observations of standard ED timestamps (arrival, bed placement, clinician evaluation, Disposition decision, and exit from ED) were conducted in a suburban ED for 4 weeks immediately before and 4 weeks after EHR implementation. Direct observations were compared with electronic timestamps to assess data quality. Differences in proportions and medians with 95% confidence intervals (CIs) were used to estimate the magnitude of effect.

Results: There were 260 observations: 122 before and 138 after implementation. We found that more systematic data errors were introduced after EHR implementation. The proportion of discrepancies where the observed and electronic timestamp differed by more than 10 minutes was reduced for the Disposition timestamp (29.3% vs 16.1%; difference in proportions, -13.2%; 95% CI, -24.4% to -1.9%). The accuracy of the clinician-evaluation timestamp was reduced after implementation (median difference of 3 minutes earlier than observed; 95% CI, -5.02 to -0.98). Multiple service intervals were less accurate after implementation. Conclusion: This single-center study raises questions about operational data quality in the peri- implementation period of EHRs. Using electronic timestamps for operational assessment and decision making following implementation should recognize the magnitude and compounding of errors when computing service times.

(C) 2013

Introduction

In medicine, the focus on data integrity has primarily been driven by the needs of research rather than clinical operations. Medical records intended for patient care require rigorous methods of chart review to obtain accurate and repeatable results [1,2]. Yet, the same methods are rarely applied when collecting data from medical charts for the purposes of clinical operations. The advent of the electronic health record has facilitated research by dramatically increas- ing the quantity of data available [3], including data pertaining to operational effectiveness. Spurred on by the ability to collect and use

? Grants/financial support: Dr Ward is funded by a research fellowship grant from the Emergency Medicine Foundation. In addition, this study was supported by an unrestricted educational grant from the Catholic Health Partners.

* Corresponding author. Tel.: +1 513 558 1045 (office); fax: +1 513 558 5791.

E-mail address: [email protected] (M.J. Ward).

data as well as financial incentives and the promise of improvED patient care, EHRs are increasingly being implemented [4].

In the emergency department (ED), data are frequently used to assess operational performance. quality initiatives to improve care for time-critical medical conditions and health systems changes to address issues such as persistent patient crowding make use of the array of information available in the medical record [5]. Electronic timestamps are commonly used as a proxy for the time an event actually occurred. Inherent in this is the acceptance of some level of error. However, whether this error is minimized and whether it is systematic are really unknown. Failure to recognize the magnitude of random error and persistence of systematic error can lead to error- prone conclusions and poor reproducibility of measurements, both of which undermine the benefits of these major information system implementations [6,7].

There are already concerns about the integrity of data in EHRs, with systematic errors being observed [8]. During the upheaval sur- rounding implementation of an EHR, it is possible that data integrity

0735-6757/$ - see front matter (C) 2013 http://dx.doi.org/10.1016/j.ajem.2013.03.027

is further diminished. Standard processes and reporting may be changed, causing the ED to be particularly vulnerable to compromised data integrity. Because this is a period when systems and processes are rapidly adapting to a whole new workflow, it is imperative that real-time use of operational data be cognizant of the possibility of error. Without understanding the errors, direction of resources toward mitigating operational disruption and threats to patient safety may be misplaced. Moreover, incorrect data can lead to erroneous assessments of both the benefits and harms of EHRs.

The primary goal of this investigation was to assess the accuracy of operational data collected during the process of EHR implementation at a single ED.

Methods

Study design, setting, and selection of participants

The hospital studied is a 200-bed, suburban, academic ED with 34000 patient visits annually. There are 24 designated ED beds, with 14 flexible-capacity beds available. Before EHR implementation, this ED used a combination of a paper chart for orders and Horizon Emergency Care ED Track Board (McKesson Corporation, San Francisco, CA). On June 12, 2011, the ED implemented EPIC ASAP (Epic Systems Corporation, Verona, WI) as one component of a simultaneous, hospital-wide EHR implementation.

After receiving institutional review board approval, data were obtained by direct observation and by extraction from the EHR. Observational data were collected by clinical study assistants (CSAs), who were trained to collect data using a case report form with an accompanying data dictionary using standardized operational defi- nitions of timestamps [9]. A definition for the source of each timestamp is provided in Table 1 before and after EHR implemen- tation. These time-and-motion observations were conducted over 4 weeks immediately before EHR implementation (baseline) and over

4 weeks immediately after EHR implementation. Patients seeking care in the ED older than 18 years and not pregnant were approached for consent upon arrival, and consenting patients were tracked until exiting the ED. Although not consecutive, CSA shifts were assigned to cover all hours of the day and days of the week to ensure an appropriate representation of patient demographics and operational patterns. Clinical study assistants were limited to observing up to 5 subjects at a time to ensure that they were able to accurately capture timestamps. This was determined through practice runs before commencing the study. Once a patient was in their bed and enrolled in the study and the CSA felt comfortable with observing an additional subject, new subjects were approached for

enrollment. New subjects were identified by standing between the emergency medical services entrance and ED lobby to observe for the next entrance. Clinical study assistants noted entrance times and subsequently approached subjects for potential enrollment. No identifiable information was collected until the subject agreed to participate in the study.

Demographic information and the following timestamps were collected on paper forms to the nearest second using stopwatches calibrated to the US Naval Observatory Master Clock: arrival, bed placement, initial clinician evaluation, disposition, and exit from the ED. These timestamps represent reproducible, highly scrutinized measures across multiple EDs [9]. Data were double entered into an electronic spreadsheet by CSAs, and entries were compared for accuracy. Electronically captured data for the same 5 timestamps were extracted weekly from the patient Tracking system in use before implementation and from the EHR after implementation.

Data analysis

Electronically captured data were initially evaluated for logic- based errors (eg, timestamps not following the expected care sequence, such as discharge occurring before admission) and for evidence of systematic errors or biases in timestamps caused by the implementation of the EHR that introduce inaccuracies into the electronic data (eg, identical exit timestamp values for all discharged patients). The electronically captured data were also reviewed for missingness of timestamps and demographic data.

Median service intervals (the elapsed duration between 2 time- stamps) were calculated using the sets of observed and electronic data: arrival-bed, bed-clinician, clinician-disposition, disposition- exit, and arrival-exit (ie, total length of stay). The differences between manually collected and electronically captured service intervals for timestamps and service intervals were calculated. The proportion of discrepancies, defined as an absolute differences greater than 10 minutes between electronically and manually collected data, was estimated. Although there is no standard threshold for what magnitude of discrepancies is significant, the difference of 10 minutes was chosen as clinically and operationally meaningful and appropriately dismissive of the trivial differences (ie, 1 or 2 minutes).

Because the discrepancies were categorized into binary data based upon a binominal outcome of either greater than or less than 10 minutes from observed values, we used ?² and Fisher exact tests to evaluate the difference in proportions of discrepancies between before and after EHR implementation. In addition, we calculated the median differences and the difference in proportions with 95%

Table 1

Definitions of timestamps before and after electronic health record implementation and the triggers used for observation and electronic documentation

Timestamp	Observation trigger preimplementation	Observation trigger Postimplementation	Electronic trigger preimplementation	Electronic trigger postimplementation
Arrival	Date and time that patient first presents to the ED.	Same	Same-documented electronically	Same-documented electronically
Bed placement Clinician	Date and time that patient is physically located within a room for the first time. The first date and time a clinician (physician	Same Same	Same-documented electronically Same-documented	Same-documented electronically Same-documented electronically
evaluation Disposition	or midlevel provider) interviews a patient either in triage or in a room. The date and time that a physician documents	Same except documented as an	electronically Same-documented	Time that admitting physician
decision	in the paper chart to discharge a patient or provides a verbal order to admit the patient	admission or discharge order by the clinician in the electronic	electronically	completes admission orders in electronic health record. Discharge
Exit from ED	to the hospital. The date and time the patient	health record. Same	Same-documented	disposition is the same. Same-documented electronically
	physically leaves their room after being admitted or discharged.		electronically

Table 2

Demographic characteristics of patients observed before/after EHR implementation

	Preimplementation (n = 122)		Postimplementation (n = 138)
Mean age (SD), y Sex (n, %) Female	47 (20) 75	61.5%	45 (18) 82	59.4%
Race (n, %) White	78	63.9%	93	67.4%
African American	41	33.6%	39	28.3%
Other/not documented	3	2.5%	6	4.3%
Acuity (n, %) Critical	0	0.0%	0	0.0%
Emergent	3	2.5%	4	2.9%
Urgent	79	64.8%	82	59.4%
Less urgent	40	32.8%	50	36.2%
Nonurgent	0	0.0%	2	1.4%
Insurance status (n, %) Commercial insurance	41	33.6%	51	37.0%
Government (Medicare/Medicaid)	50	41.0%	38	27.5%
Self-pay	29	23.8%	33	23.9%
Not documented	2	1.6%	16	11.6%
Mode of arrival (n, %) EMS	9	7.4%	13	9.4%
Disposition location (n, %)
Admit	33	27.0%	18	13.0%

Abbreviation: EMS, emergency medical services.

confidence intervals (CIs) to quantify the magnitude of effects. All statistical analyses were conducted using SPSS 21.0 (IBM Corporation, Armonk, NY).

Results

There were 260 patients observed through their ED visit: 122 before implementation and 138 after. Subjects were similar before and after EHR implementation. However, before implementation, there were more admitted patients and more patients with govern- ment insurance than after implementation, and there were more patients with undocumented insurance before implementation than after implementation (Table 2). Missing demographic data were manually taken from observed data where possible.

Four examples of systematic errors were discovered during the evaluation of the electronically captured data. First, 16 subjects did not have a disposition decision timestamp documented. Fifteen of these subjects were in the preimplementation phase. Second, for 8 days after implementation, all discharged patients (32 timestamps) had exit timestamps electronically recorded as 23:59. Third, arrival

and bed timestamps were identical for all admitted patients (50 timestamps) for the first 12 days postimplementation. Fourth, 18 patients or 13% of the postimplementation sample were manually observed, but no electronic data were available for these subjects. Unlike the other systematic errors, no predictable pattern was identified, and these cases simply appeared to have been dropped from the EHR throughout the entire postimplementation study period. For all systematic errors, the hospital information technology team was notified. For analysis, these timestamps were designated as missing because including these systematic errors would have systematically biased the differences between manually and electro- nically captured data.

We also identified that the disposition decision timestamp was differently defined before and after implementation. The definition used in this study was consistent with the literature: the time an order is placed to admit, discharge, or transfer a patient [9]. Before imple- mentation, clinicians would either verbally tell a unit clerk or docu- ment on a paper chart to disposition a patient. After implementation, the disposition decision timestamp was identical for discharged patients. However, for admitted patients, disposition decision was

Table 3

Comparison of the proportion of significant discrepancies, defined as a difference between the electronic relative to observed values of greater than 10 minutes before and after EHR implementation

preimplementation period (n = 122)

postimplementation period (n = 138)

Difference in proportions (95% CI) P value

(10 min %)^a

Total n N10 min (n, %) Total n N 10 min (n, %)

Timestamps Arrival	119	6	5.0%	122	4	3.3%	-1.8%	(-7.6% to 3.8%)	.536
Bed placement	120	8	6.7%	72	2	2.8%	-3.9%	(-10.2% to 3.6%)	.325
Clinician evaluation	119	30	25.2%	120	32	26.7%	1.5%	(-9.6% to 12.5%)	.797
Disposition decision	99	29	29.3%	112	18	16.1%	-13.2%	(-24.4% to -1.9%)	.021
Exit from ED	104	20	19.2%	89	13	14.6%	-4.6%	(-15.1% to 6.3%)	.395
Service intervals Arrival-to-bed	119	5	4.2%	72	8	11.1%	6.9%	(-0.6% to 16.5%)	.066
Bed-to-clinician	119	33	27.7%	71	27	38.0%	10.3%	(-3.2% to 24.0%)	.140
Clinician-to-disposition	99	39	39.4%	110	42	38.2%	-1.2%	(-14.2% to 11.8%)	.857
Disposition-to-exit	91	27	29.7%	102	22	21.6%	-8.1%	(-20.3% to 4.2%)	.197
Total length of stay	103	25	24.3%	89	20	22.5%	-1.8%	(-13.6% to 10.3%)	.769

Difference in proportions and 95% CIs are displayed.

^a If an adjustment for multiple comparisons were to be made, a conservative critical P value would be .005.

Table 4

Comparison of electronic with observed values for both timestamps and calculated service intervals

Preimplementation		Postimplementation			Median difference
	n	Median		n	Median	(95% CI)
Timestamps Arrival	119	0	122		0	0 (-0.94 to 0.94)
Bed placement	120	-1	72		0	1 (0.33 to 1.67)
Clinician evaluation	119	-1	120		-4	-3 (-5.02 to -0.98)
Disposition decision	99	0	112		0	0 (0-0)
Exit from ED Service intervals Arrival-to-bed	104 119	2 -1	89 72		1 -2	-1 (-1.94 to -0.06) -1 (-2.49 to 0.49)
Bed-to-clinician	119	-1	71		-5	-4 (-7.59 to -0.41)
Clinician-to-disposition	99	4	110		6	2 (0.94-3.06)
Disposition-to-exit	91	1	102		0	-1 (-1.48 to -0.52)
Total length of stay	103	1	89		-1	-2 (-2.47 to -1.53)

A median difference and 95% CI are provided. Negative values indicate electronic data capture occurred before observation. All times are in minutes.

redefined by information systems as the time the admitting physician’s orders were placed, instead of the time the ED physician placed the order.

Before implementation, all timestamps and service intervals in- cluded some discrepancies (Table 3). The timestamps with the greatest errors were the disposition decision, clinician evaluation, and exit timestamps, with discrepancies of 29%, 25%, and 19%, respectively. After implementation, the proportion of discrepancies decreased for disposition decision (29.3% vs 16.1%; difference in proportion, -13.2%; 95% CI, -24.4% to -1.9%; P = .021). No service intervals had significantly reduced discrepancies.

We also evaluated the accuracy of electronic data compared with observed data after implementation (Table 4). Clinician evaluation had a larger difference after implementation (-1 vs -4 minutes; median difference, -3 minutes; 95% CI, -5.02 to -0.98). However, three service intervals had increased differences. Arrival-bed (-1 vs

-2 minutes; median difference, -1 minute; 95% CI, -2.49 to 0.49), bed-clinician (-1 to -5 minutes; median difference, -4 minutes; 95% CI, -7.59 to -0.41), and clinician-disposition (+4 to +6 minutes; median difference, +2 minutes; 95% CI, 0.94-3.06) were all

increased after implementation (Fig.). There were no obvious trends in week-to-week data accuracy for timestamps identified.

Discussion

Our study identified ways that operational data changed and several ways that accuracy was affected by the EHR implementation. We found that immediately after implementing a new EHR, the number of systematic errors increased and resulted in the loss of data. In addition, small inaccuracies in the electronically captured time- stamps resulted in a worsening of multiple calculated service inter- vals. Results from this study of an ED undergoing EHR implementation underscore the importance of evaluating the quality of electronic data during the EHR implementation period.

Although systematic errors existed before EHR implementation, transient programming errors in the new EHR appear to have con- tributed to the frequency and nature of these errors. This was particularly evident after implementation where bed and arrival timestamps were identical, and exit timestamps were set to 23:59. In discussions with local implementation staff, all of these errors

Fig. Whisker box plots of the differences between electronic and observed data values for timestamps (A) and service intervals (B). White boxes represent differences before electronic health record implementation, and gray boxes represent the period after electronic health record implementation. The boxes represent the interquartile range; the bar within the boxes represents the median value. The whiskers represent the limit for outliers, shown as circles (o), defined as values falling between 1.5 and 3 box lengths from the end of the box. Cases shown as stars (*) are extreme values, defined as greater than 3 box lengths from the end of the box. Note: To display extreme values while providing sufficient details around the normative values, the display of the vertical axis has been transformed logarithmically in both the positive and negative directions. Note that this transformation of the axis units is only for visualization purposes; the data values themselves were not transformed.

occurred because simultaneous upgrades in the registration system and EHR unknowingly conflicted until identified after implemen- tation. Being aware of such a possible conflict and looking for systematic errors early in the implementation process can prevent the loss (and potential misuse) of operational data. Without the iden- tification and removal of these data, operational statistics would have showed slower throughput postimplementation, giving a false sense of operational degradation. In addition, further investigation revealed that a database query caused 13% of the electronic records after implementation to appear as missing when they really existed in the EHR. Hospital-wide reporting for decision-making purposes and public reporting did not include operational data for 13% of patients during this postimplementation period. Such missing data have significant operational, financial, and medico-legal ramifications, especially if this sample translates to the general population. These data were not included in this article’s results because they were used to report hospital performance publicly and our work helped to identify the missing data.

Clinician evaluation was the only timestamp that was less accurate after implementation, whereas 2 other timestamps (bed placement and exit from the ED) showed improved accuracy (Table 4). Yet, when evaluating the service intervals, 3 of the intervals were less accurate: arrival-bed, bed-clinician, and clinician-disposition reflect- ing the downstream effects of data inaccuracies. This was further compounded by the redefinition of disposition decision as the time admitting patient orders are completed, which had the effect of prolonging the clinician-disposition interval and shortening the disposition-exit interval. Accurate interval measurement is vital to quality improvement efforts. The integrity of the data, definitions of the timestamps, and the intervals that result must be considered before deciding on actions designed to improve system performance. In addition, as intervals are targeted for process improvement and, ultimately, as drivers of reimbursement, the possibility that improve- ments in one interval are offset by worsening performance in another must be taken into account. This finding also suggests that rede- fining operational timestamps can lead to inappropriate conclusions if not properly identified and interpreted a priori.

This study should be viewed as a launching point for a series of future investigations. Two primary questions arising are the fol- lowing: (i) how widespread are the data errors and (ii) for how long do they persist after EHR implementation? Based on our findings, we suspect that errors are pervasive and potentially long lasting. If this is true, the implications are extensive, raising questions about the assessment of health care system performance and im- provements to operational throughput. If data integrity issues ex- tend beyond timestamps and into clinical data, they could impact health care at the individual level and diminish research results derived from the increasing number of clinical data warehouses. We

posit that insufficient attention is paid to data integrity in EHRs, which undermines many of the benefits they may bring to a health care facility.

Limitations

This research has some limitations. First, this is a single-center study conducted with a limited number of observations. Additional observations with multiple observers and a larger number of patients, EHR vendors, and clinical settings are needed to determine the true extent of this problem. In addition, because our observations were focused on the period immediately surrounding an EHR implemen- tation, it is unclear how our results might generalize to steady state operations. We would expect some of the systematic errors due to programming to be resolved. However, bias in timestamps and more random errors would potentially remain an issue.

Conclusion

This study shows that, in the immediate period following a typical EHR implementation, systematic errors and inaccuracies are pervasive in EHR operational data. Administrators and policymakers should be aware of such possibilities during EHR implementation and should attempt to minimize the deleterious effects of EHR implementation on data integrity.

References

1. Gilbert EH, Lowenstein SR, Koziol-McLain J, Barta DC, Steiner J. Chart reviews in emergency medicine research: where are the methods? Ann Emerg Med 1996;27: 305-8.
2. Worster A, Bledsoe RD, Cleve P, Fernandes CM, Upadhye S, Eva K. Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med 2005;45:448-51.
3. Dean BB, Lam J, Natoli JL, Butler Q, Aguilar D, Nordyke RJ. Review: use of electronic medical records for health outcomes research: a literature review. Med Care Res Rev 2009;66:611-38.
4. Hsiao CJ, Hing E, Socey TC, Cai B. electronic health record systems and intent to apply for meaningful use incentives among office-based physician practices: United States, 2001-2011. Hyattsville, MD: National Center for Health Statistics; 2011.
5. National Ambulatory Medical Care Survey. Summary Tables. Hyattsville, MD: National Center for Health Statistics; 2008.
6. Gattiker TF, Goodhue DL. What happens after ERP implementation - Understanding the impact of interdependence and differentiation on plant-level outcomes (vol 29, pg 559, 2005). Mis Quart 2005;29:777.
7. Vosburg J, Kumar A. Managing dirty data in organizations using ERP: lessons from a case study. Ind Manage Data Syst 2001;101:21-31.
8. Gordon BD, Flottemesch TJ, Asplin BR. Accuracy of staff-initiated emergency department tracking system timestamps in identifying actual event times. Ann Emerg Med 2008;52:504-11.
9. Welch SJ, Asplin BR, Stone-Griffith S, Davidson SJ, Augustine J, Schuur J. Emergency department operational metrics, measures and definitions: results of the Second performance measures and Benchmarking Summit. Ann Emerg Med 2011;58: 33-40.