Medical error reporting systems have been in place for decades, with the hope that the collected reports would help us understand the nature of errors and prevent similar errors from occurring in the future. Among the various types of reporting systems, a voluntary system leaves the decision about whether to report a detected error or not to healthcare providers. Naturally, not every detected error has been reported, and we call such a phenomenon under-reporting. The level of under-reporting varies across reporters and organizations, which has prohibited us from analyzing the data and utilizing the results with satisfactory validity. The current study tried to show how to overcome this issue, using the effectiveness of a computerized prescriber order entry (CPOE) system as an example. Since CPOE is designed to catch the prescribing error, we first calculated the ratio between the odds of prescribing error reaching the patient and that of administering errors seemingly unrelated to the prescribing phase. We then compared this ratio between hospitals with CPOE and without CPOE. With this methodology, combined with adding a random intercept to control for hospital-level clustering of medication errors, we could effectively handle the varying degrees of reporting levels across hospitals, achieving a solid comparison of the effectiveness of CPOE between hospitals with and without CPOE. The final results showed that the odds of an error being caught before reaching the patient was 4.63 times higher in the prescribing phase of the medication use process among hospitals using CPOE than among hospitals without CPOE. We believe the methodology used in this study can be applied to many other topics in patient safety studies using data from voluntary medical error reports.

Keywords: voluntary medical error reporting, computerized prescriber order entry, CPOE, under-reporting, reporting and learning systems

Thousands of victims of medical errors exist in the clinical realm.^1,2 Most of these types of errors have occurred in the past, meaning the same modality is occurring again and again, harming our beloved patients and certainly discouraging healthcare professionals. Yet the flipside of the coin is that we can prevent the repeatedly occurring errors only if we have information on the mechanism—namely, how the errors occur. To address this issue, many error reporting systems have been developed and are currently in operation.^1,2 Each of these systems has various fields of interest; some collect reports from all medical errors whereas others focus on a specific part of medical care, such as surgical events or intensive care units.

The modus operandi varies extensively between completely mandatory and completely voluntary as the extremes of the continuum. Simply put, mandatory systems are mainly designed for severe events like patient deaths or hospital-acquired infection; quite often, such information is tied to reimbursement systems. On the other hand, voluntary medical error reporting systems do not mandate healthcare professionals to report errors they have detected. This voluntary nature is frequently linked to the need for anonymity among healthcare organizations and individual practitioners reporting their errors, so that they are free from the fear of reprimand. However, the price of such protection of reporters cannot be ignored; one outcome is under-reporting, which has not yet been clearly resolved, as depicted in Figure 1. When not every detected error is reported, we do not have clear information for the detecting-to-reporting ratio, which must vary across organizations or even among individual practitioners. This results in any comparison of the results across departments or organizations being unstable and, quite honestly, impossible. This reporting behavior, known to be related to organizational culture, has been measured in safety culture survey questionnaires,^3–6 yet such culture data have not been successfully utilized to adjust for the difference between detection and reporting.

Figure 1 Relationship between detected errors and reported errors.

To crack this nemesis of under-reporting, we chose one of the largest reporting system’s datasets, called MEDMARX. This national (US) voluntary reporting system collects specifically medication errors and has collected more than a million cases thus far.⁷ We chose to use MEDMARX not just because of statisticians’ need for huge and complex data, but also because—among the various types of medical errors tracked—medication errors account for a large proportion, which is easily understandable given that medication is the oldest and most frequently used venue of medical treatment.^8,9 Indeed, 65% of the population in the US receive medication prescription each year,¹⁰ and numerous adverse drug events have been identified in various medical settings. To illustrate, 50.1 events per 1000 person per year have been documented in outpatient care and 22 events have been documented in nursing facilities.^11,12 In addition to the high volume of errors, the errors also caused a wide range of patient harms, from a simple wrongfully delivered medication in both inpatient and outpatient settings that may not cause severe harm to fatal drug–drug interactions.^13,14

Figure 2 Medication use process, modified from “Preventing Medical Errors”.¹⁵

In this article, we propose a novel way to address the issue of under-reporting originating from the voluntary nature of such error reporting systems. Indeed, we pushed the envelope to the level that voluntary reporting data can be used to evaluate the effectiveness of a counter-error system in healthcare organizations. The system we focused on was computerized prescriber order entry (CPOE). We hope this article can give readers some hints for getting the most out of the sacred error reports data that countless medical practitioners spend their time reporting in the hopes of improving the safety of the whole healthcare system. In fact, we are obliged to pursue such an effort.

Before diving into the analysis, we first introduce the typical medication process, which serves as a framework of the analysis. Instead of describing it thoroughly, we rely on the Institute of Medicine’s (IOM) medication use process and the US Pharmacopeia’s definition for each step.

The CPOE system is designed to reduce prescribing-related adverse events; therefore, by definition, the downstream of medication use process, such as administering, may not be affected by whether an organization has CPOE or not.

Now that the groundwork is done, we will show how we handled the under-reporting issue in the process of analyzing the effectiveness of CPOE as an example.

Conceptual background

The premise of a prescribing error causing harm to a patient is that the error must reach the patient. CPOE is supposed to catch errors specifically in the prescribing phase; thus, if CPOE plays its part as expected, errors should be less likely to reach the patient. Yet CPOE is not supposed to influence errors in other phases of medication use process, such as administering. Therefore, we consider administering error reports as an anchor to adjust each hospital’s reporting tendency or culture. This means that we may not be able to directly quantify how well CPOE catches prescribing errors before reaching the patient, but we can compare the ratio of prescribing error reaching the patients between hospitals with and without CPOE after adjusting for the same ratio of administering error as reference, albeit reporting culture varies across hospitals. Figure 3 help readers understand this approach.

Figure 3 Conceptual diagram of errors in each phase with and without CPOE.

MEDMARX stores each error’s severity using the National Coordinating Council for Medication Error Reporting and Prevention’s (NCC MERP) category, where B is an error caught before it reaches the patients and C to I are the severity of harm that already reached the patients.¹⁷ The odds ratio that an error will fall in category B, rather than C-I, comparing the prescribing phase to the administering phase is $\raisebox{1ex}{$\text{α}$}\!\left/ \!\raisebox{-1ex}{$\text{γ}$}\right.$  divided by $\raisebox{1ex}{$\text{β}$}\!\left/ \!\raisebox{-1ex}{$\text{δ}$}\right.$ ,¹⁸ where the area of α, β, γ, and δ stand for the number of errors relative to each other. If there is a change in the detectability of errors (before reaching patients), and the change may potentially influence primarily the prescribing phase, then the ratio shown above will reflect this change. CPOE’s main effect is expected to take place in the prescribing phase; therefore, we can test whether catching errors before reaching patients in the prescribing phase is changed by comparing the odds ratios between the two situations of having and not having CPOE in use, as shown in Figure 3. Basically, it is difference in odds ratios, which we usually call the interaction term in regression formulae.

In addition to the above approach, we applied random effects to take into account any hospital-specific information that might not have been captured in an ordinary fixed effects mode, which further strengthened our endeavor to handle the varied reporting culture across hospitals.

Data collection

We used medication error data collected through the MEDMARX system across hundreds of American hospitals between 2003 (the first year MEDMARX recorded CPOE use status of each hospital) and 2007. Any healthcare professionals who found an error passed the information along to the coordinator in each hospital, and the information was then uploaded to and saved in the MEDMARX system. The results section of this paper details the characteristics of participating hospitals. As MEDMARX is an anonymous system, the information in the results section is the most definitive possible.

Model development

We began with the simplest model that contains only one covariate denoting whether a hospital has CPOE in use or not (0: no CPOE is in use, 1: CPOE is in use): $x_i$ , where i refers to an individual error. The expectation of a binary response is the probability that the response, $y_i$ , is 1, which means the error was caught before reaching the patient as category B, as previously describe.¹⁹

$E\left(y_i\text{|}{x}_i\right)=\text{Pr}(y_i=1|x_i)$

For logistic regression, the logit function was applied.

$\text{logit}\left\{\mathrm{Pr}\left(y_i=1\text{|}{x}_i\right)\right\}\equiv \ln \left\{\frac{\mathrm{Pr}\left(y_i=1\text{|}{x}_i\right)}{1-\mathrm{Pr}\left(y_i=1\text{|}{x}_i\right)}\right\}={\beta }_0+ {\beta }_1{x}_i$

where $\frac{\mathrm{Pr}\left(y_i=1\text{|}{x}_i\right)}{1-\mathrm{Pr}\left(y_i=1\text{|}{x}_i\right)}$  is the odds that $y_i$ =1 (the error was caught before reaching the patient) with the given $x_i$ . In other words, this is the expected number of category B errors for each category C-I error. Therefore, for the CPOE indicator $x_i$ :

$\ln \left\{\text{Odds}\left(y_i=1\text{|}{x}_i=1\right)\right\}-\ln \left\{\text{Odds}\left(y_i=1\text{|}{x}_i=0\right)\right\}=\left\{{\beta }_0+ {\beta }_1\right\}-{\beta }_0= {\beta }_1$

By exponentiating the above equation:

$\exp \left[\ln \left\{\text{Odds}\left(y_i=1\text{|}{x}_i=1\right)\right\}-\ln \left\{\text{Odds}\left(y_i=1\text{|}{x}_i=0\right)\right\}\right]=\frac{\text{Odds}\left(y_{{}_i}=1\text{|}{x}_i=1\right)}{\text{Odds}\left(y_i=1\text{|}{x}_i=0\right)}=\text{exp}\left({\beta }_1\right)$

Therefore, the exponentiated value of the CPOE coefficient ${\beta }_1$ means the odds ratio.¹⁹

We developed a more sophisticated model by adding various covariates to prevent endogeneity issues as much as possible. First, the prescribing phase was added to designate whether an error came from the prescribing phase or the other phases. Because we are looking for CPOE’s impact on the odds of the prescribing error being caught before reaching the patient, we included an interaction term for CPOE and prescribing phase. The method for addressing clustering effects within organizations is to apply a random effects model including a hospital specific intercept ${\zeta }_{11k}$ . The expected log odds in year j for hospital k are described below:

$\text{logit}\{\text{Pr}\left(y_{ijk=1}\text{|}{x}_{1ijk},{z^{\prime }}_{ijk},{\zeta }_{11k}\right)={\beta }_0+{\beta }_1{x}_{1ijk}+{\beta }_2{x}_{2ijk}+{\beta }_3{x}_{1ijk}{x}_{2ijk}+{z^{\prime }}_{ijk}\gamma +{\zeta }_{11k}$

To clarify, ${\beta }_1$ means the log odds ratio that an error did not reach the patient, comparing hospitals with CPOE to hospitals without CPOE for errors in the administering phase (where $x_{2ijk}=0$ ), and $x_{1ijk}$ indicates whether hospital k has CPOE in use in year j. x_2ijk indicates whether error i originated from the prescribing phase. ${\beta }_2$ is the log odds ratio of an error not reaching the patient, comparing errors in the prescribing phase to errors in the administering phase for hospitals without CPOE ( $x_{1ijk}=0$ ). ${\beta }_3$ is the interaction term that stands for the log odds ratio of an error not reaching the patient, comparing errors in the prescribing phase to errors in the administering phase for hospitals with CPOE and the same log odds ratio for hospitals without CPOE. The vector $z{\text{'}}_{ijk}$ represents all potential adjustment variables, such as hospital characteristics for year j, for hospital k, which will be controlled for in the model.²⁰

Characteristics of MEDMARX participating hospitals and collected error reports

Table 2 describes organization-level characteristics that were plugged into the previously discussed formula as covariates except for “methods available for error detection.” Note that not every hospital participated in MEDMARX for all five years (i.e., from 2003 to 2007), although the proportion of hospitals utilizing CPOE grew from 38.1% in 2003 to 50.8% in 2007. Unlike CPOE use, the proportion of healthcare facility with computer-generated medication administration records (MARs) did not vary significantly.

Table 3 describes the number of reported errors from the major phase of the medication use process. Interestingly, hospitals with CPOE showed that almost half of the errors were reported from the prescribing node (49.5%), which is in sharp contrast to hospitals without CPOE, where only 12.0% of reports were designated as prescribing errors. This phenomenon is discussed in a later section.

Effectiveness of CPOE

This section describes the impact of CPOE on prescribing whether a prescribing error reached a patient. To better understand the analysis results, we describe the results in a bar graph format in Figure 4. First, since the odds that an administering error was caught before it reached the patient were the reference, we set the odds of administering errors from hospitals without CPOE at one as reference (right most light grey bar) and depict the odds ratios between other situations (combinations of prescribing/administering and hospitals with/without CPOE) and the reference.

The odds that an error generated in the prescribing phase is caught before reaching a patient were 19.49 times (95% CI: 17.52–21.69) that of administering errors in hospitals without CPOE. This result was consistent with our intuitive expectation that errors generated in the upstream of the medication process have a greater chance of being caught before they reach the patients than downstream errors, like administering errors, thereby strengthening our approach in this study.

To examine the impact of CPOE on errors from these nodes, we then added the odds ratios from prescribing and administering errors reported from hospitals with CPOE, as indicated by the dark grey bars in Figure 4.

Figure 4 The effectiveness of CPOE in each node on catching errors before reaching the patient.

First and foremost, for administering errors, the odds ratio for hospitals with CPOE was 1.02 (95% CI: 0.68–1.52), meaning there was no statistically significant difference in the odds that an error was caught before it reached a patient between hospitals with CPOE and hospitals without CPOE. This finding suggests that CPOE did not have any influence on administering errors.

On the other hand, for prescribing errors, the odds ratio between hospitals with CPOE and hospitals without CPOE was 4.63 (95% CI: 3.09–6.95). The significantly high odds ratio observed for prescribing errors, combined with the non-significant odds ratio for administering errors, suggests that CPOE was effective in catching medication errors before the errors reached the patients. In addition, the effectiveness was confined to the errors generated in an earlier phase of the medication use process—namely, prescribing.

Medical error reporting systems have been developed to learn from defects: By analyzing multiple medical error cases, we can find patterns and even identify the mechanism describing how an error occurs. Ultimately, utilizing such information, we can redesign the healthcare process more effectively and efficiently .^21–23 With the hope of improving safety, healthcare professionals have submitted millions of medical error reports. Yet how thoroughly those reports were analyzed and applied to real-world improvement is in question. Indeed, most analyses were fundamentally descriptive, and the mechanism for determining how systems have broken down were beyond the scope of previous studies. Consequently, many data still have to be analyzed.

We sought to analyze these data ourselves and decided to devise a completely new way of using medical error reports beyond simply designating causes and contributing factors for errors or adverse events, the typical application of error report datasets. Such an ambitious goal led us to consider utilizing error reporting systems to evaluate the effectiveness of a counter-error measure—in our case, CPOE. Actually, the current study is not the first of its kind. Zhan et al. (2006) examined CPOE effectiveness with the very same dataset used in this study, MEDMARX.²⁴ Yet most previous studies failed to overcome the under-reporting issue^7,24–26 (for a better understanding, please recall Figure 1). The amount of under-reporting, which varies significantly across units or departments of a hospital and probably even more across hospitals, is related to various factors, such as whether error reports are discoverable and/or used as evidence in a court as well as senior managers’ attitude to transparency. Each organization or unit of analysis naturally holds a different level of propensity for reporting errors, which almost always undermines the validity of studies based on error reports.

We adopted two safeguards to protect us from the threat of the under-reporting issue. First, the research question of this study was whether CPOE is effective in reducing medication-related adverse events; since CPOE is designed for the prescribing phase of the medication process, we need to compare prescribing error reports from hospitals with and without CPOE. At this point, instead of making a direct comparison, we first calculated the ratio of prescribing errors to administering errors; we then compared these ratios from hospitals with CPOE and without CPOE. We assumed that each hospital has a unique reporting propensity of detected errors; thus, using administering errors that might not be related to prescribing errors as an anchor for the comparison, we could neutralize the difference in reporting patterns across hospitals.

The second safeguard against the under-reporting issue was to allow for random intercepts for each hospital in the model. Such random intercepts captured any hospital-specific characteristics that might not have been plugged into the analysis as exogenic variables. Actually, this is not a unique method of this study. Nobody can guarantee that any given model contains all the required variables, which leaves the risks omitted variable bias intact. Thus, we added a random component to the model so that any hospital characteristics not captured in the independent variable could be controlled for.

Readers not in the field of safety may appreciate the following way of thinking. The premise of this study is that CPOE does not necessarily prevent errors; rather, it can catch human errors in an earlier stage of the medication use process. To illustrate, if a physician fails to fill out a field on the screen while prescribing a medication, CPOE issues a warning to notify the physician of the need to correct the error; it is counted as an error that did not reach a patient. Usually, this kind of event is not captured by mandatory reporting systems, whose purpose is generally confined to reporting harmful events. However, voluntary error reporting systems like MEDMARX encourage the reporting of near misses that did not cause any harm to a patient.^1,2,7 The modality of collecting non-harmful errors in addition to harmful errors is supported by the causal continuum concept proposed by Myers et al. (2008): Regardless of the severity of the outcome, the mechanism of errors being generated is almost the same.²⁷

Now let us consider a notion that is popular in the safety realm: Heinrich’s law. “For every accident that causes a major injury, there are 29 accidents that cause minor injuries and 300 accidents that cause no injuries”.^28,29 All in all, collecting near misses as well as harmful events exponentially expedite the building of a sizable dataset without undermining the validity of the data. The current study was also possible due to the voluntary nature of MEDMARX, allowing the comparison of non-harmful errors and harmful errors, related to the previously mentioned causal continuum and Heinrich’s law.

However, while developing statistical models, we were discouraged by the structure of the MEDMARX data—namely, there was no detailed information regarding exactly when CPOE was implemented in a hospital. To illustrate, if hospital A began using CPOE on July 1, all errors from hospital A in the year were treated as errors from a hospital with CPOE, even though CPOE was not in use for the first half of the year. Furthermore, any learning curve for the CPOE system or burning-in period was not described in the MEDMARX data. Such subpar granulation of hospital-level information might have caused misclassification bias. However, this weakness can be interpreted as a strength. Misclassification bias usually leads the result of the analysis to be null,¹⁸ increasing type II error. Therefore, if there are statistically significant results—rejecting null hypothesis—despite the threat of misclassification, the given results can be even more supported.

We know that fully understanding the methodology of this article requires not only a strong statistical background, but also knowledge about patient safety and error reporting systems. Indeed, such ideas are difficult, yet we also know that the difficulty can never justify abandoning such precious medical error datasets, which have been collected by countless healthcare professionals through their immeasurable time and efforts. Obviously, the methodology we introduced is not a panacea, and different topics would need a different or even brand new approach. Yes, we will keep developing new approaches. We cannot resist the temptation to save lives, and we know that you cannot either, can you?

None.

None.

1. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health care system. National Academies Press (US); 2000.
2. Wachter R. Understanding patient safety. McGraw Hill Professional, USA; 2012.
3. Lee WC, Wung HY, Liao HH, et al. Hospital safety culture in Taiwan: a nationwide survey using Chinese version safety attitude questionnaire. BMC Health Serv Res. 2010;10(1):234.
4. Zimmermann N, Küng K, Sereika SM, et al. Assessing the safety attitudes questionnaire (SAQ), German language version in Swiss university hospitals-a validation study. BMC Health Serv Res. 2013;13(1):347.
5. Jeong HJ, Jung SM, An EA, et al. Development of the Safety Attitudes Questionnaire - Korean Version (SAQ-K) and its Novel Analysis Methods for Safety Managers. Biom Biostat Int J. 2015;2(1):00020.
6. Sexton JB, Helmreich RL, Neilands TB, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6(1):44.
7. Santell JP, Hicks RW, McMeekin J, Cousins DD. Medication errors: experience of the United States Pharmacopeia (USP) MEDMARX reporting system. The Journal of Clinical Pharmacology. 2003;43(7):760–767.
8. Cohen MR. Medication errors: causes, prevention, and risk management. Jones & Bartlett Learning, USA; 1999.
9. Phillips DP, Christenfeld N, Glynn LM. Increase in US medication-error deaths between 1983 and 1993. 1998;351(9103):643–644.
10. Bell DS, Cretin S, Marken RS, et al. A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities. J Am Med Inform Assoc. 2004;11(1):60–70.
11. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289(9):1107–1116.
12. Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes. Am J Med. 2000;109(2):87–94.
13. Wachter RM. The end of the beginning: patient safety five years after 'to err is human'. Health Aff (Millwood). Suppl Web Exclusives: W4-534-W4-545; 2004.
14. Williams SD, Ashcroft DM. Medication errors: how reliable are the severity ratings reported to the national reporting and learning system? International Journal for Quality in Health Care. 2009;21(5):316.
15. Institute of Medicine. Preventing medication errors. National Academy Press, USA; 2007.
16. The United States Pharmacopeial Convention, MEDMARX Taxonomy; 2005.
17. National Coordinating Council for Medication Error Reporting and Prevention. NCC MERP Index for Categorizing Medication Errors Algorithm; 2001.
18. Gordis L. Epidemiology. 3rd ed. Elsevier Saunders, USA; 2004.
19. Rabe-Hesketh S, Skrondal A. Multilevel and Longitudinal Modeling Using Stata. 2nd ed. Stata Press, USA; 2008.
20. Winkelmann R. Co-payments for prescription drugs and the demand for doctor visits--evidence from a natural experiment. Health Econ. 2004;13(11):1081–1089.
21. World Health Organization. WHO Draft Guideline for Adverse Event Reporting and Learning Systems; 2005.
22. Pronovost PJ, Goeschel CA, Olsen KL, et al. Reducing health care hazards: lessons from the commercial aviation safety team. Health Aff (Millwood). 2009;28(3):w479–w489.
23. Terzibanjan AR, Laaksonen L, Weiss M,  et al. Medication Error Reporting Systems-Lessons Learnt; 2008.
24. Zhan C, Hicks RW, Blanchette CM, et al. Potential benefits and problems with computerized prescriber order entry: analysis of a voluntary medication error-reporting database. Am J Health Syst Pharm. 2006;63(4):353–358.
25. National Patient Safety Agency. Safety in doses: medication safety incidents in the NHS; 2008.
26. Rowin EJ, Lucier D, Pauker SG, et al. Does error and adverse event reporting by physicians and nurses differ? Jt Comm J Qual Patient Saf. 2008;34(9):537–545.
27. Myers JA, Dominici F, Morlock L.  Learning from Near Misses in Medication Errors: A Bayesian Approach. Johns Hopkins University, Dept. of Biostatistics Working Papers. 2008. 178 p.
28. Taxis K, Gallivan S, Barber N, Can the Heinrich ratio be used to predict harm from medication errors? 2005
29. Heinrich HW. Industrial accident prevention: a scientific approach. McGraw-Hill, USA; 1931.