This simple article is an addendum to our previous study to handle under-reporting issues in medical error reporting systems. If you are interested in the qualitative aspect of the reported errors, you may want to refer to it. If your interest resides more in the comparing of the numbers of error reports for various situations, while controlling for the under-reporting problem, this article offers you some hints. We used computerized prescriber order entry (CPOE) as an example, comparing the difference in prescribing error reports between hospitals with and without CPOE. After controlling all potential covariates, including hospital-level clustering, we found hospitals with CPOE experienced 2.48 times more prescribing error reports than hospitals without CPOE. The result itself is meaningful, but we suggest you focus more on the methodology, which is our contribution to the patient safety society to overcome underreporting issues with voluntary error reporting system data.

Keywords: voluntary error report data, computerized prescriber order entry, CPOE, under-reporting, reporting and learning systems

Underreporting has been a serious problem for patient safety researchers and practitioners who want to analyze medical error reports data. As we never know the tendency of an individual healthcare professional or an organization as a whole in reporting the errors detected,¹⁴ any cross-sectional or longitudinal studies ought to be contaminated, although the amount of bias is uncertain. Jeong et al. developed a revolutionary methodology to indirectly control this underreporting phenomenon.^1,4,5 Figure 1 helps explain the underreporting phenomenon, which is denoted as the difference between detected and reported errors.

In our previous study, we examined how a counter-error measure, computerized prescriber order entry (CPOE), catches an error before reaching the patient.¹ In other words, we measured the CPOE effectiveness by comparing patterns of severity of reported errors between hospitals with and without CPOE. Although it worked well, that approach naturally led to an important question: If CPOE is so effective in detecting errors, are there any increases in the number of detected errors in hospitals where CPOE is in use compared to hospitals without CPOE? It is indeed a valid question.

Figure1 Relationship between detected errors and reported errors.⁶

This article answers this question using the same dataset as in our previous article. Thus, we strongly recommend you read the previous article or at least these two articles side by side.

Before we move on, let us make clear the modus operandi of CPOE: CPOE improves medication safety by making errors easily detectable, utilizing standardized data entry forms, and more importantly eliminating issues that arise from handwritten orders. As CPOE is designed for the prescribing phase of the medication use process, the administering phase gets a minimum impact from CPOE, as proven in our previous study (Figure 2).⁶

Figure 2 Medication use process, modified from “Preventing Medical Errors”.⁶

As such, we devised a methodology to catch the changes in the number of error reports from prescribing phase due to the implementation of CPOE, while controlling for the underreporting issue, the impact of which varies across hospitals. Let us directly dive into the methodology.

Data Source

The data used in this study were collected by MEDMARX, a national voluntary medication error reporting system in the US that was developed and operated by US Pharmacopeia.⁴ The dataset included medication error reports since 2003, the first year when MEDMARX collected the CPOE use status of each participating hospital, through 2007. To make the study stable, we narrowed the scope down to the inpatient setting, where the above-mentioned medication process (Figure 2) occurs in the hospital; thus, any upstream errors can be tracked and recorded. Note that the finest resolution of MEDMARX data is year, so any analysis with finer granulation was not an option.

Analysis

First, what we want to examine is the change due to CPOE in the error detection rate, which can be shown in the form of a ratio of two rates. For statistical purposes, we can refer to this as a rate ratio or incidence-rate ratio (IRR). The IRR of detected errors between the situation in which a hospital uses CPOE (i') and the situation where a hospital does not (i) can be denoted as:

$Incidence Rate Ratio=\frac{Rate of Detected Errors at Hospital i\text{'}}{Rate of Detected Errors at Hospital i}$

The rate has a count as the numerator and time as the denominator. As such, as long as two rates have the same duration of time in the denominators (e.g., one year), counts can be utilized to calculate the rate ratio without considering the issue of offset or exposure.

Because the data that we have are reported data, the actual IRR that will be examined is the IRR of counts of reported errors. The discrepancy between detection and reporting will be addressed, and detailed mathematical formulae will be presented below.

To calculate IRR, Poisson models will be utilized to analyze the counts data.⁷ It is assumed that in a specific time interval, t, medication errors occur independently. The number of error reports y in a time interval t thus follows a Poisson distribution.

$\mathrm{Pr}\left(y\text{|}\mu \right)= \frac{\text{exp}\left(-\mu \right){\mu }^y}{y!}$

where μ is the expectation of y and is depicted as

$\mu = \lambda t$           (8)

Counts (number) of prescribing error reports per year were observed for different hospitals i with different characteristics (covariates). The mean count of prescribing error ${\mu }_i$ is described with a log-linear model.⁸ In other words, if we have one covariate denoting whether a hospital has CPOE in use or not (0: CPOE is not in use, 1: CPOE is in use), $x_i$ ,

$\ln ({\mu }_i)= {\beta }_0+ {\beta }_1{x}_i$

From the above equation, by applying ${\mu }_i= {\lambda }_{i}t$ , we can obtain

${\lambda }_{i}t=\exp \left({\beta }_0+{\beta }_1{x}_i\right)=\exp \left({\beta }_0\right)\text{exp}\left({\beta }_1{x}_i\right)$

The ratio of rates of two organizations in one year, t, (or two different years of one organization) can be expressed as the ratio of two lambdas. To illustrate, ${\lambda }_i$ and ${\lambda }_{i\text{'}}$ are used for the rates of two organizations i and i’, respectively.

$\frac{{\lambda }_i}{{\lambda }_{i\text{'}}}=\frac{{\lambda }_{i}t}{{\lambda }_{i\text{'}}t}=\frac{\exp \left({\beta }_0\right)\text{exp}\left({\beta }_1{x}_i\right)}{\exp \left({\beta }_0\right)\text{exp}\left({\beta }_1{x}_{i^{\prime }}\right)}=\exp \left\{{\beta }_1\left(x_i-x_{i^{\prime }}\right)\right\}=\text{exp}{\left({\beta }_1\right)}^{\left(x_i-x_{i^{\prime }}\right)}$

Now that counts for all hospitals have the same time frame, one year, the exponentiated coefficient $\text{exp}\left({\beta }_1\right)$ is the IRR for a unit increase in $x_i$ , indicating the IRR of prescribing errors between a hospital with CPOE and a hospital without CPOE.⁸

By adding more covariates, we can develop a more sophisticated model. The expected number of prescribing errors ${\mu }_{ij}$ reported in year ifor hospital j is described as the following log-linear model:

$\ln \left({\mu }_{ij}\right)={\beta }_0+{\beta }_1{x}_{1ij}+z{\text{'}}_{ij}\gamma$

where ${\beta }_1$ means the log incidence-rate ratio between a hospital with CPOE and without CPOE and $x_{1ij}$  indicates whether the hospital j has CPOE in use in the year i. The vector $z{\text{'}}_{ij}$ stands for adjustment variables, such as all other hospital characteristics for the year i for the hospital j, and the year the data are reported. These adjustment variables are controlled for in the regression model.⁹ These characteristics include the use of computer-generated medication administration records (MARs), inpatient pharmacy availability, average monthly medication doses, and so forth. The list of these characteristics is shown in Table 3. One covariate not yet described is the count of reported errors from the administering node. Assuming that implementing CPOE has a minimal impact on the administering node, the count of reported errors from the administering phase, along with the variable for how medication errors were identified, can be effectively used as an adjusting variable for the impact of an organization’s safety culture or propensity related to the reporting of detected medication errors.

Although not shown in the previous equation, the impact of calendar year and other hospital characteristics, such as detailed facility owner information, is addressed in the results section.

For ordinary Poisson models, the independence of each count is assumed. However, in this particular analysis, we must address the fact that a hospital submitted error reports for multiple years. Therefore, counts of prescribing errors in a hospital for one year cannot be independent from those of the same hospital in a different year. The method for addressing the dependence of observations within a single organization is applying a random effects model including a hospital-specific intercept ${\zeta }_{1j}$ :⁸

$\ln \left({\mu }_{ij}\right)={\beta }_0+{\beta }_1{x}_{1ij}+z{\text{'}}_{ij}\gamma +{\zeta }_{1j}$

${\mu }_{ij}\equiv E\left(y_{ij}\text{|}{x}_{1ij},{z^{\prime }}_{ij},{\zeta }_{1j}\right)=\exp \left({\beta }_0+{\beta }_1{x}_{1ij}+{z^{\prime }}_{ij}\gamma +{\zeta }_{1j}\right)$

$=\exp \left({\zeta }_{1j}\right)\exp \left({\beta }_0+{\beta }_1{x}_{1ij}+{z^{\prime }}_{ij}\gamma \right)$

where the conditional distribution of $y_{ij}$ given $x_{1ij},{z^{\prime }}_{ij}\text{ and } {\zeta }_{1j}$ is a Poisson distribution and ${\zeta }_{1j}$ is assumed to be independent and to follow a gamma distribution.⁹ This means that the counts of prescribing error reports for a facility j at different occasions are specified as conditionally independent on the given random intercept ${\zeta }_{1j}$ and the covariates.⁸ The random intercept, ${\zeta }_{1j}$ , can be thought of as the combined effect of any omitted facility-specific covariates that could lead some hospitals to have a higher propensity to submit more medication error reports than others.⁸

Therefore, the exponentiated estimate for ${\beta }_1$ from the above random effects Poisson model denotes the IRR of prescribing errors between a hospital with CPOE and a hospital without CPOE, after controlling for the under-reporting problem.

Table 3 shows the total number of error reports from hospitals with CPOE and without CPOE broken down into four major phases of the medication use process. Please refer to the row percent. In hospitals with CPOE, 49.5% of error reports were from the prescribing phase; meanwhile, in hospitals without CPOE, only 12.0% of reports came from the prescribing phase. This table provides only the rough topography of error reports across various medication use phases, although it is still helpful. Let us move on to the main dish, Table 4.

Table 4 describes all the results from the full random effects Poisson regression model, on which we spent so many words to explain. The numbers are self-explanatory, so we focus only on the fact that the IRR for hospitals with CPOE was 2.48 (95% CI: 1.81–3.38), denoting that—when all other covariates including hospital-specific random effects were taken into consideration—hospitals where a CPOE system was in place submitted 2.48 times more prescribing errors than hospitals where a CPOE system was not in place.

This is a methodology paper, so please refer to the discussion section of our previous article for a detailed discussion about the reporting system and its relationship with patient safety improvement.¹ What we showed in this study is a suggestion of a methodology to utilize the number of error reports through voluntary medical error reporting systems as an indicator of patient safety-related situations of a healthcare organization or even beyond the individual organization level, such as a region or even a country. To our knowledge, the underreporting issue of the voluntary nature of such systems has always served as a sure hit from critics. We certainly know that there is no perfect way to control it, yet we do hope that this methodology along with the one that we introduced before¹ helps our fellow safety researchers get the most out of their error reports that have long been waiting to be analyzed.

None.

None.

1. Jeong HJ, Lee WC. A Strategy to Overcome Under-Reporting Issues of Voluntary Medication Error Reporting System: Computerized Prescriber Order Entry as an Example. Biom Biostat Int J. 2017;5(5):00145.
2. First Consulting Group. Computerized Physician Order Entry: Costs, Benefits, and Challenges, A Case Study Approach; 2003.
3. Reckmann MH, Westbrook JI, Koh Y, et al. Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc. 2009;6(5):613–623.
4. Santell JP, Hicks RW, McMeekin J, et al. Medication errors: experience of the United States Pharmacopeia (USP) MEDMARX reporting system. J Clin Pharmacol. 2003;43(7):760–767.
5. Myers JA, Dominici F, Morlock L. Learning from Near Misses in Medication Errors: A Bayesian Approach. Johns Hopkins University, Dept. of Biostatistics Working Papers. 2008. 178 p.
6. Institute of Medicine. Preventing medication errors. National Academy Press, Washington, DC, USA; 2007.
7. Belle Gv, Fisher LD, Heagerty PJ, et al. Biostatistics: a methodology for the health sciences. 2nd ed. Wiley-Interscience, USA; 2004.
8. Rabe-Hesketh S, Skrondal A. Multilevel and Longitudinal Modeling Using Stata. 2nd ed. Stata Press, College Station, TX, USA; 2008.
9. Winkelmann R. Co-payments for prescription drugs and the demand for doctor visits—evidence from a natural experiment. Health Econ. 2004;3(11):1081–1089.