This simple article is an addendum to our previous study to handle under-reporting issues in medical error reporting systems. If you are interested in the qualitative aspect of the reported errors, you may want to refer to it. If your interest resides more in the comparing of the numbers of error reports for various situations, while controlling for the under-reporting problem, this article offers you some hints. We used computerized prescriber order entry (CPOE) as an example, comparing the difference in prescribing error reports between hospitals with and without CPOE. After controlling all potential covariates, including hospital-level clustering, we found hospitals with CPOE experienced 2.48 times more prescribing error reports than hospitals without CPOE. The result itself is meaningful, but we suggest you focus more on the methodology, which is our contribution to the patient safety society to overcome underreporting issues with voluntary error reporting system data.
Keywords: voluntary error report data, computerized prescriber order entry, CPOE, under-reporting, reporting and learning systems
Underreporting has been a serious problem for patient safety researchers and practitioners who want to analyze medical error reports data. As we never know the tendency of an individual healthcare professional or an organization as a whole in reporting the errors detected,14 any cross-sectional or longitudinal studies ought to be contaminated, although the amount of bias is uncertain. Jeong et al. developed a revolutionary methodology to indirectly control this underreporting phenomenon.1,4,5 Figure 1 helps explain the underreporting phenomenon, which is denoted as the difference between detected and reported errors.
In our previous study, we examined how a counter-error measure, computerized prescriber order entry (CPOE), catches an error before reaching the patient.1 In other words, we measured the CPOE effectiveness by comparing patterns of severity of reported errors between hospitals with and without CPOE. Although it worked well, that approach naturally led to an important question: If CPOE is so effective in detecting errors, are there any increases in the number of detected errors in hospitals where CPOE is in use compared to hospitals without CPOE? It is indeed a valid question.
This article answers this question using the same dataset as in our previous article. Thus, we strongly recommend you read the previous article or at least these two articles side by side.
Before we move on, let us make clear the modus operandi of CPOE: CPOE improves medication safety by making errors easily detectable, utilizing standardized data entry forms, and more importantly eliminating issues that arise from handwritten orders. As CPOE is designed for the prescribing phase of the medication use process, the administering phase gets a minimum impact from CPOE, as proven in our previous study (Figure 2).6
As such, we devised a methodology to catch the changes in the number of error reports from prescribing phase due to the implementation of CPOE, while controlling for the underreporting issue, the impact of which varies across hospitals. Let us directly dive into the methodology.
Data Source
The data used in this study were collected by MEDMARX, a national voluntary medication error reporting system in the US that was developed and operated by US Pharmacopeia.4 The dataset included medication error reports since 2003, the first year when MEDMARX collected the CPOE use status of each participating hospital, through 2007. To make the study stable, we narrowed the scope down to the inpatient setting, where the above-mentioned medication process (Figure 2) occurs in the hospital; thus, any upstream errors can be tracked and recorded. Note that the finest resolution of MEDMARX data is year, so any analysis with finer granulation was not an option.
Analysis
First, what we want to examine is the change due to CPOE in the error detection rate, which can be shown in the form of a ratio of two rates. For statistical purposes, we can refer to this as a rate ratio or incidence-rate ratio (IRR). The IRR of detected errors between the situation in which a hospital uses CPOE (i') and the situation where a hospital does not (i) can be denoted as:
The rate has a count as the numerator and time as the denominator. As such, as long as two rates have the same duration of time in the denominators (e.g., one year), counts can be utilized to calculate the rate ratio without considering the issue of offset or exposure.
Because the data that we have are reported data, the actual IRR that will be examined is the IRR of counts of reported errors. The discrepancy between detection and reporting will be addressed, and detailed mathematical formulae will be presented below.
To calculate IRR, Poisson models will be utilized to analyze the counts data.7 It is assumed that in a specific time interval, t, medication errors occur independently. The number of error reports y in a time interval t thus follows a Poisson distribution.
where μ is the expectation of y and is depicted as
(8)
Counts (number) of prescribing error reports per year were observed for different hospitals i with different characteristics (covariates). The mean count of prescribing error is described with a log-linear model.8 In other words, if we have one covariate denoting whether a hospital has CPOE in use or not (0: CPOE is not in use, 1: CPOE is in use), ,
From the above equation, by applying , we can obtain
The ratio of rates of two organizations in one year, t, (or two different years of one organization) can be expressed as the ratio of two lambdas. To illustrate, and are used for the rates of two organizations i and i’, respectively.
Now that counts for all hospitals have the same time frame, one year, the exponentiated coefficient is the IRR for a unit increase in , indicating the IRR of prescribing errors between a hospital with CPOE and a hospital without CPOE.8
By adding more covariates, we can develop a more sophisticated model. The expected number of prescribing errors reported in year ifor hospital j is described as the following log-linear model:
where means the log incidence-rate ratio between a hospital with CPOE and without CPOE and indicates whether the hospital j has CPOE in use in the year i. The vector stands for adjustment variables, such as all other hospital characteristics for the year i for the hospital j, and the year the data are reported. These adjustment variables are controlled for in the regression model.9 These characteristics include the use of computer-generated medication administration records (MARs), inpatient pharmacy availability, average monthly medication doses, and so forth. The list of these characteristics is shown in Table 3. One covariate not yet described is the count of reported errors from the administering node. Assuming that implementing CPOE has a minimal impact on the administering node, the count of reported errors from the administering phase, along with the variable for how medication errors were identified, can be effectively used as an adjusting variable for the impact of an organization’s safety culture or propensity related to the reporting of detected medication errors.
CPOE use |
Node |
|||
Prescribing |
Transcribing |
Administering |
Total |
|
For all clinical areas |
|
|
|
|
n |
9,227 |
2,836 |
6,569 |
18,632 |
row % |
49.50% |
15.20% |
35.30% |
100.00% |
column % |
71.20% |
24.70% |
25.90% |
37.40% |
Not in use |
||||
n |
3,734 |
8,626 |
18,814 |
31,174 |
row % |
12.00% |
27.70% |
60.40% |
100.00% |
column % |
28.80% |
75.30% |
74.10% |
62.60% |
Total |
||||
n |
12,961 |
11,462 |
25,383 |
49,806 |
row % |
26.00% |
23.00% |
51.00% |
100.00% |
column % |
100.00% |
100.00% |
100.00% |
100.00% |
Table 3 Number of error reports in each phase of the medication use process1
Although not shown in the previous equation, the impact of calendar year and other hospital characteristics, such as detailed facility owner information, is addressed in the results section.
For ordinary Poisson models, the independence of each count is assumed. However, in this particular analysis, we must address the fact that a hospital submitted error reports for multiple years. Therefore, counts of prescribing errors in a hospital for one year cannot be independent from those of the same hospital in a different year. The method for addressing the dependence of observations within a single organization is applying a random effects model including a hospital-specific intercept :8
where the conditional distribution of given is a Poisson distribution and is assumed to be independent and to follow a gamma distribution.9 This means that the counts of prescribing error reports for a facility j at different occasions are specified as conditionally independent on the given random intercept and the covariates.8 The random intercept, , can be thought of as the combined effect of any omitted facility-specific covariates that could lead some hospitals to have a higher propensity to submit more medication error reports than others.8
Therefore, the exponentiated estimate for from the above random effects Poisson model denotes the IRR of prescribing errors between a hospital with CPOE and a hospital without CPOE, after controlling for the under-reporting problem.
Table 3 shows the total number of error reports from hospitals with CPOE and without CPOE broken down into four major phases of the medication use process. Please refer to the row percent. In hospitals with CPOE, 49.5% of error reports were from the prescribing phase; meanwhile, in hospitals without CPOE, only 12.0% of reports came from the prescribing phase. This table provides only the rough topography of error reports across various medication use phases, although it is still helpful. Let us move on to the main dish, Table 4.
Category |
IRR |
z |
p-value |
95% CI |
|
CPOE for all clinical areas |
2.48 |
5.71 |
0 |
1.81 |
3.38 |
Count of administering error reports |
1.01 |
22.01 |
0 |
1.01 |
1.01 |
Computer-generated MAR's |
|||||
On demand |
0.71 |
-0.65 |
0.51 |
0.25 |
1.98 |
both batch and on demand |
1.35 |
-0.9 |
0.37 |
0.7 |
2.58 |
No MAR's |
1.75 |
1.5 |
0.13 |
0.84 |
3.63 |
Owner/Operator |
|||||
Government;Federal;VA |
0.63 |
-1 |
0.32 |
0.26 |
1.56 |
Government;Federal;Other |
1.56 |
1.06 |
0.29 |
0.69 |
3.56 |
Government nonfederal |
0.46 |
-1.89 |
0.06 |
0.21 |
1.03 |
Pharmacist Availability |
|||||
On call when pharmacy closed |
0.72 |
-1.18 |
0.24 |
0.42 |
1.25 |
Not Available when pharmacy closed |
0.06 |
-2.91 |
0 |
0.01 |
0.41 |
Average Medication Doses (month) |
|||||
10,000-19,999 |
0.5 |
-2.70 |
0.01 |
0.3 |
0.82 |
20,000-39,999 |
5.13 |
4.25 |
0 |
2.41 |
10.91 |
40,000-99,999 |
6.69 |
4.44 |
0 |
2.89 |
15.51 |
>=100,000 |
14.28 |
6.62 |
0 |
6.50 |
31.37 |
Calendar Year |
|||||
2004 |
1.16 |
4.92 |
0 |
1.09 |
1.23 |
2005 |
1.54 |
12.59 |
0 |
1.44 |
1.65 |
2006 |
1.44 |
9.98 |
0 |
1.34 |
1.54 |
2007 |
1.38 |
8.46 |
0 |
1.28 |
1.49 |
Methods available for error detection |
|||||
Staff initiated written reports |
0.24 |
-4.7 |
0 |
0.13 |
0.43 |
Staff initiated electronic communication |
0.79 |
-0.99 |
0.32 |
0.49 |
1.26 |
Automatic information system detection |
0.65 |
-1.01 |
0.31 |
0.28 |
1.5 |
Random observation based reviews |
1.29 |
0.9 |
0.37 |
0.74 |
2.25 |
Download from hospital IT/MR dept. |
2.49 |
1.8 |
0.07 |
0.92 |
6.74 |
Telephone hot line |
2.82 |
4.39 |
0 |
1.77 |
4.47 |
Patient or patient's family initiated |
0.23 |
-6.27 |
0 |
0.15 |
0.37 |
Other |
1.37 |
0.89 |
0.37 |
0.69 |
2.72 |
Constant |
16.13 |
4.31 |
0 |
4.55 |
57.15 |
Random Effect |
|||||
alpha |
1.74 |
1.35 |
2.23 |
Table 4 Poisson regression result of number of prescribing error reports
Likelihood-ratio test of alpha = 0: Chibar2(01) = 13000 Prob >= chibar2 = 0.000
Reference category is government hospitals (federal military), the average monthly medication doses of which is less than 10,000, which have pharmacist 24 hours a day, 7 days a week and MARs only through batch processing, in the year of 2003.
Table 4 describes all the results from the full random effects Poisson regression model, on which we spent so many words to explain. The numbers are self-explanatory, so we focus only on the fact that the IRR for hospitals with CPOE was 2.48 (95% CI: 1.81–3.38), denoting that—when all other covariates including hospital-specific random effects were taken into consideration—hospitals where a CPOE system was in place submitted 2.48 times more prescribing errors than hospitals where a CPOE system was not in place.
This is a methodology paper, so please refer to the discussion section of our previous article for a detailed discussion about the reporting system and its relationship with patient safety improvement.1 What we showed in this study is a suggestion of a methodology to utilize the number of error reports through voluntary medical error reporting systems as an indicator of patient safety-related situations of a healthcare organization or even beyond the individual organization level, such as a region or even a country. To our knowledge, the underreporting issue of the voluntary nature of such systems has always served as a sure hit from critics. We certainly know that there is no perfect way to control it, yet we do hope that this methodology along with the one that we introduced before1 helps our fellow safety researchers get the most out of their error reports that have long been waiting to be analyzed.
None.
None.
© . This is an open access article distributed under the terms of the, which permits unrestricted use, distribution, and build upon your work non-commercially.
2 7