The area under a ROC curve (AUC) is an important summary measures useful in assessing the accuracy of a diagnostic test in discriminating true disease status when the data for measurement is paired. This assessment is most important when the AUCs of different diagnostic test procedures are compared. These comparisons are not without some problem associated with it such as the inability of some test such as the McNemar test to adjust for the possible presence of ties in the data, thereby leading to erroneous conclusions in data analysis occasioned by committing Type II error more often than not. This is evident when the use of the traditional McNemar test in data analysis yielded high value of variance and low chi-square value thereby making one to accept a false null hypothesis more often than expected. To be able to tackle this challenge, we extend the usual McNemar test adopted by adjusting for the possible presence of ties in the data when measurements of data may be on any scale. The extended McNemar test can enable one to easily estimate the probability that randomly selected pair of subjects from two diagnostic test procedures respond positive or both respond negative and it can be used to test the null hypothesis of equality of proportion of positive responses in two diagnostic test procedures. An extensive simulation study was carried out to determine the Type I error and statistical power of the existing and extended tests and the application of the tests to standard and real data, was carried out and result showed that in all the McNemar test demonstrates superior statistical power and less conservative type I error compared to DeLong area test, Bandos et al area test and the usual McNemar area test and so compares favorably.

Keywords: extended mcnemar test, positive response, correlated data, nonparametric test, diagnostic tests, type ii error

The receiver operating characteristic (ROC) curve is a standard tool used to evaluate the performance of a diagnostic test when measurement of test results are either continuous or ordinal.¹ In 1950s the methodology of ROC was first developed by electrical and radar engineers during World War II for signal detection theory in battle fields.² In an ROC curve, the true positive rate (TPR) is plotted against the false positive rate (FPR) across all possible cut-off values in other to make meaningful decision. The area under the ROC curve (AUC) is a summary index for measuring the diagnostic accuracy. AUC ranges from 0 to 1 inclusive and the greater the value of AUC close to 1, the better the discriminatory power of the diagnostic procedure. Often times, the aim of many diagnostic studies is to compare the accuracy of diagnostic tests to determine the superiority of one test over another test for a certain condition or disease when data measurement may be on any scale. Statistical inference may be based on parametric, nonparametric or semi-parametric statistics. If the statistical inference is nonparametric, the difference between correlated AUCs for paired data was first proposed by DeLong et al.,³ and it is based upon asymptotic theory for U-statistics.⁴ But the validity of this or any other method relays on large sample size and when the sample size is small, the validity of the test for the difference between two or more AUCs may not be achieved. Two permutation tests for paired receiver operating characteristic (ROC) studies currently exist: one proposed by Venkatraman & Begg⁵ and the more recent test of Bandos et al.,⁶ The test of Bandos et al.,⁶ directly tests for an equality of AUCs, while the test of Venkatraman & Begg⁵ is more general and tests for equality of the underlying ROC curves. As a result, the test of Venkatraman & Begg⁵ is less powerful for testing equality of AUCs. Both permutation tests are executed by permuting the labels of the two tests within each diseased and non-diseased subject. Such an approach implicitly assumes that both tests are exchangeable within subject and requires an appropriate transformation, such as ranks, for tests differing in scale. Bandos et al.,⁶ compared the performance of their test to that of DeLong et al.,³ using simulation and found that the permutation test had greater power than the nonparametric test developed by DeLong et al.,³ when there was moderate correlation between two tests, large AUCs, and small sample sizes.

When comparing two diagnostic procedures, the difference between AUCs is often used and to control for the sources of changes arising from changes due to subjects which represents a reasonable size of the overall changes of the AUC, a paired data is recommended. This is because paired data usually induces positive correlation between the test results of the same subjects. Based on the use of paired data, Sumi et al.,⁷ adopted the usual McNemar⁸ test for comparing two correlated marginal probability of positive responses in diagnostic test procedures. This paper is an extension of this work for evaluating the performance of two diagnostic tests in terms of the proportion of positive responses and the comparison of this method with the existing tests by DeLong et al.,³ Bandos et al.,⁶ Sumi et al.⁷

In estimating the AUC, two main factors have to be considered namely, the design of the study and the distribution of test result.⁹ Under the study design, test results or dataset can be classified into three types namely: (i) paired data (ii) unpaired data and (iii) partially paired data. For the paired and partially-paired set of data, correlation between AUCs is considered. Under the distribution type of test result, three approaches for estimating the AUCs are considered namely: (i) A parametric approach (ii) A semi-parametric approach (iii) A non-parametric approach, in this paper, our focus will be on the non-parametric method. All the approaches to estimating the AUC differ in the way the distribution functions of both populations are estimated based on their sample values. Basically the nonparametric (empirical) method of estimating AUC is stated as follows.

Given that there are two diagnostic tests, let n be the total number of subjects without disease and m as the total number of subjects with disease. Suppose represent the subjects without disease and with disease respectively. Therefore the corresponding bivariate outcomes for the two diagnostic procedures on the same N non-diseased and M diseased subjects should be Bivariate cumulative distribution functions are denoted by and their corresponding margina Bamber¹⁰ noted that the AUC is equal to Let be the AUCs of diagnostic procedures. The formula suggested by Hanley and McNeil (1982) for computing the AUC is given as

$AUC=\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^{m}g(X_i,Y_j)$   (1)

Where m=number of diseased subjects, n=number of non-diseased subjects. Also $X_{i}and{Y}_j$ are respectively the test result of the i^th and j^th subject without and with disease and g is the indicator function comparing $X_{i}with{Y}_j$ such that

$g(X_i,Y_j)=\{\begin{array}{c}1,if{Y}_j>X_i\\ 0.5,if{Y}_j=X_i\\ 0,otherwise\end{array}$  (2)

Therefore for the $b^{th}$ diagnostic test procedure the AUC can be computed as

$AU{C}_b=\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^{m}g(X_i^b,Y_j^b)$   (3)

To carry out significant test for the differences between two or more correlated AUCs, it is necessary to consider the distribution of the test result which also determines the procedure to be adopted in estimating the AUCs and its variance-covariance matrix. By the comparison of areas under the two ROC curves, we can estimate which one of two diagnostic tests is more suitable for discriminating non-diseased subjects from diseased subjects or any other two conditions of interest.⁷ Braun & Alonzo¹¹ proposed a modified rank test that does not require such a transformation and showed that the modified test has the same power as Bandos et al.,⁶ Bantis & Feng.,¹² focused on comparing two correlated ROC curves at a given specificity level. They proposed parametric approaches, transformations to normality, and nonparametric kernel-based approaches. Extensions of their methods also involved inference for the AUC and accommodating covariates. They evaluated the robustness of their techniques through simulations, compared to other known approaches and presented a real data application involving prostate cancer screening. They approaches perform satisfactorily in terms of size and power. The limitation of Bantis & Feng¹² method is that their Box-Cox version does not take into account the variability of the transformation parameter. Finally, to increase the ability to detect the crossing alternative, Yu et al.,¹³ suggested a two-stage test, where the first stage uses the test derived by DeLong et al.,³ to test the equality of the two AUCs and the second stage uses a modified area test to test two partial AUCs.

A number of tests exist for comparing two or more AUCs or proportion of positive responses for the matched sample case.

DeLong et al’s conventional nonparametric method for comparing AUCs

DeLong et al.,.,³ developed a totally nonparametric approach to compare two correlated AUCs of two diagnostic tests for paired samples of subjects by using the theory of generalized U statistics. In other words, they developed a conventional fully nonparametric approach leading to an asymptotically normal test statistic. This method is important as it helps to study the behavior of the type I error and the statistical power of the conventional nonparametric test for comparing two AUCs over a wide range of relevant parameters and against various alternatives. The test by Delong et al is limited by the fact that the AUC has an unbiased non-parametric estimator called the indicator variable that requires the comparison of all the number of subjects responding positive and negative, thus working with very large number of observations, so that computational time could be long. In estimating AUC, sigmoid function is sometimes used instead of indicator function or variable.¹⁴ However, DeLong et al.,³ method is based only on a continuous scale of measurement. The method of structural components is used to generate an estimated covariance matrix and the resulting test statistic has asymptotically a chi-square distribution.

Suppose $X_i,i=1,2,\mathrm{....},n$ denote test results for a sample of n non-diseased subjects, and $Y_j,j=1,2,\mathrm{....},m$ denote the test results for m diseased subjects. For each $(X_i,Y_j)$ pair, an indicator function is defined as follows:

$=\{\begin{array}{c}1if{Y}_j>X_i\\ 0.5if{Y}_j=X_i\\ 0if{Y}_j<X_i.\end{array}$   (4)

The average of these values for I over all nm comparisons is the Wilcoxon or Mann-Whitney U statistic:

$U=\frac{1}{mn}\sum _{i=1}^n\sum _{j=1}^{m}I(X_i,Y_j).$   (5)

Where U is equivalent to the AUC under the trapezoidal ROC curve Wieand et al.,¹⁵ obtained by connecting the ROC data points by straight lines, and the expected value of U, E(U), according to Hajian-Tilaki & Hanley.,¹⁶ is the area under the theoretical (population) ROC curve $\left(\theta \right)$ : $E\left(U\right)=\theta =prob(Y>X).$

An alternative representation, used by DeLong et al.,³ is to define the components of the U statistic for each of the n non-diseased subjects and for each of the m diseased subjects:

$Va{r}_N\left(X_i\right)=\frac{1}{m}\sum _{j=1}^{m}I(X_i,Y_j)andVa{r}_D\left(Y_j\right)=\frac{1}{n}\sum _{i=1}^{n}I(X_i,Y_j).$   (6)

Where are called “pseudo-values” or “pseudo-accuracies.” The pseudo-value  for the ith subject in the non-diseased group is defined as the proportion of Y’s in the sample of diseased subjects where Y is greater than . While for the jth subject in the diseased group is defined as the proportion of X’s in the sample of non-diseased subjects whose X is less than . can be used in place of the original diagnostic test results{X}and{Y}to construct the empirical ROC curve. The average of the sample are respectively given as

$\overline{V}a{r}_N=\frac{1}{n}\sum _{i=1}^{n}Va{r}_N\left(X_i\right)=\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^{m}I(X_i,Y_j)=U.$   (7)

and

$\overline{V}a{r}_D=\frac{1}{m}\sum _{j=1}^{m}Va{r}_D\left(Y_j\right)=\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^{m}I(X_i,Y_j)=U.$   (8)

Therefore $A\hat{U}C=U=\frac{1}{n}\sum _{i=1}^{n}Va{r}_N\left(X_i\right)=\frac{1}{m}\sum _{j=1}^{m}Va{r}_D\left(Y_j\right)$   (9)

Thus, the average of the values for $n\left\{Va{r}_N\right\}$ and the average of those for $m\left\{Va{r}_D\right\}$ are both equivalent to the U statistic, which is why there are called pseudo-accuracy measures. As was shown by Hettmansperger.,¹⁷ the estimate of variance of the U statistic (which he called W instead of U) can be expressed as the sum of variances of and a third component, $U(1-U)/nm.$ DeLong et al.,³ omitted the third component, since it is negligible when n and m are large. They explained that for a single diagnostic test, the variance of AUC is given as

$Var\left[A\hat{U}C\right]=Var\left[\hat{U}\right]=\frac{S_{T_D}^2}{m}+\frac{S_{T_N}^2}{n}.$   (10)

Where $S_{T_D}^{2}and{S}_{T_N}^2$ are respectively the sample variances for the diseased and non-diseased components and are defined as

$S_{T_N}^2=\frac{{\displaystyle \sum _{i=1}^n{[Va{r}_N\left(X_i\right)-AUC]}^2}}{n-1}and{S}_{T_D}^2=\frac{{\displaystyle \sum _{j=1}^m{[Va{r}_D\left(Y_j\right)-AUC]}^2}}{m-1}$   (11)

The null hypothesis of interest is to compare the equality of AUCs from two diagnostic test procedures when the data is paired and by extension if the period of measurement of test results are the same and the test statistic according to DeLong et al.,3 is the Z-test given as

$Z=\frac{AU{C}_1-AU{C}_2}{\sqrt{Var(AU{C}_1-AU{C}_2)}}$   (12)

Where $Var(AU{C}_1-AU{C}_2)=Var\left(AU{C}_1\right)+Var\left(AU{C}_2\right)-2Cov(AU{C}_1,AU{C}_2)$

If the two diagnostic tests are not matched to the same subjects, the two AUCs are independent and the covariance term would be zero. In other to estimates the AUCs for the two diagnostic test procedures, Delong et al.,3 considered that each variance of AUC be defined as

$Var\left(AU{C}_b\right)=\frac{S_{T_{Db}}}{m_b}+\frac{S_{T_{Nb}}}{n_b}$   (13)

Where $S_{T_{Nb}}=\frac{{\displaystyle \sum _{i=1}^{n_b}{\left[Va{r}_N\left(X_{ib}\right)-AU{C}_b\right]}^2}}{n_b-1},b=1,2.$

and $S_{T_{Db}}=\frac{{\displaystyle \sum _{j=1}^{m_b}{\left[Va{r}_D\left(Y_{jb}\right)-AU{C}_b\right]}^2}}{m_b-1},b=1,2.$

The variance of the components $Va{r}_N\left(X_{ib}\right)andVa{r}_D\left(Y_{jb}\right)$ are respectively defined as

$Va{r}_N\left(X_{ib}\right)=\frac{{\displaystyle \sum _{j=1}^{m_b}I(X_{bi},Y_{bj})}}{m_b-1}andVa{r}_D\left(X_{ib}\right)=\frac{{\displaystyle \sum _{i=1}^{n_b}I(X_{bi},Y_{bj})}}{n_b-1},b=1,2.$   (14)

Where $I(X_{ib},Y_{jb})=\{\begin{array}{c}1ifY>X\\ 0.5ifY=X\\ 0ifY<X\end{array}$

$AU{C}_b=\frac{1}{m_b}\sum _{j=1}^{m_b}Va{r}_D\left(Y_{bj}\right)=\frac{1}{n_b}\sum _{i=1}^{n_b}Va{r}_N\left(X_{bi}\right),b=1,2.$   (15)

Note here that $Y_{bj}and{X}_{bi}$ are the observed diagnostic test results for the subjects in group b diagnostic test procedures that are diseased and non-diseased respectively.

Also $Cov(AU{C}_1,AU{C}_2)=\frac{S_{T_{D_1}{T}_{D_2}}}{m}+\frac{S_{T_{N_1}{T}_{N2}}}{n}$   (16)

Where $S_{T_{D_1}{T}_{D_2}}=\frac{1}{m-1}\sum _{j=1}^m\left[Va{r}_D\left(Y_{1j}\right)-AU{C}_1\right]\left[Va{r}_D\left(Y_{2j}\right)-AU{C}_2\right]$

And $S_{T_{N_1}{T}_{N_2}}=\frac{1}{n-1}\sum _{i=1}^n\left[Va{r}_N\left(X_{1i}\right)-AU{C}_1\right]\left[Va{r}_N\left(X_{2i}\right)-AU{C}_2\right]$

Here $S_{T_{D_1}{T}_{D_2}}$ is the pooled variances of diseased test result for the first and second diagnostic test procedure or process, $S_{T_{N_1}{T}_{N_2}}$ is the pooled variances of the non-diseased test result for the first and second diagnostic test process or procedure, $Va{r}_D\left(Y_{1j}\right)$ is the variance of the positive diagnostic test result for the jth subject in the first diagnostic test process, $Va{r}_D\left(Y_{2j}\right)$ is the variance of the positive diagnostic test result for the j^th subject in the second diagnostic test process, $Va{r}_N\left(X_{1i}\right)$ is the variance of the negative diagnostic test result for the i^th subject in the first diagnostic test process and $Va{r}_N\left(X_{2i}\right)$ is the variance of the negative diagnostic test result for the i^th subject in the second diagnostic test process. When the variances are estimated, one can calculate the AUC for the two diagnostic tests and then make comparison.

Bandos et al permutation nonparametric test for comparing AUCs

Bandos et al.,⁶ derived exact and asymptotic permutation test methods to test the equality of two correlated ROC curves which are designed to have increased power to detect difference in the AUC. The test of Bandos et al.,⁶ directly tests for an equality of AUCs. This approach implicitly assumes that both diagnostic test procedures are exchangeable within subject and requires an appropriate transformation, such as ranks, for diagnostic test procedures differing in scale. Bandos et al.,⁶ compared the performance of their test to that of DeLong et al.,³ via simulation and found that the permutation test had greater power than the nonparametric test developed by DeLong et al.,³ when there was moderate correlation between diagnostic tests, large AUCs, and small sample sizes. Bandos et al.,⁶ test is limited by the fact that it requires the exchangeability of the diagnostic test procedures and do requires also the transformations of the original data. It also requires diagnostic tests that are measured on identical scales and so may prove to be less powerful in settings in which the diagnostic test results are skewed Braun & Alonzo.¹¹ If ${\left\{X_i^b\right\}}_{i=1}^n,{\left\{Y_j^b\right\}}_{j=1}^m$ be the test results of the diagnostic procedure b for n actually non-diseased and m actually diseased subjects and ${\left\{x_i^b\right\}}_{i=1}^n,{\left\{y_j^b\right\}}_{j=1}^m$ be approximately transformed test results, an unbiased nonparametric estimator for the AUC for diagnostic procedure or test b can be written as $A\hat{U}{C}_b.$ For a paired sample design, the difference in two AUCs can be estimated as,

$A\hat{U}{C}_1-AU{\hat{C}}_2=\frac{{\displaystyle \sum _{i=1}^n\sum _{j=1}^m\psi (X_i^1,Y_j^1)}}{nm}-\frac{{\displaystyle \sum _{i=1}^n\sum _{j=1}^m\psi (X_i^2,Y_j^2)}}{nm}$   (17)

Where $\psi (X_i^1,Y_j^1)-\psi (X_i^2,Y_j^2)=\{\begin{array}{c}\begin{array}{c}1,if{x}_i^1<y_j^1,x_i^2>y_j^2\\ 0.5,if{x}_i^1<y_j^1,x_i^2=y_j^{2}or{x}_i^1=y_j^1,x_i^2>y_j^2\end{array}\\ 0,if{x}_i^1<y_j^1,x_i^2<y_j^{2}or{x}_i^1>y_j^1,x_i^2>y_j^{2}or{x}_i^1=y_j^1,x_i^2=y_j^2\\ -0.5,if{x}_i^1>y_j^1,x_i^2=y_j^{2}or{x}_i^1=y_j^1,x_i^2<y_j^2\\ -1,if{x}_i^1>y_j^1,x_i^2<y_j^2\end{array}$

Being a member of U statistics, the non-parametric estimator of the AUC difference is known to be asymptotically normally distributed under quite general condition Hoeffding.⁴ Based on this property and the additional assumption of exchangeability, they constructed a simple asymptotic test procedure with test statistic

$\frac{A\hat{U}{C}_1-A\hat{U}{C}_2}{\sqrt{Va{r}_{\Omega }(A\hat{U}{C}_1-A\hat{U}{C}_2)}}\stackrel{d}{\to }N(0,1)$   (18)

Where $\Omega$ is the parameter space.

Sumi et al (McNemar Test) nonparametric method for comparing AUCs

Sumi et al.,⁷ proposed a method for comparing two proportion of positive responses. This test is based on McNemar.,⁸ for the comparison of two diagnostic tests for continuous and discrete binary scale data that are matched. Their McNemar⁸ test is based on the comparison of the equality of the proportion of positive responses in two diagnostic tests. Here each subject"s test result is either positive coded 1 or negative coded 0 on each of two diagnostic processes and interest is in testing whether the proportion of "positive" responses are the same on the first and second diagnostic procedure taken into account the correlation of the two diagnostic test results. This test is limited by the fact that it does not provide evidence of inferiority or superiority of one diagnostic test over another. Any test capable of this should have one sided alternative hypothesis Zhou et al.,¹⁸ The test assumes the use of summarized data which leads to loss of information and reliability in decisions about the data analyzed. Such summarized data could have many ties and if not adjusted for will reduce the power of any test statistic employed for the analysis. It is worthy of mentioning that McNemar⁸ test is concerned with matched pairs of dichotomous test results. Here the result of each diagnostic test are all into two categories, positive coded 1 and negative coded 0.The resulting data is presented in a 2x2 contingency table where row represents the result of one diagnostic test while the column represent the result of another diagnostic test. Here each cell represents the number of observed cases with the particular combination of test results. Depending on the scale of measurement of test results whether continuous or binary, one can compare the two test procedures by constructing a 2x2 contingency table after which McNemar⁸ test can be applied and the result compared with the result obtained using the conventional non-parametric test suggested by DeLong et al.,³ and the permutation test by Bandos et al.,⁶ For two diagnostic tests producing the continuous test results as ${\left\{X_i^b\right\}}_{i=1}^{n}and{\left\{Y_j^b\right\}}_{j=1}^m$ in the bth diagnostic test, the subjects are ordered so that ${\left\{X_i^b\right\}}_{i=1}^{n}and{\left\{Y_j^b\right\}}_{j=1}^m$ becomes the transferred results in the bth diagnostic test for n real negative and m real positive subjects. Suppose we have an optimal cut-off value of $c_b$ for bth diagnostic test, then we classify all results above as positive and results less than or equal to $c_b$ as negative so that the 2x2 contingency table can be constructed for each diagnostic procedure. The resulting table 1 is From Table 1, $A_b$=number of subjects who are diseased and who actually tested positive $(y_j^b>c_b),$ $B_b$=number of subjects who do not have disease and actually tested positive $(x_i^b>c_b),$ $C_b$ =number of subjects with disease and actually tested negative $(y_j^b\le c_b),$ $D_b$ c= number of subjects without disease who actually tested negative $(x_i^b\le c_b).$ Now each diagnostic test result is used to obtain a 2x2 contingency table based on the optimal cut-off value, so that one can verify if the diagnostic test procedure has any effect on the true observed (True) status. To test for the significance of any observed change using the McNemar8 test, one sets up a fourfold table of frequencies representing the first and the second sets of responses (test results) from the same subjects. If both diagnostic test procedures have significant effects, in other words, there are correlated, we can combine the two diagnostic test procedures thus obtaining a matched pair data from the combination of these two diagnostic tests and we obtain a contingency Table 2.

This extension is based on the previous work by Sumi et al.,⁷ who applied the usual McNemar⁸ in comparing correlated marginal probability of positive responses from two diagnostic test procedures. The usual McNemar⁸ test assumes that the data to be used are presented in a summarized form rather than being in a raw form that needs to be processed. Most times, these data may be quantitative in nature and as such may be continuous also meaning that the chances of getting any tied data is at least zero in theory but practically, there exist ties in the data. This is one of the limitations of the usual McNemar test that needs attention. It these ties are adjusted for, the power of the test statistic used for data analysis is increased. To extend the usual McNemar test adopted by Sumi et al.,⁷ to allow for the possible presence of ties in the data, let $(t_{v2},t_{v1})$ be the test results of subjects from diagnostic test 2 case 1 respectively for the vth pair of subjects who are undergoing diagnostic test 2 and 1 say respectively where v=1,2,..,N pairs of subjects in diagnostic test 2 and 1.Assuming that the data is measured on at least interval scale.

$Let{T}_v=\{\begin{array}{c}1,if{t}_{v2}and{t}_{v1}aretestresultsofsubjectsfordiagnostictest2and1respondingpositiveandnegativetotheconditionrespectively\\ 0,if{t}_{v2}and{t}_{v1}aretestresultsofsubjectsfordiagnostictest2and1respondingbothpositiveorrespondingbothnegative\mathrm{}\\ -1,if{t}_{v2}and{t}_{v1}aretestresultsofsubjectsfordiagnostictest2and1respondingnegativeandpositivetotheconditionrespectively\end{array}$ (28)

For the v^th pair of subjects in diagnostic test 2 and 1, where v=1,2,..,N,where N is the total number of pairs.

${\pi }^{+}=P(T_v=1):{\pi }^0=P(T_v=0);and{\pi }^{-}=P(T_v=-1)$ (29)

Where ${\pi }^{+}+{\pi }^0+{\pi }^{-}=1$ (30)

Therefore let $W=\sum _{v=1}^N{T}_v$ (31)

Where W is the total number of subjects in the matched pairs of subjects who test or respond positive. Based on the above specifications, the expected value of $T_v$ is

$E\left(T_v\right)={\pi }^{+}-{\pi }^{-}$ (32)

While $Var\left(T_v\right)={\pi }^{+}+{\pi }^{-}-{({\pi }^{+}-{\pi }^{-})}^2$ (33)

From equations 6 and 7, expected value of W is

$E\left(W\right)=n({\pi }^{+}-{\pi }^{-})$ (34)

Adding from equation 8

$Var\left(W\right)=N({\pi }^{+}+{\pi }^{-}-{({\pi }^{+}-{\pi }^{-})}^2)$ (35)

Note that ${\pi }^{+},{\pi }^0\text{}and{\pi }^{-}$ are respectively the probabilities that for a randomly selected pair of subjects from diagnostic tests 2 and 1, the subjects from diagnostic test 2 on the average responds positive and the subjects from diagnostic test 1 responds negative or the subjects from diagnostic test 2 and 1 both respond positive or the subjects from both diagnostic tests respond negative, or the subjects from diagnostic test 2 responds negative and subjects from diagnostic test 1 responds positive. The sample estimates of these probabilities are respectively defined as

${\hat{\pi }}^{+}=\frac{p^{+}}{N};{\hat{\pi }}^0=\frac{p^0}{N}and{\hat{\pi }}^{-}=\frac{p^{-}}{N}$ (36)

where $p^{+},p^{0}and{p}^{-}$ represents respectively the frequencies 1"s,0"s and -1"s in the distribution given in $T_v,v=1,2,\mathrm{...},N.$ That is, $p^{+},p^{0}and{p}^{-}$ are respectively the number of diagnostic test 2 and 1 subject pairs in which the diagnostic test 2 respond positive and the diagnostic test 1 respond negative or the diagnostic test 2 and 1 subjects both respond positive or both respond negative or the diagnostic test 2 responds negative and the diagnostic test 1 subject responds positive. These frequencies are expressed in terms of diagnostic tests 2 and 1 in Table 3.

There are respectively represented from Table 3 as

$p^{+}=n_{12};p^0=n_{11}+n_{22}=p^{0+}+p^{0-};p^{-}=n_{21}\text{\hspace{0.17em}\hspace{0.17em}}$ (37)

Where $p^{0+}=n_{11};p^{0-}=n_{22}\text{\hspace{0.17em}}$ (38)

are respectively the number of diagnostic test 2 and 1 subject pairs where diagnostic test 2 and 1 subjects both respond positive or both respond negative and ${\hat{\pi }}^{0+}and{\hat{\pi }}^{0-}$are the corresponding relative frequencies.

But ${\pi }^{+}-{\pi }^{-}$ measures the difference in rate of positive responses by subjects in the diagnostic test 2 and diagnostic test 1 procedure and its estimate of the sample is

${\pi }^{+}-{\pi }^{-}=\frac{W}{N}=\frac{p^{+}-p^{-}}{N}$ (39)

And the variance is estimated from Equ 35 as

$Var({\hat{\pi }}^{+}-{\hat{\pi }}^{-})=\frac{Var\left(W\right)}{N^2}=\frac{{\hat{\pi }}^{+}+{\hat{\pi }}^{-}-{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}{N}$ (40)

But the McNemar test statistic is ${\chi }^2={\left(\frac{(p_2-p_1)}{Se(p_2-p_1)}\right)}^2=\frac{{(B-C)}^2}{B+C}$ with the numerator given as

$W^2={\left(N({\hat{\pi }}^{+}-{\hat{\pi }}^{-})\right)}^2={(p^{+}-p^{-})}^2$ (41)

Now a test statistic explaining the difference between positive response rates for diagnostic test 2 and 1 subjects can be developed by noting that ${\pi }^{+}$ represents the proportion of pairs of subjects out of a total of N pairs in which the subject from diagnostic test 2 procedure and was given say T2 treatment in a given pair responds positive and the subject from diagnostic test 1 in the pair and given treatment T1 say, responds negative; ${\pi }^0$ represents the proportion of the total number of N pairs of subjects with the members of the pair both responding positive or both responding negative and ${\pi }^{-}$ is the proportion of pairs out of a total of "N" pairs in which the subject from diagnostic test 2 procedure and was given say T2 in a given pair responds negative and the subject from diagnostic test 1 in the pair and given treatment T1 responds positive. The diagnostic test 2 and 1 differential positive response rate is given as ${\pi }^{+}-{\pi }^{-}$ with their sample estimate and variance given respectively by Eqns 39 and 40. If the sampled proportion is given respectively as $p_1=\frac{A+C}{N}and{p}_2=\frac{A+B}{N}$ based on Table 1, we obtain more important and detailed information given as

$P_1=\frac{n_{11}+n_{21}}{N}=\frac{p^{0+}+p^{-}}{N}={\hat{\pi }}^{0+}+{\hat{\pi }}^{-}$ (42)

And $P_2=\frac{n_{11}+n_{12}}{N}=\frac{p^{0+}+p^{+}}{N}={\hat{\pi }}^{0+}+{\hat{\pi }}^{+}$ (43)

$where{\hat{\pi }}^{0+}=\frac{p^{0+}}{N}and{\hat{\pi }}^{0-}=\frac{p^{0-}}{N}$ (44)

$suchthat{\hat{\pi }}^0={\hat{\pi }}^{0+}+{\hat{\pi }}^{0-}$ (45)

Now the null hypothesis H0 of interest is to test that the proportions of subjects responding positive in the diagnostic test 2 and 1 procedures or treatment conditions T2 and T1 differ by some value ${\beta }_0$.This is equivalent to testing the null hypothesis given as

$H_0:{\pi }^{+}-{\pi }^{-}={\beta }_{0}versus{H}_1:{\pi }^{+}-{\pi }^{-}\ne {\beta }_0(-1\le {\beta }_0\le 1)\text{\hspace{0.17em}}$ (46)

While the test statistic is given by

${\chi }^2=\frac{{(W-n{\beta }_0)}^2}{N({\hat{\pi }}^{+}+{\hat{\pi }}^{-}-{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2)}$ (47)

Or equivalently

${\chi }^2=\frac{n{\left(\right({\hat{\pi }}^{+}-{\hat{\pi }}^{-})-{\beta }_0)}^2}{{\hat{\pi }}^{+}+{\hat{\pi }}^{-}-{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}$ (48)

which with 1 degree of freedom is approximately chi-square distributed for sufficiently large "n". The null hypothesis of equal population proportion of positive responses is rejected at the $\alpha$ level of significance in favour of the alternative hypothesis if

${\chi }^2\ge \chi {\text{}}_{1-\alpha ;1}^2$ (49)

Note therefore that under null hypothesis H0, the numerators of the extended test statistic of Equs 47 and 48 are as in the usual McNemar⁸ test statistic independent of $n_{11}=p^{0+}and{n}_{22}=p^{0-}$ the number of pairs in which diagnostic test 2 and 1 subjects in each pair both respond positive or both respond negative to the conditions of interest while for equations 47 and 48, the denominator is also independent of n11 and n22.Hence both the extended test statistic and the usual McNemar⁸ test statistic are not affected by those pairs in which the subjects in each pair both respond positive or both respond negative to the disease or treatments condition. Unlike the usual McNemar test statistic, the extended McNemar⁸ test has by specifications been adjusted and corrected for the possible presence of ties in the data. In addition, the variance of the extended McNemar test statistic in Eqn 48 is smaller than the variance of the usual McNemar test statistic stated in between eqns 40 and 41.This is because of the fact that $Var({\hat{\pi }}^{+}-{\hat{\pi }}^{-})=\frac{Var\left(W\right)}{N^2}=\frac{{\hat{\pi }}^{+}+{\hat{\pi }}^{-}-{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}{N}$ and $Se(p_2-p_1)=\frac{\sqrt{B+C}}{N}$ so that

$Var({\hat{\pi }}^{+}-{\hat{\pi }}^{-})=\frac{{\hat{\pi }}^{+}+{\hat{\pi }}^{-}-{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}{N}=\frac{n_{12}+n_{21}}{N^2}-\frac{{(n_{12}+n_{21})}^2}{N^3}\le \frac{n_{12}+n_{21}}{N^2}=Var(P_2-P_1),$

$\sin ce\frac{{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}{N}=\frac{{(n_{12}+n_{21})}^2}{N^3}\ge 0,forall\text{\hspace{0.17em}}{\pi }^{+}\ne {\pi }^{-},or{n}_{12}\ne n_{21}$

In conclusion, the extended McNemar test statistic is relatively more efficient and so is most likely to be more powerful than the usual McNemar test statistic whenever the diagnostic test 2 and 1 test results of subjects have differences in positive response rates $({\hat{\pi }}^{+}\ne {\hat{\pi }}^{-};or{P}_1\ne P_2)$ to the conditions of interest. It is note worthy that $N{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2$ is the reduced value in the variance of W since by specifications of equation 28 it has been adjusted for the possible presence of ties between the responses of diagnostic test 2 and 1 procedures. The major difference between the usual McNemar⁸ test and the extended McNema⁸ test is that there is adjustment of possible presence of tied observations in the later test, the extended McNemar⁸ test statistic will likely have smaller variance and larger calculated chi-square value than the usual McNemar⁸ test statistic, thus leading to the more chances of committing Type II error in the usual McNemar⁸ test more often than in the extended McNemar⁸ test.

We carried out computer simulations here to evaluate the performance of the extended McNemar test. We performed extensive simulations to evaluate and compare Type I errors (empirical test sizes) and statistical power of the extended McNemar⁸ test, usual (traditional) McNemar test, conventional nonparametric test of DeLong et al.,³ and asymptotic test of Bandos et al.,⁶ Here we assumed equal correlation coefficient across the two diagnostic test procedures for diseased and non-diseased test results of subjects measured on continuous and discrete binary scales and the sample sizes are 20,60,100 and 180. These test results of subject were generated from a standard bi-variate normal distribution having mean and variance respectively for the two diagnostic tests as ${\mu }_1,{\sigma }_1^{2}and{\mu }_2,{\sigma }_2^2$ when measurement of data is on continuous scale. The AUC for diagnostic test 1 and 2 procedures are respectively given as $AU{C}_1=\Phi \left(\frac{{\mu }_1}{\sqrt{(1+{\sigma }_1^2)}}\right)andAU{C}_2=\Phi \left(\frac{{\mu }_2}{\sqrt{(1+{\sigma }_2^2)}}\right)$ where $\Phi$ is the standard normal cumulative distribution function. Under binary random variable X for one diagnostic test procedure, if the test result of subject is positive, it is coded 1 and if the test result is negative, it is coded 0. If binary variables (X,Y) is assumed for correlated diagnostic test procedures, the joint distribution of X and Y is determined. The correlation coefficient $\left(r\right)$ of X and Y is determined and having the range $-1\le r\le +1.$ For data on binary scale of measurement, correlated binary test results were generated with required probabilities of positive responses $(P_1,P_2)$ to obtain specific difference $({\pi }^{+}-{\pi }^{-}={\beta }_{0}or{\pi }_b-{\pi }_t\ge {\beta }_0)$ between the probability of positive responses for the two diagnostic test procedures for the extended McNemar test and the proposed chi-square test respectively. The binary test results for the non-diseased subjects, are generated by fixing the probability of positive responses as 0.30 and 0.35. This procedure of simulating binary data is in line with the previous works of Leisch et al.,¹⁹ and Islam et al.,²⁰ who discussed the algorithm for simulating correlated binary test results. The SAS version 9 is the statistical software used to perform the simulation study.

The range of values of the correlation coefficient r for the extensive simulation for continuous test results and values of parameters (a and b) for estimating mean $({\mu }_1,{\mu }_2)$ and variance $({\sigma }_1^2,{\sigma }_2^2)$ parametrically as drawn to obtain the difference between two AUCs ranges from 0 to 0.3. For binary test results, the correlation coefficient r is also taken to range from 0.25 to 0.75 and the probability of positive responses were drawn so as to obtain the difference between probability of positive responses of subjects for the two diagnostic test procedures and it ranged from 0 to 0.2. For either binary or continuous scenario considered, we used 2000 replications in running the simulations. Table 5 compares the empirical test size (Type I error) and the statistical power of the extended McNemar⁸ test to the usual McNemar⁸ test proposed by Sumi et al.,⁷ to the conventional nonparametric test developed by DeLong et al.,³ and to asymptotic permutation test developed by Bandos et al.,⁶ for comparing two diagnostic test procedures for continuous test results. This comparison was similarly carried out for binary test results. The estimates of Type I error as well as estimates of the statistical power are obtained when the proportion of positive responses or the true AUCs for the two diagnostic test procedures are the same and different respectively as can be seen in Table 5&6. The rejection regions for the two tests are determined using 5% as level of significance.

For smaller AUCs, the extended McNemar test indicates a more conserved empirical test size (type I error) and thereafter an increased statistical power when compared to the traditional McNemar⁸ test by Sumi et al.,⁷ conventional nonparametric method by DeLong et al.,³ and asymptotic permutation test by Bandos et al.,⁶ when the test results is continuous. But when the correlation coefficient is moderate and for increased sample size for the two diagnostic test procedure, stability appears to be more in the scenario considered (continuous case) and the five tests mentioned above tends to be very close in terms of their empirical test size and statistical power. The extended McNemar⁸ test shows more false positive rate (FPR) when the correlation coefficient r is smaller. This is because the McNemar⁸ test are most suitably used when the data is correlated. However, when the correlation coefficient r is increased, the estimate of FPR reduces drastically. In the same way, when the AUCs is increased, the estimates of the empirical test size (type I error) for every sample sizes and all values of correlation coefficients can be compared. The extended McNemar test discriminates better than the traditional McNemar test by Sumi et al.,⁷ conventional nonparametric test by DeLong et al.,³ and the permutation test Bandos et al.,⁶ when the AUCs are getting higher and for lower values of correlation coefficients.

When the AUCs values are high and for moderate values of the correlation coefficient, the other three tests namely the usual McNemar⁸ test by Sumi et al.,⁷ test by DeLong et al.,³ and test by Bandos et al.,⁶ gives better statistical power than the extended McNemar test but when the sample sizes increases, the extended McNemar⁸ test provides very close statistical power to the others. In considering the binary test results in all aspects of parameter settings and for either big or small sample sizes, the extended McNemar⁸ test shows lower conservative empirical test size (Type I error) and shows higher statistical power when compared to tests by Sumi et al.,⁷ DeLong et al.,³ and Bandos et al.,⁶ Finally, in the continuous case situation, the results of the simulation shows that the proposed chi-square test and the extended McNemar⁸ test gives very close harmony of Type I error to the significant level $\alpha$ but when the values of AUCs are low this harmony yields or provides among the diagnostic test procedures moderate and very high correlation coefficient. Also having greater or higher sample sizes in the continuous case also makes the extended McNemar⁸ test have statistical power that is very comparable to other existing nonparametric methods of comparing correlated AUCs. In addition, for the discrete binary case, the extended McNemar⁸ test possesses higher operating characteristics than other existing tests considered in all the settings of parameter. The performance of the extended McNemar⁸ test may be impaired in a simulation study when the test result is continuous because of the problem of choosing or finding an optimal cut-off value for classifying the test results of subjects. To make this point clearer, we in the next section will adopt a known standard data set that already has a real cut-off value and we will conduct a bootstrap power analysis so as to compare the statistical power of all the four tests namely, extended McNemar⁸ test, usual McNemar⁸ test by Sumi et al.,⁷ conventional test by DeLong et al.,³ and permutation test by Bandos et al (Table 4&5).

In other to demonstrate the workability of the new non-parametric method (extended McNemar test) for comparing correlated proportion of positive responses, we consider a practical data set adopted from Venkatraman & Begg⁵ who carried out a distribution free procedure for comparing ROC curves from a paired experiment. This study was aimed at evaluating the performance of two diagnostic test results obtained from the anterior and posterior nodes in the cause of diagnosing Melanoma.

To demonstrate the feasibility of the extended McNemar test, we made use of the data from this study whose objective was to investigate the performance of two diagnostic test results obtained from anterior and posterior nodes for diagnosing Melanoma. The data presented in Table 4 in Venkatraman & Begg⁵ provide the results using a clinical scoring system and a dermoscopic scoring scheme. The purpose of the analysis is to determine whether the dermoscope contributes similar diagnostic information. The null hypothesis is that the dermoscope contributes the same information as the clinical scoring system. This is the same as testing the null hypothesis that the sizes of anterior and posterior nodes possess equivalent diagnostic information. Using these data, estimates of proportion of positive responses for the two diagnostic tests 1 and 2 procedures are 0.725 and 0.652 respectively and the estimated correlation coefficient between the two diagnostic tests is 0.157. To test equivalence of the accuracy of these two diagnostic tests, the conventional test by DeLong et al.,³ asymptotic permutation test by Bandos et al.,⁶ the usual McNemar⁸ test by Sumi et al.,⁷ and the extended McNemar8 test are in agreement of significant different performances yielding two tailed p-values of 0.0048,0.017,0.0028,0.0019 respectively.

Bootstrap power analysis for comparing the statistical power of tests

The bootstrap is a powerful nonparametric approach Efron.²¹ In an effort to obtain better and more specific knowledge regarding statistical power of tests, we have conducted a bootstrapping study where for each of considered sample sizes, 2000 random samples were taken from the data and rejection rates are computed.

Table 6 shows that given all sample sizes, the extended McNemar test provides the highest superior rejection rate followed by the McNemar⁸ test by Sumi et al.,⁷ and so on. At increased sample sizes, tests by DeLong et al.,³ Bandos et al.,⁶ and Sumi et al.,⁷ shows rejection rates very closed to the Extended McNemar⁸ test.

The new test for comparing correlated proportion of positive responses can be applied to real life data on gestational diabetes mellitus (GDM). Actually a random sample of 1113 pregnant women who tested positive for 50g Glucose Challenge Test (GCT) indicating that their plasma blood glucose level were at least 140 mg/dl after 1 hour. These same numbers of pregnant women were subsequently recalled and further subjected to two competing diagnostic test procedures, namely, 2-hour 75g OGTT and 3-hours 100g OGTT at various gestation periods according to the standard of World Health Organization²² and National Diabetes Data Group.²³ These two diagnostic test procedures are paired. Women who were known diabetics, or who were suffering from any chronic illness were excluded from the study. The data is measured on a continuous scale and is dichotomized using at 7.8mmol/l or at least 140 mg/dl as cut-off value which is the recommended cut-off value for diagnosing GDM WHO.²² Pregnant women whose test result is at least 7.8mmol/l is considered diseased (positive, coded 1) otherwise; they are not diseased (negative, coded 0). The data for the GDM response variables (tests results) for diagnostic test 1 and 2 procedures, namely 75g OGTT and 100g OGTT are paired and hence correlated for the 1113 pregnant women considered for this study. The null hypothesis of interest is testing the equality of the proportion of positive responses for the two diagnostic test procedures. The dichotomized data for the two diagnostic tests are as usual cross classified and presented in a contingency table to demonstrate the feasibility of the new nonparametric methods as well as the existing methods considered. We therefore obtain the sample estimates ${\hat{\pi }}^{+},{\hat{\pi }}^{0}and{\hat{\pi }}^{-},$ variance estimates and the McNemar test statistic and test the null hypothesis. In applying the extended McNemar test to the data, we evaluate the values of $T_v$ of Eqn 29 where $t_{v1}and{t}_{v2}$ are test results respectively by the subjects in the vth pair of diagnostic test 1 and diagnostic test 2 procedures for $v=1,2,\mathrm{....},1113.$ From the values of $T_v$, we have that

$p^{+}=n_{12}=270,p^0=n_{11}+n_{22}=p^{0+}+p^{0-}=134+157=291;p^{-}=n_{21}=556$

From Eqn 36, we have the sample estimates as ${\hat{\pi }}^{+}=\frac{270}{1113}=0.2426;{\hat{\pi }}^0=\frac{291}{1113}=0.2615;{\hat{\pi }}^{-}=\frac{556}{1113}=0.4995;$

$But{\hat{\pi }}^0=\frac{291}{1113}=\frac{134}{1113}+\frac{157}{1113}=0.1204+0.1411={\hat{\pi }}^{0+}+{\hat{\pi }}^{0-}.$

$AlsoW=p^{+}-p^{-}=n_{12}-n_{21}=270-556=-286.$

From Eqn 11, we have the estimated variance of W as

$Var\left(W\right)=\left(1113\right)(0.2426+0.4995-{(0.2426-0.4995)}^2)=\left(1113\right)(0.7421-0.0660)=\left(1113\right)\left(0.6761\right)=\mathrm{752.4993.}$

Therefore to test the null hypothesis of equation 46 using the extended McNemar test statistic we have from Eqn 47 with ${\beta }_0=0$ that ${\chi }^2=\frac{{(270-556)}^2}{752.4993}=\frac{81796}{752.4993}=108.69(P-value=0.0012)$ which with 1 degree of freedom is statistically significant showing that diagnostic test 1 and diagnostic test 2 do have differential effect of GDM on pregnant women. In other words, the probability of positive responses from the two diagnostic test procedures for the pregnant women differs significantly. To differ this result, we make use of the usual McNemar⁸ test which was adopted by Sumi et al.,⁷ to analyze the GDM data that the estimated variance of P2-P1 is $Var(P_2-P_1)=\frac{p^{+}+p^{-}}{N^2}=\frac{n_{12}+n_{21}}{N^2}=\frac{270+556}{{\left(1113\right)}^2}=\frac{825}{1238769}=\mathrm{0.000667.}$ Its test statistic for the H0 of Eqn 36 with ${\beta }_0=0$ is ${\chi }^2=\frac{{(270-556)}^2}{270+556}=\frac{81796}{826}=99.03(P-value=0.0028)$ which with 1 degree of freedom is also statistically significant. Even though the extended McNemar⁸ test statistic and the usual McNemar⁸ test statistic had both lead to the rejection null hypothesis, the relative sizes of the calculated chi-square values and the p-values obtained indicates that the usual McNemar⁸ test statistic as adopted by Sumi et al.,⁷ has greater chances of leading to Type II error more often than the extended McNemar⁸ test statistic. Also, we note that the estimated variance of ${\hat{\pi }}^{+}-{\hat{\pi }}^{-}\text{}is\mathrm{var}({\hat{\pi }}^{+}-{\hat{\pi }}^{-})=\frac{0.2426+0.4995-{(0.2426-0.4995)}^2}{1113}=\frac{0.7421-0.0660}{1113}=\frac{0.6761}{1113}=0.000607$ which is $0.000667-0.000607=0.00006=\frac{0.0660}{1113}=\frac{{({\hat{\pi }}^{+}-{\hat{\pi }}^{-})}^2}{N},$ smaller as expected than the variance of P2-P1 obtained when the usual or unmodified McNemar test is used.

Application of existing tests to the real life data

Applying the tests on the real life data, we obtain the following estimates of AUCs for the two diagnostic tests, the correlation coefficients between the test results of the two diagnostic test procedures and the p-values after testing for the equality of performance of the two diagnostic test procedures as.

From Table 7 results indicates that all tests showed significant difference since the p-values are less than the chosen level of significant of 5 percent at increased sample size of 1113 for the data on GDM. Overall result shows that the extended McNemar⁸ test are in agreement of significant different in their performances and therefore out performs other tests considered in this work.