Parameter and hypotheses of interest

Assume an experimental setup, where for each treatment i, there are Ji replicated experimental units (tanks, cages, animals, petri dishes, etc.) with index j = 1, …, Ji. In each experimental unit there is a number of nij biological units under observation, and the number of events of interest, xij, is counted in unit ij. These counted events may be death (or survival) of animals, mobility (or immobility) of daphnids, presence or absence of micronuclei, etc. Consider the comparison of one dose group, i = 1, to the untreated control, i = 0. Denote the unknown probability of events in the two groups by i, and parameter of interest is the risk ratio $p=\frac{{\pi }_1}{{\pi }_0}$ . Beside estimating ρ and displaying the uncertainty of this estimate in terms of a 95% confidence interval, decisions concerning one-sided hypotheses on tests on non-inferiority or superiority may be of interest. The particular choice of non-inferiority margins, ρ0, may be fixed by convention, compare¹ suggesting ρ0 = 0.75 or ρ0 = 0.8 for certain applications. In other situations, it might be a matter of debate. Although general focus is in valid two-sided confidence intervals (i.e., with coverage probability close to the nominal level), it will be further investigated whether confidence intervals do also provide valid upper and lower confidence limits and can thus be used to perform (approximate) level α test for one-sided hypotheses.

Overdispersed binomial data

There are two well-known approaches to model overdispersion in binomial data.⁸ The quasibinomial approach models overdispersion by assuming the variance-mean-dependency
$V^{QB}\left(X_{ij}\right)=\varphi n_{ij}{\pi }_{ij}\left(1-{\pi }_{ij}\right)$ ,
where ϕ is the overdispersion parameter that inflates the binomial variance term by a common fold, independent of the sample size nij. In this parameterization, the binomial assumption,
$V^B\left(X_{ij}\right)=n_{ij}{\pi }_{ij}\left(1-{\pi }_{ij}\right)$ , is met for ϕ = 1.

The beta-binomial distribution derives from a beta mixture of binomial distributions, i.e.,
${\pi }_{ij}\sim Beta(a_1,b_1)$ , and $x_{ij}\sim Binomial(n_{ij},{\pi }_{ij})$ (1)

where $E({\pi }_{ij})=\frac{a_i}{(a_i+b_i)}$  , $E(x_{ij})=\frac{n_{ii}{a}_i}{(a_i+b_i)}$ [9]. Denote the sum of the two parameters of the beta-parameters by $a_i^{*}=a_i+b_i$  . The variance of beta-binomial counts, $v^{BB}(X_{ij})$ is a function of nij, πij, $a_i^{*}$ [5,9]. When $a_i^{*}$  approaches$\infty$ , the variance of the beta-binomial counts approaches that of binomial counts. Here, the overdispersion relative to the binomial variance,$v^{BB}(X_{ij})$  , is denoted by ${\varphi }^{BB}=\frac{V^{BB}(X_{ij})}{V^B(X_{ij})}$  and is a function of nij and $a_i^{*}$  . This allows to choose $a_i^{*}$  given nij such that the over dispersion$a_i^{*}$  is a constant factor, namely ${\phi }^{BB}$  .

Main interest here is in the performance of confidence interval methods in highly controlled laboratory settings. Under such conditions it can be assumed that nij is equal for all experimental units, ij. Under such conditions, the quasibinomial assumption on the variance mean dependency coincides with the variance-mean-dependency under the beta-binomial distribution.^5,9 Hence, methods are considered that are explicitly constructed for the beta-binomial distribution, as well as methods that account for overdispersion under the quasibinomial assumption.

Confidence interval methods

Fieller-type intervals applied on the observed proportions (FN):

Naively, the t-test for ratios, with a common variance estimator and the assumption of normal distributed residuals, may be used to test the above hypotheses, treating the observed proportions as the variable of interest, $y_{ij}={\stackrel{⌢}{\pi }}_{ij}$ . The corresponding Fieller-type confidence interval can be obtained by analytically inverting the t-test statistic for ratios.¹⁰ The method assumes normal distribution and variance homogeneity for the observed proportions, which is clearly not the case in this application. This interval is referred to as FN.

Binomial MOVER-R-Wilson method after summing-up observed counts across replications (BM): It may be tempting to sum up the counts over experimental units within each treatment group, $x_i={\displaystyle \sum {}_{j=1}^{j_i}}{x}_{ij},n_i={\displaystyle \sum {}_{j=1}^{j_i}}{n}_{ij}$ and apply a confidence interval for risk ratios under the assumption of binomial distribution, i.e., ignoring possible extra-binomial variation. As a place holder for the many available options, here the MOVER-R method proposed by² is used; this is computationally simple and was among the best methods in a recent comparative study under the binomial assumption by.¹¹ It is referred to as BM.

The two above methods are merely included here to illustrate the effects of either ignoring the mean-variance-relation and skewness implied by binomial distribution (FN) or the effect of applying binomial methods (BM) when data are indeed overdispersed binomial.

Asymptotic method on the log-scale, accounting for overdispersion via residual variation (LOD): Among other methods,³ investigate and recommend a method based on the delta method applied for the log risk ratios (called MR1 therein). It can be computed from:

$n_i={\displaystyle \sum _{j=1}^{j_i}{n}_{ij}},x_i.={\displaystyle \sum _{j=1}^{j_i}{x}_{ij,}{\widehat{\pi }}_i}=\frac{x_{i.}+0.5}{n_{i.}+1},\widehat{p}=\frac{{\widehat{\pi }}_1}{{\widehat{\pi }}_0}$  ,${\widehat{u}}_i=\left(\frac{j_i}{j_i-1}\right){\displaystyle {\sum }_{j=1}^{j_i}\frac{{(x_{ij}-{\widehat{\pi }}_{i.}{n}_{ij})}^2}{n_i^2}}$  (2)

Where the variance is estimated from the residuals on the scale of the original observations. The interval is then given by:

$\widehat{p}\exp (\pm z_{1-\frac{\alpha }{2}}\sqrt{{\displaystyle \sum _{i=0}^1\frac{{\widehat{u}}_1}{{\widehat{\pi }}_i^2}}})$                                                                                         (3)

By plugging-in the observed residual variance per treatment group i, this method does not assume a particular mean-variance relation and accounts for overdispersion in a more general way. However, if the number of replications per treatment, Ji, is small, these variance estimates might be unstable. Zaihra and Paul³ additionally consider a closely related method with a sandwich-type variance estimator, which performs worse in their simulation study, and is thus ignored here.

Fieller-Bailey-type interval accounting for overdispersion via residual variance (FOD): Using the estimators above,³ follow the approach of¹⁰ and¹² that accounts for the skewed distribution of the original Fieller statistic

$z=\frac{{\widehat{\pi }}_1-p{\widehat{\pi }}_0}{\sqrt{{\widehat{v}}_1+p^2{\widehat{v}}_0}}$

 By considering the solutions of a cubic equation, with
$A={\widehat{\pi }}_0^{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}-z_{\raisebox{1ex}{$\alpha $}\!\left/ \!\raisebox{-1ex}{$2$}\right.}^2\frac{{\widehat{v}}_0}{9{\widehat{\pi }}_0^{\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}},B={({\widehat{\pi }}_1{\widehat{\pi }}_0)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.},C={\widehat{\pi }}_1^{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}-z_{\raisebox{1ex}{$\alpha $}\!\left/ \!\raisebox{-1ex}{$2$}\right.}^2\frac{{\widehat{v}}_1}{9{\widehat{\pi }}_1^{\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}}$

The interval (referred to as MR4 by³) can then be computed by

$\left[\max \left(\left(\frac{B-\sqrt{B^2-AC}}{A}\right),0\right);{\left(\frac{B-\sqrt{B^2-AC}}{A}\right)}^3\right]$

If A > 0 and B2 - AC > 0. If these two restrictions are not met, the interval has unbounded or disjoint solutions which do not provide meaningful interpretations of ρ. In the simulation study below, the interval $\left[0,\infty \right]$ is returned in such cases [3]. Again consider a closely related method that uses sandwich estimator for the variance of ${\widehat{\pi }}_i$  instead. It is not considered here.

Delta method on the log-scale under the beta-binomial assumption (LBB): Lui et al.² propose methods that are constructed under the assumption of the beta-binomial distribution. Theoretically, the variance of ${\widehat{\pi }}_i$ under this assumption is ${\widehat{\pi }}_i\left(1-{\pi }_i\right){\varphi }^{BB}\left(n_{i,C_i}\right)/n_i$ where ${\varphi }^{BB}\left(n_{i,C_i}\right)$  is the beta-binomial overdispersion factor, expressed as a function of the number under risk in each replication ij of treatment i, ${\varphi }^{BB}\left(n_{i,C_i}\right)$ and of ci, the intraclass correlation coefficient. Under beta-binomial sampling, the intraclass correlation depends on $a_i^{*}$  via $c_i=1/(a_1^{*}+1)$.²

Lui et al.² estimate the intraclass correlation using

${\widehat{c}}_i=\frac{BM{S}_i-WM{S}_i}{BM{S}_i+\left(n_1^{*}-1\right)WM{S}_i},with-n_1^{*}=\frac{n_{i.}^2-n_{ij}^2}{\left(J_i-1\right)n_i}$

Based on the between and within mean squared error of the observations,

$BM{S}_i=\frac{{\displaystyle {\sum }_{j=1}^{J_1}\frac{x_{ij}^2}{n_{ij}}-\frac{x_{i.}^2}{n_{i.}}}}{J_i-1}$ and $WM{S}_i=\frac{x_{i.}-{\displaystyle {\sum }_{j=1}^{J_i}\frac{x_{ij}^2}{n_{ij}}}}{{\displaystyle {\sum }_{j=1}^{J_1}(n_{ij}-1)}}$

This leads to estimators for the beta-binomial overdispersion factor, and the related variance of the proportion estimator under the beta-binomial assumption, for each treatment group i separately:²

${\widehat{\varphi }}^{BB}\left(n_i,{\widehat{c}}_i\right)={\displaystyle {\sum }_{j=1}^{J_i}{n}_{ij}}\left(1+\left(n_{ij}-1\right){\widehat{c}}_i\right)/n_{i.}$                                                                                 (5)

${\widehat{v}}_i^{BB}=\frac{\left(1-{\widehat{\pi }}_i\right){\widehat{\varphi }}^B{}^B\left(n_{i_1}{\widehat{c}}_i\right)}{n_i{\widehat{\pi }}_i}$                                                                                                                         (6)

The asymptotic interval relying on the delta method applied for the log risk ratio [2] can be constructed by:
$\widehat{p}\exp \left(\pm Z_1-\alpha /2\sqrt{{\displaystyle {\sum }_{i=0}^1{\widehat{v}}_i^{BB}}}\right)$                                                                                   (7)

Fieller-Bailey-type interval under beta-binomial assumption: FBB: Lui et al.² consider a Fieller-type interval and its modification according to¹² under the beta-binomial distribution:
$A={\widehat{\pi }}_0^{2/3}-z_{\alpha /2}^2\left(1-{\widehat{\pi }}_0\right){\varphi }^{BB}\left(n_0,{\widehat{c}}_0\right)/\left(9n_{0.}{\widehat{\pi }}_0^{1/3}\right),B={\left({\widehat{\pi }}_1{\widehat{\pi }}_0\right)}^{1/3}$ and
$C={\widehat{\pi }}_1^{2/3}-z_{\alpha /2}^2\left(1-{\widehat{\pi }}_1\right){\varphi }^{BB}\left(n_1,{\widehat{c}}_1\right)/\left(9n_{1.}{\widehat{\pi }}_1^{1/3}\right)$

As in Eq. (4), a meaningful interval can be calculated if A > 0 and B2 - AC > 0:

$\left[\max \left({\left(\frac{B-\sqrt{B^2-AC}}{A}\right)}^3,0\right);{\left(\frac{B-\sqrt{B^2-AC}}{A}\right)}^3\right]$                                                                       (8)

In the simulation study below, the interval $[0,\infty ]$  is returned if A > 0 and B2-AC > 0. Following [2], in case of the extreme events ${\widehat{\pi }}_i=0$  or ${\widehat{\pi }}_i=1$  , is replaced by ${\widehat{\pi }}_i=\frac{x_{i.}+0.5}{n_i+1}$  in the LBB and FBB method and their subsequent modifications.

Modifications of LOD, FOD, LBB, FBB by pooling and restricting the variance estimates: Lui et al² state, based on theoretical considerations and in the context of estimation problems that the intraclass correlation ci and the overdispersion ${\varphi }_{BB}\left(n_i,c_i\right)$  cannot fall below 0 and 1, respectively. However, their estimates may fall below the boundaries imposed by the binomial assumption. For example, the event $x_{i1}=x_{i2}=\mathrm{....}=x_i{j}_i$  may lead to the unreasonable estimates $c_i<0$  and${\varphi }_{BB}\left(n_i,c_i\right)=0$ . Moreover, interest here is in small sample laboratory experiments and estimating ci and ${\varphi }_{BB}\left(n_i,c_i\right)$ separately for each treatment from very few replications Ji may result in over fitting. Then, the assumption of a common beta-binomial overdispersion parameter ${\varphi }_{{}^{BB}}\left(n_i,{\widehat{c}}_i\right)$ may lead to a more stable estimation with small sample sizes. Therefore, the methods LBB and FBB as well as LOD an FOD are simulated with the following additional restrictions and pooling of variance estimates:

1. LBB1, FBB1 refer to methods LBB, FBB with the beta-binomial overdispersion factor restricted to be at least 1, i.e., using $max(1,{\varphi }_{{}^{BB}}\left(n_i,{\widehat{c}}_i\right))$  instead of ${\varphi }_{{}^{{}_{{}_{BB}}}}\left(n_i,{\widehat{c}}_i\right)$ in the equations (5,6)ff.
2. LBBp, FBBp refer to methods LBB, FBB with the intraclass correlation estimator using pooled observations across the groups, ${\widehat{C}}_{.}=\frac{{\displaystyle {\sum }_{i=1}^2{c}_i}{n}_i}{{\displaystyle {\sum }_{i=1}^2{n}_i}}$ and ${\varphi }^{BB}\left(n_i,\widehat{c}.\right)$  , instead of ${\varphi }^{BB}\left(n_i,\widehat{c}.\right)$ in equations (5, 6)ff.
3. LBB1p, FBB1p combine the two approaches by using the group wise variance estimators with the pooled intraclass correlation estimator and restriction of the over dispersion parameter to be$\ge 1$  , that is, using  $max(1,{\varphi }^{BB}\left(n_i,{\widehat{c}}_i\right))$ instead of ${\varphi }^{BB}\left(n_i,\widehat{c}.\right)$  in equations (5,6)ff. This procedure has been already suggested in the example evaluation of.²
4. LOD1, FOD1: refer to methods LOD, FOD, but the group wise variance estimators are restricted to be greater than or equal to the binomial variance estimate: using ${\widehat{v}}_i=\max \left(\frac{{\widehat{\pi }}_i(1-{\widehat{\pi }}_i)}{n_{i.}},\left(\frac{J_i}{J_i-1}\right){\displaystyle {\sum }_{j=1}^{J_i}\frac{{(x_{ij}-{\widehat{\pi }}_{i.}{n}_{ij})}^2}{n_i^2}}\right)$ in Eq.(2)

MOVER-R for quasibinomial profile deviance intervals (QBM): An alternative option to obtain intervals for the risk ratio would be to fit a generalized linear model (GLM) under the quasibinomial assumption $V_{QB}\left(X_{ij}\right)=\varphi n_{ij}{\pi }_{ij}\left(1-{\pi }_{ij}\right)$  , using a log-link, $n_{ij}=\log \left({\pi }_{ij}\right),n_{ij}={\beta }_i$ . Then the risk ratio can be estimated via $\exp ({\beta }_1-{\beta }_0)$ and intervals for the difference $({\beta }_1-{\beta }_0)$  can be computed by the signed root profile deviance method.^6,5 However, with the current user-level implementations in R, fitting this model (glm, stats) and obtaining profile-deviance intervals (profile, confint, package MASS) suffers from numerical difficulties if at least one of the groups shows estimated success probabilities close or equal to 1 or equal to 0.

As a numerically stable work-around, the QBM method is proposed: A GLM with the quasibinomial assumption and logit link is fitted $n_{ij}=\log ({\pi }_{ij}/(1-{\pi }_{ij}))={\beta }_i$  . If the estimated dispersion parameter in the model fit falls below$1,{\widehat{\varphi }}^{(QB)}<1$ , a binomial model is used instead (i.e., assuming $\varphi =1$  ). For the βi, (1-α)--signed root profile deviance intervals can be computed, with limits denoted $\left[l_{\widehat{\beta }0},u_{\widehat{\beta }0}\right],\left[l_{\widehat{\beta }1},u_{\widehat{\beta }1}\right]$  and estimates denoted ${\widehat{\beta }}_i$  . In R, these computations can be done in several packages, e.g. package MASS,⁵ or the add-on package mcprofile.¹³ Again, in extreme cases, the automatic search of values for the grid of parameter values for the deviance profile may fail in both packages. The signed root deviance is then computed over a pre-specified grid of parameter values,              

${\beta }_i^{*}=\left(-10,-9.5,\mathrm{...},-5,-4.8,-4.6,\mathrm{....},4.8,5,5.5,\mathrm{....},10\right)$

with elements ${\beta }_{ik}^{*}$  , k = 1,…, K. For each i and each k,

$t_{ik}=sign({\beta }_{ik}^{*}-{\widehat{\beta }}_i)\sqrt{\frac{d({\beta }_{ik}^{*},{\widehat{\beta }}_{i\text{'}})-d({\beta }_i^{*},{\widehat{\beta }}_{i\text{'}})}{\widehat{\varphi }}}$

is computed, where $sign({\beta }_{ik}^{*}-{\widehat{\beta }}_i)$  retains the sign of the difference${\beta }_{ik}^{*}-{\widehat{\beta }}_i$ , $d({\beta }_{ik}^{*}-{\widehat{\beta }}_i)$  is the deviance when replacing ${\widehat{\beta }}_i$  by ${\beta }_{ik}^{*}$  while leaving all other parameters at their ML estimates, ${\widehat{\beta }}_{i\text{'}}$ ,$d({\beta }_{ik}^{*}-{\widehat{\beta }}_i)$ is the deviance at the ML estimates, and $\widehat{\varphi }$  is the dispersion estimate with all parameters at their ML estimates. For each parameter i, a cubic spline is fitted for tik depending on${\beta }_{ik}^{*}$  and the cut points of the spline with quantiles of the t-distribution, $t_{\alpha /2,df=dfr,}{t}_{1-\alpha /2,df=dfr},dfr={\displaystyle {\sum }_{i=1}^I\left(J_i-1\right)}$ is determined by linear interpolation between fitted values. When the binomial model is used, tik is replaced by

$Z_{ik}=sign\left(({\beta }_{ik}^{*}-{\widehat{\beta }}_i)\right)\sqrt{d({\beta }_{ik}^{*},{\widehat{\beta }}_{i\text{'}})-d({\beta }_i^{*},{\widehat{\beta }}_{i\text{'}})}$

 And the quantiles of the standard normal distribution$Z_{\alpha /2},Z_{1-\alpha /2}$  , are used instead.

The interval bounds and ML estimates are transformed to the proportion scale using the inverse link,$\left[l_i-u_i\right]=\left[\frac{\exp \left(l_{\widehat{\beta }i}\right)}{1+\exp \left(l_{\widehat{\beta }i}\right)},\frac{\exp \left(u_{\widehat{\beta }i}\right)}{1+\exp \left(u_{\widehat{\beta }i}\right)}\right],{\widehat{\pi }}_i=\exp \left({\widehat{\beta }}_i\right)/\left(1+\exp \left({\widehat{\beta }}_i\right)\right)$ . These estimators and confidence limits are then used to compute intervals for ρ by the MOVER-R method.⁷ Eq. (9) of⁷ is recalled in the following as:

$\left[\frac{{\widehat{\pi }}_1{\widehat{\pi }}_0-\sqrt{{\left({\widehat{\pi }}_1{\widehat{\pi }}_0\right)}^2-l_1{u}_0\left(2\widehat{\pi }-l_1\right)}\left(2{\widehat{\pi }}_0-u_0\right)}{u_0\left(2{\widehat{\pi }}_0-u_0\right)};\frac{{\widehat{\pi }}_1{\widehat{\pi }}_0-\sqrt{{\left({\widehat{\pi }}_1{\widehat{\pi }}_0\right)}^2-u_1{l}_0\left(2\widehat{\pi }-u_1\right)}\left(2{\widehat{\pi }}_0-l_0\right)}{l_0\left(2{\widehat{\pi }}_0-l_0\right)}\right]$

Like the FOD and the FBB method, this method may yield (partially) unbounded intervals, particularly when$\widehat{\pi }=0$ , the upper limit for the risk ratio is naturally$\infty$ .

Simulation study
The beta-binomial distribution is chosen to simulate overdispersed data such that the resulting data are in line with the quasibinomial assumption, i.e., the assumption of the QBM method is met.

In the simulation, the number of experimental units per treatment group, Ji is chosen balanced, (J0, J1) = (3, 3), (5, 5) and (10, 10). The number of biological units under risk in each unit, nij , is chosen balanced nij = 10 or 20 for all i, j. Overdispersion is set at levels ${\varphi }^{BB}=1.25$  or ${\varphi }^{BB}=2$  , that is, for given nij , $a_1^{*}$ is chosen according to Section 4.2 to achieve the specified overdispersion: for each set of${\pi }_0,{\pi }_1$  , $a_i=a_i^{*}{\pi }_i$  and $b_i=a_i^{*}\left(1-{\pi }_i\right)$  in the beta distribution. The distribution of the counts xij, as well as that of the estimators of the proportions are skewed to different extent, depending on πi, especially if πi is close to the border of the parameter space. Thus, also the distribution of the estimator of ρ or log (ρ) can be skewed if π1 and π0 differ, particularly when they are close to the parameter space. Thus, the performance of large sample methods may severely depend on particular choices of π0, π1. To investigate these potential dependencies, the simulations have been run for a grid of all combinations of π0 = (0.02, 0.04, …, 0.96, 0.98) and π1 = (0.02, 0.04, …, 0.96, 0.98) that imply odds-ratio between 0.1 and 10.

Simulations have been performed with 10000 runs for all methods except the QBM method. Due to high computation times, only 5000 simulation runs are used for QBM, such that the standard error of the estimated coverage probabilities for QBM is by factor  higher than for the remaining methods. The simulation study has been performed in R 3.1.2,¹⁴ the implementation of all confidence interval methods is available in the R-package pairwiseCI, version > 0.1-25¹⁵ FBB, LBB and related methods are implemented in the function Betabin.ratio, FOD, LOD and related methods are implemented in the function ODbin.ratio, and the QBM method is implemented in function Quasibin.ratio.

Figure 1 Boxplots of simulated coverage probabilities of nominal two-sided 95% intervals, for different numbers of replications (no.rep) and two different levels of overdispersion.

Figure 2 Simulated coverage probabilities (color scale) of nominal two-sided 95% intervals over a grid of true proportions π0, π1, for two-fold overdispersion (ϕ = 2) and nij = 10 biological units in each experimental unit.

Figure 3 Simulated coverage probabilities (color scale) of nominal two-sided 95% intervals over a grid of true proportions π0, π1, for moderate overdispersion (ϕ = 1.25) and nij = 20 biological units in each experimental unit.

Figure 4 Asymmetry of non-coverage: color scale shows the proportion of cases where the true parameter is excluded by the lower limit all cases where the parameter is excluded by nominal two-sided 95% intervals over a grid of true prop proportions π0, π1, for clear overdispersion (ϕ = 2) and nij = 10 biological units in each experimental unit.

Figure 5 Boxplots and observed proportions of surviving fathead minnow larvae per tank, for the untreated control group and 5 concentrations (left), and 95% confidence intervals of the proportions of surviving larvae in the treatment groups relative to the control group (right)

Coverage probability of two-sided 95% confidence intervals
Figure 1 shows the simulated coverage probabilities of all 13 methods under comparison: the NF method can either show too low or too high coverage probability, irrespective of increasing sample size, because the relation of mean and variance in the binomial data is ignored. However, its average coverage probability is closer to the nominal level than some of those methods which explicitly account for over dispersed binomial data (LBB, FBB, LBBp, FBBp, LOD and FOD). Simply ignoring the possibility of overdispersion and assuming the binomial distribution results in too low coverage probability even for moderate (1.25-fold) overdispersion: BM method has too low coverage probabilities in nearly all settings. For very small numbers of replications (Ji=3, 5), the far too low coverage probability of the asymptotic methods for overdispersed binomial data (LBB, FBB, LOD, FBB) can be improved slightly by using a pooled variance estimator (LBBp, FBBp) and can be largely improved by setting a lower limit to their variance estimators: If we replace variance estimates suggesting under dispersion by the corresponding estimates under the binomial assumptions, the coverage probabilities of these methods are much closer to the nominal level. The QBM method is always very close to nominal coverage probability but can have slightly too high average coverage probability when overdispersion is moderate and the number of replications is small.

Figures 2 and Figure 3 shows a detailed view on the coverage probabilities in dependence on the true underlying proportions, π0 and π1. This detailed view is restricted to those 6 methods which have an average coverage probability close to 0.95. Figure 2 shows the more difficult case with substantial overdispersion (ϕ = 2), and only nij = 10 biological units in each replication. The QBM methods is close to the nominal confidence levels for a wide range of proportions, but can have too low coverage probabilities if at least one of the proportions is close to 0 or 1. The LBB1p and LOD1 methods need more replications to have coverage probabilities close to 0.95 for a similar range of πi, and still are slightly liberal for almost all πi. LBB1p and LOD1 booth have too high coverage probability for πi close to 0 and too low coverage probabilities when πi close to 1. That is test decisions based on these two methods may be conservative if hypotheses are formulated in terms of mortalities which should be low in the control group, but will be liberal when a similar hypothesis is formulated in terms of the proportion of survivors. In this simulation setting, the Fieller-Bailey-type intervals FBB1p and FOD1 have lower coverage probabilities for almost all parameter combinations considered as compared to the LBB1p and LOD1 method, respectively.

Figure 3 shows results for the less problematic case of moderate over dispersion (ϕ = 1.25) and nij = 20 biological units in each experimental unit. The coverage probability of QBM rarely falls below 0.94, but is slightly too large (between 0.96 and 0.97) if there are only 3 or 5 experimental units per treatment. The LBB1p and FOD1 method have again slightly too low coverage probability if any πi is close to 1, and slightly too high coverage probability if any πi is close to 0. The two Fieller-Bailey-type intervals have slightly lower coverage probabilities than their counterparts based on the log-delta-method.

Symmetry of non-coverage
Figure 4 shows the simulated proportion of cases, where the true parameter was excluded by the lower bound, relative to all cases where the parameter was excluded by the interval. For valid one-sided decisions, methods are preferable that exclude the true parameter equally likely by the lower and upper bound, i.e., with probability α/2 for each limit. For brevity, only the challenging setting with nij = 10 and marked over dispersion (ϕ = 2) is shown, while conclusions for the remaining simulation settings are similar: The Fieller-Bailey-type methods (FBB1p and FOD1) show a wider range of parameter settings where probability of parameter exclusion is equal between lower and upper bounds, as compared to the corresponding methods based on the delta methods on the log scale, LBB1p and LOD1. The QBM method shows asymmetric non-coverage for similar parameter settings as do the Fieller-Bailey-type intervals, i.e. when any of the true proportions is close to 0, but is clearly more symmetric than the FBB1p and FOD1 for a wide range of parameter settings where at least one πi is close to 1.

Examples
Extreme cases: Table 1 shows four extreme cases: Cases 1 and 2 represent cases where proportions are close to 0 in the control treatment as could result from testing the ratio of mortality or immobility proportions. Cases 3 and 4 show data that could arise from test systems that assume proportions close to 1 in the control, for example, survival proportions as in the fathead minnow data below.

In cases 1 and 3, data lead to variance estimates exceeding that of the binomial distribution. Then, the BM method leads to shorter intervals than all other methods. In both cases, the QBM as at least slightly wider confidence intervals than the FBB1p and FOD1 method, which might correspond to the observation that these two have too low coverage probability for small samples. In cases 2 and 4, data show a variance below that of the binomial variance. Then the methods without restriction of variance estimates to that assumed by the binomial distribution (FOD and FBB) yield considerably shorter intervals, than methods which assume that under dispersion is implausible like BM, FBB1p, FOD1 and QBM.

Fathead minnow data: The toxicity of a compound to fathead minnow larvae was investigated using an untreated control group and 5 concentrations of a compound.¹⁶ The experiment comprised 24 tanks, 4 tanks in each treatment group, each tank contained 10 larvae. The observed proportions of surviving larvae are shown in (Figure 5, left side). Analyzing the data in a generalized linear model with quasibinomial assumption, logit link shows that there are highly significant differences between the mean proportion of surviving larvae between the treatments (p < 0.0001; F-test in analysis of deviance). An estimated dispersion parameter of 1.082 suggests that the observations are at most slightly overdispersed, i.e., the data are at least roughly in line with the binomial assumption. The right side of Figure 5 shows two-sided 95% confidence intervals for ratios of the proportion of surviving larvae in the treatment groups relative to that in concentration 0. In this case, confidence limits based on the quasibinomial assumption (QBM) based on the full data including all 6 treatment groups and confidence intervals under the binomial assumption (MOVER-R method for Wilson-Score intervals, BM) do hardly differ.

Asymptotic methods based on the delta method applied on the log-scale or Fieller-Bailey type intervals have too low coverage probabilities when applied in small sample settings that are typical for many bioassays, i.e., they cover the true ratio of proportions less often than claimed by their nominal confidence level. Violation of the nominal level is most severe for small numbers of replications, low number of biological units in each replications and extreme proportions. Even for as much as 10 replications (i.e., clusters), coverage probabilities are considerably below the nominal level for wide ranges of proportions. When these intervals are then used for decisions in hypothesis tests for equivalence or non-inferiority, erroneous conclusions of equivalence or non-inferiority will occur more often than claimed by the nominal level α of such tests.

For the small number of replications and the small number of biological units per replication that are typical for some bioassays, restricting the variance estimates to that of the binomial variance (i.e., setting the dispersion parameter to 1 if under dispersion is estimated) leads to major improvements of the coverage probabilities. Further improvements can be achieved by combining the MOVER-R method with a profile deviance approach leads to intervals with better coverage probabilities for low sample sizes. However, this approach also shows too low coverage probabilities for small sample sizes and cases where one proportion is very close to 0 or 1. However, the simulation results shown here rely on the simplifying assumption, that there is a common overdispersion factor for the treatments in the experiments. Based on single data sets with few replications per treatment it will be hard to assess whether this assumption is appropriate, or whether different overdispersion factors per treatment group (as are used in the FBB, LBB method, for example) would be more appropriate. For given, highly standardized bioassay, however, available collections of historical data sets could be used to assess the plausibility of the different assumptions concerning distribution and mean-variance dependency and homogeneity or heterogeneity of overdispersion factors among treatment groups.

Bioassays usually involve several dosages. Depending on the global hypotheses to be tested, adjustments for multiple comparisons may be needed, see, e.g.¹⁷ The methods for confidence intervals discussed here can be extended to construct approximate simultaneous confidence intervals, using approaches as described in.^17-20 However, such extensions require additional investigation as some approaches involve additional approximations. This is subject to further research.