We study the influence of explanatory variables in prediction by looking at the distribution of the log-odds ratio. We also consider the predictive influence of a subset of unobserved future variables on the distribution of log-odds ratio as well as in a logistic model, via the Bayesian predictive density of a future observation. This problem is considered for dichotomous, as well as continuous explanatory variables.

AMS subject classification: Primary 62J12, Secondary 62B10, 62F15

Keywords: predictive density/probability, log-odds ratio, logistic model, predictive influence, missing/unobserved variable, kullback-leibler divergence

Odds ratio (OR) is perhaps the most popular measure of treatment difference for binary outcomes and is extensively used in dealing with 2×2 tables in biomedical studies and clinical trials. The distribution of the log of sample OR is often approximated by a normal distribution with true log OR as the mean and with variance estimated by the sum of the reciprocal of the four cell frequencies in the 2×2 table Breslow.¹ Böhning et al.² provide detailed book-length discussion on the OR. For logistic regression, ORs enable one to examine the effect of explanatory variables in that relationship.

Logistic link is perhaps the most popular way to model the success probabilities of a binary variable. Pregibon,³ Cook and Weisberg⁴ and Johnson⁵ have considered the problem of the influence of observations for logistic regression models. Several measures have been suggested to identify observations in the data set which are influential relative to the estimation of the vector of regression coefficients, the deviance, the determination of predictive probabilities and the classification of future observations.

Bhattacharjee & Dunsmore⁶ considered the effect on the predictive probability of a future observation of the omission of subsets of the explanatory variables. Mercier et al.⁷ used logistic regression to determine whether age and/or gender were a factor influencing severity of injuries suffered in head-on automobile collisions on rural highways. Zellner et al.⁸ considered the problem of variable selection in logistic regression to compare the performance of stepwise selection procedures with a bagging method.

In the present paper, our aim is to measure the predictive influence of a subset of explanatory variables in log-odds ratio of a logistic model using a Bayesian approach. We are also interested in studying the effect of missing future explanatory variables on Bayes prediction, on a logistic model as well as on the log-odds ratio.

In Section 2, we derive the predictive densities of a future log-odds ratio for both the full model and a subset deleted model. We derive the predictive density of log-odds ratio in Section 3, when a subset of future explanatory variables is missing. To derive the predictive densities we assume that the future explanatory variables $x^f$ are distributed as multivariate normal, both when these x^f's are independent or dependent. In Section 4, we discuss the influence of future missing explanatory variables by considering the predictive probability of a future response in a logistic model. This is done by assuming that the future explanatory variables $x^f$ are multivariate normal for the continuous case. Also considered is the dichotomous case. Since the predictive probabilities are not mathematically tractable for the logistic model, we use several approximations.

In Section 2 and 3 we employ Kullback-Leibler⁹ directed measure of divergence D_KL to assess the influence of variables and also the influence of future missing variables on the log-odds ratio. The form of the Kullback-Leibler⁹ measure used here is given by

$D_{KL}=\int f\left(a\text{'}{W}^f|.\right)\log \left(\frac{f\left(a\text{'}{W}^f|.\right)}{f_{{}_{\left(r+s\right)}\left(a\text{'}{W}^f|.\right)}}\right)d\left(a\text{'}{W}^f\right).$

To assess the influence of missing future variables or to measure the predictive probability in a logistic model we use the absolute difference of the two predictive probabilities.

Consider a phase III clinical trial with two competing treatments, say A and B, having binary responses. Suppose $n$ patients are randomly allocated with $n_A$ and $n_B$  patients to treatments A and B respectively. The patient responses are influenced by a covariate vector $x^{p\times 1}$ where one component of $x$ may be 1 (which covers the constant term). Let ( $Y_i$ ; $Z_i$ ; $x_i$ ) be the data corresponding to its patient, where Yi is the indicator of response ( $Y_i$ =1 or 0 for a success or failure), $z_i$  is the indicator of the treatment assignment ( $z_i=1$ )

or 0 according as treatment A or B is applied to the its patient), and $x$ is the covariate vector. We assume a logit model for the responses:

$\mathrm{Pr}\left(Y_i=1|Z_i,x_i\right)=\frac{\exp \left(\Delta Z_i+x_i\beta \right)}{1+\exp \left(\Delta Z_i+x_i\beta \right)}$ $i=1,2,\mathrm{....},n.$   (i)

Then the odds for treatments A and B with covariate vector x_i are respectively

$O_A=\frac{\mathrm{Pr}\left(Y_i=1|Z_i=1,x_i\right)}{\mathrm{Pr}\left(Y_i=0|Z_i=1,x_i\right)}=\exp \left(\Delta +x_i\beta \right)$ , $O_B=\frac{\mathrm{Pr}\left(Y_i=1|Z_i=0,x_i\right)}{\mathrm{Pr}\left(Y_i=0|Z_i=0,x_i\right)}=\exp \left(x_i\beta \right)$

and hence the log-odds ratio is

$\log OR=\frac{\log O_A}{\log O_B}=\Delta$

Let us partition

$x\beta =x_A{\beta }_A+x_B{\beta }_B+x_{AB}{\beta }_{AB}$

Where $x_A$  indicates the variables used in treatment A only, $x_B$  is for treatment B only, and ${{\displaystyle x}}_{AB}$ is for both treatments A and B. Then the model can be partitioned for treatments A and B as:

$\log O_A=u=\Delta +x_A{\beta }_A+x_{AB}{\beta }_{AB}=x_{\left(A\right)}{\beta }_{\left(A\right)}$ (ii)

$\log O_B=v=x_A{\beta }_B+x_{AB}{\beta }_{AB}=x_{\left(B\right)}{\beta }_{\left(B\right)}$ (iii)

The predictive density of future log-odds for A, $u^f$  , for non-informative prior (vague prior) with normal or any spherical symmetric errors is of Student form Jammalamadaka et al.¹⁰ and is given by

$f\left(u^f|x_{\left(A\right)}^f,data\right)\equiv St\left(n-k,x_{\left(A\right)}^f{\widehat{\beta }}_{\left(A\right)},s_{\left(A\right)}^2\left(1+x_{\left(A\right)}^{f\text{'}}{\left(x{\text{'}}_{\left(A\right)}{x}_A\right)}^{-1}{x}_{\left(A\right)}^f\right)\right)$

where ${\widehat{\beta }}_{\left(A\right)}$ is the MLE of ${\beta }_{\left(A\right)}$ , $s_{\left(A\right)}^2$  is the MLE of ${\sigma }_A^2$ and k is the number of parameters in the model (ii). See Bhattacharjee et al.¹¹ in this context. If the sample size is large then this predictive density can be well approximated by its asymptotic normal form

$N\left(x_{\left(A\right)}^f\widehat{\beta }\left(A\right), s_{\left(A\right)}^2\left(1+x_{\left(A\right)}^{f\text{'}}{\left(x_{\left(A\right)}^{\text{'}}{x}_{\left(A\right)}\right)}^{-1}{x}_{\left(A\right)}^f\right)\left(n-k\right)/\left(n-k-2\right)\right).$

Similarly one can find the same for treatment B, $v^f$ .

Let us define $w^f={\left(u^f,v^f\right)}^{\text{'}}$ and $a={\left(1,-1\right)}^{\text{'}}$ . Then the predictive density of future log odds ratio $a^{\text{'}}{w}^f$ is given by

$f\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^f,x_{\left(B\right)}^f,data\right)\approx N\left(\theta ,{\delta }^2\right)$   (iv)

Where

$\theta =x_{\left(A\right)}^f{\widehat{\beta }}_{\left(A\right)}-x_{\left(B\right)}^f{\widehat{\beta }}_{\left(B\right)}$

and

${\delta }^2=s_{\left(A\right)}^2\left(1+x_{\left(A\right)}^{f\text{'}}{\left(x_{\left(A\right)}^{\text{'}}{x}_{\left(A\right)}\right)}^{-1}{x}_{\left(A\right)}^f\right)\left(n-k\right)/\left(n-k-2\right)+s_{\left(B\right)}^2\left(\left(1+x_{\left(B\right)}^{f\text{'}}{\left(x_{\left(B\right)}^{\text{'}}{x}_{\left(B\right)}\right)}^{-1}{x}_{\left(B\right)}^f\right)\left(n-q\right)/\left(n-q-2\right)\right)$

Our interest is to measure the influence of explanatory variables in the predictive density (iv) for the following cases:

Case 1: Influence of $r$ explanatory variables $x_A^r$  of $x_A$  in treatment A.

Case 2: Influence of $r$  explanatory variables $x_B^r$  of $x_B$  in treatment B.

Case 3: Influence of $s$  explanatory variables $x_{AB}^s$  of ${{\displaystyle x}}_{AB}$ in treatment A.

Case 4: Influence of S explanatory variables $x_{AB}^s$ of ${{\displaystyle x}}_{AB}$ in treatment B.

Case 5: Joint influence of $r$  explanatory variables $x_A^r$ of $x_A$  and s explanatory variables $x_{AB}^s$ of ${{\displaystyle x}}_{AB}$ in treatment A.

Case 6: Joint influence of r explanatory variables $x_B^r$  of $x_B$  and s explanatory variables $x_{AB}^s$ of ${{\displaystyle x}}_{AB}$ in treatment B.

To see the influence of explanatory variables in log-odds ratio, we construct a reduced log-odds model deleting a subset of explanatory variables. Then we derive the predictive density of future log-odds ratio for reduced model and compare it with the predictive density (iv) for full model. It is enough to consider Case 5 for illustration. We construct the reduced model by deleting variables $x_A^r$ of $x_A$  and $x_{AB}^s$  of ${{\displaystyle x}}_{AB}$ in (ii) as

$u=\Delta +x_A^{*}{\beta }_A^{*}+x_A^{*}{}_B{\beta }_{AB}^{*}=x_{\left(A\right)}^{*}{\beta }_{\left(A\right)}^{*}$

Then the predictive density of $u^f$ is given by

$f\left(u^f|x_{\left(A\right)}^{*f},data\right)=St\left(n-k+r+s,x_{\left(A\right)}^{*f}{\widehat{\beta }}_{\left(A\right)}^{*}, S_{\left(A\right)}^{*2}\left(1+x_{\left(A\right)}^{*f\text{'}}{\left(x_{\left(A\right)}^{*\text{'}}{x}_{\left(A\right)}^{*}\right)}^{-1} x_{\left(A\right)}^{*f}\right)\right)$

The normal approximation of the predictive density is

$N(x_{\left(A\right)}^{*f}{\widehat{\beta }}_{\left(A\right)}^{*},s_{\left(A\right)}^{*2}\left(1+x_{\left(A\right)}^{*f\text{'}}{\left(x_{\left(A\right)}^{*\text{'}}{x}_{\left(A\right)}^{*}\right)}^{-1}{x}_{\left(A\right)}^{*f})\left(n-k+r+s\right)/\left(n-k+r+s-2\right)\right)$

Since no variable is missing in $\upsilon =\log O_B$ , the predictive density of ${\upsilon }^f$ is unaltered along with its normal approximation. Hence the predictive density of log-odds ratio $a^{\text{'}}{w}^f$  under Case 5 is given by

$f_{\left(r+s\right)}\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^{*f},x_{\left(B\right)}^f,data\right)\approx N\left({\theta }^{*},{\delta }^{*2}\right)$   (v)

Where

${\theta }^{*}=x_{(A)}^{*f}\widehat{\beta }{*}_{(A)}-x_{\left(B\right)}^f{\widehat{\beta }}_{\left(B\right)}$

and

$\begin{array}{l}{\delta }^{*2}=s_{\left(A\right)}^{*2}\left(1+x_{\left(A\right)}^{*f\text{'}}{\left(x_{\left(A\right)}^{*\text{'}}{x}_{\left(A\right)}^{*}\right)}^{-1}{x}_{\left(A\right)}^{*f}\right)\left(n-k+r+s\right)/\left(n-k+r+s-2\right)\\ +s_{\left(B\right)}^2\left(1+x_{\left(B\right)}^{f\text{'}}{\left(x_{\left(B\right)}^{\text{'}}{x}_{\left(B\right)}\right)}^{-1}{x}_{\left(B\right)}^f\right)\left(n-q\right)/\left(n-q-2\right)\end{array}$

To access the influence of the deleted variables we employ the Kullback-Leibler⁹ directed measure of divergence ${\text{D}}_{KL}$ between the predictive densities of $a^{\text{'}}{w}^f$ for full model (iv) and reduced model (v). The form of K-L measure used here is given by

${\text{D}}_{\text{KL}}={\displaystyle \int f_{\left(r+s\right)}}\left(a\text{'}{\omega }^f|.\right)\log \left(\frac{f_{\left(r+s\right)}\left(a\text{'}{w}^f|.\right)}{f\left(a\text{'}{w}^f|.\right)}\right)d{a}^{\text{'}}{\omega }^f$

The discrepancy measure ${\text{D}}_{KL}$ between the predictive densities (iv) and (v) reduces to

$D_{KL}=\frac{{\left(\theta -\theta *\right)}^2}{2{\delta }^2}+\frac{1}{2}\left(\frac{{\delta }^{*2}}{{\delta }^2}-\log \left(\frac{{\delta }^{*2}}{{\delta }^2}\right)-1\right)$

Here $L=\frac{{\left(\theta -{\theta }^{*}\right)}^2}{2{\delta }^2}$ is due to difference of location parameters and $S=\frac{1}{2}\left(\frac{{\delta }^{*2}}{{\delta }^2}-\log \left(\frac{{\delta }^{*2}}{{\delta }^2}\right)-1\right)$ due to difference of scale parameters of the two predictive densities (iv) and (v).

Example 1: Here we have considered a flu shot Data Pregibon.³ A local health clinic sent fliers to its clients to encourage everyone, but especially older persons at high risk of complications, to get a flu shot for protection against an expected flu epidemic. In a pilot follow-up study, 159 clients were randomly selected and asked whether they actually received a flu shot. A client who received a flu shot was coded Y=1; and a client who did not receive a flu shot was coded Y=0. In addition, data were collected on their age $\left(x_1\right)$ and their health awareness $\left(x_2\right)$ . Also included in the data were client gender $\left(x_3\right)$ , with males coded $x_3=1$ and females coded $x_3=0$ . Here we have divided whole data set into two groups A and B on the basis of gender that is group A corresponds to the male and group B corresponds to the female. We have computed $D_{KL}$ to measure the influence of the deleted variable $x_1$  in group A and B separately and the discrepancies are drawn in Figure 1.

1. Similar figure can be obtained by deleting $x_2$ . From this figure the discrepancy is less around the mean of the deleted variable.

Example 2: This is a simulation exercise. Here we have drawn sample of size 159 from bivariate normal distribution and we have used means, variances and correlation coefficient of $x_1$ and $x_2$ of the above flu shot data of size 159 for generating the sample. Now using these $x_1$ and $x_2$ , we got response that is Y values and thereafter using this whole generated data set we have computed $D_{KL}$ . Now we have repeated whole process 1000 times and computed means of $D_{KL}s$ . The mean discrepancies are shown in Figure 2. Here we get the same conclusion as in the data example.

Here the aim is to detect the predictive influence of a set of missing future explanatory variables in log-odds ratio of logistic model (i). Our interest is to detect the influence of missing future explanatory variables in the six cases pointed out in Section 2. Let in treatment A, r future variables missing from $x_A^f$ and s future variables missing from $x_{AB}^f$ be denoted by $x_{\left(A\right)}^{\left(r+s\right)f}$ . Similarly in treatment B, r future missing variables from $x_B^f$ and s future variables missing from $x_{AB}^f$ be denoted by $x_{\left(B\right)}^{\left(r+s\right)f}$ . We assume that the errors of models (ii) and (iii) are normally distributed with zero means and variances ${\tau }_{\left(A\right)}^{-1}$ and ${\tau }_{\left(B\right)}^{-1}$ , respectively. We also assume that the conditional density of $x^{*}{}_{(r)}^f$ given $x^{*}{}^f$ is independent of ${\beta }_{\left(A\right)}$ and ${\tau }_{\left(A\right)}$ and $x_{\left(B\right)}^{\left(r+|s\right)f}$  given $x_{\left(B\right)}^{*f}$ is independent of ${\beta }_{\left(B\right)}$ and ${\tau }_{\left(B\right)}$ , i.e.,

$f\left(x_{(.)}^{\left(r+s\right)f}|x_{(.)}^{*f},{\beta }_{(.)},{\tau }_{(.)}\right)=f\left(x_{(.)}^{\left(r+s\right)f}|x_{(.)}^{*f}\right)$

where $x_{(.)}^{*f}$ denotes the future explanatory variables $x_{(.)}^f$ without $x_{(.)}^{\left(r+s\right)f}$ .

Explanatory variables are continuous

We assume that $x_i^{f,}$ s are dependent and the distribution of $x_{\left(A\right)}^f$ is $\left(k-1\right)$ -dimensional multivariate normal, i.e. $f\left(x_{\left(A\right)}^f\right)\equiv N_{k-1}\left(\eta , \psi \right)$ .

The conditional density of $x_{\left(A\right)}^{\left(r+s\right)f}$  given $x_{\left(A\right)}^{*f}$ is given by

$f\left(x_{\left(A\right)}^{\left(r+s\right)f}|x_{\left(A\right)}^{*f}\right)\equiv N_{r+s}\left({\eta }_{\left(r+s\right)}^{*},{\psi }_{\left(r+s\right)}^{*}\right)$ ,

Where

$\eta =\left({\eta }^{*}, {\eta }_{r+s}\right),x_{\left(A\right)}^f=\left(x_{\left(A\right)}^{*f}, x_{\left(A\right)}^{\left(r+s\right)f}\right), \psi =\left(\begin{array}{l}{\psi }_{11} {\psi }_{12}\\ {\psi }_{21} {\psi }_{22}\end{array}\right), {\eta }_{r+s}^{*}={\eta }_{r+s}+{\psi }_{21}{\psi }_{11}^{-1}\left(x_{\left(A\right)}^{*f}-{\eta }^{*}\right)$  

and  ${\psi }_{\left(r+s\right)}^{*}={\psi }_{22}-{\psi }_{21}{\psi }_{11}^{-1}{\psi }_{12}$ .

As earlier it is enough to consider Case 5 to see the joint influence of r missing future explanatory variables $x_A^{rf}$ of $x_A^f$ and s missing future explanatory variables $x_{AB}^{sf}$ of $x_{AB}^f$  in treatment A. The density of $u^f$ when $x_{\left(A\right)}^{\left(r+s\right)f}$ is missing is given by

$f\left(u^f|x_{\left(A\right)}^{*f},\beta {|}_{\left(A\right)},{\tau }_{\left(A\right)}\right)={\displaystyle \int f\left(u^f|x_{\left(A\right)}^f,{\beta }_{\left(A\right)},{\tau }_{\left(A\right)}\right)}f\left(x_{\left(A\right)}^{\left(r+s\right)f}|x_{\left(A\right)}^{*f}\right)d{x}_{\left(A\right)}^{\left(r+s\right)f}\equiv N\left({\displaystyle \sum _{i=0}^{k-r-s-1}{x}_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}+{\displaystyle \sum _{i=k-r-s}^{k-1}{\eta }_i^{*}{\widehat{\beta }}_{\left(A\right)i},{\displaystyle \sum _{i=k-r-s}^{k-1}{\widehat{\beta }}_{\left(A\right)i}}{\widehat{\beta }}_{\left(A\right)j}{\psi }_{ij}^{*}+{\tau }_{\left(A\right)}^{-1}}}\right)$

Where ${\eta }_i^{*}$ is the $i$ th component of ${\eta }_{\left(r+s\right)}^{*}$ and ${\psi }_{ij}^{*}$  is the ${\left(i.j\right)}^{th}$ component of ${\psi }_{\left(r+s\right)}^{*}$ .

See Bhattacharjee et al¹¹ in this context. Using Taylor's expansion and improper prior density for both ${\beta }_{\left(A\right)}$  and ${\tau }_{\left(A\right)}$ , the approximate predictive density of $u^f$ when $x_{\left(A\right)}^{\left(r+s\right)f}$ is missing is given by

$f_{\left(r+s\right)}\left(u^f|x_{\left(A\right)}^{*f}, data\right)\equiv N\left({\displaystyle \sum _{i=0}^{k-r-s-1}{x}_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}+{\displaystyle \sum _{i=k-r-s}^{k-1}{\eta }_i^{*}{\widehat{\beta }}_{\left(A\right)i},{\displaystyle \sum _{i,j=k-r-s}^{k-1}{\widehat{\beta }}_{\left(A\right)i}{\widehat{\beta }}_{\left(A\right)j}}{\psi }_{ij}^{*}+s_{\left(A\right)}^2{\gamma }^{*}}}\right),$

evaluated at  ${\widehat{\beta }}_{\left(A\right)}$ and $s_{\left(A\right)}^2$ where

${\gamma }^{*}=\left(1+\frac{1}{2}{\displaystyle \sum _0^{k-1}{Q}_{ij}^{*}\left({\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)Cov\left({\beta }_{\left(A\right)i},{\beta }_{\left(A\right)j}\right)+\frac{1}{2}}{Q}_{{\tau }_{\left(A\right)}}^2\left({\beta }_{\left(A\right)},{\tau }_{\left(A\right)}\right)Var\left({\tau }_{\left(A\right)}\right)\right)$

is the multiplicative factor for the second order Taylor's approximation. If $x_{\left(A\right)}^f\text{'}s$ ’s are independent the corresponding approximate predictive density of $u^f$ is

$f_{\left(r+s\right)}\left(u^f|x_{\left(A\right)}^{*f}, data\right)\equiv N\left({\displaystyle \sum _{i=0}^{k-r-s-1}{x}_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}+{\displaystyle \sum _{i=k-r-s}^{k-1}{\eta }_i{\widehat{\beta }}_{\left(A\right)i},{\displaystyle \sum _{i,j=k-r-s}^{k-1}{\widehat{\beta }}_{\left(A\right)i}^2}{\psi }_i^2+s_{\left(A\right)}^2\gamma }}\right)$

evaluated at ${\widehat{\beta }}_{\left(A\right)}$ and $s_{\left(A\right)}^2$ , where ${\eta }_i$ and ${\psi }_i^2$  are mean and variance of the ith missing variable and $\gamma =\left(1+\frac{1}{2}{\displaystyle \sum _0^{k-1}{Q}_{ij}\left({\beta }_{\left(A\right)},{\tau }_{\left(A\right)}\right)Cov\left({\beta }_{\left(A\right)i},{\beta }_{\left(A\right)j}\right)+\frac{1}{2}}{Q}_{{\tau }_{\left(A\right)}}^2\left({\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)Var\left({\tau }_{\left(A\right)}\right)\right)$ . Since no future variable is missing in $\upsilon$ , the approximate predictive density of ${\upsilon }^f$ is same as obtained in Section 2. Thus when $x_{\left(A\right)}^f$ ’s are dependent the approximate predictive density of log-odds ratio $a^{\text{'}}{w}^f$ for $x_{\left(A\right)}^{\left(r+s\right)f}$ missing is given by

$f_{\left(r+s\right)}\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^{*f},x_{\left(B\right)}^f; data\right)\equiv {\gamma }^{*}N\left(\xi ,{\omega }^2\right)$   (vi)

Where

$\xi ={\displaystyle \sum _{i=0}^{k-r-s-1}{x}_{\left(A\right)i}^f}{\widehat{\beta }}_{\left(A\right)i}+{\displaystyle \sum _{i=k-r-s}^{k-1}{\eta }_i^{*}}{\widehat{\beta }}_{\left(A\right)i}-x_{\left(B\right)}^f{\widehat{\beta }}_{\left(B\right)}$

and

${\omega }^2=\left({\displaystyle \sum _{i,j=k-r-s}^{k-1}{\widehat{\beta }}_{\left(A\right)i}}{\widehat{\beta }}_{\left(A\right)j}{\psi }_{ij}^{*}+s_{\left(A\right)}^2\right)+s_{\left(B\right)}^2\left(1+x_{\left(B\right)}^f{\left(X_{\left(B\right)}^{\text{'}}{X}_{\left(B\right)}\right)}^{-1}{x}_{\left(B\right)}^{f\text{'}}\right)\frac{n-q}{n-q+2}$

The Kullback-Leibler⁹ directed measure of divergence between the predictive densities (iv) when no variable is missing and the predictive density (vi) when $\begin{array}{l}{\text{D}}_{KL}={\displaystyle \int f\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^f,x_{\left(B\right)}^f, data\right)}\log \left(\frac{f\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^f,x_{\left(B\right)}^f, data\right)}{f_{\left(r+s\right)}\left(a^{\text{'}}{w}^f|x_{\left(A\right)}^{*f},x_{\left(B\right)}^f, data\right)}\right)d{a}^{\text{'}}{w}^f\\ =\frac{1}{2{\omega }^2}{\left(\theta -\xi \right)}^2+\frac{1}{2}\left(\frac{{\delta }^2}{{\omega }^2}-\log \left(\frac{{\delta }^2}{{\omega }^2}\right)-1\right)\end{array}$
$-\frac{1}{2}{\displaystyle \sum _{i,j=0}^{k-1}E\left(Q_{ij}^{*}\left({\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)Cov\left({\tau }_{\left(A\right)i}, {\tau }_{\left(A\right)j}\right)\right)}-\frac{1}{2}E\left(Q_{\tau \left(A\right)}^2\left({\beta }_{\left(A\right)},{\tau }_{\left(A\right)}\right)\mathrm{var}\left({\tau }_{\left(A\right)}\right)\right)$ (vii)

If $x_{\left(A\right)}^f$ ’s are independent the predictive density of $a^{\text{'}}{w}^f$  when $\left(r+s\right)$ future variables are missing is same as (vi) and the corresponding Kullback-Leibler⁹ measure $D_{KL}$ is same as (vii) but replacing ${\eta }_i^{*}$  by ${\eta }_i$ in $\xi$ , ${\widehat{\beta }}_{\left(A\right)i}{\widehat{\beta }}_{\left(A\right)j}{\psi }_{ij}^{*}$  by ${\widehat{\beta }}_{\left(A\right)i}^2{\psi }_i^2$ in ${\omega }^2$  and $Q_{ij}^{*}\left({\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)$  by  $Q_{ij}\left({\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)$  in ${\gamma }^{*}$ , where ${\eta }_i$ and ${\psi }_i^2$ are mean and variance of the ith missing variable.

Explanatory variables are dichotomous

Here we assume that all the explanatory variables are dichotomous and independent. We assume that the errors of models (ii) and (iii) are normally distributed with means zero and variances ${\tau }_{\left(A\right)}^{-1}$ and ${\tau }_{\left(B\right)}^{-1}$ respectively. To assess the influence of the missing variables in treatment A, we consider that $x_{\left(A\right)i}^f$ is distributed as

$\mathrm{Pr}\left(X_{\left(A\right)i}^f=x_{\left(A\right)i}^f\right)={\theta }_{\left(A\right)i}^{x_{\left(A\right)i}^f}{\left(1-{\theta }_{\left(A\right)i}\right)}^{1-x_{\left(A\right)i}^f},x_{\left(A\right)i}^f=0,1, i=1,2,\mathrm{...},k-1$

The density of a future $u^f$ is

$f\left(u^f|x_{\left(A\right)}^f,{\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}\right)\equiv N\left({\displaystyle \sum _{i=0}^{k-1}{x}_{\left(A\right)i}^f{\beta }_{\left(A\right)i}, {\tau }_{\left(A\right)}^{-1}}\right).$

If $x_{\left(A\right)}^{\left(r\right)f}$ future variables are missing in treatment A, then the density of a future $u^f$ is given by

$f\left(u^f|x_{\left(A\right)}^{*f},{\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}^{-1}\right)={\displaystyle \sum _{x_{\left(A\right)k-r}^f=0}^1\mathrm{.....}\underset{x_{\left(A\right)k-1}^f=0}{\overset{1}{{\displaystyle \Sigma }}}N\left({\displaystyle \sum _{i=0}^{k-1}{x}_{\left(A\right)i}^f{\beta }_{\left(A\right)i}, {\tau }_{\left(A\right)}^{-1}}\right){\displaystyle \prod _{i=k-r}^{k-1}{\theta }_{\left(A\right)i}^{x_{\left(A\right)i}^f}}}{\left(1-{\theta }_{\left(A\right)i}\right)}^{1-x_{\left(A\right)i}^f}.$

The predictive density of $u^f$ when $x_{\left(A\right)}^{\left(r\right)f}$ is missing is given by

$f\left(u^f|x_{\left(A\right)}^{*f}, data={\displaystyle \int f(u^f|x_{\left(A\right)}^{*f})}{\beta }_{\left(A\right)}, {\tau }_{\left(A\right)}^{-1}\right)f\left({\beta }_{\left(A\right)}|data\right)d{\beta }_{\left(A\right)}$    (viii)

which is not mathematically tractable. For vague prior densities for ${\beta }_{\left(A\right)}$ and ${\tau }_{\left(A\right)}$  and using Taylor's expansion, the approximate predictive density of (viii) is

$\begin{array}{l}f\left(u^f|x{*}_{\left(A\right)}^f, data\right)={\displaystyle \sum _{x_{\left(A\right)k-r}^f=0}^1\mathrm{...}{\displaystyle \sum _{x_{\left(A\right)k-1}^f=0}^{1}N\left({\displaystyle \sum _{i=0}^{k-1}{x}_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}},s_{\left(A\right)}^2\right)}}{\displaystyle \prod _{i=k-r}^{k-1}{\theta }_{\left(A\right)i}^{x_{\left(A\right)i}^f}}{\left(1-{\theta }_{\left(A\right)}i\right)}^{1-x_{\left(A\right)i}^f}\\ \left(1+{\displaystyle \sum _{i,j=0}^{k-1}{Q}_{ij}}\left(\widehat{\beta },s_{\left(A\right)}^{-2}\right)\frac{\mathrm{cov}\left({\beta }_{\left(A\right)i},{\beta }_{\left(A\right)j}\right)}{2}+Q_{{\tau }_{\left(A\right)}^2}\left({\widehat{\beta }}_{\left(A\right)},s_{\left(A\right)}^{-2}\right)\frac{\mathrm{var}\left({\tau }_{\left(A\right)}\right)}{2}\right)\end{array}$

Since there are no missing variables in ${\nu }^f$ , the density of ${\nu }^f$ is same as that can be obtained in Section 2. Then the predictive density of $a^{\text{'}}{w}^f$ is given by

$\begin{array}{l}f\left(a\text{'}{w}^f|x{*}_{\left(A\right)}^f,x_{\left(B\right)}^f, data\right)={\displaystyle \sum _{x_{\left(A\right)k-r}^f=0}^1\mathrm{...}{\displaystyle \sum _{x_{\left(A\right)k-1}^f=0}^{1}N}}\left({\displaystyle \sum _{i=0}^{k-1}\left(x_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}-x_{\left(B\right)i}^f{\widehat{\beta }}_{\left(B\right)i}\right),}{S}_{\left(A\right)}^2+s_{\left(B\right)}^2\left(1+x_{\left(B\right)}^f{\left(X{\text{'}}_{\left(B\right)}{X}_{\left(B\right)}\right)}^{-1}x{\text{'}}_{\left(B\right)}\right)\right)\\ {\displaystyle \prod _{i=k-r}^{k-1}{\theta }_{\left(A\right)i}^{x_{\left(A\right)i}^f}}{\left(1-{\theta }_{\left(A\right)}i\right)}^{1-x_{\left(A\right)i}^f}\\ \left(1+{\displaystyle \sum _{i,j=0}^{k-1}{Q}_{ij}}\left({\widehat{\beta }}_{\left(A\right)},s_{\left(A\right)}^{-2}\right)\frac{\mathrm{cov}\left({\beta }_{\left(A\right)i},{\beta }_{\left(A\right)j}\right)}{2}+Q_{T_{\left(A\right)}^2}\left({\widehat{\beta }}_{\left(A\right)},s_{\left(A\right)}^{-2}\right)\frac{\mathrm{var}\left({\tau }_{\left(A\right)}\right)}{2}\right)\end{array}$  (ix)

Analytical solution of ${\text{D}}_{KL}$ between the predictive densities (iv) and (ix) is very difficult to obtain but numerical solution can be obtained. In Some situations it is seen that among the explanatory variables, some of the variables are dichotomous and some of the variables are continuous. Among the $k-1$ -explanatory variables, without loss of generality we assume that the first $l$ are dichotomous and the remaining last $k-l-1$ are continuous variables. We also assume that out of l dichotomous future variables last d variables are missing and out of $(k-l-1)$ continuous future variables last g variables are missing. Then the predictive density of future log-odds ratio $a^{\text{'}}{w}^f$ when d dichotomous and g continuous variables are missing is given by

$\begin{array}{l}f\left(a^{\text{'}}{w}^f|x{*}_{\left(A\right)}^f,x{*}_{\left(B\right)}^f, data\right)=({\displaystyle \sum _{x_{\left(A\right)l-d+1}^f=0}^1\mathrm{...}{\displaystyle \sum _{x_{\left(A\right)l}^f=0}^{1}N({\displaystyle \sum _{i=0}^{k-g-1}{x}_{\left(A\right)i}^f{\widehat{\beta }}_{\left(A\right)i}}+}}\\ {\displaystyle \prod _{i=k-g}^{k-1}\eta i}{\widehat{\beta }}_{\left(A\right)i}-\underset{i=0}{\overset{k-1}{{\displaystyle \Sigma }}}{x}_{(B)i}^f{\widehat{\beta }}_{(B)i}, \underset{i=k-g}{\overset{k-1}{{\displaystyle \Sigma }}}{\widehat{\beta }}_{(A)i}^2{\Psi }_i^2+S_{(A)}^2+S_{(B)}^2(1+x_{(B)}^f{\left(X_{(B)}^{{}^{\text{'}}}{X}_{(B)}\right)}^{-1}{x}_{(B)}^{{}^{\text{'}}})). (x)\\ \underset{i=l-d+1}{\overset{l}{{\displaystyle \Pi }}}{\theta }_i^{x_{(A)i}^f}{\left(1-{\theta }_i\right)}^{1-x_{(A)i}^f})(1+{\displaystyle \sum _{i,j=0}^{k-1}{Q}_{ij}}\left({\widehat{\beta }}_{(A)},s_{\left(A\right)}^{-2}\right)\frac{\mathrm{cov}\left({\beta }_{\left(A\right)i},{\beta }_{\left(A\right)j}\right)}{2}\\ +Q_{T_{\left(A\right)}^2}\left({\widehat{\beta }}_{\left(A\right)},s_{\left(A\right)}^{-2}\right)\frac{\mathrm{var}\left(T_{\left(A\right)}\right)}{2})\end{array}$  (x)

Again, analytical solution of ${\text{D}}_{KL}$ between the predictive densities (iv) and (x) is very difficult but we can obtain its numerical solution. In similar way we can derive the predictive density of future log-odds ratio when some future variables are missing in treatment B.

Example 1 revisited: This example is based on the flu shot data of Example 1. From Figure 3 we have observed same as Examples 1 and 2 that the discrepancies are less around the mean of the missing variables. Moreover we have observed from Figures 1 and 3 that the discrepancies of the missing variables are less as compared to the discrepancies of the deleted variables.

Example 2 revisited: This example is based on the simulation data of Example 2 and here we have also got same conclusion as Example 1 revisited (Figures 2 & 4).

Group A Group B

Figure 1 Three dimensional scatter plots based on real data for D_KL

when x1 is deleted.

Group A Group B

Figure 2 Three dimensional scatter plots based on simulated data for D_KL

when x1 is deleted.

Group A Group B

Figure 3 Three dimensional scatter plots based on real data for D_KL

when x_f¹ is missing.

Group A Group B

Figure 4 Three dimensional scatter plots based on simulated data for D_KL

when x_f¹ is missing.

Examples 1 and 2 revisited: In this example, we have used ${\text{D}}_{KL}$ values for real data for drawing box plots for each cases (deleted and missing). From Figure 5, we have observed that x₂ is more in uential than x₁. Moreover the discrepancies are much less in missing case than deleted case. We have got same result in simulation study and are illustrated in Figure 6.

Treatment A Treatment B

Figure 5 Box plot for D_KL based on real data.

Treatment A Treatment B

Figure 6 Box plot for D_KL based on simulated data.

We consider the logistic model as

$\mathrm{Pr}\left(y=1|x,\beta \right)=\exp \left(x\beta \right)/\left(1+\exp \left(x\beta \right)\right)$

The probability that a future response yf will be a success is given by

$\mathrm{Pr}\left(y^f=1|x^f,\beta \right)=\exp \left(x^f\beta \right)/\left(1+\exp \left(x^f\beta \right)\right)$

We assume that the conditional density of xf(r) given xf is independent of, $\beta$ where xf denotes the future explanatory variables without variables xf(r). Then predictive probabilities of yf will be a success for models are given by

$\mathrm{Pr}\left(y^f=1|x^f, data\right)={\displaystyle \int \mathrm{Pr}\left(y^f=1|x^f,\beta \right)}f\left(\beta | data\right)d\beta$

and

$\mathrm{Pr}\left(y^f=1|x^{*}{}^f, data\right)={\displaystyle \int \mathrm{Pr}\left(y^f=1|x^{*f},\beta \right)}f\left(\beta | data\right)d\beta$ respectively. Simple analytically tractable priors are not available here. Numerical integration techniques might be used for some specified priors to approximate $\mathrm{Pr}\left(y^f=1|x^f,data\right)$  and $\mathrm{Pr}\left(y^f=1|x{*}^f,data\right)$ , respectively.

Normal approximation for the posterior density

Let us suppose that the sample size is large. Lindley¹² stated that the posterior density $f\left(\beta |data\right)$ may then be well approximated by its asymptotic normal form as

$f\left(\beta |data\right)\approx N_p\left(\widehat{\beta },\sum \right)$

where $\hat{\beta }$ is the maximum likelihood estimate of β, ∑ = (-H)-1 and H is the Hessian of log L(β) evaluated at .

For the logistic model (xi), the Hessian H=(hji( $\hat{\beta }$ )) evaluated at is given by

$h_{jl}\left(\widehat{\beta }\right)=-{\displaystyle \sum _{i=1}^n\frac{x_{ij}{x}_{il}\exp \left(x_i\widehat{\beta }\right)}{{\left(1+\exp \left(x_i\widehat{\beta }\right)\right)}^2}},j,l=0,1,\mathrm{...},k,$

Where x_ij is the jth component of $x_i$ with $x_{i0}$ = 1. For given $x^f,z=x^f\beta$ will have approximately a posteriori a normal distribution with mean $b{x}^f$ = $x^f\stackrel{\wedge }{\beta }$ and variance $d_{x^f}^2=x^f\Sigma x^{f\text{'}}$ , and with probability density function $\varphi \left(z|b_{x^f}, d_{{}^{x^f}}^2\right)$ . Using the transformation we can approximate $f\left(\beta |x^f, data\right)$ by

$\mathrm{Pr}\left(y^f=1|x^f, data\right)\approx {\displaystyle \int \frac{\exp \left(z\right)}{1+\exp \left(z\right)}}\varphi \left(z|b_{{}_{x^f},d_{x^f}^2}\right)dz.$

Analytical evaluation of (4.1) is very di cult. We can however evaluate then by numerical integration techniques viz Gauss-Hermite Quadrature Abramowitz and Stegun,¹³ Normal approximation Cox,¹⁴ Laplace's approximation de Bruijn.¹⁵

If the sample size is small, the posterior normality assumption may not be accurate. Therefore, we consider Flat prior approximation Tierney and Kadane¹⁶ as an alternative approach using the Laplace's method for integrals.

Effect of the variables $x^f$

Here we assume that the future variables $x^f$ are dependent and the density of $x^f$ is p-dimensional multivariate normal i.e.

$f\left(x^f\right)\equiv N_p\left(n, \psi \right)$

The conditional density of $x_{\left(r\right)}^f$  for given $x^{*f}$ is  

$f\left(x_{\left(r\right)}^f|x{*}^f\right)\equiv N_r\left(n{*}_{\left(r\right)}, \psi {*}_{\left(r\right)}\right)$

The probability of $y^f$ as a success when $x_{\left(r\right)}^f$ is missing given by

$\mathrm{Pr}\left(y^f=1|x^{*f}, \beta \right)={\displaystyle \int \frac{\exp \left(x^f\beta \right)}{1+\exp \left(x^f\beta \right)}f\left(x_{(r)}^f{|x}^{*f}\right)d{x}_{(r)}^f}$