Testing the significance of a medical treatment on experimental subjects is very common in medical data analysis. Classical methods like the traditional analysis of variance usually assume variance homogeneity across treatments or experimental groups of subjects. However, this assumption is often violated if there exists fundamental difference between experimental groups like male and female groups of patients. In this paper, we propose to use a theoretically proved exact -test for testing the significance of a medical treatment for subjects before and after the treatment. This new exact -test is applicable regardless of variance homogeneity across groups. An illustration based on real experimental data from treatments on rats shows that the new exact  -test gives more convincing results than those from the traditional analysis of variance.

Keywords: analysis of variance; -test; gene expression data; multiple mean comparison

The significance of responses from a gene before and after an experiment can be tested by statistical methods for multiple mean comparison. While the traditional two-sample Student’s -test has been employing for comparison between two normal population means, it is assumed that the two populations have equal variances. This condition may not be always satisfied if a group of genes are correlated. Although unequal-variance two-sample -test was proposed to handle the two-sample mean comparison, it only provides an approximate solution. A natural extension to the two-sample -test for a comparison between two means is the one-way (single-factor) analysis of variance (ANOVA), which is also based on the assumption of equal variances across the populations from which the means are compared. If the equal-variance assumption is not satisfied, the conclusion from one-way ANOVA is doubtable. For example, a study on a comparison between the mean responses from different genes of rats before and after an experiment is given by Gao et al.¹ The purpose of the study is to find out which gene has a significant change between the mean responses of each rat. In the experiment, 24 pregnant rats were randomly assigned to four groups (sample size  per group) and treated with corn oil (vehicle control), 2, 10 or 50 mg/kg DEHP (Alfa Aesar). The response data from the three experiments with different doses of corn oil were collected. Under each dose, the experimental purpose is to see which gene shows a significant change before and after the experiment. This is a kind of representative experiments in medical research for identifying the significant effect from a treatment. Under the classical equal-variance normal assumption, the solution is obvious the two-sample -test for comparing the different effects from two doses, and the one-way ANOVA is employed for comparing the different effects from three or more doses. When the classical equal-variance assumption is known to be violated, there are some approximate methods available for a multiple mean comparison.^2–7 In addition to comparing the mean difference between before and after treatment for each gene, there are many other methods for analysis of gene expression data in the literature.^8–14

In this paper, we will employ a new exact -distribution-based method for the multiple mean comparison for gene-experiment data under the normal assumption on the sample data. The new exact -test does not depend on the equal-variance assumption across groups. This implies that the new -test will be especially suitable for the multiple mean comparison with known variance heterogeneity across groups. We give a summary review on the basic statistical theory for the new -test in Section 4. Details on the new -test can be referred to Liang et al.¹⁵ Section 5 presents the application of the new -test to significant gene identification compared to the classical one-way ANOVA method. Some concluding remarks are given in the last section.

Assume that there is a balanced sample design (with an equal sample size across the normal populations) to obtain i.i.d. (independent identically distributed) samples $\{x_i=(x_{i1},\dots ,x_{in}{)}^{\prime }:n\times 1,i=1,\dots ,k\}$ from the normal distribution $N({\mu }_i,{\sigma }_i^2)$ for each population $i=1,\dots ,k$ $(k\ge 2)$ . Here it is also assumed that samples from different populations $N({\mu }_i,{\sigma }_i^2)$ and $N({\mu }_j,{\sigma }_j^2)$ $(i\ne j)$ are also independent. We want to test the hypothesis of multiple mean comparison: $H_0:{\mu }_1=\dots ={\mu }_k$ $H_1:\text{at least two means differ}$ (1)

 Randomly selecting a population as population , we construct the observation matrix

$=\left(\begin{array}{cccc}{x}_{11}& -x_{k1}& x_{21}& -x_{k1}\\ x_{12}& -x_{k2}& x_{22}& -x_{k2}\\ ⋮& & \begin{array}{c}⋮\end{array}& \\ x_{1n}& -x_{kn}& x_{2n}& -x_{kn}\end{array}\begin{array}{cccc}\cdots & x_{k-1,1}& -x_{k1}& \\ \cdots & x_{k-1,2}& -x_{k2}& \\ ⋮& ⋮& & \\ \cdots & x_{k-1,n}& -x_{kn}& \end{array}\right):n\times \left(k-1\right)$ (2)

• Theorem. Let the observation matrix be given by (1.8). Define the following eigenvalue-eigenvector problem:¹⁵

$\left(\frac{1}{n}{X}^{\prime }X\right)d_i=\lambda d_i$ (3)

where $d_i=(d_{i1},\mathrm{...},d_{i,k-1}{)}^{\prime }$ for $i=(1,\cdots ,r)$ with $r=\min \left(n,k-1\right)-1$ being the number of positive eigenvalues ${\lambda }_1>\cdots >{\lambda }_r>0$ in (1.9).

Define$\begin{array}{l}{z}_i=(z_{i1},\cdots z_{in}{)}^{\prime }=X{d}_i,{\overline{z}}_i=\frac{1}{n}{\displaystyle {\sum }_{j=1}^n{z}_{ij}},\\ F_i(z_i)=\frac{n{\left({\overline{z}}_i\right)}^2}{\frac{1}{n-1}{\displaystyle {\sum }_{j=1}^n{(z_{ij}-{\overline{z}}_i)}^2}}.\end{array}$ (4)

$i=(1,\cdots ,r)$ for . Under the null hypothesis (1), $F_i$ has an exact $F$ -distribution $$F(1,n-1)$$ for $i=1,\cdots ,r=\min \left(n,k-1\right)-1$ .

Each of the $F_i$ -statistic given by (4) can be employed to test the hypothesis (1). For any given $i=1,\cdots ,r=\min \left(n,k-1\right)-1$ , reject the null hypothesis in (1) at a given level $0<\alpha <1$ for a large value of $F_i>F\left(1-\alpha ;1,n-1\right)$ , which stands for the $100\left(1-\alpha \right)$ -percentile of the traditional $F$ -distribution $F(1,n-1)$ . We suggest using the $F$ -statistic $F_1\left(z_1\right)$ in (4) associated with the largest eigenvalue in (3) based on the Monte Carlo study in Liang et al.¹⁵ The -test is called the PCA-test (principal component test).

A research project was carried out by Tianjin Medical University, China.¹ Rats were collected for experiment by four different treatments (doses) to see the treatment effects from 46 genes with sample size $n=6$ (rats) for each treatment. In the experiment on 6 rats, the ratio of organ wet weight to body weight (organ coefficient) was observed. The purpose is to evaluate organ development during the treatment. Details on the experiment and medical analysis can be found in Gao et al.¹ In one-way ANOVA, we can consider each factor level as a group or population. In the experiment on 6 male rats with 46 levels (genes), we consider if the ratio of organ wet weight to body weight has changed during the treatment.

Let

${\mu }_{1i}=\text{the average ratio of organ wet weight to body weight f or gene before treatment},$

${\mu }_{2i}=\text{the average ratio of organ wet weight to body weight f or gene after treatment}$ (5)

$i=1,\cdots ,46$ for . Then we need to test the hypothesis

$H_0:{\mu }_{1i}-{\mu }_{2i}=0\text{versus}{H}_1:{\mu }_{1i}-{\mu }_{2i}\ne 0$ (6)

For each of the two groups of rats with four treatments, we employ the Bartlett test for variance homogeneity before and after the treatments.¹⁶ The -values for testing homogeneity for those genes with significance at levels $\alpha =.05$ and $\alpha =.10$ are given in Tables 1–3 from the two groups of rats. It shows that there exists significant variance heterogeneity for those genes before and after the treatment. This implies that if one continues using the traditional ANOVA (analysis of variance) method for testing the significance of the genes after the treatment, the conclusion is doubtable because the data show a violation of variance homogeneity.

We apply both the classical one-way ANOVA $F$ -test and the new exact $F$ -test $F=F_1\left(z_1\right)$ in (4) (called PCA -test) to testing the significance for the genes in three groups. The -values from the two tests are summarized in Tables 4–7 below. The following conclusions can be summarized:

1) the red-colored genes with red $p$ -values are significant based on both ANOVA $F$ -test and PCA $F$ -test for level $\alpha =.05$ ;

 2) the red-colored genes with a red $p$ -value and a green-colored -value is significant based on PCA $F$ -test for $\alpha =.05$ or $\alpha =.10$ but insignificant based on ANOVA $F$ -test. Some of the genes are significant based on ANOVA $F$ -test for $\alpha =.10$ ;

 3) the ANOVA F-test fails to identify several significant genes at level $\alpha =.05$ or $\alpha =.10$ - those genes with red or green-colored p-values in Tables 4-7: genes Ar, Crhr2, Drd1a, Hcrtr2, Cyp19a1, Tacr3, and Trh in Table 4, Mtnr1b in Table 5, genes Gper, Grin2b, Hcrtr2, Lepr, and Mtnr1b in Table 6, genes Ar, Bdnf, Grin2a, Cyp19a1, and Tacr3 in Table 7.

The following box plots indicate there exists variance heteroscedasticity across different treatment groups. This means that the PCA-F test gives more convincing conclusions when testing the mean difference. Furthermore, the ANOVA fails to identify quite a few of significant genes.

Figures(1–4b)

Figure 1 Box plots for four significant genes in group male-neonatal.

Figure 1 Box plots for four significant genes in group male-neonatal.

Figure 2 Box plots for four significant genes in group male-ARC.

Figure 3 Box plots for four significant genes in group male-AVPV.

Figure 4 Box plots for four significant genes in group male-MPN.

The new exact -test in this paper is applicable for multiple mean comparisons without assuming homogeneity of variances across the populations. It is especially suitable for matched pair mean comparison in the situation of before and after treatments in medical research. When different experimental subjects show different responses to the treatments, it is very likely that there exists variance heterogeneity across the treatments. As a result, conclusions from the traditional ANOVA -test or the classical two-sample -test are doubtable. While there exists approximate solutions to the problem of two-sample mean comparison with heterogeneous variances, for example, Welch’s² approximate -test, and Dudewicz et al.⁷ method for an exact solution to the Behrens-Fisher problem, these methods are either based on the approximate null distribution of the test statistics or based on approximate computation of the -values. The method based on the new exact -test in this paper provides an accurate solution to the problem of two normal population mean comparison without any restriction on the population variances. The real data analysis shows the new exact -test could detect some situations of mean difference for which the traditional ANOVA -test fails. Therefore, the method based on the new exact -test in this paper is recommended to be used together with some existing methods for the same purpose in problems of multiple mean comparisons.^17–19

The new exact -test in this paper is applicable for multiple mean comparisons without assuming homogeneity of variances across the populations. It is especially suitable for matched pair mean comparison in the situation of before and after treatments in medical research. When different experimental subjects show different responses to the treatments, it is very likely that there exists variance heterogeneity across the treatments. As a result, conclusions from the traditional ANOVA -test or the classical two-sample -test are doubtable. While there exists approximate solutions to the problem of two-sample mean comparison with heterogeneous variances, for example, Welch’s² approximate -test, and Dudewicz et al.⁷ method for an exact solution to the Behrens-Fisher problem, these methods are either based on the approximate null distribution of the test statistics or based on approximate computation of the -values. The method based on the new exact -test in this paper provides an accurate solution to the problem of two normal population mean comparison without any restriction on the population variances. The real data analysis shows the new exact -test could detect some situations of mean difference for which the traditional ANOVA -test fails. Therefore, the method based on the new exact -test in this paper is recommended to be used together with some existing methods for the same purpose in problems of multiple mean comparisons.^17–1

This work was partially supported by a UIC New Faculty Start-up Research Fund R72021106, and in part by the Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science, BNU-HKBU United International College (UIC), project code 2022B1212010006.

The authors declare that there are no conflicts of interest.

1. Gao N, Hu R, Huang Y, et al. Specific effects of prenatal DEHP exposure on neuroendocrine gene expression in the developing hypothalamus of male rats. Arch Toxicol. 2018;92(1):501–512.
2. Welch BL. The generalization of Student’s problem when several different population variances are involved. Biometrika. 1947;34(1–2):28–35.
3. Turkey JW. Comparing individual means in the analysis of variance. Biometrics.1949;5(2):99–114.
4. Kramer CY. Extension of multiple range tests to group means with unequal numbers of replications. Biometrics. 1956;12(3):307–310.
5. Best DJ, Rayner JCW. Welch’s approximate solution for the Behrens-Fisher problem. Technometrics. 1987;29(2):205–210.
6. Kruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. J Amer Statist Assoc. 1952;47(260):583–621.
7. Dudewicz EJ, Ma Y, Mai E, et al. Exact solutions to the Behrens-Fisher problem: asymptotically optimal and finite sample efficient choice among. J Statist Plann & Infer. 2007;137(5):1584–1605.
8. Törönen P, Kolehmainen M, Wong G, et al. Analysis of gene expression data using self-organizing maps. FEBS Letters. 1999;451(2):142–146.
9. Brazma A, Vilo J. Minireview: Gene expression data analysis. FEBS Letters. 2000;480(2000):17–24.
10. Sherlock G. Analysis of large-scale gene expression data. Current Opinion in Immunology. 2000;12(2):201–205.
11. Yeung KY, Ruzzo W L. Principal component analysis for clustering gene expression data. Bioinformatics. 2001;17(9):763–774.
12. Slonim DK. From patterns to pathways: gene expression data analysis comes of age. Nature Genetics Supplement. 2002;32:502–508.
13. Parmigiani G, Garrett ES, Irizarry RA, et al. The Analysis of Gene Expression Data: An Overview of Methods and Software. Springer: 2003.
14. Wolf FE, Angerer P, Theis FJ. SCANPY: large-scale single-cell gene expression data analysis. Genome Biology. 2018;19:15.
15. Liang J, Tang ML, Yang J, et al. An application of the theory of spherical distributions in multiple mean comparison. In: Fan J, Pan J, editors. Contemporary Experimental Design, Multivariate Analysis and Data Mining - Festschrift in Honour of Professor Kai-Tai Fang. Springer-Verlag; 2020;189–199.
16. Bartlett MS. Properties of sufficiency and statistical tests. Proc Roy Statist Soc. (Ser. A). 1937;160:268–282.
17. Brown MB, Forsythe AB. Robust tests for the equality of variances. J Amer Statist Assoc. 1974;69(346):364–367.
18. Fang KT, Kotz S, Ng KW. Symmetric Multivariate and Related Distributions. Chapman and Hall Ltd. London and New York;1990.
19. Fang KT, Zhang YT. Generalized Multivariate Analysis. Science Press and Springer-Verlag. Beijing and Berlin;1990.