Use proper indicators of change in neuropsychological studies

doi:10.15406/mojamt.2017.03.00057

MOJ

eISSN: 2573-2935

Addiction Medicine & Therapy

Mini Review Volume 3 Issue 5

Use proper indicators of change in neuropsychological studies

Changyong Feng,^1,2

Verify Captcha

Regret for the inconvenience: we are taking measures to prevent fraudulent form submissions by extractors and page crawlers. Please type the correct Captcha word to see email ID.

Chongshu Chen,¹ Julia Z Zheng,³ Xiang Lu,¹ Bokai Wang,¹ Daryl I Smith,² Hongyue Wang¹

¹Department of Biostatistics & Computational Biology, University of Rochester Medical Center, USA
²Department of Anaesthesiology, University of Rochester Medical Center, USA
³Department of Immunology and Microbiology, McGill University, Canada

Correspondence: Changyong Feng, Department of Biostatistics & Computational Biology, University of Rochester Medical Center, 601 Elmwood Ave., Rochester, NY 14642, USA, Tel 585-275-4263

Received: June 29, 2017 | Published: July 27, 2017

Citation: Feng C, Chen C, Zheng JZ, et al. Use proper indicators of change in neuropsychological studies. MOJ Addict Med Ther. 2017;3(5):162–164. DOI: 10.15406/mojamt.2017.03.00057

Download PDF

Abstract

Some indicators have been proposed in biomedical research to index the reliable change of measurements after the intervention or treatment. A good indicator should have simple statistical property and can be easily used by people without formal statistical training. Our analysis shows that many mistakes occurred in medical publications in the application and interpretation of the indicators, especially in the statistical distributions of the indicators. In this paper we summarize some typical mistakes and give correct formulas.

Keywords: Practice effect; Normal distribution; Linear regression

Abbreviations

SDI: Standard Deviation Index; RCI: Reliable Change Index; SRB: Standard Regression-Based; PE: Practice Effect

Introduction

Repeated measurements have been widely used to study a change of outcomes in clinical neuropsychology, see Chelune et al.,¹ Du et al.,² Frerichs & Tuokko,³ Hinton-Bayre,⁴ Jacobson & Truax,⁵ Levine et al.,⁶ Ouimet et al.,⁷ Sherman et al.⁸ and Zahodne et al.⁹ To measure the change at the individual level, we first need to define a meaningful indicator of the change. A well-defined indicator should be interpretable both biologically and statistically. The statistical distribution of the indicator should be easily accessible to biomedical investigators.

Recently, Duff¹⁰ reviewed some indicators that have been used in medical journals. After reading Duff 's review paper, we have found several serious mistakes. Some of the mistakes are associated with errors that have appeared in the reference papers cited by Duff.¹⁰ Our survey shows that the statistical distributions of the indicators discussed in Duff¹⁰ are wrong. These mistakes may lead to invalid inferences and cause serious problems for biomedical investigators without appropriate statistical background. The purpose of our paper is to point out some of these mistakes, to explain the reason of these mistakes, and further to correct these errors.

Fundamentals of change indicators

In order to have an appropriate definition of the indicator for the change, we first discuss some fundamental facts. For a randomly selected individual in the experimental group, let T₁ and T₂ denote the measurements at time 1 and time 2, respectively. T₂ - T₁ is the change from time 1 to time 2. Let M₁ and M₂ be the population mean values of the measurements of the control group at times 1 and 2, respectively. The change of the individual effect in the experimental group after adjusting the practice effect (PE) is (T₂ - M₂) - (T₁ - M₁) = (T₂ - T₁) - (M₂ - M₁), see Duff.¹⁰ Suppose the standard deviations of T1 and T₂ are S₁ and S₂ with the correlation coefficient r12. Then the standard deviations of T₂ - T₁ and (T₂ - T₁) - ( M₂ - M₁) are the same and they are equal to $\sqrt{s_{1}^{2} + s_{2}^{2} 2 r_{12} s_{1} s_{2}}$ . The standardized change and the standardized change after adjusting PE are respectively. These two formulas are the fundamentals of construction of indicators of change.

$\frac{T_{2} - T_{1}}{\sqrt{S_{1}^{2} + S_{2}^{2} - 2 r_{12} S_{1} S_{2}}}$
$\frac{(T_{2} - T_{1}) - (M_{2} - M_{1})}{\sqrt{S_{1}^{2} + S_{2}^{2} - 2 r_{12} S_{1} S_{2}}}$

Mistakes in the use of indicators in medical publications

In this paper, we focus on two types of mistakes that have appeared in Duff:¹⁰

Order of operations
Statistical distribution of the indicators. Similar mistakes also appeared in some publications cited in Duff.¹⁰

Order of operations

In eqution 1 of Duff,¹⁰ the formula of the standard deviation index (SDI) is defined as

$S D I = T_{2} - T_{1} / S_{1}$ (1)

Order of operations is a fundamental part of elementary school math, see for example Ewen & Nelson.¹¹ One of the rules of the order of operations is that multiplication and division have higher priority than addition and subtraction. According to this rule, the right hand side of (1) is the same as

$T_{2} - (T_{1} / S_{1})$ (2)

For example 80-94.5/11.46 = 80 - (94.5/11.46) = 80 -8.25 = 71.75.

Anybody with knowledge of elementary school math should obtain this result. However, on p.255 of Duff,¹⁰ the calculation is 80-94.5/11.46 = -1.26, which is totally wrong.

A closer look at the Duff's¹⁰ result found that to calculate 80-94.5/11.46, he first calculated 80-94-5, which is -14.5, and then divided it by 11.46, which is -1.27. The rules of the order of operations are not fulfilled here and it appears that the incorrect assumption made that T2 - T1/S1 is the same as (T2 - T1)/S1.

It is clear that the author does not understand the rules of the order of operations. He simply assumes that T2 - T1/S1 is the same as (T2 - T1)/S1. In fact, for the measurement data, the right hand side of (1) appears to be in error. For example, suppose the unit of T1 and T2 is kilometer (km). Then the unit of S1 is also km. Physically, then the expression T2 - (T1/S1) is meaningless.

In the following discussion we examine Duff¹⁰'s formulas with the appropriate orders of operations.

For example, the SDI should be of the form

$S D I = (T_{2} - T_{1}) / S_{1}$ (3)

eqution 1 of Duff¹⁰ also reported several other indicators, which we assume are of the following forms:

The reliable change index (RCI)

$R C I = (T_{2} - T_{1}) S E D$ (4)

The RCI controlling for practice effect (PE)

$R C I P E = [(T_{2} - T_{1}) - (M_{2} - M_{1})] / S E D$ (5)

The Iverson¹¹ RCI controlling for practice effect (PE):

$R C I' P E = [(T_{2} - T_{1}) - (M_{2} - M_{1}) / S E D$ Iverson;¹¹ (6)

The standard regression-based (SRB) change formula

$S R B = (T_{2} - T_{1}) / S E E$ (7)

The expression of SED, SED Iverson,¹¹ and SEE will be discussed in relevant sections.

Statistical distributions of indicators

The paragraph designated ‘Methods for Assessing Reliable Change’ on p.253 in Duff,¹⁰ T1 and T2 are said to be the scores at times 1 and 2, respectively; and S1 and S2 the standard deviations of control group at time 1 and time 2, respectively. Under Equation 1 on the same page, S1 and S2 are said to be the standard deviations at time 1 and time 2. According to the paper, there is one control group and one experimental group. It seems that T1 and T2 should be measurements at times 1 and 2 of the experimental group, as the outcome of interest is the change of the experimental group. If this is true, S1 and S2 should be the standard deviations of the experimental group instead of the control group at times 1 and 2, respectively. It is not reasonable to standardize the measurement of the experimental group using the standard deviation of the control group unless we assume these two groups have the same standard deviations.

As discussed in sections above, the standard deviation of T2 - T1 is $\sqrt{S_{1}^{2} + S_{2}^{2} 2 r_{12} S_{1} S_{2}}$ . If (T1,T2) has a bivariate normal distribution, then T2 - T1 also follows a normal distribution. If we further assume that there is no change between times 1 and 2 (which means T1 and T2 have the same mean values), then has the standard normal distribution.

$\frac{T_{2} - T_{1}}{\sqrt{S_{1}^{2} + S_{2}^{2} - 2 r_{12} S_{1} S_{2}}}$

We discuss the distributions of some indicators listed in eqution 1in Duff.¹⁰

The distribution of SDI in formula (3): From above we know that SDI in (3) does not follow standard normal distribution. We cannot compare the SDI with the quantile of the standard normal distribution to see whether there is a significant change from time 1 to time 2. Unfortunately, on p.254, Duff¹⁰ assumes SDI has the standard normal distribution, which appears to be in error.

The distribution of RCI in formula (4): In Equation 1 of Duff,¹⁰ Hence, RCI does not follow the standard normal distribution. It also does not make sense to compare it with the quantile of the standard normal distribution to check if there is signi cant change from time 1 to time 2.

The distribution of the Iverson RCI in formula (6): According to Equation 1 in Duff,¹⁰ Hence, RCI’PE does not follow the standard normal distribution either. We cannot compare RCI’PE statistic with the quantile of the standard normal distribution to evaluate if there is a significant change from time 1 to time 2.

The distribution of SRB in formula (6): It is trivial to prove that best = r12S2/S1, not S2/S1 as has been reported in eqution 1 in Duff,¹⁰ and $S E E_{e s t} = S_{2} \sqrt{1 + r_{12}^{2}}$ . The formula of SEEest in eqution 1 of Duff¹⁰ appears to be incorrect.

Our discussions indicate that none of the indicators reviewed in Equation 1 of Duff¹⁰ follow standard normal distribution. Unfortunately, Duff¹⁰ and some medical publications compared those indicators with the quantile of the standard normal distribution to evaluate the change from time 1 to time 2. This practice is inappropriate and should be avoided in future medical research.

Correct formulas of indicators

In this section we present correct formulations for these indicators listed in Duff.¹⁰ With these formulas, the investigators can compare statistics to the appropriate quantiles of the standard normal distribution to check whether a reliable change has occurred between time 1 and time 2. Note that all indicators discussed above are in the form of the critical part is to find the correct standard deviation of the change.

Here are correct forms of those indicators

The standard deviation index (SDI)

$S D I = \frac{T_{2} - T_{1}}{\sqrt{S_{1}^{2} + S_{2}^{2} - 2 r_{12} S_{1} S_{2}}}$

The RCI controlling for practice e ect (PE)

$R C I_{P E} = \frac{(T_{2} - T_{1}) - (M_{2} - M_{1})}{\sqrt{S_{1}^{2} + S_{2}^{2} - 2 r_{12} S_{1} S_{2}}}$

The standard regression-based (SRB) change

$S R B = \frac{(T_{2} - M_{2}) - (r_{12} S_{2} / S_{1}) (T_{1} - M_{1})}{S_{2} \sqrt{1 - r_{12}^{2}}}$

Since the numerators of the RCI and the Iverson¹¹ RCI are exactly the same as those of SDI and RCIPE, respectively, we no longer need to introduce them.

Conclusion

In this paper, we point out some common mistakes related to indicators of change used in the medical literature. Some authors utilize (active language) the wrong distribution to compare the indicators and to determine whether a reliable change has occurred. Some indicators were redundant and not well defined.