Xu Cui

Even though I learned power analysis in elementary statistics class, I totally forget it because I never use it. As we have to put power analysis in our grant, I am forced to relearn this topic.

Definition: type II error:

• null hypothesis should be rejected but we don’t
• or, false negative (there is an effect but fail to detect it)

(Null hypothesis is usually boring, e.g. mean=0, or no difference between two samples etc. So null hypothesis usually means no effect, or negative. Rejecting null hypothesis is “finding positive effect” and thus usually “exciting”. )

Definition: the probability of type II error is called beta

Definition: power is 1 - beta

How to calculate beta (or power)?

You need (1) null hypothesis (2) alpha, type I error cut off, usually 0.05 (3) standard deviation (or distribution) of samples

Step:

1. calculate the decision point (to reject or accept null hypo) based on your null hypo and alpha
2. calculate beta based on the decision point above and (3)

Example:

You null hypothesis is “mean of A is 0″. You measured A 100 times. The mean is 1 and standard deviation is 10 (and thus the standard deviation of mean is 10/sqrt(100)=1). You are deciding if the null hypothesis should be rejected and what is beta if alpha is 0.05. You calculate and find the p value is 0.1587, which is bigger than 0.05 so you accept the null hypothsis.

Then what is the probability that this decision is indeed wrong (you accept null hypo but indeed null hypo is false)? or beta?

As you make decisions based on alpha = 0.05, that means your decision point of mean of sample is 1.65: if the mean of sample is bigger than 1.65, you reject null; if smaller than 1.65, you accept it (like in our case where mean is 1). So beta equals to the probability that you find mean of sample is <1.65, while the true mean is 1. And this probability is 0.74. And power is thus 0.26.

To increase power, you usually need to increase the sample size. In our case, as the standard deviation of sample is very high, we need a lot more samples. If our N = 400, then our decision point will become 0.83 and the power will increase to 0.63.