This fun and simple simulation helps us to understand the relationship between power of a t-test and effect size for small sample sizes.
Power of a binary hypothesis test is the probability of rejecting the null hypothesis when the alternative hypothesis is true.[1] Mathematically speaking,
Power = 1 – P(type II error)
Hence, as the power of a test increases, the probability of making a type II error decreases.
Effect size is a measure to quantify the difference between two groups.[2]
In my R-code, I run 10,000 simulations per effect size for different sample sizes (n = 10, 20, 30).
The plot looks like this:
What do we learn:
- For a given effect size (especially for effect sizes less than 1.5), bigger samples have greater power than the smaller samples.
- For the given sample size, power increases with effect size.
The R code is posted hereunder:
[1] Wikipedia https://en.wikipedia.org/wiki/Power_of_a_test
[2] Coe R, University of Durham – It’s the Effect Size, Stupid https://www.leeds.ac.uk/educol/documents/00002182.htm
Jui Nerurkar 