The following matrix summarizes the four possible outcomes of a hypothesis test, highlighting the probabilities of Type I error (\(\alpha\)), Type II error (\(\beta\)), and power (\(1 - \beta\)).
\[ \begin{array}{|c|c|c|} \hline & H_0\ \textbf{True} & H_0\ \textbf{False} \\ \hline \textbf{Reject } H_0 & \text{Type I error } (\alpha) & \text{✅ Power } (1 - \beta) \\ \hline \textbf{FTR } H_0 & \text{✅ Correct} & \text{Type II error } (\beta) \\ \hline \end{array} \]
Matrix of hypothesis test outcomes based on the truth of \(H_0\) and the decision to reject or not reject.
Example power statement: “At [power level]% power, this study can detect an effect size of \(\Delta\) or larger at \(\alpha\) = [significance level] with \(n\) = [sample size].”
Note: If power is close to \(\beta\), the chances of detecting a difference or making a Type II error are similar. This may occur when the effect size is small. Increasing the sample size can help.
R code:
library(tidyverse)
## Missing parameter is indicated with `NULL`
# One-sample t-test power calculation
power.t.test(n = 41, delta = .07,
sd = 0.2, sig.level = 0.05,
power = NULL,
type = "one.sample", alternative = "one.sided")
# Two-sample t-test example
power.t.test(n = 250, delta = 5,
sd = 25, sig.level = 0.05,
power = NULL,
type = "two.sample", alternative = "one.sided")SAS code:
/* Missing parameter is indicated with a period */
/* One-sample t-test */
proc power;
onesamplemeans
sides = 1
alpha = 0.1
nullmean = 20000
mean = 20450
stddev = 500
ntotal = 8
power = .; /* parameter to solve for */
run;
/* Two-sample t-test */
proc power;
*can specify mean for each (nullmean) and ntotal;
twosamplemeans
sides = 1
alpha = 0.05
meandiff = 4
stddev = 3
npergroup = 50
power = .; /* parameter to solve for */
run;
R code:
# Loop to calculate power for sample sizes from 60 to 80
powerholder = c()
samplesizes = seq(60, 80, by = 1)
for(i in 1:length(samplesizes)) {
powerholder[i] = power.t.test(n = samplesizes[i], delta = 0.07, sd = 0.20, sig.level = 0.05,
type = "one.sample", alternative = "one.sided")$power
}
# Create a data frame
powerdf = data.frame(samplesizes, powerholder)
# Plot the power curve
powerdf %>%
ggplot(aes(x = samplesizes, y = powerholder)) +
geom_line(color = "blue3", linewidth = 1.5) +
ggtitle("Power Curve") +
ylab("Power") +
xlab("Sample Size") +
ylim(0.75, 1.0) +
theme_bw()SAS code:
/* Power curve for different sample sizes */
ods graphics on;
proc power;
onesamplemeans
sides = 1
alpha = 0.05
nullmean = 0
ntotal = 60 80
mean = 0.07
stddev = 0.2
power = .; /* parameter to solve for */
plot x = n min = 60 max = 80;
run;
ods graphics off;
/* Power curve for different effect sizes */
ods graphics on;
proc power;
twosamplemeans
sides = 1
alpha = 0.05
nulldiff = 0
meandiff = 3 to 5 by 0.1
ntotal = 47
stddev = 4.5
power = .; /* parameter to solve for */
plot x = effect min = 3 max = 5;
run;
ods graphics off;