What are the steps involved in statistical hypothesis testing?

Updated May 17, 2026

Short answer

Statistical hypothesis testing is a structured process for using sample data to evaluate a claim about a population. The main steps are defining hypotheses, choosing a significance level, selecting an appropriate statistical test, calculating the test statistic and p-value, making a decision based on the evidence, and interpreting the results in context.

Deep explanation

Statistical hypothesis testing helps determine whether observed patterns in sample data provide enough evidence to reject a default assumption about a population. The process is designed around uncertainty because samples rarely represent populations perfectly.

The typical steps are:

Define the research question and hypotheses

Translate the business or scientific question into a statistical statement.

The null hypothesis ($H_0$) represents the default assumption, usually stating that there is no effect, no difference, or no relationship.
The alternative hypothesis ($H_1$ or $H_a$) represents the claim being tested, such as an improvement, difference, or relationship.

Example:

$H_0$: A new website design does not change conversion rate.
$H_a$: A new website design increases conversion rate.

The null hypothesis is not necessarily believed to be true; it is the baseline assumption that the test attempts to challenge.

Choose the significance level ($\alpha$)

The significance level defines the threshold for rejecting the null hypothesis. It represents the maximum acceptable probability of making a Type I error: rejecting a true null hypothesis.

Common choices are:

$\alpha = 0.05$ (5%)
$\alpha = 0.01$ (1%)

A lower alpha value makes the test more conservative because stronger evidence is required before rejecting $H_0$.

Select an appropriate statistical test

The choice of test depends on:

Type of data (continuous, categorical, ordinal)
Number of groups being compared
Distribution assumptions
Sample size
Whether observations are independent

Common tests include:

Goal	Common Test
Compare the mean of one sample to a known value	One-sample t-test
Compare means between two groups	Two-sample t-test
Compare proportions	Proportion z-test
Compare multiple group means	ANOVA
Test relationships between categorical variables	Chi-square test
Measure correlation	Pearson/Spearman correlation test

Collect data and calculate the test statistic

The test statistic measures how far the observed sample result is from what would be expected under the null hypothesis.

A generic form is:

TypeScript

1Test Statistic = (Observed Value - Expected Value) / Standard Error
2   ```
3
4   For example, in a two-sample t-test:
5
6   ```
7   t = (Mean_A - Mean_B) / Standard_Error
8   ```
9
10   A larger absolute test statistic indicates that the observed result is farther from the null hypothesis assumption.
11
125. **Calculate the p-value**
13
14   The p-value represents the probability of observing results at least as extreme as the sample results, assuming the null hypothesis is true.
15
16   Interpretation:
17
18   * Small p-value → data is unlikely under $H_0$ → evidence against $H_0$
19   * Large p-value → data is reasonably likely under $H_0$ → insufficient evidence against $H_0$
20
21   The decision rule is usually:
22
23   ```
24   If p-value <= alpha:
25       Reject H0
26   Else:
27       Fail to reject H0
28   ```
29
30   A p-value does not tell us the probability that the null hypothesis is true. It only evaluates how compatible the observed data is with the null hypothesis.
31
326. **Make a statistical decision**
33
34   Based on the p-value and significance level:
35
36   * **Reject the null hypothesis**: There is statistically significant evidence supporting the alternative hypothesis.
37   * **Fail to reject the null hypothesis**: There is not enough evidence to support the alternative hypothesis.
38
39   "Fail to reject" does not mean the null hypothesis has been proven true. It only means the available evidence is insufficient.
40
417. **Interpret results in the practical context**
42
43   Statistical significance does not always mean practical importance.
44
45   For example:
46
47   * A website change may increase conversion rate by 0.01% with a very large dataset.
48   * The result may be statistically significant but not meaningful enough to justify implementation costs.
49
50   Good interpretation considers:
51
52   * Effect size
53   * Confidence intervals
54   * Business impact
55   * Data quality
56   * Experimental design
57### Example workflow
58
59A/B testing commonly follows this pattern:

Define hypotheses: H0: New version has the same conversion rate. Ha: New version has a different conversion rate.

Select alpha: alpha = 0.05

Collect conversion data.

Run statistical test.

Calculate p-value.

Compare p-value with alpha.

Decide whether evidence supports changing the product.

TypeScript

1### Important concepts
2
3**Type I error**
4
5A Type I error occurs when the null hypothesis is rejected even though it is true.
6
7Example:
8
9* Concluding a new drug works when it actually does not.
10
11The probability of this error is controlled by alpha.
12
13**Type II error**
14
15A Type II error occurs when the null hypothesis is not rejected even though it is false.
16
17Example:
18
19* Missing that a new drug is effective.
20
21The probability of avoiding Type II errors is related to statistical power.
22
23**Statistical power**
24
25Power is the probability of detecting a real effect:

Power = 1 - Probability(Type II Error) ```

Higher sample sizes generally increase power because they reduce uncertainty in estimates.

Real-world example

A product team wants to know whether a new checkout flow increases purchases.

They run an A/B test:

Group A: Existing checkout flow
Group B: New checkout flow

Results:

Group A: 10,000 users, 500 purchases (5% conversion)
Group B: 10,000 users, 560 purchases (5.6% conversion)

The hypotheses are:

TypeScript

H0: Conversion rate of A = Conversion rate of B
Ha: Conversion rate of A != Conversion rate of B

A two-proportion test is performed:

Python

1from statsmodels.stats.proportion import proportions_ztest
2
3successes = [500, 560]
4samples = [10000, 10000]
5
6stat, p_value = proportions_ztest(successes, samples)
7
8alpha = 0.05
9
10if p_value < alpha:
11    print("Reject H0: checkout flow has a statistically significant effect")
12else:
13    print("Fail to reject H0: insufficient evidence of an effect")

If the p-value is below 0.05, the team has evidence that the new checkout flow changes conversion rates. They would then evaluate the effect size and business value before rolling it out broadly.

Common mistakes

* Treating the p-value as the probability that the null hypothesis is true.
* Rejecting or accepting hypotheses without defining them before looking at the data.
* Choosing a statistical test without checking assumptions such as independence or distribution requirements.
* Confusing statistical significance with business significance.
* Running many hypothesis tests without correcting for multiple comparisons.
* Increasing sample size until a desired result becomes statistically significant.
* Using a small sample size and drawing strong conclusions from weak evidence.
* Reporting only p-values while ignoring effect sizes and confidence intervals.
* Failing to consider whether the data collection process introduces bias.
* Interpreting "fail to reject the null hypothesis" as proof that no effect exists.

Follow-up questions

What is the difference between statistical significance and practical significance?
What are Type I and Type II errors?
How does sample size affect hypothesis testing?
Why do we use confidence intervals along with hypothesis tests?
How do you choose between a t-test and a z-test?
What is multiple hypothesis testing, and why is it a problem?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More Statistics interview questions