How is a p-value calculated?

Updated May 17, 2026

Short answer

A p-value is calculated by comparing the observed test statistic from a sample to the distribution of that statistic under the null hypothesis. It represents the probability of observing a result at least as extreme as the one measured, assuming the null hypothesis is true. The exact calculation depends on the statistical test being used, such as a t-test, chi-square test, or z-test.

Deep explanation

A p-value calculation has three main steps:

Define the hypotheses

The null hypothesis ((H_0)) represents the default assumption, usually that there is no effect or no difference.
The alternative hypothesis ((H_a)) represents the effect or difference being tested.

Example:

[ H_0: \mu = 100 ]

[ H_a: \mu \neq 100 ]

This means we are testing whether the population mean differs from 100.

Calculate the test statistic

The test statistic measures how far the observed sample result is from what the null hypothesis predicts, scaled by uncertainty.

For example, for a one-sample z-test:

[ z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}} ]

Where:

(\bar{x}) = sample mean
(\mu_0) = population mean assumed by the null hypothesis
(\sigma) = population standard deviation
(n) = sample size

The numerator represents the observed difference, and the denominator represents the expected variation due to random sampling.

Convert the test statistic into a p-value

Once the test statistic is known, it is compared against the theoretical probability distribution expected under (H_0).

For example:

A z-test uses the standard normal distribution.
A t-test uses the Student's t-distribution.
A chi-square test uses the chi-square distribution.
An ANOVA test uses the F-distribution.

For a two-sided z-test:

[ p = 2 \times P(Z \geq |z|) ]

The multiplication by 2 accounts for extreme values in both directions.

Example:

If:

[ z = 2.5 ]

then:

[ p = 2 \times P(Z \geq 2.5) ]

Since (P(Z \geq 2.5)) is approximately 0.0062:

[ p \approx 2 \times 0.0062 = 0.0124 ]

The p-value is approximately 0.0124.

Interpretation

A small p-value means the observed data would be unlikely if the null hypothesis were true.

Common thresholds:

(p < 0.05): often considered statistically significant.
(p < 0.01): stronger evidence against the null hypothesis.
(p \geq 0.05): insufficient evidence to reject the null hypothesis.

However, the p-value does not tell us:

The probability that the null hypothesis is true.
The size or practical importance of an effect.
Whether the result is meaningful in a business or scientific context.

Example Calculation Using a t-Test

Suppose a company tests whether a new onboarding process changes average training completion time.

Given:

Historical average completion time: 20 days
Sample average after the change: 18 days
Sample size: 25 employees
Sample standard deviation: 5 days

The t-statistic is:

[ t = \frac{18 - 20}{5/\sqrt{25}} ]

[ t = \frac{-2}{1} ]

[ t = -2 ]

With 24 degrees of freedom, the t-distribution is used to calculate the probability of observing a value at least this extreme. That probability is the p-value.

A statistical library performs the final distribution lookup:

Python

1from scipy.stats import ttest_1samp
2
3completion_times = [17, 18, 19, 21, 16, 20, 18, 17, 19, 18,
4                    22, 16, 18, 20, 17, 19, 18, 21, 16, 18,
5                    17, 19, 20, 18, 17]
6
7statistic, p_value = ttest_1samp(completion_times, popmean=20)
8
9print(statistic)
10print(p_value)

The output p-value indicates whether the observed reduction is unlikely to have occurred by random sampling variation alone.

Real-world example

A product team launches a new recommendation algorithm and wants to know whether it increases user engagement.

They run an A/B test:

Control group: 10,000 users with an average session duration of 5.0 minutes.
Treatment group: 10,000 users with an average session duration of 5.2 minutes.
Null hypothesis: The new algorithm has no effect.
Alternative hypothesis: The new algorithm increases session duration.

The team calculates a test statistic from the difference between the two groups:

Python

1from scipy.stats import ttest_ind
2
3control = [5.1, 4.9, 5.0, 5.2, 4.8]
4treatment = [5.3, 5.1, 5.2, 5.4, 5.0]
5
6stat, p_value = ttest_ind(treatment, control)
7
8print(f"p-value: {p_value}")

If the resulting p-value is 0.02, the team would typically reject the null hypothesis at the 5% significance level. This suggests that the observed increase in session duration is unlikely to be explained by random variation alone.

The team would still need to consider:

The size of the improvement.
Whether the increase improves business metrics.
Possible unintended effects, such as lower conversion rates.

Common mistakes

* Interpreting the p-value as the probability that the null hypothesis is true.
* Assuming a statistically significant result is automatically practically important.
* Choosing a significance threshold after seeing the results.
* Running many tests and ignoring the increased chance of false positives.
* Using the wrong statistical test for the data distribution or experimental design.
* Treating a large p-value as proof that the null hypothesis is true.
* Ignoring sample size effects, where very large samples can make tiny effects statistically significant.
* Forgetting whether the test is one-tailed or two-tailed when calculating the p-value.

Follow-up questions

What is the difference between a p-value and a confidence interval?
Why does sample size affect the p-value?
What does a p-value of 0.03 mean?
How do you calculate a p-value for an A/B test?
What is the difference between statistical significance and practical significance?
What happens if you run many hypothesis tests at the same time?