You reject the null hypothesis when your observed data are unlikely enough under the null that you conclude the null is not credible. The formal rule: compute a test statistic from your sample, then compare it to the null distribution it would follow if the null were true. If the probability of getting a result at least that extreme (the p-value) falls below your pre-set significance level, alpha, you reject the null. If not, you fail to reject. That single threshold decides the outcome, but the reasoning beneath it is where most thin explainers stop. This one keeps going.
This guide covers the machinery behind the decision, the quantified standards that govern it, the edge cases that break naive rules, the misconceptions that survive in professional reviews, and the practical steps that keep the decision honest.
The mechanism: why a p-value answers the question
Start with the null hypothesis. It states a specific claim about a population parameter, usually no effect, no difference, or no association. The null is not a guess that the effect is zero; it is a mathematical device that fixes the probability distribution of your test statistic. That fixation is what makes inference possible. Under the null, a sample mean has a known sampling distribution centered on the null value. A difference in proportions has a known distribution. A t statistic has a t distribution with degrees of freedom tied to sample size.
You then ask: if this distribution were the truth, how far into its tails does my observed statistic fall? The p-value is the answer measured as probability. A small p-value means the observed result would be rare under the null. Rareness alone does not prove the null false; it establishes that the null and the data are in tension.
The rejection rule formalizes that tension with a decision threshold. If p is less than alpha, reject. The test statistic crossing its critical value and the p-value dropping below alpha are the same event stated two ways. The critical value, alpha, and the p-value are locked together: set alpha first, derive the critical region, then check whether the statistic lands there.
Why does a threshold exist at all rather than a smooth verdict? Because the decision forces a binary action, and the threshold controls the long-run error rate. With alpha at 0.05, a researcher who runs many independent tests on true nulls will reject roughly 5 of every 100. That is the price of the decision, paid in false positives.
The mechanics hinge on the distribution, not on the effect size. A large effect measured with high noise can produce a large p-value. A tiny effect measured with enormous precision can produce a tiny p-value. The null test answers a probability question about the data, not a magnitude question about the world. Keep that distinction in view; it governs every edge case below.
Quantified context: thresholds, power, and sample size
The most common alpha in the scientific literature is 0.05, a convention inherited from early twentieth-century statistics and entrenched by journal practice, not by a law of nature. Particle physics uses 5 sigma, a tail probability near 3 in 10 million, because announcing a new particle demands extreme caution. Genomics runs millions of tests per study and shrinks alpha per test to control the family-wise error rate.
Sample size changes the meaning of alpha without changing its value. With 20 observations, a t test at alpha 0.05 requires a t statistic beyond roughly 2.09 in absolute value, a fairly demanding bar against noise. With 2,000 observations, the critical value drops near 1.96, and small real effects become detectable. The threshold stays fixed; the precision of the statistic moves. That is why a study with huge n can reject a null for an effect too small to matter practically.
Power is the complement of beta, the false-negative rate. A test powered at 0.80 has an 80 percent chance of rejecting the null when a specified true effect exists. Below 0.80, a nonsignificant result is uninformative: the study may simply have been too small. A common practical target is 0.80, with 0.90 preferred when the cost of a missed effect is high. Power depends on effect size, alpha, and n together, so a small pilot with 30 subjects running at 0.50 power tells you almost nothing from a null result.
An effect measured in the sample is a point estimate with a confidence interval. A 95 percent confidence interval is the set of null values that would not be rejected at alpha 0.05. This identity is exact for many standard tests and useful everywhere: if the interval excludes zero, the two-sided test at 0.05 rejects the null of zero effect. If it includes zero, the test does not reject. The interval carries more information than the p-value because it shows which alternatives survive.
Comparison table: the decision under common designs
The table below sets out the main two-sided tests at alpha 0.05 and shows what the rejection rule looks like in each. Critical values are approximations from standard distributions; exact numbers depend on degrees of freedom and test setup.
| Test | Null hypothesis | Statistic | Two-sided rejection region at alpha 0.05 | Use when |
|---|---|---|---|---|
| One-sample z test | Population mean equals a fixed value, known variance | z | |z| greater than 1.96 | Large n, variance known |
| One-sample t test | Population mean equals a fixed value, unknown variance | t | |t| beyond critical t with n minus 1 df | Small n, variance estimated |
| Two-sample t test | Two population means are equal | t | |t| beyond critical t with pooled or Welch df | Comparing two groups |
| Paired t test | Mean within-pair difference is zero | t | |t| beyond critical t with pairs minus 1 df | Before-and-after designs |
| Chi-square test of independence | Two categorical variables are independent | chi-square | Beyond critical value with (rows minus 1)(columns minus 1) df | Contingency tables |
| One-way ANOVA | All group means are equal | F | Beyond critical F with group and residual df | Three or more groups |
The pattern is the same in every row: a statistic, a reference distribution fixed by the null, a critical region, and a binary decision. Reading the table backward is a useful discipline. Ask which null value would sit exactly at the critical value; that is the boundary of the confidence interval.
Edge cases: one-sided tests, multiple comparisons, and continuity
One-sided tests shift the critical region entirely into one tail. A one-sided t test at alpha 0.05 rejects when t exceeds roughly 1.645, not 1.96, but it cannot detect effects in the opposite direction at all. The choice must be made before seeing the data. A one-sided test chosen after inspecting results halves the p-value by construction and converts ordinary noise into a significant finding.
Multiple comparisons are the largest practical hazard. Run 20 independent tests on true nulls at alpha 0.05, and the chance that at least one rejects is about 64 percent. Run 1,000, and it exceeds 99.9 percent. Standard corrections compute a family-wise adjusted alpha: Bonferroni divides alpha by the number of tests, which is conservative; Benjamini-Hochberg controls the false discovery rate instead, tolerating a defined proportion of false positives among rejections in exchange for more power. The threshold you compare against changes; the mechanism does not.
Tests that assume continuous data behave badly on discrete outcomes. With small counts, the normal approximation underlying the chi-square test overstates significance. Exact tests, such as Fisher's exact test, compute the tail probability directly and avoid the approximation error, at the cost of lower power. Similarly, a t test on heavily skewed data with small n can produce a p-value that means little, because the sampling distribution of the mean has not converged to the t distribution.
The counterintuitive detail professionals know: a p-value just under the threshold is not strong evidence, and a p-value just over is not evidence of no effect. A p of 0.049 and a p of 0.051 describe nearly identical evidence, yet a naive threshold treats them as opposite verdicts. The threshold is a decision rule, not a measurement of truth. The distance between the observed statistic and the critical value is the honest index of fragility.
Misconceptions: what rejection does not mean
Rejecting the null does not establish that the alternative hypothesis is true in any strong sense. It establishes that the data are inconsistent with the null at the chosen alpha. The alternative you tested may be wrong in form: a linear effect, for instance, when the true relationship is a threshold. Rejection points away from the null; it does not point to a specific mechanism.
A p-value is not the probability that the null is true. It is the probability of data at least this extreme, given that the null is true. Confusing those two conditions is the single most common inferential error in the applied literature and appears regularly in professional reviews that should know better. The probability of the null given the data requires Bayesian reasoning and a prior; the p-value cannot supply it.
Statistical significance is not practical significance. With a large enough sample, a difference of a fraction of a percent in a conversion rate or a test score can produce p less than 0.001 while changing nothing anyone should act on. The confidence interval width is the antidote: report the estimate and the range, not just the verdict.
A high p-value does not prove the null. Absence of evidence is not evidence of absence, especially in underpowered studies. Equivalence testing addresses this gap by flipping the null to assert an effect at least as large as a pre-defined margin; only rejection of that null supports a claim of practical equivalence.
Red flags that should trigger scrutiny before you trust a rejection: The alpha, the direction of the test, or the primary outcome changed after the data were seen. No correction appears despite many outcomes, subgroups, or time points. The result survives only after dropping observations or swapping the analysis model. The reported p-value sits just below the threshold with wide confidence intervals. The null was never precisely stated, so no distribution could have been fixed in advance.
Timeline table: the decision in sequence
The whole procedure unfolds in a fixed order. Each row marks a step where choices get locked in and cannot be revisited without invalidating the inference.
| Stage | Action | Locked-in choice | What goes wrong if skipped |
|---|---|---|---|
| 1. Framing | State the null and alternative, choose two-sided or one-sided | The hypothesis pair | Direction shopping after results |
| 2. Planning | Set alpha, target power, estimate required n | The error rates | Underpowered study, uninformative null result |
| 3. Analysis plan | Specify the primary endpoint, subgroups, and correction method | The testing family | Uncontrolled false positives |
| 4. Data collection | Follow the protocol without peeking at the primary outcome | The sample | Inflated type I error from optional stopping |
| 5. Testing | Compute statistic and p-value, compare to alpha | The verdict | Post hoc threshold changes |
| 6. Reporting | Give the estimate, confidence interval, p-value, and n | The complete record | Selective reporting and overstated certainty |
The sequence matters because the null distribution and the decision threshold must be fixed before the statistic is observed. Every step after that is mechanical. Most statistical scandals trace to a step being taken out of order, not to the math being wrong.
Practical actions: running the decision correctly
Before collecting data, write down the null value, the alternative, the alpha, and the target power. Use a power calculator to derive the sample size, and treat that number as a floor. For a common two-group comparison of means with a standardized effect of 0.3, a balanced design needs roughly 350 participants across both groups for 0.80 power at two-sided alpha 0.05; that requirement collapses to about 90 when the effect is 0.6.
At analysis time, check the assumptions the test relies on. For continuous outcomes, inspect the distribution and group variances. For counts, consider an exact test. Decide between Bonferroni and Benjamini-Hochberg before looking at which tests come close to the threshold. Report the test statistic, degrees of freedom, p-value, and effect estimate with its interval together. A verdict without an interval invites overstatement.
When the p-value lands near the threshold, treat the result as provisional and say so. If a decision hinges on a p-value between roughly 0.04 and 0.06, a small protocol deviation or a model swap can flip it. Replicate before acting. If the result is nonsignificant, check the power calculation against the effect you actually observed, and ask whether an equivalence test would answer the real question.
Use a software check to reproduce the p-value from a second implementation, especially for nonstandard tests. Most statistical packages compute standard tests identically, so a discrepancy signals a configuration error in the test specification. Document the choice of correction and the full set of tests conducted.
Limitations and the honest verdict
The null test is a tool for one narrow job: deciding whether data are compatible with a stated model. It is sensitive to every assumption in that model, from independence of observations to the correct functional form of the estimator. Violation of those assumptions invalidates the tail probability, and no threshold can rescue an invalid calculation.
The decision is also conditional on alpha, power, and the null value, all of which are human choices. Reasonable analysts can legitimately disagree about a result near the boundary. That is not a defect to eliminate but a property to disclose. Reporting estimates with intervals, the full testing family, and the sample size gives readers what they need to judge the verdict themselves.
And the result of any single study is evidence, not proof. The path from a rejection to a scientific conclusion runs through replication, independent data, and mechanisms that make the finding plausible. Treat each null test as one weight on one side of a balance, and report it as such.