post hoc test | Definition

A post hoc test is a statistical procedure used after an ANOVA to determine which specific group differences are significant.

Understanding Post Hoc Tests in Social Science Research

What Is a Post Hoc Test?

A post hoc test is a type of statistical analysis performed after an analysis of variance (ANOVA) when the overall test finds a significant difference. The term “post hoc” is Latin for “after this,” which reflects the order in which the test is used. In simple terms, after researchers find out that at least one group mean is different, they use a post hoc test to figure out exactly which groups differ from each other.

In social science research, we often compare multiple groups—for example, different teaching methods, types of therapy, or income levels. ANOVA tells us if there’s a difference somewhere, but it doesn’t tell us where. That’s where post hoc tests come in.

Why Post Hoc Tests Matter

Social science researchers often compare more than two groups. Let’s say a psychologist tests four different kinds of therapy to see which one helps people reduce anxiety the most. ANOVA might show that not all therapies have the same effect. But which therapies are actually different from each other? A post hoc test answers that question.

Without a post hoc test, researchers might guess or make incorrect assumptions about where the differences lie. This could lead to wrong conclusions or ineffective policies. Using a post hoc test adds clarity and confidence to the findings.

The Role of ANOVA and When to Use Post Hoc Tests

To understand post hoc testing, we need to start with ANOVA (Analysis of Variance). ANOVA checks whether the means of three or more groups are significantly different. If the ANOVA result is not significant, there’s no need for further testing. But if the ANOVA result is significant, then a post hoc test is needed to examine which groups differ from each other.

This process is called multiple comparisons. Without using a proper post hoc test, doing many comparisons increases the risk of finding a difference just by chance—a problem called Type I error. Post hoc tests help control this risk.

Common Post Hoc Tests

There are several types of post hoc tests. Each has its own rules and strengths. The right one depends on the research question and the nature of the data.

Tukey’s Honestly Significant Difference (HSD)

Compares all possible pairs of means.
Controls the Type I error rate well.
Works best when group sizes are equal or nearly equal.
Common in education, psychology, and sociology research.

Bonferroni Correction

Adjusts the significance level by dividing it by the number of comparisons.
Very conservative (lowers the risk of false positives).
May miss real differences if the sample size is small.
Often used in political science and public health.

Scheffé’s Test

More flexible than other tests.
Can be used with complex comparisons, not just pairwise.
Very conservative, which reduces the chance of finding false positives.
Often chosen when researchers want to be extra cautious.

Games-Howell Test

Designed for situations where group sizes are unequal and variances are not equal.
More accurate under unequal conditions.
Common in social science surveys and criminology.

LSD (Least Significant Difference)

Compares groups without strict error control.
Only appropriate if the initial ANOVA is significant.
More likely to find differences, but also more likely to make mistakes.
Usually discouraged unless used carefully.

Example Applications in Social Science Fields

Psychology

A psychologist tests the effectiveness of three different treatments for depression: medication, cognitive behavioral therapy, and group therapy. ANOVA shows a significant difference in outcomes. A post hoc test like Tukey’s HSD reveals that medication and cognitive behavioral therapy are equally effective, but both are better than group therapy.

Education

An education researcher studies the impact of four teaching methods on student test scores. ANOVA finds a difference. The Bonferroni correction shows that method A is better than methods B and D, but not significantly different from method C.

Sociology

Sociologists study the number of community service hours completed by students in different income brackets. ANOVA indicates a significant difference. A Games-Howell post hoc test shows that students from higher-income backgrounds participate less than those from lower-income backgrounds.

Political Science

A political scientist analyzes voter turnout across five regions. ANOVA shows a difference in turnout rates. A Scheffé test identifies which regions differ while accounting for multiple testing.

Criminology

A criminologist compares recidivism rates for offenders in different rehabilitation programs. ANOVA reveals a difference. A post hoc test helps identify which specific programs are more effective than others.

How Post Hoc Tests Help Reduce Errors

The more comparisons researchers make, the more likely they are to make a Type I error—thinking there’s a difference when there isn’t one. Post hoc tests control the overall error rate, keeping the research findings more reliable.

Let’s say there are five groups. That means there are 10 possible pairwise comparisons. Without adjustment, the chance of a false positive increases with each test. Post hoc methods adjust for this by either changing the significance level or using formulas that lower the chance of error.

When Not to Use

Post hoc tests should only be used after a significant ANOVA result. If ANOVA finds no overall difference, running post hoc tests is misleading. Also, post hoc tests are not a replacement for planned comparisons. If a researcher has specific hypotheses ahead of time, they should use a priori (pre-planned) comparisons instead of post hoc testing.

Planned Comparisons vs. Post Hoc Tests

Planned comparisons are decided before the data is collected. Researchers test only the specific differences they expect.
Post hoc tests are used when the researcher explores the data after seeing the results of ANOVA.

Planned comparisons are more focused, while post hoc tests are broader and more exploratory.

Interpreting Post Hoc Test Results

Most statistical software packages provide results in tables. A typical output includes:

Group pairs being compared
Mean difference
Confidence intervals
p-values

If the p-value is below a certain threshold (usually 0.05), the test shows a significant difference between those two groups.

Researchers must be careful not to interpret small differences as meaningful unless they are statistically significant. It’s also important to consider effect sizes, which show how large or important the difference is, not just whether it exists.

Visualizing Post Hoc Results

Graphs and charts can help make post hoc results easier to understand:

Bar charts with error bars show how group means compare.
Box plots give a visual of group distributions.
Letters or symbols above bars can indicate which groups are significantly different.

These visuals help audiences quickly grasp complex results, especially in presentations or published reports.

Challenges and Misunderstandings

Some researchers misuse post hoc tests by:

Running them without a significant ANOVA result.
Using too many tests, increasing the risk of errors.
Ignoring assumptions (like equal variances or group sizes).

To avoid these problems, researchers should carefully check the data before running post hoc tests and choose a test that fits the data’s conditions.

Best Practices

Check ANOVA results first. Only proceed with post hoc testing if the overall test is significant.
Select the right test. Use Tukey for equal group sizes, Games-Howell for unequal ones, etc.
Report clearly. Include which test was used, p-values, and confidence intervals.
Control for errors. Use tests that adjust for multiple comparisons to reduce false positives.
Consider effect sizes. Statistical significance isn’t everything—look at how meaningful the difference is.

Summary

Post hoc tests are essential tools for social science researchers who use ANOVA to compare multiple groups. These tests identify which specific groups differ after finding an overall difference. By adjusting for multiple comparisons, post hoc tests reduce the risk of drawing incorrect conclusions. When used properly, they add clarity and depth to research findings in psychology, education, sociology, political science, criminology, and more.

Glossary Return to Doc's Research Glossary

Last Modified: 03/22/2025