Statistical Reporting Standards: Transparent Reporting of Analyses
- Introduced by
- Statistical and methodological literature; emphasized by Cumming (2013), ICMJE, and replication crisis discussions
- Year
- 2005
- Type
- Guideline
- Family
- results-reporting
What does it do?
Transparent reporting of statistical results—including effect sizes, confidence intervals, p-values, and assumptions—is essential for scientific integrity and reproducibility.
Many published studies report p-values in isolation without effect sizes or confidence intervals, making it impossible for readers to assess the magnitude of findings. Statistical reporting standards, emphasized by Cumming (2013), the American Statistical Association, and the ICMJE, require effect sizes, confidence intervals, and discussion of uncertainty. This enables readers to judge whether findings are practically significant (not just statistically significant) and to compare effect sizes across studies in meta-analyses. Poor statistical reporting wastes research and prevents proper synthesis of evidence.
The intuition behind the method
P-values are widely misunderstood: a p<0.05 does NOT mean the result is 'important' or that the null hypothesis is 'false.' It simply means the observed data would occur less than 5% of the time if the null hypothesis were true. A trivial effect (e.g., 0.1% difference) can have p<0.001 with a large sample; conversely, a large clinically important effect (e.g., 50% reduction in symptoms) might not reach p<0.05 with a small sample. Effect sizes and confidence intervals address this: they quantify the magnitude of the effect and the range of plausible values. A 95% confidence interval of [2.1, 2.3] says we are 95% confident the true effect is between 2.1 and 2.3; if the confidence interval includes zero or a null effect value, the result is not statistically significant. This additional information—absent from p-values alone—enables readers to interpret findings meaningfully.
History
Statistical reporting standards evolved from a decades-long critique of p-value misuse in science. In 1992, the journal Epidemiology required confidence intervals alongside p-values; other journals followed. Cumming's (2013) influential paper 'The New Statistics' made the case for effect sizes and confidence intervals as superior to p-value thresholds. In 2019, the American Statistical Association published guidance cautioning against p<0.05 as a universal threshold for 'significance.' Major journals (JAMA, Lancet, etc.) now require effect sizes. The ICMJE Recommendations (updated 2023) emphasize reporting of effect sizes and uncertainty. This shift reflects lessons from the replication crisis: many published findings with p<0.05 failed to replicate because effect sizes were small, a fact hidden by p-values alone.
When to use it
Report statistics transparently in all empirical research: observational studies, experiments, surveys, clinical trials, meta-analyses. This applies to primary analyses and sensitivity/subgroup analyses. For descriptive studies without hypothesis tests (e.g., descriptive surveys), report descriptive statistics, ranges, and proportions clearly. For qualitative research, statistical reporting does not apply, but describe the sample and analysis method transparently. For modeling or simulation studies, report effect sizes and confidence intervals around key outcomes.
Limitations and considerations
- Requires space: comprehensive statistical reporting (effect size, CI, p-value, assumptions) takes more space than 'p<0.001' alone; journal limits may squeeze other content.
- Complexity: readers unfamiliar with effect sizes may find comprehensive reporting hard to interpret; education is needed.
- Interpretation ambiguity: 'meaningful' effect size depends on context (d=0.2 is small in some fields, large in others); no universal standard.
- Multiple comparisons inflate false positives: even with corrections, many hypothesis tests increase Type I error; comprehensive reporting does not solve this, only documents it.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Enables interpretation of magnitude: readers see not just 'significant difference' but 'the difference is d=0.47' (medium effect), crucial for judging practical significance.
- Facilitates meta-analysis: effect sizes allow studies to be combined; p-values alone are insufficient for meta-analysis.
- Reduces p-hacking bias: when effect sizes are reported, readers can see when effects are trivial despite p<0.05, reducing incentive for researcher degrees of freedom.
- Improves reproducibility: confidence intervals and assumptions checking enable readers to evaluate sensitivity and validity of findings.
- Supports post-hoc power analysis: reported effect sizes allow calculation of statistical power, revealing whether underpowered studies drove null findings.
Applications
- Writing a research article: plan statistical reporting before analysis; report effect sizes and CIs alongside p-values; organize results by hypothesis.
- Conducting a meta-analysis: effect sizes extracted from primary studies enable pooling; papers that report only p-values are harder to include.
- Reviewing a manuscript: check that statistics are reported comprehensively; request effect sizes and CIs if missing.
- Preparing a grant proposal: statistical power analysis (which uses effect sizes) is standard in NIH and other grant reviews; demonstrate familiarity with effect size reporting.
How it works
How it works
Report statistics in this order and format: (1) Descriptive statistics: mean ± SD (or median and interquartile range for skewed data) for each group. Example: 'Treatment group (M=45.2, SD=8.3) vs. Control group (M=42.1, SD=7.9).' (2) Statistical test name and result: Specify the test (t-test, chi-square, ANOVA, etc.), test statistic, degrees of freedom, p-value, and effect size. Example: 't(98)=2.34, p=.021, d=0.47.' (3) Confidence interval: Include 95% CI for the difference or effect size. Example: '95% CI [0.52, 3.21].' (4) Effect size: Report appropriate effect size for your test. For continuous outcomes: Cohen d (standardized mean difference), Pearson r (correlation), or eta-squared (proportion of variance explained). For binary outcomes: relative risk, odds ratio, number needed to treat. For means comparisons: report absolute difference with units. (5) Assumptions tested: state whether assumptions were met (e.g., 'Normality was assessed with the Shapiro-Wilk test, W=0.98, p=.45, indicating data are approximately normal'). (6) Corrections for multiple comparisons: if conducting multiple tests, report the correction method used (Bonferroni, False Discovery Rate, etc.) and the adjusted p-value threshold. Example: 'Bonferroni-corrected α=0.05/3=0.017.' (7) Missing data: describe how missing data were handled (deletion, imputation method). Organize results by hypothesis or research question; group related statistics. Use tables for detailed numerical results.
Quality statistical reporting meets these criteria: (1) Completeness: p-values, effect sizes, and confidence intervals are all reported; no naked p-values. (2) Accuracy: values are correct and consistent with the analysis described in Methods. (3) Transparency: the statistical method is named explicitly (e.g., 'paired t-test' not just 't-test'); assumptions are addressed. (4) Effect size: the appropriate effect size for the test is chosen (d for two-group comparisons, r for correlations, etc.). (5) Precision: values are reported with appropriate significant figures (p-values to 2–3 decimals, e.g., p=.021, not p<.05). (6) Consistency: similar results are formatted identically; a series of t-tests all report t, df, p, and d. (7) Clarity: results are presented in prose or tables in order of the research questions stated in the Introduction. Readers should be able to verify results by looking at tables and cross-checking with text.
Peer reviewers and editors check statistical reporting for transparency and compliance with field standards. Missing effect sizes, unsupported claims of 'significance,' or misreported statistics trigger revision requests. Publication bias is reduced when effect sizes and confidence intervals are reported: reviewers and readers can assess the true magnitude of findings, not just whether p<0.05. In meta-analyses, effect sizes from all studies are pooled; studies that report only p-values without effect sizes must be excluded or estimated post-hoc from other statistics, introducing error. For authors, transparent statistical reporting increases reproducibility, enables proper synthesis, and demonstrates rigor.
Common pitfalls
- Reporting p-values without effect sizes: 'p=.021' tells readers the result is 'significant' but not whether the effect is trivial (e.g., d=0.1) or large (d=1.5). Always report effect size.
- Confusing statistical and practical significance: p=.001 indicates strong evidence against the null hypothesis, but if the effect is clinically irrelevant (e.g., 0.1 mmHg difference in blood pressure), it is not practically significant. Discuss both.
- Cherry-picking p-values: reporting only the p-value that is <0.05 while hiding non-significant results (p-hacking). Pre-register hypotheses and report all tests, significant or not.
- Reporting 'trend toward significance' for p=.10 or 'approaching significance' for p=.06: these are informal and misleading. Either the result meets your threshold or it does not; discuss non-significant results honestly.
- Misreporting confidence intervals: a 95% CI is NOT the range that contains 95% of data; it is the range we are 95% confident contains the true parameter. Explain correctly to avoid confusion.
- Omitting baseline descriptive statistics: readers cannot assess whether groups were comparable or samples are representative if demographic/baseline data are not reported.
Frequently asked questions
What effect size should I report?
Choose the effect size appropriate to your statistical test: For t-tests (two-group comparison): Cohen d (standardized mean difference) or the raw difference with units. For ANOVA: eta-squared (η²) or partial eta-squared (η²p). For correlation: Pearson r or r-squared (R²). For binary outcomes (e.g., treatment success): odds ratio (OR), relative risk (RR), or absolute risk difference. For paired data: Cohen d or Hedge's g (similar to d, used in meta-analyses). If unsure, consult field-specific standards or reporting guidelines (EQUATOR Network).
Is it enough to report just the p-value?
No. The p-value alone is insufficient because it does not tell you the magnitude of the effect. A p=.001 could reflect a trivial effect (if the sample is large) or a huge effect (if the sample is small). Always report effect size and 95% CI. The combination of p-value + effect size + CI gives readers the full picture.
What if I have many tests and several p-values are 'close to significant' (e.g., p=.06)?
Report them honestly. A p=.06 is not statistically significant at α=0.05; it does not become significant by calling it a 'trend.' Discuss what the non-significant result means in context (lack of power? true null effect?). If you conducted multiple tests, report how you controlled for Type I error (Bonferroni, FDR, etc.). Pre-register hypotheses to distinguish confirmatory from exploratory analyses.
How many decimal places should I use for p-values?
Two to three decimal places is standard (e.g., p=.021, p<.001). Do not report p=.050000. Exact p-values are preferred to inequalities (p=.021 rather than p<.05) when the exact value is available from your statistical software. Use 'p<.001' only if the software rounds smaller values or if the test is one-tailed with a very small threshold.
Should I report 95% CI or other confidence levels?
Use 95% CI unless your field has a different convention (some older medical literature used 90% or 99%). The 95% CI is the standard because it aligns with α=0.05. If you report a 90% CI, explicitly state 'CI90%' to avoid confusion. Do not mix confidence levels in the same paper.
Related methods
Sources
Cumming, G. (2013). The new statistics: Why and how. Psychological Science, 25(1), 7–29.
Open DOI referenceCopied.Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2005). Editors can lead researchers to confidence intervals, but can't make them think: Statistical reform lessons from medicine. Psychological Science, 15(2), 119–126.
Open DOI referenceCopied.International Committee of Medical Journal Editors (2023). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals.
Open publication referenceCopied.