A Bayesian analysis answers decision questions that a p-value cannot: how much evidence is there for a null hypothesis, how much posterior mass is practically equivalent to zero, and does the whole answer depend on the prior you chose? All four tools below run live on the same Beta-Binomial model.
A Bayes factor compares two explanations of the same data directly: a sharp null hypothesis (θ equals a specific θ₀) against the flexible alternative your prior already describes. Unlike a p-value, it can return evidence for a null, the honest tool for "this media effect is absent."
| BF₀₁ | Evidence (Jeffreys, 1961; Kass & Raftery, 1995) |
|---|---|
| > 150 | Very strong evidence for the null |
| 20 – 150 | Strong evidence for the null |
| 3 – 20 | Positive evidence for the null |
| 1/3 – 3 | Only anecdotal / not worth more than a bare mention |
| 1/20 – 1/3 | Positive evidence for the alternative |
| < 1/20 | Strong to very strong evidence for the alternative |
The Savage–Dickey ratio. BF₀₁ = posterior density at θ₀ ÷ prior density at θ₀, valid when the prior of the alternative is continuous and matches the null's prior at θ₀ (Wagenmakers et al., 2010). It is exact for this conjugate Beta-Binomial model. Low density at a specific point can make the ratio sensitive to the chosen prior's spread, exactly why the sensitivity analysis below matters.
Even a "statistically real" effect can be too small to matter. A ROPE is a band around the null value θ₀, of half-width w you judge practically negligible; the analysis reports how much posterior mass falls inside it (Kruschke & Liddell, 2018).
Posterior predictive check. Simulate new datasets from the fitted posterior and compare with what you observed. If the observed result is a wild outlier, the model (or the question) is misfit. The theoretical distribution is shown below.
Simulate datasets using only the prior: before any data. This is a prior-predictive check: if your prior would produce absurd data, your prior is misspecified. (MacKay, 2003; Gelman et al., 2013.)
–
The honest question: would a different, still-reasonable prior change your conclusion? This profile holds the data fixed and sweeps the prior from an aggressive prior (strength 40) down to a near-uninformative prior (strength 1), tracing how the posterior mean and its 95% interval move.
When to worry: if the posterior mean or its interval crosses a substantive decision threshold across reasonable priors, your claim is prior-sensitive, report it as such and gather more data, rather than silently picking the prior that confirms the result. See van de Schoot et al. (2021) on routine sensitivity checks.