Do not use arbitrary thresholds related to 5 sigma or Bayes factor tables (Kass & Raftery)!
You should set your prior odds based on your expectations for the physical effect.
Search
Degree of surprise
Impact
LEE
Systematics
Number of \(\sigma\)
Higgs search
Medium
Very high
Mass
Medium
5
Single top
No
Low
No
No
3
SUSY
Yes
Very high
Very large
Yes
7
\(B_s\) oscillations
Medium/low
Medium
\(\Delta m\)
No
4
Neutrino oscillations
Medium
High
\(\sin^2(2\theta), \Delta m^2\)
No
4
\(B_s \to \mu\mu\)
No
Low/Medium
No
Medium
3
Pentaquark
Yes
High/very high
M, decay mode
Medium
7
\((g-2)_\mu\) anomaly
Yes
High
No
Yes
4
H spin \(\neq 0\)
Yes
High
No
Medium
5
\(4^{th}\) generation \(q, l, \nu\)
Yes
High
M, mode
No
6
\(\mathrm{v}_\nu > c\)
Enormous
Enormous
No
Yes
\(>8\)
Dark matter (direct)
Medium
High
Medium
Yes
5
Dark energy
Yes
Very high
Strength
Yes
5
Grav waves
No
High
Enormous
Yes
7
Lyons 2013 (arXiv:1310.1284), Table 1.
You need to choose a global property of the data. Not just one individual Bayes factor!
If I ask if the strain in LIGO is an elephant stomping (\(\mathcal{M}_1\)) or a boson star merger (\(\mathcal{M}_2\)), the Bayes factor will show overwhelming evidence for the boson star merger, but that doesn’t mean it’s a boson star merger.
You need to compare your model against all state of the art models (and possibly even future ones)!
You can also do posterior predictive checks to see if the data is consistent with your model.
GW200129, Gupte+ 2024 (arXiv:2404.14286), Fig. 5.
Get a better likelihood. This could involve improving your waveform model or improving your noise model.
OR
Trust that the Bayes factor you obtained dominates over your expected systematics.
Yes, and that’s the point. One should be careful to choose a reasonable prior.
The Occam penalty is a feature of the Bayes factor, not a bug. For a flat likelihood the model with a smaller prior volume is preferred.
