Bayes vs. Popper

From: Eliezer Yudkowsky
Affiliation:
Address: http://yudkowsky.net/bayes/technical.html
Date: 10 Dec 2006
Time: 22:11:54

Comments

Modern Bayesians don't consider *solely* the fit of theories to evidence, but also the simplicity of the theory. Otherwise, if you flip a coin a thousand times and get some random-looking result "HTTTHTHH...", the maximum-likelihood model is that the coin is a fixed coin which produced exactly this result. Only by penalizing the prior probability of this hypothesis can we avoid its posterior probability overwhelming that of the fair-coin hypothesis. I mention this especially because you spoke of validating a VaR model by backtesting it against the data - quite possibly some of the same data that was used to select the model *class*, even if you try to avoid contaminating the model itself. Science generally requires *advance* prediction, which is one approach to avoiding the problem of overfitting. It's also possible - from a Bayesian perspective - that a confirmed theory makes predictions about variables which are very hard to observe. It may be that after you've confirmed your relativistic theory of gravity to fourteen decimal places, it makes predictions about, say, what happens to a comet that's going to impact the far side of the Sun, in a fashion unobservable from Earth. I would say that this prediction is very likely true in real life - not at all meaningless - even though I can't figure out how to test it. With a powerful enough, massively confirmed model of a portfolio, we might be able to extrapolate well-calibrated statements about 10-day, 99% VaR. The problem is that the confirmed models aren't that powerful, and the powerful models aren't that confirmed.

Disclaimer