George Rebane
Over the year’s RR has attracted a bevy of readers with keen minds and extraordinary abilities to peer into our collective futures. The certitude of some of these Que Sera predictions have most certainly been breathtaking to behold. In this election season these prognositcators make their considerable talents known to one and all at a prodigious rate in these pages. But how good are they really, or are they just blowing some cleverly contoured smoke?
Not too long ago we ran into a similar question about ourselves as members of a small informal investor group which met about every two weeks or so to puzzle out the markets and discuss what each was investing in or considering. Assessing the future of certain critical markets was part of our regimen, and we wondered how good we were at it. The answer to that question will hold until another time, however the methodology we used to obtain the answer may be of interest in the present.
Being a ‘quant’ versed in these things, I devised a simple scoring algorithm based on the probabilistics of the boxcar (or uniform) density function. The boxcar is the simplest measure of a random variable like the predicted value of the closing price of gold in London on 31 October 2012. Of course, we can pick any other quantity we like, and I’ll get to that in a moment. But first, here is what a boxcar density looks like for some random variable x.
The boxcar says that with certainty (probability = 1), x will fall between xLow and xHigh (more formally x >= xLow AND x <= xHigh). So the length of the boxcar is simple (xHigh – xLow). Since the height of the boxcar is uniform over the range of x, its value is the constant 1/(xHigh – xLow), and it is called the ‘likelihood’ of the boxcar density.
So we picked some markets of interest and submitted our highs and lows for them at a given date. Scoring was simple. If the market value fell within your boxcar, you were awarded a score based on the likelihood value of the boxcar that you predicted. As you can see from the formula above, the shorter your boxcar, the higher your score. If you think about it, that’s the reasonable way to assess the worth of any quantitative prediction. Consider the relative worth of predicting gold to be between $1,500 and $1,550 on New Year’s Eve, and the prediction that it will then be between $1,000 and $2,000. Your score will be zero if the actual value turns out not to be in your boxcar.
To make things concrete and illustrate the simplicity of it all (see above figure, click to enlarge). I have extracted an example spreadsheet illustrating four competitors who have predicted the value of the SP500, Gold, Crude Oil, and 10-year Treasury interest rate on the given date. The actual 31dec09 values are shown as are the predicted boxcar limits and resulting scores for each competitor. The exact score value is calculated by taking the likelihood and multiplying it by the realized value of the predicted random variable (here the four markets). Total score across the markets is just the sum of the individual scores with the ‘Leader’ indicated.
We published the intermediate scores of the competition to see how everyone was doing as the end date approached. Our payoff was that the losers would pay pro-rata for a lunch or dinner attended by everyone and their guests. Lots of fun.
The Proposition. So I propose that those who wish to participate state their predictions under the soliciting post. I will do the math to report progress and tabulate the winner. Each competitor would agree to send a check for a pre-specified amount to a 501c3 (tax deductible) recipient designated by the winner and in the winner’s name. The winner will supply all the relevant recipient data.
To gauge interest, I offer the invitation for a $20 contribution to predict the first publication of the of a poll by Gallup on what fraction said Obama won the 3oct12 debate, AND the same poll’s fraction who favored Romney as the victor (we’ll leave out any ‘undecideds’).
Please feel free to suggest other such charitable prognostications we may have an interest in holding a competition – e.g. predict what fraction of voters will vote for Obama, Romney, … . The only requirement is that we have a robust, accessible, and mutually accepted adjudication source.
[4oct12 update]
“So, who do you think won, honey?”
[8oct12 update] The resulting scores for the competition are posted below. All entrants are invited to send their checks to SESF’s TechTest2012 Scholarship fund with much appreciation.
