Sandtable

After the Nate Silver talk (see previous blog post), there was some discussion around TV pundits and their predictions. Naturally, the subject drifted to football pundits, and in particular Mark Lawrenson of the BBC. Mark Lawrenson is a former professional football player, and now long time BBC pundit who regularly appears on the famed Match of the Day programme. For several years, he’s been making predictions for each EPL matchday, and pitting himself against a guest. We thought it might be fun to do a little analysis of his results, and investigate how expert this expert really is.

For each matchday, Lawro makes predictions of scores for each match to be played. A tally is kept of his correct results and correct scores. His predictions can be found at the BBC site: http://www.bbc.co.uk/sport/, or compiled at the site: http://www.myfootballfacts.com/.

For now, our analysis has focussed on match outcomes rather than specific scores. Predicting scores is much harder; think of all the combinations, versus three possible match outcomes.

Let’s get into some analysis. Here are the statistics for the last three EPL seasons, including the 2012-2013 season that finished recently. Data grabbed from: www.football-data.co.uk/.

Result	2010-2011	2011-2012	2012-2013	Average
Home Win	179 (47.11%)	171 (45%)	166 (43.68%)	45.26%
Away Win	90 (23.68%)	116 (30.53%)	106 (27.89%)	27.37%
Draw	111 (29.21%)	93 (24.47%)	108 (28.42%)	27.37%

The first thing you’ll notice, and may well be surprised by, is the just how strong the Home bias is. Recent analysis in the book ‘Scorecasting’ by Moskowitz and Wertheim claims this is due to (unintentional) referee bias. Also, you’ll notice that the averages across seasons for Away Wins and Draws are equal. Interesting.

Now, here are Lawro’s results for the last three seasons.

	2010-2011	2011-2012	2012-2013	Average
Correct Results	168 (44.21%)	179 (47.11%)	201 (52.89%)	48.07%
Correct Scores	41 (10.79%)	41 (10.79%)	44 (11.58%)	11.05%

You’ll immediately observe that Lawro is getting better at predicting results, and that his correct scores’ percentages are pretty steady.

These results really beg the question: How is Lawro getting so much better? Is he using odds or modelling to make his predictions?

Back to it. Given these results, if you were to predict a Home Win for all games (all 380 games in an EPL season), you would, on average, get 45.26% results correct. Not too bad. Better than Lawro in 2010-2011. But he has got a lot better since then.

Based on this analysis, and our discussions, we decided to develop our own strategy for predicting outcomes. We thought we would try using Home Wins as a basis.

Our simple strategy is as follows:

choose Home Win, unless home team in current bottom 3, then result is Away Win;
or pick Away Win if away team is in current top 4, and home team not in current top 4;
if no games played, i.e. beginning of the season, then pick Home Win.

The intuitive idea being that, particularly at the top and bottom, current league position is a pretty good indicator of the strength of a team, and so we combine it with Home Wins (given the enormous Home bias). We predict no draws.

We evaluated our strategy, simulating each season’s games and generating league tables as we simulate, and here are the results:

	2010-2011	2011-2012	2012-2013	Average
Correct Results	177 (46.58%)	190 (50%)	182 (47.89%)	48.16%

The results show that we are better than Lawro for the 2010-2011 and 2011-2012 seasons, and slightly better on average across the three seasons. But Lawro had a great season this year!

But how good are Lawro’s and our predictions really? We compared the results with pre-match odds for game outcomes for two high-street bookies. We grabbed the data from: http://www.football-data.co.uk.

We transformed the provided decimal odds to probabilities (1/odd), then picked the most likely outcome (highest probability) as the match outcome. If the odds reflect a good estimate of the true probability of the outcomes, we would expect them to predict results pretty well in the long run.

The results for William Hill and Ladbrokes are:

	2010-2011	2011-2012	2012-2013	Average
William Hill	191 (50.26%)	199 (52.37%)	201 (52.89%)	51.84%
Ladbrokes	191 (50.26%)	198 (52.11%)	204 (53.68)	52.02%

Observe that the bookies only predict just above 50% of the results correctly. And, yes, you are reading that right: this season Lawro did as well as William Hill!

To conclude, on the one hand, we’re impressed that our simple strategy appeared to work better than Lawro. But on the other, we’re intrigued by how much better he was this season. When does next season start?

Final note: our strategy could probably be improved through a better initialisation. Currently, we pick Home Win if no games have been played yet. It would be possible to use the previous seasons’ final league table instead, plus some exceptions for promoted teams. Exercise for the reader.

Our analysis and tools can be found at: https://github.com/tfrench/lawro/.

Nate Silver recently spoke at the Royal Geographic Society, here in London. He’s possibly the closest thing to real celebrity statistics and modelling currently has, and yet, he was quite humble in presentation. In fact, he’s the first to admit that his popular 538 Blog’s US election model is based on quite a simple approach:

Track and aggregate daily polls
Weight the polls based on their past performance, and political bias
Establish a margin of error, which shrinks as the election nears

For a statistician, this approach seems obvious, but in the space of political punditry and news media outfits this is revolutionary Of course, tracking polls, and the poll’s performance requires a higher level of effort and discipline than just merely repeating the daily polls figures. It’s my hope that the Nate Silver effect, will be on the public’s critical response to polling and political commentary. In particular, the public’s awareness of margin of error, and perhaps their appetite for news to be accompanied with some discussion of uncertainty about their results.

Of course, the public, like the audience at the RGS, loves their mystical oracles. And quite a few questions focused on Nate’s opinion ranging from the next big US Election in 2014, to Labour’s performance if the General Election were held today, and to how we can live our everyday life better using statistics and modelling (Nate does not recommend creating “a spreadsheet of kebab shops every time you want to go out for a kebab”). Nate’s responses were quite even and unsensational, which is quite refreshing to an off-the-cuff ‘expert’. In fact, his team attempted to track the UK Elections, but things that are super difficult here is the multi-party complexity (just look at the UKIP’s upset this past week), as well as there are just less frequent and common political polling being done. In light of this, the two-party. poll-rich domain of the US elections look quite simple indeed.

And this brings us to a valuable question to consider in light of Nate’s talks… what makes a fertile domain to easily (and properly) model and explore?

How about these to start:

Rich and open-data environment – baseball is great with a long history of recorded statistics, while football is harder with long fluid game play.
Explicit performance metrics – sports, elections, and gambling have discrete winners and losers, while advertising effectiveness is more wiggly to identify and measure performance
Frequent measurement episodes – multiple daily polls measuring the public’s political opinion is far more frequent domain to study, than say than large-magnitude earthquakes.

Lawro vs. Data

Nate Silver – Royal Geographical Society