Elections are a lot like sport — everyone wants to pick the winner. On one side, there are eager fans who scope out commentaries and inside stories from journalists. And on the other side, there are betting markets who use predictive modelling and experts to bias the winner. However, there is one important source that we are all rely on (to place our bet) before an election and that is the political polls.
Political polls claim to know the intent of voters before an election occurs, but with the recent Brexit surprise and the Australian federal election only days away, how much credence should we put into the polls?
Background to polling
In Australia we have a number of polling companies that have been providing election polls for more than 20 years (Newspoll, Morgan and Nielsen). As well as some newcomers (Galaxy, ReachTEL and Esssential).
These polls operate on the principle that if you survey enough people, you can infer the responses for the remainder of the population, within a margin of error. A survey of 1000 people will yield a margin of error of roughly 3%. For instance, if 50% of people polled prefer Party A, there is a 95% chance that the ‘true’ population result lies between 47% and 53%. Australian pollsters typically work within a range of 1500 and 3000, meaning that the margin of error should fall between 1.8 % and 2.5%¹.
Each pollster claims to be the source for public opinions, but which one comes out on top?
The best polls
I analysed historical polling data to find out who was the most accurate pollster. The table below shows which polling company has the lowest average error in Two Party Preferred² polls in the week before an election.
Overall, accuracy in elections between 2007 and 2013 varies between 0.4-1.2% (outside of Nielsen, marred by a particularly bad 2007 poll). Newspoll tops the list with an average accuracy of 0.43 percent.
This seems to be an acceptable level. If you get your information from Newspoll or Morgan, it is well below the margin of error. However, is this type of polling really useful? Polls are often spread across the media months, not days before an election date.
Polling accuracy ≠ prediction accuracy
Google trends shows us that interest in the polls picked up from 6 months prior to the last election in 2013. If we are going to use these polls, to base our decisions and flood our media, then lets find out how predictive they are.
Polling prediction accuracy in the months prior to an election. Data smoothed using loess
Naturally, polls are more accurate at predicting election outcomes when they are conducted near the election date. Across the pollsters, accuracy ranges from <1 percent error in the week before the election to as much as 6 percent error 9 months away.
Based on all polls across the last 5 federal elections, only 44% of the time was a poll within a 2.5% margin of error 3 months out from the election and only 50% of the time, 6 months out.
The ‘margin of prediction’
Of course, polls do not attest to being able to predict the election, merely tell us the likely result, if an election were to occur that day. But given no other knowledge, we naturally translate this into future outcomes. This prompted me to create a new ‘margin of prediction’ metric for polls. Like the margin of error provides a likely range for the population political preferences, the margin of prediction provides a likely range for future election results.
What this means is that 6 months out from an election, a Two Party Preferred polling result of 55% indicates the final election result is 95% likely to fall between 47% and 62%⁴. Unfortunately, this is hardly enough to provide any strong indication of who will actually win the election
Based on historical records, it seems that polls are a relatively reliable source of polling prediction, however, in the lead-up to an election and in the midst of election fever, you may be best off ignoring them.
- The margin of error assumes a normal distribution and a large population size. Finer adjustments also incorporate the polling result; with relatively smaller error as the result moves away from 50%. Handy calculator here
- Two party preferred polling is not taken directly, but inferred from major party preferences. This is calculated based on the preference flow in the previous election. http://blogs.abc.net.au/antonygreen/2016/04/a-few-thoughts-on-this-mornings-newpoll.html
- PhantomTrend has also collected data on all federal election polling going back to 1998 for 2PP and major parties.
- Prediction results may vary. Certain quirks of these elections (e.g., the close 2010 election or the recent swing to the Coalition after Tony Abbott was overthrown) combined with the small sample size mean these results may not extend to other elections/polls.