Calamitous poll should be seen as political entertainment not a prediction of the future

Calamitous, was how Owen Jones described the latest poll from ICM which gives the Conservative Party a 17 point lead over Labour.

This poll followed hot on the heels of a YouGov poll which suggested that the Tories had a 9-point lead over Labour. We need to do something and fast was the message from Owen Jones and members of the Parliamentary Labour Party. Did they over-react?

The two latest polls from ICM and YouGov are examples of why we should treat opinion polls as no more than a bit of political entertainment. Perhaps worthy of a headline, but in reality less capable of predicting the future than opinion pollsters would have us believe.
ICM claims a headline figure of 43% for the Conservatives with  Labour trailing on 26%, a Conservative lead of 17%, leading to headlines such as “The Conservatives' lead over Labour has widened to 17% — the second highest ever recorded” (http://uk.businessinsider.com/icm-poll-tory-lead-widens-to-17-over-labour-2016-10).

What this poll suggests is that, when asked, 43% of voters said they would vote Tory and only 26% said Labour. This poll followed a recent YouGov poll which suggested 39% of voters would vote Tory, compared to 30% for Labour. Calamitous indeed! If true. Fortunately, this is far from being the state of affairs at the moment. Labour is trailing with the public, but quite possibly not by the amount being suggested.

In fact, when “asked” by YouGov 27%, not the 39% reported, answered Conservative with 21% supporting Labour. The same question a couple of weeks later by ICM had 30% answering Conservative, not the 43% which prompted the headlines and 20% Labour. So, we might ask: how do they get these headline figures? The answer in a word: weighting.

The Independent with headline “Conservatives open up 17-point lead over Labour, according to new poll” (http://www.independent.co.uk/news/uk/politics/tory-labour-poll-conservatives-lead-17-points-theresa-may-jeremy-corbyn-icm-latest-a7354091.html ) does include the following explanation, from ICM’s Martin Boon, of why it could have been even worse for Labour: “Labour's share has only been saved from a record low by ICM's standard post-fieldwork adjustment techniques, which ordinarily help the Tories.”

Wow, thank goodness for those “standard post-fieldwork adjustment techniques” otherwise it would have been toast for Labour! The phrase used makes this sound very scientific. In fact, whilst these kind of weighting techniques are used to try to make unrepresentative samples more representative the 2015 Polling Report commissioned by the Polling Companies reached the following conclusion: “Our conclusion is that the primary cause of the polling miss in 2015 was unrepresentative samples. .. The statistical adjustment procedures applied to the raw data did not mitigate this basic problem to any notable degree.”

Historically, pollsters have tended to over-estimate Labour support. This is seen as a problem with their statistical models rather than a failure to get a representative sample in the first place. Given the polling companies failure to predict the 2015 General Election or the EU Referendum the statisticians have not simply sat back and allowed their models to continue being wrong. There is no doubt that they have been adjusting and re-adjusting their models. 

The problem is that until there is a General Election, which seems highly unlikely until after Brexit, that neither they nor we will know how effective their modelling has been. Based on the past, however, it is difficult not to feel that having plugged one gap in their model they will not have simply opened up another. If they get an election spot-on, that will be more to do with pure chance than the accuracy of their model, which will with alarming predictability be proved to be wrong in the next election. And, given that they know that they have been over-counting Labour, the answer surely, from their perspective, would be to adjust downwards for Labour and upwards for the Conservatives?

I happen to be of the opinion that unless there are absolutely compelling reasons to do otherwise that the results of surveys should be given based on the raw figures, and that the margin of error should be included to give a more accurate picture to those trying to interpret those figures. We often hear the margin of error reported as plus or minus 3%. Actually, more often we don’t hear this anymore, so that the headline poll figure is treated as the ‘real’ figure. What margin of error alerts us to is that unless you ask every single person in a population what they think about x then you have to accept that your  sample may not be representative. Assuming that the sample is random, then you have a known statistical chance of being within 3 percentage points of the correct answer. So if your sample suggests that 30% of the sample are going to vote Conservative, then the true figure is somewhere between 27 and 33%. The results should be, but never are, reported as a range any point of which could be correct. Neither, I should add, are most polling samples random.

These are the raw figures (and the margin of error range)  for the 2 recent polls:

	YouGov Poll 28-29 Sept	Range (+/- 3%)	ICM Poll 7-9 Oct	Range (+/- 3%)
Conservative	27	24-30	30	27-33
Labour	21	18-24	20	17-23
Liberal Democrat	6	3-9	6	3-9
UKIP	9	6-12	9	6-12
SNP/Plaid Cymru	4	1-7	4	1-7
Other	3	1-6	1	1-4
Would not vote	13	10-16	4	1-7
Don’t know/Undecided	18	15-21	22	19-25

There is a lot going on here, but what is worthy of note are the don’t knows and would not votes. We know, because we have the data, that turnout in General Elections has been less than 70% with the exception of 1992 and 1987 when it reached 78%. In 2015 only 66% of registered voters actually did so. However, one of the weighting techniques is to use data from those who say they don’t know and assign them to a party based on  the way they voted in 2015. But only 11% of the sample said they did not vote, another 4% would not or could not say how they voted. If that figure were true the turnout in 2015 would have been 85%. In other words, the weighting is based on either an over-estimation of those who say they voted (but did not) or on a sample which is disproportionately made up of people who do vote. If the latter, then the rationale behind the weighting is actually quite suspect. This is compounded somewhat by the 4% who say they would not vote in an election, which is way lower than the actual proportion of the electorate who do not vote in practice.

There are some complex statistical procedures at work in opinion polling and I am not suggesting that opinion pollsters are deliberately setting out to mislead the public. What I am suggesting is that, like economic forecasters, they cannot control every variable. They therefore use complex models to try to predict as best they can. That these models are occasionally right, does not make them anywhere near fullproof. Indeed, the fact that they are rarely correct two elections running is pretty sure proof that they are anything but foolproof.


Opinion polls should not be treated as facts. They are, at best, reasonable guesses of the way an election will turn out. At worst, they are complete misrepresentations and are more likely to be used to sway public opinion as to reflect it. I am not arguing that opinion polls, per se, are of no use, but that the headline figures, whilst entertaining, are not a true indication of the state of play of the parties at any time, but rather a set of figures based on some suspect, and constantly evolving statistical trickery. Now, no doubt, were a poll to be published tomorrow that gave Labour a 10% lead I would be as pleased as any other Labour supporter. But, would I believe it. It is often said, usually by those apparently doing badly in the polls, that the only polls that really matter are those which decide an election. I wouldn’t go that far, but given the apparent inability of pollsters to predict with any accuracy events in the future we might think that perhaps they would stop pretending that their polls are any more than snapshots and in this case snapshots which have been heavily photoshopped before being released to the public.