Archive for the 'Pollsters/polling' Category


Nick Sparrow, the pollster who did most to change post-1992, on poll averaging, herding and the pressure to conform

Sunday, July 5th, 2015


Why Polls End Up Saying The Same Thing

Following the General Election, the pollsters have been accused of having herd instincts.  How else do so many polling companies, acting independently, get to the same – wrong – answer?

In the final days of the campaign, the polls mainly agreed on the likely outcome, and even a late movement to Labour.  Polls of polls ironed out small differences and gave an even greater feeling of certainty.  But the natural belief that the average of independent observations is likely to be most accurate does not apply to vote intention polls.  Almost all the final polls in all general elections since the Second World War show bias and not error.  Put simply, they almost always err in one direction or the other, mainly underestimating the Conservatives.  In short, beware the average, it is only better than the worst and worse than the best.

    Nevertheless, apart from a few days, or at the most weeks after a general election, pollsters are judged by media commentators mainly on the proximity of their predictions to the average, whether that average is calculated or more vaguely expected.  That pressure is steady and, as polling day approaches, increasing. 

    A pollster with results diverging from the average will be asked by their client and others to examine every aspect of the methods for anything that might be “wrong”.  A pollster with results on the average can relax.

Those soft but sustained pressures, over the years, will tend to give greater prominence to those perfectly justifiable methods that tend to lead in the direction of conformity, and less attention may be paid to methods that lead to a greater degree of divergence.  So, the average is not only where the pollsters feel most comfortable, clients and political commentators believe the average is likely to be most right.

However, the pressure to conform to the average of the polls in turn restricts the tone of political commentary.  Common sense might have told us that the Conservatives would do very well in the General Election.   Nowadays it is more similar to a presidential election, with decisions by ordinary voters based only or primarily on the look of the leader, his aspirations for Britain, goals and ambitions.  Cameron vs Miliband was a mismatch.  Inasmuch as party and policy matter, Old Labour was so last century; the policy proposals lacking resonance in modern Britain.  The polls did not have the right smell about them.  Why did so few say so at the time?

Rather than herd instinct, the process by which pollsters and commentators influence each other may be better described as an informational cascade.  Over the long term, the publication of vote intention polls adds to the expectation of what any new poll will predict, sometimes irrespective of any other signals pointing in a different direction.  The theory would suggest that the publication of vote intention polls, strongly promoted as being reliable by the media owners who pay for them, suggesting certainty both for the overall prediction as well as small fluctuations, can rapidly influence a much larger group to accept the likelihood of a particular outcome.  At some point, the theory goes, any person with a correct prediction (however it is obtained) can be convinced, through social pressure, to adopt an alternative and incorrect view of the likely outcome.

Following a 1992 sized polling debacle, pollsters now need to take a hard look at the methods. Still relevant are the recommendations made by the Market Research Society in the report published after 1992:

“We would encourage methodological pluralism; as long as we cannot be certain which techniques are best, uniformity must be a millstone – a danger signal rather than an indication of health.  We should applaud diversity; in a progressive industry experimentation is a means of development.  No pollster should feel the need to be defensive about responsible attempts to explore in a new direction …”

Now that is a lot easier to suggest than to do.  Between 1992 and 1997 I changed from quota face-to-face interviewing to random telephone polls (“you can’t do that not everyone has a telephone”) started weighting by past voting (“you can’t do that, people imagine they voted for the party they now support – Hemmelweit et al”) and adjusted for the likely votes of those who could not or would not say who they would vote for (“you are making up the answers”).

Defiantly, and with the backing of The Guardian, as the General Election in 1997 approached I produced very different predictions to the rest, and in the process had my ear well and truly bent by many political commentators who had come to believe the average of the polls, most of which used methods in 1997 unchanged from 1992.

As it turned out the ICM prediction was most accurate, but in the run up to polling day the pressure to adopt the alternative, less accurate average of the rest, was intense.

Now, as then, pollsters should be seeking new solutions, and be unafraid of producing results very different to each other.  The average is clearly not to be trusted.  Sadly, I suggest, the likelihood is that come 2020 both pollsters and political commentators will again be converging on the average.

Nick Sparrow – former head of polling at ICM


How the Alternative Vote system could stop Burnham becoming Labour leader

Wednesday, July 1st, 2015

Many thanks to Richard Nabavi for posting this article by Peter Kellner on the previous thread.

Peter Kellner looks at how the Alternative Vote system Labour use to elect their leader might stop Andy Burnham winning, it should be remembered, that this voting system helped Ed Miliband defeat his brother five years ago.

If you’re not sure how the Alternative Vote system works, this link should help as should this link.

However the betting sentiment is moving strongly towards Andy Burnham.

Ten days ago, I wrote that we were close to a potential crossover on the Betfair exchange between Burnham and Cooper, how the latest trade shows Burnham’s implied percentage chance of winning the leadership has gone from 39% on the 21st of June to 50% this evening, whilst Yvette Cooper’s has fallen from 36% to 30% in the same period.





The Ipsos Mori issues index for June

Wednesday, July 1st, 2015



The big risers are Immigration/Immigrants and EU/Europe, which seems understandable given the focus on the EU referendum since the election. The big faller is the economy, which maybe confirmation of the fifteen year high in consumer confidence that the pollster GfK found yesterday.

For me the most interesting aspect of this polling is the below chart.

Issues EU

There’s a real difference between the ages, so the older groups are more concerned by the EU than younger ones, this as has been noted before, could help OUT win the referendum with older voters the most likely to vote.

The fieldwork for this polling ended on the 15th of June, so before the recent events in Greece and Tunisia.

The data tables are available here.



Guest Slot: Social media and shy rightwingers

Friday, June 26th, 2015

Dan Hannan Tweet

Tissue Price on the polling errors across Europe

The inquest into the polling disaster at the UK General Election continues. Matt Singh of NumberCruncherPolitics provided an excellent overview of the pollsters’ initial thoughts last week, ahead of the first meeting of the official BPC/MRS inquiry.

Some pollsters think faulty sampling was the principal cause of error, some blame turnout modelling, and one thinks a genuine late swing was the biggest single factor.

Dan Hodges channels Emile Zola in accusing the pollsters of herding, and Danny Finkelstein (£) thinks we’ll never know the true answer.

However Matt’s bet – and mine – is that Peter Kellner is right and that 2015 was a classic case of shy Tory syndrome. Peter chiefly attributes this to the Tories’ image, but I wonder whether his earlier explanation of “social satisficing” – not wanting to admit your views to a stranger for fear of being thought less of – might be nearer the mark, notwithstanding the fact that the majority of UK polls were online.

The reason for thinking this is that similar polling errors have occurred in other national elections this year. In Israel, Likud were predicted to gain 22 seats (of 120) and ended up with 30, and last week in Denmark the blue block were expected to win by 1 or 2% and actually won by 5% – with the populist DPP notably outperforming their eve-of-election polling by 3% (21% to 18%).

On more limited polling, the same pattern can be seen in Finland – with the Centre Party overestimated by about 3% at the expense of the populist True Finns and centre-right National Coalition Party; in Estonia, where the winning centre-right Reform Party were underestimated; in the Croatian presidential election, where the polls didn’t give the narrow winner Kolinda Grabar-Kitarović much of a chance (though interestingly the exit polls nailed it); and in Poland’s presidential election, where Andrzej Duda’s first round victory came as a total shock.

You could even make a case that Ireland’s marriage referendum fits the pattern, with the 62-38 victory for Yes contrasting with opinion polling expecting a 70-30 result.

The social media explanation

Is there a ready (and no doubt oversimplistic) explanation for why people all over Europe, in a variety of elections, might have been conditioned into suppressing their true intentions – even online? I think that perhaps social media – Twitter, and more importantly Facebook – has the answer.

Twitter has long been described as an echo chamber, and undoubtedly has a leftwing bias in terms of the sheer number of tweets. Dan Hannan’s tongue-in-cheek Venn diagram (at the head of this article) is to the point; the FT provided a more aesthetically pleasing proof of the same effect with some very nice network graphs.

However on Twitter you can choose who you follow and what you are exposed to. On Facebook you have to put up with your friends’ opinions. Now I am a fully paid-up PB Tory, with a social circle to match, but even I have some leftwing friends. And they didn’t shy away from signalling their virtue!

That’s anecdotal, but here is some data from the British Election Study posted by Philip Cowley of Nottingham University which confirms that left-wingers were much more likely to post content online during the election campaign:


Bes 2

NB the wider reach of Facebook – it’s by far the more important social network for communicating with the electorate at large. Ofcom estimate that there are about 35m Facebook users in the UK and only 12m Twitter users.

My supposition is that it’s easier – as in, less risk of argument or confrontation – to post left-wing opinion online. Your intentions are assumed to be good and your motives pure, whereas right-wing opinion may often carry a whiff of self-interest in financial matters and might be supposed to be xenophobic or worse in other areas.

So, there’s my overarching theory to explain the multiple failures of polling across Europe this year: an online culture in which leftwing messages get disproportionately liked or retweeted into your timeline might have helped to bring about the emergence of shy Tories. What do you think?

Tissue Price


David Miliband won the 2010 LAB members ballot by 8.8% – the final poll had had him 4% behind

Saturday, June 20th, 2015
YouGov LAB members poll Sept ’10 YouGov 1st round Actual 1st round YouGov Miliband preference Final votes
ABBOTT, Diane 11 7.3 0 0
BALLS, Ed 9 10.1 0 0
BURNHAM, Andy 10 8.6 0 0
MILIBAND, David 38 44.1 48 54.4
MILIBAND, Ed 31 29.9 52 45.6

Left wing contenders were overstated, those of the right understated

Judging by what some LAB members have been Tweeting it looks as though, maybe even tonight, we’ll see the first poll, from YouGov, of those who will be able to vote in Labour’s leadership election
at the end of August.

Is Burnham’s favourite status justified and how will the critical second and third preferences split? Maybe that will be clearer.

In the past YouGov has made leaders’ ballot a speciality after building up data on party and union membership amongst members of its polling panel . It got the Cameron-Davis battle of 2005 right to within 1%.

Last time with LAB, when EdM got elected in September 2010, a third of the electoral college was made up of the views of members and it was the trade union third that thwarted David Miliband and allowed Ed to squeeze past. This time the vast bulk of the electorate will be LAB members or those who have registered as supporters.

The above table looks at how YouGov did in its final members’ survey which was carried out about 2 weeks before voting closed in the 2010 leadership race.

To deal with the fact that this was an election under AV YouGov asked a final question as to which of the two Miliband brothers people would vote for. The figures are in the table.

As can be seen the more left wing candidates tended to be overstated and that to the final “which Miliband” question David won the members part by 8.8% against the 4% poll lead for Ed.

Will the same happen this time? Who knows but it is worth bearing in mind.

Mike Smithson


Ipsos Mori finds support at a 24 year high for remaining in the EU

Friday, June 19th, 2015

75% would vote to remain to stay in the EU whilst 25% would vote to leave

Ipsos Mori 40 year

One of the reasons I like the Ipsos Mori polling on the EU, is that they’ve been polling on the topic for nearly forty years, they have another poll out today for the the Evening Standard.

If the historic in-out referendum were to be staged now, 66 per cent say they would vote to remain members and 22  per cent would vote to quit. Excluding the don’t knows, at 12 per cent, the result is an emphatic 75 to 25.

It comes amid a rising tempo of referendum preparations. As the Prime Minister held talks with his Slovak counterpart ahead of a Brussels summit next week, seven Eurosceptic MPs from Labour, the Tories and Ukip announced moves to form the Out campaign….


….The survey for the Evening Standard used the exact wording expected to be on voting slips in the poll, due to be held by the end of 2017. In addition, half the 1,005-strong sample was asked a second question with a wording used on Ipsos MORI surveys over four decades: “If there were a referendum now on whether Britain should stay in or get out of the European Union, how would you vote?”

It found another huge majority to stay in, of 61 per cent to 27. That included 63 per cent of Conservatives and 76 per cent of Labour backers who want to stay. Almost all Ukip supporters would leave, however.

The caveats I’d emphasise are, the Tories and Cameron in particular are enjoying a post election boost, Cameron has his best leader ratings since 2010. Secondly, UKIP are on 7% in this poll, which is half of what they polled at the general election, so the OUT movement maybe under-represented in this polling which has fed into this supplementary. That said, even if we up-weighted UKIP by 100%, remaining in the EU would still have a substantial lead.

Because of the honeymoon, the proportion of respondents who are either very or fairly confident that Cameron will get a good deal in his negotiations with European leaders have increased by 12% to 38% since last November, whilst those not confident in the Prime Minister’s ability to get a good deal has fallen by 12% to 57%, this is probably another driver in the increase of voters wanting to remain in the EU.

Ipsos Mori Back Dave#

It should be remembered, ICM, the most accurate pollster when it came to the AV referendum, a year before the referendum, had AV winning 62% to 38%, so things can change and IN should not be complacent nor should Out be disheartened by just one poll.

It isn’t all good news for Cameron, 74% of the voters want ministers free to campaign against the official government position when it comes to the referendum.

The full data tables are available here.



Polling Matters / Political Betting Podcast with James Morris – EdM’s private pollster

Tuesday, June 16th, 2015

This week Polling Matters with Keiran Pedley speaks to James Morris of GQR – Ed Miliband’s private pollster during the 2015 General Election and puts to him some questions raised by PBers.

Topics covered include the following (and more):

1) How Labour’s internal polling differed from public polls?
2) What Labour’s polling did (and did not) tell the party and when?
3) Did Ed Miliband really think he was going to win?
4) Did Labour know Ed Balls was going to lose?
5) Where does Labour go from here?
6) What can GQR polling tell the BPC inquiry?
7) Will they be releasing data?

Keiran Pedley is an Associate Director at GfK NOP and presenter of the podcast ‘Polling Matters’. He tweets about politics and polling at @keiranpedley


Polling Matters/Political Betting podcast: Your questions needed

Monday, June 15th, 2015

This week Keiran will be interviewing James Morris on polling, Labour and the General Election campaign. As most will know, James was Ed Miliband’s pollster during the recent election.

Keiran will be taking questions from social media and putting them to James. If you have a question please post in the comments box. I am sure it goes without saying, keep them sensible and constructive if you want them read out.

The episode is being recorded on Tuesday and should be available Wednesday morning.

Keiran Pedley is an Associate Director at GfK NOP and presenter of the podcast ‘Polling Matters’. He tweets about politics and polling at@keiranpedley.