Yesterday, the chairman of the polling organisation ComRes, Andrew Hawkins, was busily touting a local election poll commissioned by the Coalition for Marriage on Twitter, the headline figures for which were as follows:

Conservative 31%, Labour 24%, Liberal Democrat 12%, UKIP 22%, Other 10%

Okay, so those figures look rather different to current national polling but that’s hardly surprising as this is primarily a Shire counties election in which you’d expect to see Labour, in particular, running behind the national figures because so much of the party’s electoral support is centred on urban areas where there are no local elections this year.

Nevertheless, the figures still looked a little odd to me, so I took a look at the data and then ended up having the following exchange of views with Andrew via Twitter:

https://storify.com/Unity_MoT/comres-c4m-poll

So, it’s fair to say that I wasn’t exactly convinced by Andrew’s response, which prompted me to dig a bit further into the polling data for any possible sources of bias or inaccuracy and what I found is, well, rather interesting.

For starters, a methodology note on the poll gives the following information:

ComRes interviewed 1,502 adults in areas of England where local election are taking place on 2nd May. Interviews were done online between 24th and 28th April 2013. Data were weighted to be demographically representative of all GB adults.

But hang on a second – as I’ve already noted this is primarily a Shire county election – the only exception being Bristol – so why weight the data using national demographic profiles when these will undoubtedly be different to the demographic profiles of the areas in which the elections are taking place?

ComRes, like other polling organisation, does not disclose exactly how it weights its polling or exactly which demographic factors it takes into account, although it is clear from the data that they do release that at the very least their polls are weighted by gender, age profile and social class.

So let’s look at two of those metrics in more detail, starting with age, for which I’ve generated the following graph which compares the age profiles of Great Britain (i.e. England, Scotland and Wales but not Northern Ireland) and just those areas where elections are taking place this year (both based on data from the 2011 census) and with both the unweighted and weighted age profiles of poll respondents in this particular ComRes poll:

Well, it certainly seems that we have a few issues here.

What this graph confirms, straight away, is that the age profile in the areas where elections are being held this year is skewed towards older people compared to the GB age profile for adults (over 45s up 2.69 percentage points, under 45s down by the same amount) which is no more than you’d expect for Shire counties election.

But look carefully at the ComRes samples – even after the sample has been weighted, ostensibly to be a nationally representative sample there’s an even more marked skew in the distribution toward older people. Compared to the GB distribution, Over 45s are over-represented in the ComRes weighted sample by 9.84 percentage points, most of which is due to the over 65’s group, which is 8.07 percentage points up on the national figure.

A comparison to the actual distribution for election areas is admittedly a little less out of kilter – over 45s are only over-represented there by 5.45 percentage points – but that’s still a ‘swing’ towards older voters in the polls of close to 11 percentage points.

I also took a look at ComRes’s sample weighting for social class using the NRS 2010 figures, which produced this graph:

Again, the ComRes weighted sample are off-distribution, particular in regards to the C1 class (clerical/admin & junior managerial) which is under represented by around 3.5 percentage points, giving us a distribution that places more emphasis on the AB and C2 classes than should be the case.

This is just one poll, of course, and further work is needed to confirm whether or not these apparent discrepancies in  the age and social class distributions operate consistently across other ComRes polls, but if that is the case then the general tendency for older people to be more conservative in their outlook on a wide-range of political, economic and issues (and younger people a more liberal outlook) will mean that ComRes’s polling data is likely to have a fairly substantial and in-built conservative bias across the board.

Little wonder then, that ComRes has become the polling organisation of choice for right-wing Christian organisations across a range of social issues from same-sex marriage to abortion.

The upshot of all this is, of course, that even before we come on to the bit of dubious jiggery-pokery surrounding voting intentions on this particular poll, the off-kilter weighting of the sample group will tend to favour right-wing parties (Tory & UKIP) and over estimate their overall level of support while under-estimating support for to left-wing/liberal parties, particularly the Lib Dems and Greens.

So, getting back to the issue that started all this off in the first place, what I noticed when I first looked at the data was that although ComRes initially surveyed 1502 people, when it came to figures for voting intentions as these local elections, the unweighted sample size has shrunk to just 782 – so where had the rest of the original sample disappeared to?

What ComRes did was include a question asking people whether they actually intended to vote in the election, with responses measured on 10 point scale (10 = absolutely certain to vote, 1 = certain not to vote) after which they took only the people who said they were certain to vote as the basis for their voting intention question.

Well, actually, they took almost everybody who said they were certain to vote – 828 people actually said they were certain to vote but the unweighted base for the voting intentions question is given as just 786 people, so about 5% of the data appears to have gone walkabout even before ComRes got around to weighting the sample to make ‘nationally representative (or not, as we’ve already seen).

Is this missing 5% the ‘certain to vote but don’t know who for yet’ vote? We don’t know as ComRes don’t say, they just disappeared into the aether.

The upshot of all this is that, after weighting, ComRes’s voting intentions figures are based on a weighted sample of just 765 people, or just 51.3% of weighted sample used on the other survey questions, all of whom said they were absolutely certain to vote. 80-90% certain to vote, a group which accounts for a further 16% of the original sample group, just wasn’t good enough to make the grade even though its likely that some of those people will actually vote just as some of those who said they were certain to vote won’t actually turn up on the day.

In fact, rather a lot of those people who said they were absolutely going to vote almost certainly won’t because the last time most of these councils were contested, in 2009, the actual turnout was a little over 39% – all of which means, of course, that if the sample is genuinely representative of voters, around a quarter of those who said they were certain to vote won’t get around to darkening the doors of a polling station today.

Now you should be able to see why, when Andrew Hawkins tried to justify the decision to limit polling on voting intention to just those people who’d said they were certain to vote based on a spurious looking comparison to voting behaviour at general election, I replied by pointing out that he had absolutely no evidence whatsoever to support that kind of assumption.

So where does that leave us with this poll?

Well, for starters with the suspicion that the figures for UKIP, in particular, may have been inflated by the weighting issues and the preponderance of older people in the ComRes sample group but beyond that, who knows for sure?

With what could easily be a 25% discrepancy between the number of people who said they were absolutely certain to vote in the poll and the likely turnout at the election, the final outcome of the election in terms of vote share seems likely to depend, as is so often the case, on the ability of the various parties, at a local level, to get their voters out on the day.