Having parked its [septic] tanks on Google’s lawn, the media’s efforts to generate a full blown moral panic around the existence of online pornography continued this morning with an appearance on the BBC’s Breakfast News programme by Susie Hargreaves, tbe current CEO of the Internet Watch Foundation.

To be honest, I wasn’t paying too much at the time but Hargreaves tripped my bullshit detector by casually claiming that the IWF had done some sort of research that apparently showed that 1.5 million people had accidentally run across child porn while surfing teh Interwebs.

1.5 MILLION..?

No… there’s something seriously screwy going on there but, of course, one should always check primary sources before dismissing dodgy-looking statistical claims out of hand, so off I trotted to the IWF’s website to find that they stuck out a quick statement a couple of days ago that includes the following claim:

“Our research revealed 1.5 million adults have stumbled across child sexual abuse content on the internet – but last year we received just under 40,000 reports.

Okay, so there’s the claim but where’s the actual evidence to back it up? Certainly not in this press release.

However, a bit more ferreting around on the IWF’s website did turn up another press release, dated 18 March 2013, which sheds rather more light on their “1.5 million adults” claim:

New study reveals child sexual abuse content as top online concern and potentially 1.5m adults have stumbled upon it

Hang on a second, we’ve just gone from “1.5 million adults have stumbled across” child porn to “potentially 1.5 million adults have stumbled upon it”, which rather starts to suggest that the IWF’s “study” might not be quite what they’re making it out to be and, sure enough, a little further down the page we hit paydirt:

The ComRes poll conducted among a representative sample of 2058 British adults for the Internet Watch Foundation (IWF) shows the vast majority of people in Britain think that child sexual abuse content (“child pornography”) (91%) and computer generated images or cartoons of child sexual abuse (85%) should be removed from the internet.

Riiiiight… so it’s not actually a study, it’s an opinion poll; a grade of evidence that generally sits just above the story you heard from a bloke down the pub who swears blind that his cousin’s boyfriend knows a bloke who knows the bloke that it actually happened to.

And it’s not just any old opinion poll but one that IWF have commissioned from ComRes.

The problem with opinion polls and surveys is that unless there are rigorously designed and tested to ensure before they’re used then they can be wide open to manipulation by variety of methods, the most obvious of which being the use of leading questions; questions are worded in a manner which will tend to influence survey respondents towards a particular answer, one which most favours the agenda of the person or organisation responsible for the poll.

Now, if you at all unsure how that works then this short video will explain everything you need to know:

[youtube]https://www.youtube.com/watch?v=G0ZZJXw4MTA[/youtube]

However, the downside to using leading question to cook opinion polls is that it’s all a bit obvious and easily spotted, especially if the poll is commissioned from polling company like ComRes, which is a member of the British Polling Council and, therefore, subject to its rules on disclosure and transparency, so there’s always the risk, these days, of a smartarse blogger downloading a copy of the survey questions/data and then publishing article that dismantles the poll and shows exactly how leading questions have been used to influence the results.

There are, however, a number of more subtle and less obvious ways of cooking an opinion poll, one of which is to tinker with the sample weightings on a poll so that instead of the poll being based on a genuinely representative sample of the overall population that you’re looking to survey, the sample used in the poll is actually skewed in favour of particular demographic groups within that population, typically those that are much more likely to give you the kind of answers you’re looking for than some other groups in the same overall sample.

If you look at ComRes’ website, you see that it has this to say on the subject of its approach to weighting the data in its public surveys in order to arrive as genuinely representative sample:

Data are weighted to the profile of all adults aged 18+ (including non telephone owning households). The weighting is undertaken by sex, age, social class, household tenure, work status, number of cars in the household and whether or not respondent has taken a foreign holiday in the last 3 years. Targets for the weighting are derived from the National Readership survey, a random probability survey comprising 34,000 random face-to-face interviews conducted annually.

So, based on that information, you would fully expect that the demographic characteristics of a weighted sample in a national opinion poll conducted by ComRes would very closely match would match the demographic profile from the National Readership Survey and that this, in turn, would be a close match to official demographic statistics published by the Office for National Statistics including, of course, those derived from the 2011 Census.

In other words, if the ONS  data shows that 51% of the British public are female, 20% are aged 65 or over and 10% live in rented social housing then the sample group in a ComRes national survey, after weighting, will contain as near as damn it the same proportion of women, older people and social housing tenants as makes no difference.

That’s the assumption but just about a month ago when I last looked in detail at ComRes national poll, one that had been commissioned by the Coalition for Marriage ahead of the local elections, I noticed that there appeared to be something just a bit ‘off’ about the age-group weighting in the poll – the graph below shows the extent to which the age-group weighting in that poll, which was supposed to representative of the population of Great Britain (i.e. England, Scotland and Wales) differs from the actual age profile of Great Britain recorded by the 2011 Census.

As you can see, the age profile in the ComRes sample was very heavily skewed towards older people with result people aged over 55, and particularly over 65, were heavily over-represented in the sample while those under 35 were heavily under-represented, all of which looks extremely suspicious when you consider that the poll was commissioned by an organisation which is opposed the legalisation of same-sex marriage and that the question in the poll asked people about their opinion on its legalisation and the impact that this likely to have on how they voted in the local elections.

It is no secret that, overall, there is a sizeable generational divide in attitudes towards many social issues, including  homosexuality, and the older people are much more likely to to be socially conservative in their overall outlook than younger people, so skewing the sample weighting heavily towards people aged 55 and over at the expense of people aged under 35 is highly likely to exaggerate the level of public opposition to the legalisation of same-sex marriage (and depress the level of support) reported by poll.

Having already found one poll in which the sample weighting appears to have been cooked in favour of an agenda promoted by the organisation that commissioned the poll from ComRes, I naturally decided to run the sample weightings in the IWF opinion poll through the same analysis, which generated the following graph showing, again, discrepancies in the age-group weightings.

Okay, so the pattern of discrepancies in this poll looks very different to those in the Coalition for Marriage poll but then, of course, we’re dealing a very different set of questions.

The main questions in the IWF poll ask for people’s opinions on whether certain types of notionally dangerous content should be removed from the Internet and the extent to which this content is perceived to be a cause for serious concern; giving a list of options that includes child pornography (real and cartoons/3d animations), terrorist websites, ‘hate’ websites that promote racism/homophobia, extreme/violent porn and website that ‘promote’ suicide and ‘encourage’ eating disorders.

The poll has, of course, been commissioned by an organisation whose primary line of business is tackling child pornography and so, quite obviously, it would be to the benefit of the IWF if the public saw this kind of material as being both their biggest concern and the type of content that they would most like to see removed from the internet.

And by a complete and wholly unexpected coincidence it turns out that the age-group weighting on the sample just happens to be most skewed in favour of people aged 35-44, the age group in which the people who were surveyed are most likely to be raising a family – the most recent data from the ONS shows that the average age at which women in the UK have their first child in now near enough 28 years of age while the average (mean) age of mothers for all births in 2011 was just shy of 30 years of age – although over 65s (i.e, grandparents) are also over-represented in the weighted sample to a much smaller degree.

In fact, if we drill right the way down the polling data to Table 27 we find that 58% of the people surveyed in the 35-44 group have children under 18, the highest figure for any age group, and that this same age group also has the largest number and proportion of adults with children aged 5-10 and 11-15, ages at which you’d expect parents to most concerned about protecting their offspring for ‘dangerous’ online content.

Yes, this same table does also show that the 25-34 age group, which is under-represented in ComRes’ weighted sample, had the next highest proportion of adults who have children (45%) but this distribution of children’s ages for adults in this group is heavily skewed towards under 5s and 5-10 year olds, who are much less less likely to be using the internet without parental supervision, where the skew in the 35-44 group is towards 5-10s and 11-15s, so you’d expect any under weighting in the 25-34 group to have much less of an impact on the poll results than an overweighting in the 35-44s.

So that’s now two polls we’ve looked at, conducted a mere 5-6 weeks apart by the same polling company, in which the weighted sample used in the poll is skewed in a quite specific manner, one that you would expect to generate exaggerated results that favour the primary agenda of the organisation that commissioned the poll.

Okay, that’s enough about dodgy sample weighting for now, let’s get back to the actual results of the polling where so far we’ve seen that the two main questions in the IWF poll are purely about eliciting opinions about types of content. It’s only when we get to question three that we find people were also asked whether they’d ever come into contact with or stumbled across any of the type of content listed in the other two questions, while they were online and it’s from this question that the IWF has derived it’s that 1.5 million people have, or perhaps only potentially have, run across online child pornography.

If we take the response to that question at face value then the figures tell us that: –

– 12.2% of adults have seen/encountered “very extreme/violent pornography”,

– 9.4% have seen/encountered “Hate websites, e.g. racism or homophobic”

–  6.3% have seen/encountered “Websites encouraging eating disorders”

– 4.5% have seen/encountered “Websites promoting suicide”

– 3.5% have seen/encountered “Computer generated images or cartoons of child sexual abuse”

– 3.4% have seen/encountered “Terrorist websites”, and

– 3% have seen/encountered “Child pornography”

74% of those surveyed had never seen any of the above types of content and 3% preferred to not say what they may or may not have seen online.

According to the 2011 Census the adult population of Great Britain is just over 48.1 million and 3% of that is a little under 1.43 million people, which the IWF has rounded up to 1.5 million (ignoring the usual rules on rounding) for its press releases.

That’s where the IWF 1.5 million figure comes from, but there are a few problems with this calculation.

For starters, the most recent ONS report on access to the Internet shows that 20% of British households currently have no internet access at all and 1 in 7 adults have never used a computer in their life, but as the ComRes poll didn’t include any questions about internet access or computer usage we have no idea whether or not that potential source of confounding is accurately reflected in their sample group.

If it isn’t and the sample contains a smaller proportion of people who either don’t have internet access or have never used a computer than the ONS’s figures then this will tend to exaggerate the polls findings and lead it to over estimate the number of people who claim to have seen the type of content listed in the question.

There there’s the problems of ambiguity and the lack of clear definitions in the question itself. In setting out the poll results (above) I’ve used the exact wording from the question for each of the categories and so, for example, 3.4% of the people in that survey claim to have ‘come into contact with or stumbled across’ a ‘terrorist website’ while surfing the Internet.

But what, exactly, is a ‘terrorist website’?

Well, in this particular poll, that’s entirely for the person answering the question to decide, which leaves us with a very big and wide open question as to just how reliable people’s perceptions of what is, and isn’t a ‘terrorist website’ might be, one that’s very nicely illustrated by the following screen shot from a Facebook account which shows members of the English Defence League expressing their surprise at the absolutely huge ‘mosque’ that’s been erected in Brighton without any seemingly having noticed:

Of course, the ‘mosque’ in the picture is actually the Royal Pavillion, which was built in three stages, starting in 1781, as a seaside retreat for the-then Prince of Wales, who went on to become Prince Regent and, eventually, George IV.

So, although 3.4% of the people in the survey claim to have encountered or stumbled across a ‘terrorist website’ it doesn’t automatically follow that everyone who made that claim has actually seen a terrorist website by whatever standards such things are judged by, say, the police or security services.

People’s personal perceptions of the content they say online can easily be wrong as can their subjective interpretation of terms such as ‘terrorist website’  or ‘encouraging eating disorders’, a fact of which the IWF should be fully aware given that their own figures for 2012 show that close to three quarters of the web pages reported to the organisation by the public, in the belief that they contained child pornography, were found by the IWF’s investigators not to contain child pornography.

Yes, that’s right. The IWF knows for a fact that the vast majority of ‘child porn’ reported to it by the general public isn’t actually child porn and it actively promotes the claim that 1.5 million adults have ‘encountered or stumbled across’ child porn online knowing that this figure is based on the assumption that no one who said they’d run across child porn in a dubiously constructed opinion could possibly be mistaken. This is just flat out dishonest. The organisation’s own data is telling it that the assumptions on which it’s 1.5 million claim is based are wrong and yet it still promotes that claim to the public as it were not only true but also as if it were the product of genuine research and not a number culled from a crappy and extremely amibiguous opinion poll.

In short, it’s bullshit.