Today’s Observer is running a story on research by Citizens Advice which, more than anything else, serves to demonstrate just how easily statistical information can be abused to present an alarming but potentially inaccurate picture of reality…

Domestic abuse: half a million victims ‘too terrified to come forward’

Study uncovers hidden casualties of violence, many of them too scared to report the crime

More than half a million victims of domestic abuse are too terrified to come forward and report their experiences, according to provisional estimates of research that specialists hope will quantify the true extent of the crime.

Citizens Advice has conducted research over the last year that reveals a higher level of abuse than previously reported. A pilot project in nine areas across the UK, in which clients were asked a series of routine questions when seeking help with issues such as debt and housing problems, found that 27% had experienced domestic abuse at some time since the age of 16 – three percentage points more than the national average reported for all women in the latest crime survey for England and Wales.

If extrapolated across the UK, the specialists believe that could mean there are up to 540,000 more victims of domestic abuse, the vast majority of whom would have stayed silent, according to Citizens Advice, which now plans to roll out the scheme service wide

There are, I’m afraid. a couple of problems here which require a bit of clarification, so to begin with let’s look at the the long term trend in the annual incidence of domestic violence as recorded by the British Crime Survey/Crime Survey for England and Wales since 1981.

Okay, so the solid blue line shows the recorded figures and the black dashed line the overall trend, and as you can clearly see we have a big rise in the incidence of domestic violence through the 1980s and into the early 90s – although how much of that apparent increase is a function of changing attitudes leading to women (and men, of course) being more willing to admit to having been victimised is anyone’s guess. However since the mid 1990’s we’ve seen a substantial fall in the annual incidence, albeit one that’s arrested over the last 5 years or so. Nevertheless, compared to the early-mid 1990s the annual incidence of domestic violence is down by a little over 60%, which is good news even it does that there are still far too many people being victimised by an intimate partner.

So, problem no. 1 is simply that Citizens Advice are using an estimate for for lifetime prevalence (i.e. victimisation since the age of 16) and annual levels of domestic were substantially higher in the past this will inevitably present a distorted picture of the present situation if you take its figures to indicate current levels of domestic violence, which I suspect many of the Observer’s readers will.

Now, I should stress that this doesn’t mean that lifetime prevalence estimates aren’t important and don’t have their uses. Domestic abuse can easily have long term consequences, particularly in terms of it’s impact on victim’s mental health, so if your in the business of planning local mental health service provision then these estimates are going to be extremely useful.

On the other hand, if you’re planning crisis intervention services to support people who are being victimised now and not those who were victimised maybe 20 years ago but have since got out of that situation, which appears to be the direction that Citizens Advice is going, then its the annual incidence figures and year on years trends that you should be basing your work on as these will give you a much more reliable estimate of your potential current case-load.

Lifetime prevalence estimates are not a reliable guide to current trends.

Problem no.2 lies in the claim that Citizens Advice have uncovered more than 500,000 ‘hidden’ victims of domestic abuse based on a comparison of their own estimate of lifetime prevalence (27%) with that recorded in the most recent BCS/CSEW intimate violence report (2011/12) of 24.4% – and for the record the figures by gender are 31% women, 17.6% men.

What is being completely ignored here is that both the CSEW estimate and that of Citizens Advice are based on surveys of a sample of the British public and as with any such exercise the published figures are subject to a margin of error.

So what kind of margins of error are we looking at here?

As yet, I’ve not been able to track down any figures for the CSEW lifetime prevalence estimates but for its most recent annual incidence figure (0.403% – this is the mid range figure) the 95% confidence interval range runs from 0.328% to 0.483% which, when translated in estimated numbers of victims, gives us a range which spans from 293,000 victims at the bottom end to 502,000 victims at the top end and a mid range figure of 398,000.

If – and we can’t be entirely sure of this without seeing the proper figures – the CSEW lifetime prevalence figures are subject to same margin of error then the effective range for it’s mid-range estimate of 24.4% would run from 19.8% at the bottom end to 29.2% at the top, and the Citizens Advice estimate of 27% is well within that top end range.

Let’s not forget, of course, that the Citizens Advice figure is also going to be subject to margin of error, but without sight of their actual figures there is currently no way of knowing whether this 27% figure is also a mid-range estimate, like the CSEW one, or a top end estimate – and campaigning charities have a very bad habit of using top end figure in their press releases because the biggest numbers attract the most media attention – nor what the margin of error is on their figures nor even whether their figures are based on a nationally representative sample, or have been adjusted to make them representative.

In short, the 500,000+ women that the article claims are ‘too terrified to report their experiences’ may quite easily be nothing more than a statistical artefact arising from a combination of sampling error and methodological differences between the CSEW survey and that conducted by Citizens Advice, but without sight of the actual Citizens Advice research it is impossible to be sure one way or another.

In short, without further detailed information from Citizens Advice to validate its research, we’re going to have to flag this one as another potential case of statistic abuse perpetrated by a charity which undoubtedly means well but nevertheless may be resorting to bullshit to promote its services.

Sorry…

As a final thought, if anyone at the CAB would care to send me their detailed figures (use the email link) I’d be quite happy to check them over and amend this article if/as necessary.