It’s been a while, but let’s return to the subject of the DWP’s trial of a so-called ‘voice risk analysis’ system and bring you all up to speed on what’s been happening since my last post on this particular subject.

At the beginning of May, this year, I submitted requests to each of the twenty-four local authorities taking part in the second phase of the trial, asking for a range of information on the performance of their trial systems, including the information necessary to assess how effectively, or otherwise, these systems were working.

Of these local authorities, six have failed to provide a response with the relevant statutory period of 20 working days, including two of the original first phase trial sites, Lambeth and Harrow.

A further twelve authorities in England and Wales claimed exemptions under s22 of FOIA (i.e. that the information requested is intended for publication at a later date) and all three Scottish local authorities contacted (Edinburgh, Glasgow and Aberdeen) also claimed exemptions under either s27 of Scotland’s FOIA (which is identical to s22 in the English/Welsh Act) or s30 (i.e. that the release of information would substantially prejudice the effective conduct of public affairs).

One local authority cried off on the grounds that my request would require additional work with the resulting costs to the authority exceeding the statutory maximum of £450, which is altogether a curious response as the information I asked for is nothing more than the monitoring data that other local authorities have been routinely collecting for the DWP.

This leaves two local authorities, one of which provided the data I requested, albeit not in the precise format requested as a result of their officer not quite interpreting my question correctly – nevertheless, credit where its due, they did respond positively and the information they sent has proved useful – and one local authority which complied with my request in full and without any quibbles of dissembling.

Needless to say, most of the authorities contacted will shortly be receiving letters complaining of their failure to respond within the proper time or appealing their decision to apply an exemption under FOIA.

The purpose of these requested was twofold.

First – and quite obviously – the requests for data on the current phase of the trial were intended to elicit information on the trial’s progress and the performance of the test systems.

Second, these requests were also submitted in the hope of shaking loose additional information that might provide greater insight into how these trials are being conducted and the methodology being by the DWP, about which I have some considerable concerns…

…and it’s precisely those concerns I wish to address in this article.

Let’s start with what is by far the most significant methodological flaw in the trial, one that the DWP appear to have completely overlooked.

First, it’s vitally important to understand one key fact – the ‘voice risk analysis’ system that the DWP is currently testing is NOT a single, unitary system, but one which combines two very different approaches to deception detection in what has been sold to the DWP as an integrated system.

So, the VRA system consists of two discreet components.

One of these is Nemesysco’s ‘Layered Voice Analysis’ software which, if we’re being extremely charitable, is best described as scientifically implausible and lacks any credible scientific evidence to support any of the claims made for its capabilities by its developer. It’s sole verifiable ‘benefit’ as a means of triaging benefits claims resides in its ability to engender the ‘bogus pipeline effect’ in some claimants, although this is of very limited value given that this effect works only if the claimant genuinely believes that the software actually works.

The second component is what the DWP refer to on their website as ‘behavioural analysis and conversation management techniques’. These are, in simple terms, a variety of psychologically-derived methods and approaches to the questioning of claimants about the information supplied in support of their claim that, in the hands of a skilled operative, may reveal flaws and inconsistencies in claimant’s statements of a kind that may indicate that they are acting in a less than honest manner. Unlike Nemesysco’s software-based ‘lie detector’ these techniques and practices are derived from fairly solid body of both theory and research evidence accumulated by psychologists over the last 20-25 years, primarily in relation to studies of the accuracy of witness testimony in a law enforcement context and there is evidence to show that, when used by a suitably experienced investigator, they can tip the odd in the investigator’s favour, even if these practices do not provide a foolproof method of detecting deception.

Superficially, combining these two approaches may seem to offer a reasonable prospect of tipping the odds of detecting deception even further in the investigator’s favour – providing both approaches actually work as claimed – but this ignores the fact that if Nemesysco’s software-based system were genuinely capable of living up to the claims made for it by its developer, in terms of detecting stress, deception and a wide range of other cognitive states, it would inevitably render the behavioural analysis and conversation management aspect of the combined system functionally redundant.

There is, however, another much more compelling reason for combining these two approaches – at least from Capita’s point of view.

Nemesysco’s software may well be scientifically worthless but it is, nevertheless, covered by an active patent and, therefore, of considerable potential commercial value should clients, such as the DWP, fall for the marketing hype surrounding it.

By way of a complete contrast, the psychological techniques deployed as backup to the software in the VRA system that the DWP is trialling, possess some established scientific value – there is evidence to show that these practices can elicit useful information and expose flaws in testimony of a kind that indicates that further investigation may be worthwhile. However, both the theory and research evidence that underpins these techniques have been extensively published in peer-reviewed scientific journals over last 25 years to the extent that not only would a patent application covering them be unlikely to succeed on the basis of prior art but these techniques could easily be replicated by commercial competitors using the information already available in the public domain.

Only by combining these two systems into a single package can Nemesysco’s UK licensee, Digilog UK, and its key strategic partner, Capita, close out the competition completely – remembering, of course, that Digilog UK has an exclusive licence to sell Nemesysco’s system in the UK and Ireland and Capita has a similar exclusive arrangement with Digilog UK which covers the entire public sector market and runs until 2014.

Combining the two systems together into a single package creates a very specific methodological issue that needs to be carefully controlled for when conducting trials in order to assess the effectiveness of the system, if the trials are not to produce a misleading of results. If the trial methodology is not very carefully designed to take into account the fact that what is being tested are two different deception detection systems operating in tandem then, when one looks at the final results of the trial, it will be impossible to determine whether and to what extent each individual component may, or may not, have contributed to the outcome of the trial.

One of these two systems may very well be entirely worthless and operate, in practice, at no better than chance, but if the trial methodology does not include appropriate controls then the failure of one of the components to perform in line with claims made for it by its developer may very well not be apparent when the trial data is analysed – and as far as can be ascertained at the present time, the DWP’s trials do not appear to incorporate any such controls.

A rigorous trial methodology, when testing a system which combines two different components into a single package, must necessarily assess not only how well, or poorly, the combined system performs but also how each of the individual components performs under the same basic test conditions. At the very least, in order to assess this system’s performance, the trial needs to incorporate two additional control groups, one which tests the effectiveness of Nemesysco’s LVA software when used on its own and a second one which tests the effectiveness of using the full systems ‘behavioural analysis and conversation management techniques’ in isolation. Only by comparing the performance of the combined system with the performance of its individual components is it possible to gain a full and accurate picture of the effectiveness of the system of a whole and, as importantly, obtain the information necessary to conduct a full cost-benefit analysis of the system’s performance compared to both existing methods of validating benefits claims and to the use of individual components of the full system when used in isolation.

If, as seems perfectly possible given the lack of scientific plausibility and credible evidence for Nemesysco’s claims for its LVA software, one of the two components of the full system makes little or no effective contribution to its effectiveness or, perhaps, even reduces the effectiveness of the other component by generating a significantly greater level of erroneous assessments, then testing each of its components independently of the combined system and comparing the results will very quickly identify the weak link in the system – if such exists – allowing the DWP to avoid going to the unnecessary expense of purchasing a system only half of which actually works and which, therefore, could well have been purchased at a substantially lower cost.

Nothing, in any of the trial data or other information released to date, or obtained via FOIA, indicates that the DWP’s trial methodology includes any controls of this kind and, as such, one has to regard the trials, themselves, as being fundamentally flawed.

This is rather an alarming conclusion to arrive at when one considers that the DWP have already spent around £2 million just to test the effectiveness of this system, and all the more disturbing for having also discovered, via FOIA, that while the DWP claims to be aware of both the string criticism levelled at Nemesysco’s LVA software by Profs. Ander Eriksson and Francisco Lacerda, and the negative outcomes of, to date, the only two independently conducted trials of the software (Hollien & Harnsberger 2006, and Damphousse et al. 2007), it has neither undertaken a comprehensive review of available literature nor asked the system’s suppliers, Capita/Digilog UK, to respond to any of the criticism or negative results that are already in the public domain.

I’m going to leave it there for now, but as should be apparent, the post closes with a definite ‘to be continued’ attached to it and in my next post I’ll be exposing further flaws in the trial methodology adopted by the DWP, including evidence of the miscoding of data included in the published results of the first phase of the trials and evidence of the government’s real motives for trialling this system and why these could lead to even higher levels of fraud within the benefits system.