A couple of months ago, myself and Alex Harrowell did a big of digging into the background of the company whose ‘voice risk analysis’ technology is being introduced by the DWP as a means of screening benefits claimants for the possibility that they may be committing fraud. (see here and here)

As I recall, I think we more or less came to the conclusion that there was something altogether a bit dodgy in the whole exercise and that while there may be some moderately interesting, if limited, science behind the idea that you can identify when an individual is feeling stressed, the claims being made for the system that the DWP were bring in were unlikely to stand up to scrutiny.

In short, apart from scaring a few people into behaving a little more honestly, we reckoned that the system itself wouldn’t actually do what its developers, and Israeli company called Nemesysco, claimed it would do.

Today, while catching up on some of the commentary on Ben Goldacre’s spat with LBC/Jeni Barnett, I ran across some rather intriguing additional information about Nemesysco and its proprietor, Amir Liberman, in the form of a rather unusual academic paper by two Swedish researchers,  Francisco Lacerda and Andrew Eriksson, published in 2007 in the International Journal of Speech Language and the Law, a magazine for voice experts working for the police and security services.

The paper is unusual, for one thing, because it appears to so enraged Liberman that he’s written to the researchers’ publishers and indicated that Lacerda and Eriksson could be sued for libel if they publish any more papers on the same subject.

The paper, which reviews 50 years of research into the use of lie detectors, is entitled ‘Charlatanry in forensic speech science: A problem to be taken seriously’, which should give a clue as to precisely why Liberman is rather upset about it, but there’s more to this than just a title, as the paper goes to pay particular attention to Liberman/Nemesysco’s LVA (“Layered Voice Analysis”) system, which the DWP is using via a third party supplier.

On the subject of how the system, ostensibly, works, the paper explains that:

Contrary to the claims of sophistication – ‘The LVA sofware claims to be based on 8,000 mathematical algorithms applied to 129 voice frequencies’ (Damphousse et al. 2007: 15) – the LVA is a very simple program written in Visual Basic. The entire program code, published in the patent documents
(Liberman 2003) comprises no more than 500 lines of code. It has to be said, though, that in order for it not to be possible to copy and run the program as is, some technical details like variable declarations are omitted, but the complete program is unlikely to comprise more than 800 or so lines. With respect to its alleged mathematical sophistication, there is really nothing in the program that requires any mathematical insights beyond very basic secondary school mathematics. To be sure, recursive flters and neural networks are also based on elementary mathematical operations but the crucial diference is that these operations are used in theoretically coherent systems, in contrast to the seemingly ad hoc implementation of LVA.

Oh dear, 800 lines of code sounds about the size of the project I had to turn in when I did my ‘A’ level in Compter Science more than 20 years ago (using good old BBC BASIC) and the sophisticated mathematical algorithms turn out to be nothing than you might expect a GCSE student to cope with?

The researchers, working with a Swedish journalist, also managed to track down Liberman and, in what is an extreme unusual section to find in a journal paper, entitled ‘Who is Mr Liberman?’, note that:

We might as well have asked: Who is Nemesysco, the company behind the LVA products, because Mr Liberman and Nemesysco seem to be one and the same.  Damphousse et al. (2007: 14) report as follows: ‘The LVA was developed in Israel by Amir Lieberman [sic] who applied mathematic algorithm science to voice frequencies’, giving the impression that the program is based on some advanced mathematical theory. As we have pointed out, this is far from the truth.

And the truth is?

Afer some research he [ Arne Lapidius, a Swedish journalist working in  Israel for the Swedish daily ‘Expressen’] managed to locate Mr Liberman, a 32 year old (in 2004) businessman in a small office in the town of Natania. The business appeared to be a one-man operation. Mr Lapidus interviewed  Mr Liberman about his academic background and was told that he basically had none. He has no degree (never had time to get one, he explains) but has taken some courses in marketing at an Israeli open university. As we have explained above, the LVA is a simple program written in rather amateurishly used Visual Basic. Given what we now know about Mr Liberman, that is about what one would expect rather than ‘8,000 mathematical algorithms applied to 129 voice frequencies’ (Damphousse et al. 2007: 15). What still remains for us to understand is how insurance companies, security agencies, police depart-ments can be willing to invest hundreds of thousands of dollars, pounds, and euros in equipment without ever asking who are behind the products, what are their qualifcations, what are the scientifc principles upon which the products are based.

[UPDATE]

So as to avoid any confusion regarding the passage above…

In 2004, Professor’s Eriksson and Lacerda appear to have become embroiled in a ‘controvesy’ with Nemesysco’s Scandinavian representatives when, after making a careful study of the claims made for the system, they challenged its scientific validity. In simple terms, their examination of the ‘scientific’ basis upon which the compnay claims its products operate led them to conclude that the system, itself, is a wholly ‘bogus product’. Elements of this dispute appear to have spilled over into the Swedish press, attratching the interest of Arne Lapidus, who conducted his own investigation into Mr Liberman’s background, reporting his own findings in the Expressen.

This information was then incorporated into their 2007 paper as relevant background to their own scientific examination of the system.

….

The paper also notes that Liberman’s LVA system sells, in the US, at $25,000 for a laptop (estimated cost $2,000),  his ‘amateurishly’ written software and the training necessary to use the system.

As in numerous other recent cases, when resorts to the law rather than empirical research to defend a product that supposedlt operates within the realms of science then that’s usually a sign that something is seriously amiss.

In this case, Eriksson and Lacerda put their view of Liberman’s LVA system in no uncertain terms:

The program code is part of the patent documents and may be downloaded from patents on-line. Any qualifed speech scientist with some computer background can see at a glance, by consulting the documents, that the methods on which the program is based have no scientific validity.

And yet the DWP have decided to buy into this system, as have several local authorities.

Normally I’d leave it there, but in this case I fully intend to raise this both with James Purnell and forward Eriksson and Lacerda’s paper to the National Audit Office and the Public Accounts Committee because, as far as these researchers are being concerned, the public purse is being bilked to the tune of many hundreds of thousands of pounds for a system that looks, to them, like someone’s A level homework.