Via Jack of Kent, it seems that the British Chiropractic Association have finally been stung into publishing details of the ‘plethora’ of research evidence that they claim supports the use of Chiropractic manipulation for a variety of condition other than non-specific lower back pain.
Jack’s got a copy of the full document available for download but, having checked it over, it seems that the BCA have pulled out all the stops to make its critics’ life just that bit more difficult by securing the document in question in such a way as to prevent not only unauthorised amendments but also copying and pasting information from it in the course of mounting criticisms of its content…
…and so, out comes the trusted PDF password/encryption stripper and voila, we now have a copy of the same document from which it is possible to copy/paste. (If you have any problems downloading by clicking the link, try right-clicking and choosing ‘save link as’ – WP 2.8 is behaving a tad eccentrically at the moment)
(Update: I’ve managed to fix the PDF, so it should be working properly)
So, let’s get down to business and start to examine the BCA’s evidence, and for my own contribution I’d like to make one or general points and then tackle the first three research papers cited as ‘evidence’ for the efficacy of chiropractic manipulation as a treatment for infantile colic, leaving (hopefully) the field clear for other bloggers to examine the some of the other papers cited by the BCA, both in relation to colic and other conditions.
So, general remarks first.
In total, the BCA’s research evidence runs to 29 documents and reports of which a number are largely or entirely irrelevant.
For example, the BCA cites it own code of practice and a reference in a glossary to a definition of ‘evidence based care’ as ‘‘clinical practice that incorporates the best available evidence from research, the preferences of patients and the expertise of practitioners (including the individual chiropractor him/herself)’. Research evidence is, of course, central to evidence-based medicine but this is not necessarily also true of either the ‘preferences of patients’ or even the ‘expertise of practitioners’, neither of which can be guaranteed to be founded on or derived from actual evidence.
Also joining the pile the marked ‘objection – relevancy’ are:
An article (citation 6) that discusses the conduct of medical researchers and the issue of the misrepresentation of research evidence – the BCA are nothing if not lacking in well-defined sense of irony.
Three papers (citations 8-10 inclusive) which deal with evidence for the efficacy of osteopathy rather than chiropractic.
A paper proposing a supposedly ‘rational framework’ for the care and management of excessive infant crying (citation 24).
A general paper on the nature of evidence-based medicine (citation 26).
And finally, three papers on the side effects associated with the use of nonsteroidal anti-inflammatory drugs (citations 27-29, inclusion).
So, before we’ve even got out of the starting blocks, fully one third of the ‘evidence’ cited by the BCA turns out to provide no actual evidence to support any of its claims for the efficacy of chiropractic manipulation when dealing with conditions other than back pain.
Moving swiftly on, the first three research papers cited by the BCA, Klougart et al. (1989), Mercer and Nook (1999) and Wiberg et al. (1999) were all reviewed by Stephen Hughes and Jennifer Bolton in a short paper that was published in 2002 and is available online via the BMJ’s ‘Archives of Disease in Childhood’ under the title ‘Is Chiropractic an effective treatment in infantile colic‘. (Registration required, but free of charge). The BCA do not reference this paper in its list of ‘research evidence’.
Of these three papers…
Klougart et al. is an uncontrolled prospective study that provided the first [alleged] evidence for the use of chiropractic manipulation in treating infantile colic. Although noteworthy for its size – 316 infants were recruited to the study – it lacks blinding, randomisation and a control group and is, therefore, subject to a considerable degree of bias and offers no means of assessing its findings against either a placebo or simply allowing the condition to run it natural course, colic being a condition that typically improves over time.
The BCA’s inclusion of the paper by Mercer and Nook, which claims a 93% success rate in actually curing colic, in its list of evidence is nothing short of a complete embarrassment. The paper was actually reported only in abstract and provided very little methodological information and no actual research data to support its claimed findings. It is a very small study (30 infants diagnosed with colic by a paediatrician but no information on the diagnostic criteria used) in which 15 infants were treated with chiropractic spinal manipulation and the remaining 15 were ‘treated’ with non-functional, detuned, ultrasound machine as a placebo and has a list of flaws almost as long as your arm…
No information given on dropouts RCT (level 1b). Single blinded study. Randomisation unclear. Subjective response to treatment by parents before treatment and at each subsequent consultation. Outcomes not defined. Statistically significant difference (no data given) in response to treatment between 2 groups (assumed beneficial in experimental group). Complete resolution of symptoms in 93% of infants in (assumed) experimental group. No comparative data for placebo group.
As citations go, Mercer and Nook is the next best thing to useless as a piece of evidence.
The third study, Wiberg et al. is perhaps the best designed of the three but still not without significant flaws. It is a single blinded study of 50 infants, 25 of whom were given chiropractic treatment while 25 were treated with dimethicone, a common, over the counter, colic remedy which has been shown to perform no better than a placebo. Nine of the 25 infants treated with dimethicone dropped out of the study with their parents citing a worsening of symptoms, rather than a medication bias. The big problem with this study is that the parents were fully aware of which treatment their child was receiving and the researchers failed to conduct an ‘intention to treat’ analysis prior to beginning the study, the upshot of which being that the outcomes reported are prone to a significant degree of parental bias.
The upshot of all this is that although Wiberg et al. report that the chiropractic treatment proved to be more effective than dimethicone, its impossible to say whether these result indicate that the treatment had an genuine theraputic effect or whether it merely served as a more effective placebo than the treatment administered to the control group.
Having covered the three papers cited by the BCA, it should be noted that Hughes and Bolton included a fourth paper in their review, by Olafsdottir et al. (2001), which the BCA has omitted from its list of evidence. This last study is somewhat larger than the Wiberg study (100 infants) but uses broadly the same methodology but for one very significant difference – the Olafsdottir study is double-blinded such that neither the parents or the researchers were aware of the precise treatments given to a specific infant.
Olafsdottir found no difference in the parentally reported symptom scores between spinal manipulation and placebo, hence, on would assume, the exclusion of this study from the BCA’s list of evidence.
So, from looking at just three of the colic studies put forward by the BCA its apparent that they’ve been highly selective in the choice of evidence.
This is unproblematic in the context of their current litigation against Simon Singh where, due to the adversarial nature of the courtroom, neither side is necessarily bound to advance evidence that does not support their case, but in terms of the credibility of the BCA’s claim that is adhere to the principle of evidence-based care it rather blows a hole in the reference, in their code of conduct, to using the ‘best available evidence from research’. What Hughes and Bolton concluded in their review was that the best available evidence, at the time, was that provided by Olafsdottir et al, which showed that chiropractic spinal manipulation performs no better than a placebo when used to treat infantile colic, and yet this is the one paper of the four included in that review that the BCA have failed to cite in their list of evidence.
While it would be fair to suggest that, in writing his original article, Simon Singh could have been somewhat more precise in his criticism by asserting that the BCA’s claims in regards to the treatment of colic lack credible evidence, the all-too-obvious partiality demonstrated by the BCA in its selection of ‘evidence’ does nothing whatsoever to enhance its already damaged credibility and, in my own personal opinion, fully supports Singh’s contention that the BCA happily promotes ‘bogus’ treatments.
The BCA may well believe that the three papers I’ve looked here provide evidence that supports the use of chiropractic spinal manipulation as a treatment for colic but belief alone is not proof that such treatments are not, in reality, entirely bogus and lacking in any theraputic value over and above that provided by a simple placebo.
So, that’s my bit over and done with for the moment… I wonder who else is up for picking apart the BCA’s list of ‘evidence’?
UPDATE
Jack of Kent has now put up a commentary on the covering statement that accompanies the BCA’s list of evidence, which is well worth a read…
AND SO IT BEGINS…
The Lay Scientist has a rather more comprehensive look at the amount of irrelevant material the BCA has included in its ‘plethora’ of evidence, while Prof David Colquhoun has also given the BCA’s colic papers the once over.
Thanks for your analysis of this BCA ‘evidence’. I can’t get the link to the pdf to work – it goes to something which just sais – I don’t understand your banter!
Can you e mail it me please?
Many thanks
clod
Whilst the BCA may not have to advance evidence harmful to their case in a court of law, the chiropractors at the BCA are obligued by their code of ethics not to misrepresent evidence…
“C1.6 may publicise their practices or permit another person to do so consistent with the law and the guidance issued by the Advertising Standards Authority. If chiropractors, or others on their behalf, do publicise, the information used must be factual and verifiable. The information must not be misleading or inaccurate in any way. It must not, in any way, abuse the trust of members of the public nor exploit their lack of experience or knowledge about either health or chiropractic matters. It must not put pressure on people to use chiropractic.”
I feel a complaint coming on to the GCC.
I feel a complaint coming on to the GCC.
May well be worth a shot, but regardless the BCA have made a major mistake in putting up their claimed evidence where it can be publicly shot at/down.
I don’t think they had much choice – events are becoming less in their control now.
I understand the ‘plethora’ may well have been made public anyway and that they probably felt it better for them to publish than to suffer the humiliation of someone doing it for them.
Thanks for ripping the PDF – unfortunately it registers as “corrupt” in anything I try to open it with.
Can anyone think of any logical reason why an organisation trying to defend its public image would release a media statement that no one can quote?
Excellent work!
BTW, I am not sure dimethicone works better than placebo.
I’ve now fixed the PDF..
The GCC code of practice includes a section about not bringing chiropractic into disrepute.
Do you think we should officially complain to them that the BCA is doing just that?
Might be fun just seeing how they respond to the accusation (which they have to do).
Interesting that you examine Wiberg et al. 1999 (I’ve not had chance) since this is the only (relevant) paper cited in letters to me from the GCC in response to my complaint about chiropractors claiming to treat colic.
They allude to this paper as follows:
“Wiberg JMM, Nordsteen J, Nilsson N. 1999. The short-term effect of spinal manipulation in the treatment of infantile colic: a randomised controlled clinical trial with a blinded observer, JMPT 22 (8): 517-22.”
This is the only reference I’ve seen to a trial with blinding, controls, and randomization, so it’s interesting to learn that this paper is flawed and has a less than glowing BMJ review – which the GCC omitted to mention.
It’s also interesting to learn that this is really a comparison of dimethicone treatment & chiropractic.
And no, dimethicone (or at least Simethicon = dimethicone + silica gel) does not work any better than a placebo: http://www.ncbi.nlm.nih.gov/pubmed/8008533
Unfortunately, DT, the GCC code only applies to individual chiros, not trade associations like the BCA. However, If the BCA’s members were to make claims the BCA thinks are valid, but failed to meet the GCC’s code (which requires robust scientific substantiation, then they’d be encouraging their members to break the law. What reputation, then, do they have to defend?
Of course, it if was a BCA official who was also a chiro who made those claims, then he’d presumably be in breach personally.
“…due to the adversarial nature of the courtroom, neither side is necessarily bound to advance evidence that does not support their case…”
Pardon me for acting the barrack room lawyer (Jack of Kent will probably know the answer to this) but is this right? Perhaps the rules are different in libel cases, or the judge may have directed more limited disclosure, but the general rule (“standard disclosure”) is that a party is required to disclose any documents on which he relies, or which adversely affect his own case, or adversely affect another party’s case, or support another party’s case.
See rule 31.6 of the Civil Procedure Rules – online at http://www.justice.gov.uk/civil/procrules_fin/contents/parts/part31.htm#IDATBKCC
Presuambly the BCA will be disclosing the “plethora” to the court – but perhaps someone should suggest they they include the evidence that supports Simon’s case too?
The question, really, is whether this pile of awfulness amounts to a “jot” or not, and whether the BCA realises that there is no proper evidence for their claims.
Ministry of Truth? How divinely Orwellian!
Your criticism of Klougart et al. makes me suspect that you have never received a chiropractic adjustment. If you had, then you should know that it would be impossible to fulfill your demand for “blinding” . In fact, designing a truly inert “sham” adjustment has so far eluded many sophisticated researchers in manual medicine (DOs and DCs both). You can read more about that topic at the “Problem with Placebos/Shams” page at:
http://www.chiro.org/research/ABSTRACTS/Placebos.shtml
You then blithely state that “colic being a condition that typically improves over time”. Really? And what time frame is that?
Any procedure for any illness has to face up to what is referred to as the “natural history of the disease”. If you could “prove” that colic always resolved in 2 weeks or less, then you’d have a leg to stand on, since the research results were “results occurred within 2 wk and after an average of three treatments.” However, if colic routinely last longer than 2 weeks (place your bets!) then it sounds like these researchers are onto something.
You can read the abstracts for many colic studies at:
http://www.chiro.org/research/ABSTRACTS/Colic.shtml
Then you whine about the Mercer and Nook study, right after stating that the problem with Klougart was that it lacked a control group. Well, what would you call the 15 (who were) were
Frank, I agree the issue of creating a believable placebo is a thorny one, particularly for interventions that do not lend themselves well to suitable “shams” such as chiro, but do not assume that placebo-controlled studies are not feasible. This is particularly the case for children/infants, who cannot tell the difference between a manipulation or a cuddle. And, as Olafsdottir showed, studies like this show chiro is no better than cuddling.
Colic is self-limiting, it gets better over time. How quickly? Well it seems symptoms resolve in 60 percent of infants by 3 months of age and 90 percent of infants by 4 months of age (Parker s, in “Colic”, Developmental and Behavioral Pediatrics: A handbook for primary care. Lippincott Williams & Wilkins, Philadelphia 2005).
The Mercer and Nook study is a joke. The fact that it has a “control group” (and a poor one at that) does not excuse its abysmal scientific quality in every other area.
The funding of research is not in question here – straw man fail. Anyhow, research in complementary therapies is not done primarily because the proponents refuse to contemplate doing it, not because it is hard or expensive to do. Somehow, the list of excuses as to why it cannot be done just gets longer over time.
“Satisfaction surveys”? Please say you are joking. Testimaonials and satisfaction surveys are at the bottom of the dung heap where evidence is concerned as far as I am concerned. They indicate the promoters of the therapy have no proper evidence of effectiveness. On the Internet I could probably find you testimonials and satisfaction surveys about how wonderful moon-howling therapy is at curing baldness. Would that make it true?
So you claim that “over time” you “will develop the research [we] crave”?
How come you have not developed the research anytime in the last 100 years or so? Not “time” enough for you?
I know of a crystal auratherapist who has spent 10 years learning her art. Does that make her therapies valid? According to you, it must do. So its to hell with the science, let’s just do what people have always done, hey?
Bloodletting here we come.
I believe the first Federal dollars to fund any chiropractic research didn
Having been a close observer in not one, but two, attempts by the chiropractic profession to develop its research arm, I can state categorically that the reason chiropractic has weak research, is because chiropractic is systematically excluded from science academia and biomedical research funding.
The first series of events occured in the late 1980’s through the early 1990’s, when the Foundation for Chiropractic Education and Research funded a series of fellowships to financially assist Masters-degree-holding scientists to achieve their PhD, with their research foci on chiropractic. The concept was to achieve a “critical mass” of credible researchers so that chiropractic-focused research centers could be founded. What happened instead was, after writing chiropractic-oriented dissertations, the fledgling chiropractic researchers found that they could not get hired, published, or funded if any aspect of their applications or papers referenced chiropractic in any way, and that they were socially ostracized in academia if they discussed their interest in chiropractic. Unsurprisingly, all these researchers are working in other areas now.
The second experience was at Florida State University, which has a well-established reputation for biomedical research. FSU established a College of Medicine in 2000, one of whose stated missions was to improve training of young physicians in alternative medicine. The Florida Chiropractic Association raised $1M in “seed money” from its membership and lobbied heavily until money was budgeted by the legislature to include a College of Chiropractic at FSU. This looked set to proceed in 2003-2004.
At that time, the medical school’s final accreditation was pending, and suddenly the medical school’s clinical director, an irascible orthopedist who hates chiropractors virulently, announced that he would resign if a chiropractic school were founded at FSU. He arm-twisted approximately 1/3 of the medical staff of the school to go along with him. An administrative hearing was held; the Florida Medical Association and the American Medical Association both spent hundreds of thousands of dollars getting career “experts” in “quackery” to testify.
So, chiropractic is not scientific because it doesn’t have large-scale, university-based research to back it up. But, it has lower quality research because it is “not scientific” enough to be the subject of serious academic research.
Olafdottir’s study does bring into question the effectiveness of chiropractic intervention for colic, which prior studies such as Nillson’s had supported. It is, however, underpowered, as calculation of confidence intervals compared to the p values shows. The real solution would be for a study utilizing thousands of infants treated by dozens of pediatric-board-certified chiropractors at multiple centers to be designed, funded, and performed. This would cost about 1/10th of 1% of the funding allocated by the US government to conduct one typical series of trials on one drug.
Lastly, if you had bothered to read Nilsson et. al., you would know that dimethicone drops are used precisely because they are no better than placebo, as a control group for the study, and that the study includes references to research on the natural history of colic which shows that both control and placebo groups did better than the normal progression of the untreated condition.
A good summary of the current status of chiropractic research, pitiful though it may be compared to research which is far more generously funded, may be found at: http://www.icabestpractices.org/chapter-docs.html