Homeopathic Proving- Walach plays at science again

[Badly Shaved Monkey]

Harald Walach veers erratically between trying to excuse homeopathy from the scrutiny of controlled trials by proposing ever more convoluted reasons why it still works but can't work in a trial. This is the fridge light bulb theory of homeopathic trials- we just have to believe the light goes out when the fridge door shuts, but we are precluded from drilling a hole in the side of the fridge to check because he says this will instantly stop the fridge from working at all

However, there are also times when he does some science.

Thanks to shpalman, I've come across a new one.

It's hard to know what to make of it.

They report that they went to great efforts at effective blinding.

At a first glance the graphs make interesting reading. The p-values are highly significant.

Those high p-values are odd, though, when they have been derived from non-parametric tests of treatment groups containing 10, 8 and 7 subjects.

"Results
Altogether, 25 volunteers, 6 men and 19 women, with a
mean age of 42.3 years (standard deviation SD = 6.58 years),
all medical doctors, participated in the experiment. 10 were
randomised to receive Natrum muriaticum, 8 to receive Arsenicum
album, and 7 to placebo. They experienced altogether
165 symptoms over the course of 4 days. On average
they reported 6 symptoms when taking Arsenicum album, 5
symptoms when taking Natrum muriaticum, and 11 symptoms
when taking placebo. The results are presented in figure 1,
and a selection of typical symptoms is presented in table 1.
As can be seen, symptoms typical for the respective remedy
were more frequent both in the Natrum muriaticum group
and in the Arsenicum album group, while non-specific symptoms
were more frequent in the placebo group. A non-parametric
Kruskall Wallis analysis showed significant differences
between groups (p = 0.0002), and a pre-planned separate
Mann-Whitney tests confirmed that significantly more specific
symptoms (p < 0.001) were observed in the respective groups
compared with the placebo control group."

Figure 1 can be summarised as below.

Treatment Group	Ars Alb	Natr Mur	Placebo
n	8	10	7
No. of Ars Alb Symptoms	6	0	0
No. of Natr Mur Symptoms	0	4.5	0
No. of Non-specific Symptoms	0	0	10.5
Total Number of Symptoms	6	4.5	10.5

(Thanks to Here_to_learn I can parse a table properly!)

The phrase, 'too good to be true' does rather spring to mind.

These are "mean" numbers of symptoms, though they then did non-parametric tests. Fig 1 showed markers for those means with 95% confidence intervals, which, of course, necessarily went negative (to approx -2) where the "mean" symptom number was zero. Someone will have to explain to me how you can count objects, get a mean of 0 and have a CI that goes to -2.

Gentlemen (and ladies) start your engines...

[Yuri Nalyssus]

Originally Posted by Badly Shaved Monkey

Thanks to shpalman, I've come across a new one.

It's hard to know what to make of it.

A Journal called Research in Complementary Medicine edited by Walach, H. with an article in it by Walach, H. saying homeopathy works?

All sounds perfectly pukka and above board to me .

Yuri

[Badly Shaved Monkey]

Actually, thinking about Fridge Light Bulb Theory a bit more, and like all such things it should be referred to by an acronym to make it sound grander, so FLBT, and, even better, 'flibbet' to real insiders such as ourselves, I realise that if I reverse my metaphor, or make it 'complementary' I more accurately capture the hom's thinking.

So, under 'flibbet', the homs contend that despite all we know of the physics, engineering and design of fridges and the contact switches in the doors, in fact, the light stays on when the door is shut, but that any attempt to test this assumption, for instance by drilling a hole to have a look, will immediately cause the bulb to fail.

There, I think the life has been adequately squashed from that metaphor.

[Graham Jackman]

It all looks far too neat, with no cross-over in symptoms between the treatments e.g. no-one on "active" treatment showed non-specific symptoms!
I've no interest in paying for the full paper, but we have no idea of what these symptoms were and with an average of 1.65 symptoms per day for each subject over 4 days, I don't imagine these were particularly severe or troublesome

[Fnord]

FLBT seems like a special "macro-world" case of the Heisenberg Uncertainty Principle ... at least, as I understand it ("Observing a quantum-level experiment affects the outcome").

Thus, I propose that FLBT be defined as "Attempts to measure the efficacy of a homeopathic substance renders that substance inert and ineffective."

[Rolfe]

This is actually a magical property.

I discussed this with an anthropologist a few years ago. He explained that magic must be performed in secret, as otherwise the magic would rebound on the magician. I then suggested that attempts to test magical principles by scientific methods were in effect performing magic out in the open. The corollary of this is that the magic does not want to be revealed, and thus will make very certain it is not revealed. I think I got a footnote citation in his book for that.

Rolfe.

[fls]

Originally Posted by Badly Shaved Monkey

Harald Walach veers erratically between trying to excuse homeopathy from the scrutiny of controlled trials by proposing ever more convoluted reasons why it still works but can't work in a trial. This is the fridge light bulb theory of homeopathic trials- we just have to believe the light goes out when the fridge door shuts, but we are precluded from drilling a hole in the side of the fridge to check because he says this will instantly stop the fridge from working at all

However, there are also times when he does some science.

Thanks to shpalman, I've come across a new one.

It's hard to know what to make of it.

The results don't make sense. Why wouldn't there be non-specific symptoms in all groups?

Quote:

They report that they went to great efforts at effective blinding.

Who judged the symptom diaries and how were they blinded?

Linda

[Badly Shaved Monkey]

Originally Posted by fls

The results don't make sense. Why wouldn't there be non-specific symptoms in all groups?



Who judged the symptom diaries and how were they blinded?

Linda

I'm working from my phone so can't paste in the relevant description of the blinding. It sounded good, but could I explain exactly how the data were collated and describe the paper trail in detail? No I could not.

Perhaps Rolfe might extract some (fair use) quotes for us. She has access to the text.

[Rolfe]

Originally Posted by Badly Shaved Monkey

I'm working from my phone so can't paste in the relevant description of the blinding. It sounded good, but could I explain exactly how the data were collated and describe the paper trail in detail? No I could not.

Perhaps Rolfe might extract some (fair use) quotes for us. She has access to the text.

Mmmm, can I temporarly bow out? I have a week's holiday starting in a few hours, and first I have to get through some work, and then I want to concentrate on clearing and tidying my house, which is what the leave is for. And then I have pressing matters which will be waiting for me when I get back (like two commissioned chapters for a book....). I'm trying not to get caught up in anything detailed online for the next two weeks or so.

Rolfe.

[Mojo]

Originally Posted by Fnord

Thus, I propose that FLBT be defined as "Attempts to measure the efficacy of a homeopathic substance renders that substance inert and ineffective."

That doesn't work: homoeopaths and their clients regularly report having observed effects of homoeopathic remedies, so the act of observation evidently doesn't render them inert and ineffective.

[Ivor the Engineer]

Originally Posted by Rolfe

Mmmm, can I temporarly bow out? I have a week's holiday starting in a few hours, and first I have to get through some work, and then I want to concentrate on clearing and tidying my house, which is what the leave is for. And then I have pressing matters which will be waiting for me when I get back (like two commissioned chapters for a book....). I'm trying not to get caught up in anything detailed online for the next two weeks or so.

Rolfe.

You know, sometimes I think you just don't take your online obligations seriously enough.

[Mikeuk]

How does a Homeopath define 'Placebo' -wouldn't that be a lethal cocktail of undiluted chemicals??

Maybe that explains why only 7 in the Placebo group had 11 Symptoms

[shpalman]

How much can I quote without getting in trouble?

Quote:

25 out of 59 course participants having given consent were randomised to receive either one of two homeopathic remedies, Natrum muriaticum or Arsenicum album, chosen randomly from a predetermined list of 20 remedies by an independent pharmacist despatching the substances, or an identical placebo... The list of remedies was only known to the study director (HM) and the pharmacist, not to the participants, and not to the study team (HW, RS).

Thus, all participants, the study director (HM) and other personnel involved with the administration of substances were blind as to the precise name of the remedies administered and group allocation. Substances were prepared freshly by Dolisos homeopathic pharmacy... and sent in numbered containers to the study site. The randomisation code was created by an independent researcher (RS) from the random algorithm of RITA (Randomization in Treatment Arms) [Pahlke F, König IR, Ziegler A, Informatik Biometrie Epidemiol Med Biol 2004;35:1–22.]. The code contained three group assignments with random numbers that were used to choose from the containers sent by the pharmacy. The code was kept safely by the study centre. The substances were prepared in CH30... They were then sprinkled over sugar globules, as is the dispensing rule... Placebo was sprinkled with the same amount and batch of alcohol, which is used as the carrier for the homeopathic production process.

It seems as if then all the reported symptoms across all subjects were written down, organized in a "head-to-foot" scheme, and presented to a materia medica expert who used some computer repertorisation programmes to decide which of the reported symptoms were typical of the tested remedies.

Quote:

This procedure yielded our final outcome parameter: the number of symptoms typical for each remedy. This, and the number of untypical symptoms was calculated and submitted to non-parametric statistical testing, using SPSS.

The overall number of all symptoms (per participant, on average) was about 6 in the Arsenicum alb. group, 5 in the Natrum mur. group and about 11 in the Placebo group, but they claim that the symptoms in the two homeopathy groups were specific to those remedies while the symptoms in the placebo group were "non-specific". There's a table giving an example of symptoms reported, but there is no corresponding list of claimed typical repertory symptoms for the remedies.

[Badly Shaved Monkey]

Thanks. We also need the paragraph about who did the repertorising.

[shpalman]

Quote:

Finally, a materia medica expert otherwise unconnected with the study was given the list of symptoms, ordered according to the head-to-foot scheme, and the names of the remedies tested. The expert was blind to group allocation, and he was not given the original protocols, but a collated list of symptoms which did not allow him to attribute symptoms to individuals, nor to groups. The expert used a commonly used and commercially available computer repertorisation programme (Synthesis by Archibel, Namur, Belgium) to decide for each symptom, whether it is typical for one of the two remedies tested or not. Note that through this method it was impossible for the materia medica expert to second-guess which volunteer might have produced which series of symptoms. He was strictly bound to make a decision for each individual symptom, irrespective of other symptoms.

Also, from the discussion section:

Quote:

The materia medica expert who determined whether a certain symptom was typical for one of the two remedies or not, was blind as to group allocation, and otherwise not involved with the organisation or conduct of the study. The only way this person could have cheated would have been by having access to the code and coding symptoms, accordingly. This was impossible, as the code existed only in its full version – i.e. allocation of code numbering to remedies – at the research centre and was only fully revealed once the database of symptoms was classified by the materia medica expert and the statistical analysis had been done blindly.

[Badly Shaved Monkey]

Thanks

It looks pretty tight. So, how did they achieve zero non-specific symptoms in their remedy groups?? Given what they count as a symptom I can't see how that is even possible.

[Badly Shaved Monkey]

I also want to know how a mean symptom count can be 0+/-2

[shpalman]

Zero plus or minus 2, on average per person in four days: no, I don't know how they get negative symptoms either.

[Mojo]

Originally Posted by Badly Shaved Monkey

It looks pretty tight. So, how did they achieve zero non-specific symptoms in their remedy groups?? Given what they count as a symptom I can't see how that is even possible.

Is there a claim that "proving" symptoms somehow prevent nonspecific symptoms? If not you would expect the same number of nonspecific symptoms in each group, wouldn't you?

[fls]

Originally Posted by shpalman

It seems as if then all the reported symptoms across all subjects were written down, organized in a "head-to-foot" scheme, and presented to a materia medica expert who used some computer repertorisation programmes to decide which of the reported symptoms were typical of the tested remedies.

This part is where we need the details. Was this person blind as to which remedies where given and to whom? Because I have to say that even if they claim she/he was, the results make no sense unless they weren't. Even if you agree that proving is valid, those who took the remedies should have non-specific symptoms in addition to the specific symptoms. The symptom count is similar in the three groups and they differ mostly in their assignment. This suggests that there was not a specific effect of the proving substance, but that there was specific sorting of the effects a posteriori. Also, the small number of symptoms suggests that there must have been considerable overlap with many substances. That the materia medica expert could have narrowed it down to two active substances on the basis of a handful of symptoms seems an impossible task.

Quote:

The overall number of all symptoms (per participant, on average) was about 6 in the Arsenicum alb. group, 5 in the Natrum mur. group and about 11 in the Placebo group, but they claim that the symptoms in the two homeopathy groups were specific to those remedies while the symptoms in the placebo group were "non-specific". There's a table giving an example of symptoms reported, but there is no corresponding list of claimed typical repertory symptoms for the remedies.

A more convincing analysis would have been to give the materia medica expert the information blind and then test whether or not she/he could identify which substances were given to whom.

ETA: Thanks for the info shpalman. I was too slow.

Linda

[Mojo]

Originally Posted by fls

A more convincing analysis would have been to give the materia medica expert the information blind and then test whether or not she/he could identify which substances were given to whom.

They already tried that; they couldn't:

Quote:

The materia medica expert was not able to determine the correct medicine, either in step 1 (unrestricted choice), or step 2 (restricted choice).

You wouldn't know it from the abstract though.

[fls]

So it's quite simple. The results are most consistent with blinding that has been broken. One only needs to decide whether it should be trusted that it wasn't.

Even if one decides to trust the reporting, it's hard to see how that information can be used to counteract much larger proving trials that show no difference.

Linda

[fls]

Originally Posted by Mojo

They already tried that; they couldn't:

You wouldn't know it from the abstract though.

That has always been one of my favourite articles.

That study and other previous studies contradict the results of this study. So even if you are inclined to trust these results, at best the evidence could be considered inconclusive.

Linda

[shpalman]

The materia medica expert was given

Quote:

the list of symptoms, ordered according to the head-to-foot scheme, and the names of the remedies tested.

... so only had to decide whether a given symptom was for Arse. alb. (for example, "After the end of the course unable to remember for a short while where I had parked my car – slight mental disturbance"), Nat. mur. ("Twitching right eye, for 2 minutes"), or neither ("Left ear suddenly free; I had not realised that it was blocked").

You're right that borked blinding is the simplest explanation.

[Mojo]

Here's another one: Can homeopaths detect homeopathic medicines? A pilot study for a randomised, double-blind, placebo controlled investigation of the proving hypothesis

Quote:

In this study a promising trend was observed that symptoms reported by some homeopaths may not be completely attributable to placebo.

If you squint at the results, in the dusk, with the light behind them, there might be something there.

Interestingly, given the method used ("The study design was a double-blinded, crossover trial. It consisted of a 1-week study medication period, a 2-week washout period and a further 1-week on study medication"):

Quote:

There was evidence of an order effect: subjects were much more likely to think they received active Bryonia in the first rather than the second period.

I'd still be interested in knowing whether the three dogs in the Hill et al "trial" got homoeopathy or placebo first.

[shpalman]

Hmmmm, having made notes to keep track of who did what and who knew what, I notice that the "independent researcher (RS)" who created the randomisation code also

Quote:

did the statistical analysis, wrote parts of the manuscript, participated in discussion of the data and editing of the final manuscript.

So the second author, Rainer Schneider, generated the secret code linking the remedies to the participants and also did the statistical analysis on the results after the expert had collated the symptoms with the remedies but before the blinding had been officially broken.

I'm not accusing anyone of dishonesty, but this strikes me as a possible security hole.

[Rolfe]

The thing that worries me about this is the difference between unconscious bias and deliberate fraud.

Benveniste didn't try to blind his work at all. The concept simply wasn't in the protocol. And it's the easiest thing in the world to skew a manual white cell count a little bit one way or a little bit the other. If you constantly know which sampe you're counting, it would be easy to do it without realising. That's what Maddox, Sampson and Randi found.

There are situations where an inadvertent unblinding can explain unexpected findings quite adequately. Like the smelly dogs in the herbal remedy paper I mentioned earlier. But in this one, I can't see how an accidental unblinding could cover what seems to have happened. You're talking about someone who knows perfectly well he's not supposed to know which group is which while he's making his decisions about symptoms, and yet he does know. And he goes on deciding nonetheless. You don't do that by accident, even if the fact that you know was due to an accident in the first place.

Rolfe.

[Rolfe]

Another thought, maybe someone else with access to the paper could check it for me.

Three groups. 10 salt, 8 arsenic, 7 placebo. Were these individuals treated as 25 individual people, or as three groups? Did the master prover already know how the patients were grouped before he got going? Because if he did, then he only has to guess a three-way allocation of probability. Which I think might be about a 25% chance anyway, or will someone correct me?

Given that it was a racing certainty that the placebo group would be the smallest one, maybe it was a 50/50 chance. You wouldn't need the blinding to slip much to make it a certainty from there.

I need to read the paper but not right now. Is this idea worth following up?

Rolfe.

[shpalman]

Two more things have occurred to me.

Firstly, it seems like the blinding was designed to make sure that the materia medica expert would not be able to cheat in the assignment of remedies to participants, since he only received a list of disconnected symptoms. However, I'm not sure exactly who it was who reconnected the symptoms, with their guesses as to which remedies "caused" them, back with the list of participants. But we know that the blinding code, created by RS, "was only fully revealed once the database of symptoms was classified by the materia medica expert and the statistical analysis had been done blindly..." by RS. I think it would have been better if the pharmacist had created the blinding code and not sent it to the study centre until after the statistical analysis.

So there's the possibility that RS could have just changed the which patients certain symptoms were supposed to have come from, once the symptoms were labelled with remedies. I don't know if he could have done this without HM (the lead author, who collected the symptoms from the participants via diaries and interviews) noticing. There's no innocent explanation for how this can have come about that I can think of.

Secondly, there were 7 participants receiving placebo how recorded an average of 11 non-specific symptoms each, so that's about 80 non-specific symptoms out of the reported total of 165. The 10 participants on Nat. mur. had 5 symptoms each so that's about 50; the 8 on Arse. alb. has 6 each so that's also about 50. It adds up to 180 which isn't much over the 165 reported. Given that the materia medica expert knew that what the participants had been given, it seems about right that he would assign the symptoms in these proportions. But then if RS is deliberately reassociating symptoms, now assigned to remedies, with the participants who got those remedies you might indeed get the strange result that all the non-specific symptoms, which you would expect to have been at a similar level for all groups, have been assigned to the placebo group which therefore had double the number of symptoms compared to the groups on homeopathy.

Rolfe, I'm not sure exactly what you mean in your second post. HM, who collected the symptoms, claims to have not known the group allocation or even exactly which pair of remedies out of the possible twenty had been chosen by the pharmacist.

Here are my notes from last evening:

HM: Study director (Heribert Moellinger, first author)
HW: Study designer and overseer, and editor of the journal (Harald Walach)
IP: Independent pharmacist (at Dolisos, Lausanne)
RS: Independent researcher (Rainer Schneider, co-author)
MM: Materia medica expert (Reimund Wagner, acknowledged)

HM produces list of 20 remedies
IP chooses two "at random" to give to selected participants, prepares remedies and placebos and sends them in randomly numbered containers
RS assigns random numbers to participants - "code was kept safely by the study centre"
HM as "proving director" conducts interviews with participants to verify symptoms noted by participants in their diaries
?? collates symptoms across all participants in head-to-foot scheme, sends to MM
MM collates symptoms with the two remedies using software
?? collates symptoms, now labelled according to remedies by MM, with the participants they came from
RS performs statistical analysis of symptom collation
?? breaks blinding

[fls]

How does one perform a statistical analysis before blinding was broken if all you have is a list of symptoms and no way of knowing which symptom goes with which participant or group?

Linda

[Badly Shaved Monkey]

Originally Posted by fls

How does one perform a statistical analysis before blinding was broken if all you have is a list of symptoms and no way of knowing which symptom goes with which participant or group?

Linda

They must have broken the blinding before testing because they did pre-specified U-tests after their K-W test, so they had to know which group was which at that point.

But I still don't understand what raw data they used. It seems to be counts of symptoms. So for the Ars Alb test they have 25 separate symptom counts, one for each subject. But we return to the weird fact that for the two non-Ars Alb groups the mean symptom count is zero and somehow they have been able to use this count to calculate a value for 2 SD that means some subjects had a negative number of symptoms. There is no possible list of positive integers that can yield mean=0 and 2SD=2 no matter how skewed, kurtotic or multimodal it might be. This alone suggests that something bloody peculiar is going on.

[Badly Shaved Monkey]

I think I feel a letter to the editor coming on. Just a straightforward comment and request for clarification on the 0+/-2 business. That is nice and specific. All it requires it a brief response from them to show at least one set of the raw data that generated those summary statistics.

[shpalman]

Originally Posted by fls

How does one perform a statistical analysis before blinding was broken if all you have is a list of symptoms and no way of knowing which symptom goes with which participant or group?

The authors were not blinded as to which symptom went with which participant, and after sending the list of symptoms and the names of the remedies to the materia medica expert (who was blind as to which symptoms went with which participant) they knew, in principle, which participants had symptoms typical of which remedies.

Quote:

This procedure yielded our final outcome parameter: the number of symptoms typical for each remedy. This, and the number of untypical symptoms was calculated and submitted to non-parametric statistical testing, using SPSS.

Quote:

... the code existed only in its full version – i.e. allocation of code numbering to remedies – at the research centre and was only fully revealed once the database of symptoms was classified by the materia medica expert and the statistical analysis had been done blindly.

(The only thing that was blinded was the information regarding which participants actually received each remedy.)

[shpalman]

Originally Posted by Badly Shaved Monkey

I think I feel a letter to the editor coming on.

Walach is the editor.

ETA: I hope my blog post won't prejudice matters...

[fls]

Originally Posted by shpalman

The authors were not blinded as to which symptom went with which participant, and after sending the list of symptoms and the names of the remedies to the materia medica expert (who was blind as to which symptoms went with which participant) they knew, in principle, which participants had symptoms typical of which remedies.

HM interviewed the subjects as to their symptoms, not knowing which subject had received which remedy. Somebody takes those symptoms and orders (and presumably codes) them. Each symptom is assigned to one of three treatments, and then someone re-orders them according to subject (e.g. subject 1 had symptoms 16, 23, 72 and 102, three which were coded as Arsenicum and one which was coded as non-specific). Now RS can group the subjects according to how the bulk of their symptoms were labelled (e.g. subject 1 would be grouped with other subjects whose symptoms were mostly labelled Arsenicum), and the table that was given would be consistent with forming groups in that manner (). But prior to breaking blinding, he wouldn't know which subjects would be grouped together based on the treatment they received, and so he wouldn't be able to analyze whether there were significant differences based on treatment received.

The simple answer is that blinding was broken in order to tell him which patients received which treatment. But the point is that it couldn't have happened as described, just like (as BSM points out) an analysis shouldn't yield an average of 0 symptoms with a confidence interval that includes negative numbers if it was performed as described. If we already know that the description has to be wrong on two issues, what else is it wrong about?

This may simply be a misunderstanding, but as it stands, it needs clarification.

And thank you for providing this information for me.

Linda

[fls]

I should add that it may be as Rolfe pointed out - that the researchers were given information about which subjects received the same treatment (without knowing what that treatment consisted of). Then it becomes a matter of allocating groups and a one-in-six chance of getting it right.

Linda

[fls]

Originally Posted by fls

I should add that it may be as Rolfe pointed out - that the researchers were given information about which subjects received the same treatment (without knowing what that treatment consisted of). Then it becomes a matter of allocating groups and a one-in-six chance of getting it right.

Linda

It does say that they were blind as to group allocation and remedies chosen.

So who told the Materia Medica expert to look for symptoms specific to Ars Alb and Natr Mur if none of the researchers had this information?

Linda

[Badly Shaved Monkey]

Originally Posted by shpalman

Walach is the editor.

ETA: I hope my blog post won't prejudice matters...

Oh, yes!

That's even more funny.

[shpalman]

Originally Posted by fls

It does say that they were blind as to group allocation and remedies chosen.

So who told the Materia Medica expert to look for symptoms specific to Ars Alb and Natr Mur if none of the researchers had this information?

Linda

Well, that could have been the independent homeopathic pharmacy who did that. But once all the symptoms are written down maybe it doesn't matter so much. The symptoms are going to come back from MM labelled with remedies anyway.