Double blinded homoeopathic trial 'proves' provings

[Yuri Nalyssus]

The journal Homeopathy has published the following paper:

G Dominici, P Bellavite, C di Stanislao, P Gulia and G Pitari; Double-blind, placebo-controlled homeopathic pathogenetic trials: Symptom collection and analysis, Homeopathy (2006) 95, 123–130. The abstract and links to the full paper can be found here: http://tinyurl.com/z3pfc.

The abstract reads:

Quote:

Background: Homeopathic pathogenetic trials (provings) are fundamental to homeopathy. Since most of the data from available provings have not been statistically evaluated, it is unclear how specific reported symptoms are and how they differ from those reported by people taking placebo. Method: We combine and analyse data from two different homeopathic pathogenic trials—including 10 and 11 provers, respectively, and both including 30% placebo—to test the null hypothesis that there is no significant difference between the number of symptoms in placebo and verum groups. Results: The principal results were: Placebo reported less symptoms than verum groups. Symptom distribution according to predefined classes (common symptoms increased in intensity and/or duration, cured, old, new and exceptional) was statistically different between placebo and verum group at a high level of significance (Po0.001). Compared to verum, placebo provers reported less new and old but more common (increased in duration or intensity) symptoms. Within repertory categories, other differences were detected. The two groups differ in terms of the duration of each symptom and kinetics of symptoms: most symptoms were more persistent in verum than in placebo groups and verum provers recorded a decreasing number of symptoms with time. Placebo provers did not show such a temporal pattern. Conclusions: If confirmed by other studies these results would demonstrate the nonequivalence between homeopathic medicines in high dilution and placebo and contribute to the improvement of proving methodology and evaluation. Homeopathy(2006) 95, 123–130.

This is just the sort of paper that is trumped by homoeopaths as proof that it works. Reading it, it seems that the trial was properly blinded, and has good confidence intervals and it seems to show that subjects taking the remedy showed more and varied symptoms than those receiving placebo.

Whether or not it is possible to distinguish remedy from placebo is an acid test of homoeopathy so this paper is of particular interest. Can anyone with a knowledge of statistics or who is more familiar with reading research papers than I am say what (if anything) is wrong with this trial, or is it really proof of the validity of homoeopathic provings as the authors claim.

Yuri

[geoman]

Alas, I have no subscription to the worthy journal and am certainly not shelling out 30 bucks for the paper. BUT...

do I understand the term "provers" correctly as "subjects"?; were there only 10 & 11 subjects for each trial? Doesn't seem much, especially to get a 99.9% significance level.

Also I'm not sure how to square "Compared to verum, placebo provers reported ... more common (increased in duration or intensity) symptoms" with "most symptoms were more persistent in verum than in placebo groups". To my non-medical brain, a more persistent symptom would be one which lasted longer.

p.s. Yuri, Freudian slip in saying that "it seems to show that subjects taking the placebo showed more and varied symptoms than those receiving placebo"?

[Jorghnassen]

Who peer-reviewed this? This is one of the worst abstract I've ever seen. Plus, if placebos reported less symptoms than treatment groups, wouldn't that mean that placebos were more effective than homeopathic treatment? That's just pathetic.

[Mojo]

Quote:

We combine and analyse data from two different homeopathic pathogenic trials—including 10 and 11 provers, respectively, and both including 30% placebo

Haven't read the article yet, but 30% of 21? They gave a placebo to 6.3 of the subjects?

[Mojo]

Originally Posted by Jorghnassen

Plus, if placebos reported less symptoms than treatment groups, wouldn't that mean that placebos were more effective than homeopathic treatment? That's just pathetic.

No. In proving tests, the remedy is given to healthy subjects to see what symptoms it causes in them.

The idea is that the remedy can then be used to treat patients suffering from similar symptoms to those that the remedy causes in healthy people (crazy huh?).

[Yuri Nalyssus]

Originally Posted by geoman

...do I understand the term "provers" correctly as "subjects"?; were there only 10 & 11 subjects for each trial? Doesn't seem much, especially to get a 99.9% significance level.

Yet that's what they claim to achieve, so presumably the difference between the 2 groups must have been very obvious.

Quote:

p.s. Yuri, Freudian slip in saying that "it seems to show that subjects taking the placebo showed more and varied symptoms than those receiving placebo"?

Oops, many thanks, mistake now corrected, I have an overdeveloped ridicule centre, obviously it's started to express subconsciously.

Yuri

[Mojo]

Originally Posted by Yuri Nalyssus

Yet that's what they claim to achieve, so presumably the difference between the 2 groups must have been very obvious.

Do they know about the Million Dollar Challenge?

[Kopji]

How about a test where the evaluators apply their statistics to the entire test group, and then precisely identify the placebo group as inferred from their criteria?

[Capsid]

OT: I just noticed your avatar Yuri. I'm not sure I understood that film but I did enjoy it.

[Yuri Nalyssus]

Originally Posted by Jorghnassen

Who peer-reviewed this? This is one of the worst abstract I've ever seen.

As far as I know 'Homeopathy' claims to be a peer reviewed journal but since the 'peers' would be homoeopaths it leaves itself open to accusations of publication bias. Nevertheless the figures apparently speak for themselves. Other trials show no difference between remedies and placebo so what is going on with this one - are the findings true or coincidence or are we only hearing the partial truth about the method?

Yuri

[CFLarsen]

Quote:

Placebo reported less symptoms than verum groups.
...
Compared to verum, placebo provers reported less new and old but more common (increased in duration or intensity) symptoms. Within repertory categories, other differences were detected. The two groups differ in terms of the duration of each symptom and kinetics of symptoms: most symptoms were more persistent in verum than in placebo groups and verum provers recorded a decreasing number of symptoms with time. Placebo provers did not show such a temporal pattern.

Note the very vague wording. "Most". "Less". "Increased". "Differ". But we don't hear how much it is. We also don't hear what kind of symptoms.

Sniff test.

Originally Posted by Yuri Nalyssus

Reading it, it seems that the trial was properly blinded, and has good confidence intervals and it seems to show that subjects taking the remedy showed more and varied symptoms than those receiving placebo.

Since you read the paper, perhaps you can fill us in on some numbers?

[Yuri Nalyssus]

Originally Posted by Capsid

OT: I just noticed your avatar Yuri. I'm not sure I understood that film but I did enjoy it.

One of studio Ghibli's finest. I put the avatar up a few weeks ago, No-face seemed strangely appropriate; and I do have a tendency to wander around with a blank look on my face saying 'uuh?'.

Yuri

[jon]

My library's apparently paid for electronic access and it looks like they allow me to e-mail out a copy. Don't much fancy reading this closely (and my stat's isn't great) if someone wants to PM me, though, I can e-mail out (one) copy. First come first served

[Yuri Nalyssus]

Originally Posted by CFLarsen

Since you read the paper, perhaps you can fill us in on some numbers?

I'll have a go.

Yuri

[Yuri Nalyssus]

Originally Posted by Mojo

Do they know about the Million Dollar Challenge?

I'm sure if asked they would decline on the basis that they wouldn't want to sully themselves with such folderol. (That's what the usual line is I believe)

Yuri

[flume]

They had 11 in the first trial - 3 were placebo. They had 10 in the second trial; 3 were placebo. When they did the analysis they combined the 3 placebo provers in each trial into a group of 6. They said they could do this because they wrere not analyzing specific symptoms.

The placebo provers were picked randomly from the group of 11 or 10.

I have a question - were the same people used as provers for the two tests. I think probably not, because the provers had to have taken no homeopathic remedies for 6 months, so that would mean a 6 month wait between trials. And probably it is implied. But they don't explicitly say they were not.

They do have a table listing symptoms under a long list of different homeopathic categories (e.g. mind (placebo:34 symptoms (31%); etna lava:94 symptoms (25%); hydrogen peroxide:35 symptoms (19%) (percents rounded by me), generals, head, eye, ear,... stomach (P:6(6%); EL:26(6%); HP:16(8%)....through extremities, fever, skin, back.

I think a few hypersensitive or more talkative people in the group provers of the verum remedies could throw the whole thing off (resulting in more symptoms and more interesting symptoms in that group), so that they should have repeated it with the same people who had previously gotten the remedies getting the placebo and vice versa.

[CFLarsen]

Originally Posted by Yuri Nalyssus

I'll have a go.

Yuri

Thanks.

How many people were tested?

[CriticalThanking]

I can't tell from the chart - what were the homeo products used? Etna lava and Hydrogen Peroxide, but in what strength: 30C, 1X, or something in between? Obviously at 1X there is something besides solvent present and there would be some real effects. At 30C, there should be no difference with Placebo. What are they claiming?

CT

[flume]

In the first trial (Etna lava) there were 11 people: 8 proved the etna lava; 3 proved the placebo.
In the second trial of hydrogen peroxide (or whatever their fancy name for it is) there were 10 people. 7 proved the hydrogen peroxide; 3 proved the placebo.

For their analysys they lumped the 3 placebo provers from the two trials, so they had 8 for Etna Lava, 7 for hydrogen peroxide, and 6 for placebo.
(by sex, 5/3. 4/3. 4/2 female/male)
age: EL: mean 41years(30-54); HP: mean 37 yrs (26-48); P:mean 38 yrs (30-45)

[CFLarsen]

Originally Posted by flume

In the first trial (Etna lava) there were 11 people: 8 proved the etna lava; 3 proved the placebo.
In the second trial of hydrogen peroxide (or whatever their fancy name for it is) there were 10 people. 7 proved the hydrogen peroxide; 3 proved the placebo.

For their analysys they lumped the 3 placebo provers from the two trials, so they had 8 for Etna Lava, 7 for hydrogen peroxide, and 6 for placebo.
(by sex, 5/3. 4/3. 4/2 female/male)
age: EL: mean 41years(30-54); HP: mean 37 yrs (26-48); P:mean 38 yrs (30-45)

11 people? Two groups, of 8 and 3?

Are you sure??

[Yahzi]

Originally Posted by Yuri Nalyssus

One of studio Ghibli's finest.

My 7 year old neice broke down in tears when the boy turned into a dragon.

I think she had a crush on him, and realized at that moment that he was a river, not a boy.

[tkingdoll]

Originally Posted by Yahzi

My 7 year old neice broke down in tears when the boy turned into a dragon.

I think she had a crush on him, and realized at that moment that he was a river, not a boy.

Well he was kinda hot...

Not as hot as Howl though.

[Yuri Nalyssus]

Originally Posted by CriticalThanking

I can't tell from the chart - what were the homeo products used? Etna lava and Hydrogen Peroxide, but in what strength: 30C, 1X, or something in between? Obviously at 1X there is something besides solvent present and there would be some real effects. At 30C, there should be no difference with Placebo. What are they claiming?

I'll attempt a precis.

Quote:

The authors studied data from 2 provings; symptoms were collected from the two provings, done on a small number of subjects using the same procedure, and were considered together in order to increase the significance of results.

The remedies tested were: 1] potentized Etna Lava, made from lava collected during the August 2000 eruption (volcanoes are known as sources of useful but poorly understood remedies), and 2] potentized H2O2 (hydrogen peroxide, Hydrogenium peroxidatum): (this molecule is a reactive oxygen species (ROS), responsible for tissue injury with consequent disease if not efficiently detoxified; ROS have been implicated in over 50 diseases and in the ageing process (They’d obviously never heard of these people , but that’s another story)). Both remedies were taken in 30CH potency (that’s a dilution of 1 in a hundred done 30 times) three times daily for a maximum of 2 days.

The authors state the design was a “double blind, randomized, multicentric, placebo controlled experimental study”. the trials lasted 2 months each, the 2 groups numbered 11 and 10 provers respectively. In the lava group 8 out of 11 took verum and in the H2O2 group 7 took the verum, the rest took placebo (3 in each group). Participants in the trial were the provers and their supervisers (both blinded and the authors state the verum was indistinguishable from the placebo by sight, smell or taste). The coordinators numbered the preparations and gave them to the supervisors who then distributed them blindly and randomly to the provers. Each prover had kept a diary of symptoms in the lead up to the trial and continued it while taking the test substance. The analyst received the final data from the coordinator and compared the 2 groups receiving verum with the placebo groups which were pooled to increase the statistical power.

In the discussion the authors state “The study has to be considered as preliminary, because of the limited number of subjects, but it could be an important example of proving methodology and evaluation” and later “Distribution of symptoms by predefined classes... showed that placebo and verum provers represent two different groups: the differences cannot be explained by chance alone”.

What they claim is "we have shown a particular difference between placebo and verum in pathogenic homeopathic trials" although they do go on to say more extensive studies are needed.

Re-reading the paper I see the analyst was given the results by the coordinator who was responsible for the labelling of the trial substances, a bit of a weak spot perhaps.

My problem is that it's tempting to just dismiss the whole paper on the grounds that "it can't be right, they must have made some slip up and they're just not telling us". If that were the case though, the same could be said for any paper no matter what the subject. This paper has the appearance (at least to me) of a genuine scientific trial, unlike the many howlers that are passed off as research by homoeos just desparate for positive evidence. As such it is inevitably going to be used as a stick to beat sceptics with and I'd like a few reasons why it shouldn't be taken at face value.

Yuri

[Mojo]

Quote:

...useful but poorly understood remedies

50% ain't bad...

[joller]

Originally Posted by Yuri Nalyssus

As such it is inevitably going to be used as a stick to beat sceptics with and I'd like a few reasons why it shouldn't be taken at face value.
Yuri

Scientific studies need to be repeatable.
get the same remedies that thery've used in similar potencies (as long as they're above avogadro's limit, otherwise it really isn't homeopathy per se), round up some peopletogether (looks like you don't really need that many), and verify the results.
If you get what they get, then .... then I'll ask someone else to confirm it
If It still works, and we don't detect some protocol roblems in the meantime, then I'm gonna say:
"Holy smokes, Batman!
Take me to the water, throw me in the river, I'm converted!"

[Zep]

I've just had a quick run through the figures in that table, and it raises some issues for me. Perhaps wiser heads than I can clarify...

1) The numbers in the table did not seem to be derivable from the data provided. Then I realised that it what we were shown was actually a tally of symptoms per category across all subjects. What would have been more useful would have been the per-subject tallies of symptoms. Because with such a small test population, it would take only one or two subjects to cause a severe anomaly in the overall tallies.

2) I calculated the average symptoms per subject from the table data for the placebo and the two verums, and plotted them (sorted lowest to highest to give a nice graph!). The results were very similar rankings - the categories tended to rank in the same order, and with similar proportions to the whole, for all three test products. This would indicate the subjects all recorded proportionally similar counts per categories of symptoms, regardless of the test substance - a trend that needs to be taken into account.

3) I then plotted the symptom categories as a percentage of total symptoms. This was a first basic attempt at normalisation, really. To simplify (and I allow this may be statistically invalid), I summed the two verum sets to give one "total verum" result per category. When plotted per category and compared with the placebo numbers similarly treated, the results appeared by observation to be even more similar, both per category and by absolute value. I would certainly like to see some hard statistical calculations on that side.

4) It is worth noting that over a 60-day period with 21 subjects involved, a total of 674 symptoms were recorded - just under 2 symptoms per day per subject. Given the totally subjective nature and effectively uncontrolled environment for the substances under test, there would need to be at least baseline measurements taken first to establish initial trends. While the report says these were done, they do not seem to be mentioned in the subsequent results. I suspect if baseline measurements were factored in, the results would be even more similar.

As a comment, I note that the Etna lava verum results are parallel to the others in percentage terms, but stand out in absolute terms - almost as if someone had just gone through and multiplied the base data by a constant, or added a singularly suspect data set for one subject... Either that, or the blinding may have broken in some way. Certainly the H2O2 results were almost indistinguishable from placebo. I'd be calling shenanigans on that one...

[Badly Shaved Monkey]

What Zep said.

Also, it is not clear that the 'analyst' was blinded. And, do you know what, I'll bet they weren't, even if the formal protocol says they were.

The group sizes are unbalanced, so even if they were labelled X and Y, the analyst would know that the larger group was verum.

Also, to get P=0.001 from that small number of subjects is unlikely. I'll bet they did their stats on the n of symptom numbers, 674 or whatever. BUT, these data are not independent. As has already been pointed out, uneven distribution of garrulous subjects would bias the results. Technically, this means that there are large correlations among groups of data points that would competely invalidate any stats that assume they are independently sampled.

Also, the abstract completely fails to indicate how many statistical tests were done and whether the analysis was defined before the study. If you take a large pool of data and keep running tests on it you are very likely to create some low p-values.

It's a classic cheat to get something like this published then tuck away in the small print that it is only a 'pilot' or similar. For goodness sake, this wasn't even a new piece of work. This should have been done behind closed doors as a paper exercise that would generate reasonable hypotheses to be tested properly. Instead they have shoved it out for publication where it adds yet one more item to the list of 'proofs' that homeopathy works. Yet again, we have to point out who the 'peers' are.

Also, bear in mind that this 'novel' study was performed on data that had already been shoehorned into shape for publication. Publication bias is likely to be a heavy factor.

Enough "also". It's Classic Bad Science.

Apart from all that it's a brilliant piece of ground-breaking work.

[Yuri Nalyssus]

Originally Posted by Badly Shaved Monkey

...it is not clear that the 'analyst' was blinded. And, do you know what, I'll bet they weren't, even if the formal protocol says they were.

I can't see anywhere in the paper where it says the analyst was blinded and he/she was supplied with the data by the "coordinator" who was the person who numbered and labelled the original test substances.

Quote:

The group sizes are unbalanced, so even if they were labelled X and Y, the analyst would know that the larger group was verum.

ah hah!!

Quote:

Also, the abstract completely fails to indicate how many statistical tests were done and whether the analysis was defined before the study. If you take a large pool of data and keep running tests on it you are very likely to create some low p-values.

This is the point in the paper I started to get lost:

Student t-test was used to compare data means (number of symptoms/prover) between two groups.
When data (symptoms) were distributed into categories (classes) a bivariate tabular analysis was performed and the χ2 test applied.
Confidence interval were analysed when permitted by the amount of data.
Cramer's V coefficient was calculated, when permitted, for the distribution with largest χ2.
The ‘null’ hypothesis assumed homeopathic potencies to be identical to placebo. If this was correct:
the two groups (verum and placebo) should provide a similar number of symptoms/prover;
the two variables (symptoms/classes) and groups (placebo or verum) should be independent, and symptoms distributed into the considered categories in numbers similar to the expected values;
Cramer's V, (a measure of the degree of association between the variables in the bivariate table) should be zero;
the categories of symptoms (physical, mental etc) and the time course of their appearance in the two groups should be similar.
When a statistically significant difference between data was observed by χ2 analysis, further statistical control of homogeneity was performed. The test for homogeneity answers the proposition that several individuals into a population are homogeneous with respect to some characteristic.

The problem for the lay person (ie me and more than likely most homoeos) reading this stuff is that all the stats sound very impressive even though one hasn't the faintest idea what it all means. The difference is whether to seek clarification or to swallow it hook, line and sinker. I know there are similar, larger studies out there including ones done by homoeopaths which suggest there is no difference between placebo & remedy but this one left me wondering for a while - starting to see things more clearly now though.

Yuri

[Badly Shaved Monkey]

Yuri, my eyes began to glaze over a little as well!

As far as I can see, though, if they have used chi-squared tests they have been using 'counts' of symptoms entered into the cells of their tables, but each subject potentially contributed numerous counts, which is exactly the lack of independence I suggested. However they collected their data, the only legitimate way to start doing chi-squared tests would be if they turned each subject into a count of 1, e.g. by showing some number or type of symptoms beyond a certain threshold, but of course that criterion would have had to have been predefined before the analysis. Sadly a chi-squared table with such a small number of subjects would be incredibly low-powered.

As is always the case with bad science like this, there is no way you can definitively disprove what they state as their conclusions. The data are simply too poor to make any statement one way or the other and should not have been published. but, take a politically motivated house journal and a scrutiny panel of homeopathic peers and you have a paper and another abstract to circle in the world of woo.

[Zep]

After more fiddling about with Excel...

Taking each category's symptom-count as a percentage of total symptom-count (see my point (3) above), I did a simple correlation coefficient calculation (i.e. Excel's built-in function!) of the two arrays of points. Result? R = 0.8966. And according to the bumph I have managed to unearth from my uni stats course and other places (see below), that means a strong positive correlation between the two data sets.

This suggests again that the test substance has little bearing on the shape of the data. There is merely a tendency for the test subjects to log more of the same symptoms, not any different pattern to the results.

Or put another way: The verums seem to make the provers notice and log a higher count of "normal placebo-type symptoms". Which is a unusual situation in that it indicates the provers knew when they were being given the verum (so they were aware they were supposed to be looking for more symptoms), but not what it was supposed to produce in the way of symptoms (so they logged more of the same). If so, that would clearly indicate a break of blinding, by way of knowing when they were proving a verum, but not what with (or they would have known which new symptoms to "look for").

Comments?

http://sportsci.org/resource/stats/correl.html
http://mathbits.com/MathBits/TISecti...orrelation.htm
http://www.uwsp.edu/psych/stat/7/correlat.htm

[Badly Shaved Monkey]

With such a small number of subjects it could stll be due to have more imaginative subjects in the verum group because n is too small to allow good randomisation. I mean, in the placebo group we have n=3, twice. You can barely do stats on that.

This is what it all boils down to- n was tny and unbalanced.

[Zep]

Originally Posted by Badly Shaved Monkey

With such a small number of subjects it could stll be due to have more imaginative subjects in the verum group because n is too small to allow good randomisation. I mean, in the placebo group we have n=3, twice. You can barely do stats on that.

This is what it all boils down to- n was tny and unbalanced.

All well and good - I agree.

But my discussion was more to do with interesting point that it would appear, even from these tiny set of numbers, that the provers did indeed know WHEN they were being asked to prove a verum and not a placebo, but not what the verum was they were testing. My only reasonable conclusion is that the provers of verum and placebo knew which group they were in beforehand, i.e. nowhere near properly blinded testing.

[Badly Shaved Monkey]

Originally Posted by Zep

All well and good - I agree.

But my discussion was more to do with interesting point that it would appear, even from these tiny set of numbers, that the provers did indeed know WHEN they were being asked to prove a verum and not a placebo, but not what the verum was they were testing. My only reasonable conclusion is that the provers of verum and placebo knew which group they were in beforehand, i.e. nowhere near properly blinded testing.

You mean they might not have run the trial competently? Oh, say it isn't so. My illusions are all shattered.

[Zep]

Originally Posted by Badly Shaved Monkey

You mean they might no have run the trial competently? Oh, say it isn't so. My illusions are all shattered.

[Jorghnassen]

Originally Posted by Mojo

No. In proving tests, the remedy is given to healthy subjects to see what symptoms it causes in them.

The idea is that the remedy can then be used to treat patients suffering from similar symptoms to those that the remedy causes in healthy people (crazy huh?).

OK... I guess I'll have to dilute my critical thinking skills a few Cs before I can understand the mind of an homeopath.

[Yuri Nalyssus]

Originally Posted by Jorghnassen

OK... I guess I'll have to dilute my critical thinking skills a few Cs before I can understand the mind of an homeopath.

Oh yes, because then you would become a super powerful, potentised sceptic [fiendish laughter]

Provings are described in a sceptic site here, with a 'pro' view here.

The Berlin wall is one of my favourite remedies although the Peregrine falcon comes a close second, just above anti-matter (what else would you use to treat the effects of a warp core breach). Now you see why this is so much fun!

Yuri

[Dr Adequate]

Hold on.

As I understand it, the point of "proving" a substance is that you give the subject a significant (ie non-homeopathic) dose of some substance and see if it makes him ill.

Well, why shouldn't it?

If they tried "proving", say, LSD against placebo, they'd see a huge difference. Statistically significant? You betcha. But it wouldn't go any way towards proving that sub-molecular dilutions of LSD are a cure for hallucinations.

So if their results were statistically significant, and they don't seem to be, then all they'd have proved is that eating Mount Etna is bad for you.

[Yuri Nalyssus]

Originally Posted by Dr Adequate

Hold on.

As I understand it, the point of "proving" a substance is that you give the subject a significant (ie non-homeopathic) dose of some substance and see if it makes him ill.

Well, why shouldn't it?

The trend these days is to use 30 centessimal dilutions for provings. I guess it's safer and saves people from having to eat peregrine falcons and chew bits of the Berlin wall (see above).

Sometimes even more unlikely methods are used such as meditating and dreaming about the remedies for long periods; 2 years according to one group.

Why am I explaining this to a potions professor?

I'm afraid they have an answer for everything.

Yuri

[Zep]

Dr A, the study was using 30C remedies of Etna lava, and hydrogen peroxide as the verums (pl. vera?). That is, they were diluted 1:100 thirty times over.

[Dr Adequate]

So they "prove" the substances by showing that homeopathic doses of them are bad for you?

Er ...

Haven't they just proved that homeopathic medicine makes you ill?