Dowsing & Randi's Latest SWIFT Newsletter

[Wolfman]

Okay, let me start by saying that I'm a fan of James Randi, and a huge supporter of what he does. But I consider it important to regard his statements and claims with an equal degree of skepticism to what I would regard the claims of anyone else.

In the latest SWIFT newsletter, he has this to say about dowsing tests conducted in Australia as part of his challenge:

Quote:

There are several facts about which this gentleman – and Simon Welfare, who I would have thought might have contacted me about the matter – are unaware. First, the dowsers themselves – not we – insisted that the results of all their attempts should be combined; they hoped to make the strongest possible case, and had agreed to equally divide the A$50,000 prize if they won it. In such matters, the applicants’ needs are always met when they cannot compromise the protocol or the results of the tests. Second, because of the perceived anomaly, those same dowsers were once again tested, this time in Perth, Australia, very shortly after the tests cited, because they felt they would do better in that location. Though I knew that this additional test would be done, and I approved that process, I had to return to the USA, so could not be present. They failed spectacularly, obtaining results that almost exactly mirrored the Sydney tests in reverse, so that the overall testing procedures showed null results.


However, Mr. Hart-Davis, quoting the Fairley-Welfare book, describes the 22 per cent result (the chance expectation was 10%) as being “well above chance,” without taking into account or even citing the number of trials conducted. The significance of that number is critical to making any conclusion, and it was not high enough to cause any wonder.


In any case, it is not proper to select out any section, portion, or aspect, of a set of tests to establish a significant result, unless, of course, the intention to do that is announced in advance. Though that method may be used for preliminary procedures that will help to design a final confirmatory test, it is not applicable unless it's so announced. Tossing a coin, one cannot select out a sequence of six “heads” to show a perfect score, if the goal was to obtain “heads.” That’s called, “data-selecting,” and it is the one most often-invoked no-no in amateur testing procedures…

I have two problems with Mr. Randi's 'defense' of his own claims in this regard, based on his own statements and arguments.

1) He begins by defending the numbers he publicly presented by saying that, "the dowsers themselves – not we – insisted that the results of all their attempts should be combined". However, that in no way meant that he himself could not release more complete figures -- figures which would have presented a more accurate picture of the actual results. In fact, if it had turned out that splitting the findings would have made the results look more questionable, I'm sure he would have done just that. However, in this case, since combining the results made it look less credible, he only cited those results, and did not present more specific results on each individual study until challenged to do so.

I find it difficult to see how this is any different from what those in the wonderful world of woo will do -- to present results in the manner which most favors their argument, even if (or particularly if) another more detailed explanation of those facts may weaken their claim.

2) Mr. Randi also says, "The significance of that number is critical to making any conclusion, and it was not high enough to cause any wonder", and goes on to a comparison with flipping coins, and data mining. Now, I appreciate his point -- but if this is, in fact, the case, why did he even bother conducting this test? Even if the results had shown a significant result to support the dowsers' claims, would he not have responded by simply saying that it was not statistically significant, and the test needed to be done again?

There are constant complaints that Randi's million dollar challenge is rigged, and would this not seem to support such a claim? He agreed to a test in which he stated in advance that he would pay a reward if the dowsers were able to meet a certain criteria; but then he turns around and states that such results would be essentially meaningless with such a limited number of samples to be drawn from.

Please understand, I'm no defender of woo. But if James Randi is going to appoint himself as the debunker of fraud, it seems to me that he should be held to at least an equal standard (if not higher) than that to which he holds those he criticizes. In this case, I really have questions as to whether he does that or not. While I admit that my understanding of this particular case is limited (which is why I am writing this, to gain more explanations and understanding of the actual situation), from what has been written, it seems to me that in one single article he has managed to demonstrate a willingness to print only those results that most benefit his own arguments, and to weaken the claim that his million dollar challenge is fair.

[GzuzKryzt]

In every JREF Challenge claim which proceeds to a test, there is a protocol. Said protocol defines success and failure, i.e. sets the standards.

But I do somewhat agree with you, Wolfman, the data publishing on the "James Randi in Australia" episode could have been handled in a more effective and transparent manner. Is it that hard to put together a chart?

[Wolfman]

But is Randi here not effectively arguing that, in a test in which he had agreed to the protocol, and agreed to pay the money if they met his criteria, it was in fact was not statistically relevant, and that more tests would have been required even if they had pulled it off?

It seems to me -- from the way I read it -- that he is arguing that the number of 'examples' in the first case were not enough to draw statistically significant conclusions. That many more such tests would have to be done, and the results confirmed in subsequent tests, before any conclusion could be reached as to the validity of such data.

Yet he had agreed -- again, from the way I read it -- to pay out the prize money if they had succeeded in their claims in only this one test. There was no stipulation that if this test were successful, subsequent tests would need to be conducted in order for it to be considered statistically significant.

Why would he agree to pay out money on a test which he subsequently states would not yield results that could be considered statistically significant?

Just trying to understand

[Gr8wight]

Originally Posted by Wolfman

Just trying to understand

Understand this: none of this discussion matters in any way. The dowsers themselves set the bar at 60% success rate. Had the actual success rate been 40% - dramatically above chance - they would still have failed the test.

The demonstration was not a scientific experiment, so significantly above chance was not the determining factor. The applicants set their own bar, and then failed to clear it. Case closed, they fail. Do not pass Go, do not collect one million dollars.

[Wolfman]

Sorry, but that doesn't answer my question at all, and does not render my question invalid. If they had passed the test, according to their own standards, would they have been paid? Or would they have been told that in such a limited test, the results were statistically insignificant? Because Randi's own argument above seems to be that the number of samples used was not enough to draw a reliable conclusion from.

People who do make an attempt to get the prize aren't asking themselves "What will happen if I don't pass the test?". They are asking, "What will happen if I do pass the test?". When James Randi agrees to a particular test, then afterwards declares that the results were statistically meaningless (regardless of whether or not the participants passed or not), I think it raises a valid question as to the validity of the entire challenge.

All I am doing is seeking clarification.

[robinson]

Originally Posted by Wolfman

All I am doing is seeking clarification.

For this, you will be punished, without mercy.

[Wolfman]

Originally Posted by robinson

For this, you will be punished, without mercy.

Cool

[ben m]

Quote:

It seems to me -- from the way I read it -- that he is arguing that the number of 'examples' in the first case were not enough to draw statistically significant conclusions. That many more such tests would have to be done, and the results confirmed in subsequent tests, before any conclusion could be reached as to the validity of such data.

If they actually had the powers they claimed to have, then the original test have been sufficient. Getting 60% right, just once, was (by agreement) the threshhold which "real dowsers" were said to pass every time, and which (by standard statistics) random-guessers would pass less than one time in a million.

If they didn't get 60%, but rather 22%, what can Randi say? He can't say "I have conclusively proven that these dowsers are just guessing". The statistics can't rule out the possibility that they have weaker powers---but designing a one-in-a-million test for a *weak* power requires many more trials than a one-in-a-million test for a *strong* power. If they came in saying "We have a weak power, about 2x above chance" they could have designed a test for that power. They didn't. The test they actually did, with only enough trials to confirm or refute a 6x-above-chance power, isn't sensitive to a 2x effect.

The statistical issues are quite well-defined mathematically. If someone came up to you and said, "I can predict coin tosses with 75% accuracy", would you toss four coins and see whether they got three of them? No---a random guesser will get that result one time in four. You don't want to give a million dollars to every fourth person who walks through your door, do you? No: if someone says they can predict coin tosses with 75% accuracy, you'd ask them to try their hand at 200 tosses, and then see whether they got 140-or-more right. If they got 105 right---which something like 30% of random guessers can do, too---you wouldn't change the agreement and award them a big prize for having amazing 52.5%-accurate psychic powers; you'd simply point out that they failed to demonstrate the promised accuracy.

[Starrman]

Originally Posted by Wolfman

Sorry, but that doesn't answer my question at all, and does not render my question invalid. If they had passed the test, according to their own standards, would they have been paid? Or would they have been told that in such a limited test, the results were statistically insignificant? Because Randi's own argument above seems to be that the number of samples used was not enough to draw a reliable conclusion from.

That's why Randi doesn't judge the test - the results are self-evident. If Randi agreed that 60% was the success rate, and they hit 60.1%, then they would be paid. So far, nobody has hit what they claimed to be able to hit in a test. The argument Randi is having with the dude is not weather or not the dowsers should have won, it is weather there were significant results to be found within a failed demonstration to warrant further scientific tests.

Originally Posted by Wolfman

People who do make an attempt to get the prize aren't asking themselves "What will happen if I don't pass the test?". They are asking, "What will happen if I do pass the test?". When James Randi agrees to a particular test, then afterwards declares that the results were statistically meaningless (regardless of whether or not the participants passed or not), I think it raises a valid question as to the validity of the entire challenge.

All I am doing is seeking clarification.

Randi doesn't do this - he and the claimant agree beforehand what will constitue a success. The bar is clearly set before hand - either you go over, or you don't. The key is that it is agreed on before the test takes place - any talk afterwards is just talk.

[Wolfman]

Okay, those were the answers I was looking for

And it wasn't all that difficult Thanks!

[SimonD]

Wolfman - have you watched the show that Randi refers to? It is available from the home page. Their results are the same as if they guessed, both for dowsing for water and minerals. It may help you understand why Randi is so dismissive of the claims

I do understand where you are coming from though

[Wolfman]

Simon,

Oh, I understand completely the reason why he dismissed the dowsers' results...I'm not in any way trying to defend them, I think they're completely bogus. I just had questions regarding Randi's response to the issues that were raised, and how the situation was handled in general.

And I think they've been answered quite reasonably.

[SimonD]

Fair enough, just trying to help

[robinson]

Damn, so much for the punishment.