Who runs the tests? [Archive]

TheBoyPaj

15th August 2003, 02:24 AM

I emailed the JREF with the following question:

On your web site you state:
"The JREF does not involve itself in the testing procedure, other than helping to design the protocol and approving the conditions under which a test will take place"

Yet in the actual agreement which applicants must sign it states (point 10)

"the implementation and management of the challenge will be carried out by James Randi via the James Randi Educational Foundation"

These seem to be contradictory, and may put people off the challenge. A believer friend of mine states that since the actual agreement does not state your non-involvement in the tests, they would have no grounds to complain if they later discovered that you were directly responsible for the test's administration. Could this be clarified?

--------------------------------------------------------

I received an email from Randi this morning simply stating "there is no contradiction".

Of course I have replied asking for more clarification, but in anticipation that I might recieve an equally terse reply does anyone here know how these can be reconciled? Who would actually run the formal test?

The Mad Linguist

15th August 2003, 08:34 AM

"The JREF does not involve itself in the testing procedure"

vs.

"the implementation and management of the challenge "

One is talking about the testing proceudre. One is talking about the entire setup of the challenge. No contradiction.

TheBoyPaj

15th August 2003, 08:59 AM

Is the testing procedure not part of the implementation of the challenge?

Anyway, I have a reply from Randi confirming their non-involvement in the testing.

But that does not remove the possibility that a fraudster could excuse themselves from the challenge by claiming that there is nothing in the agreement which states that the test will be impartial.

I think for this challenge to be an effective display of the weakness of the believer's argument, it should be clear in the signed agreement that the test will be administered by independent scientists with both parties approval.

Beleth

15th August 2003, 12:52 PM

"other than helping to design the protocol and approving the conditions under which a test will take place" is the phrase that clears up the contradiction.

The JREF designs the test.
Someone else executes the test.

And you should consider yourself lucky. From what I've heard, getting an answer as lengthy and as polite as "there is no contradiction" from Mr. Randi is an unusual occurrence.

Anyway.

A key part of the protocol is to come up with test conditions that both parties agree to. There may be nothing in the agreement procedure which states that the test will be impartial, but there has to be an agreement in the first place for the test to take place, and if the person being tested agrees to an impartial test, well, that's his own fault.

TheBoyPaj

15th August 2003, 03:24 PM

I agree that if someone decides to take an impartial test then they cannot be surprised by the outcome, but my concern is that the value of the challenge is diminished by the fact that the agreement is not clear on this issue. The bit of blurb on the intro page states that the JREF is not involved, but the actual agreement, the bit the people sign, does not. The JREF could feasibly change the intro page at any time as it does not constitute part of the agreement.

I got more from Mr Randi when I asked the following:

------------------------------------

I am sure you are correct and I would like to point out that I tend to agree with you on this sort of thing, but would it be possible for you to clarify how the two statements sit together? Does the JREF run the tests, and if not what is the extent of your implementation and management of the tests?

------------------------------------

I got the following reply:

------------------------------------

The actual conduct of any test is performed by other persons, though we are often asked to advise. We assist in the design, and we must -- of course! -- approve the final protocol and the personnel who conduct the test. On occasion, with the agreement and/or the request of the applicants, we have conducted the tests.

James Randi

------------------------------------

I also asked the following:

------------------------------------

Thanks for your reply. I would like to recommend, however, that you include this addition in the signed agreement to prevent a fraudster from saying that they do not want to take the challenge because they fear you will be the person actually running the test. They don't trust you, as I am sure you are aware!

------------------------------------

I got the following reply from Randi:

------------------------------------------

I don't care whether they trust me or not, and I don't care if they want to get out of taking the challenge...........!

------------------------------------------

My reply:

------------------------------------------

But don't you agree that one of the purposes of your challenge, and the fact that no one has ever passed it, is that it demonstrates to anybody who remains undecided on the issue that all paranormal claims melt away in the light of proper research? At the moment it seems you have a get-out clause in your agreement, which diminishes that demonstration in the eyes of the genuinely interested observer. I can understand that you feel you have nothing to prove, but why have the challenge at all if it is so easy to dismiss on the grounds that it would not be fair?

------------------------------------------

His reply:

------------------------------------------

Let them argue with me....

James Randi

------------------------------------------

That's the extent of my correspondence in the issue. Mr Randi doesn't seem to worry that the agreement leaves this element of doubt. I suppose it is his challenge and he can run it any way he likes, but all a fraudster has to say is "I'm not taking the challenge because he might renege on the bit about not being involved" and there would be nothing in writing to say that wouldn't happen.

The Mad Linguist

15th August 2003, 04:00 PM

I think you're barking up a dead end here, if I may mix my metaphors. I suspect that the reason the JREF's non-involvement in the tests is not part of the agreement is exactly that some applicants do want to be tested by Randi himself (a lot of the tests he reports in his commentary are of this type).

Given that the test procedure must be agreed on by both parties, in advance, as a fair test of the claimed powers, I don't see that any other specific fairness measures than this need be enumerated.

I don't think there's any element of doubt.

TheBoyPaj

16th August 2003, 01:53 AM

Then the agreement should state:

All formal experiments will be conducted by independent researchers with the approval of both parties or, in certain circumstances, by a member of the JREF, again with both parties' approval.

Without this clause the following scenario could occur:

A claimant who is genuinely under the impression that they have a paranormal ability may wish to have a crack at the challenge. But the wording of the agreement does not preclude the possibilty that is will be an unfair test, administered by people who already have ruled out the possibility of your success. This person fears that they will apply, find out that they cannot agree on a protocol that they believe will be fair, and subsequently have to pull out. This now leaves Randi with a nice story to put in his weekly commentary about the fool from Detroit (or whatever) who believes he can control fish with his thoughts and the bloke becomes a laughing stock. Pulling out of the test is seen as an admission of failure almost as much as failing the test itself, even when all the person wanted is a fair test of their abilities and a chance to win a million.

Now, imagine we have a fraudster who is conning millions. In this case the challenge is a weapon for us, the sceptics, to say "look, this person is too scared to even take a simple test of his abilities. He's clearly a con-man."
After all, that's the real point of the challenge, isn't it? It's such a simple thing, but yet they cannot pass it.
But all this fraudster has to do is state the example given above, and say that they are not going to subject themselves to ridicule if they are forced to withdraw when they find out it will be Randi conducting the test. Bingo! They have a ready-made excuse.

Randi doesn't care about what excuses they come up with. He is certain of his beliefs and knows that they're all fake anyway. But he also goes on TV to tell others about this challenge, and to hold it up as a great shining example for the masses. But this flaw in the agreement makes it shine a little less brightly IMHO.

The Mad Linguist

17th August 2003, 02:01 PM

There's no response to this.

If the phrase "the test procedure must be agreed on by both parties, in advance, as a fair test of the claimed powers" doesn't explain it, then nothing will.

Fair means fair - in any way at all. Fair in terms of the protocol, fair in terms of the target results, or fair in terms of who's conducting it. If the test is unfair, the applicant doesn't take it - they work out one that is fair.

Peter Morris

23rd August 2003, 06:22 PM

The trouble with that is that if the applicant doesn't think it fair, Randi doesn't 'work out one that's fair', he tells them that's the test I'm offering, take it or leave it.

If they agree to take the test, then that is taken as an agreement that they think its fair, ignoring their statements to the contrary.

If they don't agree to the test, Randi claims this as a victory for himself.

For an example of such, see Randi's own description of the test for Chinese medicine here: http://www.randi.org/jr/122702.html

Note TRandi's failure to work out a test that is acceptable, his 'take it or leave it' attitude, and his declaration of victory when they leave it.

Now, if the test agreement gave them the opportunity to say << I don't think the test is fair, I don't think my power works like that, but its the only test Randi offered me, and I'm ready to try it anyway >> then maybe he'd get more people willing to try it.

The Mad Linguist

23rd August 2003, 06:59 PM

Originally posted by Peter Morris

Now, if the test agreement gave them the opportunity to say << I don't think the test is fair, I don't think my power works like that, but its the only test Randi offered me, and I'm ready to try it anyway >> then maybe he'd get more people willing to try it.

But why one earth would anyone want to take an unfair test?

If the text is unfair, and they pass it, it means nothing.
If the test is unfair, and they fail it, it means nothing.

That's the whole point of making tests fair in the first place!

I read Randi's report of the Qigong applicant somewhat differently to you. I don't think it was a case of him offering "just that test".

The applicant initially attempted to diagnose Randi from a photograph, and failed. However, this is not a scientifically designed test, so even if he'd got Randi's ailments right, it wouldn't mean anything. The applicant does not appear to have outlined any notion of testing of their own. So Randi proposed a test - quick, easy, and valid.

The applicant turned this test down. Note that the applicant's daughter, in her response, did not suggest any other way of testing the claim. In fact the only really clear statement is "my father could give the diagnosis for him or her", which is a restatement of the claim, not a method for testing it.

The ball is therefore in the Qigong fellows court, to make clear what would be an adequate test of his powers, if Randi's suggestion isn't. Now we don't know if the correspondance went any further. Maybe it did, maybe it didn't.

But in general, Randi's offer of the death-test does make one thing very clear to the applicant straight away: that they cannot pass by chicanery or by manipulating the human beings that judge them, that they will have to actually pass by producing objectively verifyable results. This is why it saves Randi time because of course, none of them can produce objectively verifiable results.

Peter Morris

23rd August 2003, 07:48 PM

The ball is therefore in the Qigong fellows court, to make clear what would be an adequate test of his powers, if Randi's suggestion isn't.

It seems that there is no question of that, Randi has set the test, he has refused to alter it, the subjects may either take the test, declring it to be 'fair', or they can drop out. Altering the test isn't on the cards.

kookbreaker

23rd August 2003, 07:54 PM

Originally posted by Peter Morris
[B]The trouble with that is that if the applicant doesn't think it fair, Randi doesn't 'work out one that's fair', he tells them that's the test I'm offering, take it or leave it.

Oh yeah, like with the Penta water people, where he forced them to [/QUOTE]use the testing method they suggested. (http://www.randi.org/jr/08-24-01.html)

(Of course, the Penta water folks never took the challenge)

If they agree to take the test, then that is taken as an agreement that they think its fair, ignoring their statements to the contrary.

If they don't agree to the test, Randi claims this as a victory for himself.

For an example of such, see Randi's own description of the test for Chinese medicine here: http://www.randi.org/jr/122702.html

Hardly a clear example. Theses folks were capable of suggesting a protocol. Randi suggested one he thinks was best. They refused the protocol, and did not follow up.

kookbreaker

23rd August 2003, 07:57 PM

Originally posted by Peter Morris

It seems that there is no question of that, Randi has set the test, he has refused to alter it,

Where did you get that? Randi seems to suggest that he feels his test is the best design, but they are free to suggest a test design with equal control. ("dead" is a remarkable control).

Peter Morris

23rd August 2003, 08:23 PM

"Where did you get that"

From Randi's description of the correspondance.

"Dead is a remarkable control"

I'm not certain, but my understanding is that they claim they can diagnose from a photo, but its not divination. I think they mean they can diagnose current conditions from a current photo, or given an old photo they can diagnose conditions when the photo was taken.

Giving them an old photo of someone, whose condition has changed significantly since the photo was taken, is not a fair test IMHO.

DISCLAIMER: I'm not supporting their claims at all, just pointing out certain flaws in Randis offer of a test. Please bear that in mind.

kookbreaker

23rd August 2003, 08:41 PM

Originally posted by Peter Morris
"Where did you get that"

From Randi's description of the correspondance.

"Dead is a remarkable control"

I'm not certain, but my understanding is that they claim they can diagnose from a photo, but its not divination. I think they mean they can diagnose current conditions from a current photo, or given an old photo they can diagnose conditions when the photo was taken.

Giving them an old photo of someone, whose condition has changed significantly since the photo was taken, is not a fair test IMHO.

OK, then they must suggest a protocol where there is a definate diagnosis without any potential weasel factor, a la the assorted Larry King psychics. I.e. there's no "I sense something in the liver area" without specifics and diagnosis. In other words, remove the cold reading.

There's nothing preventing them from making their own suggestions. The way I read it, Randi is not declaring victory but rather noting that they have refused one version of a controlled test.

Peter Morris

23rd August 2003, 09:30 PM

OK, then they must suggest a protocol where there is a definate diagnosis without any potential weasel factor Ah, but would Randi accept it if they did so? Randi's own account strongly suggests otherwise.

You may interpret it differently if you wish to, but that's my understanding.

Any further discussion of the subject would be a waste of time.

The Mad Linguist

24th August 2003, 04:43 AM

Originally posted by Peter Morris

It seems that there is no question of that, Randi has set the test, he has refused to alter it, the subjects may either take the test, declring it to be 'fair', or they can drop out. Altering the test isn't on the cards.

Erm, where exactly does Randi say this?

I agree with you that the suggestion of the death test is meant to discourage. It's meant to discourage by making proponents of silly claims realise exactly what level of scrutiny they're going to face if they persist with the challenge.

At no point does the applicant actually suggest a test that could be performed. So you're right that the death test is the only one on the table at the time Randi wrote, but that's because the applicant hasn't put forward any alternative.

I'm not certain, but my understanding is that they claim they can diagnose from a photo, but its not divination. I think they mean they can diagnose current conditions from a current photo, or given an old photo they can diagnose conditions when the photo was taken.

Giving them an old photo of someone, whose condition has changed significantly since the photo was taken, is not a fair test IMHO.

You'd be absolutely right - IF the claim actually is that the Qigong fellow can diagnose conditions that the subject had at the time the photo was taken. At no point does the applicant specify this, however. If he/his daughter HAD, then of course Randi wouldn't have suggested the death test.

--------------------------------------------------------------------------------
OK, then they must suggest a protocol where there is a definate diagnosis without any potential weasel factor
--------------------------------------------------------------------------------

Ah, but would Randi accept it if they did so? Randi's own account strongly suggests otherwise.

Please quote anything that Randi says in the commentary that suggests that he would reject the applicant from the Challenge if the applicant suggested a protocol without any room for the "weasel factor".

I think it more likely he'd be overjoyed if they did so, because it would mean he could actually investigate their claim, rather than exchanging emails with psychos, which seems to take up an awfully large proportion of his working hours.

Peter Morris

24th August 2003, 06:46 AM

Please quote anything that Randi says in the commentary that suggests that he would reject the applicant from the Challenge if the applicant suggested a protocol without any room for the "weasel factor".

The fact that they object to the test offered, and Randi's reply is essentially 'take it or leave it.'

That is my impression, you are entitled to a different impression if you wish.

The Mad Linguist

24th August 2003, 07:27 AM

Well, we'll have to agree to differ. To me it seems that Randi's response was (understandably) to pour scorn on the applicant's objection, which is NOT the same thing as saying "take it or leave it".

Note that he was open to hearing again from these people (though we don't know whether he ever did). If they did get back to him with a suggestion realistic protocol, then the Challenge would have to go ahead.

Peter Morris

24th August 2003, 12:09 PM

Pouring scorn on their claims is right, he does this a lot. His rudeness certainly adds to the impression that he is not going to conduct fair tests. I don't blame anyone who gives up on an attempt to negotiate a test with him.

The Mad Linguist

24th August 2003, 12:18 PM

Being rude doesn't make him wrong.

Nor does it make him incapable of conducting a fair test - particularly since the challenge is set up so that in many cases the actual testing is out of his hands.

If I had to deal with the amount of rubbish Randi gets in his inbox, I'd be harsh and abrupt as well.

And let's be clear - it's A MILLION DOLLARS on offer here. And you seriously don't blame applicants for giving up because Randi's mannerisms tend towards the brusque? Seems foolish to me, IF they can do as they claim. Personally I'd take all kinds of abuse if it was going to get me a million...

Peter Morris

24th August 2003, 02:47 PM

If they receive a rude response to their application, they are unlikely to trust Randi to conduct the test fairly. If somebody doesn't trust Randi to conduct the test fairly, then they won't submit to it, no matter how much the prize.

The Mad Linguist

24th August 2003, 02:52 PM

Then that's their problem.

If someone really believes that the fact that Randi does not suffer fools gladly makes him untrustworthy, despite the fact that he has laid out, very specifically, in a legal agreement, the EXACT provisions which ensure that anyone can trust the JREF to administer the Challenge is fairly... well, too bad for them.

kookbreaker

24th August 2003, 03:40 PM

Originally posted by Peter Morris
If they receive a rude response to their application, they are unlikely to trust Randi to conduct the test fairly. If somebody doesn't trust Randi to conduct the test fairly, then they won't submit to it, no matter how much the prize.

Ever talk to somebody whose gone through the Phd process? You wanna talk about rude? Randi is gentle by comparison. Keep in mind that those people are not dealing with bs claims either.

"Randi was rude" is an excuse. Since he need not be present for the test all they need to do is establish a protocol. Randi isn't there to hold their tender little hands.

Peter Morris

9th September 2003, 05:36 PM

The Mad Linguist sez: If the phrase "the test procedure must be agreed on by both parties, in advance, as a fair test of the claimed powers" doesn't explain it, then nothing will.

BTW, where exactly does it say this? It's not in the claim agreement : http://www.randi.org/research/challenge.html

Where does it state the challenge is fair?

Lord Kenneth

9th September 2003, 09:09 PM

Originally posted by Peter Morris
The Mad Linguist sez:

BTW, where exactly does it say this? It's not in the claim agreement : http://www.randi.org/research/challenge.html

Where does it state the challenge is fair?

Yeah it is... look at number 1.

Segnosaur

10th September 2003, 01:44 PM

Originally posted by TheBoyPaj
Then the agreement should state:

All formal experiments will be conducted by independent researchers with the approval of both parties or, in certain circumstances, by a member of the JREF, again with both parties' approval.

Just out of curiosity, why do you insist that the people performing the test meet with the approval of both parties?

Its Randi's (or, more exactly, the JREF's) million dollars, they are the ones risking money, so they should decide on who runs the test. Also, allowing the person being tested to have a hand in choosing who runs the test might lead to a deadlock, since an applicant could easily restrict his choice to someone who is 'biased', and if the JREF refuses, he can then claim "Oh they don't want a fair test".

TheBoyPaj

11th September 2003, 10:37 AM

Because at the moment it leaves a nice get-out clause for famous mediums like S B.

All she has to say is "I'm not going to sign it because I don't know who will end up running the test. It might be someone who is obviously biased against me, meaning I would have to pull out of the test. Then Randi will crow about how I reneged on the agreement and it will ruin my reputation."

Segnosaur

12th September 2003, 11:43 AM

Originally posted by TheBoyPaj
Because at the moment it leaves a nice get-out clause for famous mediums like S B.

All she has to say is "I'm not going to sign it because I don't know who will end up running the test. It might be someone who is obviously biased against me, meaning I would have to pull out of the test. Then Randi will crow about how I reneged on the agreement and it will ruin my reputation."

But they still have that 'get out' clause; its just that instead of Sylvia saying "I won't bother applying", she could apply, then say "I only want John Edward (or some other fraud) to do the testing". The JREF says no, then she can make an even worse claim... "The JREF backed out because they were afraid of my awesome power".

Rolfe

15th September 2003, 12:20 PM

Originally posted by TheBoyPaj
All she has to say is "I'm not going to sign it because I don't know who will end up running the test. It might be someone who is obviously biased against me, meaning I would have to pull out of the test. Then Randi will crow about how I reneged on the agreement and it will ruin my reputation."

OK, does she want the money or doesn't she?

If I thought I could do anything that would get me that money (and I don't by the way), I'd be busting a gut trying to get a protocol agreed. Not crawling round trying to find ways to back out of the challenge.

That's the neat bit. It's a real quick way to sift those who genuinely believe they have the power they describe from those who, consciously or subconsciously, know they're frauds. The frauds instantly identify themselves by going into defence mode.

This even works with very rich frauds to whom a million bucks is small change, or for some reason think the money is "tainted". Never mind the money. Don't take it. Give it to charity. Just refuse it. Concentrate on the publicity coup of winning, the newspaper articles, the validation of all your disputed claims. Me, personally, I'll never criticise you again. Just go pass the test, why don't you?

But all they ever do is whine about how the test is rigged.

Rolfe.

Paul C. Anagnostopoulos

18th September 2003, 05:32 PM

Paj, if I may be blunt, I don't think JREF gives a damn if some people excuse themselves from the test for various reasons involving the personality or negative vibe of James Randi.

~~ Paul

TheBoyPaj

18th September 2003, 05:35 PM

Could I ask what you think the point of the challenge is?

Randi's not just offering his money for nothing. What does he get out of the test's existence?

UnrepentantSinner

18th September 2003, 11:11 PM

Shouldn't you be down cheering David Blaine on?

The point of the test is to get someone... anyone who claim they have superpowers to demonstrate them in a controled environment. This has been explained several times. Why is it so hard to understand?

TheBoyPaj

19th September 2003, 12:15 AM

So, what meaning do you draw from the fact that no one has taken the test? Do you see that as a victory for the skeptic's cause?

(PS. London is dirty and crowded enough without the rest of the nation travelling down to watch a man do nothing in a box)

UnrepentantSinner

19th September 2003, 04:44 AM

Originally posted by TheBoyPaj
So, what meaning do you draw from the fact that no one has taken the test? Do you see that as a victory for the skeptic's cause?

(PS. London is dirty and crowded enough without the rest of the nation travelling down to watch a man do nothing in a box)

I see the fact that people aren't able to demonstrate their claimed superpowers under controlled conditions to mean that these people don't have superpowers. I just wish more people who supposedly have real superpowers would put them up to testing under controlled conditions so we can find out if they really do have superpowers or if they're delusional or frauds as well.

TheBoyPaj

19th September 2003, 10:08 AM

And that's the point. People who think they have supernatural powers are not putting themselves up for the challenge with the excuse that they don't think it would be a fair test.

Now, we assume that it's because they know they are frauds, but then we're hardline sceptics aren't we? We don't need convincing. But I don't think the point of the challenge is just to give us something to laugh about. It's there to act as an example for those people who need convincing. It forms part of an argument which goes "all they have to do is pass this test and they win a million dollars. Isn't it odd that {insert named psychic here} refuses to take the test?".

But since the agreement does not state that the test will be fair (the word does not appear anywhere in the document) many fence-sitters who might otherwise be convinced will not find it at all odd that a psychic with a reputation to protect might not want to apply. Why should they risk being cheated?

Reb

19th September 2003, 10:34 AM

I've been wondering about this myself. If Randi/JREF is not supposed to be involved in the actual tests, then why was Randi there to beef up Natalia's blindfold? And did he not say he was present during Edge's preliminary test? There seems to be a contradiction here. I did attempt to ask him about this but received one of his famous "go away"'s.

I suspect the rules apply to the actual tests as opposed to the prelim test, but Randi denied this saying the rules apply to "all" tests. So I repeat, why was Randi involved during these two tests?

Reb

Paul C. Anagnostopoulos

19th September 2003, 11:07 AM

I think Randi was requested by Natalia's handlers. I didn't even know he tested Edge.

~~ Paul

Reb

19th September 2003, 12:11 PM

Originally posted by Paul C. Anagnostopoulos
I think Randi was requested by Natalia's handlers.
I don't remember seeing this. Do you have a cite?

I didn't even know he tested Edge.
Here is the description of the test.
http://www.randi.org/jr/032902.html
Now that I reread it, it's not clear that Randi himself was actually present during the attempts, but he does say that he spoke to at some point.

Reb

Peter Morris

19th September 2003, 01:30 PM

Here is the description of the test.
http://www.randi.org/jr/032902.html
Now that I reread it, it's not clear that Randi himself was actually present during the attempts, but he does say that he spoke to at some point.
The link you provide is to dowser Mike G.'s test.

It contains the following :
Following the "open" sequence, for each of the "blind" tests, Mr. G. and I stepped out of the library area, and two other persons randomly (by choosing a face-down card, as before) placed the target package in position, then they left the area and informed us that the target was in place. Mike and I re-entered, alone, and he made his determination while I watched carefully...
The 'I' in question appears to be Randi, since Randi is writing the article.

Reb

19th September 2003, 01:41 PM

Oh yeah! Somehow my eyes just passed over that paragraph ...

scoman

29th September 2003, 01:00 PM

Who runs the tests? The JREF, no question!

The flummery on the page proceeding the actual application, states that the JREF does not involve itself with the testing procedure. However, this point is not in the terms of the actual application. The actual application does state that Randi may be present (but he won't interact with any materials!).

Also "We consult competent statisticians when an evaluation of the results, or experiment design, is required" - Does not mean that the statistcians are completely unconnected to the JREF, or unsympathetic to the JREF.

There have also never been any failed claimants to the $1000000 challenge! Back to the flummery of the proceeding page, "Upon success in the preliminary testing process, the "applicant" becomes a "claimant." . As the application states "To date, no applicant has passed the preliminary test, and this has eliminated the need for formal testing in those cases" - therefore, there has never even been a claimant to the challenge!

TheBoyPaj

29th September 2003, 01:35 PM

You've raised an interesting point there, Scoman. (Welcome, by the way!)

"We consult competent statisticians when an evaluation of the results, or experiment design, is required"

Randi has always been very forthright in pointing out that the tests are designed so that no evaluation of results is ever required. Sure, consult a statistician while the test is being designed, so that an appropriate number of repetitions can be picked to rule out chance, but it should never be necessary to consult one afterwards. The results are supposed to be self-evident!

Can anyone explain this?

scoman

9th October 2003, 01:44 PM

Thanks paj,

"Can anyone explain this?" - apparently not!!

Scott

Rolfe

9th October 2003, 02:10 PM

Originally posted by TheBoyPaj
"We consult competent statisticians when an evaluation of the results, or experiment design, is required"

Randi has always been very forthright in pointing out that the tests are designed so that no evaluation of results is ever required. Sure, consult a statistician while the test is being designed, so that an appropriate number of repetitions can be picked to rule out chance, but it should never be necessary to consult one afterwards. The results are supposed to be self-evident!

Can anyone explain this?
Yes. It's perfectly simple. For a good example, look at the "question about statistical significance" thread on the "Science, Mathematics..." forum.

In that one, I asked about a potential applicant who has a claim that he can identify a homoeopathic preparation from a sham. You give him a preparation and he answers. Well, he can be right or wrong. It's a simple yes/no situation, he has a 50% chance of getting it right just by a random guess. Randi isn't giving anyone a million bucks for getting a 50/50 guess right.

So, how many times does he have to get it right before you're as confident as you can be that he's not just guessing? We chewed it over and came up with 18 right out of 20, because that gives a p value of <0.001. Nine out of ten doesn't quite reach that level of significance, though ten out of ten does (but three out of three doesn't). You do need statistical advice to ensure that you've set the bar high enough so that only the genuinely superpowered are going to be able to clear it.

The thing is, you have to agree this beforehand - 10 out of 10 or 18 out of 20 can be mutually agreed as the level of performance which constitutes a pass. Less than this is a fail. Once that has been agreed (and if the applicant tries to insist that 16 out of 20 should be a pass, you have to explain why you don't buy that), then the outcome is self-evident. He either achieves it or he doesn't.

Clear enough?

Rolfe.

scoman

9th October 2003, 02:54 PM

"Clear enough?" - Yes!

As long as the use of statisticians, and their identity is agreed before hand, that does answer the question.

TheBoyPaj

13th October 2003, 05:50 AM

That covers the "experimental design" bit, but not the "evaluation of the results" which should never be needed.

Rolfe

13th October 2003, 06:55 AM

Originally posted by TheBoyPaj
That covers the "experimental design" bit, but not the "evaluation of the results" which should never be needed.
It is at the most basic level. Somebody needs to have made a record of whether the applicant said "yes" or "no" at each trial, and that needs to be matched up to the record of what was given at each trial. Are the records accurate? Untamperable-with? Do they come up with a pass or fail according to the agreed criteria? I think that's all it is.

Rolfe.

Peter Morris

13th October 2003, 05:36 PM

The thing is, you have to agree this beforehand - 10 out of 10 or 18 out of 20 can be mutually agreed as the level of performance which constitutes a pass. Less than this is a fail. Once that has been agreed (and if the applicant tries to insist that 16 out of 20 should be a pass, you have to explain why you don't buy that), then the outcome is self-evident. He either achieves it or he doesn't.

Clear enough?

I understand what you are saying, and I know that's how Randi runs his tests, but it seems to me that that is completely the wrong way to do it. It seems to me absurd that with a passmark of 80% and a score of anything up to 79% Randi would declare a failure.

There should be three possible results: definite pass, definite failure, or more tests required.

Example: http://www.skeptics.com.au/journal/divining.htm

In this dowsing test, dowsers had a 1 in 10 chance of guessing the correct pipe, so you would expect a 10% hit rate. In fact they scored 22% on the water test.

This is a lot lower than the passmark of 80%, but a lot higher than chance, the odds against it are about 107-1. Statistically speaking we can say that there is a better than 99% chance that there is some real dowsing effect here.

When you get results of this type, you have to decide whether its real, or just a fluke. So you should run further tests to see if the results are replicated. If you run the test several times, and the dowsers score above 20% on every test, that would be conclusive evidence of dowsing. The fact that they boast of 100% success rate wouldn't matter.

In your hypothetical example, with a score of 16 out of 20, again that would be significantly higher than chance, I would want to see if he can repeat it with the same result.

That's how the test should be run.

Randi, of course, won't do that.