Split Thread: How are MDC protocols designed and carried out? (Split from FAQ thread)

[Bryan Shelton]

Originally Posted by SkepticScott

snuffkin wrote "I know nothing about the tests or how they would measure the abilities?" Each protocol has to be custom-designed for the claimant. Claims of dowsing, mind-reading, and telekinesis have to be tested differently. In each case, however, there would be an objective pass/fail criteria.

I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out. When random choices are a clear and obvious element of such a test (as many of them truly are), who determines whether or not the results are significant enough to win the million dollar prize? Is that up to James Randi? Do people here consider that to be fair?

[Lamuella]

Originally Posted by Bryan Shelton

I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out.

All of this information is included in the application:

http://www.randi.org/site/images/sto...2011-03-09.pdf

Quote:

If the application is accepted, testing protocols will then be developed in consultation with the Applicant, who must provide clear guidelines so that the test(s) may be properly designed and carried out to the mutual agreement of the JREF and the Applicant. The JREF may consult with experts, including statisticians, magicians, and others with specialized knowledge relevant to the claim. James Randi may or may not be present at these tests, but he will not interact with the materials used nor interfere with the protocol once a test is underway.

[gnome]

Originally Posted by Bryan Shelton

I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out. When random choices are a clear and obvious element of such a test (as many of them truly are), who determines whether or not the results are significant enough to win the million dollar prize? Is that up to James Randi? Do people here consider that to be fair?

See above... it is decided on in advance and in agreement with the applicant. For example, JREF says, "Do you think you could achieve it sixteen times out of twenty?", applicant says, "Oh yeah, that's a cinch!". JREF: "Let's make that the success threshold then?", applicant says "Agreed."

[Bryan Shelton]

Sorry, but just saying that it has to be "to the mutual agreement of the JREF and the Applicant" doesn't answer my question. What is the MINIMAL statistical significance that such a test must provide, and who determines it? Is it James Randi, or is it somebody else?

[Bryan Shelton]

Originally Posted by gnome

See above... it is decided on in advance and in agreement with the applicant. For example, JREF says, "Do you think you could achieve it sixteen times out of twenty?", applicant says, "Oh yeah, that's a cinch!". JREF: "Let's make that the success threshold then?", applicant says "Agreed."

Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.

[Lamuella]

Originally Posted by Bryan Shelton

Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.

then you and the JREF would continue negotiating until your protocol had something that would satisfy both of you, and the test would not take place until it did.

The protocol is an agreement. It's the meeting point of what you are willing to say you can do and what Randi is willing to give you money for doing. Both sides have to agree or there is no test.

[JonathanMaxBerman]

It depends on the claim. There is no one number than can be used for all experiments. In biology, for example, we often look for a confidence interval of 95%. You choose an n (number of experiments) suited to the question you are addressing. You could compare data in a variety of ways including student's t-test, anova, or simple comparrison of standard error between groups. You pick the form of analysis in advsnce based on what is appropriate for the data, and you cant change it after to get the result you want. Thats why they ask you to agree in advance- so you cant reanalyse the data in a context which is inappropriate to the data typr,but which gives a false positive.

[Bryan Shelton]

So who would I be negotiating with? James Randi, or another member of the JREF? Why is it so hard to pull this information out of everybody??

[Lamuella]

Originally Posted by Bryan Shelton

So who would I be negotiating with? James Randi, or another member of the JREF? Why is it so hard to pull this information out of everybody??

it's not hard to get the information, it's just that you don't seem to have read anything.

per the application that I linked you to earlier:

Quote:

This offer is made and administered by the JREF, and no one may negotiate or make any changes, except as set forth in writing by an officer of the JREF.

[Bryan Shelton]

Originally Posted by JonathanMaxBerman

It depends on the claim. There is no one number than can be used for all experiments. In biology, for example, we often look for a confidence interval of 95%. You choose an n (number of experiments) suited to the question you are addressing. You could compare data in a variety of ways including student's t-test, anova, or simple comparrison of standard error between groups.

So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else? What is the required degree of success OVER what would be expected purely from chance? Why is all this material (the exact rules of the test) left completely unexplained by the JREF? Who exactly would I be dealing with??

[Lamuella]

Originally Posted by Bryan Shelton

So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else?

You agree on what the criteria for success would be before the test. If you meet these criteria, you pass.

Quote:

What is the required degree of success OVER what would be expected purely from chance?

You agree on what the criteria for success would be before the test.

Quote:

Why is all this material (the exact rules of the test) left completely unexplained by the JREF?

The exact rules are explained and you have been linked to them.

Quote:

Who exactly would I be dealing with??

You would be dealing with appointed officers of the JREF.

I'm not sure what part of this is so confusing to you.

[Bryan Shelton]

Originally Posted by Lamuella

it's not hard to get the information, it's just that you don't seem to have read anything.

per the application that I linked you to earlier:

So any single "officer of the JREF" would be sufficient, as long as I could talk him into agreeing with my own view of the test?

[Bryan Shelton]

Originally Posted by Lamuella

I'm not sure what part of this is so confusing to you.

A few years ago when I first joined the forum here, I got a similar run-around about exactly how the test is performed, and the required statistical significance of the test. Nowadays, I'm less inclined to let this important issue slide by!

[Rasmus]

Originally Posted by Bryan Shelton

So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else? What is the required degree of success OVER what would be expected purely from chance? Why is all this material (the exact rules of the test) left completely unexplained by the JREF? Who exactly would I be dealing with??

Have you emailed the JREF this question? People here (well, the vast majority of them) are mere mortal forum users. We do not speak for the JREF.

I think I read somewhere that a general rule of thumb was that the preliminary and the final test should only be passed by mere chance in about 1/1000 cases each - so your chances of submitting the protocol about predicting the card without any paranormal ability would be at about one in a million.

Personally, I think they should be much higher.

People are essentially claiming they can do the impossible.

Say, e.g. I was posing my challenge to a group of tragically blind-from-birth three year olds. They cannot see, and they have no concept of seeing and they cannot read or have a concept that such a thing as letters exist.

Now I propose that i can tell what is on one of their audio-CDs without any of the usual methods: I will not play it, or even have access to a player, I will not know how or where the CD has been sorted, etc.

Of course, all I have to do is look at the CD and read what it says on top of the disk. How reliably would you think I could identify a CD that way - if it was assured that the selection that each CD was chosen from would be heterogeneous enough to make mix-ups and confusions highly unlikely?

Say there would be 20 CDs. Bringing them into some pre-defined order or naming each one correctly by chance alone would be 1:20! or 2.43290201 × 10¹⁸.

My one-in-a-million chance would allow me to try and get only 10CDs sorted right, and I would get three tries and I would only have to get it all right once.

So, that being said, whenever someone starts arguing about their chances here, I can only conclude one of two things:

They do not understand the test procedures or they want to cheat.

[Lamuella]

Originally Posted by Bryan Shelton

So any single "officer of the JREF" would be sufficient, as long as I could talk him into agreeing with my own view of the test?

At the point where the protocol is being negotiated, I'd imagine you would be speaking to whoever oversees these tests for the JREF. I believe that this is currently Banachek. The reason for the wording is so that if Banachek is not available, someone else within the JREF can perform these duties in his absence. Also, so the whole protocol doesn't have to be rewritten every time staffing changes slightly.

[Lamuella]

Originally Posted by Bryan Shelton

A few years ago when I first joined the forum here, I got a similar run-around about exactly how the test is performed, and the required statistical significance of the test. Nowadays, I'm less inclined to let this important issue slide by!

the only run around is happening in your own head.

I'm sorry that you aren't able to read documents and comprehend the information contained therein, but have you considered that this might be your problem rather than the JREF's?

[shuttlt]

Originally Posted by Bryan Shelton

Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.

It's probably best to start with what ever the remarkable thing is that you can do, and then work on a way of making it as remarkable as possible without taking up an inordinate amount of time. If you really do have a remarkable power, then you can presumably show it to an arbitrary level of certainty. If you don't, then of course how lucky the JREF demand you be is very important.

[Bryan Shelton]

A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests". Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route? What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it? It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

[Lamuella]

Originally Posted by Bryan Shelton

A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests". Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route? What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it? It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

Success is being able to do what they say they can do. Failure is not being able to do what they say they can do. The reasons why are immaterial.

Was it fair to give them a test with long odds? Sure. It's JREF's money, they can do what they want with it.

What if someone had a genuine psychic ability that they couldn't manifest on a TV show? Then they shouldn't be tested on a TV show.

I don't get what you are finding unfair about this.

[Cuddles]

Originally Posted by Bryan Shelton

Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

How is that at all relevant? They weren't claiming to do it by the normal route, so the chance they'd have if they were is utterly irrelevant.

Quote:

What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it?

What if they did? Giving them a good chance of passing without actually using their "genuine" magic powers wouldn't do anything to address that, it would simply mean they could pass without actually having magic powers at all. If people think being nervous will reduce their success rate, then they should only agree to a test that takes that into account. If they fail to do so it's entirely their own stupid fault for agreeing to be tested for something they can't actually do.

Out of interest, where else do you think this argument applies? I've taken part in quite a few competitions in various sports and music. Should I have been claiming it was all unfair because I might have been nervous during my performances? Given how incredibly difficult you seem to think it is for people to simply do what they claim, I'm frankly amazed that it's possible for professional sports, musicians, comedians, actors, and so on to exist at all. What if their natural nervousness and trepidation prevents them from performing when they're supposed to?

Of course, it's worth bearing in mind that most applicants are absolutely certain of having a 100% success rate and actually have to be argued down to a lower rate for a test precisely to take this sort of thing into account.

Quote:

I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

Perhaps it would have helped if you'd been determined to actually listen to the answers as well.

[Rasmus]

Originally Posted by Bryan Shelton

A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

A few likely explanations:

She was deluded.
She was trying to keep face.
She thought she could get away with some trick.

Quote:

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests".

How were they difficult?

Did you read my above post?
Would it have been difficult for them to perform if they really had the abilities they claimed to possess? Would it be difficult for a person to sort a few CDs if they could read the labels?

Quote:

Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

Yes.

They said they weren't going to go the normal route! The odds would make it difficult to cheat or win by chance, they would not make it difficult to pass the test with the claimed ability.

Arguably, it might still be difficult to perform the ability, but the odds assume that the ability does not exist at all. If I claimed I could keep 20 apples in the air, I could chose to juggle them. It would still be difficult, but the difficulty of juggling 10 apples is unrelated to the difficulty of levitating 10 apples.

If a challenger claimed he could levitate apples, the test would only make it difficult for him to juggle them.

Quote:

What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it?

Then what? Get some public exposure, and practice a little more. come back after a year and grab the million dollars.

Quote:

It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

How is any of this the JREF's problem?

Again, let us assume I claimed I could juggle 10 apples, and you bet me 100$ that I can't.

We talk about the bet and agree what it means for me to successfully juggle 10 apples:

At no time can I ever hold more than 2 apples in any one hand, no apple can be in one of my hands for longer than 3 seconds at a time, an apple my be nowhere else other than in my hands or the air, and specifically, an apple may not touch the ground. I have 20 seconds to get all the apples moving from their initial position oin the ground.

We agree that there will be 5 observers, each video-recording my performance on a camera. If I can show that I followed the rules, I get your money; if you can show that I failed to meet the requirements, i do not get the money.

Now, on the day of the test, I end up being terribly nervous, I keep dropping apples, totally fail to ever pick up apple number 7. etc. etc.

Do you owe me the money? Certainly not!
Are you in any way to blame for my inability to foresee that I would not be able to perform under such stressful conditions? No!

Should I whine about being treated unfairly? Neither!

Now, why don't I just spend a week or two on practising, in front of cameras and people. Maybe ask you to adjust the rules - give me a few more moments to get the apples into the air, a second longer to hold on to an apple, etc.

Should I be allowed to reduce the number of apples to 6? No, not if my claim is "I can juggle 10 apples".

Should I be allowed to have up to three apples lying on the ground at any time? No, because then I'd be juggling 7 apples and not 10.

And maybe, just maybe, I should come to the realisation that I am not able to juggle apples ...

[Aepervius]

Originally Posted by Bryan Shelton

Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

Yes it was fair, because their claim is usually much, much much better than that in accuracy.
How many psychic out there you see claiming accuracy lower than 1/10 ? Usually they claim much better accuracy than that.

As for nervousness and so forth : my impression is that JREF test are not made into public tv but with very limtied public. It smells like an excuse to explain away not doing better than chance.

[Bryan Shelton]

Originally Posted by Lamuella

I don't get what you are finding unfair about this.

I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

[shuttlt]

Originally Posted by Bryan Shelton

I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

Since they don't know what the remarkable occurrence is that they are going to have to adjudicate on, it's kind of hard to come up with hard and fast rules. Also, up against them are a bunch of professional con artists and maybe, possibly, but probably not, somebody with genuine ability. If they don't keep a free hand to try to rule out fraud, they'll be conned out of the million long before anybody with a real ability, if such a person exists, comes along.

By the way, how come the folks don't get nervous on TV and in front of big crowds until Randi puts talcum powder on their hands, or what ever it might be that would stop them cheating?

[Rasmus]

Originally Posted by Bryan Shelton

I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

They are offered a million dollars! A million! I'd have to work some 20 years and not eat or pay rent or buy as much as toothpick during that time to accumulate that much money.

If anyone was offering me a million dollars to simply demonstrate one of my (more or less) amazing abilities, I would happily submit myself to those tests. And if, actually having an ability, I'd fail to demonstrate it, I would work out what caused me to fail in my demonstration.

And then I would spend a year on getting around those factors. And then I would kindly ask the JREF if I could be tested again. I might actually be more than happy to use the exact same test protocol, because it might still be a good protocol. Maybe I had to get over my nervousness, or get used to the bright lights, or I got distracted and will simply wear earplugs ... or anything, really!

I would not waste my time whining on a forum about how the test was unfair. (Unless, of course, it was unfair. In which case I could explain precisely why that was and how it should be changed. And chances are I wouldn't blame the JREF, and I can guarantee you that my requests would not make it easier for me to pass the test in case I didin't actually have my ability!)

Quote:

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

What's unfair?

Both parties are supposed to work out a protocol that specifies the answers to these questions!

For every claim, these answers will be different, so it always, always, always has to start with the applicant stating what exactly it is they are able to do.

Judging from the various discussions I've read here so far, I am forced to conclude that paranormal abilities are tied to the inability to construct or understand a few coherent sentences, though. If the test required applicants to pass an English Language Test of any kind, there would be no challenge.

[Bryan Shelton]

Originally Posted by shuttlt

Since they don't know what the remarkable occurrence is that they are going to have to adjudicate on, it's kind of hard to come up with hard and fast rules.

Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Quote:

Also, up against them are a bunch of professional con artists and maybe, possibly, but probably not, somebody with genuine ability. If they don't keep a free hand to try to rule out fraud, they'll be conned out of the million long before anybody with a real ability, if such a person exists, comes along.

Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!

Quote:

By the way, how come the folks don't get nervous on TV and in front of big crowds until Randi puts talcum powder on their hands, or what ever it might be that would stop them cheating?

I don't know. Maybe some of them DO get nervous, even before that happens?

[Rasmus]

Originally Posted by Bryan Shelton

Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Yes. But that requires them to first claim what they can do. And since that is not known, the JREF simply leaves that point open until after someone claims what it is they can do.

I can accept that you might like it better otherwise, or if there was a fixed claim that you'd have to beat odds of precisely 1:n. But surely, this couldn't e a problem, right?

Quote:

Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!

I am not aware of a single instance of that happening.

The "numbers" are only about how likely it would be to pass the test if you performed randomly. But your claim is that you do not perform randomly!

Quote:

I don't know. Maybe some of them DO get nervous, even before that happens?

Of all the examples I am aware of - and I know I haven't followed every test, or even the majority - I am not aware of any one test where I'd say your accusations might have merit.

the one and only thing I am aware of, is allowing a magician (Banachek, I think) handle sealed envelopes during a test. In that case I would support a re-test where only people that are not famous magicians get to touch, move, or otherwise interact with the objects that are needed during the test. But that one example does not involve any obvious cooking of numbers or making things harder than neccesary.

[shuttlt]

Originally Posted by Bryan Shelton

Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!

This is where you are going wrong.

If you have the magical power to "see" the face of upturned playing cards, we might pick 5 cards from 5 randomly (we'd have to agree on how to achieve this) shuffled decks and get you to tell us what they were. If you have no powers, the probability of you getting all of them (call it P1) should be, I think, (1/52)^5 = 0.00000000263. If you can actually "see" the face of the upturned cards, the probability (call it P2) should be 1.

I would expect the JREF to seek to arrive at a test that would keep P1 as low as possible while maximizing P2. They only want to make it difficult to pass if you have no powers.

[Rasmus]

Originally Posted by shuttlt

This is where you are going wrong.

If you have the magical power to "see" the face of upturned playing cards, we might pick 5 cards from 5 randomly (we'd have to agree on how to achieve this) shuffled decks and get you to tell us what they were. If you have no powers, the probability of you getting all of them (call it P1) should be, I think, (1/52)^5 = 0.00000000263. If you can actually "see" the face of the upturned cards, the probability (call it P2) should be 1.

Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%.

Who's to say? It's magic, after all!

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?

[shuttlt]

Originally Posted by Rasmus

Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%

In my example, he could "see" the cards, so it is as close to 1 as makes no odds. Normally people's powers seem to be closer to 1 than 0 probability until they get tested.

Originally Posted by Rasmus

Who's to say? It's magic, after all!

What is magic? I don't mean anything specific by it other than winning by some means that isn't a cheap (or expensive) trick based on outsmarting the JREF. Telling homeopathic water from nonhomeopathic water would do it for me.

Originally Posted by Rasmus

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?

Sure, and then you can build an experiment to give you any probability of success you like. Keep repeating it and it becomes easier and easier for somebody with a real power to demonstrate it and harder and harder for somebody to fluke it. For some reason Byan seems to be complaining that it is unfair to make the test really hard to fluke, or am I misunderstanding?

[Lamuella]

Originally Posted by Rasmus

Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%.

Who's to say? It's magic, after all!

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?

well, in those cases you don't ask for perfection, but you ask for consistently better than chance over long periods. If I claim to be able to guess a coin flip correctly 55% of the time, obviously I won't be asked to guess the coinflip 100% of the time, but I would be asked to guess a huge number of coinflips so the chance of such a result occurring by chance is as low as possible.

[Rasmus]

Originally Posted by shuttlt

In my example, he could "see" the cards, so it is as close to 1 as makes no odds. Normally people's powers seem to be closer to 1 than 0 probability until they get tested.

I certainly agree that the claims that usually come up have no reservations or margins of error that would not be comparable with other abilities.

But, of course, even in your example, the person does not "see" the cards as such - no light is reflected from the cards in different colours towards his retina. So whatever it is, and whatever it feels like to the participant, it would not be "seeing".

Quote:

What is magic? I don't mean anything specific by it other than winning by some means that isn't a cheap (or expensive) trick based on outsmarting the JREF. Telling homeopathic water from nonhomeopathic water would do it for me.

I was just using the term to describe all or any happenings of stuff that would be classified as "paranormal".

I agree that telling which water has been treated homeopathically would qualify for a test.

But we cannot expect the challenger to do this at a success-rate of 100%, or 80% or anything. We "expect" that he performs within the boundaries of random chance, after all. We have no explanation for any other result, so it seems unjustified to expect any other result to be at 100%.

and, yes, I agree that they will usually claim as much - in which case there is no need to expect anything less than that, or significantly less.

Quote:

Sure, and then you can build an experiment to give you any probability of success you like. Keep repeating it and it becomes easier and easier for somebody with a real power to demonstrate it and harder and harder for somebody to fluke it. For some reason Byan seems to be complaining that it is unfair to make the test really hard to fluke, or am I misunderstanding?

As far as I can tell, this is what his complaint boils down to - but I am not sure if he means and understands that, or if he thinks this would actually influence the chances of a truly gifted applicant. (And I agree that it does, as long as we assume the applicant has a 100% chance and as long as that is not actually the case.)

For any ability of the applicant to perform less than perfectly but above chance, we can find a test-design that makes it arbitrarily difficult for a cheater to pass by mere chance alone yet would let us expect a genuinely gifted applicant to pass.

Usually, this would require a ridiculous number of trials, and of course it calls into question how the applicant ever discovered his ability in the first place to make claims about with any confidence.

It is easy to define my superpower as follows:

I have the ability to predict the next card drawn from a deck of cards not with the expected 1:52, but 1:51 - if and only if the card happens to be the trey of clubs.

If I genuinely had this truly amazing power ... I would never ever notice it, if I lived a million years and did nothing else but play with card decks.

[Baloney]

Originally Posted by Bryan Shelton

I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

Bryan Shelton, this link might be of some help to you:
Aesop's Fables - The Fox and the Grapes

[wardenclyffe]

Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been? How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

Ward

[Pup]

Originally Posted by Bryan Shelton

Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

After the JREF is told what the ability is, yes, and that's why the odds are agreed proposed during the negotiations, after the applicant specifies their ability.

Are you saying the JREF should be able to figure the odds, even before an applicant announces whatever their ability is?

[Rasmus]

Originally Posted by Pup

After the JREF is told what the ability is, yes, and that's why the odds are agreed proposed during the negotiations, after the applicant specifies their ability.

Are you saying the JREF should be able to figure the odds, even before an applicant announces whatever their ability is?

then they would be eligible to win their own challenge...

[Baloney]

Originally Posted by Bryan Shelton

Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Hey, stop using my name in vain!

[Rasmus]

Originally Posted by wardenclyffe

Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been? How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

Ward

http://forums.randi.org/showthread.php?t=89877

Brilliant example.

I don't know if a background discussion is available, but you can see very exact claims being made, no expected success rate for random occurrences, a quick test, and surprisingly, no winner.

To be fair, though, we would expect that a healthy person who just emptied their bladder would be found wetting themselves within 15 or 30 minutes would be almost exactly 0.0%. It is almost impossible that this should happen by mere chance.

But I agree there is a chance this could happen randomly - so it would have made things very difficult if a percentage had been pre-defined.

[Bryan Shelton]

Originally Posted by Lamuella

well, in those cases you don't ask for perfection, but you ask for consistently better than chance over long periods. If I claim to be able to guess a coin flip correctly 55% of the time, obviously I won't be asked to guess the coinflip 100% of the time, but I would be asked to guess a huge number of coinflips so the chance of such a result occurring by chance is as low as possible.

That's EXACTLY what I'm saying!

I think that James Randi was a bit unfair on that TV show, when he demanded that the "psychics" pass TREMENDOUSLY difficult tests. Of course, if they themselves had made such claims before the start of the show, then that's their own problem; one could hardly criticize Randi for demanding that they pass a test which they themselves claimed to be able to pass.

But what if their only claim is that they can "sense" the result of a coin flip only 55% of the time? Is James Randi (or an "officer" of the JREF) going to throw them out of the office, or are they going to give them a legitimate test for the one million dollar prize, as long and difficult as that would obviously be?

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...

[Bryan Shelton]

Originally Posted by wardenclyffe

Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been?

I have no idea what they are, or what they should have been.

Originally Posted by wardenclyffe

How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

I have no idea how they should have been figured out. Personally, I would have had nothing to do with such a test.