JREF Homepage Swift Blog Events Calendar $1 Million Paranormal Challenge The Amaz!ng Meeting Useful Links Support Us
James Randi Educational Foundation JREF Forum
Forum Index Register Members List Events Mark Forums Read Help

Go Back   JREF Forum » JREF Topics » Million Dollar Challenge
Click Here To Donate

Notices


Welcome to the JREF Forum, where we discuss skepticism, critical thinking, the paranormal and science in a friendly but lively way. You are currently viewing the forum as a guest, which means you are missing out on discussing matters that are of interest to you. Please consider registering so you can gain full use of the forum features and interact with other Members. Registration is simple, fast and free! Click here to register today.

Reply
Old 17th November 2012, 11:46 AM   #41
EdG
Scholar
 
Join Date: Apr 2012
Location: Arizona, USA
Posts: 108
Originally Posted by Pixel42 View Post
I was just pointing out that even if we concede your point that we shouldn't expect these abilities to be any more consistent than those of talented batsmen, chess players etc, we would still expect that they would (as with such abilities) produce results that are significantly better than random chance. And they don't.
Thanks for clarifying.
EdG is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 17th November 2012, 12:46 PM   #42
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by EdG View Post
Why? Any human ability should fall within normal parameters compared to other human abilities. Anyone can play piano after a few lessons, but only some people will reach virtuoso level after many years of study and practice.
Claimants are not expected to reach virtuoso level. They are just expected to do anything paranormal at all.

Quote:
An exceptional baseball pitcher may be defined as one who pitches a no-hitter game. There have only been 236 no-hitters in the past 111 years, so the success rate does not exactly jump significantly.

And no, despite your bizarre claim about my presumed motive, choosing pitchers or any other skilled human would not weaken my argument. Education and practice are the keys to acquiring skill in any field. If psychic skills exist, why should they be any different? Because you say so?
Perhaps there is confusion about the nature of the challenge. A claimant needs not beat anyone, unlike a baseball hitter, or pitcher, or chess player, or any of those examples.
He or she is simply expected to demonstrate something. It would be as if the baseball pitcher was only required to throw the ball a certain distance. He or she could set themselves any distance for the throw, and even demand that he or she need only succeed one time out of ten and such. And of course, the pitcher could demand that there should or should not be a hitter.

Under such test conditions one would expect that people should pass a million times in a row.
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 03:58 AM   #43
GzuzKryzt
Philosopher
 
Join Date: Aug 2005
Posts: 6,367
Originally Posted by EdG View Post
...
If psychic skills exist, why should they be any different? Because you say so?
1. There is no evidence psychic skills exist, hence a speculation about if they would be different from other skills is at best moot and at worst a colossal waste of time. As soon as there is evidence, we could but would not necessarily need to speculate. But not earlier.

2. Please do not put words in my mouth. I did not say anything "so", I merely criticized your analogy and I stand by that criticism.

Let's agree to disagree and move on.
GzuzKryzt is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 08:22 AM   #44
EdG
Scholar
 
Join Date: Apr 2012
Location: Arizona, USA
Posts: 108
Originally Posted by GnaGnaMan View Post
Under such test conditions one would expect that people should pass a million times in a row.
What an absurd claim!
EdG is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 08:28 AM   #45
EdG
Scholar
 
Join Date: Apr 2012
Location: Arizona, USA
Posts: 108
Originally Posted by William Smith View Post
1. There is no evidence psychic skills exist, hence a speculation about if they would be different from other skills is at best moot and at worst a colossal waste of time. As soon as there is evidence, we could but would not necessarily need to speculate. But not earlier.
There was no evidence the world was not flat until there was. Despite your dogmatic proclamation ("because I said so, so there!"), envisioning parameters is a tried and true method toward proving or disproving unexplained phenomena.
EdG is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 09:54 AM   #46
GzuzKryzt
Philosopher
 
Join Date: Aug 2005
Posts: 6,367
Originally Posted by EdG View Post
There was no evidence the world was not flat until there was. Despite your dogmatic proclamation ("because I said so, so there!"), envisioning parameters is a tried and true method toward proving or disproving unexplained phenomena.
Again, please do not put words in my mouth.
GzuzKryzt is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 10:43 AM   #47
EdG
Scholar
 
Join Date: Apr 2012
Location: Arizona, USA
Posts: 108
Originally Posted by William Smith View Post
Again, please do not put words in my mouth.
Sorry.

Last edited by EdG; 18th November 2012 at 10:57 AM.
EdG is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 18th November 2012, 04:35 PM   #48
Pup
Illuminator
 
Pup's Avatar
 
Join Date: Dec 2004
Posts: 4,561
Originally Posted by Musibrique View Post
Expecting an exact 100% or near 100% replication is very ridiculous and extremely conservative. Telling a psychic to pass 100 tests in a row is like telling famous baskeball player, Brian, to never miss a basket.
No, it's like telling famous basketball player Brian, who claims he can make a basket in 1 out of 10 free throws, to make 1 out of 10, 100 times in a row.

The thing that's being repeatedly tested is not the actual act (making a free-throw) but the claim (making at least 1 freethrow out of 10, in this case).

It's up to Brian to figure out that he usually gets 8 baskets out of 10 free throws but he's never tried 9 shots in a row without getting at least one basket, and make his claim accordingly.
Pup is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 20th November 2012, 04:52 PM   #49
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by Musibrique View Post
A million tests will succeed in a row? That is so unrealistic in practical terms, even via conventional research. So, if a study found a p-value of 0.001, what is the probability of getting five 0.001 p-values in a row? Simple! 1_ X 10^-14

Not even conventional research has reach those kinds of odds.
I do not believe that psychic powers exist. But I believe that such powers should be tested the same way one tests other skills.

As a result, agree with the above. There is also another related point. But if the JREF truly wants to test whether psychic powers exist, then it should also consider beta, the probability that a false null hypothesis is not rejected.

I think there's a good argument for keeping the beta reasonably low. If it's an issue of protecting the money, I'd cut down the amount of money involved to, say, $100,000 or even $50,000. I understand that there would be PR issues with doing this, but that's OT.

Last edited by FluffyPersian; 20th November 2012 at 05:57 PM.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 20th November 2012, 10:04 PM   #50
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by FluffyPersian View Post
I do not believe that psychic powers exist. But I believe that such powers should be tested the same way one tests other skills.

As a result, agree with the above.
You do? But it's nonsense!
The correct answer to that question is: We do not know.

Quote:
There is also another related point. But if the JREF truly wants to test whether psychic powers exist, then it should also consider beta, the probability that a false null hypothesis is not rejected.
What the JREF truly wants to do, AFAIU, is hammer home the point that all those professional psychics who charge a lot of money for their tricks can't deliver the goods.

Quote:
I think there's a good argument for keeping the beta reasonably low.
How do you propose to calculate beta in practice?

Quote:
If it's an issue of protecting the money, I'd cut down the amount of money involved to, say, $100,000 or even $50,000. I understand that there would be PR issues with doing this, but that's OT.
Ridiculous.
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 21st November 2012, 07:43 PM   #51
gnome
Penultimate Amazing
 
gnome's Avatar
 
Join Date: Aug 2001
Posts: 10,321
Originally Posted by FluffyPersian View Post
I do not believe that psychic powers exist. But I believe that such powers should be tested the same way one tests other skills.

As a result, agree with the above. There is also another related point. But if the JREF truly wants to test whether psychic powers exist, then it should also consider beta, the probability that a false null hypothesis is not rejected.

I think there's a good argument for keeping the beta reasonably low. If it's an issue of protecting the money, I'd cut down the amount of money involved to, say, $100,000 or even $50,000. I understand that there would be PR issues with doing this, but that's OT.
I argue that the best way to keep beta low is plenty of repetition, effectively increasing the sample size. It does not require letting more type I errors through, and thus no need to reduce the reward.

Intuitively: if someone claims they can perform a task 51 times out of 100, if you do the test 200 times, by chance they might fail to out-perform a coin toss. However, if you do the same test 2000 times, the effect should start to emerge clearly.
__________________


Last edited by gnome; 21st November 2012 at 07:46 PM.
gnome is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 21st November 2012, 09:23 PM   #52
Startz
Critical Thinker
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 428
Originally Posted by gnome View Post
I argue that the best way to keep beta low is plenty of repetition, effectively increasing the sample size. It does not require letting more type I errors through, and thus no need to reduce the reward.

Intuitively: if someone claims they can perform a task 51 times out of 100, if you do the test 200 times, by chance they might fail to out-perform a coin toss. However, if you do the same test 2000 times, the effect should start to emerge clearly.
The intuition is right, but...

If JREF wanted to allow 200 coin tosses with only a 1 in a thousand chance of success by luck, it would require 122 heads out of the 200 tosses. (Someone should check my math, I've spent the day making turkey broth.) The chance of a win for someone with a 51% chance of getting a head is only 1.8 out of a thousand.

Raise the number of tosses to 2000 and JREF would require 1069 heads. And the fictional 51% psiguy would have a 1.34 percent chance of winning. (If I did it right, you'd have to run nearly 25,000 trials before psiguy would have a better than even chance of winning.)

My limited experience with JREF is that they don't have much patience for long trials.
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 21st November 2012, 10:14 PM   #53
Rasmus
Philosopher
 
Rasmus's Avatar
 
Join Date: Jul 2005
Posts: 6,362
Originally Posted by Startz View Post
My limited experience with JREF is that they don't have much patience for long trials.
That may well be because nobody who claims an ability with such a low success-rate can ever explain how they came to know that they had that ability.

How would you notice that your coinflips are slightly biased? Who flips a coin 2000 times, keeps track and - assuming a result slightly out of the ordinary - proceeds to repeat the experiment with different coins etc?

Also, I have yet to see someone who applies and then not only understands the difficulties of such a test, but makes a reasonable effort to accomodate for them. IIRC the rules make it clear that the JREF will not pay for the testing procedures - all an applicant would have to do do get a long and complicated test done would be to pay test-team.
__________________
"Well, the religious community could not just make it up." - JetLeg
Rasmus is online now   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 22nd November 2012, 06:46 AM   #54
Startz
Critical Thinker
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 428
Originally Posted by Rasmus View Post
That may well be because nobody who claims an ability with such a low success-rate can ever explain how they came to know that they had that ability.

How would you notice that your coinflips are slightly biased? Who flips a coin 2000 times, keeps track and - assuming a result slightly out of the ordinary - proceeds to repeat the experiment with different coins etc?

Also, I have yet to see someone who applies and then not only understands the difficulties of such a test, but makes a reasonable effort to accomodate for them. IIRC the rules make it clear that the JREF will not pay for the testing procedures - all an applicant would have to do do get a long and complicated test done would be to pay test-team.
I helped--or tried to help--help on the Pavel negotiations. His claim was much higher than 51 percent but lower than 100 percent. Pavel proposed tests that would have been completed within a day at his expense. While some of his protocol design was perhaps overcomplicated, JREF was unaccommodating.

JREF is pretty clear that they're not interesting in conducting neutral scientific experiments. JREF is interesting in a challenge to debunk false claims of the paranormal, preferably in a way that educates the public.
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 22nd November 2012, 11:26 PM   #55
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by GnaGnaMan View Post
You do? But it's nonsense!
The correct answer to that question is: We do not know.
I do what? I said I don't believe in the existence of psychic powers. However, if JREF wishes to test whether such powers exist, there's no reason that a discussion of the parameters of the hypothesis tests should be deemed ridiculous.

That part of my post didn't contain a question, so it's unclear what question you're referring to.

Quote:
How do you propose to calculate beta in practice?
For particular values of the population mean and standard deviation, there are alpha-beta curves that specify the relationship between the two parameters. Of course, similar curves exist for given proportions, which would be more applicable to most MDC tests. For more details, see this text:

http://www.econ.nyu.edu/user/ramseyj.../chapter11.pdf

As for "in practice," it's unclear why practice in this context differs from the standard examples in statistics texts.

Quote:
Ridiculous.
Please clarify why reducing the prize to allow for an increased beta is ridiculous. Why is one million the magic number?

Last edited by FluffyPersian; 22nd November 2012 at 11:29 PM.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 01:40 AM   #56
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by FluffyPersian View Post
I do what? I said I don't believe in the existence of psychic powers. However, if JREF wishes to test whether such powers exist, there's no reason that a discussion of the parameters of the hypothesis tests should be deemed ridiculous.

That part of my post didn't contain a question, so it's unclear what question you're referring to.
When you click on that little arrow at the top of a quote, right next to the poster's name, it will take you back to the original post. Do that when you can't recall the context of a quote.

Quote:
As for "in practice," it's unclear why practice in this context differs from the standard examples in statistics texts.
How do you obtain your effect size estimate?

Quote:
Please clarify why reducing the prize to allow for an increased beta is ridiculous. Why is one million the magic number?
Ultimately, the JREF doesn't set a beta value.
if I understand you correctly, then your solution for decreasing beta is increasing alpha. When you can't get a real psychic to step forward by offering 1 million, then the solution is certainly not handing out smaller sums to non-psychics.
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 01:54 AM   #57
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by Rasmus View Post
How would you notice that your coinflips are slightly biased? Who flips a coin 2000 times, keeps track and - assuming a result slightly out of the ordinary - proceeds to repeat the experiment with different coins etc?
Good point.
Those questions are rhetorical but here are answers nevertheless.

The answer to the last question is: These guys:
Quote:
We analyze the natural process of flipping a coin which is caught in the hand. We prove that vigorously-flipped coins are biased to come up the same way they started. The amount of bias depends on a single parameter, the angle between the normal to the coin and the angular momentum vector. Measurements of this parameter based on high-speed photography are reported. For natural flips, the chance of coming up as started is about .51.
http://comptop.stanford.edu/u/preprints/heads.pdf

Persi Diaconis started out as a magician, btw.

The answer to the first question... Well, anyone ever notice how coin flips are really biased?
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 12:49 PM   #58
jsfisher
ETcorngods survivor
 
jsfisher's Avatar
 
Join Date: Dec 2005
Posts: 13,373
Originally Posted by gnome View Post
I argue that the best way to keep beta low is plenty of repetition, effectively increasing the sample size. It does not require letting more type I errors through, and thus no need to reduce the reward.

Intuitively: if someone claims they can perform a task 51 times out of 100, if you do the test 200 times, by chance they might fail to out-perform a coin toss. However, if you do the same test 2000 times, the effect should start to emerge clearly.
I agree with this. The question is really whose responsibility is it to include it in the protocol. The JREF is interesting in alpha, and the claimant should be the one interested in beta. Few, if any, are, though, and that may because they believe their abilities are infallible.

On the other hand, the JREF does not accept protocols that would take an excessive amount of time. The 2,000 repetitions example might be disallowed as excessive. (Didn't Pavel run afoul of this?) I'd like that particular MDC condition reconsidered.
__________________
A proud member of the Simpson 15+7, named in the suit, Simpson v. Zwinge, et al., and founder of the ET Corn Gods Survivors Group.

By the way, the Nominate button is to your right left, sort of..
jsfisher is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 01:53 PM   #59
wardenclyffe
Graduate Poster
 
wardenclyffe's Avatar
 
Join Date: Dec 2008
Posts: 1,857
Originally Posted by jsfisher View Post
On the other hand, the JREF does not accept protocols that would take an excessive amount of time. The 2,000 repetitions example might be disallowed as excessive. (Didn't Pavel run afoul of this?) I'd like that particular MDC condition reconsidered.
I think the JREF would consider protocols that take a long time, as long as the applicant is willing to compensate (monetarily) everyone involved for their time. We've seen groups of skeptics give up an afternoon or a whole day to administer a test. And remember, that's just the day of a test. It does not take into account all the time and effort of negotiating a protocol and acquiring and setting up equipment, etc.. That amount of time is something those groups seem willing to give freely in the same way that Civil War re-enactors give their time and effort for something that they enjoy.

However, when it comes to a protocol that must be administered over days/months/weeks, then it's a different story. Just like a Civil War re-enactor would expect to be compensated if he or she were participating in battle scenes for a movie which took weeks to shoot.

Eventually, a test becomes so complex and time consuming that the prize money will be eaten up by the costs of the test---and that's assuming that the applicant can actually pass the test and win the prize.

Ward
__________________
~~Na eth'er aa, ammre' en ank'aar'eith, d'emner'aa-, asd'reng'aather, em'n'err-aae...~
- Alenara Al'Kher'aat, aged 347
wardenclyffe is online now   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 03:27 PM   #60
Anerystos
Muse
 
Join Date: Jun 2007
Posts: 808
Earwig 0 again.

Back in the dim and distant days when I was a Psychology undergrad, I had a running disputation with my statistics lecturer on the subject of significance – not levels as such, but what it meant. It got to the point where on the Finals paper on Statistics and Experimental Design there was a question which meant nothing to my fellow students but was an invitation to me to expound!

Here goes.

If I carry out the same experiment 20 times – whatever the experiment – on one occasion out of the 20 I am likely to get significance at the 5% level. I say “I am likely … ” because we are getting into probability cadence: what is the probability of getting a significant result at, say, the 5% level if the effect is non-existent? It doesn't seem to me to be 5%, but ...

The significance level is actually irrelevant – it just changes how long it will take for it to happen.

Shift now to the disputation.

If I run 20 different experiments, I can expect one of those to produce a significant result at the 5% level – even if none of the 20 effects which I am investigating is “real”.

OK guys, over to you!

Last edited by Anerystos; 23rd November 2012 at 03:29 PM.
Anerystos is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 06:10 PM   #61
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by GnaGnaMan View Post
When you click on that little arrow at the top of a quote, right next to the poster's name, it will take you back to the original post. Do that when you can't recall the context of a quote.
I did look at the original post. It was unclear what you were referring to. Defining your pronouns ("But it's nonsense") would have helped.

Originally Posted by GnaGnaMan View Post
How do you obtain your effect size estimate?
The effect size is calculated the same way it would be for alpha, as the difference between the sample proportion (correct answers/total answers in a given test) and the proportion of correct answers JREF believes a psychic should be able to achieve.


Quote:
if I understand you correctly, then your solution for decreasing beta is increasing alpha. When you can't get a real psychic to step forward by offering 1 million, then the solution is certainly not handing out smaller sums to non-psychics.
If I understand you correctly, you're saying that anyone not able to achieve 100% correct answers all the time is a non-psychic. If so, you're defining away the issue we're debating here.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 06:31 PM   #62
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by Paul W View Post
Earwig 0 again.

Back in the dim and distant days when I was a Psychology undergrad, I had a running disputation with my statistics lecturer on the subject of significance not levels as such, but what it meant. It got to the point where on the Finals paper on Statistics and Experimental Design there was a question which meant nothing to my fellow students but was an invitation to me to expound!

Here goes.

If I carry out the same experiment 20 times whatever the experiment on one occasion out of the 20 I am likely to get significance at the 5% level. I say I am likely because we are getting into probability cadence: what is the probability of getting a significant result at, say, the 5% level if the effect is non-existent? It doesn't seem to me to be 5%, but
You are correct that p-values work only for supernatural abilities that differ only in quantity from normal abilities. But many abilities can be stated in this way. For example, all of us can defy gravity for half a second by jumping. However, the standard deviation of the time we spend in the air is pretty low, so spending even 15 seconds in the air would indicate some kind of ability.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 09:01 PM   #63
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by Paul W View Post
The significance level is actually irrelevant it just changes how long it will take for it to happen.
It seems to me that you're talking about the sample size, not the significance level.

As far as "waiting for it to happen,"what you're "waiting" for may never happen. The downside of drawing inferences off a smaller sample is that you're less certain that those inferences actually apply to the population. (I'm assuming that standard deviation, and effect size are held constant.)
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 09:06 PM   #64
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by FluffyPersian View Post
The effect size is calculated the same way it would be for alpha
I should be more precise here.. The effect size is calculated the same way it is in the numerator of the t-statistic, which is then mapped onto a p-value.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 23rd November 2012, 09:44 PM   #65
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
I appreciate the answers I received to the earlier question, which proved useful when I taught that particular class.

The recent discussion about the what the significance levels OUGHT to be (rather than are) should be split into its own thread, IMO. But that's up to the mods.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 12:57 AM   #66
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by FluffyPersian View Post
I did look at the original post. It was unclear what you were referring to. Defining your pronouns ("But it's nonsense") would have helped.
You were agreeing with this:
Quote:
A million tests will succeed in a row? That is so unrealistic in practical terms, even via conventional research. So, if a study found a p-value of 0.001, what is the probability of getting five 0.001 p-values in a row? Simple! 1_ X 10^-14
The problematic question/answer bolded by me.

Quote:
The effect size is calculated the same way it would be for alpha, as the difference between the sample proportion (correct answers/total answers in a given test) and the proportion of correct answers JREF believes a psychic should be able to achieve.
That's not how the challenge works and I don't think it would be appropriate to change it so.

Quote:
If I understand you correctly, you're saying that anyone not able to achieve 100% correct answers all the time is a non-psychic. If so, you're defining away the issue we're debating here.
You do not understand me correctly. I don't even know how you get the idea.
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 10:00 AM   #67
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by GnaGnaMan View Post
You were agreeing with this:

OK, now I see what you mean. I agreed that the significance level of 0.001 is far lower than that used in conventional research--a claim that does not require the (incorrect) calculation to justify it. The conventional significance level in most applications is 0.05.


Quote:
You do not understand me correctly. I don't even know how you get the idea.
My copy function isn't working at the moment, but you implied that someone isn't a psychic can't achieve a p-value of less than <0.001 happens to be set. Is that correct?

Last edited by FluffyPersian; 24th November 2012 at 10:25 AM.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 10:28 AM   #68
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by GnaGnaMan View Post
That's not how the challenge works and I don't think it would be appropriate to change it so..
That's exactly how tests of conventional treatments (throughout the social and medical sciences) are executed. Why should, say, homeopathic treatments be tested completely differently?
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 02:03 PM   #69
xtifr
Graduate Poster
 
xtifr's Avatar
 
Join Date: Apr 2012
Location: Sol III
Posts: 1,251
Originally Posted by FluffyPersian View Post
That's exactly how tests of conventional treatments (throughout the social and medical sciences) are executed. Why should, say, homeopathic treatments be tested completely differently?
Because A) this isn't a clinical trial; it's a million-dollar challenge, and B) you're allowed to include an uncertainty factor here, which absolutely wouldn't be allowed in a clinical trial. Saying, "this magical remedy works in 80% of all cases" would give you an inherent p-value of .05, and is a claim that can be tested. The point of the MDC is to get near-certainty on the claim.
__________________
"Those who learn from history are doomed to watch others repeat it."
-- Anonymous Slashdot poster
"The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore."
-- James Nicoll
xtifr is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 02:42 PM   #70
Startz
Critical Thinker
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 428
Originally Posted by xtifr View Post
you're allowed to include an uncertainty factor here, which absolutely wouldn't be allowed in a clinical trial. Saying, "this magical remedy works in 80% of all cases" would give you an inherent p-value of .05, and is a claim that can be tested. The point of the MDC is to get near-certainty on the claim.
I don't understand.
(1) Are you saying that medical tests aren't conducted on treatments that only work sometimes?

(2) What does "inherent p-value" mean?
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 03:09 PM   #71
xtifr
Graduate Poster
 
xtifr's Avatar
 
Join Date: Apr 2012
Location: Sol III
Posts: 1,251
Originally Posted by Startz View Post
I don't understand.
(1) Are you saying that medical tests aren't conducted on treatments that only work sometimes?
Huh? No. Where on Earth did you get that from?

Quote:
(2) What does "inherent p-value" mean?
It's part of the claim. MDC tests claims. If you say, "this succeeds 80% of the time", then that's what will be tested (assuming a protocol can be agreed on, and other requirements met).
__________________
"Those who learn from history are doomed to watch others repeat it."
-- Anonymous Slashdot poster
"The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore."
-- James Nicoll
xtifr is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 03:48 PM   #72
Startz
Critical Thinker
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 428
I guess I misunderstood what you meant when you said an uncertainty factor wouldn't be allowed in a clincal trial. Sorry.

If I understand the second part right you're saying the MDC would require you to beat your own claim (say 80%) by more than chance--not just do better than 50/50. That makes sense, although a claimant who believed they could do 80 percent would still be wise to ask for a protocol that only beat 50 percent. Admittedly, "claimant" and "would still be wise" doesn't seem to fit together too well.
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 04:16 PM   #73
xtifr
Graduate Poster
 
xtifr's Avatar
 
Join Date: Apr 2012
Location: Sol III
Posts: 1,251
Originally Posted by Startz View Post
I guess I misunderstood what you meant when you said an uncertainty factor wouldn't be allowed in a clincal trial. Sorry.
Oh, yeah, I guess I didn't phrase that as well as I could, but you seem to have the basic idea now.

Quote:
That makes sense, although a claimant who believed they could do 80 percent would still be wise to ask for a protocol that only beat 50 percent.
Not necessarily. Attempting to prove you can beat 50 percent with the degree of certainty needed for the MDC is going to be a lot harder, because it would require so much more testing to reach the necessary p-values for that claim. If you really think you can reach 80% consistently, it's better to ask for that, or just barely less; otherwise the testing may prove too expensive.
__________________
"Those who learn from history are doomed to watch others repeat it."
-- Anonymous Slashdot poster
"The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore."
-- James Nicoll
xtifr is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 05:34 PM   #74
Startz
Critical Thinker
 
Join Date: Nov 2004
Location: Santa Barbara, CA
Posts: 428
If I did this right, and if there isn't too much rounding error, then JREF's p=.001 can be met with 20/25 heads and the probability of doing better than that for someone with an 80% ability is 42%

But if the person claimed a 70% ability, JREF would require beating 42/60. And given a true ability of 0.8 the chance of doing that or better is 95%. So if you can go for a few more trials than it pays to claim a performance level lower than the truth.
Startz is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 09:34 PM   #75
Musibrique
Scholar
 
Join Date: Apr 2012
Posts: 52
Originally Posted by Startz View Post
If I did this right, and if there isn't too much rounding error, then JREF's p=.001 can be met with 20/25 heads and the probability of doing better than that for someone with an 80% ability is 42%
I don't believe one-tailed p-values should come in binomial experiments such as a coin. Psi in binomial experiments can be predicted as either a hit greater than chance or miss greater than chance. Thus, the proper p-value for a coin is two-tailed. Thus it would take the claimant about 21 heads in order to the pass the preliminary test. You are correct with the 80% probability though.

Quote:
But if the person claimed a 70% ability, JREF would require beating 42/60. And given a true ability of 0.8 the chance of doing that or better is 95%. So if you can go for a few more trials than it pays to claim a performance level lower than the truth.
Effects that are large can be easily detected via small sample-size whereas small effects cannot.
Musibrique is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 09:46 PM   #76
Musibrique
Scholar
 
Join Date: Apr 2012
Posts: 52
Originally Posted by FluffyPersian View Post
The effect size is calculated the same way it would be for alpha, as the difference between the sample proportion (correct answers/total answers in a given test) and the proportion of correct answers JREF believes a psychic should be able to achieve.
Where did you get this idea? There are many ways of calculating an effect-size; however, an effect-size in a binomial experiment like coins, Zener Cards, etc. is calculated by the z-value of a t-test and dividing it by the square root of N. (Effect-size=z-value\sqrt. of N)


Quote:
If I understand you correctly, you're saying that anyone not able to achieve 100% correct answers all the time is a non-psychic. If so, you're defining away the issue we're debating here.
I agree. Saying getting 59 heads out of 60 coin tosses is not evidence of psi is very absurd.

Last edited by Musibrique; 24th November 2012 at 09:48 PM.
Musibrique is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 10:28 PM   #77
FluffyPersian
Critical Thinker
 
FluffyPersian's Avatar
 
Join Date: Aug 2011
Posts: 258
Originally Posted by xtifr View Post
Saying, "this magical remedy works in 80% of all cases" would give you an inherent p-value of .05,
No, it doesn't. The significance level, which is what you're apparently referring to, is separate from the threshold proportion and the sample size.

Last edited by FluffyPersian; 24th November 2012 at 10:41 PM.
FluffyPersian is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 24th November 2012, 11:35 PM   #78
Musibrique
Scholar
 
Join Date: Apr 2012
Posts: 52
Originally Posted by xtifr View Post
It's part of the claim. MDC tests claims. If you say, "this succeeds 80% of the time", then that's what will be tested (assuming a protocol can be agreed on, and other requirements met).
A significance test does not simply test the claim, "this succeeds 80% of the time" The p-value tests the null-hypothesis, not the alternative.
Musibrique is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 25th November 2012, 03:48 AM   #79
GnaGnaMan
Critical Thinker
 
Join Date: Sep 2008
Posts: 390
Originally Posted by FluffyPersian View Post
My copy function isn't working at the moment, but you implied that someone isn't a psychic can't achieve a p-value of less than <0.001 happens to be set. Is that correct?
I don't see how you get that from what I wrote.
You could say that I think that a someone who's psychic ability is like a baseball players ability to throw a ball should have no trouble achieving virtually arbitrary p-values.
Any professional psychic should be able to achieve tiny p-values if they are able to do what they claim.
More generally, anyone whose psychic ability is so that they notice it in daily life should not have a problem with achieving low p-values in short time.

Originally Posted by FluffyPersian View Post
That's exactly how tests of conventional treatments (throughout the social and medical sciences) are executed. Why should, say, homeopathic treatments be tested completely differently?
That's fairly difficult to explain in a way that's understandable to someone who never dealt with "true believers".
A true believer will never give up his belief but always conclude that the test method is false, or that there was some foul play by nefarious forces. So when the tried and true methods of science say that homeopathy does not work...
That's what necessitates these individual protocol negotiations. And just because people are delusional, does not mean that they are stupid, even if they may have mild formal thought disorder. I think, IMHO, that much of the problem of reaching agreements comes from the fact that they navigate around negative results like a blindsighted person navigates around obstacles.

Anyways, it is not the JREF that dictates what someone should be able to do to qualify as psychic. The claimant should come up with the protocol. The JREF is only interested that whatever claim is being made cannot be achieved by normal means. That is certainly a departure from normal scientific practice but as far as normal science is concerned all that stuff is wrong/non-existent.
__________________
"I don't think it's quite fair to condemn the whole program because of a single slip-up."
GnaGnaMan is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Old 25th November 2012, 09:10 AM   #80
Musibrique
Scholar
 
Join Date: Apr 2012
Posts: 52
Originally Posted by GnaGnaMan View Post
I don't see how you get that from what I wrote.
You could say that I think that a someone who's psychic ability is like a baseball players ability to throw a ball should have no trouble achieving virtually arbitrary p-values.
Any professional psychic should be able to achieve tiny p-values if they are able to do what they claim.
More generally, anyone whose psychic ability is so that they notice it in daily life should not have a problem with achieving low p-values in short time.
Not necessarily. It actually depends on how good is the psychic. A telekinetic, for example, can be a professional of getting 55% heads/tails. However, it is most likely that the psychic will not reach a p-value of 0.001 in the small run. If the JREF wants to test his/her claim with an 80% chance of getting a p-value of 0.001, JREF would need a sample-size of 1704 coin tosses. Of course this isn't a problem with claimants who have high hit rates and claim they are professional of doing so.

Last edited by Musibrique; 25th November 2012 at 09:12 AM.
Musibrique is offline   Quote this post in a PM   Nominate this post for this month's language award Copy a direct link to this post Reply With Quote Back to Top
Reply

JREF Forum » JREF Topics » Million Dollar Challenge

Bookmarks

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -7. The time now is 06:18 AM.
Powered by vBulletin. Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
2001-2013, James Randi Educational Foundation. All Rights Reserved.

Disclaimer: Messages posted in the Forum are solely the opinion of their authors.