What's the required p-value to beat? - Page 3

FluffyPersian · 25th November 2012, 09:35 AM

Originally Posted by Musibrique

A significance test does not simply test the claim, "this succeeds 80% of the time" The p-value tests the null-hypothesis, not the alternative.

Actaully, "this succeeds 80% of the time" can be the null hypothesis But in that case, you'd have a 2-tailed test with an alternative hypothesis of

HA: Pi<>.8

In most MDC tests, on the other hand, you'd want to test whether something succeeds MORE THAN X% of the time.

However, as I said aboavae, the significance level and the population proportion being tested are two different parameters ("inputs").

xtifr · 25th November 2012, 01:46 PM

Originally Posted by FluffyPersian

No, it doesn't. The significance level, which is what you're apparently referring to, is separate from the threshold proportion and the sample size.

Yes, sorry. The point is, a claim "this works 100% of the time" will require a different protocol from the claim "this works 80% of the time".

GnaGnaMan · 25th November 2012, 02:48 PM

Originally Posted by Musibrique

Not necessarily. It actually depends on how good is the psychic. A telekinetic, for example, can be a professional of getting 55% heads/tails. However, it is most likely that the psychic will not reach a p-value of 0.001 in the small run. If the JREF wants to test his/her claim with an 80% chance of getting a p-value of 0.001, JREF would need a sample-size of 1704 coin tosses. Of course this isn't a problem with claimants who have high hit rates and claim they are professional of doing so.

Not sure how this relates to what I was saying.
It should be possible to perform 2000 coin tosses in less than 2 hours so there does not seem to be a problem with getting tiny p values if someone can really get 55% right on a coin toss.

OTOH how would one notice that ability in the first place? That would take 1000s of recorded coin tosses as well.

Also, if someone is genuinely telekinetic then surely one would be able to test this in a better way than coin tossing. It's seems gratuitous to introduce a random element. Why not just use a force sensor? I guess that's not the point, though.

Marcus · 25th November 2012, 03:55 PM

There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary).

This would be simple indeed if the various practitioners could do what they say they can do.

The fact that no one has passed the preliminary speaks volumes.

Musibrique · 15th December 2012, 09:02 PM

Originally Posted by Marcus

There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary).

This would be simple indeed if the various practitioners could do what they say they can do.

The fact that no one has passed the preliminary speaks volumes.

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

What you're saying is nothing but absurd.

Pixel42 · 16th December 2012, 12:02 AM

Originally Posted by Musibrique

the probability of an applicant reaching such p-value actually depends on the applicant's hit rate.

Which is why JREF always ask the applicant what their hit rate is, so they can take that into account as they help them design a suitable protocol.

GnaGnaMan · 16th December 2012, 02:39 AM

Originally Posted by Musibrique

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

Correct enough but it is unclear what you think this implies.
Nothing about this makes what Marcus said "absurd"-

Marcus · 17th December 2012, 04:28 AM

Originally Posted by Musibrique

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

What you're saying is nothing but absurd.

It's all about applicants being able to perform the paranormal feats they claim that they can.

If someone can really communicate telepathically or dowse for gold, the exact probabilities necessary to satisfy JREF are not that important, they just have to demonstrate their claimed abilities.

In fact, the odds are just a guideline. Not all claims can easily have probabilities assigned to them. If JREF decides a claim is paranormal, and can be tested in a reasonable amount of time, it is a potential candidate.

xterra · 17th December 2012, 09:17 AM

Musibrique and Marcus,

I believe that my post

http://forums.randi.org/showthread.p...67#post8392067

is relevant to your disagreement.

xterra

25th November 2012, 09:35 AM	#81
FluffyPersian Thinker Join Date: Aug 2011 Posts: 191	Originally Posted by Musibrique A significance test does not simply test the claim, "this succeeds 80% of the time" The p-value tests the null-hypothesis, not the alternative. Actaully, "this succeeds 80% of the time" can be the null hypothesis But in that case, you'd have a 2-tailed test with an alternative hypothesis of HA: Pi<>.8 In most MDC tests, on the other hand, you'd want to test whether something succeeds MORE THAN X% of the time. However, as I said aboavae, the significance level and the population proportion being tested are two different parameters ("inputs").
FluffyPersian Thinker Join Date: Aug 2011 Posts: 191

25th November 2012, 01:46 PM	#82
xtifr Muse Join Date: Apr 2012 Location: Sol III Posts: 586	Originally Posted by FluffyPersian No, it doesn't. The significance level, which is what you're apparently referring to, is separate from the threshold proportion and the sample size. Yes, sorry. The point is, a claim "this works 100% of the time" will require a different protocol from the claim "this works 80% of the time".
	__________________ "Those who learn from history are doomed to watch others repeat it." -- Anonymous Slashdot poster "The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore." -- James Nicoll

25th November 2012, 02:48 PM	#83
GnaGnaMan Thinker Join Date: Sep 2008 Posts: 193	Originally Posted by Musibrique Not necessarily. It actually depends on how good is the psychic. A telekinetic, for example, can be a professional of getting 55% heads/tails. However, it is most likely that the psychic will not reach a p-value of 0.001 in the small run. If the JREF wants to test his/her claim with an 80% chance of getting a p-value of 0.001, JREF would need a sample-size of 1704 coin tosses. Of course this isn't a problem with claimants who have high hit rates and claim they are professional of doing so. Not sure how this relates to what I was saying. It should be possible to perform 2000 coin tosses in less than 2 hours so there does not seem to be a problem with getting tiny p values if someone can really get 55% right on a coin toss. OTOH how would one notice that ability in the first place? That would take 1000s of recorded coin tosses as well. Also, if someone is genuinely telekinetic then surely one would be able to test this in a better way than coin tossing. It's seems gratuitous to introduce a random element. Why not just use a force sensor? I guess that's not the point, though.
GnaGnaMan Thinker Join Date: Sep 2008 Posts: 193	__________________ "Any sufficiently analyzed magic is indistinguishable from science!"

15th December 2012, 09:02 PM	#85
Musibrique Scholar Join Date: Apr 2012 Posts: 52	Originally Posted by Marcus There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary). This would be simple indeed if the various practitioners could do what they say they can do. The fact that no one has passed the preliminary speaks volumes. You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power. What you're saying is nothing but absurd.
Musibrique Scholar Join Date: Apr 2012 Posts: 52

16th December 2012, 12:02 AM	#86
Pixel42 Schrödinger's cat Join Date: May 2004 Location: Wiltshire, UK Posts: 4,455	Originally Posted by Musibrique the probability of an applicant reaching such p-value actually depends on the applicant's hit rate. Which is why JREF always ask the applicant what their hit rate is, so they can take that into account as they help them design a suitable protocol.
	__________________ "The correct scientific response to anything that is not understood is always to look harder for the explanation, not give up and assume a supernatural cause". David Attenborough.

25th November 2012, 03:55 PM	#84
Marcus Graduate Poster Join Date: Feb 2007 Location: Austin Posts: 1,866	There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary). This would be simple indeed if the various practitioners could do what they say they can do. The fact that no one has passed the preliminary speaks volumes.

16th December 2012, 02:39 AM	#87
GnaGnaMan Thinker Join Date: Sep 2008 Posts: 193	Originally Posted by Musibrique You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power. Correct enough but it is unclear what you think this implies. Nothing about this makes what Marcus said "absurd"-
GnaGnaMan Thinker Join Date: Sep 2008 Posts: 193	__________________ "Any sufficiently analyzed magic is indistinguishable from science!"

17th December 2012, 04:28 AM	#88
Marcus Graduate Poster Join Date: Feb 2007 Location: Austin Posts: 1,866	Originally Posted by Musibrique You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power. What you're saying is nothing but absurd. It's all about applicants being able to perform the paranormal feats they claim that they can. If someone can really communicate telepathically or dowse for gold, the exact probabilities necessary to satisfy JREF are not that important, they just have to demonstrate their claimed abilities. In fact, the odds are just a guideline. Not all claims can easily have probabilities assigned to them. If JREF decides a claim is paranormal, and can be tested in a reasonable amount of time, it is a potential candidate.

17th December 2012, 09:17 AM	#89
xterra Muse Join Date: Apr 2012 Location: On the outskirts of Nowhere; the middle was too crowded Posts: 651	Musibrique and Marcus, I believe that my post http://forums.randi.org/showthread.p...67#post8392067 is relevant to your disagreement. xterra