Need help defending million dollar challenge

[Dominus]

I frequent a community forum that has huge numbers of general topics from science to humor to politics to religion. Many many paranormal claims are made there, so I've begun telling people about the million dollar challenge, challenging them. I'm getting all kinds of bunk about how it's unfair, and many of the myths that people propogate are easy to deflect.

However a couple of people, one of them a self-proclaimed successful dowser(!) is giving me a hard time with an argument that's beyond my understanding. Can someone involved with the JREF or these tests help me out here? Here are the primary points made by these scoffers:

"For a statistical test, the level of significance required to win the challenge is .000001 -- a criterion that would invalidate virtually all mainstream science." (Poster claims Randi said this at a previous TAM)

The "dowser" chimed in: "Not quite. It also depends also on the power of the test to detect the effect (and the effect size it’s detecting): Remember you are detecting a standardized mean difference (which is where power and effect size come in). Typically for a lowish effect size (d=0.3) you’d need a sample size of just under 2000."

"The 0.00001 confidence level I remember reading about in a SCICOP article. I’ll see if I can dig it out. Dkyle’s point (and I hope I’m not putting words in his mouth) is that by setting such a high confidence level they are doing something that isn’t normally done in scientific studies: You normally don’t bother to set that high a confidence level because the real risk is that smaller effects are swamped by systematic errors in the experiment. How science normally takes care of the chance that the results are due to random chance and avoid systematic errors is by the reproducibility of the tests. Since by its very nature the Randi challenge depends on once set of tests, it’s already outside of that common scientific practice."

"A better form of verification would be to set a lower confidence level (0.005 would be fine since this is a commonly reported one) and then have other groups attempt reproduction: The challenge criteria would then be a certain number of groups finding the effect (remembering that absence of evidence isn’t evidence of absence)."

[Shrinker]

If I said my dad's metal detector could find a lump of iron hidden in a random shoe box 8 times out of ten, I wouldn't need to squabble over a bunch of irrelevant numbers. Either I can do it or I can't. What's his problem? Suddenly his amazing supernatural talent is a barely detectable effect? Does he tell his customers that?

[Pidge]

remembering that absence of evidence isn’t evidence of absence

Isn't that better or more fully stated as "absence of evidence is not evidence of absence, unless it would be reasonable to expect evidence"? (e.g. Sagan's example of "I've got a fire-breathing invisible dragon living in my garage" - it would be reasonable to expect some evidence to back up that claim, lest the claimant be judged to be either kidding around, or - worse - a "sandwich short of a picnic") i.e. how that test is applied depends on what is being tested.

If you claim to be able to reproducably demonstrate a paranormal effect (under strictly controlled conditions to eliminate the potential for improving the chance of a sucessful trial though means other that claimed, more colloquially known as "cheating"), then the confidence level needs to be high minimise success through chance alone, which also aids in making the success of the claimant self-evident.

e.g. a possible test for dowsing for <insert substance> would involves placing a sample of <insert substance> under one of 10 overturned opaque cups. There is a 1 in 10 chance of simply guessing which cup the <insert substance> is under in each trial. If a dowser could improve those odds to 20% for each trial, you'd need a large number trials to be able to reliably distinguish between simply guessing and a dowsing effect to the 1 in 106 confidence level (I can't do the math off the top of my head. It's been too long since I looked at it!).

The lottories have worse than 1 in 106 odds for the main prize. Guess which method of enriching oneself has so far been more sucessful for some?

[nathan]

The "dowser" chimed in: "Not quite. It also depends also on the power of the test to detect the effect (and the effect size it’s detecting): Remember you are detecting a standardized mean difference (which is where power and effect size come in). Typically for a lowish effect size (d=0.3) you’d need a sample size of just under 2000."

The dowser is closer to the truth here than the initial comment. How many trials are necessary depends on the nature of the test. Let's take the proverbial random coin toss as an example (ignoring the other thread pointing to evidence that it can be biased). If the claim is 'I can predict a coin toss', then to get a .000001 significance, the claimant would have to successfully predict a sequence of 20 coin tosses. That's relatively self contained.

The key here is that we've reduced each trial down to a binary choice. The two options differ only in their X quality.

Most dowsing claims seem to be of the form 'I can find X [in a region with many choices but only one X] all the time -- well ok, maybe 90% of the time'. so one devises a protocol where the dowser has to choose between two targets, only one of which contains X. Doing that 20 times would give a confidence of .000001 IF the claim was that the dowser would be 100% accurate. For 90% you'd have to do some more (and maths will tell you how many more).

Now, if the dowser's claim degenerates to 'I can distinguish these 52% of the time', it will take MANY more than 20 tests, but that's because the dowser's claim is not at all strong. Most dowsing claims are much stronger that that.

[egslim]

"For a statistical test, the level of significance required to win the challenge is .000001 -- a criterion that would invalidate virtually all mainstream science."
While I have serious doubts about his claim that this is true, it is also irrelevant.
Suppose the dowser and I are both tasked to find a piece of metal under one of a number of cups. He uses a dowsing rod, I use a magnet. Depending on the number of cups and our claimed accuracy a calculatable number of demonstrations is necessary to decrease the chance of succeeding through luck to a certain percentage.
As long as this number of demonstrations isn't unreasonable, I don't see any issue. If using a magnet and a large piece of iron was allowed, I would have no trouble whatsoever collecting the million. :)

The Challenge eliminates judgement, but mainstream science doesn't - because there is a margin of error in allmost any scientific measurement. Furthermore, since experimental science usually borders on the limits of technology, measuring errors are relatively large. That's not true for the Challenge, either.

Typically for a lowish effect size (d=0.3) you’d need a sample size of just under 2000."
Obviously, small effects increase the necessary sample size. If I claim a 51% succesrate where 50% would be indicated by chance, an awful lot of tests would be needed. On the other hand, while it would still be a paranormal claim, in practice I personally would consider it too insignificant to be anything but luck.

How science normally takes care of the chance that the results are due to random chance and avoid systematic errors is by the reproducibility of the tests. Since by its very nature the Randi challenge depends on once set of tests, it’s already outside of that common scientific practice."
I don't think Randi has ever claimed the Challenge was science.
Reproducibility works to eliminate cheating, errors and chance.
In the Challenge cheating is eliminated because of how the Protocol is determined.
Since the Challenge works with a discrete set of clearly distinguishable possible outcomes measurement errors are eliminated too.
And finally, the Challenge does demand reproducibility - because there are two rounds, and every round may contain as many reproductions as agreed to beforehand.

A better form of verification would be to set a lower confidence level (0.005 would be fine since this is a commonly reported one) and then have other groups attempt reproduction
Sneaky.
In theory it would be no different to either have the same person do 10 tests, or have 10 persons do one test each. You'd still need to agree beforehand how many reproductions are necessary, and how many of those need to be succesful to constitute a succes. No mathematical difference there.
However, while a single succesful reproduction would obviously fail the Challenge, it would certainly be a nice propagande coup for the dowsers.

[ChristineR]

Actually it's .001 repeated twice. .001 is a common level for unexpected or unusual results. For controversial results even higher levels are common.

If someone were to publish a study showing paranormal activity to .005 confidence everyone would brush it off as a chance result. It certainly wouldn't be worth $1,000,000.

And 2000 trials is not that impressive. Scientests do tests with more trials than that all the time and don't expect million dollar prizes.

[hellaeon]

I have no idea whats so hard about the challenge. These people know what they do is rubbish or they would have no problem being tested.

[aargh57]

I have no idea whats so hard about the challenge. These people know what they do is rubbish or they would have no problem being tested.


I think that many are self delusional and when confronted with the challenge they quickly dismiss it because of stuff they've heard (from plenty of woo sites). I think that others believe they have evidence of the paranormal but can't be tested conventionaly. Of course, some do know what they do is rubbish, but not all.

[Rasmus]

I only have one question: If the effects are so small that they are barely better than chance, then how come that the claimants know about their special abilities in the first place?

If I could predict coin tosses with even as much as 53% I would never know about it. I just would never sit down and run 10.000 iterations just for the sheer hell of it and then see that the results are 5300 vs. 4700 in my favour.

Rasmus.

[ChristineR]

Humans do not have a good intuitive grasp of probability. Using coin flips again, there is a 1/32 chance of getting five in a row, a streak which would drive some gamblers into ectasy. Ten in a row (1 in 1024) would be enough to convince most people they are psychic.

[Mongrel]

I think the other thing to mention is the odds that they quote are are Vs chance, random choosing in the case of dowsing.

Science (I believe) could score perfectly with Shrinkers dads metal detector (assuming proper calibration and a pocket of spare batteries)

[Gr8wight]

"For a statistical test, the level of significance required to win the challenge is .000001 -- a criterion that would invalidate virtually all mainstream science." (Poster claims Randi said this at a previous TAM)

The "dowser" chimed in: "Not quite. It also depends also on the power of the test to detect the effect (and the effect size it’s detecting): Remember you are detecting a standardized mean difference (which is where power and effect size come in). Typically for a lowish effect size (d=0.3) you’d need a sample size of just under 2000."

"The 0.00001 confidence level I remember reading about in a SCICOP article. I’ll see if I can dig it out. Dkyle’s point (and I hope I’m not putting words in his mouth) is that by setting such a high confidence level they are doing something that isn’t normally done in scientific studies: You normally don’t bother to set that high a confidence level because the real risk is that smaller effects are swamped by systematic errors in the experiment. How science normally takes care of the chance that the results are due to random chance and avoid systematic errors is by the reproducibility of the tests. Since by its very nature the Randi challenge depends on once set of tests, it’s already outside of that common scientific practice."

"A better form of verification would be to set a lower confidence level (0.005 would be fine since this is a commonly reported one) and then have other groups attempt reproduction: The challenge criteria would then be a certain number of groups finding the effect (remembering that absence of evidence isn’t evidence of absence)."

yada, yada, yada. The most salient points are these:
1) The test is designed with the input of the claimant.
2) The test is designed to specifically test what the claimant says he or she is able to do.
3) The level of success required for a positive result is agreed upon by both parties.
4) Prior to commencement of the test, the claimant is asked to verify that they agree that the test is a fair one.

The mumjo jumbo about confidence levels, and number of zeros to the right of the decimal point are nothing more than an attempt to deflect attention from the fact that you have asked them if they are willing to demonstrate their ability under controlled conditions, and they are not.

Additional, specific to dowsing: In a test of any dowsing claims, the test protocols almost always include an unblinded test in which the claimants are able to demonstrate that they can locate the thing they are dowsing for when they know where it is, in order to eliminate later claims that there was some kind of environmental interference.

[Rasmus]

Additional, specific to dowsing: In a test of any dowsing claims, the test protocols almost always include an unblinded test in which the claimants are able to demonstrate that they can locate the thing they are dowsing for when they know where it is, in order to eliminate later claims that there was some kind of environmental interference.

Yes, "calibration" is build into the protocol if possible. And not only for dowsing. IIRC, the test for one of the CD-enhancing-devices was done much the same way, allowing the claimant to listen to both CDs unblinded at first.

Rasmus.

[petre]

To me, the best demonstration of the challenge is that you can point to actual tests that have been conducted as examples.

For example, for an astrology applicant he states that correctly guessing the sign of 10 out of 12 subjects based on a personality test would suffice (at least for the preliminary).

Getting 12 out of 12 would seem a bit demanding, and the odds for that by random chance would be about 1/480,000,000

Getting 10 out of 12 seems more reasonable (for example, maybe you mixed up the Gemini and Taurus), yet the odds for success by random chance there are still around 1/27,500 (I think).

There's an example where random chance predicts around an 8% accuracy rate (1/12) and with an expected performance of 83.33% (10/12) you can demonstrate it with just 12 trials!

A brief review of the challenge threads will readily give several other examples of suggested protocols that seem quite reasonable if the stated ability exists as described.

Or, you could always take the other angle and make others support every claim they make as well: "For a statistical test, the level of significance required to win the challenge is .000001 -- a criterion that would invalidate virtually all mainstream science". First, I'd ask that person about the source of that quote. Suggest that if no credible evidence can be found that Randi ever said such a thing, it would be irresponsible to continue to claim that he did. Then allow that the statement itself could still be true. Note that to calculate the level of significance of an effect, that an expected behavior must be described for two cases: the effect existing, and the effect not existing. Then enjoin anyone to describe ANY "mainstream science" effect, the expected behavior if that effect did NOT exist, and a claim that the effect cannot be demonstrated to a required significance of .000001 with observed data.

If anyone does bother to make such a claim, it should be easy to demonstrate a set of test data that does indeed suffice.

[KevinM]

Just an observation but its true absense of evidence is not evidence of absense. But then again absense of evidence isn't proof of some thing either. Thats the mistake most believers make. They think since for example astrology has not been disproven there must be some thing to it(don't quible on the astrology thing its just a random example).

[drfrank]

... They think since for example astrology has not been disproven there must be some thing to it(don't quible on the astrology thing its just a random example).
That's more related to the shifting the burden of proof - believers are required to provide positive evidence of their claims, rather than assuming it true until people provide contrary evidence. Of course, when people do provide contrary evidence they still completely ignore it, but the principle's sound ;)

[Hawk one]

"For a statistical test, the level of significance required to win the challenge is .000001 -- a criterion that would invalidate virtually all mainstream science." (Poster claims Randi said this at a previous TAM)
A classic misunderstanding. I know other posters have commented here, but I am on a high horse today, and believe my own wording to be superior (Yeah, right:p):

Just tell them this: The tests are designed so that if the claimant cannot really do what he does, then there is (roughly) a one in a million chance he will succeed anyway by pure chance.

But, if the claimant can do what he claims, then there is a virtual guarantee that he will succeed in the test.

Point out this, then ask the poster who made this claim if he can understand that it is very logical that you would want to minimise the risk of someone succeeding by chance, as opposed to succeeding by having the ability that he claims he has.

Edited for a nasty spelling mistake. Dammit.

[steenkh]

Hawk one, your wording is one of the best presentations of this issue that I have seen! :clap:

[Hawk one]

Thanks, steenkh, but I do do disagree with you on one part. Take a look at this:


Point out this, then ask the poster who made this claim if he can understand that it is very logical that you would want to minimise the risk of someone succeeding by chance, as opposed to succeeding by having the ability that he claims he has.
As we can see, this sentence is too long, and also rather vague for those that doesn't already understand how the tests are designed. A rewording is most definitely in order to make it clear enough. If anyone wants to use my previous post, then replace the quoted part with:

Point out this fact to him. Then ask him if he agrees that any proper test should be designed to minimise the risk of success by chance alone.

That should be much better, when dealing with someone who doesn't know beforehand.

ETA: Also, it's not like I'm the first to make a rather concise wording of the point. There are plenty of equally good examples elsewhere in this sub-forum. I was just too lazy to find them, so I did my own wording, that's all.

[jon]

"A better form of verification would be to set a lower confidence level (0.005 would be fine since this is a commonly reported one) and then have other groups attempt reproduction: The challenge criteria would then be a certain number of groups finding the effect (remembering that absence of evidence isn’t evidence of absence)."

If what they're asking is for the challenge to pay out on events with a 0.005 probability of happening through pure chance, that would pretty much make it a lottery (actually, much better odds of winning a million than most lotteries, and you don't even need to buy a ticket...) Hell, I'd have a shot at those odds :D

If what they're suggesting is that you need a 0.005 probabality repeated twice, this would also seem to leave open the possibility of just having *lots* of groups take on the challenge - eventually, one will win through blind luck.

I guess, as has been pointed out, it's also the case that this is a challenge, not a scientific experiment. If it's scientific proof of the paranormal people want, all they need to do is cite all the high-quality, reproducible, peer reviewed scientific research that demonstrates its existence ;)

[Kimpatsu]

I have no idea whats so hard about the challenge. These people know what they do is rubbish or they would have no problem being tested.
I'm not so sure about that. I've been trying to get some self-professed psychic jealers to apply for over a year, but they refuse on the grounds that the challenge itself is a sham. Their reasoning goes like this:
There is a large pool of genuine psychics in the world, ony one of whom could win the challenge at any time. That no one has won the challenge is evidence that Randi somehow cheats winners out of the million.
When I reply that maybe the reason no one has ever won is because psychic powers aren't real, they reply that they know psychic powers are real, and that they themselves possess such powers "because of things that have happened to them in the past". (This is exactly what failed applicant Angela Patel said.)
IOW, there is no point in applying for the challenge, because they will be cheated out of the money at the end. This is axiomatic, or why else has no one ever won?
You see the problem...

[Meffy]

That no one has won the challenge is evidence that Randi somehow cheats winners out of the million.
... The winners quietly accept having been cheated out of a million dollars, avoid publicity, and recede into obscurity?

Hm, doesn't sound reasonable to me. The so-called healers' excuse doesn't stand up to much scrutiny.

[Kimpatsu]

... The winners quietly accept having been cheated out of a million dollars, avoid publicity, and recede into obscurity?

Hm, doesn't sound reasonable to me. The so-called healers' excuse doesn't stand up to much scrutiny.
Ah, but they can't use their powers for personal gain. Did you never watch "Charmed"?

[Meffy]

Ah, but they can't use their powers for personal gain.
*plays for supposed claimants a nearly worn-out tape cartridge marked "so win and give it to charity already"*

Did you never watch "Charmed"?
"Charmed," I'm sure not. ;-)

[Kimpatsu]

"Charmed," I'm sure not. ;-)
A fantasy adventure show that rabn for eight years. ending only in May ofthis year, about three sisters who are white witches. They have to juggle boyfriends and careers as well as fighting supernatural evil, but are never allowed to use their powers for personal gain. For example, when one of them gets a premonition of the winning lottery numbers, giving the numbers to an elderly couple who are destitute is fine, but when she attempts to play the numbers herself, her ticket magically disappears.
Like I wish some of these claimants would...

[Meffy]

@Kimpatsu: a belated "Ah. Thanks."

[The Atheist]

What it does (for nth time is emphasise that no matter how many people who claim to be able to exhibit paranormal abilities there are on this planet, not ONE of them will fail to think up a reason not to accept the challenge.

The question, to me, is how to separate those who know they're fakes (cheats) and those who genuinely believe in themselves (insane). The insane ones need help, the cheats need exposing.

As to answering people on other forums, it's a waste of good keyboard time; you are either arguing with nutcases or those who have a vested financial interest in keeping the myths alive - ergo, you cannot win.

By far the best plan is to become a positive poster on their forums. dream up the most insane bollocks you can and post it, they'll buy it and there's a lot less blood than with dairy cows and AK47s.

[Tamazon]

As to answering people on other forums, it's a waste of good keyboard time; you are either arguing with nutcases or those who have a vested financial interest in keeping the myths alive - ergo, you cannot win.
I am certainly finding this to be true as I am in the middle of an argument on a psychic forum myself.

http://community.discovery.com/eve/forums/a/tpc/f/6121956908/m/4611964238

I started it though, my first post is an offer to take the challenge. I have one taker, sort of. She says she has been tested by some PhD at UC Berkeley and passed with flying colors. So naturally she wants to be tested by him again following the exact same criteria. Sigh.

I think she spewed every objection to the test there is. It's not about the money. It's just a publicity stunt for Randi. The money's not there. Randi's not a scientist. She's not a trained seal to perform for my edification or amusement. Blah blah blah. She got REALLY mad at me.

On a side note Atheist, I'll be visiting your beautiful country in January. Looking into moving there. If I do, I'll be joining the NZ Skeptics.

[steenkh]

What it does (for nth time is emphasise that no matter how many people who claim to be able to exhibit paranormal abilities there are on this planet, not ONE of them will fail to think up a reason not to accept the challenge.
Actually, a few of them do try to take the challenge!

[Calcas]

I am certainly finding this to be true as I am in the middle of an argument on a psychic forum myself.


I started it though, my first post is an offer to take the challenge. I have one taker, sort of. She says she has been tested by some PhD at UC Berkeley and passed with flying colors. So naturally she wants to be tested by him again following the exact same criteria. Sigh.

I think she spewed every objection to the test there is. It's not about the money. It's just a publicity stunt for Randi. The money's not there. Randi's not a scientist. She's not a trained seal to perform for my edification or amusement. Blah blah blah. She got REALLY mad at me.

On a side note Atheist, I'll be visiting your beautiful country in January. Looking into moving there. If I do, I'll be joining the NZ Skeptics.

I'm new here but I checked out that link.

Wow. That "Dr" Headding is so full of herself it's a shame.

People like that are scum. She seems too smart to be self delusional so she must simply be a crook. And, at the expense of poor people trying to find their missing sons and daughters.

How sad.

[T'ai Chi]

Say the collective alpha is .001 for tests done by organized skeptical agencies.

So how lmany tests until we'd expect someone to win by chance?

[Silly Green Monkey]

The tests don't stack.

[T'ai Chi]

The tests don't stack.

Aren't they independent?

[drkitten]

Say the collective alpha is .001 for tests done by organized skeptical agencies.

Why are you using that number? Why should I take it at all seriously?

[Number Six]

The bottom line is that it is usually possible to design a test where (a) there is a very small chance the person will succeed if they don't truly have the powers and (b) a good chance the person will succeed if they do have the power. Naturally the JREF wants to keep the first probability small so people that don't really have the powers don't luck out and win. But if you do have the powers that shouldn't be a problem.

Suppose the JREF wants to be conservative and reduce the chance of a person without the powers winning to 1 in 1,000,000 and consequently that decreases the chances that a person with the powers will win to to 50-50. Would that really deter someone that has the powers? If you told me I had a 50-50 shot at winning a millions dollars and I could try once a year (claimants that fail the JREF challenge have to wait a year to try again), I'd be doing it as quickly as possible. I'd have a 0.999 chance of winning within the next 10 years. Even if it cost me a thousand bucks to fly to Florida and take the test (and I don't know if the JREF requires that) that'd only be 10 K over 10 years for a 0.999 chance of winning 1,000 K.

If someone had the powers and wanted to exhibit them, they could easily do so. Either nobody has the powers. Or someone has the powers but declines to exhibit them. Or, for the dislikers of the JREF, someone has the powers and agrees to exhibit them but the JREF cheats on the tests.

[drkitten]

The bottom line is that it is usually possible to design a test where (a) there is a very small chance the person will succeed if they don't truly have the powers and (b) a good chance the person will succeed if they do have the power. Naturally the JREF wants to keep the first probability small so people that don't really have the powers don't luck out and win. But if you do have the powers that shouldn't be a problem.

Suppose the JREF wants to be conservative and reduce the chance of a person without the powers winning to 1 in 1,000,000 and consequently that decreases the chances that a person with the powers will win to to 50-50. Would that really deter someone that has the powers?

Something else that should be pointed out here is that this scenario does not, in fact, describe the behavior of the JREF.

I know of no case on record where a claimant was asked to perform at a a level where their chances of winning were 50/50 if they had the claimed ability. In all cases, they are asked to perform at a level substantially below what they describe in their claim letter. If I claim to be able to dowse for water with 90% accuracy, the JREF might demand that I hit 75%. If I claim to be able to diagnose illness, the JREF may simply demand that I identify if there is an illness present. If I claim to be able to lift boulders with my mind alone, the JREF may simply ask that I lift a pebble.

The applicant is always closely involved in the design of the protocol and has no reason to accept a protocol that exceeds the boundaries of their claims.

[T'ai Chi]

Why are you using that number? Why should I take it at all seriously?

I'm using that number since that is what is typically used in JREFs tests.

And I don't care how youtake it, as usual.

[William Smith]

Say the collective alpha is .001 for tests done by organized skeptical agencies.

So how lmany tests until we'd expect someone to win by chance?

10,000.



Or preppelzall x dampfelzwubb.

[CFLarsen]

Just tell them this: The tests are designed so that if the claimant cannot really do what he does, then there is (roughly) a one in a million chance he will succeed anyway by pure chance.

But, if the claimant can do what he claims, then there is a virtual guarantee that he will succeed in the test.

I''ll steal this one.

[Hawk one]

Hehe. Feel free to replace the bold with italics in case you feel the bold stands too much out.