The tipping point

[mslxl]

Hello! I am a new participant and I have a question for all you Skeptics out there.

Suppose I take the Challenge by saying I can predict the outcome of a coin flip.

So we set up a simple procedure: I make a call, heads or tails, then someone flips a coin and notes the result. The we do it again, and again.

For three tosses, the probability of getting all the answers right is one out of eight, or 12.5%.

For four tosses, one out of sixteen, or 6.25%.

For five tosses, one out of thirty-two, or 3.125%.

For six tosses, one out of sixty-four, or 1.5625%.

Etcetera.

At what point would you say, if you were a scientist (or at what point do you say, if you actually are a scientist): "This should be investigated further. It could have happened by chance, but that probability is low enough to warrant further study."?

At what point would you say: "This is definitely paranormal. Give him the $1M."? (I understand that you don't have the authority to speak on behalf of JREF. I am interested in your private, non-binding opinion anyway.)

Are those two thresholds different? What is the cutoff point for each?

I welcome all comments. Thank you.

mslxl

[eri]

Personally, I would say being able to do it 20 times would definitely be interesting. If repeatable, of course. The JREF would require you to pass an initial test AND a final test, so if you're just hoping you'll get lucky the first time, keep in mind you'll have to get lucky twice.

2^20 = 104858(0?), or about a million-to-one odds.

[ChristineR]

The JREF challenge is in two parts, the preliminary and the final. The challenge targets are chosen to be roughly 1 to 1000 against. Getting 10 in a row right would be 1 in 1024, so to win the million you would get 10 in a row, twice on separate occasions.

Scientists use slightly more formal language but the principles are the same. This (http://news.ft.com/cms/s/5372968a-ba82-11da-980d-0000779e2340,dwp_uuid=77a9a0e8-b442-11da-bd61-0000779e2340.html) talks about the "sigma" without really going into too much detail as to where it comes from. There's no hard and fast rule, but a six or seven sigma variance is usually enough to get a scientist to say "that clearly wasn't chance."

Said scientist may say "it was a cheat, not a paranormal phenomena" but that's a different issue.

[Strider1974]

The big thing to remember here is that extraordinary claims require extraordinary proof.
If you could correctly predict 20 tosses over at least 2 different sessions under controlled conditions I would consider this paranormal and worthy of further study

[Ririon]

It's easy to win the challenge with EXTREME luck. But if that's the way you want to win a million dollars, it's easier and cheaper to do it in a casino. An investment of one dollar in a casino and a similar amount of EXTREME luck will give you one million dollars. The cost of notarized affidavits etc. is higher than one dollar, so it doesn't make sense economically. Unless you have REAL paranormal powers, of course.

[petre]

<deleted>I noted ChristineR covered the first half of what I typed...

I should also add that according to section 5.2 of the FAQ:

What you should expect is for none of the agreed-upon documents to change. There will not be any renegotiation of your abilities or how they will be tested. However, you may be required to demonstrate your claim more conclusively in the final test. This means that the test may include more trials, or may last longer. Rest assured, the test protocol itself will not be altered in ANY WAY.
So for the final test, it is quite possible that the number of trials would be increased from 10 to 20. Thus, with a roughly 1/1000 chance in the preliminary, and 1/1,000,000 chance in the final, that's a combined chance of 1/1 billion. Assuming your time is worth more than 10 cents, you've got better odds in Vegas.

I need to type faster, thanks Ririon for covering the gambling reference as well :P

[Ririon]

..
I need to type faster, thanks Ririon for covering the gambling reference as well :P
I's happened to me a couple of times, too. :) Good point about the possibility of even lower probabilities in the final test, though.

[mslxl]

I have seen six replies so far and none of them addresses my question about the DIFFERENCE between two requirements.

I am NOT thinking about winning the Challenge through luck and I have read all the rules already.

(Thank you, ChristineR, for the link. Interesting reading, and by Benoit Mandelbrot, no less, the founder of fractal geometry!)

To clarify what I am interested in, here is an example about a hypothetical research situation for a scientist. Let us say I am a research biologist trying to isolate something that would stimulate bone regeneration. I know the parameters for regular, normal bone growth. I try grafting various kinds of tissues and compounds onto bone tissue in the hope that some unusual growth will occur.

The part about me being a scientist is not true, so please correct me if I am wrong. It would be great if there were an actual research scientist posting on this board.

It seems to me that for the researcher to take interest in a phenomenon and decide that it must be investigated further, a threshold of one chance in a million, or even of one in a thousand, is not required. It seems to me that if bone growth that has a one percent chance of happening spontaneously does take place repeatedly whenever a certain compound is grafted onto the bone, then the researcher would be foolish not to get excited and look further into it.

So my question was about the DIFFERENCE between this threshold (whatever it may be in real research) and the one for the JREF Challenge.

The subtext of the question is: is there a difference between the mindset of scientists investigating the possibility of paranormal phenomena and the mindset of scientists investigating the possibility of phenomena that are not paranormal, but are not yet known?

mslxl

[ChristineR]

ANY incidence of something as important as spontaneous abnormal bone growth would be of interest to a scientist. If it only happens one time in a thousand it will not have a practical medical use, but if the researcher could figure out how to make it happen more reliably, that would be of use.

As far as practical medical application, it depends. One chance in a thousand does sound pretty slim, but if your leg has been amputated and if you can try the treatment over and over again, I imagine you'd get pretty excited about it.

[drkitten]

I have seen six replies so far and none of them addresses my question about the DIFFERENCE between two requirements.

What you're talking about is broadly the difference between a type I and a type II error (in technical terminology).

To a statistician, a "type I error" is where you mistakenly believe something that isn't true ("I can predict coin flips!") A "type II error" is the opposite, where you disbelieve something that is true ("Gee, I picked the Steelers to lose the Superbowl, so I guess I can't predict football well.") (Yes, I am aware of the more technical definitions involving terms like "null hypothesis"....)

A key aspect in experimental design is the relevant costs of type I and type II errors. For example, if we're looking at a test for safety of some new industrial process, it's much more serious to believe that it's safe when it isn't (type I) than it is to believe it's unsafe when it really is. If we think it's unsafe, we'll simply not use it, but that in and of itself won't put people at risk. If we think it's safe and use it widely, that's endangering a lot of innocent people. Conversely, if I test you for a fatal disease, it may be much more dangerous for me to get a false negative than a false positive. Left untreated, it will probably kill you -- while if I find something, it just means another round of tests and possibly some unnecessary but harmless treatment.

We can balance the statistics (technically, by setting two values called the "alpha cutoff" and the "beta cutoff") to balance the chance of these errors. These parameter are in some sense the
opposite of each other -- if you tighten the alpha cutoff, the chance of a type I error, you are loosening the beta cutoff., the chance of a type II. By convention n most fields of science, the alpha cutoff is set to 0.05 -- one chance in twenty of getting a type I error, assuming there is no actual effect. But if I was working on something really safety-critical, I would set my alpha cutoff to be a much, much smaller value.

The JREF invests substantial resources into running tests -- and of course, will pay out huge sums of money in the event a final test is passed. So from the JREF's perspective, a "false positive" on their challenge is the worst possible outcome. A false negative result isn't nearly as serious, since that just means that the applicant can try again next year. They naturally set the alpha cutoff extremely tightly -- 0.001 for the preliminary challenge, 0.000001 for the final.




To clarify what I am interested in, here is an example about a hypothetical research situation for a scientist. Let us say I am a research biologist trying to isolate something that would stimulate bone regeneration. I know the parameters for regular, normal bone growth. I try grafting various kinds of tissues and compounds onto bone tissue in the hope that some unusual growth will occur.

[...]

It seems to me that for the researcher to take interest in a phenomenon and decide that it must be investigated further, a threshold of one chance in a million, or even of one in a thousand, is not required. It seems to me that if bone growth that has a one percent chance of happening spontaneously does take place repeatedly whenever a certain compound is grafted onto the bone, then the researcher would be foolish not to get excited and look further into it.

I agree with you here, for the most part. The "costs" of running further experiments and looking further into it are pretty low, and the "costs" of not following up on this treatment are fairly substantial to public health. But on the other hand, I'd not necessarily want to start using this treatment, on actual human patients, even on an experimental basis, without better evidence, because the costs of subjecting humans to such a treatment, could be quite high.

A key difference, of course, is that the JREF are not scientists. They're educators, not researchers, as they themselves will admit. If you're not sure that an effect exists, they are not in a position to help you look for it. On the other hand, there are enough people out there who are sure, even if they're misguided, that the JREF manage to stay in the testing business....

[Strider1974]

The subtext of the question is: is there a difference between the mindset of scientists investigating the possibility of paranormal phenomena and the mindset of scientists investigating the possibility of phenomena that are not paranormal, but are not yet known?

Not if they are professionals.

So my question was about the DIFFERENCE between this threshold (whatever it may be in real research) and the one for the JREF Challenge

There is no difference. The scientists are looking for a repeatable result which can be shown to be NOT the result of chance.

[Rasmus]

So my question was about the DIFFERENCE between this threshold (whatever it may be in real research) and the one for the JREF Challenge.

The JREF challenge isn't science.

A scientist will investigate a phenomenon; wheras the JREF wants you to show that a phenomenon exists and can be classified as paranormal.

Science will look at things that are utterly possible and try to find reasonable explanations.

How high are the chances of spontaneous bone-regeneration?
So if i have an untreated bone and a treated one, and the treated one outperforms the untreated one, I would quite quickly have odds that are much much better than 1:000.

Rasmus.

[mslxl]

Sorry, this is a reply to drkitten. Still learning to use this board.

"The JREF invests substantial resources into running tests -- and of course, will pay out huge sums of money in the event a final test is passed. So from the JREF's perspective, a "false positive" on their challenge is the worst possible outcome. A false negative result isn't nearly as serious, since that just means that the applicant can try again next year. They naturally set the alpha cutoff extremely tightly -- 0.001 for the preliminary challenge, 0.000001 for the final."

Thank you very much for that distinction. It makes a lot of sense.

What you're saying, then, is that if I go to the final test and predict 19 out of 20 coin flips accurately -- or, for that matter, predict 9 out of 10 accurately at the prelim -- I won't make any money... but anybody who is interested in the advancement of science or the expansion of knowledge should grab hold of me and not let go. An interesting paradox.

mslxl

[mslxl]

This is an answer to ChristineR. Sorry, I am still learning to operate on this forum.

"As far as practical medical application, it depends. One chance in a thousand does sound pretty slim, but if your leg has been amputated and if you can try the treatment over and over again, I imagine you'd get pretty excited about it." [/QUOTE]

I did not mean how rarely it happens in the experiments. I meant how unusual the occurrence is compared to regular regeneration with nothing added. In that hypothetical situation, you can assume that the results can be repeated.

mslxl

[mslxl]

How high are the chances of spontaneous bone-regeneration?
So if i have an untreated bone and a treated one, and the treated one outperforms the untreated one, I would quite quickly have odds that are much much better than 1:000.

Rasmus.

Not necessarily. I am not talking about growing an extra bone where that would never happen naturally. I am talking about introducing a new factor that would increase the RATE of regeneration (like for example electromagnetic pulse stimulation which is used in medicine). So the effect would have to be measured statistically by comparing it to known rates of regeneration when there isn't an added factor. It would have to be expressed statistically and it could be a MILD statistical increase.

mslxl

[ChristineR]

What you're saying, then, is that if I go to the final test and predict 19 out of 20 coin flips accurately -- or, for that matter, predict 9 out of 10 accurately at the prelim -- I won't make any money... but anybody who is interested in the advancement of science or the expansion of knowledge should grab hold of me and not let go. An interesting paradox.

mslxl

Well, sort of. You don't have to claim you can get 20 of 20 unless you are actually sure you can. Say you claim you can get 9 of 10. That's a 1/512 chance, so that' s not strict enough for the prize. You'd have to go for 17 out of 20 instead.

If you did claim 10 of 10 and got 9 of 10 you'd surely be asked to replicate the experiment by persons other than the JREF, and if you agreed and could replicate a few times, you'd start to get attention. Really 1/512 is not rare enough to get the attention of anybody, but it would get the attention of paranormalists.

[petre]

Not necessarily. I am not talking about growing an extra bone where that would never happen naturally. I am talking about introducing a new factor that would increase the RATE of regeneration (like for example electromagnetic pulse stimulation which is used in medicine). So the effect would have to be measured statistically by comparing it to known rates of regeneration when there isn't an added factor. It would have to be expressed statistically and it could be a MILD statistical increase.

mslxl

I think the basics of the explaination are here, but let me try to sum up (and address your full question this time).

Almost any statistical anomoly will draw the attention of a researcher, depending on the implications of the hypothesized cause. What's important is that the end result can be demonstrated repeatedly with some level of success.

Similarly, a persistant statistical anomoly could still qualify for the JREF challenge even if it doesn't show perfect performance. If you can predict coin flips only 7 out of 10 times, but your success is consistently near that level, such a thing could qualify for the challenge. The number of trials is simply increased such that the chances of that level success by chance fall below the 1/1000 range (obviously, something with very low improvement over chance might require so many tests that it isn't feasable to do this, and such a claim would not qualify).

Note that just because an anomoly draws the attention of a researcher, it isn't necessarily of any interest either. A good one will study the data, perform repeated tests, and then undergo some kind of peer-review process.

So to that end, the challenge is like the peer-review process for the paranormal. Folks examine strange events, and when one shows itself to be reproducable it can be submitted to JREF. Sadly to date, nothing paranormal has been reliably reproducable.

[ChristineR]

This is an answer to ChristineR. Sorry, I am still learning to operate on this forum.

"As far as practical medical application, it depends. One chance in a thousand does sound pretty slim, but if your leg has been amputated and if you can try the treatment over and over again, I imagine you'd get pretty excited about it."

I did not mean how rarely it happens in the experiments. I meant how unusual the occurrence is compared to regular regeneration with nothing added. In that hypothetical situation, you can assume that the results can be repeated.

mslxl[/QUOTE]

Concerning this, again, it depends. Here we can get into some heavy statistics. Say one in a thousand spontaneously get better. Say you run your test with one hundred people, and two get better. Could it just be chance? It's possible to work out the statistics as to what percentage of one hundred person trials have two one-in-a-thousand spontaneous remissions. Is it worth investigating? It depends not only on how rare it is, but also other circumstances.

[drkitten]

What you're saying, then, is that if I go to the final test and predict 19 out of 20 coin flips accurately -- or, for that matter, predict 9 out of 10 accurately at the prelim -- I won't make any money... but anybody who is interested in the advancement of science or the expansion of knowledge should grab hold of me and not let go. An interesting paradox.


I don't see the paradox. Part of my job as an educator is to talent-spot -- to identify promising young people who have the potential to be good scholars in a particular area. I might see someone at a science fair or school essay contest or something who has the potential to be a very good scholar, but who hasn't yet done anything particularly noteworthy as of yet.

This is basically what any talent scout does, in any field. College recruiters haunt high school basketball games looking for people who have the potential to be a useful part of a team, without really caring about whether they're winning games today.

At the same time, the people who award high school basketball championships don't care about potential, they care about what youv'e accomplished.

The JREF specifically rewards accomplishment, not potential.

But the other thing you're not taking account of is that you get to define your own level of accomplishment at the JREF challenge. You don't have to claim to be 100% perfect, as long as you can accurately claim to be better-than-chance and you can demonstrate it. So if I claim to be merely a near-perfect coin caller, but that I can get 80 right out of 100, the JREF would probably accept that claim and test it on that basis. I could probably claim to be able to get "at least 75" right out of 100.... if I expect 80, that gives me a little bit of room for error and still be able to win.

But here again, we need some sort of cutoff for accomplishment. At some point, it will have to come down to a "did I win or didn't I" decision. If I expect 80, and I agree with the JREF that 75 or better will constitute a "success," then scoring only 74 will not win me the moey. I wasn't as good as I thought I was, and I didn't accomplish what I told them I would.

[mslxl]

Well, sort of. You don't have to claim you can get 20 of 20 unless you are actually sure you can. Say you claim you can get 9 of 10. That's a 1/512 chance, so that' s not strict enough for the prize. You'd have to go for 17 out of 20 instead.

If you did claim 10 of 10 and got 9 of 10 you'd surely be asked to replicate the experiment by persons other than the JREF, and if you agreed and could replicate a few times, you'd start to get attention. Really 1/512 is not rare enough to get the attention of anybody, but it would get the attention of paranormalists.

Getting 9 out of 10 correct is not a 1/512 chance. It's a 10/1024 chance, or 11/1024 if you include the particular case of getting all ten right.

"Really 1/512 is not rare enough to get the attention of anybody, but it would get the attention of paranormalists."

So a paranormalist is not "somebody"? Ahem...

[mslxl]

Similarly, a persistant statistical anomoly could still qualify for the JREF challenge even if it doesn't show perfect performance. If you can predict coin flips only 7 out of 10 times, but your success is consistently near that level, such a thing could qualify for the challenge. The number of trials is simply increased such that the chances of that level success by chance fall below the 1/1000 range (obviously, something with very low improvement over chance might require so many tests that it isn't feasable to do this, and such a claim would not qualify).

This is different from my previous understanding of the 1/1000 and the 1/1000 000 requirements. It makes more sense and it makes negotiating for a protocol easier. Thank you.

[mslxl]

Concerning this, again, it depends. Here we can get into some heavy statistics. Say one in a thousand spontaneously get better. Say you run your test with one hundred people, and two get better. Could it just be chance? It's possible to work out the statistics as to what percentage of one hundred person trials have two one-in-a-thousand spontaneous remissions. Is it worth investigating? It depends not only on how rare it is, but also other circumstances.

I think you are still misunderstanding my hypothetical situation -- and I will take the blame for not being clear enough.

When I said 1/100, I did not mean one patient out of a hundred gets better than he would have without the treatment.

I meant the overall average improvement of patients with the treatment would put them as a group within the top 1% most improved patients without the treatment. For all I know, the patients with the treatments could have ALL shown above average improvement. Or only some of them, but on the average, as a group they would be in that top 1%. Does that make more sense?

This is not a yes/no situation like flipping a coin, so it doesn't work as an analogy to that. There is no "officially normal" rate of bone growth, except as an average of actual measurements. So an increase in the rate of growth in a given population would have to be assessed as an average compared to another average -- that of the bone growth rate of people who are not receiving the treatment.

I am not a statistics person and I feel that a real statistics person could give me a tiny shove at this point and I would drown in this puddle.

[mslxl]

But the other thing you're not taking account of is that you get to define your own level of accomplishment at the JREF challenge. You don't have to claim to be 100% perfect, as long as you can accurately claim to be better-than-chance and you can demonstrate it. So if I claim to be merely a near-perfect coin caller, but that I can get 80 right out of 100, the JREF would probably accept that claim and test it on that basis.

This is the same thing that Petre said and it changes my understanding of the 1/1000 prelim and 1/1000000 final requirements. It's all a matter of negotiation and the requirements can be met through repetition. Thank you.

[tkingdoll]

Is this about abnormal spontaneous bone growth then?

There was a challenge application a while back from someone claiming to be able to make the bones in their hand grow significantly, at will.

I'll see if I can find the link.

ETA It was this one:

http://forums.randi.org/showthread.php?t=41108

[mslxl]

I don't see the paradox.

The paradox is that I might not accomplish what I said I would and therefore "lose" the Challenge, but still accomplish something that any reasonably intelligent person -- whether a scientist, a student of the paranormal or a layperson -- would deem to be so improbable that it absolutely demands further investigation. That is what I suspected when I started this thread and your answers, and others', confirm it.

[mslxl]

Is this about abnormal spontaneous bone growth then?

No, it isn't.

Hypothetical, adj. Assumed by hypothesis, supposed.

Hypothesis, n. A proposition assumed as a premise in an argument.

[drkitten]

I meant the overall average improvement of patients with the treatment would put them as a group within the top 1% most improved patients without the treatment. For all I know, the patients with the treatments could have ALL shown above average improvement. Or only some of them, but on the average, as a group they would be in that top 1%. Does that make more sense?

This is not a yes/no situation like flipping a coin, so it doesn't work as an analogy to that. There is no "officially normal" rate of bone growth, except as an average of actual measurements. So an increase in the rate of growth in a given population would have to be assessed as an average compared to another average -- that of the bone growth rate of people who are not receiving the treatment.


Happy to help. What you describe is actually a very typical type of investigation in the biological field. We gather two "sufficiently large" group of people, give one group one treatment, the other group the other treatment, and then assess the effectiveness.

I'm about to get semi-technical here, so I apologize in advance.

SImply taking "averages" is not enough here, although they're certainly informative. What is actually important (and, fortunately, calculatd as a matter of routine) is not just the average, but also the "variance," which is the amount of spread around the average that you see within the two groups. The usual measure of variance is captured in the concept of "standard deviation," which (under normal circumstances) is the distance around the average that catches about 67 percent of the cases.

For example, suppose that I measure something on a test and get an "average" of ten, and a "standard deviation" of 2. That means that about 67% of the people who took the test got between 8 and 12, respectively. About 90% of the people got between 6 and 14, and over 99% of them got between 4 and 16.

So for each group, we can calculate the average and the standard deviation of how much (e.g.) bone regrowth there is. And there is a well-established statistical procedure to determine how likely it is that both groups were drawn from identical populations -- i.e. that the difference in treatment had no effect. More accurately, it calculates, starting from an assumption that the two groups were so drawn, the probability that results as extreme as the observed would be obtained.

This is where the alpha cutoff comes in. I can say, for example, that if the chance of getting resutls as good as I got "by chance alone" is less than 5%, I will pubish (and ask for more money). Most medical journals will agree with that particular alpha cutoff. I might be more conservative and want a tighter cutoff, or I might be doing a preliminary study and I'd be happy with 10% or even 20%. But the higher the alpha cutoff, the greater chance I have of chasing a wild goose.

The reason that the variance is important is because it's an important element in assessing the level of chance. Specifically, the larger the variance, the greater range of variability you expect in the outcomes, so the better the chance of a spurious "good" finding.....

If you have a real set of numbers that you need to have crunched (for whatever reasons), I'm sure there's someone at the local mathematics department who can do the heavy lifting for you. It's not hard conceptually, just rather tedious... and a full explanation would be extremely boring for me to write and for you to read....

[tkingdoll]

The paradox is that I might not accomplish what I said I would and therefore "lose" the Challenge, but still accomplish something that any reasonably intelligent person -- whether a scientist, a student of the paranormal or a layperson -- would deem to be so improbable that it absolutely demands further investigation. That is what I suspected when I started this thread and your answers, and others', confirm it.

That's not a paradox.

A paradox would be if you won the challenge, but the very fact of you winning it proved that you don't have the ability you claim, and you therefore fail it.

And I can't think of any circumstances under which that would happen.

[tkingdoll]

No, it isn't.

Hypothetical, adj. Assumed by hypothesis, supposed.

Hypothesis, n. A proposition assumed as a premise in an argument.

Coming from someone who doesn't know what a paradox is, that's rather rich.

For a hypothesis, it's extremely specific, hence my question (you might want to take note that it WAS a question, and not an assumption).

[mslxl]

That's not a paradox.

A paradox would be if you won the challenge, but the very fact of you winning it proved that you don't have the ability you claim, and you therefore fail it.

And I can't think of any circumstances under which that would happen.

Paradox, n. 3. Any person, thing or situation exhibiting an apparently contradictory nature. (Webster's New Universal Unabridged Dictionary)

There are paradoxes in the real world, not only in logic. For example, one could say that the existence of mass hunger is a paradox, in view of the available technology and resources.

In the case we are discusssing, the paradox would be in my accomplishment, which would be "officially" not paranormal in the views of JREF and strongly suspected of being paranormal in the views of most other reliable observers. OK?

[mslxl]

Happy to help.

Thank you very much. I knew I shouldn't have skipped that stats class.:)

[tkingdoll]

Paradox, n. 3. Any person, thing or situation exhibiting an apparently contradictory nature. (Webster's New Universal Unabridged Dictionary)

There are paradoxes in the real world, not only in logic. For example, one could say that the existence of mass hunger is a paradox, in view of the available technology and resources.

In the case we are discusssing, the paradox would be in my accomplishment, which would be "officially" not paranormal in the views of JREF and strongly suspected of being paranormal in the views of most other reliable observers. OK?

It's still not a paradox, because those two things are not contradictory. The JREF's definition of paranormal is not different to that of the layperson, it's just that they need to set a parameter for testing purposes.

[mslxl]

It's still not a paradox, because those two things are not contradictory. The JREF's definition of paranormal is not different to that of the layperson, it's just that they need to set a parameter for testing purposes.

Right, they are only "apparently contradictory" as per the definition.

[Rasmus]

Right, they are only "apparently contradictory" as per the definition.

Not even that.

The JREF challenge is a bet. A bet to see if the applicant can show a specific paranormal ability under carefully pre-defined conditions.

Losing the bet doesn't have anything to do with other paranormal abilities of the applicant. It doesn't even mean that he could not do what he claimed he could do; it just means that he failed to do it then.

If I bet you I can run a marathon in under 2hrs, and then actually finish it it only 02:03:15 then what? It's still an outstanding performance, yet I lost my bet.

Rasmus.

[ChaosEngineer]

At what point would you say, if you were a scientist (or at what point do you say, if you actually are a scientist): "This should be investigated further. It could have happened by chance, but that probability is low enough to warrant further study."?

There's an even more basic question: Why did the scientist run the original experiment? This is really an economics question. He looked at these factors:
- The cost of running the experiment, including the cost of his time.
- The estimated odds of getting interesting results.
- The value of those interesting results...both the financial value, and the value of his satisfaction in adding those results to the store of human knowledge.

Obviously some of these numbers are completely subjective, and others are hard to estimate. But he'll have a vague hunch as to which experiments he wants to persue.

So: Suppose the coin-flipping experiment had never been done. The cost is very low, the probability of success is unknown, and the value of a successful result is pretty high.

After 10 flips with no obvious patterns, the scientist knows that the probability of success is on the low side. But he might just be having a run of bad luck, or he might just have a subtle influence that's only visible after a large number of throws. So he might go on for 100 flips, or 1000 flips. Eventually he'll get to the point where the probability of success seems so low that it's a waste of time for him to continue. Obviously different scientists will reach the breaking point at different times, so there's no real "optimal number of flips".

Of course, at this point in history the coin flip experiment has been done so many times that serious scientists won't waste their time on it.

The Randi Challenge is basically about the sort of experiments that serious scientists won't bother with...either because the experiments have been done countless times with negative results, or because the claimed results aren't consistent with well-tested laws of nature.

[LordoftheLeftHand]

The paradox is that I might not accomplish what I said I would and therefore "lose" the Challenge, but still accomplish something that any reasonably intelligent person -- whether a scientist, a student of the paranormal or a layperson -- would deem to be so improbable that it absolutely demands further investigation. That is what I suspected when I started this thread and your answers, and others', confirm it.

Yes, it’s possible. If you say you can predict the outcome of 100 coin flips with 100% success and you only get 99% correct you would lose. And getting 99% correct out of 100 would likely draw a lot of attention from scientists.

If you want to call this a "paradox" I suppose I won't argue with you. But what ever you call it, ultimately it would be your own fault for setting your bar too high.

Everyone has (or at least should have) a standard of evidence you must overcome to get them to believe something. The JREF's bar is pretty high, but then again they are the ones with the money on the line. The JREF certainly does not want to pay out the million on a fluke, they want to be sure you actually have the ability claimed. You negotiate the "height of your bar" with the JREF during the application process. If you claim you can predict the outcome of a coin flip 70% of the time, I bet they would accept your claim, although you will likely be required to do this over a very large number of flips (maybe several hundred). If you claim you can perform this with 100% accuracy, the number of flips would likely be greatly reduced (maybe 10-20).
LLH

[mslxl]

There's an even more basic question: Why did the scientist run the original experiment? This is really an economics question. He looked at these factors:
- The cost of running the experiment, including the cost of his time.
- The estimated odds of getting interesting results.
- The value of those interesting results...both the financial value, and the value of his satisfaction in adding those results to the store of human knowledge.

Obviously some of these numbers are completely subjective, and others are hard to estimate. But he'll have a vague hunch as to which experiments he wants to persue.

So: Suppose the coin-flipping experiment had never been done. The cost is very low, the probability of success is unknown, and the value of a successful result is pretty high.

After 10 flips with no obvious patterns, the scientist knows that the probability of success is on the low side. But he might just be having a run of bad luck, or he might just have a subtle influence that's only visible after a large number of throws. So he might go on for 100 flips, or 1000 flips. Eventually he'll get to the point where the probability of success seems so low that it's a waste of time for him to continue. Obviously different scientists will reach the breaking point at different times, so there's no real "optimal number of flips".

Of course, at this point in history the coin flip experiment has been done so many times that serious scientists won't waste their time on it.

The Randi Challenge is basically about the sort of experiments that serious scientists won't bother with...either because the experiments have been done countless times with negative results, or because the claimed results aren't consistent with well-tested laws of nature.


Let me ask you a slightly different question.

Say you are an inquisitive person without any particular credentials, but with common sense and a sharp mind. You are interested in the topic of predicting future events. Your common sense tells you it's impossible, but your inquisitive nature keeps you curious about it.

Say you are looking at performances by people who claim to be able to accomplish that coin trick. What would be the MINIMUM performance (the LEAST amazing one) that would make you feel compelled to say: "I HAVE TO find out what is going on with this person"?

mslxl

[LordoftheLeftHand]

Say you are looking at performances by people who claim to be able to accomplish that coin trick. What would be the MINIMUM performance (the LEAST amazing one) that would make you feel compelled to say: "I HAVE TO find out what is going on with this person"?

It is hard to say, because of the huge number of variables.

10 out of 10 would peak my interest, 20 out of 20 would really grab my attention. This is assuming the test was preformed once under proper circumstances.

LLH

[Rasmus]

Your common sense tells you it's impossible, but your inquisitive nature keeps you curious about it.

Just for being difficult, I dispute that that is even possible. If I *really* think it's impossible then that's that. There just wouldn't be anything to be curious <i>about</i>.

Say you are looking at performances by people who claim to be able to accomplish that coin trick. What would be the MINIMUM performance (the LEAST amazing one) that would make you feel compelled to say: "I HAVE TO find out what is going on with this person"?

Maybe 10 or 20.

Mind you, it would take a lot more than just a few repititions before I would think: "I am sure that person is not ********ting me and pulling some kind of magic trick on me; I am clearly witnessing something paranormal and have to find out what is going on with this person."

And it is very unlikely that something would occure that would make me go as far as to let something like that happen. (And if it did, I would just point that person to the JREF, since they are a lot more experienced than I am in dealing with that sort of thing.)

Rasmus.

[digithead]

Stanford statistician Persi Diaconis has shown that coin flips are not as random as they appear...

Do a google search on him and coin flips and you'll get an NPR story detailing his results...

So being able to call heads or tails isn't much of a trick or psychic phenomena...

[mslxl]

Stanford statistician Persi Diaconis has shown that coin flips are not as random as they appear...

Do a google search on him and coin flips and you'll get an NPR story detailing his results...

So being able to call heads or tails isn't much of a trick or psychic phenomena...

Fascinating... but he also says at the end of his paper that for practical purposes it is still random enough.

[digithead]

Fascinating... but he also says at the end of his paper that for practical purposes it is still random enough.

Sure, when you're choosing who gets to kickoff or receive in a football game, not when you're trying to prove that you can guess what side will come up during some coin tossing sequence. Then the issue of nonrandomness is paramount, especially when you're claiming that you can guess which side will come up and someone has shown scientifically that coin flipping is nonrandom...

Of course, if the person guessing was not involved in the flipping nor could they see the flipping, then it becomes more random. But then that's no different than someone trying to pass a true-false test by guessing. If you run enough trials, someone's bound to get it right. I'd only believe it if someone could repeatedly do it on every trial and it can be shown that that person had no prior knowledge of the test...

[hellaeon]

mate think of your testing this way

To get 17+/20 once or twice maybe is very lucky
to do so without fail, 50 times in a row is not lucky - thats something that warrants further study.

[Pup]

A real-life example is discussed here: http://206.225.95.123/forumlive/showthread.php?t=45357 (The girl with X-ray vision)

To "win" this particular test, the agreement was that the applicant had to get five out of seven correct. She got four out of seven.

No doubt she failed the test, but was it a good enough performance to warrent any further investigation of her supposedly paranormal abilities? Fifty-three pages of discussion ensue.

[drkitten]

Say you are looking at performances by people who claim to be able to accomplish that coin trick. What would be the MINIMUM performance (the LEAST amazing one) that would make you feel compelled to say: "I HAVE TO find out what is going on with this person"?


Under the circumstances as you describe it, there is no minimum number.

As you stated, it's a coin trick. Specifically, it's, quite literally, elementary sleight-of-hand to be able to "predict" coin flips. I'd be no more impressed by someone able to repeat this trick than I would be by someone being able to repeat the "tell me what card I just drew from a marked deck."

I'd need some sort of proof against cheating before I even began to entertain the question.

[petre]

Some other possibly-useful links:

http://forums.randi.org/showthread.php?t=49003
Light discussion about what level of success with a roulette wheel would pass "lucky" and become "paranormal"

http://forums.randi.org/showthread.php?t=29763&highlight=candle
Negotiating a protocol where the effect is slight (essentially where success was achieved 4 out of 6 times on average when 3 out of 6 would have been the result expected by chance).

(using the applicant names, someone else can look up the associated discussion threads too)

[mslxl]

Under the circumstances as you describe it, there is no minimum number.

As you stated, it's a coin trick. Specifically, it's, quite literally, elementary sleight-of-hand to be able to "predict" coin flips. I'd be no more impressed by someone able to repeat this trick than I would be by someone being able to repeat the "tell me what card I just drew from a marked deck."

I'd need some sort of proof against cheating before I even began to entertain the question.

Of course, I was implying a protocol in which the physical tossing is entirely out of the control of the performer.

[drkitten]

Of course, I was implying a protocol in which the physical tossing is entirely out of the control of the performer.

You mean, like a protocol where Teller throws the coin and Penn predicts it? I'm not entirely sure that I would be convinced....

But assuming that real controls are in place.... I would start to take something seriously when two independent groups managed to replicate the results at a standard (say, 5%) level of confidence.

[mslxl]

You mean, like a protocol where Teller throws the coin and Penn predicts it? I'm not entirely sure that I would be convinced....

But assuming that real controls are in place.... I would start to take something seriously when two independent groups managed to replicate the results at a standard (say, 5%) level of confidence.

Pardon my ignorance, but what's a 'level of confidence'?

[Strider1974]

Stanford statistician Persi Diaconis has shown that coin flips are not as random as they appear...

Do a google search on him and coin flips and you'll get an NPR story detailing his results...

So being able to call heads or tails isn't much of a trick or psychic phenomena...

Here is a link to the paper
http://www-stat.stanford.edu/~susan/papers/headswithJ.pdf

[drkitten]

Pardon my ignorance, but what's a 'level of confidence'?

It's a term of art -- basically, it's the alpha cutoff.

[ChristineR]

I've known several people who could control coin flips. Two used sleight of hand to reverse the coin while revealing it, one used his own special coin flipping technique.

[Admiral]

While this is an interesting discussion, does anyone else think it belongs in the Science, Medicine and Technology forum, rather than the Million Dollar Challenge one?

[William Smith]

Mslxl, please permit the following inquiry: Do you intend to apply for the JREF Challenge? With which claim?

[ChristineR]

Not all questions in this section deal with a specific challenge application. This topic is a philosophical discussion of the statistics used in the challenges and how they compare to statistics used in other circumstances (scientific research, actually believing that the claiment has ESP).

I think it's in the right place.

[mslxl]

Mslxl, please permit the following inquiry: Do you intend to apply for the JREF Challenge? With which claim?

I may. I do not wish to discuss the claim in this venue at this time.

[drkitten]

I've known several people who could control coin flips. Two used sleight of hand to reverse the coin while revealing it, one used his own special coin flipping technique.

I've also known someone who would palm a double-headed or double-tailed coin as appropriate.

[mslxl]

For anyone out there who knows combinatory analysis...

I know the number of possible outcomes for flipping a coin n times is two to the nth power. So for instance if I throw a coin ten times there are 1024 possible outcomes and my chances of guessing the whole sequence correctly are 1/1024.

What are my chances of guessing n out of p tosses correctly? For example, what are the chances of getting 8 (or better) out of 10 correct, or 15 (or better) out of 20 correct? Can someone give me the formula? I suspect the formula might require to calculate separately the number for 15 right, then 16 right, then 17 right, etc. then adding all those figures for the numerator and using the total number of possibilities as denominator. I just can't remember how to do it. It's been a while since high school.

Thank you.

[ChristineR]

Already did this for someone else. :)

http://forums.randi.org/showthread.php?postid=1416255#post1416255

If you want 15 of 20 or whatever you should be able to work out the pattern from this. It does get kind of ugly if you have to do the calculations by hand. Someone has probably written an online calculator somewhere.

[mslxl]

Already did this for someone else. :)

http://forums.randi.org/showthread.php?postid=1416255#post1416255

If you want 15 of 20 or whatever you should be able to work out the pattern from this. It does get kind of ugly if you have to do the calculations by hand. Someone has probably written an online calculator somewhere.

Thank you, ChrisitineR, but I had already done that myself by hand, for a larger number actually, and you're right, it does get kind of ugly. But what I'm asking for is the general FORMULA for n correct guesses out of p throws.

[Beth]

For anyone out there who knows combinatory analysis...

I know the number of possible outcomes for flipping a coin n times is two to the nth power. So for instance if I throw a coin ten times there are 1024 possible outcomes and my chances of guessing the whole sequence correctly are 1/1024.

What are my chances of guessing n out of p tosses correctly? For example, what are the chances of getting 8 (or better) out of 10 correct, or 15 (or better) out of 20 correct? Can someone give me the formula? I suspect the formula might require to calculate separately the number for 15 right, then 16 right, then 17 right, etc. then adding all those figures for the numerator and using the total number of possibilities as denominator. I just can't remember how to do it. It's been a while since high school.

Thank you.

If you have access to EXCEL or a similar program, it contains a function called BINOMDIST. This function will allow you to easily compute the exact probability by simply typing in the correct forumla.

For example, to compute the prob of getting 8 out 10 coin flips correctly type: 1 - BINOMDIST(7, 10, .5, 1)

Here, BINOMDIST is actually computing the probability of getting 7 or fewer out of 10 correct with a probability of .5 on each guess. You subtract from one to get the probability of getting 8 or more correctly.

eta: For an exact computational formula, look up binomial distribution on MathWorld. The computations get difficult quickly as your sample size goes up. I really recommend using EXCEL or some other software that already has the function programmed in.

[ChristineR]

http://mathworld.wolfram.com/BinomialDistribution.html

The formula you want is (3). But it won't make your life any easier if you're using a hand calculator, it just says what you already know in less space.

[William Smith]

I may. I do not wish to discuss the claim in this venue at this time.

You may. We will remain patient.

Under which circumstances might you apply?

[mslxl]

If you have access to EXCEL or a similar program, it contains a function called BINOMDIST. This function will allow you to easily compute the exact probability by simply typing in the correct forumla.

For example, to compute the prob of getting 8 out 10 coin flips correctly type: 1 - BINOMDIST(7, 10, .5, 1)

Here, BINOMDIST is actually computing the probability of getting 7 or fewer out of 10 correct with a probability of .5 on each guess. You subtract from one to get the probability of getting 8 or more correctly.

eta: For an exact computational formula, look up binomial distribution on MathWorld. The computations get difficult quickly as your sample size goes up. I really recommend using EXCEL or some other software that already has the function programmed in.

Thank you very much. And thank you for the tip on asking the reverse. I am perfectly happy letting the machine do the work.

[petre]

Might I add (using the above EXCEL formula) I deduce that predicing 138 correctly out of 230 coin flips is about a 1/1000 chance (and shows the somewhat-minimal bias of 6/10).

And predicting 333 out of 555 would give a 1/1,000,000 chance with that same 6/10 bias.

[William Smith]

Dear mslxl, I sincerely hope your meltdown in the thread http://forums.randi.org/showthread.php?t=55003 has no effect on your forthcoming Challenge Application.

Your calculations probably have finished by now.

Would you consider "discussing the claim in this venue at this time"?

[DevilsAdvocate]

What you're saying, then, is that if I go to the final test and predict 19 out of 20 coin flips accurately -- or, for that matter, predict 9 out of 10 accurately at the prelim -- I won't make any money... but anybody who is interested in the advancement of science or the expansion of knowledge should grab hold of me and not let go. An interesting paradox.It's not a paradox. An extraordinary demonstration would indicate that there MIGHT be something paranormal going on and would be worth looking into. The JREF Challenge is to show that there is in fact something paranormal--not just that there MIGHT be.

Although the 1/1000 and 1/1000000 rule has been tossed about, to my knowledge JREF has never accepted it as any standard. No standard is set because it is up to the applicants to describe what they can do. As far as I have been able to discover, in the case of claims that could be based on chance, JREF generally accepts terms that would be somewhere around 2-3 standard deviations from the norm. In the case of CARINA LANDIN, Swedish Friend of the DEAD, the applicant claimed to identify diaries as either male or female, much like flipping a coin. A protocol was accepted for the applicant to correctly identify 16 out of 20. If I recall correctly, Randi accepted even easier standards for some earlier dowsing tests.

You must have enough samples for the data to be relevant. In general, exceeding one standard deviation would be odd. Exceeding two standard deviations would certainly raise some eyebrows. Exceeding three standard deviations would certainly start hedging on proof, and would probably certainly qualify for the challenge.

[mslxl]

An extraordinary demonstration would indicate that there MIGHT be something paranormal going on and would be worth looking into. The JREF Challenge is to show that there is in fact something paranormal--not just that there MIGHT be.

Then it's impossible to ever PROVE that there is something paranormal going on, because all you'll ever have is a specific performance. Even if you perform something that has a one in a googolichtigrillion chances of happening, it would be an extraordinarily extraordinary demonstration but it still could have been luck. All you'll ever have is a MIGHT BE.

The truth about the JREF Challenge, as far as I understand it, is that it may or may not use commonly accepted criteria of scientific proof. It is entirely negotiable between the JREF and the candidate. That's what I gathered from the precisions I got right here. Please correct me if I am wrong.

[William Smith]

Then it's impossible to ever PROVE that there is something paranormal going on, because all you'll ever have is a specific performance. Even if you perform something that has a one in a googolichtigrillion chances of happening, it would be an extraordinarily extraordinary demonstration but it still could have been luck. All you'll ever have is a MIGHT BE.

The truth about the JREF Challenge, as far as I understand it, is that it may or may not use commonly accepted criteria of scientific proof. It is entirely negotiable between the JREF and the candidate. That's what I gathered from the precisions I got right here. Please correct me if I am wrong.

A far as I understand it, in short, it goes like this:

1. John Doe applies with a paranormal claim. (What "is" paranormal? See the Challenge FAQ: http://www.randi.org/research/faq.html#2.2 )
1b. In his claim, John Doe describes what he does. He describes what would constitute a successful proof of his claim - and what would count as failure.
2. Claim gets accepted.
3. Negotiations begin.
4. A preliminiary test takes place.

To determine the Applicant's claimed abilities in the prelim, the JREF uses "commonly accepted criteria of scientific proof" like the double-blind test.

I guess I understand your concern: Yes, someone may pass a prelim with no paranormal abilities just by beatings the odds.
Yes, a lot depends on the protocol negotiations.
For example, in telepathy ("transmitting" card symbols), or predicting the correct number of x amounts of spins of a roulette wheel or the good old lotto number prediction.

So far, as has been mentioned in this thread, the odds in a prelim are about 1:10,000.
For the Final Test - which of course has not happened once until today, because not one applicant passed the prelim - yet - about 1:1,000,000.
Quite unlikely, but beatable.



Perhaps a rigorous observation - using "commonly accepted criteria of scientific proof" - prevents one from stating, that there "IS" something paranormal once and for all. Especially in games of chance.
What gets undoubtely proven: At this time, at this location, this person did thing x as described in the protocol.
Same with a failure.

This does not sound very romantic, does it? No pixies, no aliens, no gods, no magic powers.
But this view helps me to define fiction and fact. It helps me define "reality". It keeps me sane.



However, I will become convinced of a paranormal activity, when someone goes Neo on me: Stopping bullets, jumping 30 ft. in the air, flying, etc.
Because there's a good chance this doesn't happen by "beating the odds". ;)

[Flange Desire]

Of course, if the person guessing was not involved in the flipping nor could they see the flipping, then it becomes more random.
Seems that this may be a flipping obvious key consideration.

But even if your stats for one trial show just a small diversion from the expected - all you need to do is demonstrate that same diversion consistantly over multiple trials. And then win the $US 1 million.

Of course, JREF is unlikely to allow such a demonstration to span too long a period of time - that might allow too much opportunity for cheating and stuff.

The point being - either the effect exists or it doesn't, and if it does then it must be measurable above the ambient noise.
If you can't differentiate if from the ambient noise, then you've just got noise.

[Pup]

Then it's impossible to ever PROVE that there is something paranormal going on, because all you'll ever have is a specific performance. Even if you perform something that has a one in a googolichtigrillion chances of happening, it would be an extraordinarily extraordinary demonstration but it still could have been luck. All you'll ever have is a MIGHT BE.

But that's pretty much the way it is with any observation of raw data, until theories are developed about WHY something is happening, and are used to predict the outcome of other experiments, which then behave as predicted, or not, causing new theories to be suggested, and so forth, until understanding is gained about the mechanism behind what's happening.

At that point, one can then say, "One in 1,000,000 people are born with a high level of a certain neurotransmitter that produces electical impulses strong enough to... which allows them to... under these conditions....." Of course whatever used to be considered paranormal--starting fires by staring, influencing coin flips, reading minds, whatever--will no longer then be considered paranormal, but the prize will have long been won and paid out.

[William Smith]

...
It is entirely negotiable between the JREF and the candidate. That's what I gathered from the precisions I got right here. Please correct me if I am wrong.

The application thread of Mr. Keeran, http://forums.randi.org/showthread.php?t=41120 might help you, too.

In my opinion, Mr. Keeran's point of view has a lot in common with yours.
Especially the part in which both of you claim that an ability which might not win the Challenge still may have a scientific interest. Unfortunately, Mr. Keeran's clever rhetorics made this thread rather long and exhausting to read.

If you make it through - fire up the coffee machine - you see a good example of how negotiations of an applicant like Mr. Keeran and the JREF may go. Should you seriously want to apply, I consider this particular thread a must-read. (I'm not JAK'in ya.)

And if you have the juice, check out the thread about Mr. Keeran in the Million Dollar Challenge Forum: http://forums.randi.org/showthread.php?t=41126
This thread will get you an insight on Mr. Keeran's agenda and his tactics. Of course, Mr. Keeran managed to stretch the discussion, the truth and the patience of the thread participants here also. Both threads paint a nice picture and let you see why - SPOILER ALERT - his file eventually got closed.

[LotusMegami]

A coin flip is not a matter of random chance. It just
appears to be, because humans cannot see all of the variables.

The ability to control a coin flip is not paranormal. Machines can do it - so a human who practiced long enough might be able to make a lot of money, assuming people would bet on something like that.

[nathan]

A coin flip is not a matter of random chance. It just
appears to be, because humans cannot see all of the variables.

Machines can do it.

Is there a predictable coin tossing machine?

[petre]

Is there a predictable coin tossing machine?

I think the idea is that a computer could predict the outcome.

A high speed camera and a computer can predict the outcome of a roulette wheel. Not perfectly mind you, but more than enough to give you the edge. It could do the same for a coin toss, assuming you were "calling it in the air".

[rjh01]

Actually I doubt that. The slightest error would mean a completely different result.

[LotusMegami]

No, I'm saying that you could build a machine that can control the flip of a coin.
There are also people who can control a flip. It's a parlor trick.

[petre]

Actually I doubt that. The slightest error would mean a completely different result.

I'm not familiar with general rotation speeds of a coin. I was thinking that with a relatively slow rotation, and a soft catch (palm of the hand) at a relatively small deviation of elevation (around waist level) it could be done (especially if the coin-flipper were trying to help). I was figguring maybe a success rate of 7/10.

Certainly, I'd expect dropping the coin onto a hard surface and letting it bounce to a stop would drive it to the realm of upredictable chaos.

[Dave_46]

A high speed camera and a computer can predict the outcome of a roulette wheel.

You don't need the camera. Just an eight bit computer built into clothing - see The Newtonian Casino by Thomas A Bass.

Dave

[nathan]

No, I'm saying that you could build a machine that can control the flip of a coin.
that's not what you actually wrote, which is why I called you on it.
Machines can do it

thanks for clarifying that you mispelt 'in theory a machine could do it, but I do not know of one that exists' :)

[William Smith]

Guys, the OP mslxl no longer participates. He pouts in his chamber because he got shellacked in his other thread http://forums.randi.org/showthread.php?t=55003

It seems this thread has also jumped the shark. Rust in peace.