Online testing of psychic - help needed.

[Pixel42]

I've been participating in a thread discussing psychic powers on another board (the general forum of a fan site, not a board specifically about the paranormal). A semi-professional psychic (tarot card reader) who has also been participating has sportingly agreed to a JREF-type test, to establish whether or not she can do successful readings over the internet. [She's quite open to the possibility that she's unconsciously picking up clues (i.e. using cold reading) to give her clients a cheap form of therapy, and there is nothing genuinely paranormal going on].

The test protocol we've come up with is:

1. Six volunteers will PM a trusted moderator with their first name, gender, age and location.

2. The mod will assign each an identifier (subject1, subject2 etc) and add two fictional subjects. He will then PM the list of identifiers and details to the psychic, who will do tarot card readings for each and post them, together with the identifier. She's agreed not to include anything which could be deduced from the minimal details she's been given on each subject, so the probability of the volunteers correctly identifying their own reading should be no better than chance.

3. Each volunteer will pick out the reading they think is theirs, and PM the mod with its identifier. The mod will post which subject picked which reading when he's received everyone's choice.

Where we need help is in calculating the probability of getting 1, 2, 3, 4, 5 and 6 correct choices out of 8 by chance (my maths is a little rusty), and deciding what the success criteria should be. Does anyone know what level of probability JREF require to be exceeded for an applicant to be deemed to have passed the preliminary test?

Any help would be appreciated.

I should make it clear that this is just a bunch of friends doing an interesting experiment in a spirit of genuine enquiry. A fair amount of trust is being placed in both the psychic and the moderator, so we understand that nothing will be conclusively proved, and our test isn't rigorous enough to be accepted by JREF as a real preliminary test. But a lot of interesting and thought-provoking discussion has already been generated, and we'd like to see it through.

[John Jackson]

As you're from the UK, why not try asking on the UK-Skeptics forum? - http://www.ukskeptics.com/forum/index.php

They do exactly the sort of testing that you're proposing to do informally.

[Loss Leader]

Your protocol sounds good for a preliminary, non-binding test.

There's still a lot of room for the individuals who have volunteered to communicate with the psychic and agree that she will put "code words" in their readings so that they can pick out which is their own.

There's also room for hot reading. After all, if I know I'm doing a reading for a 62 year-old female, I'll put, "You're nostalgic about family," or something. 24 year old male? "You sometimes can't control your libido." So, is the chance of picking one's reading really one in eight? Maybe we could say that such chance, controling for people being able to throw out several of the readings as definitely not theirs, might be one in three. That makes the chance of all six getting it right one in seventy-two thousand nine hundred. That's high, but it's not astronomically high.

You might think about having twenty or more volunteers sending the moderator their information. Then the moderator could chose six at random and not tell the volunteers which six have been chosen until after the psychic submits her readings. This should decrease the chances of cheating significantly as there's a less than one in three chance that the conspirator would actually be one of the chosen six to begin with. The amount of work needed to game even one right reading would triple and it just wouldn't be worth the effort.

Unless, of course, everybody is in on it.

[sinclairmcevoy]

Is there such a thing as a semi-professional psychic? You is or you aint. Anyone can read tarot cards.

[ChristineR]

Assuming the comparison is by chance alone, and all six subjects are shown all six readings.

There are 262,144 ways for the six to choose. The chart is number correct (0 through 6) and ways to get that result.

0 - 117,649
1 - 100,842
2 - 36,015
3 - 6,860
4 - 735
5 - 42
6 - 1

For the JREF prize, the odds are supposed to be about 1 in 1000. Five or more correct is 43 in 262,144 (1 in 6096) and four or more correct is 778 in 262,144 (1 in 336).

The general formula for p people looking at r readings and getting c correct is


{p! / [(p-c)! (c!)] } * (r-1)^(p-c)

That's what is called "p choose c" and is usually written with double high brackets with p on the top and c on the bottom, all multiplied by (r-1) raised to the power (p-c).

Which is as clear as mud, I guess.

Warning: I am not known for my lack of typos and other stupid errors. I'm known for my brilliant mind. ;)

[Pixel42]

Thanks for the responses, I think I now have all the information I need.

The psychic is doing the readings tomorrow and the mod will post the results by Tuesday. I'll let you know how we get on.

[ChristineR]

Okay, first boo-boo. I meant to say "all six subjects are shown all eight readings." Obviously it's much easier if there are only six readings to look at.

[JoeTheJuggler]

Let us know how it comes out, ok?

[Pixel42]

The psychic has now posted her readings. Unfortunately, despite being specifically requested not to include the gender/age/country information she was given in the readings, she has used 'she' or 'he' in four of the eight readings, and made it obvious that two of the subjects live in the US.

I got the moderator to PM me with the numbers of male/female/American volunteers, and have worked out that instead of all 6 subjects having 8 readings they can choose from, each subject will now (after eliminating those which can't possibly be theirs because they are of people who are a different sex or nationality) have a maximum of 6 and a minimum of 4. Averaging that out at 5 - a bit dodgy to just average it, but it's the best I can do - and feeding that new figure into the formula means that the chances of getting each number of correct choices by chance is now:

1 in 10 of getting 3 or more (down from 1 in 34)

1 in 59 of getting 4 or more (down from 1 in 337)

1 in 624 of getting 5 or more (down from 1 in 6096)

1 in 15624 of getting all 6 (down from 1 in 262144)

So using the JREF threshold of beating odds of 1 in 1000, the success criteria is now all six correct.

We've decided to continue the experiment with the new success criteria. We're now waiting for all the volunteers to read them and get back to the moderator with their choice, which might take a few days. I'll let you know the result.

[Loss Leader]

The psychic has now posted her readings. Unfortunately, despite being specifically requested not to include the gender/age/country information she was given in the readings, she has used 'she' or 'he' in four of the eight readings, and made it obvious that two of the subjects live in the US.

I said that there might be a problem like this. It can only be controled for with a larger sample size.

So, is the chance of picking one's reading really one in eight? Maybe we could say that such chance, controling for people being able to throw out several of the readings as definitely not theirs, might be one in three.

[Pixel42]

The results are in.

No-one picked the right reading.

[JoeTheJuggler]

The results are in.

No-one picked the right reading.

Any ad-hoc hypotheses from the psychics yet?

Have any of them tried the old "100% perfectly wrong is somehow significant" song and dance?

[Pixel42]

No. She just concluded that she should stick to face-to-face readings.

[Loss Leader]

The results are in.

No-one picked the right reading.


Great work. I know the "psychic" will rationalize the results but I'm sure that you got through to some fence-sitters. Congratulations.

[SRW]

Great work. I know the "psychic" will rationalize the results but I'm sure that you got through to some fence-sitters. Congratulations.

That is the most satisfying thing, getting someone to think and question the woo woo world. Good job hope you got at least one convert.

[EternalSceptic]

No. She just concluded that she should stick to face-to-face readings.

I tend to credit her for this, because of:

[She's quite open to the possibility that she's unconsciously picking up clues (i.e. using cold reading) to give her clients a cheap form of therapy, and there is nothing genuinely paranormal going on].

So she is at least realistic and was just seeking verification. Our life here would be much easier if a majority of "psychics" would act like this :)