June 2nd. commentary -the astrology protocol-

[Curnir]

Here is my sugestion for a protocol for the astrologer.
Might need a bit of tweaking and elaboration (such as "birthdates should be collected by person X, signifing the Husbands as H1, H2 H3 etc") but it's a start.

1. Take 10 randomly selected married couples.
2. Do the charts for the husbands, based on date of birth/time of birth (maybe timezone if requested... or should one convert the time to GMT first? hmm?) No further info should be given.
3. Make sure that the horoscope does not contain any reference to birthdates, starsigns etc (most wives tend to know their husbands birthday...)
4. Make 10 copies of each husbands horoscope.
5. Give each wife the horoscope of all the husbands, ask them to read through the 10 horoscopes and pick out the one belonging to her husband.

[Mojo]

Ha! You beat me to it!

[Kimpatsu]

I got there first, folks (thank the time difference), but I e-mailed the Amazing One directly.
I also explained what his Japanese correspondent meant.

[Mojo]

What did he mean?

[juryjone]

Well, that's essentially what I wrote, but I kept it at five couples. My reasoning on that had to do with the length of the horoscopes - 8 pages. Reading through five would mean 40 pages, ten = 80 pages. That's a lot to read through when you're being stopped for a few minutes in a public place. Most people would need more than an hour to read that much.

I also addressed the fact that the wives are grading the horoscopes rather than the husbands:


He's expecting the wives to know their husbands better than the husbands know themselves - playing into a stereotype, aren't we? "Obviously" women are more sensitive to the kinds of "impressions" that are given in the normal horoscope.

[Pixel42]

Looks like we're all thinking along the same lines.

I sent this email to JREF at 1 p.m. UK time:

The problem with the test protocol proposed by Mr Perron is, of course, that it fails to take into accout the well known and well understood phenomenon of subjective validation.

A simple alternative protocol which does not have this drawback, and which would probably be acceptable to Mr Perron, would be to use his proposed test of letting a subject's wife assess the accuracy of her husband's horoscope, but tweak it as follows:

1. Enlist 5 married couples to take part in the preliminary test

2. Give Mr Perron the dates/times/places of birth of the five men, identified only as subject1 - subject5, and get him to generate their horoscopes.

3. Xerox 5 copies of each of the five horoscopes and give each wife a copy of all five horoscopes. Ask them to read them all carefully, and pick out the one they think best describes their husband.

One in five of the wives should correctly identifying their husbands' horoscopes by pure chance, so I would suggest at least 3 would have to do so for this preliminary test to be judged a success. If it is, then the test would be repeated with, say, ten sets of five couples. You'd need a better mathematician than me to calculate what success rate would be statistically significant, but I would imagine anything in excess of 40% would be enough to win Mr Perron the million.

[Spektator]

I think it should be a little more interesting. In addition to the five horoscopes identified only by number, I think there should be five completely bogus horoscopes more or less randomly generated ("The placement of Mars shows that you sometimes have trouble controlling your temper, and sometimes you say things that hurt others before you realize it. However, your Saturn shows that you always regret causing pain to those around you, and fortunately you are able to pull excuses for your behavior from Uranus.") It would be interesting to see if the wives chose any of the bogus horoscopes.

[Jackalgirl]

What did he mean?

I think he meant that he has telepathy and other powers. "Terepace" is how someone might phonetically spell out "telepathy" if they were having trouble with the "l" and the "th". I think he just didn't spell out "powers". But that's my take.

I sent in a protocol too, which is almost identical to the one here. I just specified a little more detail: first, I specified that all of the husbands should be of roughly the same age (and that their lives should cover the same general historical periods) and that there should be two helpers or facilitators, whom I called "person A" and "person B".

1) person A collects the birthdate/time/location info from 10 husbands. That person randomly assigns the numbers 1-10 to the data and makes a list of what number corresponds to what data. The person then makes sure that the data is clear of any names and forwards it to person B
2) person B presents the data to the applicant, who generates the horoscope. The horoscope file will be printed out with the applicable number set up in the header (with at least three blank spaces of 12pt type) after it. The letters A-J will be assigned to each horoscope and the letters will be at the beginning of the horoscope data itself (that is, not in the header). Person B will receive the horoscopes, make a list of what letter corresponds to what number, then cut off the tops of the files (removing the numbers) before forwarding the horoscopes to person A.
3) Person A shows the wives the horoscopes and asks them to pick out the one best corresponding to their husbands (I should probably have stipulated that each wife do this alone, in a separate room, and not be allowed to "mingle" with the other wives afterwards, which I didn't put in the protocol I sent to Mr. Randi).
4) Persons A and B compare lists.

I stipulated that a) the horoscopes could not contain any "cluing" information (info referring to the date or location of the horoscopee), and that the inclusion of such data would be agreed to by all parties to invalidate the test, regardless of when that information was discovered to be present. I also stipulated that 8 out of 10 correct would constitute a success, but as I am really hopeless when it comes to statistics, I have only guessed that this would be (WAY) out of the realm of chance...

-- Kat

[Larry Coon]

(First post -- hi everyone!)

The correct protocol for testing such a claim should be obvious to anybody who has been exposed to double-blind testing in any way, and especially to those who have been readers of Swift for any length of time.

My response (e-mailed to Randi) was pretty much the same as the rest:

The most blatant flaw is that the claimant's method does not control for the Forer effect. The wives (judges of the accuracy of the claims) could judge vague and generalized statements as accurately describing their husbands. Second, the experimental group is not randomized enough -- it selects only for married men, in the company of their wives, who are at a shopping mall together. Third, the claimant is vague about what will constitute a "hit" (for the entire horoscope, not just whether individual statements within the horoscope apply to the subject). Is it just a yes/no to a final "Does this accurately describe your husband?" Or is there some number/percentage of individual statements within the horoscope that need to be deemed accurate in order for the entire horoscope to be considered a hit? Fourth, he uses irrelevant statements as justification for his method -- even if it is true that wives are not shy about being very critical of their spouses, this is orthogonal to being able to assess whether statements about their husbands are accurate (and pray tell, what if he got a wife who WAS shy about being critical of her spouse? That'd ruin the entire experiment!). Fifth, he makes references to probability with (as far as I can tell) absolutely no idea of how to actually determine the proper size of an experimental group, the number of hits required to be "beyond chance" (or even what is expected by chance alone), or any clue about all those esoteric concepts like confidence. If the wife assesses the horoscope with a simple yes/no, the sample size is five, and 60% is needed to win (his criteria), this result will be quite easy to achieve by chance alone. Sixth, and perhaps most importantly, he states that his analyses will be eight pages each. After struggling through a couple paragraphs of his writing I was ready to gouge my own eyes out -- I believe that finding even one subject capable of reading eight entire pages will be quite difficult.

A better protocol would be something like the following:

1) The claimant prepares horoscopes for everyone in the experimental group.

2) The horoscopes are "santized" if necessary -- names, birthdates and other language which could identify the subject are redacted. The claimant would be given the opportunity to verify that the "sanitized" horoscope was not altered in its "analysis." (An even better idea would be for the claimant to write his horoscopes without any names, birthdates, etc. Fat chance.)

3) The horoscopes are assigned a number by a person who has no further involvement with the testing process. Only this person knows which horoscope belongs to which subject.

4) For each subject: a) Randomize the horoscopes; b) Allow the subject to read each horoscope; c) Ask the subject to choose which horoscope is his own. (If he wants to do something like have a wife judge for her husband I suppose that'd be okay, although it introduces an extra variable which could later be used as an excuse by the claimant.) Note that since this experiment is double-blind, the person administering this step of the experiment has no knowledge of which horoscope is which.

The follow-up here is obvious -- if the subject correctly chose his own horoscope, it's a hit -- no subjective judging required for this step. The algorithms for choosing an appropriate experimental group size and the required number of hits to be sufficiently beyond chance are well established.

The only flaw in my method is that it requires the subject to read not only all eight pages of their own horoscope, but every horoscope in the experiment. Please do no conduct this experiment in a tall building in which the windows can be opened, for obvious reasons.

Randi responded that the claim uses married men, with the wives doing the deciding, and that he can't change the claim. That doesn't change my protocol -- although it makes the "reading" slightly less "cold" since he knows in advance that the subject is a married man who would accompany his wife shopping, and maybe even have some clue as to the subject's socio-economic background. Still, I responded to Randi with the following:

Hard to tell for sure from his babble what exactly constituted his claim, and what was simply supporting information. However, it looked to me that his "claim" was simply, "I WILL [PRODUCE HOROSCOPES CONTAINING THE PLANETS, THEIR RELATIVE POSITIONING AND INTERPRETATIONS OF THESE POSITIONS AND WHAT THEY INDICATE ABOUT A PERSON THEIR PERSONALITY, CHARACTERISTICS, AND NATURE] WITH A DEGREE OF ACCURACY THAT WILL BE OBSERABLY BEYOND MATHEMATICAL PROBABILITY." The stuff about malls, husbands & wives was in his second paragraph, which began, "MY PREFERED WAY TEST MY HOROSCOPES." In other words, all of that was about how his claim could be tested, and not part of the claim itself.

Admittedly, parsing a statement like that to determine exactly what he intended to be part of his claim is one of the great difficulties in your line of work (and perhaps the greatest).

[tom m.]

interesting idea... hmmmmm