Presentiment to replace the Ganzfeld?

[Paul C. Anagnostopoulos]

A conversation in another thread leads me to ask whether we are seeing presentiment experiments replacing the Ganzfeld as the current parapsychology paradigm of choice. Parapsychology has a history of hot paradigms turning cold, only to be replaced by the next hot one. Do you think we're seeing that again?

~~ Paul

[Cuddles]

Probably. It makes sense from a PR point of view. Lots of people believe in presentiment and all that nonsense about knowing who is about to phone you. Most people have no idea what a Ganzfeld is and wouldn't know one if jumped up and down on their head while wearing a pink tutu and holding a sign saying "I'm a Ganzfeld". If you're trying to promote woo, what's going to be more effective, talking about some experiments with a funny name that no-one can understand, or talking about something that lots of people think happens to them?

[Dancing David]

Especially if they can add computers to it the way they did in the auto-Ganzfeld. If they use computers it must be science.

Presentiment, lets see there are about five people who call my house, and they all have patterns of calling at certain times on certain days. Gosh it would be useful for avoiding those charity call that aren't on the 'no-call' list.

[Monza]

...Parapsychology has a history of hot paradigms turning cold, only to be replaced by the next hot one...

Isn't this because when others replicate the experiments, they do not get results showing a paranormal effect? I believe many researches have now performed the Ganzfeld experiments showing nothing more than chance. So now the paradigm shifts to something else in the elusive chase for woo.

I believe you are correct that presentiment may becuase the de rigeur choice of the paranormal researchers.

[Rodney]

I believe many researches have now performed the Ganzfeld experiments showing nothing more than chance.
Evidence?

[Jeff Corey]

None.

[Monza]

Evidence?


I stand corrected. It was a meta-analysis (by Milton and Wiseman) that showed results equal to chance. To be fair, some have criticized this analysis. Though it also seems that none of the actual experiments are without serious criticisms of their protocols.

WRT the opening post, is there indeed a decline in Ganzfeld experiments among psi researchers?

[Ersby]

I was the one who mentioned to Paul that the ganzfeld is slowly falling out of favour. There are still experiments being run, but there's been a bit of a drop since the late 90's/early 2000's. I noticed this dip myself and then when talking to a parapyschologist, he mentioned the same thing.

[fls]

I stand corrected. It was a meta-analysis (by Milton and Wiseman) that showed results equal to chance. To be fair, some have criticized this analysis. Though it also seems that none of the actual experiments are without serious criticisms of their protocols.

Regardless of what happens if you (plural) add all the experiments together, the list of experiments collected in the meta-analysis shows that you can't replicate the effect by performing the experiment. That is, any time you perform the experiment, you will probably get a result that is no different from chance. This makes it difficult to form and test any hypotheses about psi using the ganzfeld, since you won't be able to interpret negative results. That may be at least part of the reason for the purported drop off in its use.

Linda

[fls]

On another note, I am unfamiliar with the use of z/N^1/2 as a measure of effect size. Can someone point me to some information on this measure?

Linda

[Ersby]

z/N^1/2 is the Stouffer z, I believe. It's pretty standard in parapsychology. Although if you do a google search on "Stouffer z", only parapsychological papers come up.

[fls]

z/N^1/2 is the Stouffer z, I believe. It's pretty standard in parapsychology. Although if you do a google search on "Stouffer z", only parapsychological papers come up.

I think I found the clue - Stouffer's z is related to Rosenthal's r.

Thanks.

Linda

[blutoski]

Dr. Novella did expouse some thoughts about this in several podcasts. At the risk of putting words in his mouth, I'll list the steps in the cycle:


Somebody decides to test a paranormal urban legend, does it with crappy protocols, gets an amazing result, publishes it
Other paranormal researchers see the result, want to be part of this 'genuine' subfield of paranormalism, drop whatever they're doing, and start attempting to replicate, but with better protocols
Weak positive results are published; negative results are buried. As more and more results are published, and protocols tightened, the 'effect' appears weaker, until it basically becomes a statistical argument.
The assumption is that a breakthrough study is just around the corner, so research continues. Some researchers drop out, or start thinking about other paranormal urban legends.
See step one.


I think the cycle has one other aspect: nothing is ever disproven, so a future generation of researchers can decide to revisit something that fell out of fashion 30 years prior.

[Paul C. Anagnostopoulos]

I was the one who mentioned to Paul that the ganzfeld is slowly falling out of favour. There are still experiments being run, but there's been a bit of a drop since the late 90's/early 2000's. I noticed this dip myself and then when talking to a parapyschologist, he mentioned the same thing.
Indeed it was you, Ersby. We've seen this before, so I figured the cycle would have to continue.


That is, any time you perform the experiment, you will probably get a result that is no different from chance. This makes it difficult to form and test any hypotheses about psi using the ganzfeld, since you won't be able to interpret negative results. That may be at least part of the reason for the purported drop off in its use.
And part of the reason for all the previous fallings out of favor. Imagine how much fun it would be to propose hypotheses based on a theory, rather than just doing more data collection.

Maybe presentiment will be sufficiently replicable that we'll get to see a theory. However, now's your chance to predict the paradigm that will come into favor after presentiment. Any takers?

~~ Paul

[Paul C. Anagnostopoulos]

Apropos to nothing much, I found this letter interesting:

http://jeksite.org/psi/metaletter.htm

~~ Paul

[Rodney]

I stand corrected. It was a meta-analysis (by Milton and Wiseman) that showed results equal to chance.

The Milton/Wiseman meta-analysis was erroneous. Their article, "Does Psi Exist? Lack of Replication of an Anomalous Process of Information Transfer" has a footnote that reads: "For each study that used the usual method of measuring its outcome by comparing the number of hits obtained to the number expected by chance, a z score was derived from an exact binomial test. Some studies used different outcome measures involving ranking or rating the target and decoys, and in such cases the probability associated with the test statistic used (t test, etc.) provided the z score. When a study reported more than one main outcome measure, the mean z score represented the study's outcome."

Milton/Wiseman don't tell the reader that only four of the 30 studies used "different outcome measures." Those four studies included only 128 trials, whereas the other 26 studies included 1,070 trials. In those 1,070 trials, there were 301 hits, which is a hit rate of 28.1% vs. the expected hit rate of 25%. Using the binomial distribution, which is clearly the proper statistical test, the outcome is statistically significant (p =1.2%). If hit rates are imputed to the other 128 trials, there are only 29 hits (hit rate of 22.7%), but even when those results are added to the others, there are 330 hits in 1,198 trials, which is a hit rate of 27.5%. Again, using the binomial distribution, which still appears to be the proper test, the outcome is statistically significant (p =2.4%).

[fls]

The Milton/Wiseman meta-analysis was erroneous. Their article, "Does Psi Exist? Lack of Replication of an Anomalous Process of Information Transfer" has a footnote that reads: "For each study that used the usual method of measuring its outcome by comparing the number of hits obtained to the number expected by chance, a z score was derived from an exact binomial test. Some studies used different outcome measures involving ranking or rating the target and decoys, and in such cases the probability associated with the test statistic used (t test, etc.) provided the z score. When a study reported more than one main outcome measure, the mean z score represented the study's outcome."

Milton/Wiseman don't tell the reader that only four of the 30 studies used "different outcome measures." Those four studies included only 128 trials, whereas the other 26 studies included 1,070 trials. In those 1,070 trials, there were 301 hits, which is a hit rate of 28.1% vs. the expected hit rate of 25%. Using the binomial distribution, which is clearly the proper statistical test, the outcome is statistically significant (p =1.2%). If hit rates are imputed to the other 128 trials, there are only 29 hits (hit rate of 22.7%), but even when those results are added to the others, there are 330 hits in 1,198 trials, which is a hit rate of 27.5%. Again, using the binomial distribution, which still appears to be the proper test, the outcome is statistically significant (p =2.4%).

Yes, Rodney. We got it already. You know better than Bem and Milton and all those other parapsychologists as well as anyone here with training and experience in meta-analysis.

Linda

[Ersby]

Milton/Wiseman don't tell the reader that only four of the 30 studies used "different outcome measures."

Which four studies are those?

[...] but even when those results are added to the others, there are 330 hits in 1,198 trials, which is a hit rate of 27.5%. Again, using the binomial distribution, which still appears to be the proper test, the outcome is statistically significant (p =2.4%).

Are you treating all thirty experiments as one big experiment?

[Rodney]

Yes, Rodney. We got it already. You know better than Bem and Milton and all those other parapsychologists as well as anyone here with training and experience in meta-analysis.

Linda
I e-mailed Bem a few weeks ago and he agrees with my analysis. He says that the reason he, Palmer, and Broughton replicated Wiseman's and Milton's erroneous statistical methodology in their article "Updating the Ganzfeld Database: A Victim of Its Own Success?" is that, when 10 additional Ganzfeld studies were included, the results were statistically significant and so they did not find a need to debate the methodology. In my opinion, that was an unwise decision. However, if you disagree with my analysis, please explain why.

[Rodney]

Which four studies are those?
Four of the five that have a "(d)" after the "Hit Rate %" on this page: http://findarticles.com/p/articles/mi_m2320/is_3_65/ai_83262436/pg_8
The exception is the Parker & Westerlund (1998) (Serial Study), which -- as indicated by the * -- was not included by Milton and Wiseman, but was included by Bem, Palmer, and Broughton.

Are you treating all thirty experiments as one big experiment?
In effect, yes.

[Ersby]

I can't see any difference in the z scores and effect sizes reported for those four experiments when comparing Milton & Wiseman's paper and Bem et al's paper.

[Rodney]

I can't see any difference in the z scores and effect sizes reported for those four experiments when comparing Milton & Wiseman's paper and Bem et al's paper.
True, because Bem et al simply accepted Milton's and Wiseman's z scores and effect sizes, even when they were clearly erroneous. In my opinion, what Bem et al should have done is noted the numerous errors in Milton's and Wiseman's paper, but they did not.

[Ersby]

Well, I'm off work all this week, so can't get to a spreadsheet to punch in a few numbers myself. But you said you imputed the hit rate from the non-hit rate experiments. How did you do that?

[JMA]

A conversation in another thread leads me to ask whether we are seeing presentiment experiments replacing the Ganzfeld as the current parapsychology paradigm of choice.

Do you have some exemple of papers about presentiment experiments?

[Rodney]

Well, I'm off work all this week, so can't get to a spreadsheet to punch in a few numbers myself. But you said you imputed the hit rate from the non-hit rate experiments. How did you do that?
I didn't do it, Bem et al did. They state in footnote (d) to the Table that I linked to (second previous post of mine): "Hit rate not reported. Estimated from z score."

[Gord_in_Toronto]

I was going to ask what had happened with the precognition registry that was set up in the UK after the Aberfan coal tip disaster. But I see from: http://mainportals.com/precog.shtml the answer is "not much". Is this the :Randi Effect" wherein if some rational person attempts to measure some woo-woo the woo-woo goes away? :boggled:

[Dancing David]

Evidence?

Your call Rodney, making vauge statements about probability is not evidence.

[Dancing David]

The Milton/Wiseman meta-analysis was erroneous. Their article, "Does Psi Exist? Lack of Replication of an Anomalous Process of Information Transfer" has a footnote that reads: "For each study that used the usual method of measuring its outcome by comparing the number of hits obtained to the number expected by chance, a z score was derived from an exact binomial test. Some studies used different outcome measures involving ranking or rating the target and decoys, and in such cases the probability associated with the test statistic used (t test, etc.) provided the z score. When a study reported more than one main outcome measure, the mean z score represented the study's outcome."

Milton/Wiseman don't tell the reader that only four of the 30 studies used "different outcome measures." Those four studies included only 128 trials, whereas the other 26 studies included 1,070 trials. In those 1,070 trials, there were 301 hits, which is a hit rate of 28.1% vs. the expected hit rate of 25%. Using the binomial distribution, which is clearly the proper statistical test, the outcome is statistically significant (p =1.2%). If hit rates are imputed to the other 128 trials, there are only 29 hits (hit rate of 22.7%), but even when those results are added to the others, there are 330 hits in 1,198 trials, which is a hit rate of 27.5%. Again, using the binomial distribution, which still appears to be the proper test, the outcome is statistically significant (p =2.4%).


Hiya Rodney, what is the standard deviation for positive hits on the ganzfeld studies? Seriously that is a meaningful question.


28% is NOT statistically significant, it could just be random variation. What is the standard deviation rate?

What is the random match rate of a target picture to a random word string generated by the receiver?

At what point does the deviation from chance rise to any correlation above 65%, especially since you use a modified Z score.

Thanks.

:)

[fls]

I e-mailed Bem a few weeks ago and he agrees with my analysis. He says that the reason he, Palmer, and Broughton replicated Wiseman's and Milton's erroneous statistical methodology in their article "Updating the Ganzfeld Database: A Victim of Its Own Success?" is that, when 10 additional Ganzfeld studies were included, the results were statistically significant and so they did not find a need to debate the methodology. In my opinion, that was an unwise decision. However, if you disagree with my analysis, please explain why.

Why is Wiseman's and Milton's statistical methodology erroneous, other than you didn't like the results?

Any time you do a meta-analysis, you make decisions about how best to combine the data. If there are arguably several ways to do so, what makes one of those ways "erroneous"? Especially the one that follows the rules better the other?

Linda

[Rodney]

Hiya Rodney, what is the standard deviation for positive hits on the ganzfeld studies? Seriously that is a meaningful question.


28% is NOT statistically significant, it could just be random variation. What is the standard deviation rate?

What is the random match rate of a target picture to a random word string generated by the receiver?

At what point does the deviation from chance rise to any correlation above 65%, especially since you use a modified Z score.

Thanks.

:)
Your questions all relate to the probability of the results being obtained by chance. For example, with an expected hit rate of 25%, 7 hits in 25 trials could easily be due to chance. On the other hand, 301 hits in 1,070 trials is unlikely to be due to chance; specifically, there is only a 1.2% probability. There is no need to complicate the analysis.

[blutoski]

Your questions all relate to the probability of the results being obtained by chance. For example, with an expected hit rate of 25%, 7 hits in 25 trials could easily be due to chance. On the other hand, 301 hits in 1,070 trials is unlikely to be due to chance; specifically, there is only a 1.2% probability. There is no need to complicate the analysis.

You're saying that the p value is not <= .01? ie: that the results are not statically significant for a physical law?

[Rodney]

Why is Wiseman's and Milton's statistical methodology erroneous, other than you didn't like the results?

Any time you do a meta-analysis, you make decisions about how best to combine the data. If there are arguably several ways to do so, what makes one of those ways "erroneous"? Especially the one that follows the rules better the other?

Linda
I would accept Wiseman's and Milton's statistical methodology if I thought it were proper. But why use a cumbersome Stouffer z methodology when it is far more straightforward to use the binomial distribution? I just don't think you can get around the statistically significant combined hit rate by treating the 30 studies as if they are disparate, when 26 are similar. (By the way, it's not even clear what the results were in the other four studies. Wiseman and Milton show a negative z score for three of the four, but as best as I can tell, all of their negative z scores are erroneous.)

[Rodney]

You're saying that the p value is not <= .01? ie: that the results are not statically significant for a physical law?
Using the binomial distribution: Not less than .01 for the 30 studies analyzed by Milton and Wiseman, but less than .01 for the 40 studies analyzed by Bem et al, even if the most positive study is excluded.

[blutoski]

Using the binomial distribution: Not less than .01 for the 30 studies analyzed by Milton and Wiseman, but less than .01 for the 40 studies analyzed by Bem et al, even if the most positive study is excluded.

Why did they choose different studies for their analyses?

[Ersby]

I took one of the experiments that didn't use hit rate (Stanford & Frank - the only one I have the entire paper for) and put the p number into an online p-number-to-z-score converter, and it did come up with a different z score than in the two meta-analyses: -0.78 instead of -1.24.

[Ersby]

(By the way, it's not even clear what the results were in the other four studies. Wiseman and Milton show a negative z score for three of the four, but as best as I can tell, all of their negative z scores are erroneous.)

Really? I can see only one that looks odd. A closer look at the experiments involved may help.

The four in question, as listed in Bem et al, are: Stanford & Frank (1991), Kanthamani et al series 5b (1988), and Kanthamani et al series 6a and 6b (1992). (Kanthamani series 5a should also be in the list and was mentioned in the Milton/Wiseman paper because of the different way the z-score had to be calculated.)

Stanford & Frank reported "Overall (N = 58), the mean ESP score is -0.144, t(57) = -1.25, p = .216, two-tailed."

Kanthamani 5b reported a sizeable negative score in the ganzfeld condition, p=0.0658 for the ratings and p=0.0225 for the rankings (both in a negative direction, both two-tailed). 10 trials, judged 4 times for each scoring method.

Series 6a is given a negative z score, but a hit rate at chance, which is the one that strikes me as odd. I don't have the paper for this.

And series 6b is positive.

[Ersby]

Why did they choose different studies for their analyses?

A few months after the Milton/Wiseman meta-analysis was completed, a couple of successful ganzfeld experiments were presented at a parapsychological convention, and people quickly noticed if you included these in the meta-analysis, you'd get a significant result, and so Bem et al did the meta-analysis again with the new experiments included.

[Dancing David]

Your questions all relate to the probability of the results being obtained by chance. For example, with an expected hit rate of 25%, 7 hits in 25 trials could easily be due to chance. On the other hand, 301 hits in 1,070 trials is unlikely to be due to chance; specifically, there is only a 1.2% probability. There is no need to complicate the analysis.

No it isn't. That is the same as saying if you flip a coin a thousand time it will have a 500/500 distribution. But I assure you that if you try it, you will find that there are trials that vary widely from the distribution.

That is what you don't understand about probability. It is random and it is only over very large sets that you get close to the distribution of theory.

I suggest that you get something called 'percentile dice'(two ten sided dice, one for the tens place, one for the ones place). There is a one percent chance that you will roll a double zero, and then have trial of a hundred rolls. You would expect that in each trial that you will roll the double zero just once right? Then say that you record how many times you roll the double zero in each trial of one hundred rolls. What will you find? That there are trials where you roll 00 more than one time, you will have trials where you do not roll a 00 at all.

Now what is the distribution of these events where you roll more than a single 00, a single 00 and no 00?

I can state to you that there is a probability distribution based upon the number of trials. And that is where the term standard deviation sort of comes from.

Now here is the tricky part!

When you set up an experiment you have to have your independent and you dependant variables. When you want to claim that you have found a causal effect then you have to demonstrate that in your number of trial the effect was significant to have risen above the standard deviation for that number of trials. (I am mixing terms here but the idea is accurate.) So say that you have five trials, and that the positive score for that number of trials is 2 but the standard deviation for that number of trials is 3. Then you can not say that you have found a causal effect that rises above that which would be expected from random probability.

And here is the thing about the Ganzfeld meta-analysis. You are mixing methodologies, to say that you have a statistically significant event means that the studies in the analysis have to have the same protocols and methodologies for it to be meaningful.

If the Ganzfeld was true you would have to look at the standard deviation for the event of picture matching and determine that the positive matches rose above the level of probability for the trial at hand. Then you compare the number of trials that significantly were above the standard deviation. There is a way to do this and determine if it is significant.

Then there is the whole issue of certain pictures having a higher likelihood of being a match compared to others. Since that is a confounding effect that is not controlled for the results could all be skewed by the pictures chosen for the trials in the first place.

This is not adding to a simple issue Rodney, these are the simple issues that should be addressed.

[fls]

I would accept Wiseman's and Milton's statistical methodology if I thought it were proper. But why use a cumbersome Stouffer z methodology when it is far more straightforward to use the binomial distribution?

Because it is more important to consider reliability and validity than it is to make it easy.

I just don't think you can get around the statistically significant combined hit rate by treating the 30 studies as if they are disparate, when 26 are similar. (By the way, it's not even clear what the results were in the other four studies. Wiseman and Milton show a negative z score for three of the four, but as best as I can tell, all of their negative z scores are erroneous.)

As I mentioned in the last thread on this, I found considerable discrepancy (although, not large) in the reported results. I have not gone through the referenced studies to clarify this, only because it simply doesn't matter. Meta-analysis, by itself, is insufficient to serve as proof of anything, let alone psi. And it is clear that the ganzfeld experiments are unable to return a replicable effect to use for exploratory research, regardless of what happens when you add them all together. So while we can argue about the details until we're blue in the face, this study cannot answer the questions we are asking.

Linda

[Rodney]

Really? I can see only one that looks odd. A closer look at the experiments involved may help.
When I said that "as best as I can tell, all of their negative z scores are erroneous", I was referring to not just the four studies where an alternate methodology was used, but to all 30 studies included by Milton and Wiseman. For example, they give the z score for the "Broughton & Alexander Clairvoyance Series" as -0.64. However, according to Bem et al, that study showed a 22% hit rate for 50 trials. If that was the case, there were 11 hits, which -- according to my calculation -- results in a z score of -0.30.

[Rodney]

No it isn't. That is the same as saying if you flip a coin a thousand time it will have a 500/500 distribution. But I assure you that if you try it, you will find that there are trials that vary widely from the distribution.

That is what you don't understand about probability. It is random and it is only over very large sets that you get close to the distribution of theory.

I suggest that you get something called 'percentile dice'(two ten sided dice, one for the tens place, one for the ones place). There is a one percent chance that you will roll a double zero, and then have trial of a hundred rolls. You would expect that in each trial that you will roll the double zero just once right? Then say that you record how many times you roll the double zero in each trial of one hundred rolls. What will you find? That there are trials where you roll 00 more than one time, you will have trials where you do not roll a 00 at all.

Now what is the distribution of these events where you roll more than a single 00, a single 00 and no 00?

I can state to you that there is a probability distribution based upon the number of trials. And that is where the term standard deviation sort of comes from.

Now here is the tricky part!

When you set up an experiment you have to have your independent and you dependant variables. When you want to claim that you have found a causal effect then you have to demonstrate that in your number of trial the effect was significant to have risen above the standard deviation for that number of trials. (I am mixing terms here but the idea is accurate.) So say that you have five trials, and that the positive score for that number of trials is 2 but the standard deviation for that number of trials is 3. Then you can not say that you have found a causal effect that rises above that which would be expected from random probability.

And here is the thing about the Ganzfeld meta-analysis. You are mixing methodologies, to say that you have a statistically significant event means that the studies in the analysis have to have the same protocols and methodologies for it to be meaningful.

If the Ganzfeld was true you would have to look at the standard deviation for the event of picture matching and determine that the positive matches rose above the level of probability for the trial at hand. Then you compare the number of trials that significantly were above the standard deviation. There is a way to do this and determine if it is significant.

Then there is the whole issue of certain pictures having a higher likelihood of being a match compared to others. Since that is a confounding effect that is not controlled for the results could all be skewed by the pictures chosen for the trials in the first place.

This is not adding to a simple issue Rodney, these are the simple issues that should be addressed.
The most recent Ganzfeld studies have corrected past methodological problems. And, why is it improper to aggregate hits and trials and use the binomial distribution if the same protocol is used in each study?

[fls]

The most recent Ganzfeld studies have corrected past methodological problems. And, why is it improper to aggregate hits and trials and use the binomial distribution if the same protocol is used in each study?

It turns out that pooling participants leads to invalid conclusions in meta-analysis, so it is avoided. See 'Simpson's paradox'.

Linda

[latent aaaack]

So where are all these new presentiment parapsychology experiments? The only new one I've heard about is Sheldrake's telephone telepathy experiment.

[blutoski]

So where are all these new presentiment parapsychology experiments? The only new one I've heard about is Sheldrake's telephone telepathy experiment.

It's not clear if that's even a presentiment experiment, since the protocol does not clarify whether the target selected their guess before or after the caller's intention was established. It could be a telepathy experiment.

Same with the dog experiments: does the dog know 'in advance' that the owner is coming home, or does the dog know as soon as the owner has decided to come home, and is thinking about it?

We've actually considered re-doing the phone experiment here in BC, with college students as participants. We propose a much cleaner protocol.

[blutoski]

It turns out that pooling participants leads to invalid conclusions in meta-analysis, so it is avoided. See 'Simpson's paradox'.

Linda

To be honest, I think that meta-analysis is valid if the protocols are virtually identical and also very simple, and if we can be sure there's no file-drawering.

The Simpson's Paradox is not the end of the world, and in my opinion, when the results appear to contradict, it's usually the meta-analysis that's considered the best interpretation anyway.

I would reject any meta-analysis where the component trial runs used substantially different protocols, or even if they use the same protocols but are testing a batch of metrics instead of just the one in question.

I'm afraid that the big challenge I see in this type of meta-analysis is the file-drawer / positive result publication bias. In this example, waiting for two more trials shifted the aggregate results from null hypothesis to statistical significance, so you can see how even a tiny publication bias can impact meta-analysis results.

It's really hard to dismiss the proposal that runs that are negative in isolation are less likely to be brought forward for inclusion in the meta-analysis, and that this artificially inflates the aggregate result by biasing it more toward positive.

This is not a problem isolated to psi publications, and in drug trials, the solution is to pre-register all trials so negative results can't be hidden from the peer-review, literature review, and meta-analysis. It improves the profession's interpretation of the body of literature: a pattern of registering trials - but not publishing - is suspicious. Better transparency.

I don't see this rigid publication transparency in psi.

[fls]

To be honest, I think that meta-analysis is valid if the protocols are virtually identical and also very simple, and if we can be sure there's no file-drawering.

<shrug> The people with expertise in this area disagree with you and I find their reasons compelling.

The Simpson's Paradox is not the end of the world, and in my opinion, when the results appear to contradict, it's usually the meta-analysis that's considered the best interpretation anyway.

Can you give an example where this has happened?

Linda

[Paul C. Anagnostopoulos]

We've actually considered re-doing the phone experiment here in BC, with college students as participants. We propose a much cleaner protocol.
Don't forget to control for the clock desynchronization leak.

~~ Paul

[blutoski]

Don't forget to control for the clock desynchronization leak.

~~ Paul

You bet. We'd do it with both parties in the same building, but in different rooms, under immediate supervision of a mutually-agreed team. There is actually a mirror-window room available to us, which will allow the same team of observers to monitor both the sender and the receiver. Call intervals will be randomized, and the sender will have to call within a minute after being selected by the randomizer.

[blutoski]

<shrug> The people with expertise in this area disagree with you and I find their reasons compelling.

Like who?

I'm saying that you need certain circumstances for a meta-analysis to be worthwhile, and when they're met, a meta-analysis can be very useful. If the protocols mesh and the results don't need to be futzed out of the literature with tricks like regression analysis, we should take advantage of it.

I've never met anybody who believes otherwise, so I'm surprised by your statement.

Maybe I should add it to my Skeptical Urban Myths?

Just as a counterexample from my point of view, I was gratified to hear Dr. Novella explain the same thing several times in SGTTU: meta-analyses are powerful, but have limited application because it's rare to have a body of literature all done with the same protocol.




<Can you give an example where this has happened?

Linda

Here in Canada, we had an election where the loser won the minority (49%) of the ridings, but got the majority (74%) of the overall vote.

Baseball averages also come to mind. A batter can have 2 seasons with a .8 average and be the best in the league year after year, then have two bad seasons with a .2 average. If he only hit five times in the good years, but a thousand times in the bad years, the overall reality is that he's a .2 hitter even though he was best in the league two years running. It's an exaggerated example, but the informativeness of the meta-analysis has to do with compositional weighting.

[fls]

Like who?

Like the folks at the Cochrane Collaboration (http://http://www.cochrane-net.org/openlearning/HTML/mod12-2.htm), for example.

"Pooling participants (not a valid approach to meta-analysis).
This method effectively considers the participants in all the studies as if they were part of one big study. Suppose the studies are randomised controlled trials: we could look at everyone who received the experimental intervention by adding up the experimental group events and sample sizes and compare them with everyone who received the control intervention. This is a tempting way to 'pool results', but let's demonstrate how it can produce the wrong answer."

I'm saying that you need certain circumstances for a meta-analysis to be worthwhile, and when they're met, a meta-analysis can be very useful. If the protocols mesh and the results don't need to be futzed out of the literature with tricks like regression analysis, we should take advantage of it.

I've never met anybody who believes otherwise, so I'm surprised by your statement.

I didn't say meta-analysis isn't useful. I said performing meta-analysis by pooling participants can lead to invalid conclusions and so should be avoided. That is, pooling participants should be avoided, not meta-analysis. That is what Rodney and I were talking about, and is the snippet you chose to quote.

Maybe I should add it to my Skeptical Urban Myths?

Huh?

Here in Canada, we had an election where the loser won the minority (49%) of the ridings, but got the majority (74%) of the overall vote.

Isn't that an example of the opposite - the results from the individual ridings outweigh the pooled results?

Baseball averages also come to mind. A batter can have 2 seasons with a .8 average and be the best in the league year after year, then have two bad seasons with a .2 average. If he only hit five times in the good years, but a thousand times in the bad years, the overall reality is that he's a .2 hitter even though he was best in the league two years running. It's an exaggerated example, but the informativeness of the meta-analysis has to do with compositional weighting.

But in this case, the meta-analysis doesn't really contradict the individual studies, just some of them. And tiny studies would be given far less weight than huge studies, anyway. That is, considering the seasons individually, you would conclude he was a .2 hitter, even before you formally pooled the results.

Linda

[Rodney]

Like the folks at the Cochrane Collaboration (http://http://www.cochrane-net.org/openlearning/HTML/mod12-2.htm), for example.

"Pooling participants (not a valid approach to meta-analysis).
This method effectively considers the participants in all the studies as if they were part of one big study. Suppose the studies are randomised controlled trials: we could look at everyone who received the experimental intervention by adding up the experimental group events and sample sizes and compare them with everyone who received the control intervention. This is a tempting way to 'pool results', but let's demonstrate how it can produce the wrong answer."
Let's suppose a benefactor were to grant each of 20 parapsychology research groups funding for 100 ganzfeld trials, specifying the same protocol for each group. Will combining those results to obtain a total of X hits in 2,000 trials and analyzing the results using the binomial distribution produce the wrong answer? If so, why?

[fls]

Let's suppose a benefactor were to grant each of 20 parapsychology research groups funding for 100 ganzfeld trials, specifying the same protocol for each group. Will combining those results to obtain a total of X hits in 2,000 trials and analyzing the results using the binomial distribution produce the wrong answer? If so, why?

That would be closer to the large, appropriately powered study that should be done to test psi. I think that in order to get in enough trials (you'd need well over a thousand (I can't tell how much more because the tables in Cohen run out at that point)) you have to do it as a multi-center study anyway.

Linda

[Dancing David]

Let's suppose a benefactor were to grant each of 20 parapsychology research groups funding for 100 ganzfeld trials, specifying the same protocol for each group. Will combining those results to obtain a total of X hits in 2,000 trials and analyzing the results using the binomial distribution produce the wrong answer? If so, why?


That is not a meta analysis, that would be a large pool study. If the protocols were reviewed randomly by video tape that would help even more.

[Paul C. Anagnostopoulos]

You bet. We'd do it with both parties in the same building, but in different rooms, under immediate supervision of a mutually-agreed team. There is actually a mirror-window room available to us, which will allow the same team of observers to monitor both the sender and the receiver. Call intervals will be randomized, and the sender will have to call within a minute after being selected by the randomizer.
Don't you need multiple rooms, one for each caller? I don't think you want them to talk to one another, even in between calls.

~~ Paul

[Paul C. Anagnostopoulos]

That is not a meta analysis, that would be a large pool study. If the protocols were reviewed randomly by video tape that would help even more.
Thank you, yes, let's do that.

~~ Paul

[Ersby]

So where are all these new presentiment parapsychology experiments? The only new one I've heard about is Sheldrake's telephone telepathy experiment.

Presentiment is less to do with knowing who's on the phone and more to do with unconcious changes in the body just before a particular event, such as being shown an unpleasent photo or whatever. It's scores over ganzfeld, (a) it's easier to get a large amount of data, and (b) it kind of makes more sense to posit the idea that psi can detect an immediate risk in the immediate vicinity, rather than somehow know what a close friend/relative is looking at in some distant location.

Most of these I haven't read. I've had my fill of parapsychology with the ganzfeld.

Precognitive Habituation, two papers from Bem

dbem.ws/Precognitive%20Habituation.pdf
www.parapsych.org/papers/46.pdf

...One from Savva, Child and Smith

www.parapsych.org/papers/19.pdf

... And one from Gergö Hadlaczky
www.diva-portal.org/diva/getDocument?urn_nbn_se_su_diva-1017-1__fulltext.pdf

Radin has been pretty active in presentiment experiments. Here's a few...

Radin, D. I. (1997). Unconscious perception of future emotions: An experiment in presentiment. Journal of Scientific Exploration, 11(2), 163.180.
Radin, D. I. (2000, 7/19/00). Evidence for an anomalous anticipatory effect in the autonomic nervous system. Boundary Institute. Retrieved July 22, 2002, from the World Wide Web:
http://www.boundaryinstitute.org/articles/presentiment99.pdf
Radin, D. I. (2003). Electrodermal presentiments of future emotions. Paper presented at the Parapsychological Association 46th Annual Convention, Vancouver, Canada.
Electrodermal Presentiments of Future Emotions
www.scientificexploration.org/jse/articles/pdf/18.2_radin.pdf

and this page references a load of papers, too

http://publicparapsychology.blogspot.com/2007/11/brain-response-to-future-event.html

[Cuddles]

Do you have some exemple of papers about presentiment experiments?

Not yet, but we will last week.

[JMA]

In this new video:

http://www.youtube.com/watch?v=qw_O9Qiwqew

Dean Radin talks a lot about presentiment experiments.