Check my methodology - prayer study

[Thinktoomuch]

While I thank you for your concern, I ask that you not allow my theism to worry you.


No worry at all, as I already said to Linda. Anybody who commits "to behave with the utmost integrity" has my full support. And my money is after tax disposable income:rolleyes: .

[Cuddles]

I must respectfully disagree. Intelligence tests are well established as not varying under many conditions. Caffeine intake is probably the most notorious, but please reference http://www.springerlink.com/content/t25366v1554q33m0/ for an interesting summary of a recent article in the peer-reviewed _Cellular and Molecular Life Sciences (CMLS)_. Even given your claim, won't the A-B nature of the experimental design eliminate any influence? Let's say 10 out of 100 subjects cheat to improve their performance, by whatever means. Since they don't know in which trial they're receiving prayer support, they can't bias the result.

Ah, sorry, it seems I missed the bit about swapping the groups over. I agree that that should eliminate any bias between the two groups. However, I am still concerned about the sample size. Do you have a minimum number you would consider acceptable? With a very small sample, such as the 50 that Saizai proposes, if just a few people perform differently in the A and B rounds there could be a significant difference between the two groups. Sample size is of course one of the biggest concerns in health studies, but as far as I can see neither you nor Saizai have adequately addressed it. You say that you will visit churches handing out fliers, but what will you do if not enough people register? Or if too many drop out during the test?

Now, Cuddles, I really think you need to sit in a cozy armchair and ponder your statements above. You're much too smart to make this claim. We aren't interested in how it works. We are concerned that after eliminating other factors can we show with statistical accuracy that a paranormal force effected the outcome under controlled situations. If so then the claimant deserves our further consideration. I would, of course, defer to JREF for the next test if this one shows a positive result. I would not claim to have proven anything.

If you insist on maintaining your position, would you please provide a quote from the JREF FAQs or instructions that eliminates "medical studies" as "just not challenges"?

Unfortunately the FAQs are all gone at the moment, so I can't find an exact quote. I believe it says something along the lines of "Results will be immediately obvious to any observer". The point being that either you have an ability or not. If you can levitate, anyone watching can tell that either you are floating off the ground or not. If you say you can find water, then anyone can see that finding even just 19/20 hidden containers is something very special, regardless of how or why you claim you do it. Studying hundreds of people for a couple of years and then performing a detailed statistical analysis to find any small difference between two groups is just not the same. I don't know if there is anything in the challenge rules that forbids it, and in fact I suspect there is not, but it is certainly very much against the spirit of the JREF and its challengs and I very much doubt they will ever accept a test along these lines.

As I, and others, have said before, the JREF is not a research funding organistion. The challenge is there mainly as a PR tool to expose frauds and also to find evidence for paranormal abilities if any actually exist. If you want to conduct lengthy research into healing effects, this is not the place to do so. In fact, you mention that you have talked to a professor and may do the test regardless of what Saizai does. This is exactly what should be done. This sort of research belongs in universities and possibly hospitals, not in paranormal challenges.

On the issue of financial rewards, this simply cannot be acceptable. It is standard procedure to pay everyone involved in a test a small amount (in university this was normally a beer or some sweets :p), but you cannot reward people based on the outcome of the test. Paying people who give you the result you are looking for is a very clear conflict of interest. Regardless of whether you believe they couldn't influence the result, such a test could never be taken seriously and would certainly never be published.

Finally, I believe what Linda is saying is that neither you or Saizai have said how you will actually measure any positive effect, or any effect at all. In order for anyone to know if the results are valid, we have to know exactly what you plan on measuring, how you plan on measuring it and how you will analyse the results. The problem is not that the test is not valid, it is that we don't know what the test is, so we cannot tell if it is valid or not.

hope that we can disagree here and maintain our professional friendship. You have my respect, and I ask your indulgence to allow me to disagree so tersely.

With real gratitude,
Gulliver

No problem. Life would be so boring if everyone agreed all the time.

[fls]

fls,

I can't help but notice that you're using the past tense. I'll assume that you meant to use the past perfect (happening now and in the past.)

What happened in the past was the idea that it would be useful to involve myself. :) (make that a rueful smile)

You'd didn't ask me to reply regarding your concerns.

No. I felt obliged to state them since you picked up on my comment to Thinktoomuch. But I left it up to you to decide whether you were interested in looking at any of it. It is really the people you are working with (and yourself) that you need to satisfy, not me.

I'm not sure that we can repair our relationship enough to make a discourse here useful, but I'm willing to try. I've dealt with many experts over my tenure in the academia who seem to own only a sniper's rifle. I hope that you're willing to use another tool of a more cooperative nature. Please consider, for example, your original unexplained "is inadequate" sniping.

I'm sorry that I've alienated you. My comment was thoughtless. I did not realize that you were seriously considering implementing your protocol. I thought you were just treating this as an interesting exercise, since it seemed pretty obvious (to me) Saizai was not going to go along.

I haven't specified the outcome tests, so I believe that you attack a straw man. I suggest we can very much rely about the Boston short-term memory recall test that is well-documented and widely accepted even in a web-based environment. If you attack the Boston test in this protocol, I ask respectfully that you provide your reasons.

I thought you were looking at whether prayer influences health and that you were suggesting measuring blood pressure and mental acuity. I did not realize that you had decided to test the effect of prayer on normal short-term memory.

I don't understand this comment. Please tell me your reasoning that the carryover effect can't be blocked by blocking its variable in the ANOVA (or, if we chose, by a linear regression).

It can be blocked (if detected), which effectively negates the crossover part of the trial. And usually crossover trials do not have enough power (the reason for using the design in the first place) to adequately exclude the effect making the "if detected" part unreliable.

Here you're correct. We could only conclude that the results apply to churchgoers and that only if assume that prayer effects individuals near me the same as those far from me. I suggest that many accepted and useful studies ,including even the Harvard study referenced earlier in the thread (only bypass patients in several hospitals), have a more selective sample population.

I was thinking also of the generalizability of negative results.

I'm not sure how you are using the terms "selection", "sample" and "population", though.

I believe this is a conclusion that you've not adequately supported.

The statements that followed were meant to support that conclusion (that may not have been clear).

I'm sure that it cannot lay all questions about the effect of prayer to rest. But I'm sure you're aware that science progresses in small steps. This study if negative would add to the body of evidence. If positive at very small p value, we could only conclude the need for further review of the question.

We already have a lot of research on the effects of prayer. Those studies already effectively include the small step you are thinking of adding. If anyone is not yet convinced that prayer is ineffective, then the addition of a small study to the overall power is unlikely to be convincing. And as I mentioned to Saizai, a positive result (for the reasons given below) will serve to provide an out for those who wish to suggest there may be something to study (justifying further expenditure of time and money) without providing useful guidance as to what that might be.

You fail to provide any evidence or even reasoning to back up your claim, "false positive are far more likely than true positive". I kindly ask you to provide such reasoning or some evidence.

I brought up this issue in this post (http://forums.randi.org/showthread.php?postid=2542065#post2542065). This is the relevant paper (http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124) that explains this issue in greater detail.

I disagree, of course, with your comments about the relevance, as I've stated above. I don't believe that we need to concern ourselves whether improvement in short-term recall is "a good thing". The study can clearly be neutral on this.

These are the kinds of questions I would ask when assessing relevance. Why would change in short-term memory in someone without impairment be a relevant outcome (i.e. why would you hypothesize this is something that would be affected by prayer)? Is this test capable of registering improvement in the absence of impairment? Does it provide any ability to discriminate in a normal population?

Since you included the emoticon, I infer that you're not serious on this issue.

It hardly seemed fair to single you out on this issue. :)

I must respectfully disagree. It is not clear to me at all. I remain, for one, at saizai's pleasure in pursuing this study. For two, I have been quite clear, and not just in this thread, that I value constructive criticism from JREF Forum members.

May I also solicit from any Forum member their thoughts on the Gulliver protocol, especially as it relates for fls's concerns?

Gratefully,
Gulliver

Ah, I thought you were talking about the possibility of proceeding without him.

Linda

[Gulliver]

All,

I'm most grateful for your support here. I see three wonderfully insightful and kind emails in a row. I don't have the words to express how precious I find your feedback and encouragement. Please consider us all friends working to educate others, and each other.

For now I beg your indulgence. I've received enough feedback to generate many days of work. I liked how the family doctor responded with a long silence to Cuddles's and fls's comments and then stated she had some thinking to do. Please consider the Gulliver Proposal "up on blocks" while we see whether we can resolve the issues you've raised. I may need longer than some of you may consider ordinate. My cancer is confining to bed rest most days now, so please allow some extra time. I promise to report back on each item as soon as we have progress to report.

On a side (and somewhat selfish note), has anyone else noticed how quiet saizai has been of late?

Pondering and Planning,
Gulliver