Meta-analysis of JREF tests

[T'ai Chi]

What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

[Mercutio]

A. each test could be different, as the procedure depends on the claims of the claimant. So while some may be similar, I would not like to see the flexibility of the challenge subsumed to some desire for meta-analysis.

B. unless I am very much mistaken, thus far no claimant has passed even the preliminary test. Speaking as a stats teacher, I tell you there are times when no statistics are needed.

[EdipisReks]

that would be one incredibly easy and quick meta-analysis.

[T'ai Chi]

Originally posted by Mercutio
A. each test could be different, as the procedure depends on the claims of the claimant. So while some may be similar, I would not like to see the flexibility of the challenge subsumed to some desire for meta-analysis.

B. unless I am very much mistaken, thus far no claimant has passed even the preliminary test. Speaking as a stats teacher, I tell you there are times when no statistics are needed.

Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

[EdipisReks]

Originally posted by T'ai Chi


Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

why bother?

[Darat]

Originally posted by T'ai Chi
What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

Considering that you believe that science can only be done in a laboratory I suspect you have an agenda with this question rather than wanting a discussion, especially since you've said in the past such comments as

Originally posted by T'ai Chi here (http://www.randi.org/vbulletin/showthread.php?s=&postid=1870062716&highlight=challenge#post1870062716)

SRW, Randi is fine for the most part, it is his followers that are dogmatic.

Good results are obtained from science, and science is done in peer reviewed journals. Interesting entertainment, such as the challenge, is entertaining and thought provoking, but ultimately doesn't show much in the long run.



And

Originally posted by T'ai Chi here (http://www.randi.org/vbulletin/showthread.php?s=&postid=1870071836#post1870071836)


…snip…

Science is done in labs and written about in journals.

…snip…




However assuming that you are now of the opinion that the challenge is “science” and has the controls necessary for the results to be taken “seriously” I still can’t see how a meta-analysis could be done on the Challenge. Doesn’t a meta-analysis assume there is some “commonality” in the experiments and resulting data? How would you go about performing a meta-analysis on the results which are simple “fail” or “passes”?

[Paul C. Anagnostopoulos]

T'ai said:Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.
What statistics? The little Russian girl read the newspaper for awhile, then Randi put some duct tape around the mask. Suddenly she couldn't read. No statistics required.

Etc., etc., etc.

~~ Paul

[UnrepentantSinner]

This person really needs to work on his trolling skills.

:rolleyes:

[T'ai Chi]

Originally posted by EdipisReks

why bother?

Here's why.

Obviously no one has passed any tests. But let's look at all of their standardized scores.

It be interesting to see if

1. the majority of the standardized scores were positive, or

2. the majority of the standardized scores were negative, or

3. there are an about equal mix of positive and negative standardized scores,

to see if there is some overall effect, when averaging across all the tests.

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
T'ai said:
What statistics? The little Russian girl read the newspaper for awhile, then Randi put some duct tape around the mask. Suddenly she couldn't read. No statistics required.

Etc., etc., etc.

~~ Paul

Um, sure, in cases like that, we cannot get any statistics. In other cases, one can easily get statistics, such as in dowsing underground pipes, film cannisters, anything where someone claims to be able to know the contents of something, remote viewing, etc.

[Pyrrho]

As others have said, a lot of these tests don't involve statistics. Generating numbers for the purpose of a meta-analysis would simply be an exercise in assigning values where none existed, and the use of complicated formulae to find significance where none existed.

Given that all of these tests have ended in failure, I don't see any use in the meta-analysis of a series of zeroes.

[jj]

Originally posted by T'ai Chi


Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

There is no reason to assume that the statistics of such apples and oranges mean anything at all.

[jj]

Originally posted by UnrepentantSinner
This person really needs to work on his trolling skills.

:rolleyes:
Duno, he seems to be good at getting people to bite. :p

[T'ai Chi]

Originally posted by Pyrrho
As others have said, a lot of these tests don't involve statistics.


And as I've said, I realize that there are tests where no statistics are present.

But, obviously, what about the ones where there are statistics?

[/b]
Given that all of these tests have ended in failure, I don't see any use in the meta-analysis of a series of zeroes. [/B]

*sigh*, it wouldn't be an analysis of "a series of zeroes", but rather the standardized scores (of course from the tests where statistics were kept).

The point is to see if most of these standardized scores are positive, negative, or a good mix of both.

If there is nothing going on in terms of significance, we'd expect a good mix of both. Note that if that was the case, that wouldn't necessarily at all imply that something paranormal was going on (as some might want to hint at).

[Paul C. Anagnostopoulos]

If you perform a meta-analysis of a series of similar experiments (if there are enough similar ones among the lot), none of which were significant, you'd most likely reach the conclusion that the series was overall insignificant. At least, that's the conclusion I hope you'd reach.

The primary purpose of meta-analyses is to determine whether additional experiments are worth performing. But in the JREF case, additional experiments will be performed regardless. So what's the point?

~~ Paul

[billydkid]

Originally posted by UnrepentantSinner
This person really needs to work on his trolling skills.

:rolleyes:

I think the term "troll" should be reserved for those who actually deserve. I am not a big fan of Tai Chi's reasoning skills, but I do not remotely think he is a troll. There are genuine trolls in here, but using the word loosely robs it of its significance.

[Paul C. Anagnostopoulos]

I agree with Billy. I've been called a troll for being too persistent in pursuing a topic, the exact opposite of what a troll really is.

~~ Paul

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
If you perform a meta-analysis of a series of similar experiments (if there are enough similar ones among the lot), none of which were significant, you'd most likely reach the conclusion that the series was overall insignificant. At least, that's the conclusion I hope you'd reach.


I agree Paul, and I would expect that too. However, we could talk about how it might turn out all day. :) It would be interesting to see if our expectations are correct I'd say.

The majority of the standardized scores could be positive or negative for example, and the overall conclusion could be significant.


The primary purpose of meta-analyses is to determine whether additional experiments are worth performing. But in the JREF case, additional experiments will be performed regardless. So what's the point?


I personally wouldn't say that that is the primary purpose of a meta-analysis. According to how I learned it, the primary purpose of a meta-analysis is simply to accumulate experimental and correlational results across independent studies.

And to reply to another post without making another post of my own; I don't pay any attention to the 'troll-sayers'. I find that a lot of the time these people simply cannot make any valid points of their own, so they resort to a label game. I don't even read these peoples' posts after a certain grace period.

[Paul C. Anagnostopoulos]

T'ai said:I personally wouldn't say that that is the primary purpose of a meta-analysis. According to how I learned it, the primary purpose of a meta-analysis is simply to accumulate experimental and correlational results across independent studies.
In other scientific endeavors this might serve some purpose. But with psi, where the experiments are not investigations of underlying theories, the most a meta-analysis can do is encourage you to develop a theory and test it (which is not the mission of the JREF). The meta-analysis does not allow you to reach the conclusion that psi exists, because the experiments were not investigations of a theory of psi.

As I repeat ad nauseam, it is time for a theory of psi. Or give it up.

~~Paul

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
T'ai said:
In other scientific endeavors this might serve some purpose. But with psi, where the experiments are not investigations of underlying theories, the most a meta-analysis can do is encourage you to develop a theory and test it (which is not the mission of the JREF). The meta-analysis does not allow you to reach the conclusion that psi exists, because the experiments were not investigations of a theory of psi.

As I repeat ad nauseam, it is time for a theory of psi. Or give it up.

~~Paul

So, you wouldn't be interested in the results of a meta-analysis then?

I think evidence determines theories, not the other way around.

[Paul C. Anagnostopoulos]

T'ai, I think evidence determines theories, too. But not "evidence" produced by meta-analyses of experiments that had no theories to begin with.

If people think there is enough evidence for psi or dowsing or whatever, then it's time they come up with some theories, devise experiments with hypotheses based on those theories, and go for it.

~~Paul

[tedly]

I think this quote is germane.

"Parapsychology is motivated by belief in search of data, rather than data in search of explanation, says James Alcock, in an article (PDF) in the Journal of Consciousness Studies ..."

It's from
SciTechDaily (http://www.scitechdaily.com/)
about halfway down the right side of the page. I'd supply the direct link but I'm trying to plug one of my favourite web pages.

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
T'ai, I think evidence determines theories, too. But not "evidence" produced by meta-analyses of experiments that had no theories to begin with.


I'm not trying to employ meta analysis in order to come up with a theory. I'm trying to use meta analysis in order to accumulate experimental and correlational results across independent studies.

[billydkid]

Originally posted by T'ai Chi


I'm not trying to employ meta analysis in order to come up with a theory. I'm trying to use meta analysis in order to accumulate experimental and correlational results across independent studies. [/B]

There is no data. Nitwits come claiming to be able to dowse or remote view or perform telekinesis and can't - how does that constitute data?

[T'ai Chi]

Theories don't determine evidence. It is the other way around. Although, there is obviously cooperation between evidence and theories.

If there is no or incomplete evidence to warrant a theory, that is fine with me. :) However, just because something doesn't have a or an acceptable theory, doesn't mean that one shouldn't examine the evidence. Perhaps examining the evidence could lead to a theory and extend our understand about the supposed phenomenon in many ways.

As far as I know, there in fact have been theories proposed: quantum mechanics, the holographic universe theory, teleological model, thermal fluctuation model, behavior model, various electromagnetic theories, decision augmentation theory, and probably several others.

I don't know much about any of these theories, other than that they exist, and people are, in fact, working on them. (I just did a Google search).

I believe this is different from things like 'no theories of psi exist' and 'no experiments are being done to investigate these theories'.

[T'ai Chi]

Originally posted by billydkid

There is no data. Nitwits come claiming to be able to dowse or remote view or perform telekinesis and can't - how does that constitute data?

There certainly is data from some of the studies.

Say a dowser attempts to find a film cannister with water in it. There are 2 cannisters, and only one has water in it.

He does 20 trials, and gets 7 hits.

If you standardize that score, call it Z, it will be negative (in this case).

If we have another dowser who, out of 30 trials with 2 cannisters, got 17, this Z will be positive.

If we do something similar for each dowser and calculate their Z-scores, we will have a bunch of Z-scores.

If there was absolutely nothing going on but chance, we'd expect some Z-scores to be negative and some to be positive, so that the combined Z-score would wash out and be about 0.

OR, when combined, the combined Z-score could be very negative, or the combined Z-score could be very positive, meaning that the studies, when examined as a group, showed preference against chance in some way.

[thaiboxerken]

Ok. Here is the meta-analysis.

100% of all claimants failed.

People don't have superpowers.

[thaiboxerken]

If there is no or incomplete evidence to warrant a theory, that is fine with me. :) However, just because something doesn't have a or an acceptable theory, doesn't mean that one shouldn't examine the evidence. Perhaps examining the evidence could lead to a theory and extend our understand about the supposed phenomenon in many ways.


I believe this is different from things like 'no theories of psi exist' and 'no experiments are being done to investigate these theories'.

There is no evidence of "psi". There are TONS of theories on how it might work though. Although, they are not really theories, but speculations based on personal beliefs and nothing more. So.. no evidence + no theory = no psi.

[Paul C. Anagnostopoulos]

T'ai said:OR, when combined, the combined Z-score could be very negative, or the combined Z-score could be very positive, meaning that the studies, when examined as a group, showed preference against chance in some way.
Yes, but what does that mean? You don't know, because you have no theory. It could be the same flaw in the all the experiments. It could be a bunch of different flaws. It could be psi. It could be a miracle from the lord. You don't know!

Anyway, if you like this sort of thing, there are meta-analyses up the yin-yang from "real scientists."

~~ Paul

[Jeff Corey]

Or, as pointed out by D.M.Stokes in the May/June Skeptical Inquirer, it could very well be the File Drawer effect.
Great article.

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
T'ai said:
Yes, but what does that mean? You don't know, because you have no theory. It could be the same flaw in the all the experiments. It could be a bunch of different flaws. It could be psi. It could be a miracle from the lord. You don't know!

Anyway, if you like this sort of thing, there are meta-analyses up the yin-yang from "real scientists."

~~ Paul

The 'theory' is that if the applicants do no better than chance as a group, as we expect, we'd expect the combined Z-score to be around 0 and insignificant.

Right?

[Paul C. Anagnostopoulos]

T'ai said:The 'theory' is that if the applicants do no better than chance as a group, as we expect, we'd expect the combined Z-score to be around 0 and insignificant.
That's a hypothesis about the results of the experiment itself, not about any underlying theory. A zeroish Z confirms the null hypothesis. A significant Z refutes the null hypothesis. In either case, it's a big so what because the hypothesis is not interesting.

As far as dowsing is concerned, there never was anything "unexplained" that needed experimental investigation. I wander around with a stick, dig a well, and find water. I wander around without a stick, dig a well, and find water.

~~ Paul

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos
T'ai said:
That's a hypothesis about the results of the experiment itself, not about any underlying theory. A zeroish Z confirms the null hypothesis. A significant Z refutes the null hypothesis. In either case, it's a big so what because the hypothesis is not interesting.


I disagree. It certainly is interesting to many people.

If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)

[thaiboxerken]

I disagree. It certainly is interesting to many people.

Do you count yourself and the frog in your pocket as "many" people?

So far, you're the only one I know of that wants a "meta"analysis.


If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)

Only if the tests were all similar. They are not.

[Paul C. Anagnostopoulos]

T'ai said:If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)
I agree: Assuming the meta-analysis was meaningful, we should investigate further. But in the case of the JREF, further investigation simply means new claimants for the prize, who will come along regardless of any meta-analyses.

In the real world, psi researchers take the meta-analyses as evidence for psi. They are not.

~~ Paul

[Prester John]

A meta analysis could be done on all tests which produce some sort of score, correct guesses etc. There would have to be a degree of similarity in what is being measured.

If you assume no one has psychic ability then most results should show a score of 0 (no different from chance) with a few showing slightly positive or slightly negative results - normally expected variance. Some sort of graphical representation would be easiest.

It all depends on how compatable the data is really.

[T'ai Chi]

Originally posted by Paul C. Anagnostopoulos

In the real world, psi researchers take the meta-analyses as evidence for psi. They are not.

~~ Paul

I think that some people could be doing that. But yes, that is wrong.

However, if nothing is going on, we expect the results to be compatible with chance. If the results are not compatibible with chance, then something is going on.

That 'something' could be real effects or errors.

[Dub]

IMO meta-analysis would be pointless. In the tests which would create data that could be analysed statistically - such as dowsing etc, the claimant fails if they cannot perform significantly better than by chance alone. So far everyone has failed. Thus, meta-analysis of this data would simply come to the same conclusion that each individual test came to - that the claimant can perform no better than by chance alone.

[T'ai Chi]

Originally posted by Dub
In the tests which would create data that could be analysed statistically - such as dowsing etc, the claimant fails if they cannot perform significantly better than by chance alone. So far everyone has failed. Thus, meta-analysis of this data would simply come to the same conclusion that each individual test came to - that the claimant can perform no better than by chance alone.

It certainly could happen that way, you're correct.

However, since meta-analysis examines things as a group, it could turn out that, as a group, dowsers, while still failing to acheive significance in their individual tests, still have more positive scores than what is expected by chance.

(or of course, more negative scores than what is expected by chance).

[Charlie in Dayton]

I find this attempt to square the circle extremely confusing.

If the individuals as individuals couldn't do squat, why would the individiduals as a group produce kumquat salad?

QUESTION: Did the individuals who claimed they could do X actually do X?
Yes = 1, No = 0.

Nobody has ever been able to do whatever they said their personal paranormal X was.

0 = 0. 0 to the Yth power is still 0.

And (he said as he put his hand to his furrowed brow in a decent Karnak The Magnificient impersonation) don't give me this bunk about I don't understand meta-analytical techniques. I don't have to understand something that doesn't apply here.

I repeat -- 0 = 0. 0 to the Yth power is still 0.

Why is it necessary to add flour, baking soda, sugar and milk here? Them little brown things are rabbit pellets, and all the additives in the world ain't gonna convince me that the end result is chocolate chip cookies...:rolleyes:

[T'ai Chi]

Meta analysis is a sound statistical method to examine scores from similarly conducted experiments as a group.

A meta analysis is not interested in if the applicants could do the experiment or not. A MA uses a method and combines all the standardized scores of the applications into one combined standardized score.


0 = 0. 0 to the Yth power is still 0.


We aren't examining 0's (although some standardized scores could be 0's). We are looking at their standardized scores and combining them into one standardized score.

Let's just see if the combined standardized score will be around 0 (and non-significant) as expected by chance.

But... what if most of the standardized scores are positive (or negative) ? What if the results are off from what we expect by chance?

don't give me this bunk about I don't understand meta-analytical techniques. I don't have to understand something that doesn't apply here.


Why exactly doesn't a meta analysis apply to similarly conducted tests where we can obtain standardized scores? I'm trying to understand why you think it doesn't apply. I'd like something a little more applicable than the '0 * 0! + 0^500 = 0 forever!!!' type of stuff.

Them little brown things are rabbit pellets, and all the additives in the world ain't gonna convince me that the end result is chocolate chip cookies...:rolleyes:

It is a good thing I could care less about convincing you. I'm simply interested in scientifically examining the scores and seeing where that leads.

[CFLarsen]

T'ai Chi,

You need to do a little homework first, e.g. read about how the tests are designed.

Each challenge is different from any other, simply because it has to be tailored to the specific claim of the claimant.

It would be highly unfair of Randi if he insisted that Dowser A should take the same test as Dowser B, if Dowser A didn't claim the same as Dowser B.

So, there are not "standardized scores" to do a meta-analysis on.

The only thing we can look at is the results. They are all the same: "Failed". No need to look for anything there.

[T'ai Chi]

The challenges are different, of course, but the tests may be very or somewhat similar, enough to do a meta analysis on, in dowsing, and several other areas.


So, there are not "standardized scores" to do a meta-analysis on.


Sure there are. There are standardized scores in any test where there is a score recorded, and an expected score, and a measure of deviation.


The only thing we can look at is the results.


That might be the only thing you want to look at, but many analyses of the data can be done.


They are all the same: "Failed". No need to look for anything there.


So you're not interested in the question of: 'Is the combined standardized score what we would expect by chance?' I am.

[CFLarsen]

T'ai Chi,

If you feel that the tests are similar enough to do a meta analysis on, how would you do it, then? Don't ask others, I want to hear your own reasoning.

With statistics, please. Don't worry about me, if you say something I don't understand, I'll ask.

How would you perform a meta-analysis on the JREF challenges?

[Charlie in Dayton]

Originally posted by T'ai Chi

A meta analysis is not interested in if the applicants could do the experiment or not.

This is the reason for the experiments in the first place. What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?

You're trying to find out if paranormal ability X exists. If mathematical procedure Y doesn't assist you in determining if X exists, why waste the effort? This is like going out to buy a car with the major criterion being how good a sailboat it is when you drive it off the boat dock.

These tests are conducted on individuals. Why are you trying to lump things together and get a result for a group? Throw fifty dowsers together in a field, and all you're going to get are numerous flesh wounds as they stagger around stabbing each other with their bent coat hangers. What are you looking for here? Why cloud the picture with admittedly irrelevant procedures?

I'm of the opinion that you're looking for the mathematical anomaly that 'proves' the existence of some paranormal whatnot, yet in actuality proves nothing at all. "Sure, let's throw another bushel basket full of numbers in front of 'em -- if ya can't dazzle 'em with dexterity, baffle 'em with b*llsh*t!" After all, 2+2=5, for large values of 2...

This is not a question of standardized testing, or meta-analysis, or anything else other than a simple yes-or-no answer to the question "Can you perform what you claim under proper controlled conditions?" Attempts to throw some sort of mathematical analysis into this question are irrelevant, immaterial, and I personally believe to be a symptom of a hidden agenda.

Now, go peddle your rabbit pellets to someone else. I ain't interested...

[T'ai Chi]

You answer my questions, I'll answer yours.


So, there are not "standardized scores" to do a meta-analysis on.


Why do you think there are not standardized scores?


No need to look for anything there.


Are you interested in the question of: 'Is the combined standardized score what we would expect by chance?'


If you feel that the tests are similar enough to do a meta analysis on, how would you do it, then? Don't ask others, I want to hear your own reasoning.


Thanks for your concern about asking others, but I can answer your question however I deem appropriate.

There are many good references on how to carry out a meta analysis. I would consult those and proceed as recommended. In general, in those references you'd find advice like what is at the bottom of this page (http://149.170.199.144/rd/meta.htm).

[T'ai Chi]

Originally posted by Charlie in Dayton

What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?


Do you expect the combined standardized score to be what is expected by chance, or not? That certainly has bearing on the subject of the data.


You're trying to find out if paranormal ability X exists.


No, I'm not. I'm just interested in analyzing the data that others collect, and seeing where those analyses lead.


These tests are conducted on individuals. Why are you trying to lump things together and get a result for a group?


To see if we get a combined standardized score around what we expect by chance. If we get a combined standardized score that is significantly away from chance levels, then there might be something going on (effects or errors).


I'm of the opinion that you're looking for the mathematical anomaly that 'proves' the existence of some paranormal whatnot, yet in actuality proves nothing at all.


Your opinion would be incorrect then, because I'm simply interested in analyzing the data.


Attempts to throw some sort of mathematical analysis into this question are irrelevant, immaterial, and I personally believe to be a symptom of a hidden agenda.


Apparently you believe that analyzing the data in appropriate ways is innapropriate. I've already stated that I'm not out to prove anything. Your repeated insistence of hidden agendas and that I'm out to prove the paranormal is still incorrect, and will be no matter how many times you suggest it.


Now, go peddle your rabbit pellets to someone else. I ain't interested...

And as I've said, I could care less about convincing you or anybody else. I am only interested in scientifically analyzing the data and seeing where that takes us.

[CFLarsen]

Originally posted by T'ai Chi
You answer my questions, I'll answer yours.

Sure.

Originally posted by T'ai Chi
Why do you think there are not standardized scores?

Because I have not heard Randi describe them. On the contrary, he has written about how each test is designed individually. Why are you asking this question in the first place? Have you done any homework at all?

Originally posted by T'ai Chi
Are you interested in the question of: 'Is the combined standardized score what we would expect by chance?'

Nope.

Originally posted by T'ai Chi
Thanks for your concern about asking others, but I can answer your question however I deem appropriate.

There are many good references on how to carry out a meta analysis. I would consult those and proceed as recommended. In general, in those references you'd find advice like what is at the bottom of this page (http://149.170.199.144/rd/meta.htm).

I didn't ask how to carry out a meta analysis. I asked how you would do it on the JREF challenges.

I've answered your questions, you answer mine.

[Charlie in Dayton]

Let's see if I've got the hang of this multi-level quote thing...

Charlie in Dayton
What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?
T'ai Chi
Do you expect the combined standardized score to be what is expected by chance, or not?
These are NOT standardized tests that are given across the board. They are individual, and agreed to in advance by both parties. There may be similarities at times, and indeed two or more individuals may agree to take the same test (there's an excellent example of that in Randi's book Flim-Flam). BUT - the tests are adiministered individually, and whether or not they're similar, identical, or otherwise, the results apply to the INDIVIDUAL -- period.

T'ai Chi
...I'm just interesting in analyzing the data that others collect, and seeing where those analyses lead.
The analysis of each individual bit of data is to determine whether or not individual X can conclusively demonstrate claimed paranormal ability Y under proper and agreed-on-in-advance controlled conditions. Analysis of the data to any other conclusion is irrelevant to the point of the experiment, and serves to cloud the issue. What is the purpose of analyzing the performance of the group when it wasn't the group that was tested? We're not looking for group performance here. Now, if four guys walked up and said in advance that individually they can't do beans, but together as a group they can do all sorts of wonderful paranormal things, there would be a test of the group. At that time, I would seriously suggest that in addition, the members be tested individually and the individual test results be analyzed and compared to the group's performance. But it's not kosher to invent a group when there wasn't one being tested in the first place.

T'ai Chi
Apparently you believe that analyzing the data in appropriate ways is innapropriate.
Not at all. The bone of contention at this point is, what's an appropriate way to analyze the data? You want to create an entity that never existed and analyze it. These tests were neither designed for nor run on groups. Why are you so insistent on analyzing the data that way?

T'ai Chi
I've already stated that I'm not out to prove anything.
Horse puckey. If you're not trying to prove anything, why bother analyzing the data? What, you're going to get to the equals sign and stop there?

T'ai Chi
And as I've said, I could care less about convincing you or anybody else.
Meadow muffins. If you weren't trying to convince somebody somewhere, you wouldn't be here doing this.

T'ai Chi
I am only interested in scientifically analyzing the data and seeing where that takes us.
Rabbit pellets. That mantra of 'using the scientific method' wears very thin when it's bad science being used.
You're using bad science here. Stop trying to make peach pie out of a basket of apples.

[T'ai Chi]

On the contrary, he has written about how each test is designed individually.


Then an assumption of independence for doing a meta analysis is clearly met.

Randi has described the observed and expected scores (and from those you get standardized scores from something like standardized score = (observed-expected)/standard deviation) relating to dowsing (http://www.randi.org/jr/011102.html), and possibly other tests in Swift.

Why are you asking this question in the first place? Have you done any homework at all?


You're asking the question of why I asked the question?? Because I want your answer.

Far from me not doing "homework", I'm asking you because you hinted that there are not standardized scores. Obviosuly in dowsing, some ESP, and several other tests, there certainly are scores. If you have observed scores, and expected scores, and a measure of spread, you have all the ingredients of a standardized score.

Why aren't you interested in the question of: 'Is the combined standardized score what we would expect by chance?' ?


I didn't ask how to carry out a meta analysis. I asked how you would do it on the JREF challenges.


And how I'd do it would follow those guidelines that I gave in the link. Simply replace "relevant literature", "independent studies", "published information", and "studies" with "JREF tests", and proceed accordingly.

[Charlie in Dayton]

Originally posted by T'ai Chi
If you have observed scores, and expected scores, and a measure of spread, you have all the ingredients of a standardized score.


The observed score is 0.

The expected score is 0.

The spread is 0.

The standardized score is 0.

These results are consistent with each individual. The results of the group score are nonexistent, because there was no group tested as a group.

This concludes our meta-analysis.

Gee, that was simple, wasn't it? :D

[T'ai Chi]

Originally posted by Charlie in Dayton

These are NOT standardized tests that are given across the board. They are individual, and agreed to in advance by both parties.

I agree, fully, 100%, absolutely. However, dowsing tests are similar, and simply differ in numbers of pipes, or numbers of cannisters, or number of participants. The basic set ups are verysimilar. And therefore a meta analysis is appropriate and can be done.

BUT - the tests are adiministered individually,


Great! That is actually a GOOD thing, because it guarantees independence between tests, something which is required for a meta analysis.

Analysis of the data to any other conclusion is irrelevant to the point of the experiment, and serves to cloud the issue.


I disagree, completely. Data can be analyzed in any way that serves to explore a question of interest. If one wants to see if the combined standardized score is near what we expect by chance, one can do that analysis.


What is the purpose of analyzing the performance of the group when it wasn't the group that was tested?


As I've stated, we are exploring whether the combined standardized score is near what we expect by chance. It should be, shouldn't it?


, but together as a group they can do all sorts of wonderful paranormal things, there would be a test of the group.


I don't know how many times I will have to say, but I am not interested in proving that any paranormal phenomena exists.


You want to create an entity that never existed and analyze it.


Oh, I disagree. The data has always existed.

Why are you so insistent on analyzing the data that way?


Let me copy and paste my answer for the 4th or 5th time: The combined standardized score should be near what we expect by chance. Is it, or not?


Horse puckey. If you're not trying to prove anything, why bother analyzing the data?


Curiousity? Exploration? Fun? Seeing if I can analyze data? Seeing if the combined standardized score is near what we expect by chance? Applying statistical methodology in new and exciting ways? Testing the hypothesis that I can still use my calculator.


Meadow muffins. If you weren't trying to convince somebody somewhere, you wouldn't be here doing this.


Very poor attempt.

I am "here" because I am interested in exploring if the combined standardized score is near what we expect by chance, and, obviously I am replying to other posters' questions, including your own.


That mantra of 'using the scientific method' wears very thin when it's bad science being used.

Could you explain your opinion of "bad science"? Are you saying meta analysis is "bad science"?


You're using bad science here. Stop trying to make peach pie out of a basket of apples.

You have utterly failed to present a coherent case for that. You must have the standard topics of science confused with the flexible methods of science.

[T'ai Chi]

Originally posted by Charlie in Dayton

The observed score is 0.
The expected score is 0.
The spread is 0.
The standardized score is 0.
These results are consistent with each individual. The results of the group score are nonexistent, because there was no group tested as a group.
This concludes our meta-analysis.
Gee, that was simple, wasn't it? :D

Well, it was simple-minded anyway. ;)

DATA :k:

[CFLarsen]

Originally posted by T'ai Chi
And how I'd do it would follow those guidelines that I gave in the link. Simply replace "relevant literature", "independent studies", "published information", and "studies" with "JREF tests", and proceed accordingly.

Then, you have absolutely no idea what you are doing. You want to perform a meta-analysis on something that has no common ground, based on a text you clearly do not understand, but think you can just "replace" words to make it work.

You haven't done any homework regarding how the tests are being done.
You haven't exhibited any knowledge whatsoever regarding how to do a meta-analysis.
You haven't exhibited any knowledge whatsoever regarding what a meta-analysis is.
You haven't understood why you can't do a meta-analysis on the tests.

I think you just like to throw fancy terms around to appear smart. Well, you are not. Your own posts prove that.

Let's try again: In your own words, please explain how you would do a meta-analysis.

In your own words. That's where you get in trouble, isn't it?

[EdipisReks]

:hb:

[T'ai Chi]

Originally posted by CFLarsen

Then, you have absolutely no idea what you are doing.


You statement of dismissal does nothing to rationally debate your case.


You haven't done any homework regarding how the tests are being done.


What are you talking about? Where is your evidence for that claim? What do you mean by homework? Have you read any of the Randi commentaries that talk about specific dowsing tests? Do you agree or disagree that said commentaries talk about observed and expected scores? Do you agree or disagree that, with an easily calculated measure of spread, one can obtain standardized scores? I have read these, and understand statistics and how it applies to these tests, so how does that equate with my not doing my homework?


You haven't exhibited any knowledge whatsoever regarding how to do a meta-analysis.


Interesting. Are we in the same thread? Note: I don't expect you to answer rhetorical questions.

I've discussed meta analysis plenty in this thread, and gave pointers to a fairly complete link on meta analysis. The disconnect comes from you assuming I don't understand what is on the link, or how it applies to the JREF tests-something which you assume.


You haven't exhibited any knowledge whatsoever regarding what a meta-analysis is.


You are clearly incorrect. I've stated that it is to combine results from similar experiments, and I've also provided a link that explain what a meta analysis is. Do you dispute this?


You haven't understood why you can't do a meta-analysis on the tests.


No person has given any good reason why one can't. So tell us Claus, why can't one do a meta analysis? In order to say that you can't do a meta analysis, I assume you know all the specifics about meta analysis theory and application. Is it safe to say that you do know a lot about meta analysis, Claus?


I think you just like to throw fancy terms around to appear smart. Well, you are not. Your own posts prove that.


Let's not discuss what your posts show.

I want to see if the combined standardized score is near what we expect by chance. Everything else you psychics read into my words is your invention, conjured out of thin, very thin but amazingly stuffy, air.


Let's try again:


Let's not try again, Claus.

I am not required in any way to answer your questions in my own words ad nauseum if there are already sources out there explaining exactly how to carry out a meta analysis.

I'm not interesting in debating your online personality, I'm interested in analyzing some data, and discussing statistical issues. If you irrationally believe I am trying to appear smart by throwing around terms, then you have nothing to contribute to the discussion, and your further postings will not regretably be ignored.

If, on the other hand, you wish to discuss actual science, statistical, or testing issues, that would be most welcome.

[dissonance]

T'ai Chi, what's your statistics background? Graduate level? Undergrad? Interested layperson? We (OK, I) might have an easier time explaining why what your asking is inappropriate if we knew where you were coming from on this.

[tedly]

T'ai chi

If you take the meta-analysis maybe nothing is still going on.
We have a frighteningly common belief that there is something out there. Suppose we get a thousand by thousand matrix of possible interactions. If an interaction is so unlikely that it only has a one in 10000 chance of occurring, we find 100 such artifacts in our dataset. Similarly a belief so wierd that only one in a million could bite on it, has 6000 people out their that hold it as a basic tenet of their faith.

If you torture the data long enough it will confess to anything. The members seem to be sufficiently humane that they don't want to torture the data. Let go.

[CFLarsen]

T'ai Chi,

You are very right - this is not about me, it's about you.

Why are you unable to explain, in your own words, how you are going to do a meta-analysis on the JREF challenges?

It should be easy for you: You claim to have done your homework, by reading what Randi has said about the challenges. You also claim to know how to do a meta-analysis. You seem reasonably able to express yourself verbally. So, what's stopping you?

If you don't want to explain it, then just go ahead and do this meta-analysis of yours.

What's stopping you?

[69dodge]

Originally posted by T'ai Chi
I want to see if the combined standardized score is near what we expect by chance.Suppose it's not. What do you think we could conclude from that? How would it change your beliefs or your actions? In short, what difference would it make?

If the results would make no difference, there's no point in doing the meta-analysis to begin with.

[T'ai Chi]

Originally posted by CFLarsen

You are very right - this is not about me, it's about you.


Actually, I'm pretty sure it is 100% about meta analysis.


Why are you unable to explain, in your own words, how you are going to do a meta-analysis on the JREF challenges?


I don't let myself be bullied. I've given you my answer already.


just go ahead and do this meta-analysis of yours.

What's stopping you?

We were discussing the appropriateness (or not) of a meta analysis. Now you seem to want to discuss why I am not currently doing a meta analysis. I'm not going to humor your topic change.

[T'ai Chi]

Originally posted by 69dodge
Suppose it's not. What do you think we could conclude from that? How would it change your beliefs or your actions? In short, what difference would it make?

If the results would make no difference, there's no point in doing the meta-analysis to begin with.

Great questions 69dodge. If the combined standardized score is not what we expect by chance and it is statistically significant, then it would be interesting to see how it is statistically significant, and to see what that means in terms of dowsing ability, for example, and the experiments themselves.

If the combined standardized score is significant, we need to see if most of the standardized scores are positive, negative, or if they are all about average with 1 or 2 'superstar' standardized scores.

If most of the scores are positive or negative, it could mean there is systematic bias in the experiments that make the participants score too high or too low on average. This knowledge could lead into improving the experiments, designing new ones, and in general, learning more about dowsing and testing paranormal abilities.

I'd personally expect the combined standardized score to be around what chance predicts, but that is just my belief. The data itself could verify that, or not.

[T'ai Chi]

Originally posted by dissonance
T'ai Chi, what's your statistics background? Graduate level? Undergrad? Interested layperson? We (OK, I) might have an easier time explaining why what your asking is inappropriate if we knew where you were coming from on this.

Hi dissonance! I have graduate level (and yes, I did graduate :) ) and professional knowledge of statistics.

[CFLarsen]

T'ai Chi,

"Bullied"? What are you talking about?

You want to do a meta-analysis. Why is it "bullying" to ask you how you are going to do it?

There is no change of topic. If you don't want to discuss how you are going to do it, then just go ahead and do it.

What's stopping you?

[thaiboxerken]

Originally posted by T'ai Chi


Hi dissonance! I have graduate level (and yes, I did graduate :) ) and professional knowledge of statistics.

So then, maybe you have adequate knowledge to perform a meta-analysis. Feel free to fly to Jacksonville and start anytime. The JREF's records are accessible by the public.

Oh, and we'd like to see how you performed the analysis as well.

[T'ai Chi]

Originally posted by CFLarsen
, then just go ahead and do it.

What's stopping you?

The fact that I am not currently doing a meta analysis is in no way at all related with discussing the appropriateness of doing a meta analysis.

I am also not carrying out an analysis of black holes, but I can still discuss your posts, I mean, discuss cosmology.

In fact, I can personally have no intention of ever doing a meta analysis, and still be, obviously, justified in discussing meta analysis.

[CFLarsen]

Originally posted by T'ai Chi
The fact that I am not currently doing a meta analysis is in no way at all related with discussing the appropriateness of doing a meta analysis.

I am also not carrying out an analysis of black holes, but I can still discuss your posts, I mean, discuss cosmology.

In fact, I can personally have no intention of ever doing a meta analysis, and still be, obviously, justified in discussing meta analysis.

So, what is your point of this thread? Let's take a look at your opening post:

Originally posted by T'ai Chi
What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

You asked for people's opinions. Not one supported you. All gave perfectly good reasons why such an analysis would be futile.

Now, you say - after many posts where you defend the viability of just such a meta-analysis - that you are not "currently" doing such an analysis.

May I ask: What is your point, then? You really are just a troll, aren't you?

[Thanz]

Originally posted by T'ai Chi

Great questions 69dodge. If the combined standardized score is not what we expect by chance and it is statistically significant, then it would be interesting to see how it is statistically significant, and to see what that means in terms of dowsing ability, for example, and the experiments themselves.
This is where I get confused. I don't see how it is possible to take a group of scores that are not statistically significant, combine them, and somehow get a statistically significant answer. For the JREF challenges, certainly not a positive one. We know that no one has passed the preliminary test. This tells me that no one has been able to perform whatever task to a statistically significant level above chance.

Let's keep it easy and confine it to dowsers, as I think Randi has mentioned he gets a number of them. Let's say that chance would dictate they get 5 of 10 correct, and that in order to pass (to be statistically significant) they need to get 8 of 10 correct. (Note: I have no idea if 8 is the right number. It is not important to my argument whether it is or not). We know, as no one has passed, that no one has gotten 8 or more correct. If we combine all of these scores, and average them, it will still add up to less than 8 out of 10, and be non-statistically significant.

Maybe I am being too simple minded about "meta analysis", but I don't see how you can combine a bunch of results that at best hover around the chance mark and come up with a positive statistically significant number.

[Prester John]

So where would the data be then? Its a bit pointless arguing about whether a meta analysis would be useful (doable)unless you can actually look at the data.

I don't think there should be any problem with doing some sort of meta analysis. Its' quite reasonable. I think maybe people are taking a stance against the suggestion because of the suggester.

A meta analysis is a tool of science.

The only problem is how comparable the data is and how easily analysable it is, it strikes me it shouldn't be too hard...... but may take quite a while to do (compiling the data).

My background is that i am a biomedical scientist and am involved in medical and quality statistics as secondary functions of my job/proffession.

[jj]

Originally posted by Thanz

This is where I get confused. I don't see how it is possible to take a group of scores that are not statistically significant, combine them, and somehow get a statistically significant answer.

If we take one "coin flip" and get 8/10 "heads", that's on the edge of 5%.

If we take 10 such events, each one providing 8/10, we wind up with 80/100 which is, although I don't have my handy program up and running at the minute, WAY outside any normal definition of "random".

What does this mean? Well, for 'n' trials, the "noise" due to randomness is proportional to sqrt(n). The number of trials is n, so the total normalized noise is sqrt(n)/n, or 1/sqrt(n).

This means that the percentage (or per-trial) noise (as opposed to total counts) is proportional to 1/sqrt(n) when examining the whole data set.

In other words, 1% off random n 10 trials is meaningless. 1% off in 100 trials is very normal, expected randomness. 1% off random in 1,000,000,000 trials, on the other hand, looks like a real effect.

[Thanz]

Thanks jj. I get it now.

[Lucianarchy]

Originally posted by T'ai Chi
What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

First of all you need access to the data. I understand this can all be viewed, only if you go along to the JREF in Florida, and ask.

I assume it'll be found in a filing cabinet, at the bottom of some cellar, behind a door. With a sign which says "beware of the leopard.". (Apol's to Mr Adams.)

Unless it's published and peer reviewed, that is. But, it's not.

[T'ai Chi]

Originally posted by Thanz
[B]
This is where I get confused. I don't see how it is possible to take a group of scores that are not statistically significant, combine them, and somehow get a statistically significant answer.
/B]

If the majority of the standardized scores, while non significant themselves, are negative or positive, this could lead to a significant combined standardized score.

[T'ai Chi]

Originally posted by CFLarsen

You really are just a troll, aren't you?

Thanks for your concern, and the label, Claus.

This thread is about an analysis, not assigning cute labels for people. Further attempts at diverting the topic will simply be ignored.

[FutileJester]

My question would be: what measurements are we going to be subjecting to analysis?

The JREF data would be particularly problematic since even the broadest measures (like 'hits') are defined individually for each test. It's a signifigant element of the challenge that each test is custom-tailored to the exact claim made. This is, I figure, much more variation than in a typical medical meta-analysis where many measures are gathered using standard protocols, or where the same experiment is replicated with some variations. To me, it seems that adding 1 hit from a dowser to 1 hit from a blindfolded reader doesn't equal 2 hits in any meaningful way; it equals an apple and an orange, so to speak.

Can anyone think of any measures that would be meaningful across a broad range of trials? What specific conclusions could we draw from statistics on these measures?

[thaiboxerken]

First of all you need access to the data. I understand this can all be viewed, only if you go along to the JREF in Florida, and ask.

I assume it'll be found in a filing cabinet, at the bottom of some cellar, behind a door. With a sign which says "beware of the leopard.". (Apol's to Mr Adams.)

It must be accessible to the public, in accordance to law.


Unless it's published and peer reviewed, that is. But, it's not.

They are not science studies.

[jj]

Originally posted by T'ai Chi


Thanks for your concern, and the label, Claus.

This thread is about an analysis, not assigning cute labels for people. Further attempts at diverting the topic will simply be ignored.

You are quite obviously a troll.

Your conduct in other threads has been quite scurrilous, you have demonstrated a most peculiar understanding of the process of science, and your argument tactics are visibly designed to suit emotion rather than logic.

Your objection to labelling, therefore, is purely propagandistic.

Your proposed analysis is utterly invalid, you can't combine an apple, two pears, and a milkweed pod, unless you're a japanese flower arranger.

[T'ai Chi]

Originally posted by FutileJester
To me, it seems that adding 1 hit from a dowser to 1 hit from a blindfolded reader doesn't equal 2 hits in any meaningful way

Hi FutileJester,

I agree, that if a meta analysis were done, the studies would have to be combined in a way that makes sense.

Perhaps only the dowsing studies could be combined?

[Charlie in Dayton]

Here we go again...

The only paranormal ability I saw mentioned was dowsing, so let's use that, as it comes out extremely simply.

The individual can either do what they claim, or they cannot. Yes or no. 1 or 0. Black or white. Chevy or Ford. Pepsi® or Coke®. Either/or.

There is no middle ground.

Analyzing the data and subjecting it to conditions that were not in existence at the time of the test is BAD SCIENCE!!! It's rewriting the rules halfway through the game. That's not the way things are done, and you know it.

The individuals either have the paranormal ability, or they do not. What third option might a meta-analysis of the data show? Do you want to quantize it? Fine -- zero still equals zero.

What is it specifically that you think might be shown by this analysis? How do you hope to show some statistically significant result when there is NO DATA to support it?

You keep trying to put a number to all this. You can't, because YES and NO are not numbers!

You're trying to make something out of nothing here (literally). That's bad science. Stop it.

[Ed]

A quick note. This is clearly a troll. If you have some number of tests accross some number of subjects and the results for each subject are not significant there is not,, short of woo-woo statistics, that you can suddenly get significance. The use of meta analysis is for diagnostics not bailing out failed experiments. Stimson did a nice treatment of it in this forum on my thread about the Banality of Paranormal research.

[Flo]

Originally posted by Charlie in Dayton

Analyzing the data and subjecting it to conditions that were not in existence at the time of the test is BAD SCIENCE!!! It's rewriting the rules halfway through the game. That's not the way things are done, and you know it.

The individuals either have the paranormal ability, or they do not. What third option might a meta-analysis of the data show? Do you want to quantize it? Fine -- zero still equals zero.

What is it specifically that you think might be shown by this analysis? How do you hope to show some statistically significant result when there is NO DATA to support it?

You keep trying to put a number to all this. You can't, because YES and NO are not numbers!

You're trying to make something out of nothing here (literally). That's bad science. Stop it.

Actually, I've seen the same line of argument on a French-speaking forum ( www.sceptiques.qc.ca ) . The demand for a statistical analysis doesn't relate to the actual results of experiments but on the conditions surrounding them, i.e. what factors could be taken into account to explain the consistently negative results and find some sort of a hint of a beginning of positive results ("negative psy", lack of accounting for the psychological and emotional conditions of the testees, ratio of believers:unbelievers among people attending or controlling the tests, age of the captain, influence of the phases of the moon, bad feng-shui, ...).

[pgwenthold]

Originally posted by Ed
A quick note. This is clearly a troll. If you have some number of tests accross some number of subjects and the results for each subject are not significant there is not,, short of woo-woo statistics, that you can suddenly get significance.

Nonsense.

See the previous example. Suppose we run a test where the outcome 1/4 by chance. We run 10 trials, and the person gets 3 right. Is that a significant result? Not at all.

Now, suppose we test 1000 people, and they all get 3/10 right, and fail. Significant? You bet it is.

3000/10000 is far from random chance, despite the fact that not a single individual hit a significant rate.

No woo-woo statistics involved.

[Ed]

Originally posted by pgwenthold


Nonsense.

See the previous example. Suppose we run a test where the outcome 1/4 by chance. We run 10 trials, and the person gets 3 right. Is that a significant result? Not at all.

Now, suppose we test 1000 people, and they all get 3/10 right, and fail. Significant? You bet it is.

3000/10000 is far from random chance, despite the fact that not a single individual hit a significant rate.

No woo-woo statistics involved.

The stats are not woo-woo (maybe) the interpretation is. Clearly any series of random numbers will vary from chance and sometimes they might even vary significantly. This is as it should be. The difficulty arises from the source of the data. If the experiment is not well designed and if there is not a falseifyable hypothesis, you are dealing with a fishing expedition, not science. I note the elaboration of purported intervening factors (with very impressive terminology, I might add) to explicate PK and other phenomena. Fine and good except that it is not clear that there are any phenomena at all.

What it all comes down to is that there is no clear evidence of anything paranormal to begin with.

What, precisely, is the difference between the current state of paranormal research and a bunch of labs generating random numbers and trying to assign a paranormal explination to them? If you hung your professional hat on being a researcher into the paranormal, and basically all you got for your efforts is bupkus, would you not analyze the hell out of it?

As an old friend likes to say "You can torture data to say anything".

[Ed]

Originally posted by Flo


Actually, I've seen the same line of argument on a French-speaking forum ( www.sceptiques.qc.ca ) . The demand for a statistical analysis doesn't relate to the actual results of experiments but on the conditions surrounding them, i.e. what factors could be taken into account to explain the consistently negative results and find some sort of a hint of a beginning of positive results ("negative psy", lack of accounting for the psychological and emotional conditions of the testees, ratio of believers:unbelievers among people attending or controlling the tests, age of the captain, influence of the phases of the moon, bad feng-shui, ...).

This is data torture. Now, if there were some consistant effect, looking at what influences it would make sense. This is more on the order of "let's manipulate things until we get something and then we can declare success". So what they are saying, in essence, is that there is an effect, we don't need research we know it exists now we need to find out why we don't see it. This is non-falsification, not science, and religion.



Back to basics: What is the hypothesis that is being tested?

[Flo]

Originally posted by Ed


This is data torture. Now, if there were some consistant effect, looking at what influences it would make sense. This is more on the order of "let's manipulate things until we get something and then we can declare success". So what they are saying, in essence, is that there is an effect, we don't need research we know it exists now we need to find out why we don't see it. This is non-falsification, not science, and religion.

Back to basics: What is the hypothesis that is being tested?

This is exactly what I answered the guy. As could be expected, his answer was a mix of ad hominem and protests of good faith. :rolleyes:

[T'ai Chi]

Originally posted by Ed
A quick note. This is clearly a troll.


Spare us your dismissive ad hominems, and please stick to the topic.


If you have some number of tests accross some number of subjects and the results for each subject are not significant there is not,, short of woo-woo statistics, that you can suddenly get significance.


Really? If most of the standardized scores are positive (or negative), that wouldn't be something?

No one is saying that if the combined standardized score was significant that that would prove the paranormal or whatever. What they are saying is that if the combined standardized score was significant, that is, not near 0 and non-significant as we'd expect by chance, then something is going on, either an effect or problems with the design of the tests, or something else.

[Ladewig]

Why do you think that there are enough quantifiable tests in the JREF preliminary test history to see trends rather than noise?

If you did meta-analyze the data and came up with some positive result, what would you do with that information? editted to add: Never mind - I just saw your answer in the previous post.

----------------
Maybe we should meta-analyze the JREF data and then combine those results with the results of other meta analysis to produce an epi-meta-analysis? :rolleyes:

[T'ai Chi]

Originally posted by Charlie in Dayton

The individual can either do what they claim, or they cannot. Yes or no. 1 or 0. Black or white. Chevy or Ford. Pepsi® or Coke®. Either/or.


That is true.. But in many dowsing experiments they are dowsing, say, 5 film cannisters for 1 of them that has gold in it. The dowsers might do 30 trials. We'd expect them to get 30*(1/5) = 6 hits by chance alone.

Statistics are kept other than 0 or 1 outcomes, that can be analyzed.


Analyzing the data and subjecting it to conditions that were not in existence at the time of the test is BAD SCIENCE!!!


I disagee. It is good science to analyze the data, and to test the hypothesis of the combined standardized score being what we'd expect by chance.

By your rules, no one could ever do a meta analysis in any discipline.


The individuals either have the paranormal ability, or they do not.


Oh, I agree 100% with that. They either do or don't. I agree. What I am wondering is is the combined standardized score near 0 and non significant as we'd expect it would be.

How do you hope to show some statistically significant result ..


Whoah, back up. :) I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question.

[Ed]

Originally posted by T'ai Chi


Spare us your dismissive ad hominems, and please stick to the topic.

[/b]

Really? If most of the standardized scores are positive (or negative), that wouldn't be something?

No one is saying that if the combined standardized score was significant that that would prove the paranormal or whatever. What they are saying is that if the combined standardized score was significant, that is, not near 0 and non-significant as we'd expect by chance, then something is going on, either an effect or problems with the design of the tests, or something else. [/B]

Please, if they were positive there would have been results. Or, are you suggesting pooling all scores and standardizeing them? You surely don't mean that, do you? That would likely be junk statistics.

I will also ad hominum as I see fit, thank you. Your topic is moribund.

[Ed]

Originally posted by T'ai Chi


Whoah, back up. :) I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question. [/B]

Your contention regarding this type of analysis is meaningless, I'm afraid. You surely see why, do you not?

[jj]

Originally posted by T'ai Chi


That is true.. But in many dowsing experiments they are dowsing, say, 5 film cannisters for 1 of them that has gold in it. The dowsers might do 30 trials. We'd expect them to get 30*(1/5) = 6 hits by chance alone.

Statistics are kept other than 0 or 1 outcomes, that can be analyzed.


And how do you analyze the results from tests with different statistics, different expected outcomes, etc?

Note, I'm not saying you can't, strictly speaking, but I am asking you how you would make meaning from such results.

By your rules, no one could ever do a meta analysis in any discipline.


Close, but not quite. The whole problem is that it is really quite a rare case where a meta-analysis is appropriate.

Even tests of different people using the exact same methodology and such need to be tested for consistancy.

Testing the relevance of two different tests on different people at different times is just not meaningful.

Whoah, back up. :) I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question.
You're "impartial", yet you insist in other threads, for instance, that there is positive evidence for homeopathy?

Uh huh. And my name is Dilbert. Really, it is. :roll:

[T'ai Chi]

Originally posted by Ed

Please, if they were positive there would have been results. Or, are you suggesting pooling all scores and standardizeing them? You surely don't mean that, do you? That would likely be junk statistics.


I am interested in testing if the combined standardized score is near 0 and non-significant as expected by chance. Everything else said about my 'agenda' is what people invent out of thin, but amazingly stuffy, air.

Dismissing meta analysis as 'junk statistics' or 'data mining', is not correct.


I will also ad hominum as I see fit, thank you. Your topic is moribund.

Of people continuing ad hominems, I have no doubt.

Moribund. I love word of the day (http://dictionary.reference.com/wordoftheday/archive/2002/06/22.html).

[Ed]

Originally posted by jj

You're "impartial", yet you insist in other threads, for instance, that there is positive evidence for homeopathy?



Dilbert, you forget the first rule of politics when your hand is caught in the cookie jar ... Deny, deny, deny. It is similar with paranormal research ..Analyze, analyze, analyze.

In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

[T'ai Chi]

Originally posted by Ed

It is similar with paranormal research ..Analyze, analyze, analyze.


Analyzing. I guess some scientists are definitely guilty of that. :) However, they also have models that can be dismissed or kept based on the data. They also have hypotheses that can be falsified.


In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

There are tests. They have statistics. They are probably similar enough to be combined, to see if the combined stadardized score is near 0 and non-significant as we'd expect by chance.

What goes on in other threads has no bearing whatsoever for this thread. If I said in another thread that God exists and I can prove it, etc., that has no bearing on the topic of a meta analysis for the JREF tests.

I'm amazed I even have to say that to a group of skeptical thinkers.

[pgwenthold]

Originally posted by Ed


The stats are not woo-woo (maybe) the interpretation is.


How so? Are you denying that 3000/10000 is more likely significant than 3/10?









Clearly any series of random numbers will vary from chance and sometimes they might even vary significantly. This is as it should be. The difficulty arises from the source of the data. If the experiment is not well designed and if there is not a falseifyable hypothesis, you are dealing with a fishing expedition, not science.


He is talking about meta analysis of JREF challenges. Are you suggesting that JREF is a fishing expedition?

Moreover, you changed the point. The original claim, to which I responded, is that a lot of non-significant results cannot lead to a significant result. My example shows this to be clearly wrong. You can get significance in a large data set without ever having to have significance in any of the small sets.


What it all comes down to is that there is no clear evidence of anything paranormal to begin with.




This misses the point. Perhaps you can't see the evidence because all the data sets that have been tried are too small?

A psi effect that increases the probably correctly answering from 25% to 26% is a real effect, but you are going to have a hard time seeing it if you only carry out 1000 repititions. On the other hand, if you have 1000 people carry out a thousand reps, you will have a better chance of seeing it.

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.

[Ed]

Originally posted by T'ai Chi


Analyzing. I guess some scientists are definitely guilty of that. :) However, they also have models that can be dismissed or kept based on the data. They also have hypotheses that can be falsified.



There are tests. They have statistics. They are probably similar enough to be combined, to see if the combined stadardized score is near 0 and non-significant as we'd expect by chance.

What goes on in other threads has no bearing whatsoever for this thread. If I said in another thread that God exists and I can prove it, etc., that has no bearing on the topic of a meta analysis for the JREF tests.

I'm amazed I even have to say that to a group of skeptical thinkers. [/B]

Luci, your knowledge of stats is too poor to carry on. What is the probability of drawing 911 on a specific date again?

[Skeptical Greg]

Originally posted by pgwenthold



......

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.

Ahhh.. But you can dismiss the possibility that it is significant.. Like an atom of arsenic in a gallon of water..

The question becomes; " To whom, is it worthwhile to determine if it is ' there ', no matter how ' insignificant '? "

[Lurker]

Tai:

I don't see how you could a meta analysis. The claims are all so different, with such different output. Even if you grouped dowsers together, for example, even their claims and subsequent tests would be quite different.

Some dowsers dowse for water, others? Some probably say they can get within "X" meters of the water, where "X" varies for each claimant. What other differences are there that result in them getting different output for the Randi Challenge? The possibilities are endless.

Sorry, I don't see how a meta analysis could be done. If you come up with a method, feel free to try and see what happens.

Lurker

[Ed]

Originally posted by pgwenthold


How so? Are you denying that 3000/10000 is more likely significant than 3/10?

You mean 3/10 with a sample size of 10000 as opposed to 10. For a given difference n enhances significance. You miss the point. If you mix outliers in with other data enough to change the results, you have junk. If one dowser gets 10/10 and another gets 0/10 the net result is not 5/10 or 10/20 unless the experiment was designed as a multi subject effort. One outlier would (should) raise red flags all over the place.


He is talking about meta analysis of JREF challenges. Are you suggesting that JREF is a fishing expedition?

I am saying that mushing results together is done at one's peril. Again, if no challenger passed the test, why would you expect that glomming the results together would yield any different. I'd really like to understand your thinking. To put it another way [stevie wonder] Nothin' from nothin' leeeeves nothin'[/stevie wonder].

Moreover, you changed the point. The original claim, to which I responded, is that a lot of non-significant results cannot lead to a significant result. My example shows this to be clearly wrong. You can get significance in a large data set without ever having to have significance in any of the small sets.

A single subject design that calls for 100 trials might not be significant whereas the same design that produces the same scores over 10,000 trials may be. If that is what you meant I grant you that. However that is not to say that 100 subjects with 100 trials each with the same score would yield significance UNLESS the experiment was designed that way. You can aggregate to your heart's content but if it is not apples and apples you have a curiosity, nothing more (that and a note to self to design a better experiment before you go thru 100 uncontrolled replications). My beef with woo-woo's is that they might lump stuff together and declare it proof without doing the necessary homework.


This misses the point. Perhaps you can't see the evidence because all the data sets that have been tried are too small?

A psi effect that increases the probably correctly answering from 25% to 26% is a real effect, but you are going to have a hard time seeing it if you only carry out 1000 repititions. On the other hand, if you have 1000 people carry out a thousand reps, you will have a better chance of seeing it.

True. Now, why oh why over the last 50 years or so has no one done just that, or, if they have where are the results? Dosen't the absolute lack of evidence create a doubt in your mind? Forget about fancy stats, just the fact that support for ALL paranormal claims requires a dance is not the least bit troubling?

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.


No, I cannot on an intellectual level any more than I can dismiss the possibility of the existance of God. I am an agnostic on God but, given the evidence or lack thereof at hand I am an atheist. So it is with woo-woo stuff. I recognize that there is a possibility for it's existance however, given the evidence and fraud (Schwartz for example) the childlike credulity of some, I believe it is ********. This does not mean that I cannot maintain an open mind. It just means that proponents need to be rather compelling.

[jj]

Originally posted by Ed
In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

Man, oh man, Ed Save Us I hope they didn't get any of THAT in the well, for sure! :roll:

[jj]

Originally posted by T'ai Chi
They are probably similar enough to be combined, to see if the combined stadardized score is near 0 and non-significant as we'd expect by chance.


"probably".

Um, dude, some of us do this kind of analysis, you know, and speaking as one such, I dare say "probably not" is a great deal more likely.

[jj]

Originally posted by Diogenes

Ahhh.. But you can dismiss the possibility that it is significant.. Like an atom of arsenic in a gallon of water..


Actually, that particular example is a wee bit significant. We can't live without trace quantities of arsenic, you know, even if it is also a heavy metal poison... :D

Sorry, allow me to sneak away now:

Pedant, Pedant
Pedant
Pedant pedant pedant

(think pink panther)

[gnome]

In proper statistics, if you believe the effect you are testing for has not been found because the sample size is too small, you resolve that not by pooling the results of already-done tests, but by performing a NEW test with the correct sample size.

Anything else is cooking the books. Seriously, what if I set about doing this and found out that the results of the last 100 tests were statistically insignificant, but the results of the last 50 tests are significant. I'd be awfully tempted to write my paper highlighting the last 50 tests, if I were biased towards the hypothesis. I might even do so without consciously meaning to be deceitful.

The only way to be sure this is not happening (by unconscious bias or malfeasance) is to always test again and get new data, if you intend to change the conditions.

[T'ai Chi]

Originally posted by Ed

Luci, your knowledge of stats is too poor to carry on. What is the probability of drawing 911 on a specific date again?

LOL, you hypothesizing I'm Lucianarchy?

You're simply resorting to a 'I don't know the topic so I will just name call.' tactic.

I'm very glad that all these messages are archived. :)

[T'ai Chi]

Originally posted by gnome
In proper statistics, if you believe the effect you are testing for has not been found because the sample size is too small, you resolve that not by pooling the results of already-done tests, but by performing a NEW test with the correct sample size.

Anything else is cooking the books.


I am not testing for any effect. I am simply wanting to see if the combined standardized score is near 0 and non-significant as we'd expect.

[Ed]

Maybe you are, maybe you are not. Your knowledge of applied stats is as bad as his so I thought .....

As far as staying on topic, I think that your question has been addressed, and that early on. You do see why what you are suggesting is silly, do you not?

[Ed]

Originally posted by T'ai Chi


I am not testing for any effect. I am simply wanting to see if the combined standardized score is near 0 and non-significant as we'd expect. [/B]

You might or might not expect this. You really don't get it do you? And if you get a number (which you surely will) what then? How would you interpret it? On what basis would you pool the data?

Your absence of specifics and blase assertions sure look like Luci. Combined with the stats thing, I dunno.

[Pyrrho]

Originally posted by T'ai Chi
LOL, you hypothesizing I'm Lucianarchy?

You're simply resorting to a 'I don't know the topic so I will just name call.' tactic.

I'm very glad that all these messages are archived. :)
You wouldn't know it was a skeptic board, would you? With skeptipsychics trying to divine sockpuppetness.

[T'ai Chi]

Originally posted by Ed

You might or might not expect this. You really don't get it do you? And if you get a number (which you surely will) what then? How would you interpret it?


It obviously depends on what the combined standardized score is. If it is non-significant, it is what we expect it to be by chance. If it is significant, then the experiments could be improved or there could some some small but present anamoly that would beed to be investigated further.

[T'ai Chi]

Originally posted by Ed
Maybe you are, maybe you are not. Your knowledge of applied stats is as bad as his so I thought .....


You've already addressed me as "Luci" by saying:


Luci, your knowledge of stats is too poor to carry on. What is the probability of drawing 911 on a specific date again?


so you obviously aren't thinking maybe here.

As far as staying on topic, I think that your question has been addressed, and that early on. You do see why what you are suggesting is silly, do you not?

Please, Ed, actually explain what you mean by "silly". So far no one has adequetly addressed why seeing if the combined standardized score is near 0 and non-significant as we'd expect is in any way unscientific.

[gnome]

Originally posted by T'ai Chi

I am not testing for any effect. I am simply wanting to see if the combined standardized score is near 0 and non-significant as we'd expect. [/B]

Perhaps the term "effect" is not the best word here, but it doesn't change the argument that you are testing a hypothesis (that the standardized scores over several tests will be greater than chance levels) and you must use new data.

I should modify what I've been saying, however--it has been pointed out to me IRL that such meta-analysis might be useful in designing a new test, as long as the results of the meta-analysis are not held up as significant in themselves.

[Ed]

Originally posted by T'ai Chi


You've already addressed me as "Luci" by saying:

[/b]




Please, Ed, actually explain what you mean by "silly". So far no one has adequetly addressed why seeing if the combined standardized score is near 0 and non-significant as we'd expect is in any way unscientific.
[/B]

You are not doing a meta analysis, the way it is generally understood. You are pooling data from different "experiments" and deriving some number from them. You then say "if the number is big, design a new test". That is all ok, it ain't science and you are probably violating various assumptions (homogeneity of varience, possibly) you might or might not have experiments that are comperable in terms of method and control, who knows. You are getting a bunch of numbers is all, any result is meaningless. Why would you go thru this sort of crap and not design a simple experiment with adequite sample in the first place. I'll suggest a reason. A well controlled experiment will never be done when you can draw specious conclusions from crappy data and thus muddy the water. That is, I am afraid, SOP for paranormal research.

So go right ahead and z-score away, it is in all probability, meaningless.

That is what I meant by silly. Something out of the Ministry of Irreproduceable Results, headed by John Cleese.

[Ed]

Originally posted by Pyrrho

You wouldn't know it was a skeptic board, would you? With skeptipsychics trying to divine sockpuppetness.

You referring to me or the Luci entity that I am in communication with. If not a sock then Channeling, fer sure ... Could Luci have died and come back? How ironic.

[T'ai Chi]

Originally posted by Ed

You are not doing a meta analysis, the way it is generally understood. You are pooling data from different "experiments" and deriving some number from them. You then say "if the number is big, design a new test".


I'm not doing any test at all. I am simply discussing the possibility and the interpretation of the results from such a test if one were to occur.

If I did it, if the combined standardized score was significant, then I'd say there is something that we need to look into, because we'd expect it to be non-significant.


That is all ok, it ain't science ..


I don't think you much or any justification for saying that. The discipline of Statistics is essentially the method of the scientific method.


..and you are probably violating various assumptions

..you might or might not have experiments that are comperable in terms of method and control,


I agree 100%. We don't know for sure, certainly, without seeing the JREF database of tests (and/or other skeptical organizations that test), of course.


You are getting a bunch of numbers is all, any result is meaningless.


If a sound meta analysis is carried out, the results are anything but meaningless. Why do you think meta analyses are done in the first place. I disagree 100% with the statement that "any result is meaningless". If the combined standardized score is significant, that would be meaningful.


Why would you go thru this sort of crap and not design a simple experiment with adequite sample in the first place.


JREF, and other skeptical organizations, already did the tests.


I'll suggest a reason. A well controlled experiment will never be done when you can draw specious conclusions from crappy data and thus muddy the water. That is, I am afraid, SOP for paranormal research.


From what I've read, there have been soundly designed experiments in anomalous mental phenomena with repeatable results. Also, Honorton, I believe, examined the common claim against psi research of 'poor experimental design led to the better scores' and found there to be no significance in that relationship.


That is what I meant by silly. Something out of the Ministry of Irreproduceable Results, headed by John Cleese.

Everyone is entitled to their opinion.

[FutileJester]

Originally posted by T'ai Chi
The discipline of Statistics is essentially the method of the scientific method.

The axioms of the scientific method don't depend on statistics. Statistics are mathematical tools that support the practical pursuit of science. Regardless, the question is not whether or not stats are important, but whether or not this particular application of statistics should be expected to produce useful data.

If a sound meta analysis is carried out, the results are anything but meaningless. Why do you think meta analyses are done in the first place. I disagree 100% with the statement that "any result is meaningless". If the combined standardized score is significant, that would be meaningful.

Standardized score of what measure? Let's limit ourselves to dowsers for simplicity. What measure could we analyze and what information would be suggested by various scores? Is there a concrete example along the lines of "the score for measure X was much greater than Y, which suggests Z" that you can give?

I'll admit to some ignorance here, but it seems to me that 'useful' meta-analysis involves more than the principle measure of a test. The original aspirin studies presumably measured pain relief; but of course many measures are taken on participants in a medical study. The meta-analysis was able to show a connection between aspirin use and heart attacks because of these adiitional measures.

AFAIK we don't have any other additional measures for the JREF data. If we did it's possible some interesting areas for study could be revealed. For example, it could be found that out of all categories of claimants, dowsers lived perceptibly longer. Or that clarivoyants have the strongest religious convictions. But without this additional data, we're basically stuck with hits. And I don't yet see any way that meta-analyzing the hits would be suggestive of anything concrete enough to focus a new study on.

[slimshady2357]

Hi Whodini! :w2:

Adam

[Ed]

Originally posted by FutileJester


The axioms of the scientific method don't depend on statistics. Statistics are mathematical tools that support the practical pursuit of science. Regardless, the question is not whether or not stats are important, but whether or not this particular application of statistics should be expected to produce useful data.



Standardized score of what measure? Let's limit ourselves to dowsers for simplicity. What measure could we analyze and what information would be suggested by various scores? Is there a concrete example along the lines of "the score for measure X was much greater than Y, which suggests Z" that you can give?

I'll admit to some ignorance here, but it seems to me that 'useful' meta-analysis involves more than the principle measure of a test. The original aspirin studies presumably measured pain relief; but of course many measures are taken on participants in a medical study. The meta-analysis was able to show a connection between aspirin use and heart attacks because of these adiitional measures.

AFAIK we don't have any other additional measures for the JREF data. If we did it's possible some interesting areas for study could be revealed. For example, it could be found that out of all categories of claimants, dowsers lived perceptibly longer. Or that clarivoyants have the strongest religious convictions. But without this additional data, we're basically stuck with hits. And I don't yet see any way that meta-analyzing the hits would be suggestive of anything concrete enough to focus a new study on.

Luci simply wants to lump data to muddy the water.

[T'ai Chi]

Originally posted by Ed

Luci simply wants to lump data to muddy the water.

Ed, you are on ignore now for obvious reasons. Unfortunately, you resorted to absurd claims of me being Luci.

If I am Luci, (I'm not), that has no bearing at all on the topic of meta analysis anyway, something which all skeptical thinkers should obviously know.

[thaiboxerken]

Originally posted by T'ai Chi


If I am Luci, (I'm not), that has no bearing at all on the topic of meta analysis anyway, something which all skeptical thinkers should obviously know.

True, but there is no reason to keep trying to reason with you, as you have completely ignored ALL relevant points as to why a meta-analysis could not be done.

So, we may as well call you names, Whodini.

[Ed]

Originally posted by T'ai Chi


Ed, you are on ignore now for obvious reasons. Unfortunately, you resorted to absurd claims of me being Luci.

If I am Luci, (I'm not), that has no bearing at all on the topic of meta analysis anyway, something which all skeptical thinkers should obviously know.

Of course it does. It means that rational discusion is not possible.

Notice how you are Luci-like in ignoring every comment that is critical and keep up with the tedious mantra of standardized scores without the slightest notion of what you are talking about. Keep ignoring the questions, just keep on repeating.

[Ladewig]

This thread has gone on for long enough.

T'ai Chi, if you want to analyze the data, then go to the JREF headquarters and analyze the data. Performing that type of meta-analysis is outside the stated purpose of the The Foundation. Furthermore, even it were something within the Foundation's charter, the funds are too limited to devote to a task that might give a result other than zero.

Lastly, I would suggest that before you book an airline ticket, ask yourself, "why do I think there are enough quantifiable tests in the JREF history that trends will be visible above the noise."

[T'ai Chi]

Originally posted by Ladewig
This thread has gone on for long enough.


Perhaps because people reply? ... ;)


T'ai Chi, if you want to analyze the data, then go to the JREF headquarters and analyze the data.


The fact that I am merely suggesting the idea, has nothing to do with me going to Florida, and the fact that I am not personally doing the analysis has nothing to do with discussing the idea of doing one.

It would be nice if the data was publicly available (without specific names of people involved, of course) though, then we could all analyze and compare our results.


Performing that type of meta-analysis is outside the stated purpose of the The Foundation.


Sure, I agree with that. That is not the purpose of the JREF.

Question: Is it outside of the stated purpose of "The Foundation" to allow outsiders/independent investigators access to their dataset?

[T'ai Chi]

Originally posted by FutileJester

The axioms of the scientific method don't depend on statistics.


It would have been more proper of me to say that the 'scientific method' and the disciplines of Statistics have many overlapping areas: inference, hypothesis testing, quantitative, experimental design, issues about bias, randomization, models, etc.


Standardized score of what measure?


Something like: number of times correct idendtifying the film cannister that has gold in it, or identifying the pipe that has water flowing through it.


If we did it's possible some interesting areas for study could be revealed. For example, it could be found that out of all categories of claimants, dowsers lived perceptibly longer. Or that clarivoyants have the strongest religious convictions. But without this additional data, we're basically stuck with hits. And I don't yet see any way that meta-analyzing the hits would be suggestive of anything concrete enough to focus a new study on.

Perhaps we could look at score by gender, or score by age, or score by time spent learning dowsing, for starters. I'm personally not too interested in relating their scores to any other variables (although it is a very interesting area to explore). I'm just interested in seeing if the combined standardized score is near 0 and non-significant as we'd expect it to be.

[Lucianarchy]

T'ai, do you have any evidence that the data you refer to actually exists?

[Ed]

Hmmmm..... Talking to yourself, Luci?

[FutileJester]

Originally posted by T'ai Chi
Something like: number of times correct idendtifying the film cannister that has gold in it, or identifying the pipe that has water flowing through it.

Okay, so basically we're analyzing hits, however they are defined for an individual test. I still don't see what useful results we can derive from an analysis of hits. To re-ask an unanswered question from before, can you give an example along the lines of "the score for measure X was much greater than Y, which suggests Z"? Or specifically for this case, "the standardized score for dowsing hits was X, therefore Y".

The purpose of a meta-analysis can't be just to wave our hand in the air afterwards and say "look we found something". It should suggest correlations not originally seen which can then be separately tested in their own controlled studies. What studies, in principle, could be suggested by the results of analyzing dowsing hits?

Perhaps we could look at score by gender, or score by age, or score by time spent learning dowsing, for starters. I'm personally not too interested in relating their scores to any other variables (although it is a very interesting area to explore).

I agree, probably not much interesting to be found there. My point is that these types of studies, looking for correlations between measures that are available but which were not the principle concern of the study, seem much more likely to provide usable results. I doubt enough data of that sort is consistently available in the trial data in any case.

[T'ai Chi]

Originally posted by Ed
Hmmmm..... Talking to yourself, Luci?

Wow, quite abnormal behavior for a skeptic...

You might want to rethink your strategy, as I can assure you, that Lucianarchy and I are quite different people entirely.

It seems like you are simply acting based on your deep-set beliefs, so carry on.

[Ed]

Originally posted by T'ai Chi


Wow, quite abnormal behavior for a skeptic...

You might want to rethink your strategy, as I can assure you, that Lucianarchy and I are quite different people entirely.

It seems like you are simply acting based on your deep-set beliefs, so carry on.

Address some of the issues raised on your silly thread. Luci would never do that. That will be proof.

[T'ai Chi]

Originally posted by Ed


Address some of the issues raised on your silly thread. Luci would never do that. That will be proof.

So you are admitting that you don't have proof but yet you already made the claim of 'T'ai Chi = Lucianarchy'???

That is not skeptical at all I would say.

Excuse away.

[Lord Kenneth]

Originally posted by Ed


Address some of the issues raised on your silly thread. Luci would never do that. That will be proof.

Ta'i Chi is Whodini, not Luci.

[FutileJester]

Ta'i Chi, I know you have no obligation to jump through hoops when I ask a question, but it's frustrating to have you engage me in a discussion and then ignore my questions about your viewpoint. I think I'm giving you much more benefit of the doubt here than most others, but that's a hard attitude to maintain when you take time to answer personal slights but won't answer (repeatedly) good questions about the topic.

For the third time, and for the benefit of those of us who do not know as much about meta-analysis as you claim to, could you please answer the following questions:


What kind of conclusions could we, in principle, make based on the meta-analysis? Looking for statements along the lines of "the standardized score for dowsing hits was X, therefore Y".
What further studies could be suggested by the results of analyzing dowsing hits?


If we can't answer these ahead of time, then I can't help but agree with the conclusion others have already reached - it's just a fishing expedition.

[T'ai Chi]

Originally posted by FutileJester

For the third time, and for the benefit of those of us who do not know as much about meta-analysis as you claim to, could you please answer the following questions:


Sorry FJ, I get distracted from the good questions by the people who insult, etc.

We could make the conclusion of: 'the combined standardized score was significant or nonsignificant.'

This could lead to possibly finding ways to improve the tests.

[FutileJester]

Originally posted by T'ai Chi
We could make the conclusion of: 'the combined standardized score was significant or nonsignificant.'

I know that finding a significant score means that the score is significant. :rolleyes: But what could it mean if the score is significant?

This could lead to possibly finding ways to improve the tests.

Such as? I know meta-analysis is a tool to suggest future tests, I've said so myself. The question is, which improvements or new studies could in principle be suggested by the analysis you're proposing?

My point with the earlier examples involving secondary variables was that meta-analysis using these variables makes sense, since we can easily imagine a result and what tests that result would imply. For instance, a result might be that claiming to be a dowser is correlated with strong religious conviction. We can then easily design a test that collects data about dowsing claimants and religious beliefs.

I can't think of any results from analyzing dowsing hits that would suggest a specific test or improvement. But I'm the first to admit that I don't know a lot about this, which is why I'm asking. Is there a hypothetical but specific result that would imply a specific test or improvement?

[Ed]

Originally posted by Lord Kenneth


Ta'i Chi is Whodini, not Luci.

You sure?

[T'ai Chi]

Originally posted by Ed


You sure?

Why get an opinion from a 3rd party... Dontcha trust me? I trust you.

Even though I know you are really Luci.


*wink*
:) :) :)

[T'ai Chi]

Originally posted by FutileJester

I know that finding a significant score means that the score is significant. :rolleyes: But what could it mean if the score is significant?


It is hard to say for sure, FJ, without actually examining the data. So, I don't know. :(

[FutileJester]

Originally posted by T'ai Chi
It is hard to say for sure, FJ, without actually examining the data. So, I don't know. :(

But don't you see that this is a fatal flaw in your proposed analysis? With other proposed (but uninteresting to us) analyses we could easily imagine ahead of time possible results.

You pointed out earlier the parallels between statistics and science; in this vein, doing a meta analysis without anticipating possible conclusions is like doing an experiment with no falsifiable hypothesis. It's fishing. Painful experience has shown that it just doesn't lead to reliable results.

In fact this is a principle I use at work - never run a test if you don't understand what the results will mean. I've seen (slightly) younger engineers go through a cycle of getting confused, running a test they don't really understand, and consequently getting more confused. Contrast this with the more experienced engineers, whose most stinging criticism of a propsed test is, "But what will that tell us?" If you can't answer that to their satisfaction, they won't run the test. Experience says it will be a waste of time.

[thaiboxerken]

Originally posted by Ed


You sure?

I'm sure, this is from another thread.

http://www.randi.org/vbulletin/showthread.php?s=&postid=1870085040#post1870085040


Posted by: Thaiboxerken
I didn't like you when you posted as Who, and I still don't like you. Go meditate on that.



Response from T'ai Chi:

Well I like you! *smooch* So there.

I will go meditate though. Thanks!

No mention of "I'm not Who" in there anywhere.

Damn trolls. Who, you said you were done with the forum, why did you come back?

[T'ai Chi]

no mention of "I'm not Who" in there anywhere.


Um, that is because I tend to ignore responding to childish jabs.

Except this one. ;)