Little Evidence for Effectiveness of Scientific Peer Review

[zakur]

Little Evidence for Effectiveness of Scientific Peer Review (http://bmj.com/cgi/content/full/326/7383/241/a)

Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research, concludes a systematic review from the international Cochrane Collaboration.

Yet the system, which has been used for at least 200 years, has only recently come under scrutiny, with its assumptions about fairness and objectivity rarely tested, say the review authors. With few exceptions, journal editorsand cliniciansaround the world continue to see it as the hallmark of serious scientific endeavour.

Published last week, the review is the third in a series from the Cochrane Collaboration Methods Group. The other reviews look at the grant application process and technical editing.

Only the latter escapes a drubbing, with the reviewers concluding that technical editing does improve the readability, accuracy, and overall quality of published research.

The Cochrane reviewers based their findings on 21 studies of the peer review process from an original trawl of only 135. These were drawn from a comprehensive search of biomedical print and online databases, and information received from bodies such as the World Association of Medical Editors.The full-text Cochrane Group report:

Editorial peer-review for improving the quality of reports of biomedical studies (http://www.update-software.com/Cochrane/MR000016.pdf)
Jefferson TO, Alderson P, Davidoff F, Wager E

[arcticpenguin]

Who in Ed's name is "The International Cochrane Collaboration"? What rating do they have from the Imperial Arcticpenguin Commission?

Almost half of the available research focused on the effects of concealing the identity of reviewers and/or authors, which, the Cochrane authors conclude, has little impact on quality. Few studies assessed the impact of peer review on the importance, usefulness, relevance, or quality of research. Only one small study tested the validity of the peer review procedure itself.

On the basis of the current evidence, "the practice of peer review is based on faith in its effects, rather than on facts," state the authors, who call for large, government funded research programmes to test the effectiveness of the system and investigate possible alternatives.

[BillHoyt]

Originally posted by arcticpenguin
Who in Ed's name is "The International Cochrane Collaboration"? What rating do they have from the Imperial Arcticpenguin Commission?



AP,

They're part of a growing evidence-based-medicine movement. I don't know many details, but they can be found at www.cochrane.org

Cheers,

[wollery]

Can't speak for the biomedical sciences, but peer review certainly makes a difference in astronomy/astrophysics. To see that you just have to look at the number of papers that appear on the astro-ph website (which allows papers on before they are reviewed) that don't make it into a peer reviewed journal.

That's not to say that every paper that gets published is good or advances the field. Just that bad science and erroneous results get filtered out, which is the point of peer review. I've read numerous papers which say little or nothing that's new, in fact really good papers that contain new ideas or groundbreaking results are pretty rare. The vast majority just add a tiny bit to the existing knowledge.

Without peer review people would be free to write whatever they fancied and, since funding largely depends on the number of published papers, they probably would.

[hammegk]

Hmm, I believe this fits right here.

Myth #2: Science is a self-correcting system.

\corollary An integral part of this self-correcting system is the peer review process.

Open scrutiny by a critical community is an important part of science (perhaps the most important), and peer review is a vital part of this process. So this is not so much a myth, as warning against placing blind faith in peer review.

Consider this study by Michael Mahoney.8 He prepared two papers that were identical in methodology, but different in which theory the results supported. He sent these papers out for review by reviewers who had earlier expressed support for either the theory supported by the experimental results, or the theory refuted by the experimental results.

Mahoney found two things: First, reviewers were more likely to reject papers that did not support the theory they favored. Second, reviewers were on average more critical of the methodology in the papers that did not support their prior views---even though the methodologies were identical. That is, the reviewers in Mahoney's study held conflicting theories to a higher standard.

http://www.rpi.edu/~sofkam/papers/skeptik.html

[daver]

I'm not sure that holding conflicting theories to a higher standard is a bad thing. Presumably there's a fair amount of evidence for the existing theory; experiments that support that theory are less surprising. This is a variant of extraordinary claims requiring extraordinary proof.

[apoger]

Peer review is certainly useful.

Is there a process that can replace peer review and provide better results? Possibly. If these people can convince the community to abandon peer review for a better process, then more power to them.

Constant challenges leading to improvments is the hallmark of science. If the methodology of science can be improved, good!

On the other hand starting with a premise of "Little evidence for" isn't going to get applause in most circles. If they want to make arguments for changing the system then they will have to present massive evidence that the new method is superior.

[jj]

Originally posted by hammegk
Hmm, I believe this fits right here.



http://www.rpi.edu/~sofkam/papers/skeptik.html

Now I understand what kind of troll you are. You just don't like anything, and you're happy to tear down.

Please tell me what testable, falsifiable, verifiable, repeatable things you build up, ok?

To the first point, peer review works reasonably well in the signal processing area. It's not perfect, but we manage to avoid most duplication and most wrongness. What doesn't get caught in the first shot at reviews does get caught later, the readership is rather enthusiastic about that.

[JamesM]

Originally posted by apoger
If they want to make arguments for changing the system then they will have to present massive evidence that the new method is superior.

Unfortunately, they don't have an alternative system, they merely looked at biomedical studies that evaluated certain tweakings of the peer review system (e.g. having reviewers sign themselves, blinding reviewers and so on).

The BMJ's description of peer review getting a "drubbing" is a bit misleading as the study didn't say the peer review system was worse than having no system, just that there was no evidence to support the scientific community's "faith" in it working well.

What I want to know is, was the Cochrane Library paper peer-reviewed? And if it was, does this mean we should accept/reject its findings?

[hammegk]

Originally posted by jj


Now I understand what kind of troll you are. You just don't like anything, and you're happy to tear down.


Well, doggone it, you think I'm a troll. :( Such is life.

I know I'm a skeptic, and we wouldn't want any of them here would we -- unless of course they agree with you, perhaps?

[jj]

Originally posted by hammegk

Well, doggone it, you think I'm a troll. :( Such is life.

I know I'm a skeptic, and we wouldn't want any of them here would we -- unless of course they agree with you, perhaps?

Yeah, yeah, you're a skeptic. You "don't believe" some things.

On the other hand, let's hear your actual problems with peer review, not your pointing to somebody else's problems, for instance.

Let's say, perhaps, how the article misses the point that errors are also caught after publication.

Let's say, perhaps, how it is entirely reasonable that very unusual theories require more proof.

Let's say, perhaps, how does this paper, working in a field where the scientific method isn't applied as well as it might be, indeed, apply to, say, an IEEE Journal?

In other words, your attack above was arranged to have a sense of universality.

Now justify that.

[hammegk]

jj, does shooting the messenger usually help?

No, I cannot discuss the effectiveness -- or not -- of IEEE reviews. I can suggest a hypothetical which may be far from the mark, but anyway, here goes.

Let's see if we can postulate some interesting effect I'd hope IEEE has some interest in; say prediction of electron tunneling. If someone proposed an approach based on meta-analysis that seemed at odds to whatever the theory dujour might be, would it be treated diiferently than PEAR results?

I do not propose I believe that PEAR demonstrates anything and what little I know of it suggests it may well be seriously flawed. However there seems to be a gut-level non-scientific reaction as much as a scientific rebuttal.

Do you propose that no one could possibly ask a question in your fields of expertise unless their expertise matches or exceeds your own? I don't think so, fwiw.

[BillHoyt]

Originally posted by hammegk
Let's see if we can postulate some interesting effect I'd hope IEEE has some interest in; say prediction of electron tunneling. If someone proposed an approach based on meta-analysis that seemed at odds to whatever the theory dujour might be, would it be treated diiferently than PEAR results?
But exactly these discussions do occur and have occurred in many other professional journals, sir. Meta-analysis is a dicey affair. BMJ, as just one example, has dedicated much space to the argument over its scientific value. Furthermore, the contoversial nature of meta-analysis is not discussed only when results differ from expectations. Neither has it much to do with who is doing the analysis. The tool itself is dicey. It has been seriously abused by those who don't understand its many inherent weaknesses.

Cheers,

[gnome]

It seems reasonable to me that supporters of peer review must allow the peer-review process itself to be subject to scrutiny and questioning. :D

On the other hand, I suspect it will be a long time before someone comes up with an alternative that doesn't have worse problems.

Perhaps the process will be slowly refined over time, with occasional "leaps", as science itself is.

Originally posted by BillHoyt

But exactly these discussions do occur and have occurred in many other professional journals, sir. Meta-analysis is a dicey affair. BMJ, as just one example, has dedicated much space to the argument over its scientific value. Furthermore, the contoversial nature of meta-analysis is not discussed only when results differ from expectations. Neither has it much to do with who is doing the analysis. The tool itself is dicey. It has been seriously abused by those who don't understand its many inherent weaknesses.

Cheers,

Dear BillHoyt,

There is nothing wrong with carrying out a meta-analysis. It is a perfectly valid statistical technique. There might be objections on non-statistical grounds though.

Sincerely,

S. Holmes

[jj]

Originally posted by gnome
It seems reasonable to me that supporters of peer review must allow the peer-review process itself to be subject to scrutiny and questioning. :D


Which you can count on to happen, at least in the fields I see papers in. I've had authors argue with me, other reviewers, the editor, reviewers argue with the editor, each other, etc.

At least somewhat often, something comes of it.

Sometimes, though, something was just wrong.

Both happen.

[jj]

Originally posted by Sherlock Holmes


Dear BillHoyt,

There is nothing wrong with carrying out a meta-analysis. It is a perfectly valid statistical technique. There might be objections on non-statistical grounds though.

Sincerely,

S. Holmes

Exactly WHAT do you think "Meta-analysis" is, or has somebody usurped this one to mean something else, too?

[BillyJoe]

It seems to me that doing Peer Review is the only way to do science so therefore we'd better damn well do Peer Review properly if we're going to do science properly. The article is pointing out that we are not doing it properly in the field of medical research not that we should abandon it as a useless tool.

[BillyJoe]

Originally posted by jj
Exactly WHAT do you think "Meta-analysis" is....? That would be the lumping together of fatally flawed with methodologically sound clinical trials, calculating the average and pretending you have a result that is, in any way, meaningful.

:(

[hammegk]

Originally posted by BillyJoe
That would be the lumping together of fatally flawed with methodologically sound clinical trials, calculating the average and pretending you have a result that is, in any way, meaningful.

:(

That defines the problem away, but does the problem itself also disappear? How do we determine fatally flawed.?Would it be-- peer review? The subject we are discussing the effectiveness of. :eek:

Can you say, circular? :D

But I would agree it's statistics drawn from multiple studies. ;)

[BillyJoe]

Originally posted by hammegk
That defines the problem away, but does the problem itself also disappear? What problem?
What is disappearing?
I'm merely giving a cynical definition of "meta-analysis".

Originally posted by hammegk
How do we determine fatally flawedAw, yer know, when they forget to double-blind or use placebos or forget to acount for demographics.....
Stuff like that.

Originally posted by hammegk
Would it be-- peer review? The subject we are discussing the effectiveness of. Yeah, yer know, that useless stuff listed above.

Originally posted by hammegk
Can you say, circular? Yes, but will I? ;)

Originally posted by hammegk
But I would agree it's statistics drawn from multiple studies.........averaged over methodologically sound AND fatally flawed clinical trials. ;)

[hammegk]

How many scientists does it take to change a lightbulb?










































None. They use them for double-blind studies.

[BillyJoe]

How many light-bulbs does it take for you to see the light?

[hammegk]

Several if I'm trying to find something meaningful from you, here.

[BillyJoe]

Keep trying then, you never know one day you might.....see......the......

[hammegk]

;)

Originally posted by jj


Exactly WHAT do you think "Meta-analysis" is, or has somebody usurped this one to mean something else, too?

Dear jj,

The subject of meta-analysis is brought up, and you ask if I know its definition rather than discussing the actual issues with meta-analysis.

Consider for a moment that people other than the holy who contribute to IEEE journals know what it means.

Awaiting the green-cheese,
Sincerely,

S. Holmes

Originally posted by BillyJoe
[meta-analysis] would be the lumping together of fatally flawed with methodologically sound clinical trials, calculating the average and pretending you have a result that is, in any way, meaningful.


Dear BillyJoe,

Your description of meta-analysis completely missed the target. This is where you need to stop reading scoffer's comments for a moment and read accounts of what meta-analysis actually is.

A meta-analysis is the combining of results (effects) from similar experiments (similar design, measured in the same way, and some other criteria) to get an overall effect. This is a completely valid statistical technique.

Many believers go one way and say that the meta-analysis proves something is real, and the scoffers go the other way and say that it proves nothing and even that it is a flawed technique. Both groups are seriously mistaken and lack experience.

Sincerely,

S. Holmes

[BillyJoe]

Originally posted by Sherlock Holmes
Your description of meta-analysis completely missed the target...... A meta-analysis is the combining of results (effects) from similar experiments (similar design, measured in the same way, and some other criteria) to get an overall effect. I will have a look but I don't think you are correct.

At the very least, there are different ways of performing meta-analyses which probably represent attempts to get results that are meaningful!

However, I think you would find it difficult to find a sufficiently large collection of sufficiently similarly conducted clinical trials to make a sufficiently meaningful meta-analysis possible. ;)

[jj]

Originally posted by Sherlock Holmes


Dear jj,

The subject of meta-analysis is brought up, and you ask if I know its definition rather than discussing the actual issues with meta-analysis.

Consider for a moment that people other than the holy who contribute to IEEE journals know what it means.

Awaiting the green-cheese,
Sincerely,

S. Holmes

Mr. "Holmes", if we are to engage in a dialog, we must know what YOU think it is.

IF you won't tell us, we can't communicate.

Now, I'm sure you'll try to slide down the side of a definition if we don't pin you down, so let's get on with it. On the other hand, I'm afk a week, so we can do this later, if you like.

Now, I know several kinds of meta-analysis. Some are good, some are bad, some are silly. Some are in fact involving metalogical thinking, some are simply misnamed.

But you won't tell me which one you mean.

Why?

[jj]

Originally posted by Sherlock Holmes

Your description of meta-analysis completely missed the target. This is where you need to stop reading scoffer's comments for a moment and read accounts of what meta-analysis actually is.

A meta-analysis is the combining of results (effects) from similar experiments (similar design, measured in the same way, and some other criteria) to get an overall effect. This is a completely valid statistical technique.


Ahh, so you're talking about large-sample statistical analysis, combining incommensurate samples.

Thank you. Dismissed. You may reapply later if you wish.

Originally posted by jj


Ahh, so you're talking about large-sample statistical analysis, combining incommensurate samples.

Thank you. Dismissed. You may reapply later if you wish.

Dear jj,

Why am I "dismissed"? Notice that I also said combining of studies where things were set up and measured in the same or similar way, so I don't know where you got "incommensurate" from.

Sincerely,

S. Holmes

[BillHoyt]

Originally posted by Sherlock Holmes


Dear BillHoyt,

There is nothing wrong with carrying out a meta-analysis. It is a perfectly valid statistical technique. There might be objections on non-statistical grounds though.

Sincerely,

S. Holmes

Might I suggest you read the BMJ discussions on meta-analysis? The objections are on both statistical and non-statistical grounds.

Cheers,

[Quasi]

Hey,

I suggest you read some Karl Popper, especially what he said about circular reasoning and metaphysical theories which cannot be tested (hint, they are completely worthless.) This is why science works, you can actually do stuff and predict it.