Credible sources = Credible Information?

[jayman]

I'm with the Cleveland Skeptics meetup group and The Cleveland Freethinkers. We have a mailing list and have discussions/debates. Here is a thread a guy posted recently and I wanted to get everyone's thoughts on it.

Thanks,
Josh



http://www.sciencenews.org/view/generic/id/41827/title/Ouch!_Way_worse_than_plagiarism

Ouch! Way worse than plagiarism

A little over a week ago I wrote a two-parter on software that has uncovered hundreds of instances of apparent plagiarism in biomedical science. Copycatting someone else’s work is lazy at best; more likely it’s just amoral. But the current issue of Anesthesiology News highlights an even more egregious type of fraud: blatant fabrication of medical data.

In two investigative news stories, Adam Marcus describes the case against anesthesiologist Scott S. Reuben. This prominent Massachusetts pain researcher is accused of faking data that served as the basis for a minimum of 21 published medical studies. At least plagiarists “borrow” data that are ostensibly real and therefore might have some medical validity. Fabricated data benefit no one but the author who is looking to bolster his reputation by fattening his portfolio of published studies.

<snip>

People always point to peer review as the gold standard for vetting research — confirming that it is not only important but also solid. As the Reuben incident points out, peer review is far from perfect. Reviewers assume their colleagues won’t cheat to get their name in print. In this case, that’s proved to be a pervasive and dangerous assumption.

------------------------------------------------------------------------------

Like I was saying...
True information can come from sources that are 'not credible' and likewise false information can come from 'credible' sources. It is the actual information that must be critically analyzed. Blanket denial or acceptance of information based on whether the source is 'credible' or not is not a logically valid argument.

[Jozen-Bo]

I'm a little lost here? The focus of this thread is inquiring whether or not we should trust all sources of information dubbed credible?

[paximperium]

Like I was saying...
True information can come from sources that are 'not credible' and likewise false information can come from 'credible' sources. It is the actual information that must be critically analyzed. Blanket denial or acceptance of information based on whether the source is 'credible' or not is not a logically valid argument.

Tim

Is it solipsistic in here... or is it just me?" It's not solipsistic at all, it is the standard skeptical and scientific attitude. Always validate and reproduce the study in question before accepting its findings especially if it is novel or makes no sense.

Science and medicine is very conservative for a good reason. The evidence must be overwhelming before it is widely accepted.

[jayman]

I'm a little lost here? The focus of this thread is inquiring whether or not we should trust all sources of information dubbed credible?

Yes.

[jayman]

I'm a little lost here? The focus of this thread is inquiring whether or not we should trust all sources of information dubbed credible?

It's not solipsistic at all, it is the standard skeptical and scientific attitude. Always validate and reproduce the study in question before accepting its findings especially if it is novel or makes no sense.

Science and medicine is very conservative for a good reason. The evidence must be overwhelming before it is widely accepted.

Sorry, that solipsistic quote is something the guy signs off on all of his e-mails.

[desertyeti]

Yes.

Then no.

[Jozen-Bo]

Then, though a source be credible, that does not make certain that it is moral and that it could forge in order to profit, I have to wonder how much of this sort of thing goes on in politics?

[athon]

I think the problem lies in this strawman:

People always point to peer review as the gold standard for vetting research — confirming that it is not only important but also solid. As the Reuben incident points out, peer review is far from perfect. Reviewers assume their colleagues won’t cheat to get their name in print. In this case, that’s proved to be a pervasive and dangerous assumption.

It's a frequent point that gets brought up, and is quite simply wrong. Just because it is in a journal, it isn't taken to be irrefutable or even necessarily correct. It is simply found to be internally consistent.

The gold standard of science involves more than just publishing. That simply puts an idea out there for discussion. The media then jumps on novel ideas as if they've been 'proven', sensationalising the process beyond its limits.

Ideally, once a conclusion has been published, the idea is for somebody else to find it interesting enough to attempt to repeat it. After that, more variables are tweaked...and science slowly moves on.

The science methodology is not a punctuated equilibrium. Every other week, papers are published with radical new ideas that never make it much further. The real test is if they can be replicated faithfully. Journals don't decide the truth - they simply decide if what is written is consistent with practice and philosophy.

Athon

[Kevin_Lowe]

Published cientific papers aren't infallible oracles. They're just worthy of a higher degree of belief than unreviewed studies, anecdotes, personal opinions and whatnot.

[bpesta22]

What matters immensely is the reputation of the journal. Some are absolute crap; others are highly elite. I think you'd be far less likely to find this stuff in an elite journal.

I'd say the best predictor of how "good science" an article is would be the reputation of the journal that published it.

[blutoski]

I'm with the Cleveland Skeptics meetup group and The Cleveland Freethinkers. We have a mailing list and have discussions/debates. Here is a thread a guy posted recently and I wanted to get everyone's thoughts on it.

Well, it's a strawperson argument.

He's saying because peer-review isn't perfect at detecting fraud, that peer-review shouldn't be considered the last word in whether a finding is true.

He's correct.

This is why scientists don't consider peer-review to be the last word in whether a finding is true. As Athon points out, independent replication is another important aspect.


The problem is that he seems to implying that because peer-review is not perfect, it has no value, which is not true. He is correct that it is inappropriate to assume that a peer will not fabricate information.

However, this is not a problem specific to peer-review. All publications have this risk. If he believes that the possibility of fabrication is a reason to ignore articles, it follows that we should never waste time reading anything that has been published in any format. I can't imagine how science would progress in such a world.

Like I was saying...
True information can come from sources that are 'not credible' and likewise false information can come from 'credible' sources. It is the actual information that must be critically analyzed. Blanket denial or acceptance of information based on whether the source is 'credible' or not is not a logically valid argument.

"Blanket" (deductively) no; but inductively, credibility matters. Laypersons usually can't evaluate the papers, so need to depend on experts' reputations.

See: [Skeptical MythConceptions - part 1 - Authority (http://blog.bcskeptics.info/?p=7)]

[Cuddles]

What matters immensely is the reputation of the journal. Some are absolute crap; others are highly elite. I think you'd be far less likely to find this stuff in an elite journal.

I'd say the best predictor of how "good science" an article is would be the reputation of the journal that published it.

Unfortunately, that's not necessarily the case. While plagiarism may be possible to detect, when you have hundreds of thousands of papers to check against it becomes rather difficult, especially if that includes ones that aren't available online and that you may not have a subscription to. And simply rewriting something in your own words would be enough to avoid almost any cursory examination.

As for fraud, that's not even possible to detect in principle. For example, when I was in school we had to perform titrations in chemistry. In one particular experiment I accidentally knocked a beaker over near the end of the class and ruined the whole thing. Since there wasn't time to start again, I just made up some numbers that I knew were close to the expected ones. I got the best mark in the class. And this was in a class of 20 or so other students under the supervision of a teacher. Now move that to a private laboratory with no-one constantly keeping an eye on you. As long as you say that you performed a particular experiment, how is anyone to know different? Especially a reviewer who may never have heard of you or even visited the country you're in?

Reviewers don't just assume that other people won't cheat out of naivety, they simply have no way of knowing if a person is cheating, in the vast majority of cases at least. However, as has already been said, this isn't a "pervasive and dangerous assumption" because publication is just a small part of the scientific process.

One thing I find strange is that every time I see this being brought up, the person doing so seems to think it is a shocking new revaltion that no-one would have thought possible. It isn't. Cheating and fraud in science has been around for as long as cheating and fraud everywhere else. The Korean cloning guy (I forget his name), cold fusion, Piltdown man, Z-rays, and the list goes on. And yet here we are, with more advanced science and technology than ever before. Maybe the system could be improved in some ways, but it's a hell of a long way from useless.

However, this is not a problem specific to peer-review. All publications have this risk. If he believes that the possibility of fabrication is a reason to ignore articles, it follows that we should never waste time reading anything that has been published in any format.

And therefore by his own logic we shouldn't bother reading his article.:)