File Drawering on Psych Med Studies

[Dancing David]

I just heard on NPR that a study has been published which would indicate that studies which are positive of psychiatic medication usage are published at a rate of 90% as compared to studies which are negative of use which are published at 8%.

So there is a definite bias in publishing research that supports the use of medication.

:(

[Paul C. Anagnostopoulos]

Man, that's some bulging file drawer if it's true.

~~ Paul

[Garrette]

I was listening to the same story. It surprised me only in that I thought someone on this forum had said that Big Pharma has to pre-register all med trials to avoid this.

[Dancing David]

Man, that's some bulging file drawer if it's true.

~~ Paul


Not really, I think that the same story also said the number of negative reports is less than the positive, I will have to wait until they post it.

The main thing i want to know is how they measured the 'effect' versus placebo.

They probably used a really gross tool like the Beck depression Inventory, which is good at detecting depression but not for grading it.

[sthomson]

I was listening to the same story. It surprised me only in that I thought someone on this forum had said that Big Pharma has to pre-register all med trials to avoid this.

Yep. That's how they found out about the publication bias, isn't it? By looking at the results of the registered trials vs. the results of the published trials?

[blutoski]

I was listening to the same story. It surprised me only in that I thought someone on this forum had said that Big Pharma has to pre-register all med trials to avoid this.

That might have been me.

My impression from the wording in the OP is that a "positive" paper submitted to a journal has a 90% chance of being published, while a "negative" paper has an 8% chance of being published.

As opposed to: 90% of published papers are positive, 8% of published papers are negative.

I'll have to get a copy of the report the program was discussing, because I think they may be confusing publication bias with file-drawering.

Publications can't publish every study that is submitted, and there is evidence that they publish trials of new treatments that show positive results over all the new treatments that show negative results, because it's evidence treatment may need to change, and practitioners should know. Sometimes, an accepted treatment is tested and shown not to work, and this is new information instead, and is an example of a reason to publish a negative study. This is publication bias: it's more likely that a new treatment is being tested, and it's more likely to publish new treatment research if it's positive. That's all.

File-drawering is different. This is the tendency for a research party such as a pharmaceutical company to start research and either stop it when the results look unpromising, or to complete the research, find unsatisfactory results, and simply discard it. The file drawer problem is not just that they are unavailable in high-profile journals, but that they have not been submitted to any journal at all and are completely unavailable. In this day and age of crap journals and the internet, if you really want something to be available, it can be. File-drawering is caused by the deliberate decision to keep these results from inclusion in literature reviews.

Registration does not mean that all trials will be published in peer-reviewed journals, but it does increase the likelihood that trials which are started will be known, whether they get published or not. In principle, the trial results would still be available in the corporate archives and available for review and consideration. If not, then there's a paper trail that the company has hidden results, and it's reasonable to assume the results were negative.

[Garrette]

Yep. That's how they found out about the publication bias, isn't it? By looking at the results of the registered trials vs. the results of the published trials?Possibly, but I didn't catch it. Then again, I was busy with other things and only half-listening.

And with that, I have exceeded my expertise in this area and so bow out, with thanks to Blutoski for his post.

[blutoski]

Just to give an example from the Skeptical side, we're working right now on an attempt to replicate Sheldrake's phone precognition experiment.

Now, skeptics are more likely to be biased toward expecting a negative result, so the positive-negative situation is a bit reversed for us, but the principle is the same.

If the results are positive (if precognition hypothesis is supported), I might be tempted to pretend the whole thing didn't happen. Not sure how I'd do this, but let's say I buried the results so nobody could know about it. That's be file-drawering.

OTOH, let's say that we got a negative result (precognition not supported) and I submitted it to a bunch of psi publications. If they rejected the submissions for reasons other than failing peer-review (ie: if they agreed the methods were sound &c, but just felt it wasn't the right time or place to publish these results), that'd be publication bias. We could always publish it in a Skeptical magazine instead, and it would be available for anybody who wanted to do a literature review.

[sthomson]

If the results are positive (if precognition hypothesis is supported), I might be tempted to pretend the whole thing didn't happen. Not sure how I'd do this, but let's say I buried the results so nobody could know about it. That's be file-drawering.

Mostly, what I've seen happen is a bit more complicated. Let's say you do the experiment once, and the results come out positive. You think about it for a bit, locate a potential design flaw, tweak the experiment, then come out with a negative result. Your original positive result gets file-drawered, and you publish your negative result in a peer-reviewed journal. Or, perhaps you get a another positive result, find another design flaw, and repeat the test again.

This is how science works, right? Continually eliminating sources of bias from experiments to get better results. But, it also leads to "file drawer" results. Your preconceived notion of what the results should be dictates when you stop looking for experimental flaws. Now, a good scientist will eventually say, "Maybe my preconceived notion was wrong" in the face of repeatedly successful results, and an even better scientist will ask, "How may this negative result have been biased by the experimental set-up?" after a negative result, but this does not occur all the time. And, in fields such as parapsyhology where the statistical difference between a positive and negative result is so small, this sort of unintentional file-drawer bias is even more likely to occur.

(I don't mean to pick on you, blutoski - I merely used "you" in the illustrative sense. And of course, I would never accuse skeptics of exhibiting such biases more than believers do.)

[blutoski]

Mostly, what I've seen happen is a bit more complicated. Let's say you do the experiment once, and the results come out positive. You think about it for a bit, locate a potential design flaw, tweak the experiment, then come out with a negative result. Your original positive result gets file-drawered, and you publish your negative result in a peer-reviewed journal. Or, perhaps you get a another positive result, find another design flaw, and repeat the test again.

This is how science works, right? Continually eliminating sources of bias from experiments to get better results. But, it also leads to "file drawer" results. Your preconceived notion of what the results should be dictates when you stop looking for experimental flaws. Now, a good scientist will eventually say, "Maybe my preconceived notion was wrong" in the face of repeatedly successful results, and an even better scientist will ask, "How may this negative result have been biased by the experimental set-up?" after a negative result, but this does not occur all the time. And, in fields such as parapsyhology where the statistical difference between a positive and negative result is so small, this sort of unintentional file-drawer bias is even more likely to occur.

(I don't mean to pick on you, blutoski - I merely used "you" in the illustrative sense. And of course, I would never accuse skeptics of exhibiting such biases more than believers do.)

I think I understand your point: researchers can file-drawer unfavourable results with sincere intentions - not just deception.

I understand this concern, but propose that if there's a real methodological flaw, the paper would be rejected by a reviewer anyway, and the impact on total knowledge is zero.

It's the rejection of valid research results that impacts science's understanding of the phenomenon in question.

Or are you saying that maybe it's common that a researcher gets an unfavourable result and then rationalizes it away with an excuse of bad methodology, even though the study may be OK? I can see how that would be a problem, as the removal of good research from the body of literature impacts understanding.

However, this problem would be significantly mitigated with the prior registration approach, since the study would be available for peers to examine and see if the researcher's methodology rejection was sensible.

[Paul C. Anagnostopoulos]

I understand this concern, but propose that if there's a real methodological flaw, the paper would be rejected by a reviewer anyway, and the impact on total knowledge is zero.
It would be nice if they did the parapsi world a favor and published the flaw. Then maybe it wouldn't happen again.

~~ Paul

[Paul C. Anagnostopoulos]

Say Blutoski, have you got a protocol we can review?

~~ Paul

[blutoski]

It would be nice if they did the parapsi world a favor and published the flaw. Then maybe it wouldn't happen again.

~~ Paul

Absolutely, so I stand corrected when I said that the impact of suppression would be zero.

One of the things about science methodology is that it is its own field of study that benefits from the discussion that follows failed experiments.

[blutoski]

Say Blutoski, have you got a protocol we can review?

~~ Paul

I wonder if it'd make more sense for us to create a seperate thread for discussing a protocol?

Probably not appropriate for the Challenge forum, though.

[sthomson]

I understand this concern, but propose that if there's a real methodological flaw, the paper would be rejected by a reviewer anyway, and the impact on total knowledge is zero.

Hopefully! But flaws may be of the type that a reviewer would not necessarily spot. I don't know much about medical trials, but I do understand that pharmacology corporations have a vested interest in getting positive results!

However, this problem would be significantly mitigated with the prior registration approach, since the study would be available for peers to examine and see if the researcher's methodology rejection was sensible.

The methodology in trials like this is rather fixed, I believe. However, I think the problem generally stems from the fact that a positive and negative result are not black and white. Drug trials often rely on measures of statistical significance between similar populations, right? So it's possible that one test shows a statistically significant difference, and another test won't. The drug companies aren't going to keep testing drugs after they get positive results. This is on top of any sort of intentional cover-up of bad trials.

[blutoski]

The methodology in trials like this is rather fixed, I believe.

The only thing that's fixed is placebo-controlled double-blinded, randomized.

Beyond that, methodology is very dependent on the indication and drug.



However, I think the problem generally stems from the fact that a positive and negative result are not black and white.

That's the definition of an acceptable study, though: unambiguous outcome metrics. One can quibble over binary thresholds, of course.

I wouldn't call it a 'problem,' though.




Drug trials often rely on measures of statistical significance between similar populations, right? So it's possible that one test shows a statistically significant difference, and another test won't.

Happens all the time.

Most Phase I trials come out negative, but those that come out positive go to Phase II. Most Phase II trials are negative. It follows that most Phase II trials contradict the Phase I outcome.




The drug companies aren't going to keep testing drugs after they get positive results. This is on top of any sort of intentional cover-up of bad trials.

Yes, that's the point of preregistration: to expose 'cover-ups,' and make them more or less impossible.

The other thing to keep in mind is that often enough, other parties are retesting their drugs, too. eg: competitors, academia. Nobody just takes, say, Roche's word for it.

[Paul C. Anagnostopoulos]

I wonder if it'd make more sense for us to create a seperate thread for discussing a protocol?
Indeed. Start a new thread to discuss your protocol.

~~ Paul

[Dancing David]

That might have been me.

My impression from the wording in the OP is that a "positive" paper submitted to a journal has a 90% chance of being published, while a "negative" paper has an 8% chance of being published.

As opposed to: 90% of published papers are positive, 8% of published papers are negative.

I'll have to get a copy of the report the program was discussing, because I think they may be confusing publication bias with file-drawering.

That was my statement, not the story or the article's, I did it deliberately to generate interest and grap attention. Shameless pandering on my part.

Mea culpa.

It is a publication bias for sure.

[Dancing David]

Here is a link to the audio file:
http://www.npr.org/templates/story/story.php?storyId=18177115

[blutoski]

That was my statement, not the story or the article's, I did it deliberately to generate interest and grap attention. Shameless pandering on my part.

Mea culpa.

It is a publication bias for sure.

Fair enough.

Just to repeat myself: there may be innocuous reasons for publication bias.

One of the debates a few decades ago was about the definition of 'information'. An example is: when you read a newspaper, would you expect to find a headline like "Sahara Hot and Dry?" It is hot and dry, and this information is 'submitted' to weathermen every day. But they don't publish it.

On the other hand, if it were to suddenly snow on the Nile, a headline like "Sahara Cold and Wet" would be expected. It'd be published in a New York minute. Wuxtry!

So, the point is that there doesn't have to be an intentional suppression of negative findings so much as the evaluation that their publication is not very important.

The point is that a journal gets more submissions than they can print, and they have to select them for informational value. Trials that show a new drug doesn't work may not be useful to the journal's audience, whereas a trial showing a new drug does work will be valuable - it can lead to a scramble for duplication studies and move the prospective drug forward. Some MDs may even begin to prescribe it off-label.

The situation where a negative study would be valuable is if the drug in question is already in common use, and now there's evidence it doesn't work or doesn't work well compared to another option. These are much rarer than positive results from a study of a new drug.

[blutoski]

DP

[SteveGrenard]

So wait a minute. If a study is started and early results looks like it is unpromising and it is stopped, it makes a certain amount of sense. Why would a commercial, profit making enterprise want to continue spending a ton of money on something that predictably will be unpromising?

And if it is unpromising and the study stopped for this reason doesn't it also mean that the company will not be submitting the drug for marketing approval or for approval to add a new indication?

[calebprime]

-Doctors rely on the prevailing wisdom and articles in medical journals.

-The medical journals are publishing a higher percentage of studies of trials which seem to show that antidepressants are effective, than studies which show them to be ineffective.

-These are not new drugs we are discussing. They are the antidepressants already sold under the names Paxil, Zoloft, etc.

-There are at least 4 areas of concern with SSRI's--sexual dysfunction, withdrawal symptoms, overall effectiveness compared to other options, long-term effects (and long-term effectiveness). There are others as well.
Suicidality, akathisia.

-For these areas of concern, the drug companies somehow didn't study them, or exaggerated the benefits and downplayed the risks. The studies were short-term. I have read of various methods the drug companies used to slant their data. (This is subtle stuff for me--so I don't recall all the details--but I've read about this quite a number of times.)

-example: Withdrawal symptoms were at first written off as the return of depression. Sexual dysfunction was blamed on depression--or supposedly could be treated with other drugs (!)

-Ought reports that SSRI's are ineffective be considered less interesting than those saying that they are effective? I don't understand.


Whether this is publication bias or "File-drawering", isn't this important evidence that doctors have received skewed information?

Someone more knowledgeable than me, please confirm or explain why not.

[fls]

-Doctors rely on the prevailing wisdom and articles in medical journals.

They also rely on the FDA-approval process and post-marketing surveillance. The FDA had access to the studies even if they were not all published in medical journals. Regardless of what is published, the uses for which a particular drug is approved will have evidence backing that use.

-The medical journals are publishing a higher percentage of studies of trials which seem to show that antidepressants are effective, than studies which show them to be ineffective.

That really only influences the perception of overall effectiveness. And from a practical point-of-view, the absolute effectiveness of a medication doesn't really matter when it comes to prescribing for an individual. You don't really expect an identical response to what was found in an RCT, as an RCT represents a somewhat contrived situation. You just need to know about relative responses - whether it's likely to have a response better than placebo or other anti-depressants, and in what conditions (mild vs. moderate vs. severe depression, for example). Publication bias becomes more relevant when you are talking about off-label uses.

-These are not new drugs we are discussing. They are the antidepressants already sold under the names Paxil, Zoloft, etc.

Yes, but it includes the information that went in to deciding whether or not the drugs were approved - i.e. at the time of the studies they were new drugs.

-There are at least 4 areas of concern with SSRI's--sexual dysfunction, withdrawal symptoms, overall effectiveness compared to other options, long-term effects (and long-term effectiveness). There are others as well.
Suicidality, akathisia.

-For these areas of concern, the drug companies somehow didn't study them, or exaggerated the benefits and downplayed the risks. The studies were short-term. I have read of various methods the drug companies used to slant their data. (This is subtle stuff for me--so I don't recall all the details--but I've read about this quite a number of times.)

-example: Withdrawal symptoms were at first written off as the return of depression. Sexual dysfunction was blamed on depression--or supposedly could be treated with other drugs (!)

To be fair, that is the point of post-marketing surveillance. It is difficult to discover these effects before there is widespread use of the drug. Side-effects are looked for in Phase I, II, and III trials, but to rule-out uncommon effects or to thoroughly discover the effects of long-term use would be prohibitively expensive and time-consuming. It's a balance between making useful drugs available and ensuring they are as safe as possible.

And regardless of whether drug companies try to suppress the information, the process is too transparent, and too many other people, like physicians, are aware of the information, for it not to become public knowledge.

-Ought reports that SSRI's are ineffective be considered less interesting than those saying that they are effective? I don't understand.

I think Blutoski answered this question well, already. Unless the information is likely to change practice, it's not particularly interesting regardless of whether it's positive or negative.

Whether this is publication bias or "File-drawering", isn't this important evidence that doctors have received skewed information?

Someone more knowledgeable than me, please confirm or explain why not.

Doctors generally assume that they are receiving skewed information, though. This is sort of a formal confirmation of already established suspicions. There tends to be an ongoing re-evaluation of drugs over time, sometimes even over-turning prior practice. That this kind of information comes to light shows that the system is working.

Linda

[Dancing David]

On the whole 'effectiveness' issue, the scale that is used to asses depression is crucial, the Beck Depression Inventory is a good gross assessment tool, and it is not a great way to assess levels of depression.

Here is a brief technical discussion of the BDI:

http://www.swin.edu.au/victims/resources/assessment/affect/bdi.html

Here is the Wikipedia article:

http://en.wikipedia.org/wiki/Beck_Depression_Inventory#BDI


This is a very useful tool for assessing the presence of depression, however it will not give you as accurate a picture as a functional analysis or personal interview. People are notorious for under and over scoring.

(For example)

When I did interviews about domestic violence you would ask people a very standard set of question, one of which was “Has there been physical violence?"

Now it is very important when you ask this question to listen and then follow up with a series of behavioral indicators and criteria. I am very serious that people would say "No" and then answer "Yes" to most of the follow-up questions.

Did they strike you? Did they slap you? Did they grab you and restraint you? Did they ever bruise you when they grabbed you or restrained you? Did they throw things? Did they break things? Did they block the doorway? Did they cut you? Did they burn you? Did they choke you?

So this can be a real problem in any information gathering, seriously many victims would say "No" and then when you ask if they have been choked they would say "Yes".

Then there is the issue of how you grade events as an individual who is living with depression. You get used to being depressed and so how you answer questions kind of gets skewed after a while, especially if you have had it for a long time. You begin to notice that 'good' days are really 'not so terrible days'. And so when someone rates their level of sadness this can totally mess up the scoring.

[robinson]

Just watched a News clip about this. Said 94% of published studies state antidepressants work great, while 96% of unpublished studies show little or no benefit from the same drugs.


http://cosmos.bcst.yahoo.com/up/player/popup/?rn=3906861&cl=5989719

[robinson]

Anybody know how to get to the unpublished studies?

[The SkepDoc]

Here's the abstract: http://content.nejm.org/cgi/content/abstract/358/3/252

I've read the whole article. It doesn't say anything new. We always knew about publication bias, and that's why efforts have been made to get all trials registered. The authors point out that their review did not show that antidepressants were ineffective, it just showed that they may not be quite as effective as we thought. I was intrigued that there did not appear to be any publication bias for Prozac in the table they printed iin the article. Prozac was the first, prototypical SSRI. I wonder what the lack of bias for that particular drug means. I think one of the authors of the new Science Based Medicine blog at http://www.sciencebasedmedicine.org/ plans to discuss this study soon. Watch for it!