What's wrong with science (the profession)?

[Cynric]

OK, so we spend a lot of our time here dissecting the shortcomings of the various woo delusions, but for a change let’s apply our incisive minds closer to home. What’s wrong with science?

I propose limiting this to the profession of science – but if anyone has wider ranging views please join in.

Perhaps we could avoid any dazzling insights along the lines of “science doesn’t know everything,” and “Galileo was persecuted once you know.” Science never claimed to know everything (quite the reverse), and Galileo was persecuted by religious fundamentalists for putting reason before faith (and being a cantankerous old fart about it).

So, I’ll kick things off with a pet hate of mine: current attitudes to measuring scientific success are corrosively anti-science; the biggest problem being the dreaded Impact Factor.

For those who have not come across it, the Impact Factor is a metric for determining the relative scientific impact of a particular journal. That is, journals publishing papers which are widely read, assimilated and cited by other scientists (i.e. impact on how they pursue their own research), have a high impact factor.

I think you can see where the administrator’s went with this excellent metric: if you publish in journals with high impact factors, you must be doing good science. This is despite the fact that the chap who invented it (Eugene Garfield) has repeatedly stated:


The source of much anxiety about Journal Impact Factors comes from their misuse in
evaluating individuals, e.g. during the Habilitation process. In many countries in Europe, I have found that in order to shortcut the work of looking up actual (real) citation counts for investigators the journal impact factor is used as a surrogate to estimate the count. I have always warned against this use.

[From an article on Garfield’s extensive and interesting website: http://www.garfield.library.upenn.edu/]

According to the Institute for Scientific Information website, there are 5969 journals listed in the Science Citation Index. Of these, 93 have an impact factor >10. Most learned society journals (e.g. Biochemical Journal, Journal of Physiology, Philosophical transactions of the Royal Society) have an impact factor in the 3-6 range. But this is all basically irrelevant. There are ~6 journals that matter. In the basic biological sciences these are Nature, Science and Cell, and in the medical sciences, New England Journal of Medicine, Journal of the American Medical Association and the Lancet.

The blunt and brutal truth is that these days a publication in one of these 6 journals means the difference between getting tenure and not. Of course, ultimately the best way to succeed is to do good science. But these days, you’d better do it fast and publish it in one of those 6 journals, or you may not get another chance.

Why does any of this matter? After all, tenure is not the only means for securing good science. Well, I think this attitude for measuring scientific success has 3 major downsides:

1) Haste.
There is a lot of pressure on young scientists to publish fast, and publish frequently (and in high impact journals). That is why the literature is filled with papers of questionable value. Minimal experiments are carried out in order to “prove” the pet hypothesis (another massive irritant of mine – hypotheses should be tested not “proved”!). Claimed data often turn out to be irreproducible. Overlapping sets of results are published in several locations (so called Salami slicing). Basically, studies are not finished in a complete fashion, because it takes too long and someone may scoop you. If you don’t publish a steady stream of papers, your career is sunk. It is hard to imagine Watson and Crick getting tenure in today’s climate, given how long they took to publish any results.

2) Hype.
The 6 journals mentioned above are particularly interested in big discoveries. Consequently, many authors give a politician-like spin to their results in order to attract the top 6 and secure their future. Big claims, which are often simply wrong, take precedence over careful work.

3) Fabrication.
When your career rests on a big paper in a big journal, the temptation is to get the results you need by any means. The recent cases of Hendrick Schon and Hwang Woo Suk illuminate the great personal benefits of dishonesty (until found out, of course). Most research scientists have anecdotal experience of questionable practices by junior or senior colleagues, most of which go unnoticed, but pollute the literature.

The present senior faculty in universities did not face this climate to anything like the same extent. Many who have made huge breakthroughs took a good length of time to develop their ideas, and the full significance of their discoveries only became apparent years after publication. I fear that they will be replaced by those scientists most adept at the flashy “presentational” side of scientific discovery, rather than the slow, steady accumulation of evidence that leads to confident advancement in knowledge. This could lead to an analogous situation to the erosion of quality evident in humanities departments, where relativism has ousted predictive models for analysis of language. After all, grand claims and slapdash experiments are easier than real research.

So, is it a problem? What's the solution?

[shrugs]. I dunno. Wha’d’ya think? Huh?

[T'ai Chi]

I'm not too keen on the war applications.

[Cynric]

I'm not too keen on the war applications.

Of technology?

I suggest you talk to the politicians and generals that start wars, then.

[T'ai Chi]

I suggest you talk to the politicians and generals that start wars, then.

Apparently you believe me sharing what I dislike implies that I have to do stuff. I don't share your belief.

[JamesM]

When your career rests on a big paper in a big journal, the temptation is to get the results you need by any means. The recent cases of Hendrick Schon and Hwang Woo Suk illuminate the great personal benefits of dishonesty

On the other hand, plagiarism has always struck me as immensely pointless. Either the work you're doing is sufficiently important that other people will attempt to replicate it, and after they fail, you get rumbled; or, it's not that important, in which case, why bother?

[Cynric]

On the other hand, plagiarism has always struck me as immensely pointless. Either the work you're doing is sufficiently important that other people will attempt to replicate it, and after they fail, you get rumbled; or, it's not that important, in which case, why bother?

I guess you mean fabrication rather than plagiarism (sorry for being a pedant). In the long term, fabrication is pointless. Ultimately, your contribution will be forgotten as useless. Unless you guess correctly, of course, and fabricate evidence for a real phenomenon. I think most fabricators fall into this category - they are sure that their hypothesis is right, they just don't have the time/skill/equipment/patience to carry out the experiments.

However, in the short term, fabrication can be immensely rewarding. Competitors failing to reproduce your work is very common. Some very eminent professors have a reputation of flitting from field to field, publishing a "big" paper that turns out to be junk, and leaving the rest of the field to tidy things up over several years. But were they fabricating, or just selective with data, or sloppy? By the time it's rumbled, they're onto something new and they still have a stellar publication record...

[Strider1974]

My pet hate in science is how politics is involved in funding.
Throw enough money at a subject and you will get a whole host of so called scientists willing to prove your cause.
Just look at eugenics

[Floyt]

It has been suggested that open access publishing might do its part in relegating the impact factor to the use intended by Garfield - i.e., the measuring of the reputability of a journal. OAP is frequently still having a tough time at the moment, though. After all, there's a sizable industry based on subscription journals.

Another thing I've heard mentioned would be a kind of negative offset at the level of the individual author, your "salami number", so to speak, factoring cases of slicing, rushed publication etc. into your rating. Nice idea in theory, but has "TOXIC" written all over the implementation :boggled:

[Outhere]

My pet peeve about scientists is that they do what I would probably do; take the government's money and develop weapons that they themselves have moral qualms about, with good reason. Einstein was the best ditherer in this area and I think he could stand for 20th century man (yes Man). I shouldn'ta done it, but I done it and what do I do now? Or rather, what do you poor slobs do now?

[athon]

The political side of science indeed has major issues. As a social field (which in our perception always takes a relatively long time to evolve) it is relatively new; journals and the influence of science in the modern world is a mere infant. And it has had some big teething issues.

In the future, we are going to rely increasingly on this system as one of measure and record, and so it needs to become more robust. But I don't see this as a problem with the method of science, rather how it is practically implemented in today's global society.

For me, one of the biggest impediments continues to be selective publishing of successful experiments. Failed experiments that are either abandoned or left unsubmitted can contribute a great deal to science, yet are ignored in favour of the reward of success.

Change is needed, and will happen, if slowly.

Athon

[Rob Lister]

My pet hate in science is how politics is involved in funding.
Throw enough money at a subject and you will get a whole host of so called scientists willing to prove your cause.
Just look at eugenics

Nothing wrong with eugenics as a science. So long as the goal (to discover truth, to build upon it) remains pure, so too will the science of it. Politics does drive goals but that applies to all sciences, not just eugenics.

I think eugenics has great potential. Especially now that we can actually take more advantage of advances in it than ever before.

[Strider1974]

Rob I think (hope) we are referring to different things.
I used the term 'eugenics' as referenced below

"eugenics is often used to refer to a movement and social policy that was influential during the first half of the twentieth century"

"the international scientific community has mostly disassociated itself from the term "eugenics", sometimes referring to it as a pseudo-science"

http://en.wikipedia.org/wiki/Eugenics

[geni]

The conflict between falsificationism (which I tend to feel is the best model for describeing how science should work) and Feyerabend's scientific anarchy which describes how historicaly science has worked.

[T'ai Chi]

This is also pretty horriffic:

http://en.wikipedia.org/wiki/Tuskegee_syphilis_study

[CFLarsen]

Apparently you believe me sharing what I dislike implies that I have to do stuff. I don't share your belief.
But then, you are not interested in solving the problems, but only whining about them.

[Chris Hyland]

I quite like the idea of the H-index:
arxiv.org/PS_cache/physics/pdf/0508/0508025.pdf


My pet hate in science is how politics is involved in funding

Yeah I know, for example the EU wont fund malaria research because there's not enough oppurtunity for profit.

[Darat]

....snip...

Yeah I know, for example the EU wont fund malaria research because there's not enough oppurtunity for profit.

Yes it does: http://ec.europa.eu/research/health/poverty-diseases/fp6projects_en.html

[Cynric]

It has been suggested that open access publishing might do its part in relegating the impact factor to the use intended by Garfield - i.e., the measuring of the reputability of a journal. OAP is frequently still having a tough time at the moment, though. After all, there's a sizable industry based on subscription journals.

Another thing I've heard mentioned would be a kind of negative offset at the level of the individual author, your "salami number", so to speak, factoring cases of slicing, rushed publication etc. into your rating. Nice idea in theory, but has "TOXIC" written all over the implementation :boggled:

I think open access is a great idea. The Public Library of Science (PLOS) does genuinely seem to be making an impact. It will be interesting to see how the OA journals rate in impact factor in the future.

If I was in a more secure position, personally, I would be very tempted to publish only in OA journals. As things stand, however, that would have a detrimental effect on my efforts to secure a permanent job. Which is one of the reasons for my long whinge.

[Cynric]

I quite like the idea of the H-index:
arxiv.org/PS_cache/physics/pdf/0508/0508025.pdf


The H index is one of the better measures for individuals, I agree.

For those who don't want to follow the link, the basic idea is to find the number (n) of publications by a scientist that have been cited n times. i.e. if you've published 20 papers which have been cited (by other papers) at least 20 times, then your H index is 20. It is very hard to increase one's H index without publishing high quality papers that will be cited frequently.

[Cynric]

Apparently you believe me sharing what I dislike implies that I have to do stuff. I don't share your belief.

Apparently you believe that a suggestion implies compulsion.

[lenny]

For those who don't want to follow the link, the basic idea is to find the number (n) of publications by a scientist that have been cited n times. i.e. if you've published 20 papers which have been cited (by other papers) at least 20 times, then your H index is 20.
so 20 20's and 20 0's gives the same H as 19 200's 1 20 and 20 18's?

that is not a complaint; and i am now curious what the distribution of #cites looks like for "average" scientists. i expect a one or two "big" ones, a few moderate ones, and many 2, 1, 0's. the question is how well does the H index discriminate (ignoring the silliness of counting citations instead of scanning/reading the bloody papers)? esp among younger scientists. but small n always means small H, no? so it seems H is just your number of "frequently cited" papers, slightly better than your "number of papers" nevertheless, but a bit hard on the young scientist who's papers have not been in print very long, has few papers overall, and has made a few highly cited (and hopefully interesting) remarks.

there is no penalty for a vast number of zero citation papers, but also no reward for a moderate number of highly cited papers. i personally would like a score which tended to decrease the noise level by rewarding those who publish fewer, more interesting papers...

do self citations count? if yes, then we can work out special cases, like someone who cites all their previous work, and is cited by no one else...

and do you feel it really has to be restricted to single number?

[Cynric]

so 20 20's and 20 0's gives the same H as 19 200's 1 20 and 20 18's?

there is no penalty for a vast number of zero citation papers, but also no reward for a moderate number of highly cited papers. i personally would like a score which tended to decrease the noise level by rewarding those who publish fewer, more interesting papers...

do self citations count? if yes, then we can work out special cases, like someone who cites all their previous work, and is cited by no one else...

and do you feel it really has to be restricted to single number?

It's not perfect - nothing is - but the H index is a good measure of consistency. You're right, both those scenarios would give an H of 20, but I imagine the second is quite unlikely to occur. One observation in it's favour is that the top ten H index scientists are those widely accepted as having had a big and lasting impact. It penalizes the savants that have one big hit, but it rewards those that build a steady reputation within a field.

There is a discussion here:

http://www.nature.com/nature/journal/v436/n7053/full/436900a.html

but it's restricted access. I'll get the details at work tomorrow and post brief highlights.

Self citations do count, but you'd have to be going some to push your score up significantly. From what I remember the "average" number of citations for a paper is ~20. Junior faculty typically have an H of 10-20, senior reach >40, highly cited scientists push into the hundreds.

I don't think any one factor is good for rating individuals, but some index of citations is a good start, as it's a hard one to bias. You can have a dozen wild Nature papers, but if nobody believes them the citations will dry up quickly.

[lenny]

One observation in it's favour is that the top ten H index scientists are those widely accepted as having had a big and lasting impact.i take your point, but i am not sure that this is in it's favour, as none of those ten are going up for tenure or likely to be lost in the midst of 100 applicants for the same job. anything in it's favor in terms of evaluating young/pre-tenure scientists?I'll get the details at work tomorrow and post brief highlights. thanks [edit to fix "quotes" & add "thanks".]From what I remember the "average" number of citations for a paper is ~20. wow! that seems remarkably high to me! what field?
You can have a dozen wild Nature papers, but if nobody believes them the citations will dry up quickly.agreed. i would also be interested in a statistic that counted non-self citations 10 years after publication rather highly...

[Cynric]

i take your point, but i am not sure that this is in it's favour, as none of those ten are going up for tenure or likely to be lost in the midst of 100 applicants for the same job. anything in it's favor in terms of evaluating young/pre-tenure scientists? thanks

True. I don't know how it would pan out for young scientists - probably a lot of clustering below 10.
As you point out, the strategy for assessing "success" would have to vary with the age/position of the scientist. I suppose highest citation number would be a goodish measure for youngsters, but then being on a big review with a star supervisor might skew that too.


wow! that seems remarkably high to me! what field?

snip

agreed. i would also be interested in a statistic that counted non-self citations 10 years after publication rather highly...

So, the quote of ~20 comes unattributed. I heard it from a former director of the institute where I currently work, and it refers to the average (probably mean) of citations per paper for papers 10 years old. The field is biomedicine, and so 20 isn't massive. Physics and maths (you know, proper science ;)) have a lot lower citation rates.

Do you think self-citation is a problem? I agree that it has something of a masking effect, but it also shows a dedication to a chosen subject. I find people who stick on a given theme and develop it over a decade preferable to the jacks of all trade that flit back and forth.

[Jimbo07]

The thing I hate most about science is the thing I like best.

Science does not provide certainty. It must fight in the arena of public opinion with one hand tied behind its back, through onslaughts from the religious, woos and 'sophisticated' liberal arts students. :mad:

[blutoski]

So, is it a problem? What's the solution?

[shrugs]. I dunno. Wha’d’ya think? Huh?


The expression is "Publish or perish." It's a problem.

My solution: I got a "day job" in IT until I can take more risks with long-term research projects.

Alternatively, there's always college teaching. Good salary, flexible hours, minimal supervision, usually none of the soap opera that revolves around research, but there's an opportunity now and then.

[Cynric]

The expression is "Publish or perish." It's a problem.

My solution: I got a "day job" in IT until I can take more risks with long-term research projects.

That takes some courage if your interests are in "mainstream" academia. Getting back into a research position after a hiatus can be tricky - another problem with the current professional situation for scientists. I think this is also the reason for low salaries (at least in the UK - see current lecturer strike), as there are always more candidates prepared to do the job than there are jobs. Rather than aiming for the best candidates, therefore, the universities will prefer good candidates who will tolerate low pay.


Alternatively, there's always college teaching. Good salary, flexible hours, minimal supervision, usually none of the soap opera that revolves around research, but there's an opportunity now and then.

Unfortunately, I'm too impatient to teach. I admire people who can do it, and am very grateful that they do.

[blutoski]

That takes some courage if your interests are in "mainstream" academia. Getting back into a research position after a hiatus can be tricky - another problem with the current professional situation for scientists. I think this is also the reason for low salaries (at least in the UK - see current lecturer strike), as there are always more candidates prepared to do the job than there are jobs. Rather than aiming for the best candidates, therefore, the universities will prefer good candidates who will tolerate low pay.

Well, I'm in a special situation: my wife is finishing her residency in about a year, which is just the right time for me to register for a postdoc in W2007. I'm crossing my fingers that this will mitigate the 'gap' in my CV from these relatively underproductive years.



Unfortunately, I'm too impatient to teach. I admire people who can do it, and am very grateful that they do.

Chacun a son gout. It's actually part of the appeal for me, and I look forward to teaching. Bill Nye's my role model. I think there's a bit of the showman in me.

[lenny]

So, the quote of ~20 comes unattributed. I heard it from a former director of the institute where I currently work, and it refers to the average (probably mean) of citations per paper for papers 10 years old. The field is biomedicine, and so 20 isn't massive. Physics and maths (you know, proper science ;)) have a lot lower citation rates.
i just computed my thesis professor's H; ivy league, "proper science", strong international reputation, now nearing retirement: H ~ 33.
Do you think self-citation is a problem? I agree that it has something of a masking effect, but it also shows a dedication to a chosen subject. I find people who stick on a given theme and develop it over a decade preferable to the jacks of all trade that flit back and forth.
if your median citation rate is 20-ish then self citations are probably in the noise; if it is 2 or 3, then self citations can have a significant impact.

it would be interesting to see the IF of Nature broken down to exclude Nature, or to show the distribution of citations by origin...

[lenny]

As you point out, the strategy for assessing "success" would have to vary with the age/position of the scientist. I suppose highest citation number would be a goodish measure for youngsters, but then being on a big review with a star supervisor might skew that too. well, if we goto two d, then we can consider the cites per paper, and the max citation of any paper by any other author on the paper. that will spot the potential problem, but of course the young guy might have written that review paper...

i am told that for major evaluations in Italy, you have to state your "percentage of contribution" for every paper you want considered, and the statement must be SIGNED by every coauthor for each paper.

also, i was wrong to suggest that there is no disincentive for publishing lots of papers: there is no incentive to publish a paper you expect to get fewer cites than your current H, so splitting up ideas or repeating them can be discouraged. but then so will interesting, pedagogically useful, little dead-end insights.

[rockoon]

The current system of peer review is messed up.

Not that the idea of peer review is bad, but that humans are involved and that makes for some messed up backstabbing, theft, and so on.

[luchog]

The main problem with science as a profession is that there aren't enough hot chicks doing it.

Galileo was persecuted by religious fundamentalists for putting reason before faith (and being a cantankerous old fart about it).

Not quite. Galileo wasn't so much persecuted as butted heads with a major political figure, who was just as pigheaded as he was. And not for putting reason before faith, but for claiming that his theories, which were based nearly as much on complete woo-woo nonsense as on scientific observation, invalidated religion completely and the Roman Catholic church specifically; and used it as the basis for rather viscious attacks on the church. His support for his theory contained several major errors, which other scientists of his time had noticed and therefore failed to support him because of it. (For example, Galileo completely denied the effect of the moon on tides; a theory supported by Kepler and most of Galileo's contemporaries).

Pope Urban VIII was also not the most reasonable of me, and reacted poorly to Galileo's attacks. Geocentrism, while Urban's preference, was not official church doctrine, the pope was not speaking ex cathedra, and his actions were highly controversial within the church.

Although heliocentrism was dismissed, and even prohibited, for a short time; it was never done so ex cathedra, only by the personal preference of a couple of popes. The church at the time was, in fact, one of the strongest proponents of the physical sciences, and many prominent cardinals not only accepted, but actively promoted Copernican heliocentrism.

Rather than persecution of the noble scientist standing up to the repressive religion; it was more or less just a clash of a**holes who were both more concerned with personal agendas than truth.

[drkitten]

The current system of peer review is messed up.

Not that the idea of peer review is bad, but that humans are involved and that makes for some messed up backstabbing, theft, and so on.

I'm afraid that the alternative seems to be substantially worse. If you don't have "peer-review," then you'll have science and scientists still being judged by humans, but by humans with even less understanding of the material, and just as much backstabbing, theft, and other humorous pursuits.

[Cynric]

I'm afraid that the alternative seems to be substantially worse. If you don't have "peer-review," then you'll have science and scientists still being judged by humans, but by humans with even less understanding of the material, and just as much backstabbing, theft, and other humorous pursuits.

Boy, I wouldn't like to read your reviews ;)

Peer-review as it currently stands is crap. Frankly. And I say that as author and reviewer.

In my opinion the "expert" referees should be limited to only comment on the technical competence of the manuscript, and it's originality (i.e. the work hasn't been previously published). So much skulduggery is hidden in the "would this be of general interest to other readers" questions of the top journals. How many manuscripts have been sunk by snooty reviewers disparaging the general significance of a paper?

Not to drown the thread in personal anecdotes, but I once sent a manuscript to one of the 6 journals above (clue it rhymes with hate-ya). It was an analysis of the kinetics of my favourite enzyme. The referee commented that as I'd only deduced a phenomenon from kinetic analysis, it was all hypothetical and unworthy of publication. I resubmitted it to a second-tier journal, and one of the referees said it was the most interesting paper they'd read in the field for 10 years.

Now I suspect the truth lies somewhere between these extremes, but with comments like that, how are you supposed to act? Should I have argued the toss with the first journal? It's not my sort of thing, but people do it, and it sometimes works. I've even heard of some researchers who phone the press before submission, and then ring the editor and lay on the "the press are pushing for details of this work," crapola.

I also wonder if some referees get a reputation as cantankerous farts, and are used by editors to kill papers they're unsure of. I wouldn't have wanted my paper refereed by Galileo.

[Cynric]

The main problem with science as a profession is that there aren't enough hot chicks doing it.


There are plenty in the biological sciences. Personally, I find them rather distracting in a completely-unprofessional-would-never-admit-it kind of way. Summer's the worst. They're just so pretty... *sigh*


Not quite. Galileo wasn't so much persecuted as butted heads with a major political figure, who was just as pigheaded as he was. And not for putting reason before faith, but for claiming that his theories, which were based nearly as much on complete woo-woo nonsense as on scientific observation, invalidated religion completely and the Roman Catholic church specifically; and used it as the basis for rather viscious attacks on the church. His support for his theory contained several major errors, which other scientists of his time had noticed and therefore failed to support him because of it. (For example, Galileo completely denied the effect of the moon on tides; a theory supported by Kepler and most of Galileo's contemporaries).

Pope Urban VIII was also not the most reasonable of me, and reacted poorly to Galileo's attacks. Geocentrism, while Urban's preference, was not official church doctrine, the pope was not speaking ex cathedra, and his actions were highly controversial within the church.

Although heliocentrism was dismissed, and even prohibited, for a short time; it was never done so ex cathedra, only by the personal preference of a couple of popes. The church at the time was, in fact, one of the strongest proponents of the physical sciences, and many prominent cardinals not only accepted, but actively promoted Copernican heliocentrism.

Rather than persecution of the noble scientist standing up to the repressive religion; it was more or less just a clash of a**holes who were both more concerned with personal agendas than truth.

Yes, I read all about his murky dealings in The Sleepwalkers. He brought the whole thing on himself. The cantankerous old fart.

[Thing]

A difficulty I find with science, in UK universities at least, is that it's managed by other scientists who have worked their way up through a mixture of ability, luck, patronage, personality defect, workaholism etc but are firmly convinced that they got there through shear merit. They therefore treat the people they are managing as inferior versions of themselves who have failed to get where they (the managers) now are because they're not clever or hard-working enough.

I'm currently reading 'Hard facts, dangerous half-truths and dangerous nonsense' by Jeffrey Pfeffer and Robert I Sutton, which is about evidence-based management. The more I read the more it strikes me that scientists, whom one might expect to naturally instinctively take up evidence-based management without even being told, instead do anything but.

[lenny]

Peer-review as it currently stands is crap. Frankly. And I say that as author and reviewer. that sounds like American English to me; in British it would be something like:

Peer-review as it currently stands is somewhat lacking.
(And I say that as author, reviewer and editor.)

but who is interested in improving it? like the media, many journals are forced into a position of selling journals, that introduces many pressures.

reviewers get annoyed when they invest time explaining the errors in a manuscript, twice, (for different journals) and then see it appear in a third, with even notation (and boarder-line typo's) uncorrected.

one could, for example, require a pgp encrypted tag on all manuscripts; this would not bias the review process, but IF the manuscript was accepted then all previous reviewers could be notified immediately, and their documented reviews could be published and/or posted.

i would not want to bias the review process, but i would want authors to be humiliated, if indeed they deserved to be.

that said, it may not be in the business interests of Nature, Science or PRL to do this. why should they devote resources to this? (association journals might jsutify this). we ("science") may want fewer papers, better papers: but who has this as a business interest?

[lenny]

A difficulty I find with science, in UK universities at least, is that it's managed by other scientists who have worked their way up through a mixture of ability, luck, patronage, personality defect, workaholism etc but are firmly convinced that they got there through shear meritwhich field?

maths/stats/physics seems to be mixed (to an american eye); some hold that their success is due to merit, others are refreshingly insecure and well aware of the mixture you specify. what is it like in the states these days?

[Tez]

I quite like the idea of the H-index:
arxiv.org/PS_cache/physics/pdf/0508/0508025.pdf


The H index is fine for comparing people within a field. But where I am they are talking about adopting it as a measure across the whole physics department which I think would be unfair to people who work in smaller areas and therefore generate less citations.

[Tez]

If I was in a more secure position, personally, I would be very tempted to publish only in OA journals. As things stand, however, that would have a detrimental effect on my efforts to secure a permanent job. Which is one of the reasons for my long whinge.

In my experience you almost never get an academic job "blind" - i.e. by just applying somewhere no-one knows you personally but they look at your publications and want to interview/hire you). People get hired where they are known personally - often where they were a student or postdoc. This is not without reason - hiring someone is picking a collegue for the next 30 years or so. My biggest advice would be to ensure you are talking to the right people, and doing so frequently. Published papers are important too of course, but there are people I know who have been hired based primarily on work which appeared on a preprint arxiv (www.arxiv.org). Certainly the things I was most known for up to a couple of years ago were never even submitted to a journal.

[Tez]

A difficulty I find with science, in UK universities at least, is that it's managed by other scientists who have worked their way up through a mixture of ability, luck, patronage, personality defect, workaholism etc but are firmly convinced that they got there through shear merit. They therefore treat the people they are managing as inferior versions of themselves who have failed to get where they (the managers) now are because they're not clever or hard-working enough.

Well, I sure hope that isnt a valid description of me!

I tell people frequently that I was extreeeeemely fortunate to get a permanent position where I am. First off I was lucky that on the interview panel (consisting only of hard-core experimentalists) for a crazy fellowship application I wrote (that normally goes to applied physicists and I wrote an application based around foundational issues in quantum mechanics) there was one guy who pushed hard for me. Reasons unknown. The fellowship gave me funding for 5 years. Then a year after I arrived one of the senior faculty left for a professorship elsewhere. In hiring a replacement the head of the department managed to swing the funding for three new junior people to replace him, as long as two of them had fellowship money for about 5 years worth. There was another guy (better than me) who was also on a fellowship, so I was damn lucky he swung the funding for both of us. And so on.

Sometimes I get a kind of "survivors guilt" because I see many young postdocs who are better physicists then me struggling to get a job. It seems that persistence is actually the main thing a postdoc needs (I was nearly 7 years out of my PhD before getting tenure, a guy visiting me at the moment is 11 years out and only last week got offered a tenure track job!).

[LordoftheLeftHand]

My beef with science and scientists: Lab coats creep me out!

LLH

[Cynric]

My beef with science and scientists: Lab coats creep me out!

LLH

Psst! It's OK. Real scientist's don't wear them.

But don't tell Health and Safety ;)

[Cynric]

that sounds like American English to me;

That, Sir, is fighting talk! :D

(Damn Yankees must have infected my fragile little mind with their straightforward manner, and colourful idioms)


in British it would be something like:

Peer-review as it currently stands is somewhat lacking.
(And I say that as author, reviewer and editor.)

If, by British, you mean the Queen's Good English, then I submit that your alternative phrasing would only be true of a document written by a civil servant wishing to disparage peer-review in the most robust terms.

However, it is true that the blessed anonymity of the forum does tend to make one more forthright than usual...


but who is interested in improving it? like the media, many journals are forced into a position of selling journals, that introduces many pressures.

reviewers get annoyed when they invest time explaining the errors in a manuscript, twice, (for different journals) and then see it appear in a third, with even notation (and boarder-line typo's) uncorrected.

one could, for example, require a pgp encrypted tag on all manuscripts; this would not bias the review process, but IF the manuscript was accepted then all previous reviewers could be notified immediately, and their documented reviews could be published and/or posted.

That's a nice idea. Publish the spectrum of reviews as supplementary material with a paper. Although, it's probably not in the best interest of editors taking a chance on a borderline paper.

With respect to notifying reviewers, this seems to be coming in with the advent of electronic submissions. I usually get an email when a decision has been reached on a paper I've reviewed nowadays, and can read the other referee reports.

Regarding my earlier point, have you ever, as an editor, been tempted to send a paper you disliked to a kiss-of-death reviewer?

[lenny]

That's a nice idea. ... With respect to notifying reviewers, this seems to be coming in with the advent of electronic submissions. I usually get an email when a decision has been reached on a paper I've reviewed nowadays, and can read the other referee reports.for submission to a first journal that works fine, but if the manuscruipt is rejected and then submitted to a second journal the thread is ?always? broken in the current system. the paper may then appear, uncorrected, in the second or third journal; this need not be withouth penalty for the authors, or reward to those who supplied thoughtful reviews along the way.

[lenny]

Regarding my earlier point, have you ever, as an editor, been tempted to send a paper you disliked to a kiss-of-death reviewer?no, but i have used kiss-of-reality reviewers. early on i read each paper closely enough to ask for clarification with a few pointed questions myself if i thought something was just plain wrong; like/dis-like was not much of a factor.

i'd also like the scoring stats of reviewers to be held somewhere, for papers but much more importantly for grants. panels know that some reviewers always give low scores and provide insightful comments for the proposals they like or hate; naive panels who rely on some simple average score for pre-sorting the proposals (or score based administrative decisions made before the panel even sees the proposal) can have a very negative impact.

[TruthSeeker]

for submission to a first journal that works fine, but if the manuscruipt is rejected and then submitted to a second journal the thread is ?always? broken in the current system. the paper may then appear, uncorrected, in the second or third journal; this need not be withouth penalty for the authors, or reward to those who supplied thoughtful reviews along the way.

I'm in a small enough field that I've been asked to review the same manuscript more than once as it makes its way down the impact factor scale. When that happens, I let the editor know I've reviewed it previously and whether or not it has changed.

Also, I know of colleagues who send their negative reviews from previous submissions as "letters to the editor" when an uncorrected paper appears.

[Tez]

no, but i have used kiss-of-reality reviewers. early on i read each paper closely enough to ask for clarification with a few pointed questions myself if i thought something was just plain wrong; like/dis-like was not much of a factor.

i'd also like the scoring stats of reviewers to be held somewhere, for papers but much more importantly for grants. panels know that some reviewers always give low scores and provide insightful comments for the proposals they like or hate; naive panels who rely on some simple average score for pre-sorting the proposals (or score based administrative decisions made before the panel even sees the proposal) can have a very negative impact.

I know the American Physical Society keep all sorts of scoring stats on reviewers. They are quite open about doing it (and what stats they keep), though I dont think they'll actually give you them! But internally across their many journals they're available to all their editors I believe. Grant reviewing is much more important, it would certianly be useful to have such stats (though as you say, in most fields you get to know the people and how cranky they are...)

[JamesM]

Reviewing papers can be such a pain, if I get sent the same paper twice, where my recommendations have been ignored, I generally just tell the editor and offer to send my original review. I suppose it would be more vindictive to agree to review and then just submit "I thought this was crap the first time I saw it, and it hasn't got any better the second time round".

A variation on the H index, m, which supposedly measures how 'hot' the field you're working in is: arXiv link (http://www.arxiv.org/abs/physics/0604216).

[Ben Best]

The present senior faculty in universities did not face this climate to anything like the same extent. Many who have made huge breakthroughs took a good length of time to develop their ideas, and the full significance of their discoveries only became apparent years after publication. I fear that they will be replaced by those scientists most adept at the flashy “presentational” side of scientific discovery, rather than the slow, steady accumulation of evidence that leads to confident advancement in knowledge. This could lead to an analogous situation to the erosion of quality evident in humanities departments, where relativism has ousted predictive models for analysis of language. After all, grand claims and slapdash experiments are easier than real research.


There are real grounds for your concern, but I think that you overstate the case in alarmist terms. Science has objective reference that is harder to define in the humanities. There is only so much "flash" you can present in science in the absence of content.

[lenny]

There are real grounds for your concern, but I think that you overstate the case in alarmist terms. Science has objective reference that is harder to define in the humanities. agreed, but isn't the question (and part of Cynric's worry) one of (a) how we quantify this "objective reference" for young researchers and (b) the time scale on which we can sort out who is serious from who is mere flash (or even more worrying, are we using evaluation procedures which push the potentially serious towards merely being flash?).There is only so much "flash" you can present in science in the absence of content.agreed. but once flash has "tenure" identifying it as such does little to advance science. no?

[neil]

Most spelling and grammar errors do not cause misunderstanding, but on rare occasions a disaster can occur, so we should ask for clarification if the matter is possibly important. Since the big success that President Bill Clinton had with lies, half truths and false inferences, things have been going down-hill. We need to maintain real integrity and honesty, not just the appearance. Neil

[Cynric]

agreed, but isn't the question (and part of Cynric's worry) one of (a) how we quantify this "objective reference" for young researchers and (b) the time scale on which we can sort out who is serious from who is mere flash (or even more worrying, are we using evaluation procedures which push the potentially serious towards merely being flash?).agreed. but once flash has "tenure" identifying it as such does little to advance science. no?

This is a big one for me. I feel it's insidious influence, and I know colleagues do too. We've starting thinking about doing experiments in terms of forming the next paper, rather than answering the next question. IMO thinking in those terms (i.e. every set of experiments is a "story") is not very sound, and leads to the trend where scientists set out to prove their new hypothesis rather than test it.
Not flash, necessarily, but they are usually grand hypotheses.

[Cynric]

no, but i have used kiss-of-reality reviewers. early on i read each paper closely enough to ask for clarification with a few pointed questions myself if i thought something was just plain wrong; like/dis-like was not much of a factor.

That's reassuring. Perhaps I am getting a little paranoid.

The wolf's at the door, though.

Gah! What am I doing wasting time on a forum? :jaw-dropp

;)