Scientific illiteracy.

[delphi_ote]

Dunning's latest episode of Skeptoid really upset me. Specifically here:
I tend to be cautious of claims coming from sources dedicated to promoting them. The scientific method starts with a null hypothesis, not with a preconceived notion to justify; and that process invariably produces data that do not support the conclusion, and theories tend to change over time as a result.
If this were true, we should be skeptical of Darwin, Hawkings, Newton, and Einstein! They all had ideas that they very much believed to be true without direct evidence. They later proposed or conducted experiments and research to prove those ideas. The idea that you should be skeptical of a scientist devoted to an idea is laughable. Darwin spent years of his life tracking down data to support his evolution by natural selection hypothesis. He was very much dedicated to proving this hypothesis to be true. He was also very much devoted to promoting his hypothesis. Investing in an idea is the very core of good science.

Dunning's caution targets the scientific method itself. Grade school science class taught us that science begins with a hypothesis, not a null hypothesis. It begins with an idea we suspect might be true. This is "a preconceived notion to justify." Look up the definition of the word: "a tentative theory about the natural world; a concept that is not yet verified but that if true would explain certain facts or phenomena."

A hypothesis is the formal representation of our inspiration. It's what leads us us to test our idea repeatedly in an attempt to disprove it. This is where the concept of a null hypothesis enters the picture. Once we know what we want to prove, we try to design tests that isolate the measurement of our hypothesis from everything else. Noise and random chance and expectation and error are all lumped together into one big counter-hypothesis. The outcome of our experiment is either, "Maybe there's something to my idea," or "Nope. Probably some other explanation." The null hypothesis is sort of an intellectual foil for our hypothesis. It is a very important concept, but it only has meaning in light of the alternative hypothesis.

I've been feeling a bit ill of late about scientific illiteracy in self-proclaimed spokespersons for the "skeptical movement." It makes me feel especially upset in light of our intolerance for scientific illiteracy in the general public. This kind of mistake is extremely serious. The premise of skepticism is that we apply the scientific method and logic as best we can to everyday life. Dunning is clearly not an expert in the scientific method, but he picks up a microphone every week and tells people they should live by that method and excoriates those who do not. This makes me very uncomfortable identifying myself as a "skeptic."

[Bob Klase]

Grade school science class taught us that science begins with a hypothesis, not a null hypothesis. It begins with an idea we suspect might be true.


Science doesn't begin with a hypothesis at all. It begins by asking a question or making an observation(s). Then comes research to gather data related to the question or observation. Then a hypothesis is formed that is consistent with the and research data.

So it's true that it doesn't begin with a null hypothesis- it begins with a hypothesis that seems to explain the observations.

[Upchurch]

If this were true, we should be skeptical of Darwin, Hawkings, Newton, and Einstein! They all had ideas that they very much believed to be true without direct evidence. They later proposed or conducted experiments and research to prove those ideas.
Could you expand a bit on what you mean by this? Hypothesizing is not the same thing as belief without evidence. In fact, theoretical physics of the kind Einstein and Hawkings practices are very much based on physical evidence and then extrapolated out using known principles. Einstein, specifically, used known data (the orbit of Mercury, for example) that was inconsistent with Newtonian physics to correct Newtonian physics.

The stuff that was predicted and proven later (i.e. gravitational lensing) were logical consequences of their ideas. Einstein would have been the first to reject his own theory if it had been shown to be wrong.

No amount of experimentation can ever prove me right; a single experiment can prove me wrong.

Newton was a slightly different beast, but he did have a large body of data compiled by Tycho Brahe and the work of Johannes Kepler that he worked from.

I can't speak much about Darwin's history, but I do know he spent many, many years collected data before publishing his ideas.

[tkingdoll]

I don't really have any comment about the first part of your post, because I'm the last person qualified to talk about scientific method.

However, this bit interests me:

I've been feeling a bit ill of late about scientific illiteracy in self-proclaimed spokespersons for the "skeptical movement." It makes me feel especially upset in light of our intolerance for scientific illiteracy in the general public. This kind of mistake is extremely serious. The premise of skepticism is that we apply the scientific method and logic as best we can to everyday life. Dunning is clearly not an expert in the scientific method, but he picks up a microphone every week and tells people they should live by that method and excoriates those who do not. This makes me very uncomfortable identifying myself as a "skeptic."

I can sympathise with that somewhat, not specifically about Brian as I don't listen to the podcast, but I don't like any sort of dogma attached to my skepticism and I think even phrases like "skeptical movement" are a bit of a misnomer. There are communities of people who have common interests, but often I observe that that common interest is an anger towards anything or anyone who believes something daft. Which is fine, I get that way myself often, but that can easily lead to a position of 'no compromise'.

Unfortunately, a palatable argument has to be one softened by compromise, which is why so much science is reported badly in mainstream media. So to have a holier-than-thou attitude like the one you suggest Brian extols isn't much use to the general public, and will only win over those who already feel themselves intellectually superior. But outside of that narrow margin, there is no 'movement'. Organised skepticism in whatever incarnation isn't going to have an impact on society at large unless it gets some whopping finance behind it and gets down from the ivory tower. The latter is happening a bit though, and the reason it's happening is because younger, non-science-background folks are getting involved, and that means a watering down of the dogma. This can only be a good thing. Podcasts reach a much wider audience than print and are free, so again, that's a good thing and a valuable resource. But a watering down of the dogma, reaching for a more palatable skeptical message, carries a risk, and that risk is a miscommunication or even a misunderstanding of the science. This is bound to happen when non-scientists talk about science. Again, see the media for details.

But the best thing about many skeptics is that they're willing to say "I was wrong" or "I learned something new and changed my opinion". So, rather than being an indication that something is rotten at the heart of skepticism (I don't think it is), perhaps this is simply an opportunity for a skeptical podcaster to share his learning with his audience?

I don't know. Like I say, you could write about my practical experience of the scientific method on the back of a postage stamp. With a thick pencil. In CAPS. Does that mean I'm not a skeptic? No. Does that mean I can't talk about science? No. It does mean I should be careful what I say about science, and I generally am, but heck, everyone screws up once in a while. I recently wrote a post on Skepchick and managed to misread a really important number during my research (it was tiny print, dammit), screwing up the maths of the entire post. Doesn't mean I don't know anything or shouldn't talk about numbers, though. It just means I had to fess up, correct my error, and hope that everyone doesn't think I'm a jerk.

As for self-identifying as a skeptic, I tend to think of it as a handy shortcut within communities like this one, but it has little relevance outside of that. Labels are silly. We're all just people who think stuff.

[drkitten]

If this were true, we should be skeptical of Darwin, Hawkings, Newton, and Einstein! They all had ideas that they very much believed to be true without direct evidence.

Huh? They -- all of them -- had lots of evidence for their ideas at the time they propsoed them.

Granted, much of the evidence was in the form of reason and argumentation. But that's still evidence. If you have facts about the world that someone else has found, but you think you have a explanation from first principles that explains those facts, you still have evidence supporting your explanation.

[articulett]

I'm with you Delphi-- I don't listen to dunning because I've noticed a couple of similar error.

We test the null hypothesis for sure-- we don't "start with it"-- When the data supports the hypothesis, you look for errors, you repeat, you test the null hypothesis, you attempt to falsify, you run double blind studies or use other methods to remove the possibility of error or bias, you subject it to pear review, you set up "if x is true then we should expect to see y tests"-- but you don't start with nothing. How can you design a test if you don't have some sort of hypothesis. We might not believe that psychic powers are true, that doesn't make them impossible to test. We might have a theory as to how the "magic is achieved" or how "noticing the hits" skews the results-- but that is not a "null hypothesis"-- that is a hypothesis which we can either gather data in support of or to falsify.

There's been a couple times when Dunning says stuff that isn't really up to snuff from my perspective.

Yes, we don't seek to confirm the truth we want... we follow the evidence where it goes unlike faith based belief systems--but that doesn't mean we start with a "null hypothesis" which means the opposite of one's hunches. We start with a hypothesis, and we seek to falsify it using the null hypothesis (to see if we can prove our theory wrong)-- but that is different than "starting with a null hypothesis" and different than starting with "no opinion" if that is what he meant instead.

[delphi_ote]

Could you expand a bit on what you mean by this? Hypothesizing is not the same thing as belief without evidence.
Gah! My browser ate my previous response. But yes. I can clarify.

First of all, I completely agree with your summarization of these great scientists' lives, and I appreciate that you summarized them so well. It will be handy to refer back to. Of course I did not mean to imply that a hypothesis was the same thing as belief. These great minds all rooted their thinking in established facts.

But as you said, these scientists had ideas that were extrapolations of known facts. Based on the facts at hand, they saw a hint of a much deeper truth. At the time, there was insufficient evidence to prove whether their inspiration was correct or naive. But they still dedicated years of their lives to the idea. Even the best of scientists have no way to know whether their hypothesis is true or not until it is tested. Even then, it needs to go through peer review and be subjected to further experimentation before the scientific community at large accepts the idea. Big ideas take a lifetime (sometimes many lifetimes) to prove or disprove. Mendel, for example, died long before his important work was accepted by the world. In fact, he was essentially brushed off by the scientific world. But he still woke up every day and continued his painfully slow and meticulous experiments. We're still conducting experiments to validate Einstein's ideas today.

This podcast was about Philip Zimbardo (http://en.wikipedia.org/wiki/Philip_Zimbardo). Brian Dunning said he was suspicious of Zimbardo because he dedicated his life to the notion that unethical behavior is a product of a person's environment. Dunning implied that a real scientist doesn't start with a notion he wants to prove is true. That is patently false. Zimbardo supporting his ideas through his life is perfectly acceptable and normal for a scientist. The community at large and further experimentation will decide if Zimbardo was mistaken, but there's nothing inherently unscientific about vigorously supporting and pursuing a hypothesis.

I'll give two modern examples of very good modern scientists who have dedicated significant parts of their lives to ideas that are not yet proven: Günter Wächtershäuser (http://en.wikipedia.org/wiki/G%C3%BCnter_W%C3%A4chtersh%C3%A4user) and Gerald Joyce (http://en.wikipedia.org/wiki/Gerald_Joyce). Both of them have ideas on how life started. Neither idea is proven (in fact, they're probably mutually exclusive) but both research and argue in favor of their pet idea. There's nothing wrong with this. It's how scientists work.

In the words of Thomas Henry Huxley, "It is a popular delusion that the scientific enquirer is under an obligation not to go beyond generalization of observed facts...but anyone who is practically acquainted with scientific work is aware that those who refuse to go beyond the facts, rarely get as far."

[Fnord]

If this were true, we should be skeptical of Darwin, Hawkings, Newton, and Einstein! They all had ideas that they very much believed to be true without direct evidence. They later proposed or conducted experiments and research to prove those ideas. The idea that you should be skeptical of a scientist devoted to an idea is laughable.


Being skeptical of scientific claims is one of the major foundation stones of the scientific method. Without a healthy dose of skepticism, any bearded old man with a Bachelor of Science degree could call himself a scientist, and make any claim he wished without fear of being refuted and losing his funding.

And note the phrase "... healthy dose of skepticism...". I use it here to imply that only the claims, the methods used to arrive at those claims, and the data used to support those claims should be examined, and not the religious, political, or personal beliefs of the claimant.

So, if another Darwin, Hawkings, Newton, or Einstein were to make a claim, that claim should be examined on its own merits, and not taken as The Truth merely because the new Darwin, Hawkings, Newton, or Einstein said so.

Leave blind faith in authority figures to the religionists.

[Quath]

I think we need to separate out how scientists work and how science should work. I think Einstein is a good example in this.

Early in his career, he wrote about Brownian motion as proof of the atomic theory of matter. He never mentions Brownian motion, but says in his paper that there should be a process that exhibits the properties that he describes. He didn't want to act like he was explaining an experiment (Brownian motion) with his theory. He wanted to act like he was doing the opposite. But really, the experiment came first, then Einstein's treatment of the theory and then further new experiments are proposed and tested.

Einstein did the same thing with special relativity. He had heard about the Michelson–Morley experiment and some other such experiments. However, he would sometimes claim he had not heard of them when he formulated his theory. He justified his theory by logical positivism. He said if we can not measure Absolute Time, we should not assume it exists. So he wrote his paper with the knowledge of experiment and with some philosophy he agreed with and claimed that there should be some experiments that demonstrate what he came up with. He proposed new experiments as well.

Much later in life, Einstein opposed Quantum Mechanics because it violated his personal belief system that the universe should be deterministic and look designed. He didn't like that QM showed weird results like the future affecting the past. His fellow scientists would quote back to him the idea of logical positivism which Einstein claimed led him to special relativity. His reply was that the same joke should not be told twice.

He fought long and hard for his belief system that the universe should appear designed and ignored and rejected the possibility of something else. Einstein was on the loosing side of this debate (though what he did was greatly valuable to science).

So science gets done even though the people are human and do not quite follow it as well as they should. There are self corrections in science that make it work despite the error prone attempts of humanity.

The way real science is suppose to work is you test while trying very hard to remove any bias. You may or may not succeed. If you don't succeed well enough, another scientist will eventually figure it out and your error will be made public. So as a scientist, I am very skeptical of another scientist pushing a vision because they probably have not done a good job removing bias. Some of the best scientists I have known will admit to their bias, remark on how they tried to adjust for it and then give the pros and cons.

[articulett]

I find Zimbardo a better scientist than Dunning. He has given us a lot of very interesting knowledge and is pretty good at devising tests for refining that knowledge.

[delphi_ote]

So as a scientist, I am very skeptical of another scientist pushing a vision because they probably have not done a good job removing bias.
This mentality drives me absolutely bonkers. If you remove all bias, you have nothing to test. Why is it we are so terrified of inspiration in this "skeptical movement"?

Sorry folks, but you have to have some courage and take some risks in science (and life.) You can't always be right. If you're going to brave new frontiers of knowledge, you have to step out on a limb and support an untested idea. You'll also have to watch that idea get torn to shreds on a regular basis (even if you're right.)

Science isn't just about logical purity. It also takes courage, vision, discipline, and dedication. We are, after all exploring. Scientists labor at the very edge of our understanding of reality. As Einstein said, "If we knew what it was we were doing, it would not be called research, would it?"

Einstein turned out to be wrong about quantum mechanics. He labored long and hard, because he thought the world behaved in a different way. All this 20/20 hindsight about the end of his career turns my stomach. It's tragic that he was mistaken, but he was following the same intuitions and genius that caused him to discover relativity. As I said before, we can't always be right about all things. Even great minds take risks when they go beyond what is already known.

Think again of Mendel alone counting wrinkled peas, pursuing an idea nobody else cared about or understood. He could've been wrong about it all. He could've been completely wasting his time. I'm sure he had doubts. But he kept counting. For years. Without any reward or recognition. And he changed the world.

The difference between the pseudo-scientist and the scientist is not that one advocates vigorously for an idea and the other is dispassionate. The difference is that the scientist tries to test their ideas and subjects them to critical review and the pseudo-scientist does not. The scientist takes a beautifully crafted labor of love and hands it to an insatiable mob to be ripped apart. The pseudo-scientist protects his dilapidated construct at all costs.

It's starting to seem to me that some skeptics would prefer not to build anything in the first place. That's not science. It's intellectual cowardice. It's clinging to safe shores and throwing rocks at everyone who gets in a boat. Are we really so terrified of being wrong once in a while that we shy away from people who are inspired?

[delphi_ote]

So, if another Darwin, Hawkings, Newton, or Einstein were to make a claim, that claim should be examined on its own merits, and not taken as The Truth merely because the new Darwin, Hawkings, Newton, or Einstein said so.

Leave blind faith in authority figures to the religionists.
There was nothing in my argument that was an appeal to authority. Maybe my point was buried a bit deeper in my post and you missed it. Sorry about that. My point was not that these are authorities beyond question. My point is that they're just solid examples of good scientists who labored and maintained inspiration in the face of uncertainty and criticism.

[articulett]

This reminds me of people who think that you should have no opinion as to someone's guilt or evidence in a trial. Just the fact that most people who go to trial have enough evidence to go to trial and the fact that most people tried are guilty, should lead most people to presume that the person is more likely guilty than not. But that doesn't mean that you can't listen to the evidence and change your mind or find that the state has not proven the case beyond a reasonable doubt.

You must be more careful and take extra precautions when you are aware of these tendencies and you must be careful that you don't have a vested interest in a particular answer or that you've taken measures to correct such things. All humans have biases-- we invented the scientific method and all sorts of error correcting mechanisms so that we might find out the facts in as unbiased a manner as possible. We hone the method so it's better and better at this. But we don't start out with no direction or idea or thought of what we think we might show or what we are trying to determine. If our hypothesis is correct, we can use it to learn more. If it's wrong, we can use it to eliminate possibilities to find out what is right.

But, like Delphi_Ote-- I think Dunning stated in incorrectly and confusingly... The scientific method is the best error correcting method we have-- the best method we have for getting at the truth despite our hopes and biases and so forth... but scientists don't start from the null hypothesis. Scientists start with a problem or question-- they develop a hypothesis and test it... testing the null hypothesis is part of the testing procedure. It isn't a starting point. I can't imagine any scientist saying it is.

I do teach science. I do teach the "scientific method". He mispoke in effort to show how science is not like religion I suppose. But his example was wrong and should be subject to the same skeptical scrutiny he's subjecting Zimbardo to.

[deoxy-2-ribose]

Stephen J Gould, an amazing scientist, is a perfect example of this topic. He proposed a new theory of evolutionary mechanisms that were different than the consensus. He viewed the fossil record and created a theory based on his observations. He then set about adding credibility to his theory (notice I did not say prove) by showing how it fits the data. He had an agenda. He worked to prove his notion that Punctuated Equilibria was a valid theory. He was also the first to admit that scientists are prone to following their own crusades, sometimes in the face of the evidence (Mismeasure of Man). This is what the peer review system is all about.

Scientists do start with a preconceived hypothesis most of the time. Good scientists then test this hypothesis and either prove it false, or it passes the first test and onward they go to more hypotheses and more experiments.

I think the original poster was correct in calling Mr Dunning on the carpet about this statement; however, he does advocate people doing their own investigations and tells us not to take his word for it, so I think his misstep is an honest one.

[Quath]

This mentality drives me absolutely bonkers. If you remove all bias, you have nothing to test. Why is it we are so terrified of inspiration in this "skeptical movement"?

Sorry folks, but you have to have some courage and take some risks in science (and life.) You can't always be right. If you're going to brave new frontiers of knowledge, you have to step out on a limb and support an untested idea. You'll also have to watch that idea get torn to shreds on a regular basis (even if you're right.)
There is nopthing wrong with persuing an idea, even one that you want to be true. But you have to work to remove bias. One example is the astrophysics community. Sometimes they want to observe a star based on the extreme limits of detection. The bias is to find a star or elements in a star. So to overcome this, they encode their data so when they go through it to remove outliers and bad measurements, they have no idea if it helps or hurts the claim they may have been hoping for. So they recognized their bias and they worked to remove that bias as best as they could.

Peer review would have also caught it eventually. But the goal should be to do good science. not sloppy science that will be cleaned up later on.

Another example that is not about science, but about evidence is Pres. Bush's belief that Iraq had WMDs. He had a bias and he selected all evidence that supported his bias and ignored evidence that went against it. Peer review among his staff had the same biases and resulted in the same result. Eventually experimental evidence in the form of war showed the bias. We need to encourage good results the first time around and not let crusades and biases lead us down dead end paths.

[bpesta22]

Lol on Gould being an amazing scientist.

It seems like even people here in this thread can't agree on what science is / is not, so maybe it's unfair to criticize Dunning.

For example, my understanding is that falsification has nothing to do with the reason we test nulls. I thought the reason was purely practical-- we know precisely the value the null predicts; 0 difference between groups.

Often, we don't know the exact size of the effect for the alternate hypothesis (and if we did, why do the experiment?). So, we don't test it directly, only because we don't know it's effect size. Instead, we assume the null is true, which lets us calculate the probability of the observed data precisely (given that we expect zero difference).

If the observed difference is improbable, then maybe the null is unreasonable, and so we reject it.

If the observed difference is not all that surprising, then we don't reject the null (which is different from accepting it, which I think is an important distinction here).

None of this is motivated by the fact that we can't prove theories; only falsify them. That's a completely separate issue. No amount of support for a theory proves it, and confirmed predictions are nothing more than faulty reasoning: if Theory A is true then B should follow. B follows, so Theory A is true-- that's the fallacy of affirming the consequent!

This is why we falsify, but again that has nothing to do with the null hypothesis. If Theory A is true then B should follow. Instead, Not B follows. Therefore Theory A is not true. This is the valid inference (modus tollens).

So, it's the logic of conditional inferences that force us to falsify, but it's apples and oranges from why we test nulls.

Am I right?!

[bpesta22]

ETA: I agree with Quath that we should separate the individual scientist from the method. For sure the individual is biased, and that's probably a good thing. But your whacky ideas won't be accepted unless the unbiased scientific method suggests they're merited.

[UnrepentantSinner]

You know, evolution is just a theory.

[Fnord]

You know, evolution is just a theory.


And a soufflé is just eggs. ;)