Brain Evolved Preference for Anecdotal Evidence

[skeptigirl]

Some of you know I have been an advocate for addressing the underlying causes of people's false beliefs and not to only address the false information they believe. Here is a brief comment from Shermer published in SciAm that adds to our knowledge base on the function of the brain that contributes to why people believe anecdotal evidence more readily than scientific evidence.

How Anecdotal Evidence Can Undermine Scientific Results (http://www.sciam.com/article.cfm?id=how-anecdotal-evidence-can-undermine-scientific-results)

I assume this is a no brainer for many of you. But I was wondering what solutions people thought might address this problem.

I have brought the problem to people's attention when trying to persuade them to think more critically. I have given them examples of the fallacy involved. But that isn't enough to convince many people that their personal experience upon which they drew a conclusion was not a reason to draw said conclusion.

There must be a more effective way to address this barrier to learning than just pointing it out to people.

[Ysidro]

What you need is an anecdote...

[paximperium]

"My flu was cured by taking Homeopathy Pill X" Vs. An analytic metaanalysis of homeopathic remedies Vs. placebo in the treatment of influenza

"Peter Poppoff saved Mrs. X from cancer." Vs. "Peter Poppoff is a fraud. More news at 11."

"You know that Sylvia Brown? She saved little Timmy from being kidnapped." Vs. "Eeew...that James Randi is an atheist. Atheist are liars."

[skeptigirl]

What you need is an anecdote...

You know, that actually is a valid point. I do use anecdote examples to illustrate the fallacy when I point it out to people using an anecdote to justify an unsupportable belief. Trouble is, it is not their anecdote.

[skeptigirl]

"My flu was cured by taking Homeopathy Pill X" Vs. An analytic metaanalysis of homeopathic remedies Vs. placebo in the treatment of influenza

"Peter Poppoff saved Mrs. X from cancer." Vs. "Peter Poppoff is a fraud. More news at 11."

"You know that Sylvia Brown? She saved little Timmy from being kidnapped." Vs. "Eeew...that James Randi is an atheist. Atheist are liars."One example of what I was thinking of comes to mind. I had a long discussion with a friend trying to explain to her why the fact that she had some symptoms at some point after getting a flu shot did not mean the flu shot caused the symptoms.

These kind of personal experience, temporal events in particular are the hardest to counter.

With examples like your last one, the issue is more one of selective credibility assigning to our various sources of information. There is a different process involved in simply choosing which sources are credible and which aren't.

[paximperium]

One example of what I was thinking of comes to mind. I had a long discussion with a friend trying to explain to her why the fact that she had some symptoms at some point after getting a flu shot did not mean the flu shot caused the symptoms.

These kind of personal experience, temporal events in particular are the hardest to counter.

With examples like your last one, the issue is more one of selective credibility assigning to our various sources of information. There is a different process involved in simply choosing which sources are credible and which aren't.

Good point. People tend to trust their own senses but our senses and brains are easily fooled especially when it comes to correlation/causation fallacies.

The best we can do is to teach others that our senses are fallible and our brains are easily fooled...that may awaken them to learning and using the scientific method in their lives.

PS: Well the flu shot can cause muscle aches and pains and can cause a brief fever...a mini-flu.

[Wowbagger]

It is not enough to point out the fallacy of anecdotal evidence, because other weaknesses of the human mind evolution take effect, such as cognitive dissonance, and sunk cost strategies, etc.

[cj.23]

It's not a no brainer for me. Actually I find it hard to agree with much of it.

Well a personal experience is by definition NOT anecdotal: anecdotes are not necessarily anecdotal.

Nor actually is Shermer's first example; that is a false correlation. Correlation does not require causality, so the vaccine is unrelated to the onset of autism, though they do correlate - and as correlation often does mean causality, it is a failure to understand that it that the correlation is false that causes the problem. However, it is nothing to do witrh anecdotes, or anecdotal evidence! It's actually misapplied logic and bad science. :)

Again Wigmore seems to have engaged in false correlation, based on empirical evidence but utterly misleading. She presented her evidence, but it was nonsense - she was wrong. (Whether she could have told that at that period in history I know not, given the complexity of the biochemistry involved.) However her methodology seems to have been observation and then a false inference, with strange religious ideas supporting this mistaken chain of reasoning. It is not as I understand it however anecdotal?

And finally, anecdotal reasoning in no way undermines science: just the public acceptance of science? It just undermines faith in scientific and presumably efficacious cures, among those who would prefer to pursue chimerical junk? Science continues quite happily?

I like the evolutionary logic that false positives having less danger than false negatives are usually ignored though, and as always I have a great deal of respect for the excellent Michael Shermer, but I can't make much sense of why the original article was retitled as it was.

cj x

[cgallaga]

It may be confounded by the apparent fact that conscious processing takes place after the facts. So people quickly make up their mind about what happened - after the fact - and then are certain that is what happened at the time it happened, not recognizing that it is really what they consciously decided after the fact.

ETA in an odd twist, it seems to me that scientific thinking is trying to circumvent a part of the mechanism of natural selection. That anecdotal rapid decision and action, without knowledge is probably part of what evolved us to the point where we could start thinking about things more analytically. Now we are trying to over ride that in favor of this thoughtful analysis. Its kind of like dieting, for the mind.

[cj.23]

Actually I'd better check this - maybe it's a UK/USA thing?

My UK understanding of anecdotal evidence is "hearsay, evidence given that is not direct witness testimony; or the result of scientific research undertaken to demonstrate a claim"

Does it mean something different in the US?

cj x

[paximperium]

Actually I'd better check this - maybe it's a UK/USA thing?

My UK understanding of anecdotal evidence is "hearsay, evidence given that is not direct witness testimony; or the result of scientific research undertaken to demonstrate a claim"

Does it mean something different in the US?

cj x

an·ec·dote Audio Help /ˈænɪkˌdoʊt/ Pronunciation Key - Show Spelled Pronunciation[an-ik-doht] Pronunciation Key - Show IPA Pronunciation
–noun
a short account of a particular incident or event of an interesting or amusing nature, often biographical.
http://dictionary.reference.com/search?q=anecdote&x=0&y=0

It often meant here as a personal story or experience.

[cj.23]

an·ec·dote Audio Help /ˈænɪkˌdoʊt/ Pronunciation Key - Show Spelled Pronunciation[an-ik-doht] Pronunciation Key - Show IPA Pronunciation
–noun
a short account of a particular incident or event of an interesting or amusing nature, often biographical.
http://dictionary.reference.com/search?q=anecdote&x=0&y=0

It often meant here as a personal story or experience.

Yeah, that is an anecdote here too, usually a personal experience related, though not necessarily. Hence my assertion anecdote is not necessarily equal to anecdotal evidence, as a personal anecdote related by a victim of a crime would be admissible in court, being a short statement of their own direct experience - whereas anecdotal evidence, as in "I heard on the bus that Bob saw the UFO" is not?

Well anecdote means the same in the uK and USA anyway! Thats good to know. Cheers!

cj x

[Roboramma]

Re: the OP:

One time I was talking to someone, and I presented really good, solid evidence for what I was saying, but he didn't believe me. When I asked why, he told me a story about his grandmother.
So I think you're right.
:P

[skeptigirl]

Good point. People tend to trust their own senses but our senses and brains are easily fooled especially when it comes to correlation/causation fallacies.

The best we can do is to teach others that our senses are fallible and our brains are easily fooled...that may awaken them to learning and using the scientific method in their lives.

PS: Well the flu shot can cause muscle aches and pains and can cause a brief fever...a mini-flu.Paxi, I hope that last line was sarcasm.

With the exception of very young children, research indicates it is a myth. (http://archinte.ama-assn.org/cgi/content/abstract/156/14/1546)

[skeptigirl]

It is not enough to point out the fallacy of anecdotal evidence, because other weaknesses of the human mind evolution take effect, such as cognitive dissonance, and sunk cost strategies, etc.So if it is not enough, is our only hope then to wait for the old to die out and hope to reach the young at an earlier age teaching them how to overcome the brain's shortcoming? :p

[robinson]

I have brought the problem to people's attention when trying to persuade them to think more critically. I have given them examples of the fallacy involved. But that isn't enough to convince many people that their personal experience upon which they drew a conclusion was not a reason to draw said conclusion.


That is an example of anecdotal evidence. "I brought the problem, I gave them examples, It wasn't enough". Somebody tells a short account of their personal experience, and concludes that they did something to try and change another persons mind, but it didn't work. Or rather, it wasn't enough, I need to do more.

Whatever. We don't always recognize when we do the same thing we find flawed in others.

Like starting off a thread about anecdotal evidence with anecdotal evidence. I would suspect deliberate irony, to bring about a humorous response, but that might be giving too much credit.

It does show how deeply ingrained humans are when it comes to observing what happens, and foolishly believing what they observe counts as evidence. They must be trained, taught, to doubt their own experience, and only trust experts who do experiments and use science to determine reality.

Especially parents. They see their kid fall out of a tree, and they assume the injuries that appear right after the event, were caused by the fall. They don't realize that until proper experiments and evidence are provided, they can't know that it was the fall.

Kids at that age often have broken arms, head trauma, and bleed profusely, just because one thing happened, then the injuries appear, doesn't mean they are connected.

Of course parents might come around and stop being so dumb, if we could only show them the evidence. Sadly, they probably would still insist the fall led to the injuries. Same for hot objects, sharp pointy things and electric sockets.

Stupid parents. They trust their own experience over an expert who was paid a lot of money to show that nothing we inject into a child ever causes any harm, it is always just coincidence that kids get sick and develop serious symptoms right after multiple injections.

I don't think anything can really be done about it. Except maybe jailing anyone who questions authority, and refuses to allow their kids to be injected.

Oh sure you have those Amish with there lack of autism. But it is just coincidence that they refuse all vaccines. Or it could be genetics. How do we know? Well, we don't,but somebody said it was genetics so that is good enough.

Same for that other group of religious nutjobs who refused all vaccines. Sure they have almost no autism, but again, it is just coincidence. Same for those isolated tribes that never get vaccinated.

It is either coincidence or genetics. It can't be anything anyone did, despite how it seems. Parents just need to shut up and quit rocking the boat.

Kids just naturally go from normal to autistic, at the exact same time they get a buttload of vaccines. Stupid experience, always getting in the way of theories and evidence.

Oh, and all those stupid Doctors and researchers that find a connection between vaccines and health problems, they are all stupid too. Like those European countries that banned mercury. They are so dumb. They think mercury might be hurting kids.

Ha! That can't be true, because everybody agrees it can't be true.

What? No no, trust me. If a scientist or a Doctor says something, it must be true. Unless it is one of those Doctors that says something we don't agree with, then they are wrong.

Oh sure we could do animal studies, inject a bunch of monkeys and chimps babies and mothers with vaccines and adjuvants to actually see if they can cause neurological problems, but since we don't have any evidence to show mercury or vaccines causes any problems, what would be the point? I mean, it said right there in the OP article that "On the one side are scientists who have been unable to find any causal link between the symptoms of autism and the vaccine preservative thimerosal". See? They looked, but can't find any evidence.

So what would be the point of doing research and actually testing the stuff out? Plus that kind of research is expensive and stuff.

Oh sure, you might think adjuvants would be required to be tested before we use them, but we have been using them since 1947 or so,
so why worry about it now? Same for mercury. It has been used for a long time, without any testing, why would you want to actually do all that hard science to find out what it can do to developing babies and pregnant moms and stuff?

I mean, if you inject a bunch of pregant chimps and then inject the little chimp babies and study what happens, you might hurt the little chimp babies. Or worse, waste time and money. Because we already know the answer, we don't need to do any experiments, studies, all that scientific crap.

Scientist found no evidence, and that closes the case. If you look real hard, and there is nothing to show any evidence, why would you want to do animal studies and all that expensive and time consuming really hard stuff? If you already know, you don't have to do anything.

Besides, they stopped using mercury, except for when they still use it (every year, but only for babies and pregnant moms), because the Flu is dangerous, so mercury is OK then. But it can't be mercury, because we stopped using it. What? Something else in the vaccine? Don't be absurd. Next you will want extensive animal studies done on all vaccines and adjuvants and everything. Do you think we are made out of money?

Besides, testing all those combination's of vaccines would be a lot of work.


But anyway, people are just so dumb, they think they know there kids intimately, and can tell when something effects them and makes them sick. They just need to realize that Pharmaceutical companies and researchers never lie, and if they tell you something, it has to be true.

So Parents, stop believing your own experience, and start trusting the Government and the Drug Companies. They are only here to help you.

And stop asking for animal studies and all that crap. We told you, we found NO EVIDENCE!! Why can't you just believe? Why are you so skeptical of our claims? Trust us, we never hurt a single child, and we are never wrong about stuff we give kids and moms.


And if you say different, you are wrong. Because we said so.

[INRM]

One should note that not all anecdotes are wrong, and while anecdotes can lead to superstitions which can even spawn religions and all sorts of strange beliefs, there are actually other reasons which produce and lead to religious thought as well (the desire to believe in things bigger than ones-self, to trust their parents, and even due to neurotransmitter responses to comforting thoughts)... still, I wouldn't give anecdotes too much weight all the time anyway.


INRM

[skeptigirl]

Yeah, that is an anecdote here too, usually a personal experience related, though not necessarily. Hence my assertion anecdote is not necessarily equal to anecdotal evidence, as a personal anecdote related by a victim of a crime would be admissible in court, being a short statement of their own direct experience - whereas anecdotal evidence, as in "I heard on the bus that Bob saw the UFO" is not?

Well anecdote means the same in the uK and USA anyway! Thats good to know. Cheers!

cj xThere may be a difference in what I am talking about and what you are talking about if you consider legal definitions of evidence. I am talking about evidence one uses to draw conclusions. The admissibility of evidence in a court is a completely different matter.

Personal experience is a more precise term for what I am referring to. 'Anecdotal evidence' is the scientific term for a personal experience one uses as evidence.

One can also believe such personal anecdotes by proxy such as the grandmother's anecdote that Roboramma described in the above post. The 911 truthers that believe the anecdotes of witnesses have more significance than the testimony of structural engineers would also be considering anecdotal evidence (or in that example, eye witness testimony). On the other hand, 911 truthers who prefer a rogue structural engineer's testimony over an 'establishment' structural engineer is not an example of what I am talking about, even though in those two examples, trust one assigns to different sources is also a factor.

Some people have a very hard time believing that the conclusions one draws from either their own or someone else's personal experience is subject to all sorts of errors. Take the flu shot example. Lots of people get sick after flu shots. We give flu shots during the time of the year when commonly acquired infections are the most frequent. Kids return to school. The few children passing things to an occasional neighbor, cousin, or sibling over the summer are suddenly in confined classrooms with dozens of kids. Then those kids go home and infect their parents who by October are passing the infection on all over the community.

The conclusion the flu shot was the cause of the illness is common but mistaken conclusion. How do we know? Using the scientific method rather than basing assumptions on uncontrolled experiences, we take a large enough group of people, blindly randomize them into two groups (or 3 if you want a no-treatment arm of the study) and give half a flu shot and the other half a shot of saline. Then you collect the data on who got sick and then compare the group who got the vaccine with the group(s) that didn't.

Lo and behold the rate of illness is the same in both groups as was found in the study I cited in a previous post.

Scientific method leads you to a valid conclusion. Anecdotal or personal experience in this case leads you to a false conclusion. Your personal experience was unable to control for other variables affecting the outcome while the scientific method was. But some people are unable to recognize that fact even when presented with clear incontrovertible evidence. That personal experience was so convincing they cannot overcome its influence.

[skeptigirl]

That is an example of anecdotal evidence. "I brought the problem, I gave them examples, It wasn't enough". Somebody tells a short account of their personal experience, and concludes that they did something to try and change another persons mind, but it didn't work. Or rather, it wasn't enough, I need to do more.No, it is not an example of anecdotal evidence. Simply conveying information is not what is meant by the term, anecdotal evidence.



....
It does show how deeply ingrained humans are when it comes to observing what happens, and foolishly believing what they observe counts as evidence. They must be trained, taught, to doubt their own experience, and only trust experts who do experiments and use science to determine reality.This certainly illustrates how trust in sources of information can be affecting attempts to teach people the principles of critical thinking.



Especially parents. They see their kid fall out of a tree, and they assume the injuries that appear right after the event, were caused by the fall. They don't realize that until proper experiments and evidence are provided, they can't know that it was the fall.

Kids at that age often have broken arms, head trauma, and bleed profusely, just because one thing happened, then the injuries appear, doesn't mean they are connected.... I have pondered the fact we can trust that swinging a hammer into a glass window and watching it break is enough to draw a reliable conclusion while getting sick after a flu shot is not. Just because some personal experiences are valid evidence for conclusions doesn't mean all personal experiences are.

After all, some of our conclusions likely had to be correct in order to provide a selection pressure for a brain function that looked for causal relationships. As Shermer hypothesized in the OP piece, there was a benefit when the conclusion about the causal effect was correct (that bear is going to eat me if I don't get out of here) and not as much of a negative effect when the conclusion about the cause was not correct (if I perform this prayer ritual the harvest will be better).

In the case of immediate trauma after a high force impact, we know from experience those events are causal. In the case of the flu shot causing illness symptoms, the assumption of causality is made without consideration of the other potential variables and with only false beliefs there is experience supporting the assumptions.



I don't think anything can really be done about it. Except maybe jailing anyone who questions authority, and refuses to allow their kids to be injected.

....Off topic ranting serves no purpose here.



What? No no, trust me. If a scientist or a Doctor says something, it must be true. Unless it is one of those Doctors that says something we don't agree with, then they are wrong.That is also not consistent with critical thinking. It is a straw man no one is advocating.



Oh sure we could do animal studies, inject a bunch of monkeys and chimps babies and mothers with vaccines and adjuvants to actually see if they can cause neurological problems, ....More off topic useless ranting. Take it elsewhere.

Goodness, this post is full of a number of examples of both the trust issue and it's affect on critical thinking and the trust of anecdotal evidence over scientific evidence. It contains classic examples of both.

As I said, 'giving one an example', referred to presenting scientific evidence whether it be by simply explaining what evidence exists as I did above, or providing examples of research such as in the citation I referred to. An anecdote is not simply conveying information as you seem to be defining it as.

The trust issue is not something I want to spread this thread into covering. It is important enough that it deserves another thread. But one thing at a time please.

[krelnik]

There must be a more effective way to address this barrier to learning than just pointing it out to people.

What you need is an anecdote...

Exactly. I suppose this isn't exactly practical in every debate situation, but what about creating an anecdote for the person, that they can experience first hand, and then demonstrating later that what they saw was incorrect, or led them to a false conclusion?

Magic tricks come to mind, but I suppose some people might be turned off by that. "Oh, magicians get paid to fool people. How does that relate?" Plus there is the problem that not everyone is proficient.

I can't find the article, but recently I was reading something about false memories (I think) that was an experiment involving children to show how easy it is to get kids to believe false things. With a little simple storytelling, a moderator got most of room full of kids to believe that there was a magic animal hiding in a box that he had earlier demonstrated was empty.

Maybe we could brainstorm some scenarios in this thread.

[krelnik]

Thinking about what I just proposed some more, I think you would need some different scenarios depending on the nature of the anecdote you were trying to debunk.

For instance, many alternative medicine anecdotes are fundamentally based around a post-hoc ergo propter hoc fallacy. "I took this magic potion and my gout went away." That would probably the most fundamental scenario, and probably pretty easy to do. Just set up some false correlations in the room. Maybe prearrange with a confederate to turn the lights on and off on a pre-arranged cue?

But other anecdotes are built around selective memory situations, typically counting hits and ignoring misses. "Every time I drank the magic potion, this happened." Are you sure it was every time? Did you write them all down? This is a classic example of where challenging the person can be problematic, because many people will just assume you are accusing them of being a liar. A good scenario here would be handy and doable, I think.

Then there are perceptual ones, where someone thinks they saw something but they really didn't. For instance, someone who is synaesthesic seeing "auras" around people. Pareidolia and optical illusions might play into some of these. Many UFO reports are based on optical illusions. These are harder because there are so many of them, and each is a bit different.

Are there other ways anecdotes can get created that would require a unique scenario? It's late here, I'll sleep on it...

[GreyICE]

[INDENT][INDENT]I think I found a solution to this problem. No no, I'm serious this time. It is elegant, simple, and effective. It will not only soothe the irrational fears of parents, but silence the critics of vaccines once and for all.

Anybody can do it.

Vaccines are safe, and don't cause health problems for kids. But some people are skeptical, they claim all kinds of crazy things, blaming aluminum and other substances in vaccines. They also blame the number of vaccines, as well as suspecting mercury of causing neurological problems.

So, to show them how wrong they are, and to show how safe vaccines are, simply inject yourself with them, to show parents and doubters how safe they are. Okay. I volunteer. You pay any expenses and costs (since I'm apparently risking my life and sanity here), and I'll donate my time (no travel expenses, but I'm in NYC. Be creative). Give me a fairly balanced IQ test, general checkup, and overall wellness test. Then give me the recommended dosage for my gender, age, and body weight of vaccines, in a similar pattern to the one children receive. After all, we give out adult vaccines all the time.

Repeat said tests after an appropriate length of time has passed.

I bet we could find 20-50 volunteers for you. Easy enough, since you're paying all the costs.

Or we can just believe the people who do this.


I can already hear the responses. I'm not a child (sorry, that was a long time ago). One person isn't a large enough sample (duh). [Existing medical condition A], which I had in the first checkup, has gotten worse/not improved as much as expected/etc.

[skeptigirl]

Exactly. I suppose this isn't exactly practical in every debate situation, but what about creating an anecdote for the person, that they can experience first hand, and then demonstrating later that what they saw was incorrect, or led them to a false conclusion?

Magic tricks come to mind, but I suppose some people might be turned off by that. "Oh, magicians get paid to fool people. How does that relate?" Plus there is the problem that not everyone is proficient.

I can't find the article, but recently I was reading something about false memories (I think) that was an experiment involving children to show how easy it is to get kids to believe false things. With a little simple storytelling, a moderator got most of room full of kids to believe that there was a magic animal hiding in a box that he had earlier demonstrated was empty.

Maybe we could brainstorm some scenarios in this thread.Given the right circumstances, this is a perfect example. I have no doubt that most people who believed in astrology who had the good fortune to be in one of those demonstrations where everyone secretly got the same horoscope and were asked if it fit them, came away with a much better understanding of the cold reading trick than had someone just told them about it.

[skeptigirl]

Thinking about what I just proposed some more, I think you would need some different scenarios depending on the nature of the anecdote you were trying to debunk.

For instance, many alternative medicine anecdotes are fundamentally based around a post-hoc ergo propter hoc fallacy. "I took this magic potion and my gout went away." That would probably the most fundamental scenario, and probably pretty easy to do. Just set up some false correlations in the room. Maybe prearrange with a confederate to turn the lights on and off on a pre-arranged cue?

But other anecdotes are built around selective memory situations, typically counting hits and ignoring misses. "Every time I drank the magic potion, this happened." Are you sure it was every time? Did you write them all down? This is a classic example of where challenging the person can be problematic, because many people will just assume you are accusing them of being a liar. A good scenario here would be handy and doable, I think.

Then there are perceptual ones, where someone thinks they saw something but they really didn't. For instance, someone who is synaesthesic seeing "auras" around people. Pareidolia and optical illusions might play into some of these. Many UFO reports are based on optical illusions. These are harder because there are so many of them, and each is a bit different.

Are there other ways anecdotes can get created that would require a unique scenario? It's late here, I'll sleep on it...This is excellent. This is just the kind of brainstorming I was hoping for.

So we have the power of personal experience to consider, the selective credence given some anecdotes over others, and selective memory of correlation to consider.

[Soapy Sam]

Scenario:-
Three Stone Age hunters, X, Y and "M'Boggy", are up a mountain when a thunderstorm starts.
X and Y hunker down under a ledge.

Third one (young and a bit daft), dances in the rain, waving a spear and yelling abuse at the sky.

Third man gets struck by lightning.

Conclusions of his colleagues:-
1. Getting struck by lightning is to be avoided.
2.Getting struck by lightning may be caused by
a)Dancing around in the rain.
b)Abusing the sky.
c)Being called "M'boggy"
d)Using a cellphone

Q1:-Which of these conclusions is scientific?
Q2:- If the tale is repeated EXACTLY AS OBSERVED, is the tale itself anecdotal or a scientific report?
Q3:- If the tale is repeated EXACTLY AS OBSERVED, are conclusions a-d anecdotal or scientific?
Q4:- How many people have to be struck by lightning to constitute a statistically valid sample for analysis?
Q5:- Assume you are hunter X. As Shaman of the tribe, you are required to establish the actual threat of lightning . You are also required to spread the word to avoid further deaths. Do you:-
i)Found a research project?
ii)Tell a scary story?
iii)Suggest that dancing in the rain may be no riskier than hunkering under a ledge?
iv)Invoke the Gods of Planet X?

[Ivor the Engineer]

<snip>

Q4:- How many people have to be struck by lightning to constitute a statistically valid sample for analysis?

<snip>

This question highlights one of the problems of relying on Frequentist statistical analysis.

Real men are Bayesians.

[Cuddles]

This thread is not for discussing vaccines. If you wish to do so, either take it to one of the many that already exist or start your own.

[paximperium]

This question highlights one of the problems of relying on Frequentist statistical analysis.

Real men are Bayesians.

Exactly. There are no real absolutes.
Bayesian Probability Theorem is the way to go!!!

[skeptigirl]

Scenario:-
Three Stone Age hunters, X, Y and "M'Boggy", are up a mountain when a thunderstorm starts.
X and Y hunker down under a ledge.

Third one (young and a bit daft), dances in the rain, waving a spear and yelling abuse at the sky.

Third man gets struck by lightning.

Conclusions of his colleagues:-
1. Getting struck by lightning is to be avoided.
2.Getting struck by lightning may be caused by
a)Dancing around in the rain.
b)Abusing the sky.
c)Being called "M'boggy"
d)Using a cellphone

Q1:-Which of these conclusions is scientific?
Q2:- If the tale is repeated EXACTLY AS OBSERVED, is the tale itself anecdotal or a scientific report?
Q3:- If the tale is repeated EXACTLY AS OBSERVED, are conclusions a-d anecdotal or scientific?
Q4:- How many people have to be struck by lightning to constitute a statistically valid sample for analysis?
Q5:- Assume you are hunter X. As Shaman of the tribe, you are required to establish the actual threat of lightning . You are also required to spread the word to avoid further deaths. Do you:-
i)Found a research project?
ii)Tell a scary story?
iii)Suggest that dancing in the rain may be no riskier than hunkering under a ledge?
iv)Invoke the Gods of Planet X?

I think you are confusing the idea here of what is and what isn't scientific.

The science involved is making the observation and proposing an explanation. But if one then stops there and acts on the conclusion without further attempts to verify or replace it with a better conclusion, now you have left the scientific process and are operating in auto-brain mode. The conclusion could be right or wrong in either case. It is what you do next that delineates the scientific processing brain the from natural fallacious thinking brain.

It's just like skeptics thinking there is something inherently wrong with anecdotal evidence. That isn't so. It's how you use the anecdotal evidence that determines whether it is valid evidence or not.

Anytime we ask people to recall events such as interviewing people recently in the salmonella outbreak, we are using anecdotal evidence but within the scientific process. If an individual were to conclude, "the source must have been the restaurant because I never eat there and now I am ill" that would be a natural but unsupportable conclusion.

The public health investigators, OTOH, interview ill people and well people who ate with them. They collect the anecdotal evidence systematically. They observe a pattern. The ill people were much more likely to have eaten tomatoes than the well people. Then they look for a source of the outbreak to verify the hypothesis. They find no source. In the meantime more people become ill despite a sharp decrease in tomato consumption. The hypothesis fails. The investigators go back and re-evaluate the data and collect more data. It seems maybe the peppers are also a potential source. And on it goes.

In the meantime, the unscientific person remains convinced the source was the restaurant in spite of the fact the evidence does not support that conclusion. Both the public health investigators and the ill person used anecdotal evidence. The individual cannot see why his/her personal experience is not evidence of his/her conclusion.

[cgallaga]

In the meantime, the unscientific person remains convinced the source was the restaurant in spite of the fact the evidence does not support that conclusion. Both the public health investigators and the ill person used anecdotal evidence. The individual cannot see why his/her personal experience is not evidence of his/her conclusion.

And therein is the rub of purely anecdotal evidence. I'll wager that not one person who got sick from tomatoes will willingly go back to or recommend the restaurant they got sick in, same for the spinach of a few years ago. But they probably would be all to happy to take an organic farming tour to Mexico or California.

ETA: The pirnt being we evolved to act on anecdote, but science gives us a more accurate (though also more time consuming) method of knowing truth. So do you leap to dogmatic conclusion from your own anecdote, or do you perhaps use it to formulate a test, and then accept what empirical evidence from rigrous testing demonstrates to be most likely?

[cj.23]

There may be a difference in what I am talking about and what you are talking about if you consider legal definitions of evidence. I am talking about evidence one uses to draw conclusions. The admissibility of evidence in a court is a completely different matter.

Personal experience is a more precise term for what I am referring to. 'Anecdotal evidence' is the scientific term for a personal experience one uses as evidence.

Right, yeah I was thinking legally. Problem is that I have seen all observations and personal experience classed as anecdotal, and as such then statements like "my father is called Gunnar" become ancedotal and considered less valuable. Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline. Ditto Darwin. If he had ignored his personal observations in the Galapagos, and not then formulated theories based upon his own observations, no theory of evolution. And so on and so forth. Alll of modern science was developed with observations as a key criteria. So anecdotal clearly means something far more specific than "observation based" - though increasingly I see it misued that way - Sagan did it for instance - it means hearsay or unrelaible second hand testimony, the stuff of rumour. If Soapy saw the locjh Ness monster eating petunias in his living room tonight,and reported his experience, that would not be anecdoatal evidence. If I said Soapy had had that experience and i knew cos Larsen mentioned it to me while we were buying laptops, that would. :)

So anecdotal evidence must be defined very strictly - and as the term is taken from law, I have used that definition. Here is a modern American definition of the type I regard as nonsensical


Anecdotal Evidence: Anecdotal Evidence is information you obtain from a subjective report, an observation, or some kind of example that may or may not be reliable. In addition, anecdotal evidence is not scientifically valid or representative of a larger group or of conditions in another location.


That is so wide as to cover almost any empirical observation. So an experiment ceases to be anecdotal when? When you type it up? If a naturaluist informs me they saw a Turdus merula in their garden this morning does that mean I should dismiss it out of hand as a mere anecdote? If so the forum birdwatching is a pretty pointless enterprise, because seeing a Turdus merula is not an incredible claim where I come from I can assure you! WHat about written anecdotes? When Darwin discusses his brother experience at the seance at their home, and Huxley and his brothers initial enthusiasm in his letters, do I assume this is mere anecdotal evidence and ignore it? It is an anecdote, that is clear, but nope, it's not anecdotal evidence.

Anecdotal evidence properly defined in law is hearsay as far as I can see, and in it's literal sense is just a "story told about something usually based on personal experience." I ses no reason to not accept it, one simply requires more caution perhaps, but to a historian that would probably seem a bit odd. Its the most relaible evidence available in many cases - a letter by Darwin carries more weight than a newspaper article about him?


One can also believe such personal anecdotes by proxy such as the grandmother's anecdote that Roboramma described in the above post. The 911 truthers that believe the anecdotes of witnesses have more significance than the testimony of structural engineers would also be considering anecdotal evidence (or in that example, eye witness testimony).

However we have a relationship one presumes between eyewitness testimony and the structural engineers reports. While much eyewitness testimony is doubtless flawed, as observations are, I would assume the structural engineers reports simply explain what eyewitnesses experienced on the day? If there was a large disagreement between the eyewitness reports and the official theory, as in some early models of the Hindenburg disaster, we might well want to revise till we find agreement, or a clue to the oddities of the eyewitness reports?

Take the classic case, the assasination of JFK. Eyewitness reports are confusing and contradictory, and the physical evidence is clearly vital, but we might expect some degree of agreement in a correct theory?


Some people have a very hard time believing that the conclusions one draws from either their own or someone else's personal experience is subject to all sorts of errors. Take the flu shot example. Lots of people get sick after flu shots. We give flu shots during the time of the year when commonly acquired infections are the most frequent. Kids return to school. The few children passing things to an occasional neighbor, cousin, or sibling over the summer are suddenly in confined classrooms with dozens of kids. Then those kids go home and infect their parents who by October are passing the infection on all over the community.

Surely one might also argue that in some people a hypersensitive immune reaction gives minor flu like symptoms (but no flu)? I'm assuming flu is a dead vaccine, and sure the correlation of dates is not necessarily significant - for exactly the reasons you say. It would be simple to conduct a trial where the vaccine was administered at a different time of year, or patients were isolated from possible disease vectors?


The conclusion the flu shot was the cause of the illness is common but mistaken conclusion. How do we know? Using the scientific method rather than basing assumptions on uncontrolled experiences, we take a large enough group of people, blindly randomize them into two groups (or 3 if you want a no-treatment arm of the study) and give half a flu shot and the other half a shot of saline. Then you collect the data on who got sick and then compare the group who got the vaccine with the group(s) that didn't.

Lo and behold the rate of illness is the same in both groups as was found in the study I cited in a previous post.

Nice study, but the possibility of a hidden variable remains. Saline solution is just as likely to provoke an immune response as the actual vaccine - it's not the virus making people ill, but there bodies response to the process of vaccination. If you include the no treatment arm however you have a perfectly sound study, and I would fully accept the results. :)


Scientific method leads you to a valid conclusion. Anecdotal or personal experience in this case leads you to a false conclusion. Your personal experience was unable to control for other variables affecting the outcome while the scientific method was. But some people are unable to recognize that fact even when presented with clear incontrovertible evidence. That personal experience was so convincing they cannot overcome its influence.


Yes, but we are now back to my original point - neither of the thinsg cited in the article were actually anecdotal. In both cases what happened was simply a false correlation, as in the flu vaccine example. And this happens just as much in real science as in pseudo-science.

Imagine I perform a study of the population of sticklebacks in a river, and measure the variable pollution froma sauce factory. As pollution increases, the stickleback population drops drastically. Applying careful statistics i find the correlation is significant.

If however I fail to note the increase in the population of stickleback-in-Worcester-sauce-loving herons, I may well have made a false corellation. Sure my brain is programmed to accept this - I have noted a correlation between say snogging my girlfriends sister and being dumped, putting my hand in a fore and it burning, and getting drunk at office parties and humping off the roof and physical injury. In almost all cases correlation does indicate causality - in a few it soes not. :)

Many years ago I became ill three times after visiting a burger joint near my house. I had eaten there for ears on and off with no ill effects, but now i seemd to be getting sick every time. Why? Well I thought maybe the new management had let hygeine slip. It semed a bit unlikely as the staff were mainly the same though. I thought it through, and kept going. And I was ill again, a fourth time. I switched to chicken burgers, and was fine. It took weeks of patient experimentation and a couple more queasy nights before I discovered it was nothing to do with beefburgers (the false correlation), or hygeine. It was that some items were served with a blue cheese sauce, and for reasons I have still not sicovered eating blue cheese sauce makes me really ill. I tested this by buying aj jar, and waiting till my girlfriend used some in a meal - and siure enough I was ill, even though I did not know it was in there.

So my point here is the problem is nothing to do with anecdotal evidence: it's actually false correlation, where the apparent correlation is actually explicable by another variable. The problem is in our lives and in science the overwhelming majority of correlations are significant, which renders us prone to pay attention. Correlation does not require Causality: but in most cases it implies it. When it doesn't, then we have a problem, and make mistakes. :)

And Soapy's post said all this much more elegantly I felt, but hey I like whittering. :)

cj x

[drkitten]

Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline.


Well, that's one reason that history isn't a form of science.

Ditto Darwin. If he had ignored his personal observations in the Galapagos, and not then formulated theories based upon his own observations, no theory of evolution.

I don't think it's fair to characterize Darwin's observations as "anecdotal"; part of the key that made them more than mere anecdote was their systematicity.




Anecdotal Evidence: Anecdotal Evidence is information you obtain from a subjective report, an observation, or some kind of example that may or may not be reliable. In addition, anecdotal evidence is not scientifically valid or representative of a larger group or of conditions in another location.

The key word that you missed in that is "representative of a larger group." A single temperature measurement, especially a temperature measurement that we have reason to believe is atypical, is not "representative." A set of temperature measurements collected over a wide area or a long period of time, is representative.

One easy way to distinguish the two -- if you can put error bars on it, it's not anecdotal.


Surely one might also argue that in some people a hypersensitive immune reaction gives minor flu like symptoms (but no flu)? I'm assuming flu is a dead vaccine, and sure the correlation of dates is not necessarily significant - for exactly the reasons you say. It would be simple to conduct a trial where the vaccine was administered at a different time of year, or patients were isolated from possible disease vectors?

And, again, this is where the anecdote of an adverse reaction (which you accept might be caused by a hypersensitive immune system -- i.e. an unrepresentative situation, or an unusual and unrepresentative circumstance) can, properly collected, generate useful data.

Basically, 'cause you can then put error bars on the data.:D

[borealys]

What you need is an anecdote...

Someone upthread mentioned cognitive dissonance, and I suspect that's a big part of what happens. People will nod and smile and agree, oh, of course, that's bad reasoning, but can't connect it to themselves. We have big ol' blind spots when it comes to our own thought processes, and unless we can accept that even smart people make mistakes in their reasoning sometimes, our minds will do whatever it takes to save us from the cognitive dissonance that would come from even noticing anything that would make us look irrational or stupid.

I have no idea whether it actually makes a difference or not, but whenever I can, I like to use an anecdote about myself to make my point. I tell about my own mistakes in reasoning without the slightest bit of embarrassment, because, after all, everyone makes those kinds of mistakes at some point or another. They don't make me any less smart, and, by implication, they don't make anyone else less smart, either. (In fact, I can get a bit smug at being smart enough to have caught myself, but that's a whole other issue. ;))

We live in a mistake-averse society. It's only a thought, but I suspect that if we can find ways to make people less afraid of being wrong, their ability to spot their own errors and correct them will improve, whether those errors are due to overreliance on anecdotes, correlation-causation confusion, false memories, confirmation bias, or anything else.

[SezMe]

ETA in an odd twist, it seems to me that scientific thinking is trying to circumvent a part of the mechanism of natural selection. That anecdotal rapid decision and action, without knowledge is probably part of what evolved us to the point where we could start thinking about things more analytically. Now we are trying to over ride that in favor of this thoughtful analysis. Its kind of like dieting, for the mind.
Maybe it's an odd twist but since mankind's ability to reproduce is no longer controlled by natural selection it is appropriate that our rapid anecdotal response behavior be replaced by analytic thinking. IOW, natural selection has favored those who were able to overthrow natural selection!

[Miss_Kitt]

Paxi, I hope that last line was sarcasm.

With the exception of very young children, research indicates it is a myth. (http://archinte.ama-assn.org/cgi/content/abstract/156/14/1546)

SkeptiGirl: The cited study does not say that nobody got flu-like symptoms; it simply says that patients getting a placebo shot instead ALSO got flu-like symptoms at about the same rate. So I think Pax's post was correct, but incomplete. Something along the lines of, "...a flu shot can cause [symptoms] -- of course, so can getting a saline injection. :) " might be clearer.

The flu vaccine is not what's causing the symptoms, but they are occurring. Psycho-somatic effects aren't imaginary! They're just caused by some still-unknown mechanism(s). Some subjects get actual rashes from placebos! And some get symptom relief. There's a world of interesting stuff to be discovered in the realm of mind-body effect--that science has not yet figured out the mechanism doesn't make it mythical. And recognizing that such effects occur isn't surrendering to woo explanations of them, either.

I know, it's a minor detail, but I am fascinated by placebo effect and have had to change how I think and talk about it to be accurate. The real world is so much more amazing than any fantasy ever written!

ETA: On the larger issue of convincing versus just showing how a method is faulty, a counter-example can be helpful. On the issue of Correlation is not Causation, I like to point out that there is a strong correlation between espresso drinks consumed by parents and kids' test scores.

After a moment, I explain that this is because AFFLUENCE is strongly correlated to both! Wealth enough to make $3 coffee drinks a common expenditure means one or both parents are making good money; that is strongly linked to literacy and level of education. Parental literacy and level of education are linked to kids' scholastic achievement. It would be erroneous to credit some caffeine-induced Life Energy that jumps from the parent's aura to the kid's as the source of the initial correlation. (Editor's note: I assume in the UK this would not work, since the percentage of coffee junkies is so much lower.) Usually, this example works because people can see how there IS a real correlation, and yet no causation.

[cgallaga]

Maybe it's an odd twist but since mankind's ability to reproduce is no longer controlled by natural selection it is appropriate that our rapid anecdotal response behavior be replaced by analytic thinking. IOW, natural selection has favored those who were able to overthrow natural selection!

Hmmm memetics supercedes genetics...

[Crundy]

I'd like to see critical thinking taught as a one-off general studies topic in schools. Not "God doesn't exist", or "All psychics are phonies", just to tune people's ******** detectors before they start making their own money and become targets for the scammers. e.g. why anecdotal evidence means nothing, and applying occams razor to situations.

[Ivor the Engineer]

<snip>

ETA: On the larger issue of convincing versus just showing how a method is faulty, a counter-example can be helpful. On the issue of Correlation is not Causation, I like to point out that there is a strong correlation between espresso drinks consumed by parents and kids' test scores.

After a moment, I explain that this is because AFFLUENCE is strongly correlated to both! Wealth enough to make $3 coffee drinks a common expenditure means one or both parents are making good money; that is strongly linked to literacy and level of education. Parental literacy and level of education are linked to kids' scholastic achievement. It would be erroneous to credit some caffeine-induced Life Energy that jumps from the parent's aura to the kid's as the source of the initial correlation. (Editor's note: I assume in the UK this would not work, since the percentage of coffee junkies is so much lower.) Usually, this example works because people can see how there IS a real correlation, and yet no causation.

In the UK this would not work because we have the marking of our kid's tests organised by incompetent US companies and give degrees free with boxes of cereal.

[Ivor the Engineer]

<snip>

e.g. why anecdotal evidence means nothing, and applying occams razor to situations.

Anecdotal evidence is what keeps you alive.

[Crundy]

Anecdotal evidence is what keeps you alive.

Yes, my friend also told me that, so I guess it's true.

[skeptigirl]

SkeptiGirl: The cited study does not say that nobody got flu-like symptoms; it simply says that patients getting a placebo shot instead ALSO got flu-like symptoms at about the same rate. So I think Pax's post was correct, but incomplete. Something along the lines of, "...a flu shot can cause [symptoms] -- of course, so can getting a saline injection. :) " might be clearer.But that is also false according to the evidence.

The flu vaccine is not what's causing the symptoms, but they are occurring. Psycho-somatic effects aren't imaginary! They're just caused by some still-unknown mechanism(s).You are forgetting that we know what is causing the symptoms. It is simply the coincidental timing of flu shots which are given during the peak in other mild infectious diseases.


Some subjects get actual rashes from placebos! And some get symptom relief. There's a world of interesting stuff to be discovered in the realm of mind-body effect--that science has not yet figured out the mechanism doesn't make it mythical. And recognizing that such effects occur isn't surrendering to woo explanations of them, either.

I know, it's a minor detail, but I am fascinated by placebo effect and have had to change how I think and talk about it to be accurate. The real world is so much more amazing than any fantasy ever written!Where is your evidence that the occurrence of symptoms after flu shots are psychosomatic? I can show you other evidence that is not the case.

The way to sort out the effect of something such as a flu vaccine is to subtract the background rate. If you get zero when you subtract the background rate then the variable you are testing caused zero % of the effect.

Now there is the issue of a no test arm in a study. That would be the true background rate. And many studies don't include a no treatment arm. But if you had an ideal study you would start with a large group, you would randomly assign members of the group to a no treatment arm, a placebo arm and a flu shot arm. If the rates of mild systemic symptoms was the same in all three groups then the flu shots caused ZERO % of the systemic symptoms. If the placebo and flu shot groups had more systemic symptoms than the no treatment group, then you would be seeing the effect of getting a shot.

Only if the flu shot group had more systemic symptoms than the other two groups, then that would be evidence the effect was due to the flu vaccine.

You seem to be making a common mistake here. You are attributing cause to a variable, (getting the shot), when the evidence suggests some other variable, (more likely frequency of upper respiratory infections at the same time we give flu vaccine) is responsible for the effect.

There is no evidence the effects are psychosomatic. While one needs that no treatment arm to confirm that, the assumption the symptoms are coming from the shot are unsupported. Such a conclusion is short sighted. It fails to consider the obvious variable, the frequency of common respiratory infections in Oct and Nov.

You are correct to consider placebo effect as one possible variable. And I have an advantage of knowing that study is but one of several pieces of very good evidence that confirm side effects from flu vaccine, psychological or not, are 99% myth. But I'm pretty sure you'd have a hard time finding any evidence that the placebo effect contributes to systemic symptoms after flu shots. What you will find, however, is people pay more attention to their symptoms after a shot and in a study. So that headache which would have been forgotten a week later is instead remembered if the patient associated it with the flu shot.


ETA: On the larger issue of convincing versus just showing how a method is faulty, a counter-example can be helpful. On the issue of Correlation is not Causation, I like to point out that there is a strong correlation between espresso drinks consumed by parents and kids' test scores.

After a moment, I explain that this is because AFFLUENCE is strongly correlated to both! Wealth enough to make $3 coffee drinks a common expenditure means one or both parents are making good money; that is strongly linked to literacy and level of education. Parental literacy and level of education are linked to kids' scholastic achievement. It would be erroneous to credit some caffeine-induced Life Energy that jumps from the parent's aura to the kid's as the source of the initial correlation. (Editor's note: I assume in the UK this would not work, since the percentage of coffee junkies is so much lower.) Usually, this example works because people can see how there IS a real correlation, and yet no causation.I've tried these kinds of examples. But there is something about the convincing personal experience that blocks the person seeing the correlation.

[skeptigirl]

Maybe it's an odd twist but since mankind's ability to reproduce is no longer controlled by natural selection it is appropriate that our rapid anecdotal response behavior be replaced by analytic thinking. IOW, natural selection has favored those who were able to overthrow natural selection!That natural selection forces are no longer operating in the human species is an unsupportable conclusion. But maybe you could take this to another thread.

[skeptigirl]

Someone upthread mentioned cognitive dissonance, and I suspect that's a big part of what happens. People will nod and smile and agree, oh, of course, that's bad reasoning, but can't connect it to themselves. We have big ol' blind spots when it comes to our own thought processes, and unless we can accept that even smart people make mistakes in their reasoning sometimes, our minds will do whatever it takes to save us from the cognitive dissonance that would come from even noticing anything that would make us look irrational or stupid.

I have no idea whether it actually makes a difference or not, but whenever I can, I like to use an anecdote about myself to make my point. I tell about my own mistakes in reasoning without the slightest bit of embarrassment, because, after all, everyone makes those kinds of mistakes at some point or another. They don't make me any less smart, and, by implication, they don't make anyone else less smart, either. (In fact, I can get a bit smug at being smart enough to have caught myself, but that's a whole other issue. ;))

We live in a mistake-averse society. It's only a thought, but I suspect that if we can find ways to make people less afraid of being wrong, their ability to spot their own errors and correct them will improve, whether those errors are due to overreliance on anecdotes, correlation-causation confusion, false memories, confirmation bias, or anything else.
I think people's information filters do play a big part here. That personal experience is heavily favored by one's brain. And there is something that keeps people supporting their original conclusions in spite of contradictory evidence.

Teaching critical thinking skills early on in childhood education is probably the most effective option.

[cj.23]

Actually, a rather obvious variable on the flu thing springs to mind. Given that most flu vaccines are administered i'm guessing at doctors surgeries, where in the UK at least you will wait with other patients, many of whom are in for mild seasonal disorders, might one simply not contract the cold or whatever while waiting for the shot in a disease rich environment? :)

I recall the appalling legends which prevented many people seeking vaccination against flu in the uS on a couple of occasions historically- Gina Kolata's excellent book Flu deals with them well, and the public health implications as I recall.

Also, flu vaccinations only protect against the most virulent strains predicted for that season, so quite likely many people who are vaccinated will go on ot get flu perfectly naturally?

cj x

[skeptigirl]

Right, yeah I was thinking legally. Problem is that I have seen all observations and personal experience classed as anecdotal, and as such then statements like "my father is called Gunnar" become ancedotal and considered less valuable. Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline. Ditto Darwin. If he had ignored his personal observations in the Galapagos, and not then formulated theories based upon his own observations, no theory of evolution. And so on and so forth. Alll of modern science was developed with observations as a key criteria. So anecdotal clearly means something far more specific than "observation based" - though increasingly I see it misued that way - Sagan did it for instance - it means hearsay or unrelaible second hand testimony, the stuff of rumour. If Soapy saw the locjh Ness monster eating petunias in his living room tonight,and reported his experience, that would not be anecdoatal evidence. If I said Soapy had had that experience and i knew cos Larsen mentioned it to me while we were buying laptops, that would. :)

So anecdotal evidence must be defined very strictly - and as the term is taken from law, I have used that definition. Here is a modern American definition of the type I regard as nonsensicalI think I have addressed this in previous posts here. I am used to using the term, anecdotal evidence, because it has a specific meaning in the medical sciences. But to avoid confusion, I think 'personal experience' fairly well describes what I am referring to for those who are not used to the term, anecdotal evidence, as it is used in medical science.



...However we have a relationship one presumes between eyewitness testimony and the structural engineers reports. While much eyewitness testimony is doubtless flawed, as observations are, I would assume the structural engineers reports simply explain what eyewitnesses experienced on the day? If there was a large disagreement between the eyewitness reports and the official theory, as in some early models of the Hindenburg disaster, we might well want to revise till we find agreement, or a clue to the oddities of the eyewitness reports?...

Take the classic case, the assasination of JFK. Eyewitness reports are confusing and contradictory, and the physical evidence is clearly vital, but we might expect some degree of agreement in a correct theory?Eye witness evidence and scientific observations are not really what I am referring to here. But since you've brought this up, it is worth pointing out that a trained observer is different from a random observer. And some eyewitness accounts are going to be more reliable than other accounts. For example you can identify a person you witnessed if you know them while you may not make an accurate identification if it was someone you didn't know.



Surely one might also argue that in some people a hypersensitive immune reaction gives minor flu like symptoms (but no flu)? I'm assuming flu is a dead vaccine, and sure the correlation of dates is not necessarily significant - for exactly the reasons you say. It would be simple to conduct a trial where the vaccine was administered at a different time of year, or patients were isolated from possible disease vectors? ...

Nice study, but the possibility of a hidden variable remains. Saline solution is just as likely to provoke an immune response as the actual vaccine - it's not the virus making people ill, but there bodies response to the process of vaccination. If you include the no treatment arm however you have a perfectly sound study, and I would fully accept the results. :)My flu shot example is so classic that even people in the thread are arguing the anecdotes over the evidence. These conclusions are even less supportable than Miss Kitt's. I think it's safe to say you won't find any evidence that saline provokes an immune response. And I already explained in an above post that there is no evidence the vast majority of people get any reaction from flu shots other than a mild sore arm in a few people and extremely rare reactions such as an allergic reaction.

People do have vasovagal syncope after shots and blood draws. That is generally unrelated to what is in the syringe. But people do not in any study I have ever seen, get placebo illness symptoms from vaccines. I think the range of placebo effect is a lot narrower than is being suggested here. That symptoms are paid attention to is known to cause a perceived increase in symptoms following a flu shot. But again, that doesn't completely explain the widespread myth that flu shots cause illness symptoms. Those supposed after shot symptoms are 99% myth. They simply don't happen. Many people attribute any illness they get within a month or even longer of a flu shot as being caused by the shot. And, they selectively remember so that they will tell you they got sick "after every shot" when in reality they have just forgotten the shots that were not followed by symptoms.


Yes, but we are now back to my original point - neither of the thinsg cited in the article were actually anecdotal. In both cases what happened was simply a false correlation, as in the flu vaccine example. And this happens just as much in real science as in pseudo-science.

Imagine I perform a study of the population of sticklebacks in a river, and measure the variable pollution froma sauce factory. As pollution increases, the stickleback population drops drastically. Applying careful statistics i find the correlation is significant.

If however I fail to note the increase in the population of stickleback-in-Worcester-sauce-loving herons, I may well have made a false corellation. Sure my brain is programmed to accept this - I have noted a correlation between say snogging my girlfriends sister and being dumped, putting my hand in a fore and it burning, and getting drunk at office parties and humping off the roof and physical injury. In almost all cases correlation does indicate causality - in a few it soes not. :) There may be some areas where correlation more often than not indicates causality. But it is not so common when it comes to things that tend to have lots of potential variables such as in medical research.


Many years ago I became ill three times after visiting a burger joint near my house. I had eaten there for ears on and off with no ill effects, but now i seemd to be getting sick every time. Why? Well I thought maybe the new management had let hygeine slip. It semed a bit unlikely as the staff were mainly the same though. I thought it through, and kept going. And I was ill again, a fourth time. I switched to chicken burgers, and was fine. It took weeks of patient experimentation and a couple more queasy nights before I discovered it was nothing to do with beefburgers (the false correlation), or hygeine. It was that some items were served with a blue cheese sauce, and for reasons I have still not sicovered eating blue cheese sauce makes me really ill. I tested this by buying aj jar, and waiting till my girlfriend used some in a meal - and siure enough I was ill, even though I did not know it was in there.If your assumption about cause was correct, then the public health would likely have been alerted to outbreaks of illness associated with that restaurant. I think this shows one of the problems I am talking about. Your study was flawed. Yet you consider it was a valid study.



So my point here is the problem is nothing to do with anecdotal evidence: it's actually false correlation, where the apparent correlation is actually explicable by another variable. The problem is in our lives and in science the overwhelming majority of correlations are significant, which renders us prone to pay attention. Correlation does not require Causality: but in most cases it implies it. When it doesn't, then we have a problem, and make mistakes. :)... cj xIn other words, you can't see the flawed thinking going on here. I don't mean to be rude about it. But these are exactly the kind of false conclusions I am talking about.

[cj.23]

Hey skeptigirl, no offence at all. OK, so we are dealing here actually simply with the arguments for what Guyatt & Sackett called evidence based medicine, is that right? I read Archibald Cochrane's book from the mid 70's when I worked at the College of Health (1991-1994), and lived through many of the controversies over these issues. The article strikes me as claiming a far wider application than just medicine however, and still evidence based medicine is simply that supported by widespread statistical randomized/double bllind trials as far as I recall - the name is misleading, as all diagnosis are evidence based after all. I have no doubt of the efficacy of EBM, or the excellent medical statisticians out their, like FLS who posts on this forum, but I still do not think the article makes sense. I just wanted you to know a) you are not at all rude, I probably need to clarify and b) that I intend to reply, just been busy and had not seen your latest till just now.

cj x

[skeptigirl]

Hey skeptigirl, no offence at all. OK, so we are dealing here actually simply with the arguments for what Guyatt & Sackett called evidence based medicine, is that right? I read Archibald Cochrane's book from the mid 70's when I worked at the College of Health (1991-1994), and lived through many of the controversies over these issues. The article strikes me as claiming a far wider application than just medicine however, and still evidence based medicine is simply that supported by widespread statistical randomized/double bllind trials as far as I recall - the name is misleading, as all diagnosis are evidence based after all. I have no doubt of the efficacy of EBM, or the excellent medical statisticians out their, like FLS who posts on this forum, but I still do not think the article makes sense. I just wanted you to know a) you are not at all rude, I probably need to clarify and b) that I intend to reply, just been busy and had not seen your latest till just now.

cj xThe thread is about addressing the underlying causes of why people fail to think critically rather than just addressing the facts people have wrong. The thread discusses one example, brain functions. We know that some characteristics of the human brain frequently lead to false conclusions about causal relationships when only coincidental relationships actually have occurred. In other words people naturally look for explanations for the things they observe. In order to think critically and therefore draw more valid conclusions, we need to recognize the underlying mechanisms which cause people to go wrong.

I used the term, anecdotal evidence. It confused some people. My reference to the term, 'anecdotal evidence' in medical research was merely to clarify the definition for the discussion. Trained observers and the nature of eye-witness accounts are not what the discussion is about. The discussion is about the strength of personal experience (more than just witnessing something) and how that impacts people's beliefs to the degree they don't believe the scientific evidence even when it is more reliable.

I used the 'flu shot false conclusions' as an example. But it caused some problem in the discussion because many people in the forum also are affected by experiences, or people they trust's experiences when it comes to false beliefs about flu vaccine. It would seem that to discuss how personal experience can lead to false conclusions, one must remember, no one is immune. Our brains are designed to draw these conclusions.

That is why the subject of underlying mechanisms for barriers to critical thinking skills needs to be discussed in the first place. And, no, this is not strictly about medical research. Medical beliefs happen to be particularly rife with personal experiences. However, so are psychic experiences and ghost encounter experiences.

[Big Les]

Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline.

I've wrestled with this (because as a fellow historian I DO think that anecdote is the same thing or on the same order as the "anecdotal evidence" that skeptigirl and Shermer discount), and I think you simply have to accept that accurate history and archaeology just aren't possible. The overwhelming majority of the past is essentially lost to us. Just think about your own life, and how much of a true sense of it some future historian could get from only a handful of documents (for example).

This doesn't make it futile. Imperfect, even fundamentally flawed, understanding, is still worth seeking. And history is always more about those of in the present than those dead and gone. But it does make it (anecdotal evidence) the best we have in terms of the past, and the worst we have in terms of establishing present day events. Still not worthless, but the lowest form of evidence.

Then there's the issue of accepting anecdote regardless - for some things, this risk is worth taking. If someone tells you your house is on fire, a) there's precedent and plausibility. and b) you have a lot to lose by ignoring the evidence and little to lose by following it.

When it comes to what we collectively call "woo", these are things that have never been proven to occur by science, and so anecdotes surrounding such events can (sceptics agree) be rejected out of hand.

Far from crippling it, scepticism is a boon to history, because it means you lower your expectations of what's achievable, you get closer to the truth, and you can still speculate based on the evidence, as long as you make clear that's what you're doing.

[skeptigirl]

I've wrestled with this (because as a fellow historian I DO think that anecdote is the same thing or on the same order as the "anecdotal evidence" that skeptigirl and Shermer discount), ...It is interesting that however clear one describes one's view, people ignore what you say and churn out their own versions.

I do not discount anecdotal evidence. Half the research in medicine relies on anecdotal evidence. What you fail to understand here is the difference between systematically collected anecdotal evidence, and using un-systematically collected stories without controlling for other possibilities meaning you take everyone's word for it that they know something caused something.

It's important that people for example, report beliefs about symptoms they think their flu shot caused. Then it is up to the researchers to test the hypothesis that the shot and not something else caused the symptoms. You don't just say, "OK, that must be a correct conclusion and therefore it is a fact."


When it comes to what we collectively call "woo", these are things that have never been proven to occur by science, and so anecdotes surrounding such events can (sceptics agree) be rejected out of hand.

Far from crippling it, scepticism is a boon to history, because it means you lower your expectations of what's achievable, you get closer to the truth, and you can still speculate based on the evidence, as long as you make clear that's what you're doing.You have made a false statement here that these anecdotes are dismissed out of hand. To the contrary, they are followed up on with research at least in medicine. If the research shows the conclusion to be wrong, then the individual's conclusion about the relationship between the two things is a false conclusions.

If you wear your hat backward and your ball team wins the game, did your hat make that happen? Was your backward hat required in order for that win? If it happened a couple times, lots of people believe the hat actually had an influence. That doesn't mean it did.


So it would help if you got your facts straight here about who is dismissing what evidence and which conclusions.