How well would you do on a jury? [Archive]

Bikewer

31st October 2009, 07:08 AM

Being relatively innumerate, I read the book "Innumeracy" some time ago. It was primarily concerned with how mathematical information was presented in the media, though it would apply in this sort of application as well.
Unlikely you'd get such a presentation of information in a jury trial. Both prosecutors and defense attorneys want to make such presentations as simple as possible. They well know that if one side says..."The chances of this DNA belonging to anyone other than the accused is billions to one!" the other side will attempt to skew the statistics the other way....
I got 3 out of 5, which isn't bad for someone who flunked algebra 1.....

Nick Bogaerts

31st October 2009, 07:23 AM

New scientist got question 5 wrong as well, which is astonishing given how much it has been discussed.
They picked up on part of the error, but not all of it. If the odds of a child dying of cot death are 1 in 8500, the odds of a child dying of cot death given that the child has died are much higher than 1 in 8500.

Wolfman

31st October 2009, 07:37 AM

I'd hate to be the guy who wrote an article like this...you just know that he's gonna' be getting tons of emails from both qualified and wannabe mathematicians out there, giving him numerous reasons why he's wrong.

The more important issue -- at least to me -- is that the article does quite effectively demonstrate how misunderstanding these statistics can lead to wrong conclusions (particularly the issue of the guy who's test results indicate he has a rare disease...the fact that it is actually over 99% likely that the results are wrong is one that the vast majority of people would miss).

Puppycow

31st October 2009, 07:37 AM

Just 2 out of 5 here.

For some reason I got the meaning of "less than 1 in 12 million" and "more than 1 in 12 million" mixed up in my mind. I was thinking about the denominator instead of the value of the fraction itself. I think I had the correct principles in my mind, but I had a brain fart in selecting the answers.

Puppycow

31st October 2009, 07:39 AM

McHrozni

31st October 2009, 07:42 AM

Unlikely you'd get such a presentation of information in a jury trial. Both prosecutors and defense attorneys want to make such presentations as simple as possible. They well know that if one side says..."The chances of this DNA belonging to anyone other than the accused is billions to one!" the other side will attempt to skew the statistics the other way....

Not only that, but for the given case, a population must be estimated. Getting a match of 1:3,000,000 from the general population for a crime on Pitcairn Island is different than in Mexico City. This was not supplied, yet it would have to be. Chances can be higher or lower than 1:3,000,000 in such cases.

Also about that interracial couple, we are told what was the chance of them occurring, but not what the population looked at was. We are then to assume the estimate was skewed in favor of them being guilty. Why? That's something the defense must show, not jury.

Only 4) and 5) are actually sensible with the latter being borderline so, and for useless reasons: 4 because it's inexact, and 5 because of relations. Given that we were asked to conclude the estimate was skewed in the favor of the prosecution in the interracial couple question, why can't we assume at least one of the two children who died was adopted?

Stupid quiz, really.

McHrozni

McHrozni

31st October 2009, 07:48 AM

The more important issue -- at least to me -- is that the article does quite effectively demonstrate how misunderstanding these statistics can lead to wrong conclusions (particularly the issue of the guy who's test results indicate he has a rare disease...the fact that it is actually over 99% likely that the results are wrong is one that the vast majority of people would miss).

That one also struck me as totally misleading. The question stated the test was 99% accurate. That to me would say there is 99% chance of diagnosing the disease accurately, and giving 1% false exclusions. Many of the medical tests used don't give either false inclusions and false exclusions (barring laboratory mistakes), but just one of the two. Yet, we're supposed to know somehow that 1% error rate is false inclusions, and not false exclusion.

McHrozni

Elaedith

31st October 2009, 08:00 AM

For number 1 the answer identified as correct is 'More than 1 in 3 million' but this depends on the population being larger than 3 million and there is nothing about the size of the population in the account. Again question 2 assumes a relatively populous area and there is no such information given. Number 3 is kind of silly because there are presumably independent reasons for the suspect being suspected of being the murderer, and the murderer is thought to have the disease, and so surely this increases the prior probability of the suspect having the disease. However, the explanation treats the suspect as a random sample from a population of 10000.
Question 5 correct this depends on the assumption that each cot death is an independent event, whereas in practice they will not be.
The only answer I don't have problems with is for 4.

Elaedith

31st October 2009, 08:02 AM

That one also struck me as totally misleading. The question stated the test was 99% accurate. That to me would say there is 99% chance of diagnosing the disease accurately, and giving 1% false exclusions. Many of the medical tests used don't give either false inclusions and false exclusions (barring laboratory mistakes), but just one of the two. Yet, we're supposed to know somehow that 1% error rate is false inclusions, and not false exclusion.

McHrozni

I noticed that too. I had to assume the 1% is both false positives and negatives but it is very sloppy not to say so.

DevilsAdvocate

31st October 2009, 08:33 AM

Ok. I did what Puppycow did. Their answer to number 1 is correct.

1 in 3 million people will have a DNA match. If the population is 60 million, that means that statistically there are 20 people with a DNA match. The DNA on the knife must have come from one of those 20 people. The suspect has a match. So, excluding the suspect, there are 19 other people who’s DNA is could be. The probability that the DNA is from one of those other people is 19 in 20 (95%). That is more that 1 in 3 million (.00003%).

It doesn’t matter if the population is less than 3 million. If the population were 3 million, there would be one other person with a DNA match. So there would be a 1 in 2 (50%) chance that the DNA on the knife is someone else’s.

If the population of “other people” were just one, and the probability of “anyone else” having a DNA match is 1 in 3 million, then there be a one-three-millionth in 2 chance (.0000167%), which is less than 1 in 3 million (.00003%). But if there were 2 other people, there would be a two-three-millionth in 2 chance (.00003%), which would be exactly 1 in 3 million.

So as long as the population of “other people” is more than 2, the correct answer is:

B. More than 1 in 3 million

bluesjnr

31st October 2009, 08:40 AM

We were not even told the age of the two children who died in the cot death scenario, yet no "not enough info" option. I manged three/five so I'm above (by a considerable margin) average at flawed quizzes. Can't quite work out what that says about me other than.

"Your Honour, we find the defendant.....................guilty"

DevilsAdvocate

31st October 2009, 08:58 AM

Unlikely you'd get such a presentation of information in a jury trial.Yeah. And some would be presented the other way around. For example with number one, the defense may point out that given the probability of 1 in 3 million DNA match and 60 million people, there are 20 other people that could match and there is a 95% chance that the DNA on the knife belongs to one of those other people, and a 95% chance that the DNA is someone else’s establishes reasonable doubt.

Of course, that is 95% out of the people with a DNA match (the people whose DNA could be on the knife), but still only 1 in 3 million (.00003%) of all people if the suspect were picked from the total population at random.

CptColumbo

31st October 2009, 09:01 AM

I don't want to nitpick, but they mentioned something I know about.
SHAMBLING sleuth Columbo always gets his man. Take the society photographer in a 1974 episode of the cult US television series who has killed his wife and disguised it as a bungled kidnapping. It is the perfect crime - until the hangdog detective hits on a cunning ruse to expose it. He induces the murderer to grab from a shelf of 12 cameras the exact one used to snap the victim before she was killed. "You just incriminated yourself, sir," says a watching police officer.

If only it were that simple. Killer or not, anyone would have a 1 in 12 chance of picking the same camera at random. That kind of evidence would never stand up in court.
The episode they are refering to is "Negative Reaction" with Dick Van Dyke as the murderer.

I don't remember how many cameras there were on the shelf in the last scene, but 12 seems about right. However, they were mixed together, with the camera in question behind several others. Van Dyke's character physically pushed aside other cameras to pick up the one that incriminated him.

Wolfman

31st October 2009, 09:05 AM

I don't want to nitpick, but they mentioned something I know about.

The episode they are refering to is "Negative Reaction" with Dick Van Dyke as the murderer.

I don't remember how many cameras there were on the shelf in the last scene, but 12 seems about right. However, they were mixed together, with the camera in question behind several others. Van Dyke's character physically pushed aside other cameras to pick up the one that incriminated him.
Actually, I think that in most cases, the 'evidence' Columbo comes up with wouldn't be enough to convict with. What tends to happen instead is that he comes up with largely circumstantial evidence which nevertheless compels the guilty individual to confess to their guilt. Thus, the 'odds' of him choosing that particular camera are irrelevant...if it is sufficient to convince him to confess.

McHrozni

31st October 2009, 09:13 AM

Yeah. And some would be presented the other way around. For example with number one, the defense may point out that given the probability of 1 in 3 million DNA match and 60 million people, there are 20 other people that could match and there is a 95% chance that the DNA on the knife belongs to one of those other people, and a 95% chance that the DNA is someone else’s establishes reasonable doubt.

This is called the defenders' fallacy. It assumes all those people are equally likely to have committed a crime. That is highly unlikely - in the UK, for example, a vast majority will have a solid alibi ("I was at home in Inverness with my pals the night of the murder in Kent, sir") and no motive. It is always difficult to determine a target population, but saying 60 million is reasonable because it's the population of the UK is no more right than some other estimate.

McHrozni

DevilsAdvocate

31st October 2009, 09:45 AM

For question two I answered “more than 1 in 12 million” simply because there is no witness testimony that would identify that the murderers were a couple. The statistics are based on “couples”. The man and woman could have been friends, associates, business partners, etc. If the odds of there being a matching “couple” is 1 in 12 million, then the odds of there being any two people with matching descriptions that might commit a murder together (whether they are couple or not) must be much higher.

The wording of the expert testimony is also unclear. I would read it as saying that statistically a matching couple occurs 1 in every 12 million couples. The suspects would be 1 in 12 million. There would need to be another 12 million couples for it to be likely that there is another match. Wikipedia says the Combined Statistical Area of Los Angeles is about 18 million people. Even if 80% of those people are married, you would only have 14.4 million married people, which would be 7.2 couples. It is MOST likely that there are no other matching couples.

I also think that it is misleading to say that, given a reasonably sized population where there would be one other matching couple, that the odds that they are innocent are 50%. It makes it sound as if the fact that the suspects match the description only gives equal weight that may be guilty or innocent and that isn’t any evidence either way. That is not true, unless the police has a database of all couples and found two matches and flipped a coin on which couple to prosecute. Given the rarity of the description of the couple, the odds that they were innocent and were just wrong place at the wrong time or connected to the murders by some coincidence are extremely unlikely.

Professor Yaffle

31st October 2009, 09:53 AM

I'm very good at identifying what the testers want me to say is the right answer, even when the question is flawed.

Aitch

31st October 2009, 10:39 AM

Question 2 also assumes that no-one can alter their appearance.

Which is my excuse for getting it wrong. ;)

DevilsAdvocate

31st October 2009, 10:45 AM

The way I figured number three was that if 1 in 8500 families has a child die, and we know that families can have more than one child, the odds of a particular child dying is more than 1 in 8500. For example, if the average family has 3 children, and the death from SIDS is evenly distributed among families, then the odds of any particular child dying is 1 to 25500. Of course the family had at least two children, so the odds of one child dying is at least double, which may or may not be more than 1 in 8500. Not having any other information or testimony about how many children they had, distribution of deaths per family, likelihood of multiple deaths per family, I had to conclude that it could be more than 1 in 73 million, and could be less than 1 in 8500 (but only if the family had many other children or some other link that wasn’t provided such as living conditions, mother’s health, hereditary, etc.).

I understand where I went wrong on question 5 now.

Interesting quiz. :)

CptColumbo

31st October 2009, 11:05 AM

Actually, I think that in most cases, the 'evidence' Columbo comes up with wouldn't be enough to convict with. What tends to happen instead is that he comes up with largely circumstantial evidence which nevertheless compels the guilty individual to confess to their guilt. Thus, the 'odds' of him choosing that particular camera are irrelevant...if it is sufficient to convince him to confess.I agree, I just wanted to give more detail to the example they gave. Van Dyke's character wasn't just picking a number between 1 and 12, he sought out the camera that was eventually chosen.

Beerina

31st October 2009, 11:29 AM

I got Q3 wrong, but I don't think I got it wrong. They said the test is 99% accurate, but that only 1 in 10,000 actually has the disease. They suggest that the test, being 99% accurate, will thus mis-label 1% of the population. Since only 1 in 10,000 actually has the disease, that means for every 100 people considered positive, only 1 will actually be positive.

And what I mean by that is it impacts the definition of 99% accuracy. So let's say you're a scientist who's developed this test. How can you claim it has 99% accuracy when only about 1 in 100 positives are actually accurate.

This delves into decision theory, or partitioning theory. Let's say you want to separate the population into two groups. Let's use this example, and say that 1 in 10,000 people go into one, the other 9,999 go into the other.

Knowing nothing else, you could create the decision "rule" that you should simply presume everybody belongs in the big group. After all, you'd only be wrong 1 in 10,000 times, or an accuracy rate of 99.99%

Note that is a far superior test than the 99% one listed at trial. However, it is also completely useless because it does not select for the group we actually want, the 1 in 10,000.

So the scientist works on it, and comes up with some kind of chemical or biological analysis. If his 99% is to mean anything, it must therefore mean that 99% of the people he selects as belonging to the positive group are, in fact, positive.

Again, we could just get a 99.99% "accurate" test by simply assuming nobody has the disease. So this 99% of the scientist's test has to apply to something other than the overall partitioning accuracy.

The correct answer, therefore, would either be that we can rely on 99 of 100 people who tested positive as actually being positive which is exactly what should be asked of the scientist who made the test.

The other answer is that we simply do not have enough information. It may be correct to disregard the test results for that reason, but the conclusion listed in the question's "correct" answer, that such a test definitely implies 100 of 101 people who test positive are, in fact, negative is not warranted.

Sorry, partitioning, convex hulls, nearest neighbor, neural network gradient descent algorithms with simulated annealing, it's all gud!

taiyo

31st October 2009, 12:28 PM

Question 3 (the "99% accurate test") is badly written and the answer is wrong (it cannot be answered from the information given). I work with diagnostic tests so I have to be able to answer questions like this instantly. The technical definition of the accuracy of a diagnostic test is the percentage of test results that would be correct (positive and correct or negative and correct) if the entire population was tested. I would not expect jurors to know that definition.

They tell you that one person in 10,000 in this population has the disease. Let's assume the whole population is 1 million. 100 will have the disease and 999,900 won't. With a 99% accurate test, 990,000 people will get the correct result and 10,000 will get the incorrect result.

The 10,000 incorrect results can happen in different ways. For example, all 100 people with the disease might get an incorrect negative result (0% sensitivity), while 9,900 healthy people get incorrect positive results (99.0010% specificity). In this case, every positive result would be a false positive and the chance of the suspect having the disease given a positive test result is 0% (0% positive predictive value).

Alternatively (in the scenario the question answer assumes), all 100 people with the disease might correctly test positive (100% sensitivity), while 10,000 healthy people get incorrect positive results (98.9999% specificity). In this case, there are 10,100 positive results, 100 of them correct. So the chance of the suspect having the disease is 1 in 101, or 0.9901% positive predictive value.

Depending on the test sensitivity, the chance of the suspect having the disease is between 0% and 0.99%. Moral of the story: the rarer a disease, the less likely it is that a positive screening test result is correct.

If jurors are getting things wrong based on this kind of information, then it's not the jurors who are at fault.

New scientist got question 5 wrong as well, which is astonishing given how much it has been discussed.
They picked up on part of the error, but not all of it. If the odds of a child dying of cot death are 1 in 8500, the odds of a child dying of cot death given that the child has died are much higher than 1 in 8500.

Yeah I read that one and thought "what?!". If you looked at the cause of death for 8500 children I expect that more than one of them would have died of cot death (a lot more, depending on the age of the children). That one is just plain wrong.

I also got the domestic violence/murder one wrong too, but I thought the last 2 answer options were really unclear - they both seemed to mean the same thing to me.

taiyo

31st October 2009, 12:38 PM

Beerina - just seen your reply. A scientist who developed a test would never just say it is "99% accurate" and leave it that, for exactly the resons you describe. They would explain exactly what the test can and can't do, by describing sensitivity, specificity, positive/negative predictive value, etc. When you test for more and more rare diseases, it gets more and more likely that a positive result is a false positive, no matter how great the test is. So the scientist would say "my fantastic test has an excellent sensitivity and an excellent specificity and an excellent negative predictive value [the test would be very good at ruling out the disease], but because it is a rare disease the positive predictive value is only 1%". The purpose of a test like that is to pick out a small group of people to offer a super duper gold-standard test that really will give the right answer, but which you can't practically offer to everyone.

fls

31st October 2009, 12:38 PM

Well, I got zero out of five. And I actually understand this stuff. Which goes to show you how poorly they presented the information. Part of the problem is that many of the explanations require the assumption that the suspects are drawn randomly from a population, rather than identified by other means based on police work.

Question 1: There isn't enough information. The population under consideration is not "population of the UK", because, of course, that is not the population to which the test would be applied. Instead, it should be "population identified as suspects by the police and who subsequently undergo testing". In that case, whether or not it is more or less than 1 in 3 million depends upon whether or not the police tend to be right somewhat more or less than half the time.

Question 2: There isn't enough information. The explanation contradicts the information provided in the question. We are specifically given the information that racial mixing is rare. Even if there was no other information used to identify this couple, it is unlikely that the "correct area" referred to represented a population of 12 million. The real answer depends upon the probability that the perpetrators were local and were married. However, the chance that they are innocent is "less than 1 in 12 million" is probably also a correct answer, as it is unlikely that more than 12 million people had to opportunity to commit the crime in the first place.

Question 3: Suffers from the same problem as the first.

Question 4: I might give them this. Technically they did not give us the baseline murder rate in the US. However, I would expect people to realize that nowhere near 150,000 women are murdered every year in the US.

Question 5: This suffers from the same problem as the first and third questions. The population under consideration is "dead infants", not all infants in the UK. Then it requires information about the probability that a dead infant has been murdered vs. succumbed to SIDS.

Linda

Professor Yaffle

31st October 2009, 12:46 PM

I think I *wanted* to check "not enough info" for most of them - but I knew that wouldn't be the answer they were after.

fls

31st October 2009, 12:48 PM

taiyo

31st October 2009, 12:56 PM

A 99% accuracy means that the test will identify 99 out of a 100 people with the disease and 99 out of a 100 people without the disease. The 99% accuracy shows up as 1% of the population (i.e. 100 out of 10,000 people) are mis-labelled as having the disease.

Linda

I'm afraid not - it's a common fallacy. 99% accuracy does not mean 1% of the population are mis-labeled as having the disease. It means that 1% of the population (with or without disease) get the wrong result. There is a big difference. See my post above.

fls

31st October 2009, 01:01 PM

I think I *wanted* to check "not enough info" for most of them - but I knew that wouldn't be the answer they were after.

You're forgiven. :) I always over-think multiple choice questions, as well, so I also try to answer according to what they're after instead of what the best answer is. I did a very poor job of figuring that out this time, though, because that was what I was trying to do.

Linda

fls

31st October 2009, 01:06 PM

I'm afraid not - it's a common fallacy. 99% accuracy does not mean 1% of the population are mis-labeled as having the disease. It means that 1% of the population (with or without disease) get the wrong result. There is a big difference. See my post above.

I agree. I was trying to keep it simple to make the point that the mis-labelling really did only represent 1%. I realize that an accuracy will represent some combination of sensitivity and specificity, unless it is stated in way that makes it clear that it is talking about one or the other.

Linda

taiyo

31st October 2009, 02:37 PM

I agree. I was trying to keep it simple to make the point that the mis-labelling really did only represent 1%. I realize that an accuracy will represent some combination of sensitivity and specificity, unless it is stated in way that makes it clear that it is talking about one or the other.

Linda
I understand - it actually makes very little difference in this case because the accuracy is so high and the disease rare. But obviously you'd get real discrepancies if those figures were different.

Lucky

31st October 2009, 08:08 PM

Conditional probability is my bread and butter, and the person who set this test seriously does not understand it. Several people have pointed out that it's ... I was going to say 'flawed', but that would be too kind. It's absolutely atrocious! Four of the questions either are ambiguous or have the wrong answer (and the other one's a bit iffy).

Q1
The suspect's DNA profile matches that on the handle. An expert witness says that the probability of anyone else being a DNA match is 1 in 3 million.
...
What are the chances that the DNA on the knife came from someone other than the suspect?
'Probability of anyone else being a DNA match' is completely ambiguous. Does it mean the probability that >= 1 matching person exists, or the probability that a random person will match? We're meant to assume the latter, but I'd say the wording rather suggests the former.

A slightly more subtle problem is that we can't estimate the probability of a match within a population (at probability per person of 1 in 3 million) if we have no idea of the size of the population. The population of the UK is a red herring - why not consider the population of the world, say, or the city where the murder occurred? What we're actually interested in is the population of plausible suspects, which is not in the millions for most murders. The correct answer is that, if a person suspected for some independent reason also matches the DNA profile (and there's no independent reason why this person should have a high probability of a match), then it's virtually certain to be his DNA.

Q2
What are the chances that the couple are innocent of the murder?
Again, to get the 'right' answer you have to assume that there's no other reason to suspect the couple, and the first people found who match the description will be charged with the murder. Highly improbable (one would hope).

Q3
The suspect has been tested for the disease, using a test that is 99 per cent accurate, and the test was positive.
Ridiculous to differentiate between 1 in 100 and 1 in 101, and anyway the correct answer is 1 in 102 (assuming the most reasonable interpretation of '99 per cent accurate').

Q5
What is the probability that the two children did both die of cot death?
Their 'answer' very, very wrong. It's true that it's a serious mistake to assume the two deaths are probabilistically independent (it would clearly be wrong if they'd both been murders!). However, the main point is that we're not interested in the probability that some random family will suffer two cot deaths, but the conditional probability that this double death, having happened, is the result of cot death rather than murder. So the right answer is 'We don't have enough information to say' (though 'More than 1 in 8500' wouldn't be wrong).

Myriad

31st October 2009, 10:02 PM

I was misled by the wording of the first question. The key part reads: "An expert witness says that the probability of anyone else being a DNA match is 1 in 3 million." By "anyone else" it appears they meant "any other given individual." But I read it as "any other person in the world."

Respectfully,
Myriad

Wowbagger

31st October 2009, 10:12 PM

Folks, this is New Scientist we're talking about....

..of course their gonna get their articles and quizzes wrong!!

Duh.

Dr. Imago

31st October 2009, 10:46 PM

A 99% accuracy means that the test will identify 99 out of a 100 people with the disease and 99 out of a 100 people without the disease. The 99% accuracy shows up as 1% of the population (i.e. 100 out of 10,000 people) are mis-labelled as having the disease.

Linda

Well, you're being a bit too simplistic. We don't really talk about "accuracy" in medicine, do we? We speak of "sensitivity" and "specificity", as well as positive predictive value and negative predictive value.

Care to explain the difference, Linda? It's actually quite relevant to this discussion, as DNA testing is just another form of a screening test. :)

~Dr. Imago

Eyeron

1st November 2009, 12:28 AM

Isn't guilty by statistics a poor way to see if a person is really guilty?

quixotecoyote

1st November 2009, 12:51 AM

The number 2) is inexact to the point of being fatally flawed. The rest are kindof okay, though stupid in their limits.

McHrozni

This.

I haven't read the thread yet, so I assume most of these will be said, but I want to write right now.

I fell for the Prosecutors Fallacy on question 1, although I did think it was phrased in a misleading fashion.

I had to put not enough information for 2 because the question asked if they were innocent, not how many other suspects there were. It's especially weird because they acknowledge this in their explanation of the correct answer.

3 appears to be ********. It shouldn't matter what the overall prevalence of the disease in the population is. If you test positive on a test that has a 99% prediction success rate, there's a 99% chance you have the disease. eta: upon reading this thread, it appears to be a problem with the wording, not with the math as they intended to represent it.

4 I liked.

5 seems to be ********** up because nothing in the question suggested the events should be treated as dependent.

Wolfman

1st November 2009, 01:09 AM

3 appears to be ********. It shouldn't matter what the overall prevalence of the disease in the population is. If you test positive on a test that has a 99% prediction success rate, there's a 99% chance you have the disease. eta: upon reading this thread, it appears to be a problem with the wording, not with the math as they intended to represent it.
I don't think so...I'm no mathematician, and others have pointed out problems with this that I wasn't aware of, but the basic logic of it seems fairly obvious.

One in ten thousand people will have this disease...so in a sample population of one million people, 100 people will have the disease.

99% of people tested will get accurate results, but 1% will not...so in a sample population of one million people, 10,000 people will get false results.

So...which is more likely...that you are one of the 100 people who have the disease, or that you are one of the 10,000 people who will get a false result? The odds are rather massively on your side that you're in the latter group.

ETA: I know and acknowledge the errors that others have raised above (and I've likely made some errors in my own argument). For example, differentiating between false positives and false negatives. But the basic logic behind the argument is solid.

ETA2: Another way of looking at it (and slightly more accurate)...if the test is 99% accurate for positive and negative results...then out of a sample population of one million people, 1 person who has the disease (1% of 100) will get a false negative, and 99 people will get a true positive. On the other hand, out of the same population, 9999 people who don't have the disease (1% of 999,990) will get false positives, and the remainder will get true negatives. Thus, out of a total of 10098 positive results from the entire population, only 1% (or actually, 0.99%) of them are actually correct.

Brian-M

1st November 2009, 01:47 AM

Yeah I read that one and thought "what?!". If you looked at the cause of death for 8500 children I expect that more than one of them would have died of cot death (a lot more, depending on the age of the children). That one is just plain wrong.

They don't provide any relevant statistics in the question. They provide us with the chances of any given baby dying of SIDS, when what we need is the chances of any given dead baby having died of SIDS... two very different figures.

I noticed In the explained answer (but omitted in the question) they give the odds of a second child dying of cot death being as high as 1 in 60. Well, if they can use relevant statistics not included in the question to arrive at the answer, then so can I.

According to Wikipedia, the United States has an infant mortality rate of 6.3 per 1000, and SIDS is responsible for 0.543 deaths per 1000 live births.

So that's SIDS mortality of 543 per million live births,
And an infant mortality of 6300 per million live births.

543/6300 = 1/116 chance of any given infant death being caused by SIDS.

So we have a 1/116 chance of the first baby dying of SIDS and a 1/60 chance of the second baby dying of SIDS.

1/116 times 1/60 = 1/6960

That gives us a 1 chance in 6960 that any two dead babies both died of SIDS. Compare this to New Scientist's answer. They're not even close to being right. :D

edd

1st November 2009, 02:52 AM

Isn't guilty by statistics a poor way to see if a person is really guilty?

No, I think it's the correct way, although it's not normally so obviously statistics at work in assessing evidence and it's not often so straightforward to calculate.

Bad statistics are certainly a poor way to assess guilt, I'll give you that.

Wolfman

1st November 2009, 03:51 AM

Isn't guilty by statistics a poor way to see if a person is really guilty?You would consider evidence like fingerprints and DNA to be valid methods in helping determine a person's guilt, wouldn't you? Well, the actual value of such evidence is based upon the statistical likelihood of someone else having fingerprints/DNA that similarly match the evidence at hand.

ArmillarySphere

1st November 2009, 04:09 AM

According to Wikipedia, the United States has an infant mortality rate of 6.3 per 1000, and SIDS is responsible for 0.543 deaths per 1000 live births.

So that's SIDS mortality of 543 per million live births,
And an infant mortality of 6300 per million live births.

543/6300 = 1/116 chance of any given infant death being caused by SIDS.

So we have a 1/116 chance of the first baby dying of SIDS and a 1/60 chance of the second baby dying of SIDS.

1/116 times 1/60 = 1/6960

That gives us a 1 chance in 6960 that any two dead babies both died of SIDS. Compare this to New Scientist's answer. They're not even close to being right. :D
I think you slipped a decimal point there - The odds are 1/11.6, not 1/116.

So the chances of both babies being dead of SIDS are 1/135 - again, disregarding linking factors such as genetics, environmental factors etc. And if it can be established that one of the infants really died from SIDS, the other infant's odds are at least as high as 1/11.6.

Elaedith

1st November 2009, 04:09 AM

Professor Yaffle

1st November 2009, 04:14 AM

Re question 3: Ben Goldacre did a couple of good articles on this area - which was the only reason I had any idea of what anser they were wanting.

http://www.badscience.net/2006/12/crystal-balls-and-positive-predictive-values/

http://www.badscience.net/2009/02/datamining-would-be-lovely-if-it-worked/

sol invictus

1st November 2009, 04:17 AM

Ok. I did what Puppycow did. Their answer to number 1 is correct.

No, the answer to number 1 cannot be determined because it is too poorly worded (if you like, the correct answer is D, not enough information). The question states " An expert witness says that the probability of anyone else being a DNA match is 1 in 3 million." That might mean that a randomly selected person has a 1 in 3 million chance of matching as well or better than the subject. Or, it might mean that there is a 1 in 3 million chance that anyone else in the world has as good a match. Those are very different.

A 99% accuracy means that the test will identify 99 out of a 100 people with the disease and 99 out of a 100 people without the disease. The 99% accuracy shows up as 1% of the population (i.e. 100 out of 10,000 people) are mis-labelled as having the disease.

Linda

The accuracy of a medical test like that cannot be characterized by a single percentage. I'm a little shocked that a doctor wouldn't know that.

There are two possible cases - you have the disease or you don't - and two possible outcomes for the test - positive (the test says you have it) and negative (the test says you don't). Hence there are two different error rates, false positives and false negatives, and therefore the statement that such a test is "99% accurate" is almost totally meaningless.

JWideman

1st November 2009, 04:47 AM

Nevermind how likely it is that the suspect actually has the disease. Let's assume he does. How is this at all an indicator of guilt without knowing a LOT of other details? Is the disease one which is present in a disproportional number of the population, like sickle cell anemia? In that case, if the suspect is African-American, the odds are only 1 in 5000. If Latino, 1 in 36000. And if neither, there's your man.

Toke

1st November 2009, 04:51 AM

I assumed that the 99% accuracy of the test were both false positive and negative since nothing else were written.

I got q5 wrong, Guess I was assuming something about what brought the case to police attention in the first place.

fls

1st November 2009, 04:52 AM

Well, you're being a bit too simplistic. We don't really talk about "accuracy" in medicine, do we?

We do among those of us involved in relevant areas in research and education, but I agree that "accuracy" is little used among those who are just referring to the application of EBM to clinical practice.

We speak of "sensitivity" and "specificity", as well as positive predictive value and negative predictive value.

Yes, because we are usually dealing with a specific situation - a positive or negative test result and the need to consider a condition ruled-in or ruled-out.

Care to explain the difference, Linda? It's actually quite relevant to this discussion, as DNA testing is just another form of a screening test. :)

~Dr. Imago

Accuracy refers to the extent to which a test is correct - correctly identifying those people who have the condition and correctly identifying those people who do not have the condition. None of the other measures take all that information into consideration.

One of the ways in which accuracy can be measured is a simple count of all the correct answers out of the total number of answers. If the sensitivity and specificity are identical, then this number will be representative. If they are not, then the final result will depend upon the base rate of the condition, even though sensitivity and specificity do not, which obviates the usefulness of the number (it ends up as a weighted average instead of just an average). One can set the base rate to 50% in order to derive numbers which can be compared.

Receiver Operating Characteristics (ROC) curves are used to determine accuracy. There are other measures of accuracy which relate to reliability and internal consistency, such as Cronbach's alpha or Pearson-product moment correlation.

Accuracy is also used less formally (which I think was the case in question 3) to refer to one of the other measures you mentioned or to something more vague. And one cannot derive any of the other measures from a reported accuracy unless the possibilities are limited - as they are in this case by a number that is very high.

ETA: It should be noted that this explanation is very simplistic and incomplete. This is a huge area of study and by attempting to form some sort of summary for you, I don't wish to give the impression that I have somehow captured the entire topic.

Linda

fls

1st November 2009, 04:56 AM

The accuracy of a medical test like that cannot be characterized by a single percentage. I'm a little shocked that a doctor wouldn't know that.

That's okay. I'm not at all shocked that a physicist (?) doesn't know about some of the technical terms used for diagnostic test development.

There are two possible cases - you have the disease or you don't - and two possible outcomes for the test - positive (the test says you have it) and negative (the test says you don't). Hence there are two different error rates, false positives and false negatives, and therefore the statement that such a test is "99% accurate" is almost totally meaningless.

I agree that if you don't know what it means, then you won't know how to apply it, which is yet another reason why these were poor questions.

Linda

sol invictus

1st November 2009, 05:15 AM

That's okay. I'm not at all shocked that a physicist (?) doesn't know about some of the technical terms used for diagnostic test development.

I've read perhaps two dozen refereed articles in medical journals that reported test accuracies. That's not a large sample, but they all broke things down clearly into two ratios, or sometimes in a 2x2 table.

I've never came across a convention for combining those into one (not to say there isn't one, this isn't my field), nor do I see why such a convention would be either useful or desirable except perhaps in very specific circumstances.

Care to enlighten me on those "technical terms" you referred to? Let me point out the quiz and article don't even use the same terms consistently: in one place it says "a test that is 99 per cent accurate"; in another "the test is 99% certain".

fls

1st November 2009, 05:42 AM

I've read perhaps two dozen refereed articles in medical journals that reported test accuracies. That's not a large sample, but they all broke things down clearly into two ratios, or sometimes in a 2x2 table.

I don't know what you read, but that wouldn't surprise me. The application of evidence-based medicine to diagnostic tests involves the 2x2 tables you describe. "Accuracy" isn't useful under these circumstances (as I mentioned in post #55 to Dr. Imago) unless it is being used to refer to one of those measures instead. So I would only expect to see it used in certain kinds of articles.

I've never came across a convention for combining those into one (not to say there isn't one, this isn't my field), nor do I see why such a convention would be either useful or desirable except perhaps in very specific circumstances.

Do you have any familiarity with ROC curves? Or is that more of an engineering thing?

Care to enlighten me on those "technical terms" you referred to?

You can read post #55 above. If you are interested in this topic, I can suggest some reading material. If your interest is to suspect that I'm talking crap, I don't have any need to get between you and your desires. :)

Let me point out the quiz and article don't even use the same terms consistently: in one place it says "a test that is 99 per cent accurate"; in another "the test is 99% certain".

I already agreed that the quiz and article are poor. Please take that to mean that I have no interest whatsoever in defending them. :)

Linda

Brian-M

1st November 2009, 06:10 AM

I think you slipped a decimal point there - The odds are 1/11.6, not 1/116.

So the chances of both babies being dead of SIDS are 1/135 - again, disregarding linking factors such as genetics, environmental factors etc. And if it can be established that one of the infants really died from SIDS, the other infant's odds are at least as high as 1/11.6.

You're right... I messed it up. :o (But even then I was still closer to the correct answer than New Scientist. :) )

You had me confused with that 1/135 for a moment until I realized you'd substituted the 1/60 with 1/11.6 as well (which makes sense).

Thabiguy

1st November 2009, 06:21 AM

I assumed that the 99% accuracy of the test were both false positive and negative since nothing else were written.

I guess that's a reasonable assumption, but even then the answer they give is wrong.

Let's calculate: out of every 100 positive people, 99 will test positive and 1 negative. And out of every 100 negative people, 99 will test negative and 1 positive.

Out of every 10,000 people, 1 is positive and and 9,999 negative.

So, in a population, the proportions will be as follows:

1. (9999/10000)*(99/100) = 98.9901% will be negative and test negative.
2. (9999/10000)*(1/100) = 0.9999% will be negative and test positive.
3. (1/10000)*(99/100) = 0.0099% will be positive and test positive.
4. (1/10000)*(1/100) = 0.0001% will be positive and test negative.

The question, as the pose it, is this: The suspect was tested positive. What is the probability that the suspect is positive? - These are the offered choices:

A. 1 in 1
B. 99 in 100
C. 1 in 100
D. 1 in 101
E. less than 1 in 101

The fact that the suspect was tested positive means they belong to group 2 or 3. So the probability that the suspect is positive (belongs to group 3) is 0.0099% / (0.0099% + 0.9999%), which is exactly 1 in 102.

The correct answer would therefore be E, and not D as they claim. The difference may seem small, but seeing that they actually differentiate between C and D, that's not a valid excuse.

sol invictus

1st November 2009, 06:25 AM

I don't know what you read, but that wouldn't surprise me. The application of evidence-based medicine to diagnostic tests involves the 2x2 tables you describe. "Accuracy" isn't useful under these circumstances (as I mentioned in post #55 to Dr. Imago) unless it is being used to refer to one of those measures instead.

It still isn't as useful as the 2x2 table (or just two ratios, same thing).

Do you have any familiarity with ROC curves? Or is that more of an engineering thing?

I know what they are, yes. They contain much more information than a 2x2 table and/or two ratios (they're equivalent to an infinite or large set of such tables, where some other parameter varies over that set), and so are even less well characterized by a single "accuracy".

You can read post #55 above.

"One of the ways in which accuracy can be measured is a simple count of all the correct answers out of the total number of answers."

I'm confused by your language. That makes it sound as if there was such a thing as a single parameter "accuracy" for these tests, and the example you're giving is one way to get at it. But there is no such single parameter. Probably you just meant that that's one way of defining a quantity that is sometimes termed "accuracy"?

Note that with that definition, the quiz question cannot be answered (I realize you probably agree with that, I'm just pointing it out).

If you are interested in this topic, I can suggest some reading material. If your interest is to suspect that I'm talking crap, I don't have any need to get between you and your desires. :)

You're clearly not talking crap, and I retract any implication that you didn't understand this. In the past I've encountered many doctors that didn't, which might have made me jump to an unwarranted conclusion.

taiyo

1st November 2009, 06:46 AM

Well, you're being a bit too simplistic. We don't really talk about "accuracy" in medicine, do we? We speak of "sensitivity" and "specificity", as well as positive predictive value and negative predictive value.

Care to explain the difference, Linda? It's actually quite relevant to this discussion, as DNA testing is just another form of a screening test. :)

~Dr. Imago
I know I'm not Linda :) but here's my explanation:

Imagine you do a test on a group of 100 people who have the disease you're looking for. Imagine 90 of them test positive (true positives, TP) and 10 test negative (false negatives, FN). You would say the test has 90% sensitivity, i.e. it will pick up the disease in 90% of people who have it.
[sensitivity = TP/(TP+FN)]

Now imagine you do the test on 100 people who don't have the disease. Imagine 95 of them test negative (true negatives, TN) and 5 of them test positive (false positives, FP). You would say the test has 95% specificity, i.e. it will give the correct negative result in 95% of healthy people.
(specificity = TN/(TN+FP)].

Accuracy, positive predictive value and negative predictive value depend entirely on how common the disease is, i.e. they depend on the population being tested.

Accuracy (diagnostic accuracy) is just how many people overall get the correct result. If the disease affects 50% of people, then a sample of 200 people would contain on average 100 people with the disease and 100 people without (like the 2 groups above). In that case you'd have 185 correct results (90 TP and 95 TN) and 15 incorrect results (10 FN and 5 FP). The (diagnostic) accuracy is (no of correct results)/(total no of results), or (TP+TN)/(all results). In this case it's 185/200, which is 92.5% accuracy. You cannot back-calculate from accuracy - it can't tell you about sensitivity and specificity. "Accuracy" doesn't mean much on its own.

You have a total of 95 positive results, but only 90 are true positives. The positive predictive value is 94.7% (94.7% chance that a positive result means the person has the disease). [PPV = TP/(TP+FP)]

You have 105 negative results, but only 95 are true negatives. The negative predictive value is 90.5% (90.5% chance that a negative result means the person doesn't have the disease). [NPV=TN/(TN+FN)]

However, if the disease only affects 1 in 10 000 people, you get a diagnostic accuracy of 95%, positive predictive value of only 0.2% and negative predictive value of 99.9999%.

This is good revision for my exams!!! :)

There's a good "official" explanation here (sorry I can't post urls so you need to replace the "hxxp" with "http")

hxxp://ifcc.nassaro.com/index.asp?cat=Publications&scat=eJIFCC_&suba=Vol_19_No_4&subx=Measures%20of%20diagnostic%20accuracy%20basic %20definitions&zip=1&dove=1&zona=full&numero=&aq=1

fls

1st November 2009, 07:11 AM

It still isn't as useful as the 2x2 table (or just two ratios, same thing).

Yes. When I say "accuracy isn't useful" please take that to include "it isn't as useful".

I know what they are, yes. They contain much more information than a 2x2 table and/or two ratios (they're equivalent to an infinite or large set of such tables, where some other parameter varies over that set), and so are even less well characterized by a single "accuracy".

"Accuracy" with respect to ROC curves is the area under the curve, so each ROC curve can be characterized with a single "accuracy". An accuracy of 1 is perfect, while an accuracy of 0.5 is useless. Again, you cannot use this number to figure out the answer to question three (it doesn't tell you sensitivity and specificity for a particular test), but it does have other applications. And it has the value of being independent of the base rate.

"One of the ways in which accuracy can be measured is a simple count of all the correct answers out of the total number of answers."

I'm confused by your language. That makes it sound as if there was such a thing as a single parameter "accuracy" for these tests, and the example you're giving is one way to get at it.

Yes.

But there is no such single parameter.

In that case, that's just me talking crap again. :)

Probably you just meant that that's one way of defining a quantity that is sometimes termed "accuracy"?

Do I need to review the concept of 'operational definitions"?

Linda

Kestrel

1st November 2009, 07:17 AM

The 3 million to one odds against DNA belonging to someone else assume that the suspect doesn't have an identical twin. Knowing this, I got too clever on this question and selected "not enough information".

fls

1st November 2009, 07:17 AM

There's a good "official" explanation here (sorry I can't post urls so you need to replace the "hxxp" with "http")

hxxp://ifcc.nassaro.com/index.asp?cat=Publications&scat=eJIFCC_&suba=Vol_19_No_4&subx=Measures%20of%20diagnostic%20accuracy%20basic %20definitions&zip=1&dove=1&zona=full&numero=&aq=1

http://ifcc.nassaro.com/index.asp?cat=Publications&scat=eJIFCC_&suba=Vol_19_No_4&subx=Measures%20of%20diagnostic%20accuracy%20basic %20definitions&zip=1&dove=1&zona=full&numero=&aq=1

(I thought it was a post count of 15 before you were able to post links?)

Linda

sol invictus

1st November 2009, 07:26 AM

"Accuracy" with respect to ROC curves is the area under the curve, so each ROC curve can be characterized with a single "accuracy".

That's not the same as your other definition.

And it has the value of being independent of the base rate.

Here's what wiki says about it, in a list of three or four other ways to get a single number out of a ROC curve:

* the area under the ROC curve, or "AUC", or A' (pronounced "a-prime") [2]
<snip>
The AUC is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one.[3]It can be shown that the area under the ROC curve is equivalent to the Mann–Whitney U, which tests for the median difference between scores obtained in the two groups considered if the groups are of continuous data. It is also equivalent to the Wilcoxon test of ranks. The AUC has been found to be related to the Gini coefficient(G) by the following formula[4] G1 + 1 = 2xAUC, where...

Yes.

And yet you've now given two inequivalent definitions for it which can differ just about maximally under some circumstances.

fls

1st November 2009, 07:54 AM

That's not the same as your other definition.

Yes. In post #55, I mentioned that there are several different ways to measure accuracy. First I defined it as the extent to which a test is correct and then I mentioned several different ways in which this is done. The 'several different ways' can be thought of as operational definitions. The article which taiyo linked to also describes, in greater detail, different ways of making reference to accuracy. Which way you choose depends very much upon what you are going to do with the information.

Here's what wiki says about it, in a list of three or four other ways to get a single number out of a ROC curve:

And yet you've now given two inequivalent definitions for it which can differ just about maximally under some circumstances.

I agree (and I have said this from the beginning) that there are different numbers that can be used to represent the idea. I'm not sure what maximal differences you are referring to, since all that I have made specific reference to will characterize accuracy in a similar way (i.e excellent, good, fair, etc.). I agree that measures which are sensitive to the base rate will give quite varying results depending upon what that base rate is. That is why I have specifically mentioned whether or not a particular measure is sensitive to the base rate, and held that to be a disadvantage to the measure, if it cannot be got around.

Also, I understand that sometimes people make reference to PPV, NPV, sensitivity, specificity, LR+'s and LR-'s as measures of "accuracy". These measures necessarily only consider one aspect - people with or without the condition, people with or without a positive test results, circumstances where you are ruling-in or ruling-out a disease. I earlier distinguished the difference between these measures of "accuracy" which take only one aspect into account and measures which take all of that information into account. I personally consider the former use redundant.

The key point, though, is that if you are giving a specific number for "accuracy", then it means that you had some sort of operational definition in mind. If it's not obvious what that might be, then you need to specify that definition. It looked like the authors of Question 3 meant that 99% to apply to those who did not have the disease, based on the choices that were given. They just did a poor job of conveying that. And it was a poor use of 'accuracy' since it referred only to the correctness of a test in the absence of the condition, not the correctness of the test.

Linda

AndyD

1st November 2009, 08:03 AM

My nearest city has a population of 25,000 and my nearest capital only houses 1.6 million. I had no chance with the population estimates I was supposed to just assume in a couple of questions.

It seems that most questions added "further information" (population, genetics...) in the explanations without "need more info" being the correct answer. Weird.

I FALED :)

fls

1st November 2009, 08:05 AM

I FALED :)

Join the club. I got 0 out of 5.

Linda

taiyo

1st November 2009, 10:20 AM

The key point, though, is that if you are giving a specific number for "accuracy", then it means that you had some sort of operational definition in mind. If it's not obvious what that might be, then you need to specify that definition.

Exactly. "Accuracy" will mean completely different things to the person applying a diagnostic test, a person developing a laboratory method used in performing a test, and the average person on the street trying to figure out what the results mean. To answer all the questions in the quiz you have to make assumptions, and that should not be the case - either the assumptions should be specified or the data should be presented in a way that does not require assumptions.

http://ifcc.nassaro.com/index.asp?cat=Publications&scat=eJIFCC_&suba=Vol_19_No_4&subx=Measures%20of%20diagnostic%20accuracy%20basic %20definitions&zip=1&dove=1&zona=full&numero=&aq=1

(I thought it was a post count of 15 before you were able to post links?)

Linda

Oh yes! I wasn't keeping count - thanks!

Lucky

1st November 2009, 11:08 AM

There are two possible cases - you have the disease or you don't - and two possible outcomes for the test - positive (the test says you have it) and negative (the test says you don't). Hence there are two different error rates, false positives and false negatives, and therefore the statement that such a test is "99% accurate" is almost totally meaningless.
In the absence of any other information we can only assume that 99% is both the true positive rate (sensitivity) and true negative rate (specificity). I agree this assumption is unrealistic in a medical test (and also misleading), but it's quite usual to see it in worked examples 'for simplicity' (though the wording usually makes it plainer). However, their 'answer' bafflingly assumes that 99% is the specificity only, and the sensitivity is 100%. This makes their sums slightly wrong - the answer should be 1 in 102, not 1 in 101.

I FALED :)
Join the club. I got 0 out of 5.

Linda
No - as I pointed out (http://forums.randi.org/showthread.php?postid=5263454#post5263454), the person who set the test failed. And it's not just a matter of poor wording, it's a basic lack of understanding of conditional probability. I guess it was concocted by a science journalist who skimmed over the material without really getting the point (the questions and analyses have all been published elsewhere). Rather an embarrassment for the New Scientist, as the article is trying to show that the average ignorant clod on a jury doesn't understand conditional probability - unlike the clever folks at New Scientist.

fls

1st November 2009, 11:18 AM

No - as I pointed out (http://forums.randi.org/showthread.php?postid=5263454#post5263454), the person who set the test failed.

Yeah, but I beat you by seven posts. :)

http://forums.randi.org/showthread.php?postid=5262689#post5262689

Linda

Lucky

1st November 2009, 02:24 PM

Yeah, but I beat you by seven posts. :)

http://forums.randi.org/showthread.php?postid=5262689#post5262689

Linda
You missed a few of the errors - there were plenty to choose from! :)

Q1: I agree with your criticism as far as it goes, but an equally important error is that 'the probability of anyone else being a DNA match' is as least as likely to mean 'the probability that at least one other matching person exists' as 'the probability of a match per random person'. And I believe the ambiguity is there because the author doesn't clearly see the distinction (or the importance of it) - even though that's precisely what the question is about!

Q2: I agree - there are several reasons why the answer should be 'We don't have enough information to say'. New Scientist are totally confused here. They state in the question that 'the odds of finding another interracial couple matching the description are 1 in 12 million'. This isn't even ambiguous - it can only mean 'the probability that another such couple exists'. Then, in the answer they say 'there may well be plenty of couples who match the description'! If they were asking 'What are the chances that another couple could fit the description' then we've already been told - 1 in 12 million.

But not only have they given the wrong answer, they've asked the wrong question! The question 'What are the chances that the couple are innocent of the murder?' can't be answered by knowing the probability of finding someone else matching the description - we need to know what other reasons the police have for suspecting them.

Q3: As they clearly think the difference between 1 in 100 and 1 in 101 is important, they should have noticed that the answer is actually 1 in 102. Their assumption of 100% sensitivity for the test, having given the 'accuracy' as 99%, is obviously unjustified - I suspect it was just a blunder.

Q4: This question is wrong in a much more subtle way than the others, and I don't want to get bogged down in it - especially as most of the non-mathematicians/medics posting here seem not to understand the glaring errors in the other questions.

Q5: This was a really disastrous blunder by the New Scientist. You correctly pointed out that the population of all infants, and the expected number of cot deaths, can't tell us the probability that the deaths that occurred were cot death. But it really needs emphasising that, again, they answered the wrong question. 'What is the probability that the two children did both die of cot death?' is most emphatically not the same as 'What is the probability of a double cot death happening to a random family'. They are different by several orders of magnitude. Again, this is what the article is meant to be explaining!

It was particularly stupid of both Roy Meadow and the New Scientist not to realise that the 'rarity' argument applies just as much to murder as to cot death - they are actually arguing that it's highly improbable (by millions, or hundreds of thousands, to one) that the babies could have died!

taiyo

1st November 2009, 03:04 PM

Q3: As they clearly think the difference between 1 in 100 and 1 in 101 is important, they should have noticed that the answer is actually 1 in 102. Their assumption of 100% sensitivity for the test, having given the 'accuracy' as 99%, is obviously unjustified - I suspect it was just a blunder.
Just want to stress that the answer for question 3 is not 1 in 102, it's anything between 1 in 101 and 1 in infinity. The answer of 1 in 102 assumes 99% sensitivity. The answer of 1 in 101 assumes 100% sensitivity. They are both assumptions. The sensitivity could be 27% for all we know.

Cainkane1

1st November 2009, 03:27 PM

The New Scientist has an article on statistical/probabalistic evidence in court: http://www.newscientist.com/article/mg20427311.500-probably-guilty-bad-mathematics-means-rough-justice.html but before you read it, they ask you to take a short quiz.

I surprised myself by getting 5 out of 5 (though I am sure I can rely on you all to nitpick and decide that their correct answers are wrong ;)).

I didn't take this test but i've been on one grand jury and one regular jury. I helped send some bad guys to prison including a murderer named Randy Dobbs.

Dragoonster

1st November 2009, 03:36 PM

I got them all wrong, probably not for noble reasons--I don't understand probability. But the first one bugged me because it was vague and asked for the chance of anyone else, seeming to include all who would even be proven to not have matching DNA.

My answer was less than 1 in 3 million chance. I figured that in a population of 3 million, we know that the guy on trial has the DNA match, so he'd be the expected 1 in 3 million, meaning the likelihood someone else in a population of 3 million having the match would be 0 in 3 million (or I figured probability doesn't like 0 chances so somewhere less than 1). Then, if the population was 6 million he'd be one of the two expected, and so on, where he sort of takes up the spot of the expected other 1 in 3 million chance. Well, I suck at math and stuff but that answer seemed to make the most sense given the question as asked.

Careyp74

1st November 2009, 03:45 PM

sol invictus

1st November 2009, 04:08 PM

Yes. In post #55, I mentioned that there are several different ways to measure accuracy. First I defined it as the extent to which a test is correct and then I mentioned several different ways in which this is done. The 'several different ways' can be thought of as operational definitions.

I still do not understand your language. These definitions are inequivalent. That means they measure different quantities. Either there are simply many meanings to "accuracy" - in which case it's not a useful technical term in this context - or there is one underlying "accuracy" (which is what you seem to be implying), in which case at most one, or none, of these definitions is a good measure of it. If it's one, which? If it's none, why do you think this mysterious quantity exists at all?

I agree (and I have said this from the beginning) that there are different numbers that can be used to represent the idea. I'm not sure what maximal differences you are referring to, since all that I have made specific reference to will characterize accuracy in a similar way (i.e excellent, good, fair, etc.). I agree that measures which are sensitive to the base rate will give quite varying results depending upon what that base rate is.

For example, yes.

I earlier distinguished the difference between these measures of "accuracy" which take only one aspect into account and measures which take all of that information into account. I personally consider the former use redundant.

None of them (at least none that you've mentioned so far) take all that information into account. It's impossible to fully characterize a continuous distribution (like a ROC curve) or a 2x2 table with one number. All you can do is take some kind of weighted average.

The key point, though, is that if you are giving a specific number for "accuracy", then it means that you had some sort of operational definition in mind. If it's not obvious what that might be, then you need to specify that definition.

Fully agreed. My point (apart from simply trying to understand your views on this, which I'm having trouble doing) is that those "operational definitions" are all you have. Those are the definitions - there aren't any others.

fls

1st November 2009, 04:15 PM

You missed a few of the errors - there were plenty to choose from! :)

I agree. I wasn't trying to be inclusive. I sorta tried to pick the most relevant errors, but even then I sometimes defaulted to the easiest or shortest to explain. I got question 3 wrong for a different reason than the reason that I stated for it being wrong, for example. I was sincerely trying to answer the way that I thought they wanted it answered, so even though an ordinary consideration of the results of a diagnostic test was the wrong way to go about it, I tried it that way and got the 'less than 1 in 101' that others have already pointed out is only correct answer for that formulation.

Q4: This question is wrong in a much more subtle way than the others, and I don't want to get bogged down in it - especially as most of the non-mathematicians/medics posting here seem not to understand the glaring errors in the other questions.

For me, it was the realization that either abuse increased the risk or I should expect to see 500 women murdered in my city each year. I knew that the latter couldn't be true. However, this makes use of the sort of numeracy that Paulos talks about in his book Innumeracy, rather than information contained in the question. I'd be interested in knowing what you considered erroneous, though.

Q5: This was a really disastrous blunder by the New Scientist. You correctly pointed out that the population of all infants, and the expected number of cot deaths, can't tell us the probability that the deaths that occurred were cot death. But it really needs emphasising that, again, they answered the wrong question. 'What is the probability that the two children did both die of cot death?' is most emphatically not the same as 'What is the probability of a double cot death happening to a random family'. They are different by several orders of magnitude. Again, this is what the article is meant to be explaining!

I agree. This was the most disastrously wrong. The rest you could almost argue away with a naive audience and some shifty footwork. This one cannot even remotely be salvaged.

Linda

marting

1st November 2009, 04:25 PM

Yep, it's an astonsihingly bad test but hopefully the innumerate readers will at least have an increased inclination to question their assumptions should they be jurors. For those more mathematically inclined it provided some amusement.

Lucky

1st November 2009, 04:52 PM

Yeah, I got two right, which means I be dum. maybe. Can someone explain this in elementary school language so that my feeble brain can understand it:

"If the population is greater than 3 million (as an example, the UK's population is around 60 million), more than 1 person will have that profile.

Hence, the odds that the DNA on the knife did not come from the suspect must be higher than 1 in 3 million."

yeah it would be more than one person, it would be around 20. Still a 1 in 3 million chance though, right?
The handful of us in this thread who understand conditional probability have been explaining that the quiz, and the article, are rubbish. See for example this post (http://forums.randi.org/showthread.php?postid=5262689#post5262689) from fls, and this one (http://forums.randi.org/showthread.php?postid=5263454#post5263454) and this one (http://forums.randi.org/showthread.php?postid=5265376#post5265376) from me.

What Q1 is trying (and failing) to demonstrate is that the probability of a person found to have a matching DNA profile not being the real source is not at all the same as the probability of some random person being found to have a matching DNA profile.

If the expected number of matching people is 20 (out of the population of possible suspects, not the population of the UK - that's where the question went badly wrong), then (given no further information to help us pick the guilty party), each one of them has a probability of 1 in 20 of being the source of the DNA, i.e. a probability of 19 in 20 of being innocent.

taiyo

1st November 2009, 05:30 PM

fls

1st November 2009, 05:49 PM

I still do not understand your language. These definitions are inequivalent. That means they measure different quantities. Either there are simply many meanings to "accuracy" - in which case it's not a useful technical term in this context - or there is one underlying "accuracy" (which is what you seem to be implying), in which case at most one, or none, of these definitions is a good measure of it. If it's one, which? If it's none, why do you think this mysterious quantity exists at all?

The definition of accuracy is extent to which a test is correct - correctly identifying those people who have the condition and correctly identifying those people who do not have the condition (both, not one or the other or sometimes both). This is the meaning of accuracy. The next step is to consider how this is measured or quantified, which is where the idea of operational definitions comes into play.

It might help to consider when these measures are equivalent (please let me know if I need to fill out any of these steps in more detail). The 'count' I mentioned earlier (which is called "diagnostic effectiveness" in taiyo's link) is equivalent to Youden's index when the base rate is 50% and Youden's index is equivalent to the distance from the diagonal or chance line to the ROC curve at that particular cutoff point.

None of them (at least none that you've mentioned so far) take all that information into account. It's impossible to fully characterize a continuous distribution (like a ROC curve) or a 2x2 table with one number. All you can do is take some kind of weighted average.

I think that you are looking at it from the opposite direction. I am talking about whether information about positive and negative tests in people with and without the condition are needed in order to form the number (i.e. whether that information needs to be taken into account), and I think that you are talking about whether two tests with the same number can be distinguished when it comes to knowing the values for the information that was taken into account (as has been mentioned several times, they can't)?

Fully agreed. My point (apart from simply trying to understand your views on this, which I'm having trouble doing) is that those "operational definitions" are all you have. Those are the definitions - there aren't any others.

The difference between the meaning and an operational definition may not be particularly obvious when we are talking about diagnostic tests with a distinct gold-standard against which they are tested (i.e. the operational definitions are very close to the meaning). Where this shows up more distinctly are comparisons for which there isn't a gold-standard, so that 'accuracy' no longer has a distinct operational standard (e.g. agreement between experts, comorbidity indices, health status). In that case, the difference between meaning and measurement (when you are wracking your brains trying to figure out a way to validate your measure :)) may make more sense.

Linda

bpesta22

1st November 2009, 06:02 PM

I got 1 wrong, can someone help me understand why?

It didn't occur to me that the 1 in 3 million thingy should then be applied to a population to estimate how many people are matches. I guess that's clever. But, wouldn't the estimates be expected values (it's not certain that 3 people in 9 million exist that match in a population of 9 million; 3 is only the most expected value, but it could be 2, 4, 0 or some other number)?

If england is 60 million people, it's certain there are exactly 20 with this profile? Seems very odd to me, unless I am missing something about DNA testing. I am used to tests with much higher error rates (IQ tests!) so I also "factored in" the notion that a DNA test might not be 100 % accurate (the suspect's profile as reported by the test might not be a 1:1 perfect mapping of his actual DNA profile, as known by God).

If using the expected value as a given (there must be 1 person in every 3 million with this profile) then what would the logic suggest if the population were only 1 million. Would that prove that this guy must be innocent, because no whole person would be expected to have the profile!? Or would it imply there's a mistake in the test, which means it isn't perfectly reliable, which then supports the idea of adjusting the p value based on the test's unreliability?

I also got the base rate question wrong which is most embarrassing because I immediately recognized it as such and have given lectures specifically on it at least 50 times. I'd like to think it was sloppy reading on my part.

One other way to look at it for those who don't like looking at .0009 decimals, etc.

100,000,000 people in the population / .01% base rate. This means that 100 people have the disease. 99 of them will test positive (one will be a false negative, but that's irrelevant here).

999,900, though, don't have the disease, so 1% of them will test false positive-- 9999.

So, of 10,098 positive tests, only 99 have it. 99/10098 = .0098, which I guess is close enough to 1 in 102.

negativ

1st November 2009, 06:23 PM

The only thing I got out of that was renewed frustration at my utter lack of number skills.

Aside from that, any lawyer whose case depends on a jury's thorough understanding of statistical probability is a lawyer who has just lost the case.

taiyo

1st November 2009, 06:26 PM

bpesta22

1st November 2009, 06:32 PM

@bpesta22 - Actually, the answer to question 3 isn't even 1 in 102. That assumes 99% sensitivity. The sensitivity isn't stated. The answer could therefore be anywhere between 1 in 101 and 1 in infinity. See post: http://forums.randi.org/showpost.php?p=5262664&postcount=28

The example is almost exactly from Paulos' innumeracy. I think most people could be forgiven for not knowing the trick.

When I lecture on it, I do say that: if a test is 99% accurate, then of 100 people with the disease, there'd be 99 hits and 1 miss. And, of 100 without, there'd be 99 correct rejections and 1 false alarm. Even with the clarification, students still vastly over-estimate the probability when the base rate is low.

DevilsAdvocate

1st November 2009, 08:15 PM

No, the answer to number 1 cannot be determined because it is too poorly worded (if you like, the correct answer is D, not enough information). The question states " An expert witness says that the probability of anyone else being a DNA match is 1 in 3 million." That might mean that a randomly selected person has a 1 in 3 million chance of matching as well or better than the subject. Or, it might mean that there is a 1 in 3 million chance that anyone else in the world has as good a match. Those are very different.You are correct, sir. If read the second way, then there is a 1 in 3 million chance that another one person exists that has a DNA match. If such a person exists, there is a 50% chance that the DNA is that person’s and not the suspects. Therefore, there is a 1 in 6 million chance that the DNA on the knife belongs to someone other than the suspect.

From later in the article:
In the 1991 rape trial of Andrew Deen in Manchester, UK, for example, an expert witness agreed on the basis of a DNA sample that "the likelihood of [the source of the semen] being any other man but Andrew Deen [is] 1 in 3 million."

That was wrong. One in 3 million was the likelihood that any innocent person in the general population had a DNA profile matching that extracted from semen at the crime scene - in other words, P(E | H). The express witness stated the facts wrong. It appears the authors of the quiz were attempting to re-create the witness’s error by giving a fact to the effect that statistically “1 person out of every 3 million people will match this DNA” and see if you make the same error as the witness in translating that into the probability that the DNA belongs to someone other than the suspect. But the author also made the mistake of stating the fact wrong, or at least stating it in an ambiguous way.

Incidentally, according to this article (http://www.timeshighereducation.co.uk/story.asp?storyCode=94921&sectioncode=26), at the retrial, Deen plead guilty to one offense (I assume rape—he was originally convicted of 3 rapes) and the other two charges were dropped.

fls

1st November 2009, 08:57 PM

I got 1 wrong, can someone help me understand why?

I started out trying to explain what they think they mean, but their question and explanation is so awful that there isn't really any way to make sense of it.

Linda

Brian-M

2nd November 2009, 03:45 AM

"If the population is greater than 3 million (as an example, the UK's population is around 60 million), more than 1 person will have that profile.

Hence, the odds that the DNA on the knife did not come from the suspect must be higher than 1 in 3 million."

yeah it would be more than one person, it would be around 20. Still a 1 in 3 million chance though, right?

It would all depend on how they came to match his DNA. If they tested his DNA because he was a suspect, then the probability that someone else with the same DNA fingerprint being the guilty party is extremely tiny.

But if they just started testing people at random until they found a match, with an estimated 20 possible matches in a population of 60 million, then there is only 1 chance in 20 that the DNA was his.

If using the expected value as a given (there must be 1 person in every 3 million with this profile) then what would the logic suggest if the population were only 1 million. Would that prove that this guy must be innocent, because no whole person would be expected to have the profile!?

It doesn't mean that only one third of a person in a population of 1 million would be a match... it means that there is a 1 chance in 3 that any person out of that million is a match.

metzomagic

2nd November 2009, 11:11 AM

Lucky

2nd November 2009, 01:31 PM

I'm going to roll in here, and respectfully disagree with even the posters such as Lucky who obviously know their stuff. I agree with the analysis of the stats experts here on all but Question 3, which to refresh your collective memory was put as follows:

Q.3) A man has been murdered, and various pieces of evidence mean that we can be certain that the murderer had a particular disease.

The disease is rare; only 1 in 10,000 people have it.

The suspect has been tested for the disease, using a test that is 99 per cent accurate, and the test was positive.

What is the probability that the suspect really has the disease?
Now... this suspect supposedly wasn't just rounded up at random. There are (unspecified) pieces of evidence that implicate him in the murder. And the murderer left traces at the scene that indicate the murderer had this rare disease, which occurs on average in only 1 in 10000 of the general population.

So I maintain that if the suspect tested positive for this rare disease, and there are *other pieces of concrete evidence linking him with the murder*, then the chance he has the disease is maybe not 99 out of 100, but pretty darn close.

If the guy *wasn't a suspect in the murder*, then I agree with the 1 in 102 assessment. So I think the question is framed in a way that makes it entirely useless as a pure statistics exercise.
What you are saying is that the relevant prior probability isn't the population base rate of 1 in 10000 - and I agree, but I thought it would just add to the confusion if I tried to make this point along with everything else I was trying to explain.

The basic question is one that New Scientist could have cribbed from any number of medical education sources. But they've dressed it up to fit the forensic theme, and in doing so they have introduced more information. They seem not to understand one of the fundamental concepts in probability - more information in principle changes the probability.

We are all agreed that the test has a positive likelihood ratio (http://www.childrens-mercy.org/stats/definitions/likelihood.htm) of ~100. We're told it's certain that the murderer has the disease, so the true prior probability is the same as the (pre-test) probability that the police have the right suspect (the base rate of 1 in 10000 being negligible in comparison, one would hope). So, using the additional information that he's a suspect, his probability of having the disease is a lot more than 1 in 102 (though we can't say 99 in 100, or anything definite). I amend my analysis of Q3: it's 1 in 102 if we answer it in the way they meant it to be answered!

Lucky

5th November 2009, 03:58 PM

The technical definition of the accuracy of a diagnostic test is the percentage of test results that would be correct (positive and correct or negative and correct) if the entire population was tested.
Just want to stress that the answer for question 3 is not 1 in 102, it's anything between 1 in 101 and 1 in infinity. The answer of 1 in 102 assumes 99% sensitivity. The answer of 1 in 101 assumes 100% sensitivity. They are both assumptions. The sensitivity could be 27% for all we know.
There are many meanings to "accuracy". It depends on the context. When specifically talking about a diagnostic test, "diagnostic accuracy" means "if you tested the entire population, how many people would get the correct results". That's it.
...
Only "diagnostic accuracy (sometimes called diagnostic efficiency) has a specific definition.
...
In the question posed in the quiz, the only sensible interpretation of the term "accuracy" is "diagnostic accuracy".
This will seem like a nitpick after the major errors from New Scientist we've been discussing here, but I (strongly!) disagree with you.

It is not true, as you claim, that 'accurate' (or 'accuracy') has a defined technical meaning as a single-value summary of a medical test's effectiveness. Almost all the examples of 'diagnostic accuracy' I found cite both sensitivity and specificity (or equivalents), and even 'diagnostic efficiency' is mostly used more loosely, to include two or more measures.

We are told that the test is '99% accurate'. We are not told whether this is sensitivity, specificity, both, or an overall value such as the one you are claiming for 'diagnostic efficiency'. We should consider what interpretation a reasonable (and knowledgeable) person is likely to make. Given that we need both sensitivity and specificity in order to answer the question, and also that any reasonable definition of 'accuracy' must take into account both false positives and false negatives, I'd say 'both sensitivity and specificity' is by far the most natural. (And I'd say 'specificity only' is by far the least.)

Actually, you are wrong in saying the answer could be 1 in infinity (i.e. 0) under any assumption. If the test gives no information then the answer is 1 in 10000. An answer of 0 would require a sensitivity of 0, and sensitivity obviously can't be less than the false-positive rate (1 - specificity) - else you could just reverse the meaning of the test results. (And if specificity can be whatever you like, then the answer can be as high as 100%.)

They may not have made a conscious assumption about sensitivity - probably reasoned that it makes no significant difference to the answer whether we take it as 99% or 100%. As, of course, it wouldn't in a real medical example - we'd round the answer to 1 in 100 in either case.

Unlike the other errors in the quiz, I think this really is a case of poor wording.

taiyo

5th November 2009, 07:22 PM

I was taking the definition from the IFCC (http://www.ifcc.org/index.asp?cat=Publications&scat=eJIFCC_&suba=Vol_19_No_4&subx=Measures%20of%20diagnostic%20accuracy%20basic %20definitions&zip=1&dove=1&zona=full&numero=&aq=1):
Diagnostic effectiveness (accuracy)
Another global measure of diagnostic accuracy is so called diagnostic accuracy (effectiveness), expressed as a proportion of correctly classified subjects (TP+TN) among all subjects (TP+TN+FP+FN). Diagnostic accuracy is affected by the disease prevalence. With the same sensitivity and specificity, diagnostic accuracy of a particular test increases as the disease prevalence decreases. This data, however, should be handled with care. In fact, this does not mean that the test is better if we apply it in a population with low disease prevalence. It only means that in absolute number the test gives more correctly classified subjects. This percentage of correctly classified subjects should always be weighed considering other measures of diagnostic accuracy, especially predictive values. Only then a complete assessment of the test contribution and validity could be made.
Although when you read the full page there is plenty of ambiguity. They talk about 'diagnostic accuracy' in the loose sense, then go on to give the particular definition above. My interpretation has always been that if someone talks about 'accuracy' in the context of a diagnostic test and actually attaches a numerical value to it, it is assumed to be calculated according to the more specific definition. But I kind of agree with you - in the real world it's not a great idea to make assumptions like that. In the New Scientist test I assumed that definition because no other definition was given and it seemed like the most sensible interpretation.

Actually, you are wrong in saying the answer could be 1 in infinity (i.e. 0) under any assumption. If the test gives no information then the answer is 1 in 10000. An answer of 0 would require a sensitivity of 0, and sensitivity obviously can't be less than the false-positive rate (1 - specificity) - else you could just reverse the meaning of the test results. (And if specificity can be whatever you like, then the answer can be as high as 100%.)

Not sure about that. Purely theoretically, it could be the world's worst test, set up completely wrong and guaranteed to give a negative result in anyone with the disease. I know it's daft and you would just reverse the results, but mathematically it's possible. It wouldn't matter what the false positive rate was in healthy people - anyone with a positive result would be guaranteed to not have the disease.

Since the test is said to have a 99% accuracy, you can't have a specificity of 100%, as you have to account for 1% of the results being wrong, and they can't all be false negatives, as less than 1% of people have the disease. So there must always be plenty of false positives in the healthy group - enough to swamp the true positives.