PDA

View Full Version : Poll: Accuracy of Test Interpretation

Pages : [1] 2 3

Wrath of the Swarm
27th April 2004, 09:48 AM
Let's say that I went for an annual medical checkup, and the doctor wanted to know if I had a particular disease that affects one out of every thousand people. To check, he performed a blood test that is known to be about 99% accurate. The test results came back positive. The doctor concluded that I have the disease.

How likely is it that the diagnosis is correct?

It's best if you don't sit down to work it out. Just give your honest opinion about what you think is likely. If you happen to know the formula that gives the correct answer, feel free to use it.

Wrath of the Swarm
27th April 2004, 09:51 AM
S

P

O

I

L

E

R

S

I'll post the correct answer in three days when the poll shuts down.

27th April 2004, 10:07 AM
Too easy, but what's the chances of you having the disease if you get three positives in a row?

Wrath of the Swarm
27th April 2004, 10:14 AM
A lot of doctors can't do this problem correctly. I expect the forumites to do better, but I think I'll still make my point.

(After three positive results, the chances of having the disease are about 99.9%.)

 (This assumes, of course, that each test is independent of the others, which is not a realistic assumption. Still, tests with such high degrees of accuracy are generally unrealistic as well.)

Rolfe
27th April 2004, 10:28 AM
Hmmm, big piece of information missing. Are you showing any clinical signs of the disease, or is your probability of infection that of the general background population?

The thing is, to get the predictive value of a test (which is what Wrath is asking), you need to know the incidence of the condition in the population representative of the individual being tested. This is obviously higher if that population is "sick people with clinical signs typical of the disease in question". In fact, the relevant figure is the clinical probability that this individual is affected.

I also need to know if the figure of "99% accurate" refers particularly to specificity. Specificity is the percentage of positive results which are correct (i.e. identify an affected individual) and is the important parameter here. Sensitivity, on the other hand, is the percentage of negative results which are correct (i.e. identify an unaffected individual). The two together may be loosely combined and referred to as "accuracy", but if we are talking about a positive result, it is the specificity figure one needs to know.

However, if we assume for the moment that Wrath's probability of being affected is the same as the general population, that is 0.1%, and that the specificity of the test is 99% (I don't really care about sensitivity in this situation), a positive result is 90.98% likely to be wrong and 9.02% likely to be correct.

So there is only (approximately) a 9% chance that Wrath actually has the condition.

(I'm sorry, I cheated, I have a spreadsheet on my computer to produce this information and a pretty graph that demonstrates how the predictive value of a positive or a negative result varies with clinical probability of infection, and sensitivity and specificity of the test.)

http://www.b5-dark-mirror.demon.co.uk/graph.jpg

Rolfe.

geni
27th April 2004, 10:29 AM
It's no good becase of the way Wrath of the Swarm worded the question there is more than one correct answer

Rolfe
27th April 2004, 10:31 AM
Originally posted by geni
It's no good becase of the way Wrath of the Swarm worded the question there is more than one correct answerWell, I did state some assumptions. Did I leave anything out?

Rolfe.

Wrath of the Swarm
27th April 2004, 10:31 AM
No, there's only one correct answer.

Rolfe, you are both an idiot and a jerk. Now that you've posted an answer, the poll is completely useless.

(Oh, and your requests for more information are pointless. The information provided is more than sufficient to answer the question. Besides, this is the question that doctors have such trouble with - I don't think adding more would actually help them.)

Rolfe
27th April 2004, 10:33 AM
Now that it's been explained, you don't get to feel so superior. What a shame.

Rolfe.

Wrath of the Swarm
27th April 2004, 10:34 AM
You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

Deetee
27th April 2004, 10:35 AM
So what is Wrath's "point", anyhow?

Since in his hypothetical scenario he deliberately had the doctor jump to the wrong conclusion, I suspect he is just out to try and rubbish the poor medics again.

I suspect that docs know a bit more about sensitivity, specificity, predicitive values, odds ratios etc than WoS gives them credit for, anyway.

geni
27th April 2004, 10:36 AM
Originally posted by Rolfe
Well, I did state some assumptions. Did I leave anything out?

Rolfe.

Doing the maths the way I am suggests that you get wrath's answers by assuming the error is in false posertives. Playing around with false negativs screws the results.

geni
27th April 2004, 10:37 AM
wrong button

Rolfe
27th April 2004, 10:38 AM
That was exactly the assumption I made. That the 99% "accuracy" (meaningless term, really) quoted was actually a specificity figure. If it included the sensitivity and the specificity was actually something different, then the result is void.

Anyway, five out of five posters already got it right while I was composing my post and before I hit the send button.

Rolfe.

geni
27th April 2004, 10:41 AM
Originally posted by Wrath of the Swarm
You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

Er no beacuse most of the time information wont be presented the way you presented it (If it is You find the person responcible and tell them to get into the advertising bussness if they already are in the advertising bissness then why were you beliving them in the first place?).

All you've done is present a waiter pocketing the tip type problem

Brian the Snail
27th April 2004, 10:41 AM
Originally posted by Wrath of the Swarm
You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

Actually, before Rolfe revealed the answer there were at least 4 votes cast, all of which gave the correct answer.

So what makes you think that most people here would have got the answer wrong?

Wrath of the Swarm
27th April 2004, 10:41 AM
The accuracy covers both true positives and true negatives. If I had specified the rate of alpha error only, you would have needed to know the beta error. But since I didn't, you didn't.

Wrath of the Swarm
27th April 2004, 10:45 AM
Originally posted by Deetee
So what is Wrath's "point", anyhow?

Since in his hypothetical scenario he deliberately had the doctor jump to the wrong conclusion, I suspect he is just out to try and rubbish the poor medics again. It must be nice to be able to psychically determine what someone's position is before they state it and tear apart the holes in the arguments they haven't made yet.

Have you considered trying out for the million? A debating technique like that could make you a very rich person.

Soapy Sam
27th April 2004, 10:48 AM
And to add to the chaos, I just guesstimated 10% before I read the rest of the thread. So it's unanimous.
Wrath- I'd suggest you are playing to the wrong audience ith this one. Most folk here will have been reading J.A.Paulos and Martin Gardner for years. You said you expected us to beat GPs and you were right, but I think you underestimated by how much. You would really need a more random test group. (And a larger one of course).

Wrath of the Swarm
27th April 2004, 10:50 AM
Originally posted by Brian the Snail

Actually, before Rolfe revealed the answer there were at least 4 votes cast, all of which gave the correct answer.

So what makes you think that most people here would have got the answer wrong? Most medical students (and a significant fraction of doctors) get the question wrong.

Given that people here can check the other answers, take the time to work out the answer fully, and generally are better educated in statistics and critical thought, I figured many more people than normal would get the right answer. I expected the numbers of wrong answers to still be significant, though.

Since I clearly stated that I would post the correct answer when the poll cleared, it was extremely rude of Rolfe to post the explanation. After all, if there's the least chance it could be used to support a position she doesn't like, she couldn't permit it to continue unchecked, could she?

Rolfe
27th April 2004, 10:58 AM
Now, to take this on to where it needs to go, can you see the point I'm making, which is perhaps more interesting?

The figure quoted is correct if Wrath's clinical probability of being affected is that of the general population. Actually, that is difficult to imagine unless the condition in question is symptom-free in the vast majority of cases. Most interesting diseases do show some clinical signs at some stage.

Now, if Wrath is clinically symptom-free, then his probability of being afffected is the probability that someone showing no symptoms is affected. If only half of all sufferers show some clinical signs, this is already down to 0.05%, as only 1 in 2,000 of the asymptomatic population is affected.

The probability that the doctor is right is in fact only 4.72%, in this situation.

On the other hand, if the reason his doctor wanted to test him is that he came in demonstrating clear clinical signs suggestive of the condition in question, then his probability of being affected is the probability that anyone showing these clinical signs has the condition. This depends a lot on how pathognomonic the clinical signs are for the disease. But let's say he was a very typical case, and that 80% of people with these presenting signs actually have the condition.

Now look at the graph, and what it does over at the right-hand side, at the 80% probability of infection level (hint, it's the line that is almost indistinguishable from the 100% abscissa at that level).

There is a 99.75% probability that the doctor is right.

This explains why it is vital to take the real likelihood that the patient you are looking at is affected into consideration when interpreting tests like this. That is the conclusions you have come to from your clinical examination and history-taking. Otherwise, if you use a figure for incidence in the general population regardless of the individual's own circumstances, positive results are always judged to be very probably wrong and negative results to be very probably right.

Not much point doing the test if that's how you think.

In fact, it's a good illustration that it's statistically valid to say that if the test gives you the answer you were expecting, it's probably right, but if it gives you a result you weren't expecting, be very cautious. In practice, the unexpected result has to be re-checked by a reference method.

If you are screening well people, it will be the positive results you regard with suspicion, but if you are testing on a strong clinical evidence the positive result is pretty safe to accept, and you may well want to check a negative (depending on how suspicious you were in the first place, refer to graph again).

Rolfe.

JamesM
27th April 2004, 10:58 AM
Now that the answer's been given, can someone tell me: is the answer exactly 10% or just close to 10%?

Wrath of the Swarm
27th April 2004, 10:59 AM
Originally posted by Rolfe
The thing is, to get the predictive value of a test (which is what Wrath is asking), you need to know the incidence of the condition in the population representative of the individual being tested. This is obviously higher if that population is "sick people with clinical signs typical of the disease in question". In fact, the relevant figure is the clinical probability that this individual is affected. Actually, this isn't even correct.

Rolfe is confusing the utility of clinical indications with test accuracy. The significance of the test's results to the diagnosis depends upon the proportion of the population that actually has the condition. Its accuracy does not depend on that value.

Wrath of the Swarm
27th April 2004, 11:00 AM
Originally posted by JamesM
Now that the answer's been given, can someone tell me: is the answer exactly 10% or just close to 10%? The answer is about 9%. Since there were a limited number of poll options, 10% was the correct answer.

Rolfe
27th April 2004, 11:00 AM
Originally posted by JamesM
Now that the answer's been given, can someone tell me: is the answer exactly 10% or just close to 10%? 9.02% with the number of "decimal places" set to 2.

Rolfe.

geni
27th April 2004, 11:02 AM
Originally posted by Wrath of the Swarm
The accuracy covers both true positives and true negatives. If I had specified the rate of alpha error only, you would have needed to know the beta error. But since I didn't, you didn't.

But by playing around with the false posertive negative numbers I can keep the total accuracy at 99% while geting a number of different answers to you question (particuly sine you put the about in I can get the chances up to to 100% (10,000 tested wrong 9 times all false negatives) (ok that quite a bit higher than 99% acuricy unless you have ver big error bounds

Wrath of the Swarm
27th April 2004, 11:03 AM
Obviously, performing more than one test increases the chance of getting the correct result significantly.

But Rolfe can't distinguish between the accuracy of the test and the usefulness of combining it with another selection procedure. She's also ignoring the important point that many conditions (like certain kinds of cancer) don't have obvious symptoms.

I'm just glad she's a vet in another country instead of a doctor here. She wouldn't have made it through med school, of course, so I suppose the actual risk she would pose is minimal.

Wrath of the Swarm
27th April 2004, 11:05 AM
Originally posted by geni
But by playing around with the false posertive negative numbers I can keep the total accuracy at 99% while geting a number of different answers to you question (particuly sine you put the about in I can get the chances up to to 100% (10,000ed test wrong 9 times all false negatives) Since I didn't specify different values for alpha and beta error, the single value I have for the accuracy holds for both.

Thanks for trying, though.

geni
27th April 2004, 11:07 AM
Originally posted by Wrath of the Swarm
Since I didn't specify different values for alpha and beta error, the single value I have for the accuracy holds for both.

Thanks for trying, though.

Simplify it's two years since I did sats I fail to see why it should hold for both rather than the sum of both.

Wrath of the Swarm
27th April 2004, 11:09 AM
Because I gave you an overall accuracy. Given any particular input, the test has a 99% chance of giving the correct answer. That holds whether the person has the disease or not.

In reality, tests don't always have equal chances of false positives and false negatives. That's not the case for the hypothetical test, though.

Rolfe
27th April 2004, 11:17 AM
Originally posted by Wrath of the Swarm
The significance of the test's results to the diagnosis depends upon the proportion of the population that actually has the condition. Its accuracy does not depend on that value. NO, no and thrice no.

This is the whole point. This is the mistake most likely to be made by young graduates who have been brainwashed by statistics of the sort Wrath is peddling.

Now do you see why I went for the answer early and widened the question?

Predictive value of a test depends on the sensitivity, the specificity, and the prevelance of the condition in a population representative of the patient in question. ("Accuracy" is a meaningless term in the context of a test of this nature.)

The prevelance of the condition in a population representative of the patient in question means the prevelance of the condition in a population presenting clinically exactly as the patient in question presents. That is, the clinical probability that the patient in question has the disease.

For example, people often quote the incidence of FeLV (feline leukaemia virus) as 1% in the population as a whole. But if you confine your FeLV testing to cats presented chalk white with a lymphocyte count of > 20 x 10<SUP> 9</SUP>/l, the proportion of correct positive results you get will be a hell of a lot less than the 66.89% the sums Wrath would like you to do might suggest. Let us assume that the specificity of the FeLV test is about 98% (as it is). Since these cats are perhaps 70% likely to be infected (that is, if the only cats you test are in that group, 70% of the cats you test will be infected), only 0.87% of your positive results will be wrong.

Conversely, if you spend all your time screening healthy pedigree cats from tested-negative households (people do do this, prior to breeding), where there is maybe only a 1 in 10,000 chance that a cat has sliped the net and become infected, 99.51% of your positives will be wrong.

Experienced clinicians understand this. People who've just blindly read a rather superficial statistical explanation of predictive value don't, and routinely underestimate the reliability of a ositive result form a clinically sick uindividual with suggestive clinical signs.

I know it's hard to get your brain round, Wrath, but do try.

Rolfe.

yersinia29
27th April 2004, 11:19 AM
Originally posted by Wrath of the Swarm
Most medical students (and a significant fraction of doctors) get the question wrong.

Nothing you say can be trusted at all. Nobody should believe a word you say unless you provide links and evidence backing up your claims.

Wrath of the Swarm
27th April 2004, 11:19 AM
Well, since this example has been ruined by that whore Rolfe, let's go over the math.

p = fraction of people who have the condition
x = error rate of the test

(1-p)x = fraction of false positives
p(1-x) = fraction of true positives

When are these values equal?

x - xp = p - xp
x = p

When the accuracy of the test is equal to the proportion of the population that has the condition, any positive result has a 50% chance of being correct. The calculation becomes more complicated if we presume there are different error rates for false and true positives, of course.

Luciana
27th April 2004, 11:21 AM
This thread has been reported but I see no breaking of the forum's rules. No action will be taken.

Civility, however, is always desirable...

Rolfe
27th April 2004, 11:21 AM
Originally posted by Wrath of the Swarm
Because I gave you an overall accuracy. Given any particular input, the test has a 99% chance of giving the correct answer. That holds whether the person has the disease or not.

In reality, tests don't always have equal chances of false positives and false negatives. That's not the case for the hypothetical test, though. This is meaningless. You didn't say that the test was both 99% sensitive and 99% specific. I had to assume it before I could even begin.

Suppose the test was 100% specific and only 98% sensitive (bloody good test if it managed that). Would you, by that reasoning, still call that "99% accurate"? However, in that case, all positive results are correct, so the doctor knows he's right a priori.

Rolfe.

Wrath of the Swarm
27th April 2004, 11:25 AM
Originally posted by Rolfe
NO, no and thrice no.

This is the whole point. This is the mistake most likely to be made by young graduates who have been brainwashed by statistics of the sort Wrath is peddling.

Now do you see why I went for the answer early and widened the question?

Predictive value of a test depends on the sensitivity, the specificity, and the prevelance of the condition in a population representative of the patient in question. Wrong.

If there are diagnostic criteria that must be met before a test is performed, that's performing two different tests. One just isn't done in a laboratory. We then must consider the error rate of the initial screening by symptoms. After all, surely it's not falliable.

What Rolfe describes (winnowing the population before lab tests are performed) is good medicine, but she's incorrectly asserting what she's doing. We're talking about whether the test is correct or not, but she's talking about whether clinical judgments based on its result are correct, and that's a completely different issue.

Wrath of the Swarm
27th April 2004, 11:28 AM
Originally posted by Rolfe
This is meaningless. You didn't say that the test was both 99% sensitive and 99% specific. I had to assume it before I could even begin. I did say that. I said the test is 99% accurate. That sets both values. If I said that the test would correctly identify a person with the condition 99% of the time, then there wouldn't be enough information for anyone to answer the question - you'd know the false negative rate, but not the false positive. But that isn't what I said.

It's a good thing you can look up the answers on a chart, because you sure as hell can't handle the concepts involved.

ceptimus
27th April 2004, 11:33 AM
Hmmm. Seeing as the cat is already out of the bag :) let's work this out using a population of 1,000,000 people. As the disease affects one out of every 1,000 people, we know that 1,000 people will be infected.

Of the one thousand people that are infected 990 will be told they have the disease and 10 will test negative.

Of the remaining 999,000 people who don't have the disease, 1% (i.e. 9,990) will be told they tested positive and the remaining 989,010 will be told they tested clear.

So a total of 990 + 9,990 = 10,980 people will be told they tested positive and of those only 990 people will really be ill.

So if you are told you tested positive for the disease, the chances that you actually have it are:

990 / 10,980 = 9.016393443 %

Paul C. Anagnostopoulos
27th April 2004, 11:37 AM
Wrath said:
It must be nice to be able to psychically determine what someone's position is before they state it and tear apart the holes in the arguments they haven't made yet.
You mean there wasn't a hidden agenda here? Wow, fooled me, too.

~~ Paul

Wrath of the Swarm
27th April 2004, 11:40 AM
Correct.

This is why people shouldn't be overly concerned about screening tests that return positive results. One positive HIV test doesn't mean very much - which is why when someone is found to be HIV positive, a second round of testing commenses with a more expensive but higher-quality test that's less likely to give the wrong answer.

Of course, it's always possible that some unlucky person will get a false positive for multiple tests... but that's not as bad as the poor saps who get a false negative and never go on for more testing.

Anyway, it has been shown that a very large number of medical students have problems with this question - and even doctors interpreting the results of things like mammograms, PSAs, and HIV tests. A lot of research has gone into ways of presenting test data that are less likely to cause people to reach the wrong conclusions. When results are returned in terms of population frequency, people are much less likely to misunderstand what the tests mean.

Rolfe
27th April 2004, 11:43 AM
Hmmm, accuracy determined as the percentage of overall tests performed which are correct, irrespective of whether they are positive or negative.

This depends absolutely on the population you choose to test.

If you are testing overall a population which has a low disease incidence, you will come to the conclusion that virtually all your positives are wrong and virtually all your negatives are right. Thus so long as the test has good specificity, that is not spewing out too many false positives (99% is bloody brilliant), it will seem to have great "accuracy" no matter how bad the sensitivity.

If almost all the patients you test are unaffected, almost all your negative results will be right even if the test is actually missing quite a high proportion of affected individuals. You could have a sensitivity of only 50%, missing half of the true positives, but still claim 99% "accuracy" in this way. And it has been done.

However, such a test will be useless to you if you are testing sick individuals you suspect of having the disease. It willl miss half of the real cases.

This is why the term "accuracy" is meaningless. First it is made up of sensitivity and specificity, whch will almost certainly be different, and secondly if you're looking at overall numbers of "correct" results, you can get any answer you want just by choosing how you use the test.

Lousy sensitivity - display the figures of how it performs as a well-animal screen. You can't lose.

Lousy specificity - assume that the user will only be using it where the condition is strongly suspected on clinical grounds. It may still not look wonderful, but you can make it a lot rosier than it really is.

I've seen both ploys used to make things look better than they are. I'm wise to it. That's one of the reasons they ask me to scrutineer papers submitted to a number of professional journals.

Oh, hold still for one of the only two jokes I ever invented all by myself.

________________________________________

<CENTER>New! Cutting-edge technology! Statistically proven!

<FONT SIZE="+3">THE
NEG-TEST™</FONT>

Over 99.5% of all negative results guaranteed correct!*

NEVER produces a false positive!

Simple and inexpensive!

<FONT SIZE="-1">Method: Simply take the Neg-Test™ ballpoint pen provided, find the cat's clinical record, and write the words "FeLV negative". That's it. No need to take a blood sample, no messy reagents, no fiddly timing, no laboratory skill required.</FONT>

Change to the Neg-Test™

* <FONT SIZE="-2">Statistics only valid when the prevalence of infection in the population being tested is less than 0.5%.</FONT></CENTER>

____________________________________________

OK, you can quit with the hysterical laughter now. :D

Rolfe.

Wrath of the Swarm
27th April 2004, 11:48 AM
Originally posted by Paul C. Anagnostopoulos

You mean there wasn't a hidden agenda here? Wow, fooled me, too.
The "hidden agenda" was to permit me to begin a discussion of why people in general (and sometimes physicians) have problems with the question. Also to demonstrate that certain people aren't nearly as knowledgeable as they think they are.

For the record: I approve of most of modern medicine, and disapprove of most of alternative "medicine". It's the stuff I don't approve of in modern medicine that bothers me - and the unwillingness of some to admit that it could be made better.

pgwenthold
27th April 2004, 11:53 AM
Originally posted by ceptimus
Hmmm. Seeing as the cat is already out of the bag :) let's work this out using a population of 1,000,000 people. As the disease affects one out of every 1,000 people, we know that 1,000 people will be infected.

Of the one thousand people that are infected 990 will be told they have the disease and 10 will test negative.

Of the remaining 999,000 people who don't have the disease, 1% (i.e. 9,990) will be told they tested positive and the remaining 989,010 will be told they tested clear.

So a total of 990 + 9,990 = 10,980 people will be told they tested positive and of those only 990 people will really be ill.

So if you are told you tested positive for the disease, the chances that you actually have it are:

990 / 10,980 = 9.016393443 %

Read what Rolfe has written. This will be the case if the test is administered to everyone. However, if you administer the same test only to those who show other clinical signs, then the incidence of the _tested population_ is far higher than 1/1000.

Let's use a pregnancy test as an example. Suppose the pregnancy test is 99% accurate (in both directions) and that 1/1000 women are pregnant. If every woman is given a preg test, then 90% will be false positive.

OTOH, suppose that the only women who get tested are those who have missed a period. Now, there are lots of reasons to miss a period, but the main one is pregnancy. Let's say that 80% of the time when a woman misses a period, it is because of pregnancy. Thus, if only the women who have missed a period are given an exam, then 800/1000 would be pregnant. For a 99% test, 792 of the pregnant women would test positive, but 2 non-pregnant would test positive. Thus, the probably of being pregnant, given a missed period and a positive preg test, is 99.75%.

This is the same point that Rolfe has been making. Tests are not carried out in a vacuum.

Now throw in a woman who has not only missed a period but is also suffering morning sickness. At that point, the positive test is even more solid.

Rolfe
27th April 2004, 11:58 AM
Originally posted by Wrath of the Swarm
I said the test is 99% accurate. That sets both values.Please be more clear, Wrath.

When you say that the test is 99% accurate, do you mean that 99% is the arithmetical mean of the sensitivity and the specificity, or do you mean that 99% of the results you get when you actually do the test are correct?

If the former, then I submit that 99% accurate could describe a test with 98% sensitivity and 100% specificity. In which case the doctor would be right anyway.

If the latter, then it would depend entirely on the percentage of the tested population which is actually being affected, and on the (possibly differing) values of sensitivity and specificity. (In the usual scenario, the majority of the tested population is assumed to be unaffected. This means that a test with great specificity will always look very good, no matter how lousy the sensitivity, while a test with great sensitivity may look diabolical if the specificity is poor.)

Thus one can be misled into thinking that good specificity is what matters and to hell with the sensitivity - especially for screening well patients.

In fact the opposite is true. You need almost perfect sensitivity quite desperately. Because you don't want to have to keep doubting and re-checking all your negative results, which will after all be in the large majority. If you can trust a negative result to be highly unlikely to miss an affected individual, then double-checking all your positives, within reason, isn't too much of a chore.

A test with 99.5% sensitivity and only 95% specificity is much more use to me in a screening situation than one with 99.5% specificity and only 95% sensitivity. That's because I can virtually rely on the negatives and only have to recheck 5% (or a bit more) of my results, the positives, with the latter. With the former, I can't rely on either the positives or the negatives.

But the former has better "accuracy" according to the second definition, in a mostly-unaffected population.

However, I'd settle for knowing which definition of "accuracy" you were using, for a start.

Rolfe.

Rolfe
27th April 2004, 12:02 PM
Originally posted by pgwenthold
This will be the case if the test is administered to everyone. However, if you administer the same test only to those who show other clinical signs, then the incidence of the _tested population_ is far higher than 1/1000.

Let's use a pregnancy test as an example. Suppose the pregnancy test is 99% accurate (in both directions) and that 1/1000 women are pregnant. If every woman is given a preg test, then 90% will be false positive.

OTOH, suppose that the only women who get tested are those who have missed a period. ....pgwenthold, I think I'm in love with you.

You know, I have to explain this concept to two groups of people. Those who haven't originally heard it Wrath's way, for whom the light bulb comes on almost at one, and those who have heard the "predictive value" spiel without really thinking about what representative of the population in question actually means. They have a great deal of trouble, usually.

Rolfe.

Mercutio
27th April 2004, 12:03 PM
Originally posted by Wrath of the Swarm

Anyway, it has been shown that a very large number of medical students have problems with this question - and even doctors interpreting the results of things like mammograms, PSAs, and HIV tests. A lot of research has gone into ways of presenting test data that are less likely to cause people to reach the wrong conclusions. When results are returned in terms of population frequency, people are much less likely to misunderstand what the tests mean. It seems to me you have been asked for your sources for this more than a couple of times in this thread.

There is, of course, a long line of research on cognitive heuristic use (your problem is one example of the "base-rate fallacy" within this literature). I don't know which sources you are refering to, but I am guessing it is probably Kahneman & Tversky, one of several different publication dates...

Anyway, this paper (http://www.amstat.org/publications/jse/v9n3/keeler.html) sums up quite a bit of the research--I don't see your particular claim in it, but I only did a quick once-over of the paper. If you have another source or sources in mind...please cite them.

Rolfe
27th April 2004, 12:07 PM
Originally posted by Wrath of the Swarm
It's a good thing you can look up the answers on a chart, because you sure as hell can't handle the concepts involved. Sorry, I just realised what that implied.

Wrath, I wrote the spreadsheet. From scratch. In order to produce that graph I posted earlier, in order to demonstrate the absolute importance of assuming the correct value for the x-axis when deciding whether a result can be relied on or not.

Before anyone gets twitchy, yes, the graph was scanned in from a book. But as I am the author of the book, I think this is allowed, yes?

Rolfe.

drkitten
27th April 2004, 12:08 PM
Originally posted by Wrath of the Swarm
I did say that. I said the test is 99% accurate. That sets both values. If I said that the test would correctly identify a person with the condition 99% of the time, then there wouldn't be enough information for anyone to answer the question - you'd know the false negative rate, but not the false positive. But that isn't what I said.

It's a good thing you can look up the answers on a chart, because you sure as hell can't handle the concepts involved. [/B]

The situation isn't as clear-cut as you seem to think, Wrath. It's simply sloppy writing to cite one number and to assume that it applies equally to both the alpha and beta error rates. Another equally legitimate interpretation is that the test has a 99% accuracy rate in practice, but that figures aren't available to support breaking them out into false-positive and false-negative rates.

The standard terminology exists for a reason. Use it.

Your central point, however, is well-taken. This is a classic med-student error. I believe, however, that most experienced physicians have seen enough to know abou this error. Have you relevant evidence on medical error rates? The JREF forum is hardly typical of medical practitioners in either math sophistication or medical training.....

Wrath of the Swarm
27th April 2004, 12:10 PM
Originally posted by Rolfe
Hmmm, accuracy determined as the percentage of overall tests performed which are correct, irrespective of whether they are positive or negative.

This depends absolutely on the population you choose to test. No. Think about what you're saying. If all that matters is whether the result is correct, the distribution of the condition in the population is irrelevant unless there are different error rates for positives and negatives.

In this hypothetical 99% accurate test, it doesn't matter one bit whether everyone tested doesn't have the condition, everyone has the condition, or there's some intermediate state. 99% of the results are accurate, and 1% are not.

The strength of conclusions drawn from the results will depend on the population - but that's not what we're talking about. The power of the test is not the same as its accuracy.

Thank the beneficient powers you don't deal with people.

Wrath of the Swarm
27th April 2004, 12:13 PM
Originally posted by Mercutio
It seems to me you have been asked for your sources for this more than a couple of times in this thread. Yes, I know.

The basic problem is a classic one. I'm trying to find the sources in which I read about the implications for screening tests several years ago.

If I recall correctly, doctors get the right answer more frequently than the general population, but they still tended to reach grossly wrong conclusions about whether a particular patient had a disease. I believe they overestimated the power of the tests significantly.

If I find some good sources on the subject, I'll get back to you.

Wrath of the Swarm
27th April 2004, 12:19 PM
Originally posted by drkitten
It's simply sloppy writing to cite one number and to assume that it applies equally to both the alpha and beta error rates. Another equally legitimate interpretation is that the test has a 99% accuracy rate in practice, but that figures aren't available to support breaking them out into false-positive and false-negative rates. Overall accuracy includes both forms of error. If it's not stated that the probabilities can be further broken down, then there's no reason to presume that they do.

It's not sloppy. I avoided unnecessary complexities (which people are now trying to hide behind, I see.)

Wrath of the Swarm
27th April 2004, 12:22 PM
Well, I found this (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=11036111) at PubMed. I've found several other references to the research finding that doctors frequently have problems with Bayesian inferences, but not the research itself.

It's common knowledge within the profession, though. Let me keep looking.

geni
27th April 2004, 12:23 PM
Originally posted by Wrath of the Swarm
Overall accuracy includes both forms of error. If it's not stated that the probabilities can be further broken down, then there's no reason to presume that they do.

Context is import. In the JREf forums making that kind of asumption with this kind of question is probaly means you are getting the wrong answer (check the puzzels section if you don't belive me)

drkitten
27th April 2004, 12:26 PM
Originally posted by Wrath of the Swarm
Yes, I know.

The basic problem is a classic one. I'm trying to find the sources in which I read about the implications for screening tests several years ago.

If I recall correctly, doctors get the right answer more frequently than the general population, but they still tended to reach grossly wrong conclusions about whether a particular patient had a disease. I believe they overestimated the power of the tests significantly.

If I find some good sources on the subject, I'll get back to you.

Daniel Kahneman won the Nobel Prize for this some years ago. The classic work on cognitive errors in the general public is Judgement Under Uncertainty, but I assume you have something more specific for medical professionals?

Wrath of the Swarm
27th April 2004, 12:28 PM
9. For the base-rate neglect question, the important finding from these studies (see also Hogarth and Einhorn, 1992, and Robinson and Hastie, 1985) is that the order in which people get the information makes a difference. Although it shouldn't make any difference what order they get information in, subjects usually put greater weight on the most recently received information (Adelman, Tolcott, and Bresnick, 1993, with military intelligence experts dealing with realistic military intelligence problems; Tubbs, Gaeth, Levin, and Van Osdol, 1993, with college students on everyday problems such as troubleshooting a stereo; Chapman, Bergus, Gjerde, and Elstein, 1993, with medical doctors on a realistic diagnosis problem). In more ambiguous situations the first impression had a lasting effect (Tolcott, Marvin, and Lehner, 1989).11. Does it matter that people cannot accurately revise numerical probabilities (Christensen-Szalanski, 1986)? The deeper study of what people actually do, as called for by Koehler, can provide perspective. What do doctors do, for example, when ideally they should be forming hypotheses and revising hypothesis probabilities as they gather evidence?

12. It is not that they do a numerical integration more complex than Bayes' Theorem to revise probabilities (Gregson, 1993), as Hamm's (1987) explorations show. Doctors thinking aloud about cases don't even speak explicitly of probabilities (Kuipers, Moskowitz, and Kassirer, 1988), though when they are induced to do so it improves their decisions (Pozen, D'Agostino, Selker, Sytkowski, and Hood, 1984; Carter, Butler, Rogers, and Holloway, 1993).

13. Nor do doctors rely exclusively on learning probabilities from experience, like rats learning the contingencies on a lever (Spellman, 1993). While some of their knowledge is based on this kind of experience (Christensen-Szalanski and Beach, 1982; Christensen- Szalanski and Bushyhead, 1981), doctors have to know what to do with both the common diagnoses (8 out of 10) and the rare ones (1 in 10,000). Though in some situations, where people experience an event repeatedly, they can implicitly learn a base rate, in other situations, where people do not experience an event repeatedly but rather learn about it abstractly, they may also be able to take account of a base rate—but if they cannot, the consequences may be important.

14. How, then, do doctors usually handle diagnostic problems? Experts generally organize their extensive knowledge into mental scripts (Schmidt, Norman, and Boshuizen, 1990), complex rules that function with the speed of recognition to provide responses for familiar and unfamiliar situations. Explicit calculation of Bayesian probabilities is not a strength of this type of rule (cf. Hamm, 1993). Instead, experts' accuracy may be a function of the recognition processes, which can bring ideas to mind optimally (Anderson and Milson, 1989). Or accuracy may be due to well-tuned judgment processes governing response choice (Chapter 8 of Abernathy and Hamm, 1994).

15. If doctors' scripts are used accurately, producing results similar to those that wise use of Bayes' theorem would produce, this is due not only to the feedback of experience but also to reflection and to others' criticism (Chapter 11 of Abernathy and Hamm, 1994). Any form of argument can be applied toward justifying a change in a script, including arguments based on probabilistic analysis.

16. For example, when the screening tests for HIV first came out, Meyer and Pauker (1987) warned against ignoring the base rate, i.e., against assuming that someone with no risk factors has AIDS if their screen is positive for AIDS. Guided by such explicit discussion of the probabilities, and by individual cases of people devastated by false positive HIV screens, doctors' shared scripts were adjusted until now they don't recommend that patients be screened unless there are risk factors. The "1993 script" produces behavior that is, for the most part, consistent with a Bayesian analysis. Individual doctors using the script need neither think about probabilities nor understand the Bayesian principles. They just think of the rules, or of cases in which the script is implicit (Riesbeck and Schank, 1989). Note, of course, that this scenario depends on there being someone who understands the probabilistic principles and can shape the script that everyone else will use.

From this site (http://dieoff.org/page19.htm)

Paul C. Anagnostopoulos
27th April 2004, 12:29 PM
Wrath said:
Thank the beneficient powers you don't deal with people.
Sheesh.

~~ Paul

pgwenthold
27th April 2004, 12:38 PM
Originally posted by Rolfe
pgwenthold, I think I'm in love with you.

"First you have to move that damn cat."

Oh, sorry.

hey, I have a high affinity for vets (my wife begins fourth year rotations in 2 weeks). However, I'm not a cat person. We wouldn't get along. Besides, the aforementioned wife wouldn't go for it.

You know, I have to explain this concept to two groups of people. Those who haven't originally heard it Wrath's way, for whom the light bulb comes on almost at one, and those who have heard the "predictive value" spiel without really thinking about what representative of the population in question actually means. They have a great deal of trouble, usually.

Actually, I am one of the latter, and am very familiar with the John Allan Paulos take on the matter. However, you made a good point about testing populations. Since I don't know much about the tests for feline leukemia, I figured I'd put it in terms that most people would recognize.

Wrath of the Swarm
27th April 2004, 12:53 PM
This site (http://yudkowsky.net/bayes/bayes.html) has a nice discussion of the issue in simple terms. More importantly, it references research studies and asserts that the problem has been replicated many times.

Okay, so it's not a stellar reference... but I think it proves my point. My problem is what medical resources don't discuss the issue much - you'll find a lot more if you do a general Google on "do doctors have problems with Bayesian reasoning?"

Rolfe
27th April 2004, 12:56 PM
Originally posted by Rolfe

When you say that the test is 99% accurate, do you mean that 99% is the arithmetical mean of the sensitivity and the specificity, or do you mean that 99% of the results you get when you actually do the test are correct?

If the former, then I submit that 99% accurate could describe a test with 98% sensitivity and 100% specificity. In which case the doctor would be right anyway.

If the latter, then it (gets considerably more complicated....)This seems to have been missed. Please address. (Unless you did while I was writing this post, sorry, carried away again.)

I seem to have posted once assuming Wrath meant the former, then a second time assuming he meant the latter, then I see from yet another post that maybe he means the former after all.

Not hiding behind anything, Wrath.

Reference to simplistic form of the explanation that Wrath is peddling, at least the one that has caused me the most grief over the years.

JACOBSON, R. H. (1989) How well do serodiagnostic tests predict the disease status of cats? J. Am. Vet. Med. Ass. 199 (10), 1343-1347.

My pet hate quote from this pile of misinformation:A negative test result .... is reliable in predicting that a cat does not have the infection/disease.

.... negative test results are good prognosticators of non-infected cats even if the sensitivity .... of the test is not good.The example he used was that a sensitivity of 90% was just peachy, because in his scenario (only 1% of the "population" infected), 99.9% of the negative results would still be correct. He even remarked that even if the specificity was only 20% (!), this was OK because >99% of the negative results would still be correct.

It was at this point I was driven to grasp him methaphorically by the throat and point out that 90% sensitivity was still missing 10% of all infected cats, and I really didn't want that. 20% sensitivity is missing an incredible 80% of the infected cats, and there's no way this can be acceptable except by his crazy logic (which Wrath never extrapolated to, but it's where it goes if you don't rein it in).

Of course, this is where my "NEG-TEST<SUP>TM</SUP>" was born. The reductio ad absurdum of his premise is that if the sensitivity is zero, nevertheless, 99% of the negative results are still correct. The NegTest has a sensitivity of zero. I just tweaked it a little by reducing the hypothetical incidence of infection to 0.5% (not unreasonable in a healthy population, and iin fact if you are talking closed and tested pedigree breeding establishments even 0.5% is a gross libel).

I don't know whay the reductio ad absurdum wasn't spotted when the paper was published, but it wasn't.

This is the reason I have this explanation honed - it seems to imply that sensitivity doesn't matter, and so long as specificity is good (not too many false positives), you're laughing.

Of course, as I said above, the opposite is the case. For a viable screening test, you must be able to trust your negatives, not just trust to luck that the cats you're testing are in fact negative! If you can trust the negatives, you only need to get the (relatively few) positives double-checked. No problem. If you know that the bloody test is missing 10% or more of the infected cats, why do it at all?

In fact Jacobson did say something perfectly sensible in his paper.When evaluating a serodiagnostic test result, the veterinarian should first consider whether the cat is at high risk (from a high prevelance group) or low risk (from a low prevalence group) for the condition under consideration.The problem is that he didn't understand that "population" doesn't mean "the village where the cat lives", it means "cats like this one". Including the clinical presentation.

He spent the entire five pages only looking at the left-hand side of the graph, because he couldn't imagine a (geographical) population with more than about 10% incidence of infection. Of course, he isn't a veterinarian. He simply didn't think about the selection-to-test based on clinical presenting signs and the "population" you will be testing if you choose (as many vets do) to test only cats presenting with clinical signs suggestive of infection by the virus.

Once you think about that scenario, you realise you are way up to the right-hand-side of the graph, and positive results become relatively reliable while negative results are untrustworthy as hell.

And of course a "population" in this sense can be one cat, with all its features which put it closer to one side or the other of the graph. Indeed, the bottom line is you don't think about what other cats you tested or didin't choose to test that day, or week, or year, you assess that cat as an individual with probability/risk of infection of x.

I know it's not easy to get your head round, especially if you've got the sloppy version strongly pre-conceived. But it would be nice if Wrath at least read my posts.

Rolfe.

Rolfe
27th April 2004, 01:02 PM
Sorry, this is unworthy, but I'm getting a bit narked. (I only just saw the word I suspect led to the reporting of the thread, and yes, I'm not terribly flattered.)

Is it relevant that Wrath had to search PubMed to find something to back him up, after he'd been called on it? Whereas I reached for journals already in my bookcase, and was able to illustrate my point with a graph copied from a book of which I am in fact the author?

Rolfe.

Wrath of the Swarm
27th April 2004, 01:02 PM
I have read them. The question is whether you ever do - or if you think about what you read.

The quote you've presented from this person you argued with is utterly correct. "A negative test result .... is reliable in predicting that a cat does not have the infection/disease. .... negative test results are good prognosticators of non-infected cats even if the sensitivity .... of the test is not good."

That's the simple truth. It may not be a desirable test in a wider sense because it misses infected cats... but that has nothing to do with the points you quoted.

Obviously, the population tested will affect the results, in the sense that if everyone has the condition, there's no such thing as a false positive (or if everyone lacks it, there's no false negative and so forth). What's important is the accuracy of the test (whether in general or in distinguishing between alpha and beta error), because that is an objective, universal property of the test that doesn't change across populations.

Your hysterical rantings don't echo, Rolfe.

Wrath of the Swarm
27th April 2004, 01:07 PM
Why perform a test that misses 10% of the infected patients?

Maybe because not performing the test misses 100% of the infected patients by definition.

Because tests with low false negative rates usually also have high false positive rates? Because overall accuracy in diagnostic testing is extremely difficult to accomplish?

No, obviously he's just performing the test to make money off of his victimized clients, and you're swooping in to save the day! ;)

ceptimus
27th April 2004, 01:27 PM
If we have a test for a disease that doesn't exist in the population (say we have a test for smallpox - and smallpox has been eradicated) then no matter how 'good' the test is, any positives that it reports will be wrong.

I think when Wrath said the test was 99% accurate, and gave no further information, then you have to assume that out of every 100 people tested, whether they have the disease or not, 1 person will be told the incorrect result.

I realize this is not a likely scenario in the real world - I was just treating it as a puzzle. I like puzzles.

Rolfe
27th April 2004, 01:34 PM
Originally posted by Wrath of the Swarm
Your hysterical rantings don't echo, Rolfe. Anyone else care to say if I'm ranting hysterically?

Wrath, you made two classic errors of presentation when you posted that question. With, I note, the not-very-well-hidden agenda of showing how clever you are and how stupid medical professionals are.

The first was the one which was obvious to everyone, where you quoted an "accuracy" figure which was meaningless as it stood, without stating that you were implying that sensitivity and specificity were both 99%.

I assumed this was just a sloppy way of saying that specificity was 99%, because sensitivity wasn't relevant to the question anyway. You have however dug yourself a deeper and deeper hole by declaring that this "99% accuracy" is some sort of combined sensitivity and specificity figure. This is a meaningless concept. You can't simply take an arithmetical mean of the sensitivity and specificity and call it "accuracy", and to assume (and to assume that we would assume) that they were equal is ludicrous.

100% sensitivity, 50% specificity. 75% accuracy?
100% specificity, 50% sensitivity. 75% accuracy?
75% sensitivity, 75% specificity. 75% accuracy?

These are three very different products, and nobody in their right mind would consider them all under the same banner, as "75% accuracy". By the way, if these were all that was available, which would you choose to stock, and why?

We can go round the houses on this relatively minor point all night.

However, the more important point is, why did the doctor decide to do the test? He knows that, and he will take it into account in deciding whether to believe a positive result or not.

Scenario 1. Wrath goes to the doctor for an insurance medical, feeling fine. Doctor checks him over carefully, and finds nothing wrong. But the insurance form requires that he tick the box to test for Galloping Varicella, as a routine.

In that situation, a 9.02% probability that the test Wrath described is correct is actually an overestimate unless all people with Galloping Varicella are clinically normal. What you need is the incidence of Galloping Varicella in the clinically normal population - which will undoubtedly be less than the incidence in the population as a whole, which of course includes those who are in the last stages of terminal disease from the condition.

Scenario 2. Wrath goes to the doctor for an insurance medical, feeling fine. Doctor checks him over carefully, and notes a couple of worrying things. He has a mole in the middle of his back, where he can't see it, and there is a faint but just perceptible cast to his left eye. The doctor knows that these two features, found together, are very suggestive indeed of Galloping Varicella, in fact he is about 80% sure Wrath has the condition. Although nothing is said about it on the insurance form, he decides to perform the test. Of course, knowing that there is still a 20% chance he is wrong, he just tells Wrath that the test is "routine", to save possibly unnecessary worry.

In that situation, there is a 99.75% probability that a positive result is right.

Do we criticise the doctor if he decides to break the bad news at that point?

This is a classic error. Look at a problem from outside, ignoring the inbuilt assumptions with regard to way of working that people build up over the years. Assume one scenario, and only one, because you don't have the experience to imagine any other. Then ambush some professionals with your assumed scenario, and completely fail to realise that they may be (consciously or unconsciously) answering the question from the point of view of a different scenario, the scenario they are familiar with.

The fact is, there is always a reason for testing, and that reason is part of the interpretation. Insurance requirement, although no suspicion? Strong clinical suspicion? Wrath believes that doctors do the test without thinking about this. They don't. But it's often so instinctive that you can make them look unreasonably stupid by pulling a Wee Kirkcudbright Centipede (http://sniff.numachi.com/~rickheit/dtrad/pages/tiKIRKCPED.html) on them. (Note, that text is all that is available, but it is the result of a bad transcription from a sung version by an American who didn't understand the lingo. For a start, the dance is the "Palais Glide", not the "parlor glide". What were they thinking of!)

And just to note this again, five out of five posters got the "right" answer before I posted a syllable, so Wrath's stated purpose of showing how bad the people on this forum are at this sort of reasoning was doomed from the start.

But we're having a much more entertaining discussion now, aren't we? :D

Rolfe.

Late edits only for spelling typos.

Dragon
27th April 2004, 01:37 PM
Originally posted by Wrath of the Swarm
This site (http://yudkowsky.net/bayes/bayes.html) has a nice discussion of the issue in simple terms. More importantly, it references research studies and asserts that the problem has been replicated many times.

Okay, so it's not a stellar reference... but I think it proves my point. My problem is what medical resources don't discuss the issue much - you'll find a lot more if you do a general Google on "do doctors have problems with Bayesian reasoning?"

From the link (using breast cancer as an example) - Figuring out the final answer always requires all three pieces of information - the percentage of women with breast cancer, the percentage of women without breast cancer who receive false positives, and the percentage of women with breast cancer who receive (correct) positives. Oh dear - how many pieces of information in your OP?

Dragon
27th April 2004, 01:48 PM
On the subject of Bayes' Theorem - this thread (http://www.randi.org/vbulletin/showthread.php?s=&threadid=39384) that ca3799 has just started in GS&P got me thinking - surely it applies to polygraph testing? Can't find anything on http://antipolygraph.org/ about it yet - hmmm....

pgwenthold
27th April 2004, 02:02 PM
Originally posted by Rolfe

And just to note this again, five out of five posters got the "right" answer before I posted a syllable, so Wrath's stated purpose of showing how bad the people on this forum are at this sort of reasoning was doomed from the start.

But we're having a much more entertaining discussion now, aren't we? :D

Oh, but it is fun. The clown came in as pompous as could be intent on showing us how much smarter he was than everyone else, and then after fooling no one, got schooled big time.

After having it blow up in his face, then he runs to the literature (where, as demonstrated by Dragon, blows it again), apparently abandoning his attempted exercise.

Entertaining it has been, hmmmm?

Wrath of the Swarm
27th April 2004, 02:11 PM
Originally posted by Dragon
From the link (using breast cancer as an example) - Oh dear - how many pieces of information in your OP? All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

It's nice to see that most of the people who bothered responding did indeed choose the correct answer, although now we'll never know how many formuites would have chosen differently. A shame.

Dragon
27th April 2004, 02:17 PM
Originally posted by Wrath of the Swarm
All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

...
Nope - we had to assume what you meant by "accuracy". Rolfe has explained this to you already.

Rolfe
27th April 2004, 02:23 PM
Just a comment.

The first (permanent) part of my sig line wasn't chosen by accident.

Rolfe.

Wrath of the Swarm
27th April 2004, 02:25 PM
Originally posted by Rolfe
Wrath, you made two classic errors of presentation when you posted that question. With, I note, the not-very-well-hidden agenda of showing how clever you are and how stupid medical professionals are. Pointless character assassination. The problem certainly doesn't make me look any smarter - I failed it the first time I saw it, many years ago.

It does make you look dumber, but since you're not a medical professional that doesn't quite count, does it?

The first was the one which was obvious to everyone, where you quoted an "accuracy" figure which was meaningless as it stood, without stating that you were implying that sensitivity and specificity were both 99%. I wasn't "implying" it - it's a consequence of what I said. Sloppy interpretation.

You have however dug yourself a deeper and deeper hole by declaring that this "99% accuracy" is some sort of combined sensitivity and specificity figure. This is a meaningless concept. You can't simply take an arithmetical mean of the sensitivity and specificity and call it "accuracy", and to assume (and to assume that we would assume) that they were equal is ludicrous. No one's claimed anything about an arithmatic mean - except you.

And it's certainly not ludicrous for the alpha and beta rates to be equal. It's somewhat improbable in the same sense that it's improbable for any two values to be the same, but there's no reason it can't happen.

That's two 'misinterpretations' made by you.

These are three very different products, and nobody in their right mind would consider them all under the same banner, as "75% accuracy". Except anyone who uses English in the standard manner. Of course, we weren't considering such a situation - the accuracy of the test was established without reference to alpha and beta rates.

We can go round the houses on this relativly minor point all night. But that would involve discussing how wrong your statements are and how pointless your objections have been, and we don't want that.

In that situation, a 9.02% probability that the test Wrath described is correct is actually an overestimate unless all people with Galloping Varicella are clinically normal. What you need is the incidence of Galloping Varicella in the clinically normal population - which will undoubtedly be less than the incidence in the population as a whole, which of course includes those who are in the last stages of terminal disease from the condition. That's two tests. The first test involves an examination of the obvious symptoms - it has an accuracy all its own. Considering the results of test B in the light of test A is perfectly reasonable medical practice - but it's not an effective way to determine the accuracy of test B.

You have continually ignored this point, I suspect because you know you've made a major error and want desperately to direct attention away from it.

In the question, there were no other tests specified other than the one I gave you all information on. The potential ability to perform other tests has no bearing on the question I asked.

In that situation, there is a 99.75% probability that a positive result is right. True. When we discuss that situation, we'll drop you a line.

Wrath believes that doctors do the test without thinking about this. Liar. You have no idea what I believe, so you make up a position for me that you know you can successfully attack.

The point is that not this is how medical testing is performed. The point is that doctors fail to answer the question correctly. Being the self-appointed forum apologist for the field of medicine in general, you leap to explain how doctors base their judgments on additional clinical data, blah blah blah... ignoring the point that THEY CAN'T CARRY OUT A SIMPLE MATH PROBLEM.

Wrath of the Swarm
27th April 2004, 02:29 PM
Originally posted by Dragon
Nope - we had to assume what you meant by "accuracy". Rolfe has explained this to you already. If I say that I can identify a randomly-chosen card while blindfolded with 80% accuracy, would you have problems understanding that as well? Would you demand I offer accuracy ratings for each type of card?

The test has 99% accuracy; without further specificiation, that means that any response it gives has a 99% chance of being correct and a 1% chance of being wrong. There's your alpha and beta rates right there.

No further categorization is given; none is needed. You had all the information needed to answer the provided question.

geni
27th April 2004, 02:41 PM
I didn't. I don't know the standard equations so was trying to work it out from scratch and ran into an equation with two unknows which is of course unsolverble. You card analogy is false since the way you have stated it the question being asked is different from t he one in this thread.

Wrath of the Swarm
27th April 2004, 02:45 PM
Two unknowns? Then you certainly weren't working the problem properly.

It is perfectly reasonable to talk about a test that has an accuracy rating. Not all tests have different rates of false positives and false negatives - and even if they did, we don't always care.

When given an accuracy and the population prevalence, you had sufficient information to solve the problem. I could have made it more complex and somewhat more realistically probable, but that not only wasn't necessary, it would have invalidated my point that doctors were unable to answer the question correctly. If I changed the question, why would I bring up those studies?

Rolfe
27th April 2004, 02:51 PM
Sob.

There are an infinite number of answers to this question, based on the information Wrath didn't clarify. However, Wrath chooses to declare only "his" answer to be correct.

Once again, Wrath, the scenario you post cannot exist in the abstract. You didn't tell us why the doctor did the test.

If it was for no other reason than because it was a box that had to be ticked (for example in an insurance medical), your 10% probability of the positive result being correct is in fact an overestimate, because you didn't tell us the incidence of the condition in the population with no suspicious clinical signs (less that 0.1% obviously, but we don't know how much less).

If it was because you came in with clear clinical symptoms suggestive of the condition, then the probability of the positive result being correct is pretty high (depending on a number of clinical factors).

Forgive me if I'm inclined to assume that if the doctor decided the result was correct, it might have been because he knew he was in the latter scenario.

You cannot put forward a hypothetical situation like this, then get miffed when people point out that your "correct" answer is only correct if a number of details which you haven't specified are exactly the way you have tacitly assumed them to be.

You did imply that the doctor's appointment was for a routine checkup, without any particular presenting signs. That's fine. But you didn't say why the doctor wanted to check for the condition. Now you impose more conditions than you originally stated, that he was just doing it for fun, or the greater enrichment or the laboratory, or (more likely) the test was a condition of an insurance policy or an employment contract. It could easily have been because the doctor's clinical acumen smelled a very aromatous rat.

You, however, want to assume the scenario that makes the doctor look a fool.

Now, for God's sake put me out of my misery and tell me what the bloody blue blazes that "99% accuracy" figure is supposed to mean. Stop assuming that sensitivity and specificity are equal, I know and you know and the entire medical laboratory profession knows that only hypothetical tests come like that.

So, for a real-life test, like the ones I deal with every day, which have unequal sensitivity and specificity, how are you calculating what you call "accuracy"?the accuracy of the test was established without reference to alpha and beta ratesI assume that by "alpha and beta rates" you mean sensitivity and specificity - that's OK, we obviously come from areas with a different vocabulary. But I'd like to get this clear. So, if not like that, how in all that's holy was it established?

(Note, this part of the argument is not of my making. I originally assumed that Wrath meant 99% specificity, since specificity was the only figure relevant to the sum he had set. It's Wrath himself who keeps saying now that the 99% somehow incorporates both sensitivity and specificity. Not my problem if he can't then explain how.)

Wrath. Who can't carry out a simple maths problem? People here got it right. And if you still think I used a crib, I'll repeat that I wrote the spreadsheet I used myself, years ago, and only mentioned it to explain why I could do multiple scenarios of the problem relatively quickly.

We made the assumptions you wanted us to make. We got the "right" answer by your lights. However, we also realised where you were mistaken, which was in assuming that the reason for carrying out the test was irrelevant to the doctor's decision as to whether to go with the result or not.

Deal with it.

Rolfe.

yersinia29
27th April 2004, 02:56 PM
Wrath is a troll guys. Dont waste your time. He wont address your points, and will just continue to spill his bile. He gets a philosophical thrill out of trying to obfuscate the arguments involved.

Wrath, I'm still waiting for your Bayesian analysis of MRI and x-rays.

Wrath of the Swarm
27th April 2004, 02:58 PM
Why the doctor performed the test has no bearing on the correct answer! Does it matter why Farmer Brown took away three apples from the box that held seven? No!

And there aren't an infinite number of answers. Without specifying different values for alpha and beta, we consider only error. Alpha and beta values follow from the overall accuracy.

Again: the hypothetical test was 99% accurate, so there was a 99% chance that any result it came up with would be correct. That tells you what the alpha and beta rates are - they're equal in this particular case.

Even you aren't stupid enough not to recognize this, so I'm forced to conclude you're being intentionally deceptive in order to support your 'point'.

yersinia29
27th April 2004, 03:10 PM
Originally posted by Wrath of the Swarm
Why the doctor performed the test has no bearing on the correct answer!

Of course it does you fool.

The incidence of lupus in the overall population of women is x%

The incidence of lupus in women with a butterfly rash, photosensitivity, and Raynaud's phenomenon is x+y%

doctors dont just run lupus tests on random women. Therefore, the incidence thats used in calculating false positives and other parameters depends on x+y, NOT x.

ceptimus
27th April 2004, 03:13 PM
Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

geni
27th April 2004, 03:20 PM
Originally posted by Wrath of the Swarm
Two unknowns? Then you certainly weren't working the problem properly.

We we need to know what 99% accurcy means. It means that some of the tests are giving false posertive or negatives. Therfore the inacurcies are due to either false posertives or negatives. What is is the ratio of these inacrucies ah can't work that one out on the data given problem is unsolverble.

It is perfectly reasonable to talk about a test that has an accuracy rating. Not all tests have different rates of false positives and false negatives - and even if they did, we don't always care.

But in this case we do case because it can have a big effect on the answer.

You used to work for edexcel didn't you? The problem as stated is unsolverble.

geni
27th April 2004, 03:22 PM
Originally posted by ceptimus
Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

Nope try pluging in the figurers for what hapens when the inacucay is entiry due to false negatives.

Rolfe
27th April 2004, 03:25 PM
Originally posted by ceptimus
Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers. If you thought that was clear, you don't understand the question.

What Wrath intended to be assumed was reasonably clear, because we know how his mind works, and by assuming that, the desired result was obtained.

However, he was dishonest because his unstated assumption was that the scenario was such that the doctor was wrong in assuming the positive result to be correct. It is equally if not more likely, in real life, that the scenario was not that assumed by Wrath, and that the doctor had a perfectly valid reason for assuming the result to be right.

Rolfe.

Wrath of the Swarm
27th April 2004, 03:32 PM
Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with.

Wrath of the Swarm
27th April 2004, 03:35 PM
Oh, and by the way: Rolfe is an excellent example of why the medical practitioners generally failed to answer the question properly:

They assumed facts not in evidence, and had excessive confidence in the ability of doctors to make accurate judgements.

Of course, Rolfe is not a qualified medical professional. So her inability to interpret a simple question properly means little.

ceptimus
27th April 2004, 03:37 PM
I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong result, regardless of whether the persons being tested are diseased, healthy, or any mixture of the two.

It follows from this assumption (which is the only sensible one to make, given how the original question was phrased) that the error rates for both false positives and false negatives is 1%

I think your familiarity with the subject is making you try to read things into Wrath's question that simply were not there.

Geni - if you look back through the thread, you will see I gave a simple worked out solution in my first post. Wrath gave quite sufficient information to allow the question to be answered fully. As I already said, if you wish to ask different questions (or choose to believe that Wrath did) then they will likely have different answers.

geni
27th April 2004, 03:40 PM
Originally posted by Wrath of the Swarm
Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Appeal to authority logical fallicy

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

next it going to be 100,000 european doctors isn't it I can just tell
Context is everything.

Point 3: The question, as it stands, is perfectly comprehensible.

If by that you mean that I can guess what you mean then yes. However without makeing this guess the problem is unsolverble

Wrath of the Swarm
27th April 2004, 03:45 PM
Originally posted by geni
Appeal to authority logical fallicy No, you fool! The next bit is the appeal to authority! That's just the "appeal to keeping experimental modalities the same".

If by that you mean that I can guess what you mean then yes. However without makeing this guess the problem is unsolverble I quite agree. The problem is completely unsolverble. No one can solverb it!

Flan flan flan flan...

geni
27th April 2004, 03:47 PM
Originally posted by ceptimus
I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong result, regardless of whether the persons being tested are diseased, healthy, or any mixture of the two.

It follows from this assumption (which is the only sensible one to make, given how the original question was phrased) that the error rates for both false positives and false negatives is 1%

(my Italics) I see no reason to assume. The error would be enogh to get the question throw out of an exam paper. The way Wrath of the Swarm persented the question made it clear that it was ment to throw you. In such cases it is vital that the question is sound and makes sure that the person trying to solve it does not have to make any assumptions. In this case an assumption had to be made for which I saw no reason to belive such an assumption should be relible. As such the question was unsolverble.

ceptimus
27th April 2004, 03:51 PM
Originally posted by geni

(my Italics) I see no reason to assume. The error would be enogh to get the question throw out of an exam paper. The way Wrath of the Swarm persented the question made it clear that it was ment to throw you. In such cases it is vital that the question is sound and makes sure that the person trying to solve it does not have to make any assumptions. In this case an assumption had to be made for which I saw no reason to belive such an assumption should be relible. As such the question was unsolverble. I think you are being unfair. If I told you a remote viewer was asked to view whether someone was in a town or the country, and they were right 99% of the time, what would you assume then?

geni
27th April 2004, 03:52 PM
Originally posted by Wrath of the Swarm
No, you fool! The next bit is the appeal to authority! That's just the "appeal to keeping experimental modalities the same".

they both are appeals to authority it's just the second one contians an appeal to popularity as well. You didn't keep the experimental modalities the same since you changed the context.

Rolfe
27th April 2004, 03:54 PM
Originally posted by Wrath of the Swarm
Why the doctor performed the test has no bearing on the correct answer! Does it matter why Farmer Brown took away three apples from the box that held seven? No!

And there aren't an infinite number of answers. Without specifying different values for alpha and beta, we consider only error. Alpha and beta values follow from the overall accuracy.

Again: the hypothetical test was 99% accurate, so there was a 99% chance that any result it came up with would be correct. That tells you what the alpha and beta rates are - they're equal in this particular case.This is confusing and conflating my two separate assumptions about what Wrath meant by "accurate" that I'm barely capable of disentangling them. It's now clear that Wrath had even less idea about what he was talking about than I realised. Breathtaking.there was a 99% chance that any result it came up with would be correctDo you realise that you've just soundly contradicted yourself? The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%.

There are only two ways I can see to get this "accuracy" figure.

An arithmetical mean of the sensitivity and specificity. If they were equal, then that would be right enough. But you've explicitly denied that this is how you calculate the figure.

Or the percentage of tests carried out in practice which are correct (positive or negative). This would seem more likely for a figure you now relabel as "error", but to calculate this you need all of the sensitivity, the specificity and the incidence of the condition in the population being tested.

Dream of a thousand cats (with apologies to Neil Gaiman).

1000 cats. Incidence of FeLV infection 10% (for whatever reason).
FeLV test, sensitivity 98%, specificity 95%.

We have 100 infected cats, and 900 uninfected cats.

Of the 100 infected, 98 are true-positive and 2 are false-negative.
Of the 900 uninfected, 855 are true-negative and 45 are false-positive.

Total results:
143 positive, of which 68.5% are correct.
857 negative, of which 99.8% are correct.

1000 results, of which 47 are wrong. Therefore 95.3% of the results on this population are correct. With the positives much more likely to be wrong than the negatives, as is quite often the case, special circumstances pertaining to individuals with very pathognomonic clinical presentations notwithstanding.

And you can see that if you plug in different values for the three original variables, you can get a wide variety of different answers.

OK Wrath. These are two ways of calculating "accuracy" to definitions I can comprehend. Now would you please do me the maths for your derivation of 99%?

And it's quite ridiculous to assert that because you gave only one figure, we should assume the same figure applies to sensitivity and specificity. This pretty much never happens in the real world. To say that since only specificity was relevant to the question, you therefore meant to say "specificity", is reasonable and it's what I originally assumed.

But if 99% is some calculated figure from sensitivity and specificity, I at least want to know how you are going to calculate it when the two values are not equal.

Rolfe.

geni
27th April 2004, 03:56 PM
Originally posted by ceptimus
I think you are being unfair. If I told you a remote viewer was asked to view whether someone was in a town or the country, and they were right 99% of the time, what would you assume then?

If 999 people in your sample were in the town and 1 in the country I know excatly what I would assume. The assumption can totaly mess up the results and as such is serious.

Wrath of the Swarm
27th April 2004, 03:57 PM
Originally posted by Rolfe
Breathtaking.Do you realise that you've just soundly contradicted yourself? The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%. Um, no.

The point of the thread was that, for a particular individual who had been given a positive result, there was only about a 10% chance they actually had the disease.

The chance that the test would give out the correct result was still 99%. But the disease was sufficiently uncommon that the chance the test would wrongly give a positive was much greater than the chance of a true positive.

Do you understand that the set of people given the test and the set of people who tested positive are not the same?

Would it help if I typed more slowly?

Doo yoou unnnderstaaaannnnd?

ceptimus
27th April 2004, 03:59 PM
Originally posted by Rolfe
The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%.No. Out of every 100 tests, 1 gave the wrong answer. You are misreading what Wrath said.

ceptimus
27th April 2004, 04:02 PM
This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

Rolfe
27th April 2004, 04:12 PM
Originally posted by Wrath of the Swarm
Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with. Trawling through the ad-homs to get to the argument, such as it is....

Point 1. I don't care whether the same flawed question was used to ambush doctors. Appeal to authority. Geni spotted the flaw too while I was typing my initial post, so it wasn't exactly subtle.

Point 2. Same as point 1, appeal to authority.

Point 3. Only if you make the effort to figure out the unstated assumptions.

Point 4. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.(a) Yes, there are plenty of tests that are used as screens. But whether or not that is the case in this particular instance is something you didn't see fit to tell us.

(b) Kindly define "error rates of the test". Show me the maths. I showed you mine. Specificity and sensitivity are independent of the composition of the population being tested. That is why they are the figures to look for when assessing a product. You can then plug these in to different "populations" to get positive and negative predictive value, which are. Your arguments seem to be slewing wildly between one definition and the other, which your lack of defining what you mean by either "accuracy" or "error rate" simply obfuscates completely.

Point 5. Combining sensitivities, specificities and clinical probability of infection to get an estimated predictive value for an individual test on an individual patient is more complex, I agree. Which is why I presented it as a graph (see page 1). You are making it needlessly complicated dragging in differential probabilities for individual clinical signs. While this might be a further refinement, to say "the clinical probability that this patient is affected is x%, to my most educated guess" is a perfectly workable way to go about it, and much superior to "people in general have a y% incidence of this condition" when you are dealing with a specified individual. It's not that hard.

Point 6.You're only making yourself look more like a fool the more you continue this, Wrath. Admit you were wrong and get it over with.I agree.

Rolfe.

geni
27th April 2004, 04:19 PM
Originally posted by ceptimus
This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

Problem is that the sensitivity and specificity are the two things that make up the accucery. The result is I end up with an equation looking something like this:

P<sub>1</sub>+P<sub>2</sub>=X now I know X but I don't know either of the other two values so I'm slightly stuck.

Rolfe
27th April 2004, 04:21 PM
Originally posted by ceptimus
This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear. Wrath made no mention, but he should have.

In fact, only the specificity is required for the calculation Wrath posed. Therefore I initially assumed that the sloppily-used "accuracy" figure was intended to be specificity. And stated this assumption clearly. It is Wrath himself who is denying this is what he meant.

Now, please think long and hard about the different things your second paragraph might mean, and the ways in which it is not "perfectly clear". Have another look at the "Dream of a thousand cats".

You are assuming that by 99% accurate, we can assume that 1% of unaffected individuals will test false positive, and 1% of affected individuals will test false negative.

That is simply an invalid assumption. Real tests in the real world have different values for these two figures, and they have to be quoted separately. You might say as a sweeping generalisation that a test was "highly accurate" if both figures were very good, but there's no meaningful way to combine them to a single "error rate" unless you do the entire thousand cats dance.

I'm not confused. I do this for a living. I have published a chapter in a book about it. And got very good book reviews from eminent professors, by the way.

I know what Wrath assumed, and I know what he wanted us to assume. That was clear from the first post. What is being discussed is the way this was set up without making these assumptions clear, and the fact that if you make other, equally valid assumptions, you get a completely different answer to the one Wrath wanted us to get.

Rolfe.

Rolfe
27th April 2004, 04:33 PM
Originally posted by Wrath of the Swarm
The point of the thread was that, for a particular individual who had been given a positive result, there was only about a 10% chance they actually had the disease.

The chance that the test would give out the correct result was still 99%. But the disease was sufficiently uncommon that the chance the test would wrongly give a positive was much greater than the chance of a true positive.Quit with the ad-homs, it just gives me eyestrain.

For a particular individual who gets the test, there are all sorts of different probabilities that he is actually affected. Depending on the assumptions you make.

Now, could you do me the maths again (oh sorry I mean for the first time) to demonstrate how you arrive at "the chance that the test will give out the correct result is 99%". Accuracy, error rate, I don't care what you call it, just TELL ME HOW YOU WORK IT OUT.

Now tell me how, if you are treating every individual the same, that is as members of this "population" with 0.1% incidence, you can still simultaneously declare that the chance the test has given out the correct result is only 9.02%.

Where are these test being done that have the 99% probability of being right, and what's it about this particular patient that gives him only a 9.02% chance of getting a correct result?

(Hint: You are going to have to consider the people getting the negative results here. I want to see the maths. And I want to bottom line to come out at 99% exactly, using the parameters you yourself have set.)

Rolfe.

Wrath of the Swarm
27th April 2004, 04:38 PM
Real tests in the real world do not necessarily have different values for the two numbers. They do frequently.

Your inability to comprehend his point makes the rest of your claims even more suspicious than they already are.

For my argument to be valid, I would have to use the same question as was used in the studies. That's a basic point of experimental design - which Rolfe clearly knows nothing about. You can't test the validity of an experiment without recreating its structure.

The second point *is* appeal to authority... just as Rolfe's claims about having written a book are appeals to authority. Who is more credible - psychological researchers, medical doctors, and statisticians, or Rolfe?

There are no unstated assumptions. The only person with assumptions is Rolfe, who can only think by rote and can't comprehend that alpha and beta values do not have to be specified, nor do they even have to be different.

Your attention-whoring is not going unnoticed.

Wrath of the Swarm
27th April 2004, 04:41 PM
Originally posted by Rolfe
Now tell me how, if you are treating every individual the same, that is as members of this "population" with 0.1% incidence, you can still simultaneously declare that the chance the test has given out the correct result is only 9.02%. Strawman. The conclusion from a positive result that a given person has the disease has the 9.02% chance of being correct.

The chance of the test being wrong for any person is 1%. The chance of the test being wrong for a particular subset of people isn't necessarily the same.

Don't you understand any statistics at all?!

Paul C. Anagnostopoulos
27th April 2004, 04:47 PM
Mentioning one's own book is an appeal to authority?

~~ Paul

Rolfe
27th April 2004, 04:49 PM
Originally posted by Paul C. Anagnostopoulos
Mentioning one's own book is an appeal to authority?Probably. One has to counter the ad-homs somehow.

Rolfe.

geni
27th April 2004, 04:51 PM
Originally posted by Wrath of the Swarm
For my argument to be valid, I would have to use the same question as was used in the studies. That's a basic point of experimental design - which Rolfe clearly knows nothing about. You can't test the validity of an experiment without recreating its structure.

The test is already invalid as a repeat due to the change in context. Thinking that you have equiverlence suggests that you havn't though th through properly

There are no unstated assumptions

Even your supporter dissagrees with you here. You assumption is that p<sub>1</sub>=p<sub>2</sub>. You can see this in you own calculations back on page one

Wrath of the Swarm
27th April 2004, 04:59 PM
Originally posted by Paul C. Anagnostopoulos
Mentioning one's own book is an appeal to authority? The implication is that she's an expert on the subject, as she had cause to write a book.

Since she's demonstrating a complete lack of comprehension in this thread, I'd hate to read her book.

geni: Nothing about the context was changed. May I ask what you think changed between my question and the one presented in the studies?

steve74
27th April 2004, 05:02 PM
Originally posted by Wrath of the Swarm
Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with.

With regard to point 1: I'd like to know how you know the question you asked was the same question asked in the research, when you fully admit that you don't can't remember which research you read?

I'd guess you are referring to Casscells et al. (1978) (certainly the most famous example of a base rate neglect study in med students) where a similar example was given to a group of faculty, staff and fourth-year students at Harvard Medical School. Only 18% got anywhere near the correct answer. The question, in their study, was phrased rather more exactly than in your question, specifically:

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?_ ____%”

As you can see this study was quite specific in mentioning false positives rather than your vague talk of 'accuracy'. Your question is underspecified as has been pointed out to you many times.

Casscells, W., Schoenberger, A. and Grayboys, T. (1978). Interpretation by physicians of clinical laboratory results. New England Journal of Medicine, 299, 999-1000.

Of course if you were referring to another study I'm sure you'll cite it.

Interesting Ian
27th April 2004, 05:04 PM
Too easy, but what's the chances of you having the disease if you get three positives in a row?

Just looked at it now and I thought . .ummm . .surely it's very obviously 10%. But when I voted I scracely thought that 13 out of the previous 14 would have voted the same as me! :eek:

I'm agreeing with everyone. This is seriously worrying :( ;)

Rolfe
27th April 2004, 05:10 PM
Originally posted by Wrath of the Swarm
Strawman. The conclusion from a positive result that a given person has the disease has the 9.02% chance of being correct.

The chance of the test being wrong for any person is 1%. The chance of the test being wrong for a particular subset of people isn't necessarily the same.

Don't you understand any statistics at all?! Wrath, it seems I understand them better than you.

You are simply continuing to assert that under the very restricted conditions you impose, this figure is correct. What I am trying to get through to you is that in the real world, these conditions do not apply.

Just tell me one actual diagnostic test which has been through proper sensitivity and specificity testing and has come out with identical results for both. And I don't want vague manufacturers' claims, I want your actual studies, with actual patients and actual reference testing for comparison. I can find you plenty that aren't identical. (Unfortunately they're in a box in my office, not on the Net, because the Veterinary Record has only dragged itself into the IT age this year.)

[Digression. If you go through the entire thousand cats for 0.1% incidence, 99% sensitivity and 99% specificity, you do indeed get 99% of results correct for that particular combination. Though I doubt if you knew that. I'd still like to see you do the working.]

No, it's easier than that JUST TELL ME THE CALCULATION YOU USE TO DERIVE THAT 99% ACCURACY/ERROR RATE FIGURE. Can you do it? Given that even you can't possibly assert that EVERY test has equal sensitivity and specificity.

Or simply admit that you meant to say "99% specificity" in the first place (because that was all we needed to know), but were a bit loose in your terminology.

Now, back to the more interesting question.

If you test everybody in the world, and pool their results, then your figure is correct. 9.02% of the positive results are correct. (And 99.99898% of the negative results are correct, to save you a job.)

Guess what. We don't care. We don't test everybody in the world, and even if we did, we wouldn't be testing them as unidentified zombies, but as individuals with their own characteristics.

If the individual in question is in a group less likely than the whole to be affected (that is clinically normal) then we reduce the probability of the positive result being correct accordingly, by plugging in the correct incidence in the group to which this patient belongs. As the prevalence to be considered has to be the prevalence in the population to which the patient belongs.

But if he is in a group more likely to be affected, we increase the probability of the positive result being correct.

The bare probabilities applicable to the population of "everybody in the world" may be of interest to statisticians, but they are only a starting point when making a clinical decision.

If you had worded the question in a totally abstract way, asking for the percentage of positive results which would be right assuming that the incidence in the population being tested was 0.1% (and we'd managed to agree that it was the specificity which was 99%), then fine.

But you can't ask a question about an individual patient, with vital information which you simply leave out, and continue to assert that this validity still maintains.

Rolfe.

Paul C. Anagnostopoulos
27th April 2004, 05:13 PM
Ian said:
I'm agreeing with everyone. This is seriously worrying
That is because this thread is causing a rampant, free-floating vortex in the space/time continuum. Consider the players. Consider the opinions. Consider the personalities. Nothing like this has ever occurred in this universe before.

~~ Paul

Wrath of the Swarm
27th April 2004, 05:33 PM
There is no vital information being left out. You've given everything you need to know about the conclusion and the factors affecting the test.

Symptoms are irrelevant. They matter only to presorting, which is a form of test. Combining two tests makes everything much more complex, and it's not the situation asked about in the research (I mentioned several posts after I complained about not being able to find it that I was looking in the wrong place for references).

What if the disease we were discussing was HIV, or a similiar infection with few (if any) obvious symptoms?

The test has objective and universal error rates that are sometimes the same for alpha and beta and sometimes different. In this example, they are the same. We do not want to consider extraneous details that would make answering the question harder - you can barely manage this one as it is.

I am pointing out that, when presented with a simple question involving the use of a diagnostic test with known accuracy in a particular generic situation, the vast majority got it wrong.

I suspect that if you had asked them, most of them would have predicted they'd get it right.

This is the problem. Rolfe, as the resident mindless-defender-of-the medical-status-quo, denies that there is a problem and attacks that which makes the existence of the problem clear. When dealing with people whose positions are grossly incorrect, her mindless rancor actually aids her. But she can't tell the difference - she'll attack anything and everything.

Rolfe
27th April 2004, 05:34 PM
Originally posted by steve74
The question, in their study, was phrased rather more exactly than in your question, specifically:

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?_ ____%”Now that is valid. State clearly that the figure being given is the "false positive rate", which is simply 100 - specificity (or to be absolutely exact, the other way around, that is specificity is defined as 100 - the false positive rate). So, specificity is 95%. Perfectly clear.

"ASSUMING YOU KNOW NOTHING ABOUT THE PERSON'S SYMPTOMS OR SIGNS."

Exactly. State that clearly, and we understand the question. There is only one possible answer. In this case, the probability of the result being correct is only 1.87%. (We can say nothing about the probability of a negative result being right because we have not been told the sensitivity.)

I can see how only 18% were anywhere near. The combination of quite poor specificity (95% isn't very good) and very low disease incidence pushes the number very low, and if you're guessing and not working it out then it's not very intuitive.

Wrath's basic premise isn't so wrong. Left to themselves without being led through the arithmetic, a lot of medics and vets do make a wrong guess. However, that is one of the things clinical pathologists are for. You ring me up and you say, do I trust this, or do I have to do further testing.

Or in the real world, where knowing nothing about the patient's clincial signs or the reason the test has been requested isn't on the agenda, the intuition jumps to a negative test in a patient who probably doesn't have the disease is probably right
a positive test in a patient who probably dosn't have the disease is probably wrong
a positive test in a patient who probably does have the disease is probably right
a negative test in a patient who probably has the disease (least firm situation as getting right over to the right-hand side of the graph isn't that common clinically, but certainly should not be taken at face value, and needs follow-up)Let the experienced clinician do it by true instinct, and they will be right most of the time without working it out, so long as you allow them all the information they usually have.

Ask the question in a deliberately statistical manner, without allowing a full calculation, and the intuitive answer may well be wrong.

The danger lies in drumming the bare statistics into people (as Jacobson did) without qualifying it. The result is often that the simplified position of "negative results are always right, positive results are always wrong" might be the take-home message.

Which is where the NegTest came in.

More common is the (also false) position that negative results are always right, and positive results always have to go to the reference method. Not so bad, but it leads to unnecessary doubt being raised about the obvious true-positives in the obvious clinical cases, and underappreciation that even your negative results can be wrong, again especially if the patient is showing clinical signs.

So without fully understanding the complete range of possibilities, as Wrath clearly doesn't, you end up with people who are making worse seat-of-the-pants assumptions if you approach the problem this way, than you'd get if you left them alone.

Just to recap. Wrath got the question wrong. That's quite obvious. He didn't state the terms as clearly as the original did, if this is the original which is being quoted. (And if he has an original which asks his exact question, first show me the reference and second, it was a bad study.)

Yes, we could see what you meant. But please don't continue to assert that this was the only possible way the question could be taken.

Rolfe.

Rolfe
27th April 2004, 05:45 PM
Originally posted by Wrath of the Swarm
What if the disease we were discussing was HIV, or a similiar infection with few (if any) obvious symptoms?More ad-homs. Wrath, this is a substitute for rational argument, not part of it.

Now you're doing the "what-if" game. You didn't tell us anything, originally. So you've been shown the wide range of possibilities which might arise, depending on the precise nature of the "what-if" you want to plug in.

It comes to the same thing, anyway.

If the doctor had no reason for testing other then ticking the box for the insurance company, your figures are in the right ball-park. Subject to subtracting the percentage of HIV-infected people with recognisable clinical AIDS from the "incidence in the population being tested" figure.

If on the other hand the doctor noticed a lesion suspicious of (at an extreme example) Kaposi's sarcoma that the patient had overlooked, that bet is completely off.

The original study correctly and honestly stated that (however improbably) the people being asked were not to know anything about the clinical signs. Wrath didn't appreciate the necessity for this part, and so we have all these futile attempts at self-justification.

Rolfe.

Wrath of the Swarm
27th April 2004, 05:50 PM
If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost. We're also no longer talking about a single test, but a combination of two tests that probably aren't independently distributed.

You're completely incapable of admitting that you could be wrong, much less actually so.

Medical professionals simply aren't very good at thinking explicitly and rationally through their job. They rely on instinct and acquired knowledge.

That is a major weakness. It means that they will be very poor at detecting flaws in established medical techniques - and they certainly are.

In some cases, it also means that they will deny reality as long as they don't have to admit they don't know what they're talking about.

I'm done. Peace out.

steve74
27th April 2004, 05:59 PM
Originally posted by Wrath of the Swarm
If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost. We're also no longer talking about a single test, but a combination of two tests that probably aren't independently distributed.

You're completely incapable of admitting that you could be wrong, much less actually so.

Medical professionals simply aren't very good at thinking explicitly and rationally through their job. They rely on instinct and acquired knowledge.

That is a major weakness. It means that they will be very poor at detecting flaws in established medical techniques - and they certainly are.

In some cases, it also means that they will deny reality as long as they don't have to admit they don't know what they're talking about.

I'm done. Peace out.

Your're going Wrath? So soon?
That's a shame because I thought you might stick around and answer my question as to why you claim to be posing the exact same question posed in a study that you, er, can't even remember the name of.

Your basic point stands that some doctors need better training in probability theory but your example was badly phrased and your unwillingness to admit that fact makes you look very foolish indeed.

Interesting Ian
27th April 2004, 06:09 PM
Originally posted by ceptimus
Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

I agree.

Rolfe
27th April 2004, 06:11 PM
Originally posted by Wrath of the Swarm
If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost.YOU were talking about raw probabilities, your problem is you never made that clear. You assumed the doctor was not allowed to have information he clearly did have, and demanded without even specifying that this information (which in the real world would be a vital part of the decision-making process) should be disallowed. And got badly tangled as a result. (Once again, we now know the original study did not make that mistake.)

Yet again you've used the term "overall accuracy" without specifying what the heck you mean. Probability that any random positive result is right, with these given parameters? (positive predictive value)
Probability that any random negative result is right, with these given parameters? (negative predictive value)
Percentage of affected patients who will test positive? (specificity)
Percentage of unaffected patients who will test negative? (sensitivity)
These are all clearly defined terms. Please quit with this "99% accuracy", "error rate", "overall accuracy" without saying which of these, or which function of these you refer to. Your persistent refusal to get your terminology correct has not only been the cause of all this from the start, it demonstrates clearly that you only have the most superficial grasp of the subject.

Your refusal to explain how you would calculate this "accuracy" figure you initially specified as 99%, reveals your lack of understanding of what you are talking about.

Your off-the-cuff formulation of the problem with two crucial errors in it with respect to the original was perhaps understandable, in someone with no practical experience of the field, but your repeated refusal to admit to these errors, or to examine the effect of failing to define these terms on the range of answers that can be admitted, is juvenile and ignorant.

The problem is that most of us saw where you were at the very beginning, and what your unstated assumptions were, and your agenda in putting the question the way you did. It was a piece of cake to solve the arithmetic, for your assumptions.

The interesting bit has been to look at how limiting these assumptions were, and how different the answers might be if you allow equally or indeed even more probable assumptions. Then when we see that an experienced clinician might be instinctively operating from the second set of assumptions, while the statistician wants to chain him down to the unrealistic first set of assumptions, we get a better overall appreciation of the implications of the exercise as a whole.

Wrath, however, will never understand this. For someone who seems to pride himself on his thinking, he's distressingly "in the box", and unable to accommodate his understanding of the subject to take account of assumptions other than those he made himself.

I'm off to bed, goodnight.

Rolfe.

Rolfe
27th April 2004, 06:13 PM
Originally posted by Interesting Ian
I agree. (with Wrath, that is.)I rest my case.

Rolfe.

Interesting Ian
27th April 2004, 06:46 PM
Originally posted by Rolfe
I rest my case.

Rolfe.

So Rolfe, what have I done to annoy you?

Dancing David
27th April 2004, 08:44 PM
Originally posted by Wrath of the Swarm
All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

It's nice to see that most of the people who bothered responding did indeed choose the correct answer, although now we'll never know how many formuites would have chosen differently. A shame.

Some of us vote before we read the thred, that is why I am the sole vote at 70%, which was just my guess.

I think that it is important in testing to know which way he test runs, Does it run to false positives or to false negatives.

Most people who use the OTC pregnancy tests are encouraed to find out that they generaly test positive in error and not negative in error.

And here in the US it depends on your test results and your insurance. I have known people who were sent home when they had cardiac enzymes indicating they were having a severe MI, because they didn't have insurance.(Happened in Decatur Illinois, two months ago to an african american, self employed and uninsured, 55 years old.)

Rolfe
28th April 2004, 01:14 AM
Originally posted by Interesting Ian
So Rolfe, what have I done to annoy you? Sorry Ian, nothing personal.

But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

Rolfe.

Brian the Snail
28th April 2004, 01:24 AM
Originally posted by Rolfe
But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

No, I think Ian's was the probabilty of finding random sequence in infinite series type-thing argument.

But I think the two questions are pretty much equivalent anyway. Or at least equally as silly.

ceptimus
28th April 2004, 02:19 AM
I suppose we are really arguing about semantics.

If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered.

As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.

If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

Rolfe
28th April 2004, 03:08 AM
Originally posted by ceptimus
I suppose we are really arguing about semantics. Yes, we are pointing out that when posing a defined problem like this, it is essential to use clearly-definied terms. These terms are defined for a very good reason, and failure to use them, or too define the terms you are using will make your question meaningless.Originally posted by ceptimus
If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered. ASSUMPTION? Ceptimus, it is a simple fact that false positive rates and false negative rates for tests like this are completely different variables. They will only be equal by pure (and unlikely) chance. In fact, as Wrath pointed out at one stage, there is a tendency for them to vary in opposite directions, so that improvements in the false-positive rate lead to deteriorations in the false-negative rate, and vice versa.

Specifying the two different values separately (or at least specifying the value of the parameter you need for the caculation in question) is a fundamental necessity when putting a question of this nature. It's not optional.Originally posted by ceptimus
As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.Yes, indeed, this was clearly the assumption that Wrath wanted to be made. And you will note that I made this assumption in my first post, explicitly.

What I am trying to point out is that making the assumption was essential before the calculation could be done. To turn round and say, "oh well, it's obvious that Wrath intended that assumption to be made" is completely missing the point.Originally posted by ceptimus
If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.Yes and no. If Wrath had been explicit in his question, we wouldn't be having this part of the discussion, we would be majoring on the more important aspect of how legitimate it is to assume that no account is taken of the reason for carrying out the test when deciding whether or not the result is reliable.

However, the way you suggest posing the question is not acceptable either. To say something like"The test is always 99% accurate, regardless of what percentage of those tested have the disease" I'm afraid still demonstrates a deep misunderstanding (or lack of understanding) of the problem. You are still failing to use a defined term, and failing to define the term you are using. To do the calculation as set, it is necessary to be told, explicitly, the specificity (or false positive rate) for the test. Obfuscations that don't distinguish between false-positives and false-negatives, which are as I said two completely separately-defined variables, are insufficient.Originally posted by ceptimus
I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.Absolutely not. This area of test interpretation is a particular interest of mine. As I said, I have published a book chapter about it, which was singled out for praise by an eminemt reviewer. No matter who it had been who had posed the question using undefined terms and with some basic assumptions unspecified, I would have reacted in exactly the same way.

Like it or not, "semantics" (or correct use of explicity-defined terms) is the be-all and end-all of this type of problem.

Rolfe.

ceptimus
28th April 2004, 04:03 AM
Originally posted by Rolfe
However, the way you suggest posing the question is not acceptable either. To say something like"The test is always 99% accurate, regardless of what percentage of those tested have the disease" I'm afraid still demonstrates a deep misunderstanding (or lack of understanding) of the problem.No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.

Why do you assume I don't understand? That is rather insulting. This is a very simple statistics problem - I wouldn't be involved at all except that I didn't like the attack on Wrath.

geni
28th April 2004, 04:23 AM
Originally posted by ceptimus
I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

It's got more to do with going through certian sections of the uk exam system in the last few years.

You admit that is the problem is to be solverble you have to make an assumption?

You think this is a reasonble assumption to make. I fail to see why. If you treat the problem as a mathmatical abstract there is no reason the make the assumption and if you treat the problem as a real world situtation then there is no reason to make the assumption. The only time that you would make the assumption was if you have to give an answer or you think of it as puzzle and give the answer that the person setting it thinks is correct.

geni
28th April 2004, 04:25 AM
Originally posted by ceptimus
No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.

Agreed. It is a slight round about way of saying it but the way you've set it up now there is only one correct answer.

Wrath of the Swarm
28th April 2004, 04:27 AM
...and I'm back. Morning, all.

I think I've said all I need to about the problem itself, so let's turn our discussion to a somewhat more interesting topic that's come up: Rolfe.

Did you notice that Rolfe posted an answer to the question openly, then brought up real-life facts which aren't relevant to the question at all, while claiming that she did this so that *I* couldn't feel smarter than everyone else?

Further, did you notice that she refuses to admit that the points she's brought up had no bearing on the question? She insists that actual doctors possess information that somehow changes the nature of the answer - which certainly isn't the case.

Consider: doesn't Rolfe spend a lot of time here debunking the rather pathetic arguments for homeopathy? Do you think that she does it out of a concern for truth, or because they make easy targets for her to demonstrate her knowledge on?

Who's really interested in being perceived as smarter than everyone else?

exarch
28th April 2004, 04:27 AM
Originally posted by ceptimus
I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong resultSo I will assume that all those wrong results are false negatives. Which means that a positive result on the test is 100% correct. Problem solved.

Oh, but apparently, 100% is not one of the possible answers :confused:

exarch
28th April 2004, 04:37 AM
Originally posted by ceptimus
This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.Well Ceptimus, if you put the question like that, this means that of every 100 positive results you get, only 1 will be wrong, and as such, your chances of being the false positive ar 1/100. The distribution of the affliction among the population isn't even relevant any more.

Can you not see the need for specifying the occurance rate of false positives and false negatives?

Prester John
28th April 2004, 04:40 AM
Sensitivity - proportion of individuals with the disease who are correctly identified by the test.

Specificity - proportion of individuals without the disease who are correctly identified by the test.

Accuracy is not a term used for desribing diagnostic tests.

geni
28th April 2004, 04:43 AM
Originally posted by exarch
Oh, but apparently, 100% is not one of the possible answers :confused:

But neither is the answer you get if you make Wrath of the Swarm's assumption

Wrath of the Swarm
28th April 2004, 04:44 AM
You're wrong in claiming that the population distribution no longer matters. The population distribution determines how many positive results will occur in a sample of a certain number of individuals. The relative proportion of true positives and false positives matters to the question.

If only one in a million people has the condition being tested for, then if a million people are tested, 99% accuracy means that roughly 10,000 people will receive false positives and only about one person a true positive. If it's one in a thousand, 99% accuracy means that about 10 people will receive a false positive and one person a true positive.

You don't see a difference in the ratios 10,000:1 and 10:1?

Rolfe
28th April 2004, 05:06 AM
Originally posted by ceptimus
No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.Your wording tells me nothing of the kind. Sensitivity and specificity are independent, and defined, variables. You have introduced a term which is not defined in this context, "accuracy". I have repeatedly asked Wrath to explain how this "accuracy" is calculated. I've explained exactly how sensitivity, specificity, and positive and negative predictive values are calculated. I need to know how you are assuming "accuracy" is calculated from the same basic data.

I realise that you don't understand from your consistent failure to grasp that this is an important question which you need to know the answer to.

(I'll reiterate that the original question from the paper Wrath seems to have been referring to didn't use the word "accuracy" at all, it referred to "a false positive rate of 5%", which implies a specificity of 95%, and this is absolutely correct presentation of the terms.)

Wrath or Ceptimus, please explain how I would calculate the term you refer to as "accuracy" in this context.

Rolfe.

JamesM
28th April 2004, 06:04 AM
Just out of interest, a very similar problem, but with the false positive and true positive rates specified, is given in the entry for Bayes' Theorem (http://en.wikipedia.org/wiki/Bayes%27_theorem) over at Wikipedia.

Interesting Ian
28th April 2004, 06:07 AM
Originally posted by Rolfe
Sorry Ian, nothing personal.

But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

Rolfe.

I was quite definitely correct. One instinctively would think my position was wrong. And it wasn't that argument you said above. That was a similar argument in another thread which I did not read.

Interesting Ian
28th April 2004, 06:09 AM
Originally posted by ceptimus
I suppose we are really arguing about semantics.

If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered.

As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.

If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

I'd just like to say I absolutely agree with this.

Wrath of the Swarm
28th April 2004, 06:29 AM
"Accuracy" is the proportion of correct test responses to total test responses.

(Just what the English definition of the word would imply.)

For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.

As we have explained to you many times.

geni
28th April 2004, 06:37 AM
Originally posted by Wrath of the Swarm
For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Not stated in the original question. So you admit the orginal was imposible to answer with the data given?

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.

You did not state this in you original question therefor it was imposible to answer.

Wrath of the Swarm
28th April 2004, 06:39 AM
But it was possible.

Let's say I flip a coin (heads or tails) and then someone guesses which side came up.

I can specify an accuracy of the guessing without stating alpha and beta rates because they're the same.

This test is no more likely to mess up when dealing with a positive than a negative. Instead of stating accuracies for positives and negatives, all I need to do is state the accuracy.

geni, are you going to respond to my post where I pointed out that your claim about the population proportion not mattering was wrong?

Wrath of the Swarm
28th April 2004, 06:46 AM
Are we really expected to believe that Rolfe does not understand what it means for a test to be accurate?

There are plenty of tests whose error rates aren't dependent on the nature of the answer. This hypothetical test is one of them.

Now, since I consider Rolfe to be utterly unable to actually think, and only capable of applying the correct principles by rote, I have no problem with accepting that she genuinely does not understand the concept of test accuracy.

But those of you who claim Rolfe is both intelligent and honest - how do you account for this?

geni
28th April 2004, 06:52 AM
Originally posted by Wrath of the Swarm
But it was possible.

Lots of things were posible

Let's say I flip a coin (heads or tails) and then someone guesses which side came up.

I can specify an accuracy of the guessing without stating alpha and beta rates because they're the same.

In fact you can't because you failed to specify that it was a fair coin (you relly should try working for edexcel) However since most coins are fair I would probably be ok making that assumption. For most clinical tests I have no reason to belive the two values are the same.

This test is no more likely to mess up when dealing with a positive than a negative. Instead of stating accuracies for positives and negatives, all I need to do is state the accuracy.

This is not information I had avaible in the oringinal question

geni, are you going to respond to my post where I pointed out that your claim about the population proportion not mattering was wrong? [/B]

Since I'm not entiry sure what you are talking about this is going to be difficult. However I think think you mean my reply to ceptimus. If the accury remains the same for any population proportion then I can slove the problem through simultanious equations ( the equations look something like this:

Acccurary for proptionA= p<sub>1</sub>+p<sub>2</sub>
Acccurary for proptionb= p<sub>1</sub>+p<sub>2</sub>

accurary is of coure fixed and we can have any value for and and b making the problem solverble.)

Wrath of the Swarm
28th April 2004, 06:58 AM
You're missing the point. Whether the coin was fair or not has nothing to do with whether the guessing procedure has different accuracies for heads or tails.

You're also not responding to my earlier question. Why did you claim the population proportion was irrelevant when it was still vitally important?

geni
28th April 2004, 07:07 AM
Originally posted by Wrath of the Swarm
You're missing the point. Whether the coin was fair or not has nothing to do with whether the guessing procedure has different accuracies for heads or tails.

By thid logic I can win the million no problem I have a coin that lands on heads 60% of the time and claim I can predict it's fall with 60% accuricy and then guess heads everytime. To put it another way with a fair coin random guessing would give you 50% with a non fair coin random guessing would give you less than 50%

You're also not responding to my earlier question. Why did you claim the population proportion was irrelevant when it was still vitally important?

Where did I say this?

Wrath of the Swarm
28th April 2004, 07:10 AM
Oh, I'm sorry. That was exarch, wasn't it?

I take it back. I have trouble telling Rolfe's groupies apart from one another.

 Actually, I take back the taking back. The original comment was exarch's, but you've proceeded to make similar claims - you're implying you can get a 60% accuracy by guessing the same way each time in a particular case, but you're ignoring the effect of the actual population on your strategy's success.

Which is in essence the same claim exarch is making.

Wrath of the Swarm
28th April 2004, 07:13 AM
But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.

This test does.

pgwenthold
28th April 2004, 07:15 AM
Originally posted by Wrath of the Swarm
Are we really expected to believe that Rolfe does not understand what it means for a test to be accurate?

Considering that Rolfe knows an awful lot about it, then it is not surprising that she would not know what you mean by accuracy.

Your naive approach is too simplistic for someone who really knows what they are doing.

This is what makes it really funny, because the point of your exercise was to show that you know what you are doing and those silly doctors don't. Yet this doctor has schooled you up and down fourteen times to nowhere. The irony is so great.

Wrath of the Swarm
28th April 2004, 07:17 AM
But she hasn't. Her argument is inherently flawed.

If you mindlessly accept that everything she says is correct, well then - clearly every time she gets into an argument with anyone, she'll wipe the floor with them!

But she simply isn't correct.

steve74
28th April 2004, 07:20 AM
Wrath, why can't you just admit the question was worded too vaguely?

You claim that you worded the question exactly as it was given to the test subjects:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

But you'd previously admitted you couldn't even remember the source of the research:

The basic problem is a classic one. I'm trying to find the sources in which I read about the implications for screening tests several years ago.

If I recall correctly, doctors get the right answer more frequently than the general population, but they still tended to reach grossly wrong conclusions about whether a particular patient had a disease. I believe they overestimated the power of the tests significantly.

If I find some good sources on the subject, I'll get back to you.

It has been ably demonstrated to you by many posters why you are wrong but you refuse to admit it. I can only conclude one of 2 things:
1. You lack the intellectual ability to see the correctness of others' arguments or,
2. You're a troll.

Which is it?

Wrath of the Swarm
28th April 2004, 07:23 AM
Neither.

I finally found the sources that duplicated the question (I even pointed them out, remember?).

The original source was a professor of mine, many years ago.

geni
28th April 2004, 07:24 AM
Originally posted by Wrath of the Swarm
But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.

This test does.

Assuming a reasonble szie population yes I would beacuse I've force dthe population to be 60% on way and 40% the other. If you change the population you have to do it by changing how much the coin is rigged.

Wrath of the Swarm
28th April 2004, 07:32 AM
No, I don't.

You see, the population of people given in the example is chosen from the wider population. The subpopulation might not have the same distribution as the population as a whole does. (This was actually one of Rolfe's points, remember?)

So if I toss this unfair coin ten times, it won't necessarily come up heads six times and tails four (or vice versa). It might come up heads all ten times.

If you do nothing but state one possibility over and over (say, guessing heads every time), your accuracy will depend entirely on what population is given to you. If the population is all heads, you'll be 100% accurate. If the population is all tails, you'll be 100% inaccurate. You'll be X% accurate if the population fed to you is X% heads.

The hypothetical test given originally has a 99% accuracy, no matter what subjects are fed to it. That's what accuracy means - it's no good using a concept that depends on the population distribution if you don't know what that is, and we can't presume beforehand that a doctor will face any particular population.

exarch
28th April 2004, 07:37 AM
Originally posted by Wrath of the Swarm
I am pointing out that, when presented with a simple question involving the use of a diagnostic test with known accuracy in a particular generic situation, the vast majority got it wrong.

I suspect that if you had asked them, most of them would have predicted they'd get it right.What was it you were saying earlier about there being no hidden agenda in your question? Seems like Woo of the Swarm has started slamming doctors again. Just wait long enough, and the true purpose emerges, as always :rolleyes:

This is the problem. Rolfe, as the resident mindless-defender-of-the medical-status-quo, denies that there is a problem and attacks that which makes the existence of the problem clear. When dealing with people whose positions are grossly incorrect, her mindless rancor actually aids her. But she can't tell the difference - she'll attack anything and everything.No Swarm, the problem is you thinking you are so much smarter than you really are. You think you're infallible, and it makes you cocky. And in this instance you have to start back peddaling at a speed you've never had to before, because for once you had the misfortune to run into someone who actually is an expert in the subject you're trying to woo us over with.

Wrath of the Swarm
28th April 2004, 07:42 AM
But it's not limited to doctors. Most people given this question get it wrong, and most of them overestimate how many people get it right.

The only hidden agenda here is the one you're projecting.

And this "expert" is making an invalid objection. You can't just assert that the objection is valid because she's an "expert" - that's one of those nasty arguments from authority, remember?

Rolfe has suggested that the question cannot be answered without more data. This is not the case. She's so used to thinking about tests whose accuracy is dependent on the nature of the result that when she didn't see that information, she reflexively asserted it was necessary.

pgwenthold
28th April 2004, 07:42 AM
quote:
--------------------------------------------------------------------------------

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

--------------------------------------------------------------------------------

I think this statement basically illustrates the problem at hand here.

Wrath initially assumed that the fact that doctors apparently performed poorly on this question indicates their problems with cog thinking.

However, as Rolfe has demonstrated (by actions more than words), there are certain assumptions built into the question. Wrath, in his naivity, thinks the assumptions are obvious. However, to a more knowledgable reader, the proper assumptions are not at all obvious. Thus, an alternate explanation for the poor performance by the doctors is that they did not share the assumptions of the question writer.

So we have two possible explanations: one is that doctors have very poor cognative ability. The second is that the question requires too many assumptions that will not be shared between the question writer and an expert answering the question.

I see no reason to think that either or even both explanations account for what was observed. There are probably doctors that lack the cognative skills, and there were probably doctors who knew so much about it that they couldn't answer the question.

Wrath of the Swarm
28th April 2004, 07:43 AM
Exarch: you've claimed that the population distribution was irrelevant to the question.

Can you explain for us why you made the statement and whether it is correct?

Wrath of the Swarm
28th April 2004, 07:46 AM
Originally posted by pgwenthold
I think this statement basically illustrates the problem at hand here.

Wrath initially assumed that the fact that doctors apparently performed poorly on this question indicates their problems with cog thinking. No, it demonstrates that people in general have problems with thinking. (By the way, what's "cognitive thinking"?

However, as Rolfe has demonstrated (by actions more than words), there are certain assumptions built into the question. No, there are certain assumptions built into the answerer - namely, Rolfe.

Alpha and beta errors are more complicated concepts that simple accuracy. The question gave an accuracy for the test that did not refer in any way to these more complicated concepts. It simply stated the chance that the test would be wrong about any result it gave.

Rolfe is conditioned to expect the more complex concepts, and therefore she is rendered unable to think about the simple ones?

Riiight.

steve74
28th April 2004, 07:58 AM
Originally posted by Wrath of the Swarm
Neither.

I finally found the sources that duplicated the question (I even pointed them out, remember?).

The original source was a professor of mine, many years ago.

Stop bullsh1tt1ng, Wrath. You have not referenced one single study in this thread. You have referenced a few review articles, which you had to Google for. So, once again I ask you, which study was this wording used in? You must know the answer because you claimed:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Rolfe
28th April 2004, 07:59 AM
Originally posted by Wrath of the Swarm
"Accuracy" is the proportion of correct test responses to total test responses.

(Just what the English definition of the word would imply.)

For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.Funnily enough, I do know what accuracy means. It is a defined term in clinical biochemistry, where you are dealing with concentrations of analytes, rather than just positive or negative. It is defined as how well the results from the method under test match up to the "true" concentrations in the samples, defined as the concentrations as measured by the designated reference method. This is actually a much more difficult subject than the sensitivity/specificity one, because you can get different values for r depending on the spread of concentrations you have in your test group of samples, and because it's not just the correlation coeficient that matters, it's how well the line of best fit matches up with the line of coincidence (so you have to look a the slope and the y-intercept as well). And at some point you do have to give in and concede that your "reference" method has a degree of inaccuracy within it too.

This is one of the two cardinal characteristics of a biochemistry assay. The other is precision (inter and intra-assay), defined as the consistency of results when performing repeat assays on the same sample, and measured by the coefficient of variation.

Accuracy (in this sense) is a minefield and a nightmare to pin down statistically, let's not go there.

But this is the context in which it is a defined term in the field of laboratory analysis. In biochemistry assays (measuring concentration of analyte) the cardinal characteristics of the assays are accuracy and precision, as described.

In serology testing (positive/negative results) the cardinal characteristics of an assay are sensitivity and specificity. Accuracy is not a defined term and has no meaning in this context.

You see, Wrath, while ordinary conversation may be able to use these words as it pleases, clinical laboratory work requires very careful use of the terms as they are defined, otherwise terrible misunderstandings may ensue. As you are finding out. In this context "the ordinary English definition of the word" is irrelevant.

You are now revealing that the sensitivity and the specificity of the test are the same. However, as Geni has said, you didn't include that information in your original problem. You used an undefined word "accuracy" which you still have not told us how you are calculating.

To get the "percentage of correct test responses to total test responses" you need to do the entire thousand cats calculation, and sensitivity, specificity and prevalence all have to be stated before you can even start. So I submit that this isn't a clear or complete answer.

If you care to refer to the start of the thread, you may notice that my assumption was that since all we neeed to know for the purpose of the calculation set was the specificity, I was going to assume that by "accuracy" you actually meant specificity. I really didn't care whether the sensitivity was the same or not, because I didn't need to know that to do the sums. Rather sloppy use of terminology, but something easily clarified, or so I thought.

But no, you have to make this more and more complicated, and in so doing it becomes more and more obvious that your understanding of the subject is really rather superficial.

Now either admit that since all we needed to know for the sum was the specificity, your "accuracy" figure should be taken to mean specificity, or please explain in detail how I would calculate this novel "accuracy" term you've introduced, from the beginning.

As an example (shining, I have to say - some of this is so sparely expressed I thought for half a moment it was wrong, but it ain't.):POSITIVES/NEGATIVES, SENSITIVITY/SPECIFICITY, PREDICTIVE VALUES, PREVALENCE.

TRUE POSITIVE: a person who tests positive and has the disease.

FALSE POSITIVE: a person who tests positive but does not have the disease.

TRUE NEGATIVE: a person who tests negative and does not the disease.

FALSE NEGATIVE: a person who tests negative but does have the disease.

SENSITIVITY: the percentage of people with the disease for whom the test is positive.
Sensitivity = TP / (TP + FN)

SPECIFICITY: the percentage of people without the disease for whom the test is negative.
Specificity = TN / (TN + FP)

POSITIVE PREDICTIVE VALUE (PPV): the percentage of people with the disease who test positive for the disease
PPV = TP / (TP + FP)

NEGATIVE PREDICTIVE VALUE (NPV): the percentage of people without the disease who test negative for the disease
NPV = TN / (TN + FN)

PREVALENCE: the percentage of the population who have the disease

courtesy,
BillyJoeThese are all the standard definitions of the terms used by people who understand the subject. Just define your use of "accuracy" to the same standard, please.

Rolfe.

Wrath of the Swarm
28th April 2004, 08:00 AM

The examples linked used the same question as I did, and as I remembered my professor using.

geni
28th April 2004, 08:01 AM
Originally posted by Wrath of the Swarm
No, I don't.

You see, the population of people given in the example is chosen from the wider population. The subpopulation might not have the same distribution as the population as a whole does. (This was actually one of Rolfe's points, remember?)

So if I toss this unfair coin ten times, it won't necessarily come up heads six times and tails four (or vice versa). It might come up heads all ten times.

If you do nothing but state one possibility over and over (say, guessing heads every time), your accuracy will depend entirely on what population is given to you. If the population is all heads, you'll be 100% accurate. If the population is all tails, you'll be 100% inaccurate. You'll be X% accurate if the population fed to you is X% heads.

Only relivant if you are dealing with real clinical situations Since I dont and never will do I don't care

The hypothetical test given originally has a 99% accuracy, no matter what subjects are fed to it. That's what accuracy means - it's no good using a concept that depends on the population distribution if you don't know what that is, and we can't presume beforehand that a doctor will face any particular population.

But you didn't state this in the orignal question did you? You once again are giving me more data to play with which makes the question answerble. The orginal question only delt with one population. It didn't sate that the answer was the same for all populations.

Wrath of the Swarm
28th April 2004, 08:04 AM
Calculating? Rolfe, you don't understand. The accuracy of the test is the value we calculate other values from in the presented problem.

In any situation, the test is 99% likely to give the correct answer. That's the whole point of the question - people get confused between the idea that the test is right 99% of the time and the idea that the chance of it being accurate in a specific, particular case is only 9%.

If you're told only that the test is 99% accurate, that holds no matter what other conditions apply. From this, it follows that the alpha and beta rates are identical.

There is no information missing. You're just not capable of deriving information you weren't presented with.

Hellbound
28th April 2004, 08:25 AM
Originally posted by Wrath of the Swarm
There is no information missing. You're just not capable of deriving information you weren't presented with.

Um, you did notice that she got the right answer the first time, yes?

So, what are you trying to say here? That the answer you claimed was right is not? Because, obviously, she had to derive the information to get that answer.

Or did little angels whisper in her ear? What are you trying to assert?

She not only derived the information, she then went on to explain how this information should not have to be derived, that the terms used are not applicable in the real-world, exactly how those terms should be used, why the 9% figure does not apply as an overall (because real-world tests are not typically given willy-nilly), and several other bits of info that you did not mention and still seem unable to grasp (or unable to admit that she is right).

I'm sorry, Wrath. Your ego is so big it's blocking your view. This thread would have been over three pages ago, without bickering and with some useful statistical knowledge, if you weren't too much of an a\$\$ to admit someone might know more than you.

Rolfe
28th April 2004, 08:27 AM
Originally posted by Wrath of the Swarm
Calculating? Rolfe, you don't understand. The accuracy of the test is the value we calculate other values from in the presented problem.

In any situation, the test is 99% likely to give the correct answer. That's the whole point of the question - people get confused between the idea that the test is right 99% of the time and the idea that the chance of it being accurate in a specific, particular case is only 9%.

If you're told only that the test is 99% accurate, that holds no matter what other conditions apply. From this, it follows that the alpha and beta rates are identical.

There is no information missing. You're just not capable of deriving information you weren't presented with. Wrath, you're digging deeper and deeper into "make it up as you go along".

Nobody will ever tell you in practice that any serology test is "99% accurate", because accuracy isn't a defined term in the vocabulary of this problem. BillyJoe has kindly posted a list of (almost) all the defined terms and how they are defined, which I reproduced above. These are the words we professionals use when talking about these things.

You keep talking about "people getting confused", but from the evidence presented so far on this thread, the most confused person is yourself. I think because you have learned by rote a particular example case which has a non-intuitive answer, and which can thus, by careful phrasing of a trick question, be used to ambush medical types. However, you have not really got anywhere close to coming to grips with the permutations and variabilities possible unless the parameters of the question are extremely carefully nailed down in advance.

Since you made the mistake of not nailing down the parameters of the question when you originally posed it, you are coming up against aspects of the problem you never even thought about.

Now, can I ask again. What do you mean by "alpha and beta" rates? I'm assuming this is another way of saying sensitivity and specificity, but I've never encountered this before, so would you mind confirming that this is the case, and revealing which is which?

And about that "accuracy" figure again. When I said, how do I calculate it, I meant that if I were characterising an assay, how would I derive the figure from the raw data? The same way BillyJoe detailed how you would derive figures like the specificity or the positive predictive value from a set of raw data.

Raw data for characterisation of the test is quite easy.

Group of patients, x% of whom have the disease (prevalence, as I keep saying a mobile and artificial concept), and you already know by some other means (reference method) which is which. Test them all. Some of the affected patients will test positive, TP. Some of them will test negative, FN. Some of the unaffected patients will test negative, TN. Some of them will test positive, FP. This is your starting point. I can derive all the defined characteristics of any assay lfrom these numbers - have to, it's all you're going to get. BillyJoe showed you how this is done. I want the same level of understanding from you about how you derive your "accuracy".

Rolfe.

geni
28th April 2004, 08:28 AM
Originally posted by Wrath of the Swarm
If you're told only that the test is 99% accurate, that holds no matter what other conditions apply. From this, it follows that the alpha and beta rates are identical.

Once again this is more information that is avaible in the orgianl question.

exarch
28th April 2004, 08:29 AM
Originally posted by ceptimus
If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.If the test is always 99% accurate, wouldn't the accuracy of your positive result be 99%?

That was my initial thought, because the test didn't specify false positives or false negatives explicitely. I ASSUMED an equal distribution between false positives and negatives though, which eventually led me to the 10% number , but obviously, that's a big assumption to make, seeing how wildly the results can vary depending on how you interpret that "accuracy" value of 99%.

steve74
28th April 2004, 08:31 AM
Originally posted by Wrath of the Swarm
Neither.

I finally found the sources that duplicated the question (I even pointed them out, remember?).

The original source was a professor of mine, many years ago.

Your attempts at evasion would be funny if they weren't so wearisome. You posted three links:

Let's say that I went for an annual medical checkup, and the doctor wanted to know if I had a particular disease that affects one out of every thousand people. To check, he performed a blood test that is known to be about 99% accurate. The test results came back positive. The doctor concluded that I have the disease.
(My emphasis)

Nowhere in any of those three links is the term accuracy or accurate used in the way you use it. By contrast all three do use the terms false positive and false negative.

Again, I remind you that you claimed:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Yet you have still not cited a source for this question. If the question was not copied directly from a study, then fine, just admit this and the question can be discussed on its own (de)merits. But don't lie and say it was a question posed to doctors in a study, when you are unable to cite this study.

Edited fur speelin mistuks

exarch
28th April 2004, 08:41 AM
Originally posted by Wrath of the Swarm
Let's say I flip a coin (heads or tails) and then someone guesses which side came up.

I can specify an accuracy of the guessing without stating alpha and beta rates because they're the same.How do you get a false positive when flipping a coin? How exactly does that work? I would really like to know?

Wrath of the Swarm
28th April 2004, 08:50 AM
Originally posted by Huntsman
Um, you did notice that she got the right answer the first time, yes? Yes, but she claimed she could only get the right answer by assuming information that wasn't presented to her. That is a lie.

The question doesn't ambush anyone. Yes, the information given to doctors is generally more complex than that. That makes the question easier to answer, not harder.

Unless of course you've learned to get the answer by rote, in which case changing the presentation screws everything up.

exarch
28th April 2004, 08:51 AM
Originally posted by Wrath of the Swarm
But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?Hang on, are you saying that something which has a 60% chance of occuring does not have a 60% chance of occuring under all circumstances? And where exactly did you study statistics? Did your degree come in a box of cheerios too?

Wrath of the Swarm
28th April 2004, 08:56 AM
I will admit that I can't find a source that used precisely the same question. That is the question I remember being given (quite vividly), but I can't demonstrate that it was ever used in a study.

I retract the claim and admit I was wrong to make it.

Posted by exarch:Hang on, are you saying that something which has a 60% chance of occuring does not have a 60% chance of occuring under all circumstances? And where exactly did you study statistics? Did your degree come in a box of cheerios too? You ignorant prat.

If the coin has a 60% chance of coming up heads, that does not mean that in a sample of tosses, heads will occur 60% of the time.

Anyone with even the slightest knowledge of statistics would know that. Hell, the majority of high school students would know that.

How old are you, again? Are you sure you're permitted on the net without parental supervision?

exarch
28th April 2004, 09:09 AM
Originally posted by Wrath of the Swarm
The hypothetical test given originally has a 99% accuracy, no matter what subjects are fed to it. That's what accuracy means - it's no good using a concept that depends on the population distribution if you don't know what that is, and we can't presume beforehand that a doctor will face any particular population.But this is the exact opposite of what you were saying earlier. If the population fed to the test are all healthy people, an accuracy of 99% means 1% false positives. How can this same test also be 99% accurate when a 50/50 sample is fed? By chance, in this case, you'd get 0.5% false positives and 0.5% false negatives. But with other "accuracies", the numbers don't work out that nicely.

When determining the odds of a particular positive being a false positive, the overall accuracy is meaningless unless you specify the odds of said false positive to occur.

In other words, a 1/1'000'000 occurance of an illness with a test of 95% accuracy cannot be 95% accurate when testing only sick people as when testing only healthy people, or a random population sample.

Accuracy as you describe it simply means that 1 out of every 100 tests is defective, and gives the opposite result to the truth. This type of logic works just dandy in case of puzzles, in the puzzle section, but it simply doesn't apply to reality, which, no matter how hard you want to deny it, is the not so very well hidden alterior motive of this poll: pointing out that, evil doctors are stupid for trusting test results, and you are not :rolleyes:

exarch
28th April 2004, 09:12 AM
Originally posted by Wrath of the Swarm
And this "expert" is making an invalid objection. You can't just assert that the objection is valid because she's an "expert" - that's one of those nasty arguments from authority, remember?Actually, it's not an "argument from authority" fallacy when the authority referenced really *IS* an authority on evaluating clinical test results.

I think your denial of her authority on this subject could in fact be seen as an ad hom :)

Wrath of the Swarm
28th April 2004, 09:13 AM
That's not the opposite of what was said. It's what was said.

The test's accuracy does not depend on the population incidence. If all sick people are tested, 1% of the responses will be wrong. If all healthy people are tested, 1% of the responses will be wrong. It doesn't matter.

We need to specify alpha and beta rates only if the chance of 'positive' being wrong is different from the chance of 'negative' being wrong regardless of what the population is.

exarch
28th April 2004, 09:14 AM
Originally posted by Wrath of the Swarm
Exarch: you've claimed that the population distribution was irrelevant to the question.Strawman.

Edited to elaborate before Woo of the Swarm jumps on this:
I said the distribution of the disease among the population didn't matter right after I stated that the "accuracy" of the test might simply apply to false negatives, in which case all positives are always positives, and the doctor is always right in assuming the test result to be correct.

I also stated that if the test itself is always 99% accurate, then your positive result, no matter how big or small the population, is always correct 99/100 times. Again, distribution among the population is not relevant in this case. But this situation doesn't occur in reality, only in theoretical puzzles. Since this is a theoretical puzzle though, you should have clarified.

Wrath of the Swarm
28th April 2004, 09:18 AM
But she's generalizing from analyzing results given in a particular manner to a completely different type of question, and she's failing.

She may be an expert in an extremely narrow band of competence, but she can't do basic statistics when taken outside of that narrow band.

If the accuracy of the test can be established without referencing alpha and beta rates, then it follows that the alpha beta rates are equal. This is a very simple and basic point that Rolfe does not seem to understand.

Wrath of the Swarm
28th April 2004, 09:20 AM
Originally posted by exarch
Well Ceptimus, if you put the question like that, this means that of every 100 positive results you get, only 1 will be wrong, and as such, your chances of being the false positive ar 1/100. The distribution of the affliction among the population isn't even relevant any more.

You're a liar, exarch. 'Strawman' indeed. You didn't even do the arithmetic correctly - if ceptimus phrased the question as he did, your conclusion is wrong.

yersinia29
28th April 2004, 09:21 AM
Wrath, you are a total fool.

The incidence of disease X in the general population is W.

The incidence of disease X in a population with symptoms A, B, and C is W + Y.

The incidence of disease in those 2 groups is NOT the same.

Deetee
28th April 2004, 09:23 AM
Come on chaps - isn't it time to quit?

Wrath can retire (with some egg-on-face, having been so keen to show that all doctors are fools that he forgot sometimes others, himself included, can be shown to be fools too), and Rolfe can calm down and accept that although she is the queen of predictive values, her considerable talents would be better directed at woo-fighting than semantic arguments with Wrath.

Wrath of the Swarm
28th April 2004, 09:28 AM
Originally posted by yersinia29
Wrath, you are a total fool.

The incidence of disease X in the general population is W.

The incidence of disease X in a population with symptoms A, B, and C is W + Y.

The incidence of disease in those 2 groups is NOT the same. Who mentioned symptoms A, B, and C? Checking for those symptoms is another test - and we've already established that combining two tests gives a much more accurate result than either seperately.

Wrath of the Swarm
28th April 2004, 09:32 AM
Originally posted by Deetee
Wrath can retire (with some egg-on-face, having been so keen to show that all doctors are fools that he forgot sometimes others, himself included, can be shown to be fools too) There's just three problems with this.

1) The point is to show that people are really bad at statistics, including doctors.

2) According to the presented arguments, Rolfe is the one who's wrong.

3) According to the presented arguments, Rolfe is the one who's wrong.

Now, I realize that this is only two problems, but the second is so important I thought it should be mentioned twice. I'll say it again:

If it's possible to establish an accuracy for the test as a whole, then the alpha and beta rates (chances of false positive and false negative, respectively) MUST BE EQUAL.

There's no other possibility. No one needs to 'assume' they're the same - it is mathematically impossible for it to be otherwise. If this is not the case, then the accuracy of the test becomes a variable dependent on the population given to it, and it is then impossible to consider how good the test is independent from a particular sample. That is fairly useless, and so it's not used when talking about the test itself.

steve74
28th April 2004, 09:33 AM
Wrath, I'm glad you've finally admitted you were wrong about the origin of the question. It was big of you to do so. Although, you have a lot more to apologise for in this thread with regard to your juvenile and spiteful use of ad homs.

With regard to whether the information given in the question was sufficient to answer the question, I have only this to say. Yes, it was fairly easy to make an assumption about what you were getting at and give the answer, as you defined it. But, it is also true that an assumption had to be made and that while this assumption was the most likely one, it certainly wasn't the only way to interpret the question.

Your question was underspecified. This has been pointed out to you many times by many posters, all of them far more able than I to make this clear. My suggestion to you is to accept that you are incorrect on this matter and bow out with what tattered remains of dignity you still have.

Wrath of the Swarm
28th April 2004, 09:40 AM
No, you're still wrong about that, I'm afraid. There was only one correct and contextually viable way to interpret the statement.

The only way it's possible to establish the accuracy of a test independent of a particular sample is for the alpha and beta rates to be the same. Otherwise, the alpha and beta rates need to be known specifically, because the overall accuracy will depend on who is being tested.

We didn't know whether (in the example) I had the disease or not. I gave an accuracy for the test - from this, we know that alpha must equal beta. There is no other possibility.

Rolfe expected the format she was accustomed to, and interpreted the difference as an error. This is incorrect.

If you can show me a way in which an accuracy can be established independent of a sample and alpha can be made not equal to beta, I will admit that I'm wrong. But you simply can't do that because it's not logically possible.

Regarding insults:

Considering that Rolfe and Friends have slugged an even greater number of insults at me, and that Rolfe is an attention-whoring quack... well.

yersinia29
28th April 2004, 09:49 AM
Originally posted by Wrath of the Swarm
Who mentioned symptoms A, B, and C? Checking for those symptoms is another test - and we've already established that combining two tests gives a much more accurate result than either seperately.

You idiot, doctors dont test people randomly in the population. They run tests on people present with specific symptoms.

ceptimus
28th April 2004, 09:51 AM
Well Wrath, as you know, I agree with you. But I don't see that you need to keep insulting Rolfe and her followers, even when they insult you.

Rolfe knew what you meant, and is only quibbling because she wants to prove you wrong.

Rolfe's supporters are wrong, but so far they don't fully understand why.

I think you would convince more lurkers if you left out the insults.

Wrath of the Swarm
28th April 2004, 09:52 AM
Really? So what symptoms induce them to offer HIV testing? Or mammograms? Or PSAs?

Anyway, this is irrelevant. The population the test will be used in is irrelevant to how accurate it is if alpha and beta are equal. Clearly it affects the conclusions drawn from the test, but not the test itself.

Wrath of the Swarm
28th April 2004, 09:54 AM
You might be right, ceptimus. On the other hand, if these people aren't convinced by logical arguments but are swayed by the presence of insults... well, they're like Rolfe's followers.

Do I really want that kind of person telling people I'm right? Having gullible people proclaim my correctness doesn't say much for me.

The actual skeptics on these forums will think about the issues raised and reach a conclusion whether I insult Rolfe or not. And since I feel like insulting Rolfe... well, what's the harm, really?

exarch
28th April 2004, 09:55 AM
Originally posted by Wrath of the Swarm
If the coin has a 60% chance of coming up heads, that does not mean that in a sample of tosses, heads will occur 60% of the time.Ignoring the ad homs ...

Sure, but if you guessed heads every time, you'd still have a better chance of being right more often. It could be tails every time, but the odds of that happening are extremely small. An all tails sample would also not be representative of the average sample, which would, on average, contain 60% heads.Originally posted by Wrath of the Swarm
But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.Sure enough, but that's why you have to specify the rating at which false positives occur. Because the amount of false positives will depend on the population, while the accuracy as you describe it, according to your own words:
Your method does not have an error rating independent of the population distribution.

This test does.So if accuracy is not dependent on the population distribution, then neither is the ratio between false positives and false negatives, in order to receive the same final independent "accuracy" rating. so how is it calculated?

geni
28th April 2004, 09:59 AM
Originally posted by ceptimus

Rolfe's supporters are wrong, but so far they don't fully understand why.

Really? Show that mathmaticly that what WOTS calls alpha and beat have to be equal. You amited before that you had to make an assumption to get your answer.

exarch
28th April 2004, 10:04 AM
Originally posted by Wrath of the Swarm
If it's possible to establish an accuracy for the test as a whole, then the alpha and beta rates (chances of false positive and false negative, respectively) MUST BE EQUAL.There we are, you finally said it.

WRONG !! :D

If the overall accuracy of the test is 95% that says nothing about the amount of false positives or false negatives. Indeed, a test can be 95% accurate, and of every 100 tests, come up with 5 false negatives. here is nothing that says they MUST BE EQUAL.

I think this is the tricky bit you still don't get.

it is mathematically impossible for it to be otherwise. If this is not the case, then the accuracy of the test becomes a variable dependent on the population given to it, and it is then impossible to consider how good the test is independent from a particular sample.Now do you understand why it is important to give rates for false positives and false negatives? Do you understand why Rolfe is puzzled by your use of the word "accuracy"? Why your initial question was poorly worded?

Wrath of the Swarm
28th April 2004, 10:08 AM
Let alpha and beta be non-equal. The chance of getting a false positive is not the same as getting a false negative.

Therefore, the accuracy of the test is a function of the sample population fed to it. For example, if the entire population is positive, all error will consist of false negatives. If the entire population is negative, all error will consist of false positives.

Since we've established that the chance of false positives isn't the same as the chance of false negatives, the test's performance varies depending on what kinds of people are fed to it.

That means that a general statement about its accuracy cannot be made.

This does NOT mean that there are no tests for which general statements about their accuracy cannot be made.

IF a general statement about the test's accuracy is made, THEN the alpha must be the same as the beta for the statement to be valid. These tests are perfectly possible - common, even. Thus there is no reason for the statement to be presumed invalid - and since we're talking about a hypothetical test, anything logically possible I establish about it is correct and valid - ergo, the alpha is the beta.

No "assumptions" are necessary.

Wrath of the Swarm
28th April 2004, 10:11 AM
Originally posted by exarch
If the overall accuracy of the test is 95% that says nothing about the amount of false positives or false negatives. Wrong.

First, we're not talking about the 'amount', we're talking about the proportion. Completely different concept.

Secondly, the overall accuracy of the test cannot be established - or even spoken about - without reference to a specific test population if the proportion of false positives is different from the proportion of false negatives.

ceptimus
28th April 2004, 10:12 AM
Part 1

Geni, if I tell you a test is 99% accurate, regardless of the incidence of the disease in the population, then it follows that alpha = beta.

I assume you accept this?

Part 2

The wording of Wrath's question did tell you the true incidence of the disease in the population, so there is room for you to quibble if you wish.

However, if you do quibble, then the question becomes unanswerable.

I assert that almost any question, no matter how carefully stated, can be spoilt by pointless quibbling. You almost have to use only the language of mathematics to avoid this.

Now if your quibble allows you to come up with a different valid answer, I don't mind that, but if it merely allows you to declare the question unanswerable I think it's pointless.

Wrath of the Swarm
28th April 2004, 10:19 AM
Ah, but I told them the incidence in the general population.

Each patient, taken individually, is their own population. Either the patient has the disease, or they don't.

If alpha != beta, then without knowing whether the patient is positive or negative, the overall accuracy is not equal to the accuracy in a particular case. (And indeed, if we know whether the patient is sick or not, we can be 100% sure of the conclusion we draw no matter what the test says!)

In order to draw a meaningful conclusion, we would have to know what the test result was and the alpha or beta rate.

But the alpha rate IS the beta rate in the particular example since the test itself has a known accuracy.

exarch
28th April 2004, 10:25 AM
Originally posted by ceptimus
Part 1

Geni, if I tell you a test is 99% accurate, regardless of the incidence of the disease in the population, then it follows that alpha = beta.Or alpha = 0 or beta = 0

Part 2

The wording of Wrath's question did tell you the true incidence of the disease in the population, so there is room for you to quibble if you wish.

However, if you do quibble, then the question becomes unanswerable.

I assert that almost any question, no matter how carefully stated, can be spoilt by pointless quibbling. You almost have to use only the language of mathematics to avoid this.Obviously this wasn't necessary, as someone posted a link to a similar question that wasn't ambiguous.
I solved the question by making an assumption, based on knowing exactly what WotS was getting at, but as someone who posts regularly in the puzzles forum, you of all people should know that the wording of a puzzle is particularly important if you want to find the right answer. (Remeber the prince with the poison in the boxes?)

There probably wouldn't have been a problem if WotS would have just conceded that he had worded it wrong. He just can't admit he's wrong. It took us 5 pages to finally get him to do that.
Right now, I'm just messing with him :p

ceptimus
28th April 2004, 10:29 AM
No exarch. alpha or beta can't be zero. Here's a simple way to see why.

Say everyone in the population has the disease. The test is 99% accurate remember.

Now say no one in the population has the disease. The test is 99% accurate remember.

As regards the unambiguous question, post a copy of it here, and we'll see if we can find a quibble.

Wrath of the Swarm
28th April 2004, 10:32 AM
If alpha != beta, then we can't say the test has an accuracy.

We said the test has an accuracy.

Therefore, alpha == beta.

28th April 2004, 10:47 AM
This thread has been bizarre, to say the least.

I would say my reaction to the initial post was very similar to exarch's and ceptimus'. I just assumed he meant the error rates were the same. To tell the truth it seemed obvious to me.

It's interesting that you see people come into this thread and insult WOS for various things, most of which amount to insulting him for continuing on this ridiculous thread for so long. But not many people are saying much about Rolfe continuing the thread for just as long....

There was a problem stated, with a reasonable assumption (IMO) it's easily solved. Without the assumption, it cannot be solved. 5 pages later people are still talking about this?

WOS, you think the problem was stated in an acceptable way. OK.
Rolfe, you think the problem was not clear enough. OK.

We done now? :rolleyes:

Wrath of the Swarm
28th April 2004, 10:57 AM
The statement that an assumption is necessary is a false one.

I just want that to be admitted, and this thread can end peacefully.

Dancing David
28th April 2004, 10:59 AM
Nanny nanny boo boo , seems to be the main reason this thread has gone on.

I say flarn! I thought it would tell me if I had the illness 70 % so I was way off.

Rolfe
28th April 2004, 11:03 AM
Originally posted by Wrath of the Swarm
Secondly, the overall accuracy of the test cannot be established - or even spoken about - without reference to a specific test population if the proportion of false positives is different from the proportion of false negatives.Wrong.

The "accuracy" of the test cannot be established, or even spoken about, until you have clearly defined what you mean by "accuracy". This you have not done.

This subject is a very well-defined one, with its own well-defined vocabulary. So if I were to refer to the "positive predictive value" for example, I would be justified in believing that anyone familiar with the subject would know exactly what I meant by this. If someone unfamiliar with the subject asked me for a definition, it would be easy to provide one.

BillyJoe kindly supplied a list of concise definitions of nearly all the terms recognised for discussing this problem. I'll repeat them again now.

POSITIVES/NEGATIVES, SENSITIVITY/SPECIFICITY, PREDICTIVE VALUES, PREVALENCE.

TRUE POSITIVE: a person who tests positive and has the disease.

FALSE POSITIVE: a person who tests positive but does not have the disease.

TRUE NEGATIVE: a person who tests negative and does not the disease.

FALSE NEGATIVE: a person who tests negative but does have the disease.

SENSITIVITY: the percentage of people with the disease for whom the test is positive.
Sensitivity = TP / (TP + FN)

SPECIFICITY: the percentage of people without the disease for whom the test is negative.
Specificity = TN / (TN + FP)

POSITIVE PREDICTIVE VALUE (PPV): the percentage of people with the disease who test positive for the disease
PPV = TP / (TP + FP)

NEGATIVE PREDICTIVE VALUE (NPV): the percentage of people without the disease who test negative for the disease
NPV = TN / (TN + FN)

PREVALENCE: the percentage of the population who have the disease

courtesy,
BillyJoe

The two extra that BillyJoe didn't mention are:

FALSE POSITIVE RATE: 100 - specificity

FALSE NEGATIVE RATE: 100 - sensitivity.

Nowhere in that list do the terms "accuracy" or "alpha and beta values" appear.

The way these characteristics of a test are evaluated is by taking a group of patients, x% of whom have the condition (known about by other means, the reference method). Test them all by the test you are evaluating. Some of the affected people will test positive, the true-positives (TP). Some of them will test negative, the false-negatives (FN). Some of the unaffected people will test negative, the true-negatives (TN). Some of them will test positive, the false-positives (FP).

This is all the information you get about the test. From this you have to derive the numbers that describe the test to its users. Above, BillyJoe has detailed exactly how this is done for the terms we actually use (and I added a couple more, and defined them).

Now, you will note that the sensitivity and specificity values are completely independent of the proportion of affected people, x. So long as you have enough in each group (affected and unaffected) to give a good representation of test performance, the proportions are irrelevant. Switch them round and the values for sensitivity and specificity will remain the same. And THEY DO NOT HAVE TO BE EQUAL FOR THIS TO BE TRUE.

This is why these two figures are the cardinal descriptive terms of the test. They are absolute values which describe the test independently of population prevalence (which may vary widely).

However, if you actually want to examine the implications of these values for the probability of the test being right in populations of differing prevalence (or as I like to think of it, in patients with differing probabilities of being affected), you need to factor in your assumed prevalence, and derive the PPV and the NPV. And indeed, sometimes the results you get for particular permutations of this sum are somewhat counter-intuitive.

Wrath is persistently referring to "accuracy", and to "alpha and beta values". Now it is impossible to talk about this rationally unless all terms are defined. And by defined, that means explain how you get the number from the results you got when you did the evaluation described in the bold paragraph. Because that is all the information you will ever have (although you may improve on the validity of the exercise by increasing the number of individuals involved).

I thought alpha and beta values were just another way I've never heard of of expressing sensitivity and specificity, and I still suspect that from the way Wrath is using the words. But he won't even confirm that, or which is which.

He persistently refuses to explain what he means by "accuracy".

I think I'm beginning to get it better in the more recent posts. It's one of the suggestions I considered earlier. "Accuracy" is defined by Wrath as the figure you get for both sensitivity and specificity if these two parameters happen to be equal.

No wonder this isn't a defined term! It's a completely useless term. To dream up a term like this which can only be used in the improbable fluke of TP / (TP + FN) happening to come out equal to TN / (TN + FP) in the evaluation scenario described above is meaningless. To then castigate everyone else who doesn't intuit this remarkable definition from your ill-defined scenario, is arrogant and unjustified.IF a general statement about the test's accuracy is made, THEN the alpha must be the same as the beta for the statement to be valid. These tests are perfectly possible - common, even.All right, Wrath, name six. I've already said this pages earlier. In the real world in which most of us actually practise, real tests don't come like that. And if one happens to come like that, it's just a coincidence (and a coincidence which might well just disappear if you extend the numbers of patients in your evaluation study and publish revised and better estimates of the figures). Not worth coining a special defined term for, which is why nobody did. Until Wrath came along.

Now this is boring me too. It isn't the aspect of the problem I really wanted to talk about. I think the use of the word "accuracy" was just a piece of sloppy terminology for "specificity" in the first place. I was perfectly happy all along just to substitute the word "specificity" for "accuracy" in the original question and call it quits on that aspect. Because we don't need the sensitivity value at all to do the sum! It doesn't have to be the same as the specificity, it doesn't have to be anything in particular. If you just say, the test specificity is 99%, you can carry on.

Which is what I was trying to do, honest. (Because there is a lot of carrying on to do, in fact.)

However, we do have something to clear up properly before we do. I've stated that sensitivity and specificity values are a constant property of the test, and do not change with prevalence of disease in the population being tested. And that they not only do not have to be equal, it is no more than a mildly interesting (and unusual) coincidence if they are. I can be told the values for any given test, and I know they describe the performance of the test as they are.

This is why they are the parameters you would quote when asking the initial question the thread is about. In fact the question was simply, "given a test with specificity 99%, what is its positive predictive value in a population with a prevalence of disease of 0.1%?" Answer, no arguing, 9.02%. No need for the sensitivity even to be mentioned, you don't need it. And without the dressing-up as a doctor's appointment, soluble as a pure statistics question.

So, do I have agreement that sensitivity and specificity are absolutes, do not have to be equal (and in fact will probably not be equal), and do not depend (equal or not) on the prevalence of disease in the population used to evaluate the test?

I'm asking this because I've seen some posts that make me think this isn't clear in everyone's minds, and if we don't get it clear we could be in for another three pages of cross-purposes.

Rolfe.

geni
28th April 2004, 11:07 AM
Originally posted by Wrath of the Swarm
The statement that an assumption is necessary is a false one.

I just want that to be admitted, and this thread can end peacefully.

Lets see the maths then. Show me how you can caluculate a unique answer from the information in the intial question.

Rolfe
28th April 2004, 11:10 AM
Originally posted by Wrath of the Swarm
The statement that an assumption is necessary is a false one.

I just want that to be admitted, and this thread can end peacefully. Not the faintest chance. And once we've cleared this up, it's only the start, because it's not what I wanted to talk about anyway.

Now will you please read the post I was writing when you posted this? I can't help feeling that if you'd actually read what I was saying several times before, you'd have got it by now.

You're simply being Humpty Dumpty, making words mean what you've decided they mean, in a field where every primer is prefaced by a list of defined terms in order to avoid misunderstandings like the one you created. By using and sticking to a word which simply isn't defined, because it's a useless concept which would never apply to any but the most flukey real-life tests - then asserting that since you'd used that word, everyone should have known that the test you were using was one of these flukey I've-never-seen-such-a-thing examples.

Rolfe.

Rolfe
28th April 2004, 11:16 AM
Originally posted by ceptimus
I assert that almost any question, no matter how carefully stated, can be spoilt by pointless quibbling. You almost have to use the language of mathematics to avoid this.It would have been nice if this question had been carefully worded. Then we could have moved on to the interesting bit. But it wasn't, in two crucial aspects. And "accuracy" is the simple and boring one.

Wrath, you don't need to use the language of mathematics. You only need to use the already-defined language of the field you are presuming to discuss, as I've pointed out to you too many times already. But by driving headlong with your insistence in your own words and your own meaning, you have created a problem which could have been solved in about four words on page 1.

"Yes, I meant specificity."

There, that wasn't so hard, was it?

Rolfe.

Wrath of the Swarm
28th April 2004, 11:19 AM
That would be a scathing indictment, if only I had used words in ways other than what they were generally accepted to mean.

Test error isn't always affected by the type of response the test gives. In such cases, the test itself can be said to have a specific accuracy.

When the error does change depending on the response, then the test has a specific alpha and beta error rate.

You insisted that such information is necessary to answer the question, when it is not. This behavior on your part demonstrates that you really don't understand the concepts involved, other than in the rote-learning sense.

I'm sure that you can follow a learned algorithm as long as conditions don't fall outside a narrow, prescribed range. But change the nature of the questions presented to you just slightly and... well, we've seen the results.

Wrath of the Swarm
28th April 2004, 11:24 AM
Originally posted by Rolfe
Wrath, you don't need to use the language of mathematics. You only need to use the already-defined language of the field you are presuming to discuss, as I've pointed out to you too many times already. Translated: "Mathematical terminology frightens and confuses me. I want to stick with terms used in medicine, because I've learned them by heart and can refer to them without having to think and stuff."

 Stupid typos.

Rolfe
28th April 2004, 11:34 AM
Originally posted by Wrath of the Swarm
You insisted that such information is necessary to answer the question, when it is not. This behavior on your part demonstrates that you really don't understand the concepts involved, other than in the rote-learning sense.No, Wrath. Why don't you try again to understand it.

All I said was that I had to assume that when you said "accuracy", you actually meant specificity. I was perfectly prepared to do that, and did so. For the purpose of the question I do not care what the sensitivity is. I do not need to know. I never asked you.

However, you have spent the best part of eight pages asserting that "accuracy" must mean that sensitivity and specificity are the same.

Which is a virtually meaningless concept.

You say you are using the word in its "generally accepted" usage. But the only usage which means anything in this discussion is one defined before you have the discussion. Quite a lot of terms are. A term for "the figure you get when sensitivity and specificity happen to be the same" is not defined anywhere, because it's no practical use.

In fact "accuracy" does have a defined meaning in the context of clinical laboratory analysis, as I posted earlier. It just doesn't happen to be of any relevance to this problem.

Your insistence that we accept that sensitivity and specificity must be equal whenever you choose to use the word "accuracy", rather than simply acknowledging that only a specificity value is needed for the problem and the sensitivity is irrelevant and could be anything, demonstrates that you do not understand the concepts involved, and are trying to handle them with tools developed for rather different purposes, and coming unstuck.

Rolfe.

Wrath of the Swarm
28th April 2004, 11:39 AM
But I didn't mean "specificity", I meant accuracy. It's a more powerful concept with strict standards of application.

Admit it: you are so used to thinking about provided alpha and beta values - named 'sensitivity' and 'specificity', respectively - that you mistakenly assumed your familiarity with them meant they had to be present, and their absense meant a mistake had been made.

But your ego and your need to always be perceived as correct won't let you admit the error, will it?

'Whore' is the wrong word for you. I can respect an honest prostitute. You have no intellectual integrity whatsoever.

MRC_Hans
28th April 2004, 11:41 AM
WOTS: It is a pity that you are so aggressive, because I guess you have some knowledge, and perhaps even some interesting viewpoints. Unfortunately, it is all lost in abrasiveness.

Hans

drkitten
28th April 2004, 11:44 AM
Originally posted by Wrath of the Swarm
Translated: "Mathematical terminology frightens and confuses me. I want to stick with terms used in medicine, because I've learned them by heart and can refer to them without having to think and stuff."

I don't think that the term "accuracy" means what you think it does in mathematics, either.

Semantic quibbles aside : what was your original point going to be?

Wrath of the Swarm
28th April 2004, 11:49 AM
First, my name is not Mr. Fluffer-nutter Fuzzy Wuzzy. It's WRATH OF THE SWARM. If you were expecting kid gloves and gentle talk, you've come to the wrong poster. If you're looking for compassion, you've come to the wrong universe.

Second, if you can't look past the personalities in an argument to see the arguments, you're a fool. You're no different from the believers that get suckered by personalities and pretty speeches and swallow any argument disguised in them.

yersinia29
28th April 2004, 11:53 AM
Remember guys, this Wrath fool is the same idiot who said that its OK if people get killed from "alternative" medicine if the gene pool is improved.

Expecting logic to come from such a dolt is an exercise in futility.

MRC_Hans
28th April 2004, 11:53 AM
WOTS: Certainly :rolleyes:, but BESIDES showing off your intrinsic aggressions, do you HAVE A POINT?

Hans

Wrath of the Swarm
28th April 2004, 11:54 AM
Originally posted by drkitten
I don't think that the term "accuracy" means what you think it does in mathematics, either.
http://mathworld.wolfram.com/Accuracy.html

Nice try. Well, it wasn't actually a very good try at all, but you get points for effort. Well, since it was a throwaway attempt at a stab in the dark, no you don't.

Rolfe
28th April 2004, 12:07 PM
Originally posted by Wrath of the Swarm
But I didn't mean "specificity", I meant accuracy. It's a more powerful concept with strict standards of application.OK, more ad-homs. Enough.

I would like to point out again that I run a diagnostic testing laboratory. That is run. I choose the serological tests we will offer, and decide which is going to perform best according to the demands our clients will make on it. I evaluate tests, and decide which are the most appropriate for our requirements. I have published on the subject, in a refereed journal. Oh, and by the way, not one of these tests has equal sensitivity and specificity.

And every day, I am responsible for "signing-out" scores of serological test results from real patients (and the arithmetic works just the same way in animals as it does in humns). I am also responsible for advising my colleagues in practice on the true reliability of any test result they have received. I make my living doing this. So far, I have not exactly gone bankrupt.

On the contrary, I am considered something of an expert in the field. I have been invited to speak at conferences on the subject. As well as the refereed publications, I have pubished a text-book which included a chapter covering just this issue, and an eminent professor chose that chapter for especial praise in a very flattering book review.

Wrath, we may disagree. But I do not appreciate language like the following.'Whore' is the wrong word for you. I can respect an honest prostitute. You have no intellectual integrity whatsoever. Interestingly, we know nothing of your qualifications, your background in this or other fields, or your reason for being interested (indeed, aerated) about the subject. All we know is that you're so sure you're right you won't even consider any alternative.

Now, to your point above. Please explain in what way a term which is only applicable to the rare and coincidental cases where specificity and sensitivity happen to be the same, is "more powerful" than the standared expressions of sensitivity and specificity. It's certainly a strict standard of application - it's only applicable in rare and unusual cases. But what makes it powerful?

Rolfe.

MRC_Hans
28th April 2004, 12:09 PM
That definition of accuracy obviously does not fit the "accuracy" discussed here. The traditional definition of accuracy (as specified on the referred site) is not dependent on the probability of a given value.

That the "accuracy" of a medical test IS dependent on the probability of a given result is new to me (and seems odd to me) but I bow to the authoritiess here.

Hans

exarch
28th April 2004, 12:16 PM
Originally posted by Wrath of the Swarm
If alpha != beta, then we can't say the test has an accuracy.

We said the test has an accuracy.

Therefore, alpha == beta.If the test is 99% accurate at spotting an affliction with a 1/1000 occurence, the "alpha" value is 0.00001. In other words, 99.99999 specificity. Very unlikely :rolleyes:

Your test is only 9% accurate.

Rolfe
28th April 2004, 12:18 PM
Originally posted by Wrath of the Swarm

http://mathworld.wolfram.com/Accuracy.html

Nice try. Well, it wasn't actually a very good try at all, but you get points for effort. Well, since it was a throwaway attempt at a stab in the dark, no you don't. Not a relevant definition to the subject under discussion. In fact it's a general form of the definition I posted earlier which is used in the biochemistry field. Note the coupling with the related concept, precision.

However, the only terms you can use to discuss the very precise field you are involved in now are those which can be derived from the various permutations of the numbers of true-positives, false-positives, true-negatives and false-negatives in your test population.

If you can't define your term that way, it is not a useful term.

I'm not throwaway stabbing in the dark, I'm simply requiring you to do what I require any student to do before discussing the topic. Understand the definitions of the terms used, and be able to define mathematically with reference to these four figures what any other term they want to use actually means.

Rolfe.

Rolfe
28th April 2004, 12:22 PM
Originally posted by MRC_Hans
That the "accuracy" of a medical test IS dependent on the probability of a given result is new to me (and seems odd to me) but I bow to the authoritiess here. No, Hans, it isn't. "Accuracy" as Wrath is trying to define it, is simply not a useful concept in this field. Which is why the word isn't used.

Specificity and sensitivity are not dependent on the probability of a given result. However, it has been found useful also to consider the positive and negative predictive values, which are.

It's explained back where I quoted BillyJoe's explanation, because he was kind enough to deminstrate how all the terms we use are derived from four basic figures derived in the initial evaluation of the test. Wrath's "accuracy" cannot be so defined, therefore it is not a useful concept.

Rolfe.

Wrath of the Swarm
28th April 2004, 12:24 PM
But if you always assume that what an "authority" says is true, you'll never find evidence that what they say isn't actually correct.

The degree to which a test's results match the actual values of the test subjects is the test's accuracy. The standard definition given in the link is precisely what we're talking about - we're discussing binomial distributions, not continous ones. The degree to which something is free from error is its accuracy. If we can say that the test's results will match reality 99% of the time, regardless of the nature of the result itself, we can say that the test is 99% accurate. If the specific result matters, we can't say that - we need to specify alpha and beta values.

The accuracy of the test is one thing; the accuracy of the conclusion drawn from that test's result is another. That's the whole point of the question - it's counterintuitive.

exarch
28th April 2004, 12:26 PM
Originally posted by ceptimus
No exarch. alpha or beta can't be zero. Here's a simple way to see why.

Say everyone in the population has the disease. The test is 99% accurate remember.

Now say no one in the population has the disease. The test is 99% accurate remember.

As regards the unambiguous question, post a copy of it here, and we'll see if we can find a quibble.You're right. if the accuracy is the occurence of false positives, it can't be.
Ceptimus, this whol thing hinges on WotS's misunderstanding between "accuracy" and "specificity" (occurence of false positives).

WotS's test is in essence only 9% "accurate" at positively identifying someone as having the affliction, not 99%.
It is 99% specific though.

Rolfe
28th April 2004, 12:28 PM
Originally posted by Wrath of the Swarm
That's the whole point of the question - it's counterintuitive. And that's the point I'd have liked to pursue from the start - that it's only counter-intuitive if you load the question in a specific way.

However, your ridiculous insistence in porting this "accuracy" concept from whatever field you do have familiarity with to a field you clearly understand little about has cause this discussion to stall for about six or seven pages.

What use to man or beast is a term which can only be applied to the rare and flukey situation where sensitivity and specificity happen to be equal?

Rolfe.

Wrath of the Swarm
28th April 2004, 12:30 PM
Originally posted by Rolfe
I'm not throwaway stabbing in the dark, I'm simply requiring you to do what I require any student to do before discussing the topic. Understand the definitions of the terms used, and be able to define mathematically with reference to these four figures what any other term they want to use actually means.

Those four concepts are derivable from the provided information in this particular case. This case was chosen because it was a simple example - in reality, things are often more complicated, but it should be easier to answer the question than it would be otherwise.

Doctors generally deal with more sophisticated forms of error than we were here. That doesn't mean the simpler form of error is wrong or invalid in any way.

Rolfe doesn't understand these points in a mathematical sense. She learned the procedure doctors need to know and memorized it. Now she can plug values into the algorithm and generate results, but she doesn't understand what she's doing at all.

Rolfe
28th April 2004, 12:34 PM
Originally posted by exarch
You're right. if the accuracy is the occurence of false positives, it can't be.
Ceptimus, this whol thing hinges on WotS's misunderstanding between "accuracy" and "specificity" (occurence of false positives).

WotS's test is in essence only 9% "accurate" at positively identifying someone as having the affliction, not 99%.
It is 99% specific though. Exarch, please read the bit that BillyJoe provided. The useful data is getting buried under Wrath's ad-homs and blind assertions here, and I don't have time to type it all again.

You have to be able to define how you derive any numerical description of the test you are using, in terms of only four variables you derive when you evaluate the test (TP, FP, TN and FN). "Accuracy", if you want to use the term, has to be defined in that way too.

Wrath's usage can't be defined like that, because it is only applicable to a particular subset of situations where TP / (TP + FN) happens to come out equal to TN / (TN + FP). Which virtually never happens in practice.

You know how Geni says, give me the maths for that? Well, this is a situation where you have to be able to "give the maths" for how you derive any attribute of the assay you quote. If you can't, don't quote it.

Rolfe.

Rolfe
28th April 2004, 12:35 PM
Originally posted by Wrath of the Swarm
Rolfe doesn't understand these points in a mathematical sense. She learned the procedure doctors need to know and memorized it. Now she can plug values into the algorithm and generate results, but she doesn't understand what she's doing at all. Wrath, you don't have a bloody clue what you're talking about, do you?

Rolfe.

MRC_Hans
28th April 2004, 12:38 PM
WOTS: I will give you one more try. What is you point here? I don't give a d*mn what you think about Rolfe, it is simply not interesting. Do you have some point about medical testing (other than extreme figures yield extreme results)?

Hans

Wrath of the Swarm
28th April 2004, 12:39 PM
Originally posted by Rolfe
And that's the point I'd have liked to pursue from the start - that it's only counter-intuitive if you load the question in a specific way. No, the general result is counter-intuitive. We don't need to "load" the question at all - tests performed by doctors are rarely even 99% accurate, and they frequently involve relatively uncommon conditions.

Even considering that lab tests are often ordered because of worrisome symptoms (with the identification of such symptoms serving as a crude pre-test which alters the population sample), such tests have significant margins of error. This needs to be acknowledged and understood.

What use to man or beast is a term which can only be applied to the rare and flukey situation where sensitivity and specificity happen to be equal? Mathematicians seem to find it useful indeed.

In a broader sense, we can even discuss accuracy when alpha is not equal to beta, but we can't use that value in discussions of cases where the subject falls into a specific category. It doesn't help to know that a test is 99.9% accurate in general if it's only 20% accurate at detecting a very rare condition and we're concerned that a person might have have that condition.

exarch
28th April 2004, 12:44 PM
This thread has been bizarre, to say the least.

I would say my reaction to the initial post was very similar to exarch's and ceptimus'. I just assumed he meant the error rates were the same. To tell the truth it seemed obvious to me.

It's interesting that you see people come into this thread and insult WOS for various things, most of which amount to insulting him for continuing on this ridiculous thread for so long. But not many people are saying much about Rolfe continuing the thread for just as long....The only problem I see is Wrath of the Swarm immediately attacking Rolfe for pointing out an innacuracy in the wording of the problem. And his insistence that the wording of his problem was not wrong, and that he had proof of this, but that Rolfe doesn't know what she's talking about, being one of the "ignorant doctors".

I would have let it slide if it had been anyone else, probably because anyone else would have conceded right away that Rolfe actually has a point, and not have started insulting either. The problem I have is with WotS's attitude. The holier than thou "I know it better than all of you", and his unwillingness to admit even a small mistake. He's very quick to point out even the tiniest problem in everyone else's posts though, but he gets extremely upset if you do the same to him. My last 8 or 9 posts in this thread are just an attempt to toss more oil on the fire. Maybe rubbing it in might humiliate him just enough to think twice in the future before immediately ad-homming everyone who dares to disagree, falsely assuming he himself doesn't make mistakes.

It's extremely annoying and it doesn't further the conversation one bit. Which has been shown extensively on this thread I think :), where everyone else is nitpicking HIS post and HIS statements.

I doubt he'll learn though.

exarch
28th April 2004, 12:51 PM
[DOUBLE POST]

Wrath of the Swarm
28th April 2004, 12:52 PM
Originally posted by MRC_Hans
WOTS: I will give you one more try. What is you point here? I don't give a d*mn what you think about Rolfe, it is simply not interesting. Do you have some point about medical testing (other than extreme figures yield extreme results)? The first point is that people aren't very good at interpreting the results of statistical tests - even people who do so regularly and who make life-or-death decisions on the basis of such tests.

The second point is that doctors generally do not have that great an understanding of what they do, or the strengths and weaknesses of the methods they were taught. As I pointed out quite some time ago in this thread when someone asked me this question previously, that's why doctors aren't very good at detecting when there's something wrong with treatments.

For treatments that can be carefully tested by people not prescribing them (like drugs), it's a bit easier. Surgical treatments are especially known for being used without strong experimental support for their safety and effectiveness, and many turn out to be useless or even harmful.

I remember reading an article in Discover magazine some years ago about a surgical treatment for strokes that involved rerouting blood through veins. It was considered an effective and appropriate treatment, and it was used for something like thirty years. The author even discussed practicing the complicated procedure of tying off and reconnecting vessels on rats in school. Then a comprehensive study was conducted to see if patients who had the procedure were better off. Not only did they not improve, the procedure turned out to increase the rate of further strokes significantly.

The procedure was abandoned, but the author's point was that for decades, everyone was convinced that it was helpful and beneficial. Surgeons swore by it. And the whole time it was worse than doing nothing at all.

I don't believe the article is still online. If you search through library archives, you can probably find a copy of it.

Wrath of the Swarm
28th April 2004, 12:55 PM
Originally posted by exarch
The only problem I see is Wrath of the Swarm immediately attacking Rolfe for pointing out an innacuracy in the wording of the problem. And his insistence that the wording of his problem was not wrong, and that he had proof of this, but that Rolfe doesn't know what she's talking about, being one of the "ignorant doctors". But that's my point - it's not an inaccuracy.

If I had merely made a mistake, I would have accepted it. But Rolfe's claims are wrong.

And for the record, I find Rolfe's "I'm an expert, so if what I say is nonsensical or contradicts basic logic, you must just be too ignorant to understand" attitude to be completely offensive. So is her attitude that any criticism toward any aspect of medicine is an attack on the field as a whole and its practitioners in general.

Rolfe
28th April 2004, 12:59 PM
Originally posted by MRC_Hans
WOTS: I will give you one more try. What is you point here? I don't give a d*mn what you think about Rolfe, it is simply not interesting. Do you have some point about medical testing (other than extreme figures yield extreme results)?Hans, there is a point. I'd like to get to it. In fact it is what Wrath touches on in the post immediately below yours here.

Now, I might see a way through.

Wrath. Do you accept that the sensitivity value of the test in question is irrelevant to the problem you posed?

Do you accept that you intended it to be assumed that the specificity value was 99%? (Seems to follow, if you intended it to be assumed that specificity was equal to sensitivity and that both were equal to 99%.)

Do you therefore accept that I don't honestly care what the sensitivity is? Not for the purpose of your question. You can have it as 99% if you like. Or 100%, or 20%. I don't care because it doesn't matter.

So can we continue on the assumption that you want us to know that specificity is 99%?

To go on stalling just because you also want to assert that the sensitivity is 99% too (even though the figure is irrelevant to the calculation), seems rather sterile by now.

Rolfe.

28th April 2004, 01:00 PM
Having trawled these many pages of semantic tittle-tattle, I still find the need to post the same comments as I would have done about 4 pages ago!

I do not claim expertise in this field, but I have never heard or read the term 'accuracy' used in the context of diagnostic testing except and only in the casual use of the word as a qualitative judgement.

Rolfe has beaten me to the point that 'accuracy', with 'precision', has an exact meaning in measurement of numerical variables.

Diagnostic testing, reported as 'positive' or 'negative' does not employ the term 'accuracy' among its jargon of specificity/sensitivity, NPV and PPV and that jargon can completely define the behaviour of a diagnostic test without resort to use of the term 'accuracy'.

WotS' link to a Wolfram site rather confirmed the use of 'accuracy' in the sense described above and not as it relates to binary diagnostic testing in which the values are counts and proportions not continuous variables.

I stand to be corrected, but what it would require is a reference explicitly and formally linking specificity and sensitivity and something called accuracy.

Rolfe
28th April 2004, 01:05 PM
Originally posted by Wrath of the Swarm
The first point is that people aren't very good at interpreting the results of statistical tests - even people who do so regularly and who make life-or-death decisions on the basis of such tests.

The second point is that doctors generally do not have that great an understanding of what they do, or the strengths and weaknesses of the methods they were taught. ....Oh wow, no agenda here at all.

Since these are all asssertions, which Wrath would like to get to by his selective use of statistics and rigged scenarios following his own little train of thought, can we leave them as points still to be demonstrated, or not, as we discuss the wider implications of the test?

Wrath has frequently told us that he is annoyed that the "accuracy" question was holding up what he wanted to do with the thread, which was make doctors look stupid. I can see how he wants to do it, but I think he has to actually do it rather than just assert "this is what I will prove".

After all, it might not be entirely as he imagines.

Rolfe.

28th April 2004, 01:06 PM
Originally posted by exarch
The only problem I see is Wrath of the Swarm immediately attacking Rolfe for pointing out an innacuracy in the wording of the problem. And his insistence that the wording of his problem was not wrong, and that he had proof of this, but that Rolfe doesn't know what she's talking about, being one of the "ignorant doctors".

I would have let it slide if it had been anyone else, probably because anyone else would have conceded right away that Rolfe actually has a point, and not have started insulting either. The problem I have is with WotS's attitude. The holier than thou "I know it better than all of you", and his unwillingness to admit even a small mistake. He's very quick to point out even the tiniest problem in everyone else's posts though, but he gets extremely upset if you do the same to him. My last 8 or 9 posts in this thread are just an attempt to toss more oil on the fire. Maybe rubbing it in might humiliate him just enough to think twice in the future before immediately ad-homming everyone who dares to disagree, falsely assuming he himself doesn't make mistakes.

It's extremely annoying and it doesn't further the conversation one bit. Which has been shown extensively on this thread I think :), where everyone else is nitpicking HIS post and HIS statements.

I doubt he'll learn though.

While I agree with most of your post, and (Mandy forgive me!) I even support your oil throwing in a way :D, I do think you're missing something.

You see, everything Rolfe has said was said by page one or two. The fact that Rolfe is also willing to continue this nonsense for 6 pages does not reflect well on him/her either. Why continue? What is the point?

But then again, why am I still reading this thread? Why am I posting to it? Hmmmmmmmmmmmmmmmmm........

28th April 2004, 01:08 PM
p.s. I did get WotS' original question right even though I am only an idiot clinician.

p.p.s. WotS, you've been asked before and I still haven't seen an answer, or I may have missed it in all the pages of bickering. Please explain what you mean by alpha and beta probabilities and tell us from what field this is standard jargon, because it isn't standard for me. I wondered whether you were talking about alpha and beta error rates in relation to hypothesis testing, but that doesn't seem to fit your usage.

exarch
28th April 2004, 01:08 PM
Originally posted by Wrath of the Swarm
http://mathworld.wolfram.com/Accuracy.html

Nice try. Well, it wasn't actually a very good try at all, but you get points for effort. Well, since it was a throwaway attempt at a stab in the dark, no you don't.Accuracy

The degree to which a given quantity is correct and free from error. For example, a quantity specified as 100 ± 1 has an (absolute) accuracy of ± 1 (meaning its true value can fall in the range 99-101), while a quantity specified as 100 +/- 2% has a (relative) accuracy of +/- 2% (meaning its true value can fall in the range 98-102).

The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits.
Odd. I don't see anything regarding statistics, false positives or alpha/beta here :confused:

Rolfe
28th April 2004, 01:08 PM
Sorry, double post. (Evil software!)

Wrath of the Swarm
28th April 2004, 01:13 PM
'Accuracy' has a meaning in terms of the mathematics that underlie medical testing. Whether the term is ever brought by in the field of medical testing or not is irrelevant. It has a simple, easy-to-understand meaning that can be applied in this example with a minimum of thought.

Rolfe: You can't weasel out that easily. You insisted that the question could not be answered without my explicitly providing the rate of Type I error (although you used different terminology) and proclaimed that my question was ambiguous because of my ignorance.

The data I gave you was more than sufficient to answer the question, and you know it. Your mealy-mouthed evasions wouldn't be nearly as mealy if you weren't aware of how deep a hole you've dug yourself into.

You're simply wrong, and no amount of complaints about "ad homs" (they're insults, toots, not ad hominem attacks, not that you can tell the difference) or attempts to troll for sympathy as a poor little misunderstood fighter for skepticism and justice will change that.

28th April 2004, 01:13 PM
Looking again at the Wolfram site, I don't think it is about testing at all.

http://mathworld.wolfram.com/Accuracy.html

http://mathworld.wolfram.com/Precision.html

In the realm of laboratory testing accuracy mean the closeness of the mean measured value to the true value of the test parameter measured by some gold standard or known a priori. Precision is effectively the same as reproducibility.

Maybe I am labouring under a misapprehension here and Rolfe can correct me, but if I am right the Wolfram citation is some very narrow mathematical programming use of these terms not how they are used in statistics generally.

Rolfe
28th April 2004, 01:19 PM
You see, everything Rolfe has said was said by page one or two. The fact that Rolfe is also willing to continue this nonsense for 6 pages does not reflect well on him/her either. Why continue? What is the point?First, Wrath is also continuing with the same assertions he was making on about page 2. Just avoiding defining what he means. While he continues, why should I not continue?

This is my academic subject. Wrath has continually asserted that I am stupid, and ignorant, and worse. I feel that to retire while he is still taking that attitude is to concede to him, and I will not do that. He is trying to pontificate on a subject he is not familiar with, and I will not have it appear by my retiring that I give the slightest acknowledgement to his ill-conceived and ill-thought-out arguments.

Also, there is a more interesting discussion waiting in the wings. One which may make this look like a tea-party given Wrath's temperament, but interesting nonetheless.

Now if we can only agree that Wrath meant to imply that both specificity and sensitivity were 99%, and as sensitivity doesn't matter an iota for the purpose of the problem he sets, he might as well have said specificity is 99% from the start, we migth just manage to go on.

Rolfe.

yersinia29
28th April 2004, 01:20 PM

You have no evidence for any of the idiotic claims you make.

Quit referring to it as "evidence" and call it what it is: your own personal opinion and nothing more.

exarch
28th April 2004, 01:20 PM
Originally posted by Wrath of the Swarm
But that's my point - it's not an inaccuracy.No, it's an "accuracy" :D

Wrath of the Swarm
28th April 2004, 01:21 PM
alpha: chance of making a Type I error (incorrectly dismissing the null hypothesis)

beta: chance of making a Type II error (failing to correctly dismiss the null hypothesis)

In medicine, the null hypothesis is usually that the person lacks the condition being tested for. (Sometimes not, of course, but generally so.)

So alpha represents the chance of giving a positive response when the patient is actually negative. (A false positive.)

Beta represents the chance of giving someone a negative response when they're actually positive. (A false negative.)

For many simple tests, alpha is equal to beta. For more complex ones, alpha and beta aren't necessarily the same. Generally, it is considered better to have as few false negatives as possible, even if it results in false positives. Medical testing is usually biased towards false positives for this reason.

Wrath of the Swarm
28th April 2004, 01:23 PM
Accuracy, in the Wolfram site sense, is a concept involved with measurement - and ultimately, all testing is just a subcategory of measurement.

The definition provided applies.

28th April 2004, 01:31 PM
Originally posted by Rolfe
First, Wrath is also continuing with the same assertions he was making on about page 2. Just avoiding defining what he means. While he continues, why should I not continue?

Because you're better than him? Because it's futile. Because you have more important things to do.

If someone calls you a name in the street and continues to stand there repeating the insult, do you stand there for a few hours and continue to explain how he is incorrect? It seems a futile stance to me.

This is my academic subject. Wrath has continually asserted that I am stupid, and ignorant, and worse. I feel that to retire while he is still taking that attitude is to concede to him, and I will not do that. He is trying to pontificate on a subject he is not familiar with, and I will not have it appear by my retiring that I give the slightest acknowledgement to his ill-conceived and ill-thought-out arguments.

I understand what you are saying, but how long are you willing to continue? Should WOS have stamina to boggle the mind and can keep this up for the next 10 weeks, you still going to be here on page 666 saying the same things? I just think that people who feed trolls have no right to complain about them.

Also, there is a more interesting discussion waiting in the wings. One which may make this look like a tea-party given Wrath's temperament, but interesting nonetheless.

Then move along. Is WOS the only person this new discussion could take place with? Is that why you are waiting?

Now if we can only agree that Wrath meant to imply that both specificity and sensitivity were 99%, and as sensitivity doesn't matter an iota for the purpose of the problem he sets, he might as well have said specificity is 99% from the start, we migth just manage to go on.

Rolfe.

Why do you need this to go on. It's patently obvious that the basic message is true. Move on.

Your refusal to do so, just makes you look like WOS to me. You can't let go.

BUT! I don't want you to think I'm unfairly picking on you in this idiotic affair. It's just that many people are pointing out what a jerk WOS is being, it's easy to see :D

I think though, that you display similiar behaviour in not being able to let this go and move on (to what you admit is the more interesting bit!). I suppose, it's really just me wanting to vent about the troll feeders that give them just what they want. More and more and more and more and more posts to respond to.....

Rolfe
28th April 2004, 01:37 PM
Originally posted by Wrath of the Swarm
Rolfe: You can't weasel out that easily. You insisted that the question could not be answered without my explicitly providing the rate of Type I error (although you used different terminology) and proclaimed that my question was ambiguous because of my ignorance.

The data I gave you was more than sufficient to answer the question, and you know it. Your mealy-mouthed evasions wouldn't be nearly as mealy if you weren't aware of how deep a hole you've dug yourself into.OK, note that I tried to move this on, but Wrath won't have it. (He also admits he's being deliberately insulting, but then we're told to cultivate a thick skin so I will let that run off.)

NOW HEAR THIS. THE PROBLEM AS IT STANDS IS NOT SOLVABLE UNLESS THE SPECIFICITY VALUE FOR THE TEST IS PROVIDED.

I'm finding Wrath's insistence on using his own words rather than the words clearly defined at the beginning of all primers in the subject (including American ones, the example in front of me now is American) annoying, but we will proceed.

As several people said at the start, the result might be different depending on how you have combined sensitivity and specificity to get this "accuracy" figure. (x<SUB>1</SUB> + x<SUB>2</SUB>) / 2 = 99%, fine but now we no longer know what either x<SUB>1</SUB> or x<SUB>2</SUB> were. And since only one of those is required for the solving of the problem, we are stymied.

The only way to proceed is to assume that we do know, because there is an underlying assumption that x<SUB>1</SUB> = x<SUB>2</SUB>. Finally, Wrath confirmed this. His very use of the term "accuracy" was to be taken to imply that (however unlikely this may be in practice), x<SUB>1</SUB> must be equal to x<SUB>2</SUB>. Because otherwise this "powerful" term couldn't be used at all.

So, we had a problem that required we know the specificity. Wrath told us a figure which we sort of gathered might be (specificity + sensitivity) / 2. Several people protested that this wasn't really good enough. About five pages later it gradually became obvious from Wrath's posts that since the term was indeed meaningless unless sensitivity was equal to specificity, then we were intended to assume that this was the case.

So, rather than simply tell us that specificity was 99%, he told us (we finally figured) that (specificity +sensitivity) / 2 was 99%. Then he admitted that the extra wrinkle was that sensitivity = specificity, implied by his particularly unique use of terminology.

OK, if (specificity +sensitivity) / 2 = 99%, and specificity = sensitivity, we are finally able to deduce that specificity (the term we need the value for) is 99%.

WOW.

And now Wrath is saying that we don't need the specificity at all? Now this is just plain wrong, whichever way you slice it. The figure you need for the calculation is specificity, and because Wrath chose to make it a very uphill battle to discover for sure how he meant us to derive that, doesn't alter the necessity for its being derived.

Who is in the hole?

Rolfe.

exarch
28th April 2004, 01:38 PM
Originally posted by Wrath of the Swarm
they're insults, toots, not ad hominem attacks, not that you can tell the differenceFrom here (http://en.wikipedia.org/wiki/Ad_hominem)

An ad hominem argument, also known as argumentum ad hominem (Latin, literally "argument against the man"), is a fallacy that involves replying to an argument or assertion by attempting to discredit the person offering the argument or assertion.

Wrath of the Swarm
28th April 2004, 01:42 PM
We've been over this.

For any patient, under any condition, the chance of the test producing an incorrect result is 1%. The accuracy for any patient, under any condition, is 99%.

Of the people who don't have the condition, 99% will receive a negative result and 1% will receive a positive.

Of the people who do have the condition, 99% will receive a positive result and 1% will receive a negative.

This is almost an idea situation. Very low alpha, very low beta. They don't get much more predictive power than this.

Rolfe
28th April 2004, 01:44 PM
Because you're better than him? Because it's futile. Because you have more important things to do.

I understand what you are saying, but how long are you willing to continue? Should WOS have stamina to boggle the mind and can keep this up for the next 10 weeks, you still going to be here on page 666 saying the same things? I just think that people who feed trolls have no right to complain about them.I do have more important things to do, you're right. But I'm also a teacher (though I no longer hold a university teaching post, I still teach this subject at post-graduate level).

If I'm failing to get my point across, if there is one of the audience (or it looks like more) who don't understand me, I like to try to do better, one more attempt to make the situation clear.

Rolfe.

Rolfe
28th April 2004, 01:48 PM
Originally posted by Wrath of the Swarm
For any patient, under any condition, the chance of the test producing an incorrect result is 1%. The accuracy for any patient, under any condition, is 99%.This is almost a meaningless statement.

Wrath, unless you tell me how you get this particular 99% from TP, FP, TN and TP numbers gathered from the evaluation testing of the assay, we can't even discuss it.

Rolfe.