Poll: Accuracy of Test Interpretation

[Rolfe]

I really do have a catalogue of laboratory tests to proofread. You know, some of these things that give concentration results, where accuracy and precision are the watchwords, and others that give positive/negative results, which we judge by assessing sensitivity and specificity.

So, either Wrath admits that the figure required for the problem he posed is the specificity (however derived), and the accuracy doesn't matter and doesn't need to be equal to the specificity for the problem still to be meaningful....

Or he explains how his "accuracy" figure would be worked out as a numerical value from the findings of the evaluation testing of the assay, which we recall are limited to:
number of patients testing true-positive
number of patients testing false-positive
number of patients testing true-negative
number of patients testing false-negative.

Because that is how it is done. Binary assays are tested in that way, those are the results you acquire, and those are the only data you have to derive all the numerical terms you want to define the assay.

BillyJoe showed how all the terms we use are derived from these four figures. Knowing how they are derived makes us able to use them "accurately". So tell us how to use "accuracy" in that way.

Or simply agree to move on, the only relevant figure needed for the problem was specificity, you took a helluva roundabout way to give it to us but we got there, now let's look at that 9.02% number and its implications.

Rolfe (off to proofread, see you in the morning - or 4 am if insomnia strikes!).

[ceptimus]

Originally posted by Rolfe
This is almost a meaningless statement.

Wrath, unless you tell me how you get this particular 99% from TP, FP, TN and TP numbers gathered from the evaluation testing of the assay, we can't even discuss it.

Rolfe. Well, here's one way simple enough for all to understand:

Try the test on 100,000 people known to have the disease, the test says that 99,000 have it, and 1,000 don't.

Estimate of probability of a false negative is 1,000 / 100,000 = 1%


Now try the test on 100,000 people known to be free of the disease, the test says 99,000 are clear and 1,000 of them have it.

Estimate of probability of a false positive is 1,000 / 100,000 = 1%

As both figures have turned out the same we combine them together and say the test is always 99% accurate.

Now quibble away.

[Rolfe]

Originally posted by ceptimus
As both figures have turned out the same we combine them together and say the test is always 99% accurate.

Now quibble away. Flying visit.

Most of the time in fact nearly all of the time both figures do not turn out to be the same. So your definition needs to be able to cope with that to be any use in the real world.

How does it do that?

Rolfe.

[ceptimus]

Originally posted by Rolfe
Flying visit.

Most of the time in fact nearly all of the time both figures do not turn out to be the same. So your definition needs to be able to cope with that to be any use in the real world.

How does it do that?

Rolfe. It doesn't need to, as IN THIS CASE they are both the same. Of course, IF they were not both the same, then both would have to be given.

There are plenty of real world tests where the chance of a false positive IS the same as the chance of a false negative, even though this may not be true of medical tests.

The fact remains that Wrath's question, as posted has a definite answer. You may not like him, and you may believe he has a hidden (or not so hidden) agenda, and find it distasteful to admit that he is right, but the mathematics speaks for itself. Saying that real world situations are almost always different to Wrath's hypothetical situation, does not prove him wrong.

I think you (Rolfe) cast the first stone in this thread, by posting the answer and your chart, and so spoiling Wrath's poll. I think both of you are equally guilty of hurling insults at each other.

Anyway, I am happy to continue to debate for as long as is necessary, and it doesn't make me angry - I enjoy it. I might still learn something about statistics, though I've not learned any statistics from this thread yet.

[exarch]

Originally posted by ceptimus
As both figures have turned out the same we combine them together and say the test is always 99% accurate.Why combine them at all?

Anyway, the real issue is 9.02%, as has been pointed out (or, alternately, 90.98%).

Even though even that figure is purely theoretical, since a case like this will only pop up in obligatory tests, like the physical you get when joining the army, or testing for HIV infection.
What do you think is the first thing people who test positive for HIV do? Get a second opinion I think ...

Does anyone know how well a second test performs. Are those "accurate"?

[steve74]

I had decided to cease posting to this thread after Wrath admitted he didn't take his question from a study involving doctors:

I will admit that I can't find a source that used precisely the same question. That is the question I remember being given (quite vividly), but I can't demonstrate that it was ever used in a study.

I retract the claim and admit I was wrong to make it.


But Wrath wouldn't let it go and accused me of being a liar. When I pointed out the blatant lie he told in this thread (http://www.randi.org/vbulletin/showthread.php?s=&postid=1870429347#1870429347) in the TRSOTTTWND he had no reply over there. Normally I'd let it go (hell, I've had fun today making Wrath eat his words) but when someone accuses me of lying, it becomes a point of principle.

Wrath, you lied in this thread. You claimed:

I finally found the sources that duplicated the question (I even pointed them out, remember?).

That is a lie. Look at the first quote in this post. You contradict yourself. You have lied and yet you accuse me of being a liar. Wrath, you are a bad joke. Its time for you to own up to your lie and apologise for claiming that I lied.

Edited to add: I apologise for derailing this thread but when someone lies in a thread I feel it is important that they are called up on it.

[Rolfe]

Originally posted by ceptimus
It doesn't need to, as IN THIS CASE they are both the same. Of course, IF they were not both the same, then both would have to be given. It does need to. A descriptive parameter for the test which is only applicable in the unlikely chance of the two values being the same is useless.

You can see the artificiality of the situation from the way you posted your calculation (which was clear, thank you).Try the test on 100,000 people known to have the disease, the test says that 99,000 have it, and 1,000 don't.

Estimate of probability of a false negative is 1,000 / 100,000 = 1%

Now try the test on 100,000 people known to be free of the disease, the test says 99,000 are clear and 1,000 of them have it.

Estimate of probability of a false positive is 1,000 / 100,000 = 1%
You had to force the two groups to be the same, or your scenario simply wouldn't have worked. But in real life there is no reason at all why they should be the same. They are not independent variables, because (as Wrath pointed out) there is a tendency for improvements on one side to cause a deterioration on the other - efforts to eliminate false negatives tend to increase false positives and vice versa. They tend to be opposite variables, which actually makes it less likely that they will happen to be identical in any given example.

Wrath has been asserting that this "accuracy" definition is a "more powerful" way to describe the test. However, you have just shown quite clearly that the term simply cannot be used at all except in the very unlikely chance that false positive rate and false negative rate are identical. This is precisely why it is not used. There is in fact no meaningful way to combine the two sides for the general case, and give an overall "accuracy" figure which does not vary with prevalence. This is why it is standard practice to quote sensitivity and specificity separately.Originally posted by ceptimus
There are plenty of real world tests where the chance of a false positive IS the same as the chance of a false negative, even though this may not be true of medical tests. This may be so. But we are not talking about other situations here, this discussion is specifically about medical tests. For the purpose of going on to discuss the competency or otherwise of doctors, as Wrath has told us several times. Therefore the deliberate use of a defining term which is simply not applicable to the medical testing situation is perverse, to put it mildly.Originally posted by ceptimus
The fact remains that Wrath's question, as posted has a definite answer.No, Ceptimus, it doesn't.

You cannot calculate a positive predictive value (which was in effect the question) unless you know the SPECIFICITY. That is the percentage of unaffected patients who test positive. (We don't care, for this question, how many affected patients test negative, we don't need to know.)

Wrath did not provide this information in a way that could be understood without making assumptions.

My assumption was, OK, I know I need the specificity figure. I've been given something called "accuracy", which I know is not a meaningful concept in the context of this type of testing. I will assume that this is just sloppy terminology, and that what I have actually been given is in fact the specificity. I did this, and got the expected result.

However, Wrath explicitly denies that this is what happened. He states that he deliberately used this "accuracy" term because it is "more powerful", because it incorporates both sensitivity and specificity. I ask again, how can a term be "more powerful" for the purpose of (real-world, in which we actually live and in which the terms we choose to use have to have general utility) test description, when it cannot be used at all in most (real-life) instances?

("The very fact that I used the term at all should have told you that I was referring to the rare and unrealistic situation of equal specificity and sensitivity!" Oh, God give me strength!)

Others reading the thread realised that "accuracy" must somehow incorporate sensitivity and specificity - and started speculating how. Arithmetical mean? This loses information, as Geni pointed out, but not only that. As Wrath pointed out, you can't just give equal weight to sensitivity and specificity when in reality more unaffected patients will be tested than affected, and so a higher rate of false positives will be disproportionately reflected in the "wrong" results. This notion is actually dragging the discussion back towards the concept of the predictive value, which is in fact the answer, not the question.

(In fact, this is the reasoning that led to the adoption of the predictive value calculation as another way to characterise test performance, with all its advantages and disadvantages which we will no doubt get to some time next week.)

So, we're left scratching our heads. Wrath has told us the "accuracy", specifically denying that he just meant to say "specificity". We need the specificity figure. How to get it? We don't know what Wrath means by "accuracy", because as you've just demonstrated, it's not a term which has a meaningful definition which can be used to characterise real-life tests.

Now we begin to suspect that the only way to get any sense out of this is to assume that Wrath must mean that both sensitivity and specificity are 99%. This is such an unlikely situation in real life (for medical tests, but remember, we are specifically dealing with medical tests, and we are trying to test those used to handling medical tests), that it hadn't really occurred to us that he could possibly mean that. But maybe he does.

Yes, he does. Since the question cannot be answered without the specificity value, Wrath must have given us the specificity value, we're told. OK, that's what I thought, but no, "accuracy" isn't specificity. But I still have to be able to deduce the specificity from this "accuracy" value. Arithmetical means run into the sands of the predictive value. So there is only one other possible explanation. He means that for this remarkable test, sensitivity and specificity are identical, therefore he can produce this all-in-one "accuracy" figure (which we've never heard used in the real world, for reasons already gone into).

Can you see that this inevitably involves an assumption? Either that Wrath meant specificity when he said accuracy, or that he meant that specificity must equal sensitivity for this very singular assay, and therefore accuracy means both sensitivity and specificity, which brings us back to where we were, accuracy here means specificity.

However you slice it, it is an assumption. Necessitated by Wrath's choice to parachute-in this "accuracy" figure, rather than simply use the standard terminology for the discipline which is applicable to all tests, not just a (tiny) subset.

Recall, Wrath kept declaring that he was posing the question in exactly the way it had been put to the medical personnel in the studies he was replicating (from memory). But when sources were finally produced, none of them quoted an "accuracy" value. In fact the popular choice was the "false positive rate" (100 - specificity), which is in effect the same information, but the term is more intuitive in its meaning than specificity itself. No problem with this, it's the right way to do it.

Why did Wrath choose to do it differently?

I think, because he knows pure statistics, but not applied statistics. He was trying to do it from memory, and didn't understand what any medical person would automatically know - that tests are usually not of equal sensitivity and specificity, and that specificity is the term you need to know. He therefore got into a huge tangle which was entirely unnecessary.

I'm sorry if I've still failed to explain it to you. Doesn't the fact that none of the source studies introduced an "accuracy" figure reveal anything, even if my explanations are inadequate?

Yes, I threw an early stone, overtly. But it was as a response to an unprovoked covert stone, Wrath's OP. I knew exactly where he was going. I'd quite like to go there and explore that place. But to set out to trap/criticise the medical community and yet not to employ the unambiguous terms that are provided for the purpose of this question, and which it turns out were employed by his sources, got my back up from the first moment. If Wrath is going to set himself up as the oracle in the medical statistics department, then for goodness sake use the correct medical statistics in the OP!

However, that wasn't my main reason for pulling the trigger on that sloppily-put and simplistic problem. My main reason was that Wrath had shown us a scenario deliberately designed to make the doctor look stupid. First he had built into the scenario a clinical examination, which led to the test request, but he did not tell us the reason for the request. This is hugely dishonest, because it is this reason that would influence the doctor's decision to accept the test result as correct. Which we're told he did.

Wrath tells us that the doctor examined the patient. Then he decided to order the test. Then he received a positive result. Which he decided was correct.

He wants us to assume that there was no special reason for the ordering of the test. That the figure to be assumed for "prevelance" in the case of this particular patient is the 0.1% figure for the population as a whole. And that the doctor therefore jumped to a wrong conclusion for no good reason.

But these are all assumptions. We can make equally (if not more) valid assumptions fron the same data.

The doctor ordered the test because of something he observed while examining the patient. When he got the positive result he was perfectly well aware that the condition had only a 0.1% incidence in the general population, but he knew that the probability of this patient having the disease was much greater than that, so the predictive value of this result in this patient was much higher than the baseline 9.02%, indeed high enough to make it a racing certainty.

The reason I say this is the more likely scenario is that Wrath couldn't just ask us the probability that the positive result was correct, oh no. He chose to tell us that the doctor decided it was correct, and ask us the probability that the doctor was wrong. Now Wrath has a very low opinion of doctors. But I don't think they're as stupid as he assumes. If part of the information I'm explicitly given is that "the doctor chose to believe that the result was correct" I feel I am entitled to use this information to reflect on the entire problem. If he made this decision, might not this be an indicator that he'd requested the test because of clinical suspicion, not as a routine?

Ceptimus, wording of these problems is of crucial importance. You mustn't tell the audience too little, or too much. Here, we were told too little, in that we had to guess at the specificity figure we needed, and we're kept in the dark as to the reason for requesting the test. And we're told too much, in that we're told what the doctor concluded. We didn't need that to be able to work out the basic maths, but once we've been told it, it introduces other possible assumptions which may affect the interpretation of how the problem should be viewed.

Again, when we look at the source material, we find that the question isn't "how likely is it that the doctor was wrong?" It's "how likely is the test to be wrong?" Much more neutral, but not Wrath's style.

And (certainly in the example quoted by Steve74, which looked like the original 1978 study) there has to be some way of indicating that we are not allowed to take signs and symptoms into consideration (explicitly stated in the 1978 question), or in fact that the patient is low-probability and we should use a low-probability in calculating predictive value.

Wrath missed that part completely.

Now is we ignore Wrath's justifications for his wording, how could it have been worded to pose the same question, but to rein in all those assumptions and keep the reader on the desired train of thought? Easy.

Last month I had to go for an insurance medical. The doctor could find nothing wrong with me, but the insurance company required that I have a blood test for a particular disease, which actually has a 0.1% incidence in the clinically heathy population. The test, which has a false positive rate of only 1%, came back positive. What is the probability that I really have the disease?

That's how you do it (and believe me, I've set this one for clinicians often enough, I know how careful you have to be to stop them jumping all over the question - but this one is bomb-proof).

Note that I explicitly identified the patient in question with the population I gave the incidence figure for. No weaselling that the prevalence isn't necessarily valid for that individual. It's the right prevalence. And by the way, no silly "accuracy", you have a number there from which you can directly derive the specificity. And I didn't tell you what the doctor thought. Why should I? You're the doctor! Come on, what do you think? I've been very careful to give you NO reason to go for a high probability that the result is right. Will you still fall for it?

And quite often, they do.

And that, actually, is the start of the class, not the end.

So, I went for Wrath. This is no more than he dishes out - he frequently tells posters they can expect no mercy from him. The reason I did it is that he was setting out to have a go at medical comprehension of statistics, but he himself had been extremely sloppy in his wording of the question. A question which is regularly used in the sort of classes I teach, and the parameters of which are well-known.

I wish he'd worded the question better. We might have had a much more constructive discussion by now. But Wrath takes no prisoners. When he himself is less than perfect, why should we refrain from retaliation?

Rolfe. Hoping to start the real discussion soon.

[Wrath of the Swarm]

They can be the same. They often are. They don't need a reason to be identical any more than they need a reason to be different.

Indeed, the term 'accuracy' cannot meaningfully be applied unless alpha and beta are equal - but that is precisely why it's more powerful. When it does apply, it says a great deal.

No assumptions need be made. We know the disease incidence in the general population. We know how frequently healthy people will test as positive, as well as how frequently sick people will test as positive, because we were told how often the test is wrong: 1% of the time.

Your assumption, Rolfe, was that the error rate must be dependent upon the possibility being examined. This is not the case. It's not even the general case. It's simply the case that applies most often in medical situations.

In your ignorance, and arrogance, you decried this as a mistake, an ambiguous statement. There's no ambiguity about it. The statement made can be valid only if the Type I and Type II errors are equally likely - and since we were discussing a hypothetical test whose properties are determined by fiat, there is no reason for any logically consistent properties we assign to it to be considered invalid.

The basic point holds even if we don't use an example with identical alpha and beta rates. People aren't able to answer the question correctly no matter what permutations are used.

The simple truth is that the example was meant to illustrate that most medical professionals do not understand basic issues of statistics and are not qualified to make judgements regarding them. The irony is that your witless babbling has proven my point better than I could ever have done.

Thank you, Rolfe.

[epepke]

Originally posted by Wrath of the Swarm
Your assumption, Rolfe, was that the error rate must be dependent upon the possibility being examined. This is not the case. It's not even the general case. It's simply the case that applies most often in medical situations.

I think you're accurate in your assessment of medical professionals. I've presented a similar conundrum to several, and it usually results in a lot of blustering and tap-dancing.

However, vets tend to be smarter than physicians anyway. For one thing, at least in the US, it's harder to get into vet school than med school. For another, vets have to deal with multiple species, none of whom can speak in the normal way. For another, they're permitted to retain much more of their humanity than docs are.

In any event, I get the impression that Rolfe is mostly expostulating on the vagueries of trying to reduce statistics to a single number. I hope you consider saving your anger for people like HopkinsMedStudent.

[Wrath of the Swarm]

I will save my anger for all those who speak against the truth, whether it be Rolfe, Hopkins, or myself.

[Badly Shaved Monkey]

WotS

I think may still have missed it if you have provided a source for the conventional use of the formal term 'accuracy' in the sense in which you have used it, which, unless I am mistaken, is a single variable that can be cited if and only if false positive and false negative rates are the same and then it is numerically equivalent to both. It still looks like a term you have made up on the fly and invoked in order to set a question that actually only required specificity to be specified.

As I have already asked, can you provided a citation to a source in which 'accuracy' defined and used in this particular sense? This would simply let you show that your usage is conventional even if it may be unfamiliar to Rolfe, me and several others here.

[MRC_Hans]

Originally posted by Rolfe
I do have more important things to do, you're right. But I'm also a teacher (though I no longer hold a university teaching post, I still teach this subject at post-graduate level).

If I'm failing to get my point across, if there is one of the audience (or it looks like more) who don't understand me, I like to try to do better, one more attempt to make the situation clear.

Rolfe. For the record, I understood it around the bottom of page 1.

If you test for something rare on a large population, then even a very small percentage of false positives will create more positives than the real positives.

This is basic in all testing. At some point the errors generated by the test surpasses the real faults, at which point it is usually pointless to continue testing.

What I STILL fail to understand is the point of this discussion. What is it's relevance to the real world?

Hans

[Brian the Snail]

Originally posted by Badly Shaved Monkey
Looking again at the Wolfram site, I don't think it is about testing at all.

http://mathworld.wolfram.com/Accuracy.html

http://mathworld.wolfram.com/Precision.html

In the realm of laboratory testing accuracy mean the closeness of the mean measured value to the true value of the test parameter measured by some gold standard or known a priori. Precision is effectively the same as reproducibility.

Accuracy as defined on the Wolfram site seems to have the same meaning as what is more commonly called "error" in physics. So if for example I make a measurement of a physical constant, and it has a value 3.0 with an error of 1%, then it means that the true value has a 95% probability of lying in the range 2.97 to 3.03 (assuming that I'm using a 2 sigma error bar). This seems to match your definition of accuracy for medical lab. testing.

WOS seems to be using "accuracy" in a different sense, corresponding in some way to probabilites of finding type I and type II errors in significance testing. However, this seems to me to have a very different meaning to the definition of accuracy contained in the Wolfram site.

[MRC_Hans]

Brian, yes that is precisely what accuracy means. What WOTS is talking about is really what I would call predictive value, which is the ability of a given measurement method to predict the distribution of an actual set of values. Of course, this could be termed the overall accuracy of a given instance.

When working with very extreme distributions as in the example he gives, this can give results that are counterintuitive.

Why he feels that this in any way justifies abusive behavior is beyond me.

Hans

[steve74]

Wrath, I'm still waiting for the apologies. You need to apologise to the people who've posted in this thread for telling a blatant lie. You also need to apologise to me for calling me a liar.

You got caught telling a blatant lie to hide your discomfort that you had no sources to back up your point. You thought you could bullsh1t the newbie and that I wouldn't check; you were wrong. Admit you got caught out and apologise.

For those who missed it, Wrath's lie:

I finally found the sources that duplicated the question (I even pointed them out, remember?).

[Brian the Snail]

Originally posted by MRC_Hans
Brian, yes that is precisely what accuracy means. What WOTS is talking about is really what I would call predictive value, which is the ability of a given measurement method to predict the distribution of an actual set of values. Of course, this could be termed the overall accuracy of a given instance.

Well, now I'm confused, since WOTS originally posted the link the Wolfram site to show that he's using the term accuracy in the correct sense mathematically. He posted this on page 6, in reply to drkitten who said "I don't think that the term "accuracy" means what you think it does in mathematics, either."

Originally posted by Wrath of the Swarm

http://mathworld.wolfram.com/Accuracy.html

Nice try. Well, it wasn't actually a very good try at all, but you get points for effort. Well, since it was a throwaway attempt at a stab in the dark, no you don't.

But the link seems to define "accuracy" differently to how WOTS defines it.

Perhaps he could clarify why there appears to be a difference.

[Prester John]

At the risk of being labelled a groupie, i find that Rolfe is talking to me in the langauge i understand. I work in a lab, always have done and have performed the tests (not that there are many yes/no answers in bacteriology but anyway) of the kind in question.

I think Rolfe nailed it earlier, when she suggested that this may be a pure statistics vs applied medical statistics argument.

Tests are very rarely done in isolation, for screening programmes, there should be a multilayered approach, with the initial test aiming to have a low false negative rate, and a confirmatory test, preferably using a different technology having a low false positive rate.

[exarch]

Originally posted by Wrath of the Swarm
The basic point holds even if we don't use an example with identical alpha and beta rates. People aren't able to answer the question correctly no matter what permutations are used.Wrong WotS, If the alpha or beta error values are 0, the overall "accuracy" as you call it, could still be seen as being 99% (with 1% of the total test population testing either false positive in the former case or false negative in the latter instead of 0.5% of both, or is that 1% of both?).

[exarch]

Originally posted by MRC_Hans
For the record, I understood it around the bottom of page 1.

If you test for something rare on a large population, then even a very small percentage of false positives will create more positives than the real positives.

This is basic in all testing. At some point the errors generated by the test surpasses the real faults, at which point it is usually pointless to continue testing.

What I STILL fail to understand is the point of this discussion. What is it's relevance to the real world?Relevance to the real world? Well, as Rolfe pointed out on page 1, there is no relevance, except that doctors aren't staticticians.

What's really going on is a lot of people getting even with Wrath of the Swarm for constantly nitpicking the slightest details in their posts. Further more, Instead of just admitting right away that his question might have been worded more clearly, he did his usual trick of trying to bluff his way out, and as such started digging himself a hole he couldn't get out of any more without losing face.

People have now caught him on a number of lies, one being his statement that he worded the problem "exactly" as it is being given to medical students (it wasn't, as shown by steve), the other being that the conventional meaning of the word "accuracy" in mathematics is "exactly" the same as the meaning he's claiming it has (also shown not to be true, by the wolfram website no less, a link he provided himself, but obviously didn't bother to read).

Right now, it's up to wrath to either concede and take a big bite of humble pie, or continue his false assertions and discredit himself for all future arguments. Let's just say we're getting pretty close to making a Larssen-list of Wrath's lies. Maybe we should call it a wrath-list? a WotS-list? :D

[Wrath of the Swarm]

I was wrong about the research - at least, I couldn't find sources that used the question in the same way, and I got overeager and thought I had.

The Wolfram definition of accuracy is perfectly consistent with the way I've used the word.

And I'm not bluffing - my argument has been correct. If you were more interested in learning the truth instead of attacking someone you don't like, you'd have realized that by now.

[steve74]

Originally posted by Wrath of the Swarm
I was wrong about the research - at least, I couldn't find sources that used the question in the same way, and I got overeager and thought I had.

The Wolfram definition of accuracy is perfectly consistent with the way I've used the word.

And I'm not bluffing - my argument has been correct. If you were more interested in learning the truth instead of attacking someone you don't like, you'd have realized that by now.

Oh come on now, Wrath, you don't expect me to take this seriously do you? Let me remind you what you wrote:

I finally found the sources that duplicated the question (I even pointed them out, remember?).

Are you seriously trying to say that you genuinely thought that you had found the correct sources and pointed them out, when you hadn't? Do you often have memories of events that didn't happen?
Overeager? Don't make me laugh. You told a blatant lie and then had the audacity to accuse me of lying. Apologise for lying on this thread, Wrath, and apologise for calling me a liar.

[Wrath of the Swarm]

I made a mistake. It happens sometimes when I read sources too quickly.

Lies are intentional attempts to deceive. This was just a random act of stupidity on my part. I shouldn't have said I found the particular question until I was sure I had.

[Wrath of the Swarm]

Originally posted by steve74
With regard to whether the information given in the question was sufficient to answer the question, I have only this to say. Yes, it was fairly easy to make an assumption about what you were getting at and give the answer, as you defined it. But, it is also true that an assumption had to be made and that while this assumption was the most likely one, it certainly wasn't the only way to interpret the question. Liar.

Your question was underspecified. This has been pointed out to you many times by many posters, all of them far more able than I to make this clear. Again, liar.

[Paul C. Anagnostopoulos]

Wrath said:
I will save my anger for all those who speak against the truth, whether it be Rolfe, Hopkins, or myself.
What you might consider is that the truth is sometimes not so obvious, and even with some give and take may not reveal itself easily.

~~ Paul

[Wrath of the Swarm]

What has become obvious is that "bandwagoning", where people take assertions about an argument and run with them regardless of whether they're true or not, is a major problem with this board.

It doesn't matter whether assumptions are necessary for the original question (they are not) or whether this question describes a grossly improbable situation (it does not) -

- people looking for errors so they can attack will seize on the suggestion of error and begin treating it like fact.

I have to say, it doesn't bode well for these boards being able to have rational responses when dealing with potential woos. When a valid argument that appears similar to previous invalid ones comes along, it's going to be as virulently attacked as the nonsense.

[geni]

Ok show mathmaticly how you reach a unique solution based on the information avaible in the orginal question.

[steve74]

Originally posted by Wrath of the Swarm

With regard to whether the information given in the question was sufficient to answer the question, I have only this to say. Yes, it was fairly easy to make an assumption about what you were getting at and give the answer, as you defined it. But, it is also true that an assumption had to be made and that while this assumption was the most likely one, it certainly wasn't the only way to interpret the question.

Liar.

Your question was underspecified. This has been pointed out to you many times by many posters, all of them far more able than I to make this clear.

[/b] Again, liar. [/B]

Wrath, time and again you have demonstrated your ignorance. But I have to say I'm still amazed that you don't even know the meaning of the word 'liar'. I made those quotes believing in their veracity. I still believe in their veracity and stand by them 100%.
So how can I be a liar?

Your conduct in this thread has consisted of blatant lies, a complete unwillingness to admit your untruths (until you were forced to do so), shoddy research (at best) and a boneheaded refusal to admit the wrongness of you argument.

You are intellectually bankrupt.

[Badly Shaved Monkey]

Originally posted by Brian the Snail


Accuracy as defined on the Wolfram site seems to have the same meaning as what is more commonly called "error" in physics. So if for example I make a measurement of a physical constant, and it has a value 3.0 with an error of 1%, then it means that the true value has a 95% probability of lying in the range 2.97 to 3.03 (assuming that I'm using a 2 sigma error bar). This seems to match your definition of accuracy for medical lab. testing.

No, I don't think the Wolfram site is using a definition of accuracy that is like thaht used in lab testing. I thought it was from it's first paragraph.

"The degree to which a given quantity is correct and free from error. For example, a quantity specified as 100 ± 1 has an (absolute) accuracy of ± 1 (meaning its true value can fall in the range 99-101), while a quantity specified as has a (relative) accuracy of (meaning its true value can fall in the range 98-102)."

I would recognise this as a definition of accuracy, but the explanation goes on to say;

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

Consider that second paragraph.

I weigh a car and find it weighs 1.295 tonnes. The accuracy by Wolfram's definition "is given by the number of significant decimal (or other) digits to the right of the decimal point in x" i.e. 3 places of decimals. If I weigh it on another set of scales that report a weight of 1,294,864.4g. This has only 1 place to the right of the dp. Does that make it less accurate? In truth it neither makes it more nor less accurate because a single number cannot convey accuracy you need a measure of spread. But to use the Wolfram definition again, is 1.295 +/- 0.002 tonnes more accurate than 1,294,864.4 +/- 0.3g.

So I'm still left confused as to what the Wolfram definition, in its full form, is used for.

(Edited for clarity)

[Wrath of the Swarm]

Oh brother.

Chance that someone will have the disease: .001
Chance that someone won't have the disease: .999

Chance that the test will give an incorrect answer in any particular circumstances: .01
Chance of correct answer: .99

Chance of getting a false positive: (chance disease-free)(chance of error) = .999(.01) = .00999

Chance of getting a true positive: (chance disease)(chance of correctness) = .001(.99) = .00099

Percentage of correct positive diagnoses: (true positives) / (true positives + false positives) = (.00099) / (.00099 + .00999) = (.00099) / (.01098) ~ .0901639

Conclusion: If the test results come back positive, there is only about a 9% chance that the patient really has the disease.

[geni]

Originally posted by Wrath of the Swarm

Chance of getting a false positive: (chance disease-free)(chance of error) = .999(.01) = .00999

Chance of getting a true positive: (chance disease)(chance of correctness) = .001(.99) = .00099


There is not enough information in the intial question to do the above without making assumptions.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
Oh brother.

Chance that someone will have the disease: .001
Chance that someone won't have the disease: .999

Chance that the test will give an incorrect answer in any particular circumstances: .01
Chance of correct answer: .99

Chance of getting a false positive: (chance disease-free)(chance of error) = .999(.01) = .00999

Chance of getting a true positive: (chance disease)(chance of correctness) = .001(.99) = .00099

Percentage of correct positive diagnoses: (true positives) / (true positives + false positives) = (.00099) / (.00099 + .00999) = (.00099) / (.01098) ~ .0901639

Conclusion: If the test results come back positive, there is only about a 9% chance that the patient really has the disease.

I don't think any of us who have posted to this thread are disagreeing with this arithmetic. But some of those who voted do not!

[Wrath of the Swarm]

Originally posted by Badly Shaved Monkey
"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.
It would be used in the context of chemistry or physics.

For example, if you were performing an equation to calculate the yield of a particular chemical equation, the answer you get cannot be more accurate than the least accurate measurement of the ingredients. If the measuring device you used to parcel out the ingredients is accurate to only one-tenth of a basic unit, the result cannot be given as a value accurate to more than one-tenth of that basic unit, even if the math gives you more digits than that.

Another example: if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand) by a physical constant with eight significant decimal places, the resulting answer must be rounded off to three decimal places. Digits after that are not considered meaningful, as the uncertainty in the measured value is greater than they are.

[MRC_Hans]

Originally posted by Badly Shaved Monkey


No, I don't think the Wolfram site is using a definition of accuracy that is like thaht used in lab testing. I thought it was from it's first paragraph.

"The degree to which a given quantity is correct and free from error. For example, a quantity specified as 100 ± 1 has an (absolute) accuracy of ± 1 (meaning its true value can fall in the range 99-101), while a quantity specified as has a (relative) accuracy of (meaning its true value can fall in the range 98-102)."

I would recognise this as a definition of accuracy, but the explanation goes on to say;

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

Consider that second paragraph.

I weigh a car and find it weighs 1.295 tonnes. The accuracy by Wolfram's definition "is given by the number of significant decimal (or other) digits to the right of the decimal point in x" i.e. 3 places of decimals. If I weigh it on another set of scales that report a weight of 1,294,864.4g. This has only 1 place to the right of the dp. Does that make it less accurate? In truth it neither makes it more nor less accurate because a single number cannot convey accuracy you need a measure of spread. But to use the Wolfram definition again, is 1.295 +/- 0.002 tonnes more accurate than 1,294,864.4 +/- 0.3g.

So I'm still left confused as to what the Wolfram definition, in its full form, is used for.

(Edited for clarity) Mmm, I'm not THAT well versed in English terminology, but I'll give it a try anyhow:

Accuracy: As was already explained earlier, the 95% confidence range of a measurement. This includes various noise in the physical system, so 100 +-1% means that if the meter reads 100, then the actual value is, with 95% certainy between 99 and 101.

Precision: This, I think, is what I would call resolution. Suppose the meter used above showed the result with four decimals: 100.0000 . In this case the extra decimals would not influence the accuracy, because it would still be +-1%. However, if the resolution was only three digits (no decimals), then we could not get +-1%, because the impresicion in the readout alone would be +-1.

Hans

[Wrath of the Swarm]

Originally posted by Badly Shaved Monkey
I don't think any of us who have posted to this thread are disagreeing with this arithmetic. But some of those who voted do not! Ah, but people are disagreeing. They assert that an assumption is being made when I use the chance that the test is wrong in any particular case (1%) for the chances that the test is wrong when its response is positive and when its negative.

The point is that, since it is possible to give the test an accuracy that's independent of a sample population, those probabilities are necessarily equal to each other and to the overall accuracy.

Rolfe and her bandwagoning partners in ignorance insist that this is an assumption and that the relevant information wasn't provided in the question. This is false.

[Brian the Snail]

Originally posted by Badly Shaved Monkey

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

Consider that second paragraph.

I weigh a car and find it weighs 1.295 tonnes. The accuracy by Wolfram's definition "is given by the number of significant decimal (or other) digits to the right of the decimal point in x" i.e. 3 places of decimals. If I weigh it on another set of scales that report a weight of 1,294,864.4g. This has only 1 place to the right of the dp. Does that make it less accurate? In truth it neither makes it more nor less accurate because a single number cannot convey accuracy you need a measure of spread. But to use the Wolfram definition again, is 1.295 +/- 0.002 tonnes more accurate than 1,294,864.4 +/- 0.3g.

So I'm still left confused as to what the Wolfram definition, in its full form, is used for.

Yes, I was rather confused by the second paragraph as well. I would define accuracy and precision in much the same way as Hans does. So, for example, if I do a measurement my device might give an answer on the digital readout of 543243546, which is very precise, but if the error (or accuracy) is 10%, then anything past the second digit is meaningless. So I'm not sure what the Wolfram page is going on about really.

And none of this is particularly relevant to this thread, I must say.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
It would be used in the context of chemistry or physics.

For example, if you were performing an equation to calculate the yield of a particular chemical equation, the answer you get cannot be more accurate than the least accurate measurement of the ingredients. If the measuring device you used to parcel out the ingredients is accurate to only one-tenth of a basic unit, the result cannot be given as a value accurate to more than one-tenth of that basic unit, even if the math gives you more digits than that.

Another example: if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand) by a physical constant with eight significant decimal places, the resulting answer must be rounded off to three decimal places. Digits after that are not considered meaningful, as the uncertainty in the measured value is greater than they are.

Ah, that's not it. What I was trying to point out that the Wolfram definition depends on using the decimal place as an absolute reference point about which toassess 'accuracy'. The point of the example I gave was to show that the position of the d.p. is arbitrary, which means that a definition of accuracy the depends on the choice of position of the d.p. cannot be generalised so it is not a very useful definition. That's why it seems to be an odd way to define it.

Wait a minute;

"if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand)"

Is the immplicit assumption that everyting has been reduced to scientific notation with only one digit to the left of the d.p?

Under that assumption it would be generalisable, so 1.346 x 10^2 tonnes has the same accuracy as 2.543 x 10^15 W or 1.654 x 10^1 N.

That, I think, is logically consistent.

[Brian the Snail]

Actually, just to add something, precision does have the meaning of "the number of significant figures" in computer programming. For example, single precision real numbers have 8 digits, while double precision numbers have 16. So I can understand that part anyway.

[Rolfe]

Again, this illustrates the difference between pure and applied (in this case medical) statistics. The Wolfram site gives very pure definitions, and doesn't discuss their use in context. In contrast, when what is required is a meaningful and standardised language to discuss and evaluate very particular contexts, things can look a lot different.

I'm not disputing the definition or arithmetic given for "precision" above, but in the particular context of a biochemistry or haematology assay, precision describes the repeatability of the results from the assay in question. It is assessed by calculating the coefficient of variation of a number of replicate measurements, CV = (SD/mean) x 100 (%). For routine assays we try for less than 5% CV, often we actually achieve around 2% in practice.

(Number of significant figures also comes in here though, because obviously the better the precision the more significant figures you can legitimately report to.)

Accuracy, the twin concept with precision, describes how well the results compare to the "true" result from a reference method. More difficult to pin down statistically, in fact some authors have suggested abandoning the numbers and just eyballing difference plots, but usually one calculates the correlation coefficient for a group of samples with a range of concentrations, generally hoping for better than 0.99 in a routine assay, with the line of best fit lying very close to the line of coincidence.

These two concepts are the gods in the biochemistry and haematology laboratories, as sensitivity and specificity are the gods in the serology laboratory with its binary assays (OK, forget titre results for now). We have to understand how to derive these parameters, and how to interpret them. We have to sit exams about it.

These concepts may be unfamiliar to those whose statistical background isn't in the precise area being covered. But they are nevertheless well defined and well studied, and are essential to allow professionals to have a meaningful conversation about assay performance.

Having an understanding of pure statistics doesn't necessarily equip one to understand specialist areas of applied statistics without at least some prior familiarisation with the methods in use and the conventions adopted. Which I'm now certain that Wrath will never understand.

Rolfe.

[Wrath of the Swarm]

Translation: "I'll talk about how complex and sophisticated my understanding of a limited and highly specific form of statistics is so that no one will pay attention to the fact that I don't understand the simple stuff."

Further translation: "The information wasn't given to me in the form I have been conditioned to expect, so obviously I'm the only one who understands statistics here."

[Wrath of the Swarm]

Originally posted by Rolfe
Having an understanding of pure statistics doesn't necessarily equip one to understand specialist areas of applied statistics without at least some prior familiarisation with the methods in use and the conventions adopted.

Um, it's actually the other way around - understanding a very limited adaptation of the field doesn't necessarily equip one to understand the field as a whole.

Not that this is relevant, because we're not discussing some esoteric point of advanced statistical analysis. This is the basic stuff, the stuff they teach in the first two weeks of introductory courses.

You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it. Without numbers labeled "specificity" and "sensitivity" you're not able to "plug and chug".

[Dancing David]

There does seem to be an issue in the way people are 'accuratly" characterising the others in this thread.

Thank goodness the Ivory Tower has a popcorn machine....

[Rolfe]

Originally posted by Wrath of the Swarm
I made a mistake. It happens sometimes when I read sources too quickly.

Lies are intentional attempts to deceive. This was just a random act of stupidity on my part. I shouldn't have said I found the particular question until I was sure I had. Yes, Wrath, you made a mistake. The mistake wasn't just reading your source too quickly, it was not referring to your source when you formulated your question. Yes, I'm sure it was a random act of stupidity. You shouldn't have written the question until you'd checked that the wording really was identical to that used in whatever study you thought you were referencing.

The only "sample question" we've actually seen quoted in this thread from a published study was the 1978 one referenced by Steve74.“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?”"False positive rate". Nothing about "accuracy". Just like the other studies on the same subject, you'll find. Not one uses the term accuracy in the question.

Wrath didn't refer to the source study when he formulated his question. So he didn't notice that "accuracy" wasn't the term presented. Not fully understanding that his concept of "accuracy" was inapplicable to a binary assay in which sensitivity and specificity tend to vary inversely and will only be equivalent by very occasional coincidence, he loosely stated "accuracy" rather than the required value, specificity.

And people started to jump all over him.

It would have been quite easy to admit the mistake at the time, it wasn't a capital offence, and we could have moved on to the interesting discussion, which (as many people seem to realise) is the concept of predictive value, its variability with "prevalence", and most important of all, how to choose the most applicable figure for prevalence to get the true predictive value for an individual patient's result.

But Wrath won't admit it. He's resorted to declaring that since this silly "accuracy" concept is obviously quite meaningless unless sensitivity and specificity are equal, then he was being very clever in giving us even more information than we needed! By his choice of this term, he was telling us not only what we needed to know, that the specificity was 99%, but something we didn't give a damn about for the pupose of the calculation, that is that the sensitivity was also 99%.

Yeah, right.

What he was actually telling us was that in spite of the stated purpose of this thread being to allow Wrath to attack doctors' comprehension of medical statistics, Wrath himself has little familiarity with medical statistics. Otherwise he would have known perfectly well that the term "accuracy" is never used in this context, and why, and he would have understood that the figure he should supply was the specificity.

Wrath, people in glass houses shouldn't throw stones. Whether you realise it or not, you are currently standing out in the open surrounded by many small shards of broken glass. And it's starting to rain.

Rolfe.

[Hellbound]

Originally posted by Wrath of the Swarm

Um, it's actually the other way around - understanding a very limited adaptation of the field doesn't necessarily equip one to understand the field as a whole.
Yes, and of course we should always use the general when discussing the specific? Next time I write a medical report, I'll use organism instead of patient, should be obvious what I'm talking about, right?

Not that this is relevant, because we're not discussing some esoteric point of advanced statistical analysis. This is the basic stuff, the stuff they teach in the first two weeks of introductory courses.
Yes, the basic stuff. Not the branch of statistics developed to specifically work with medical testing, with procedures and terminology designed to prevent confusion and allow for precise discussion. It's like using pliers instead of a wrench to loosen that nut.

You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it. Without numbers labeled "specificity" and "sensitivity" you're not able to "plug and chug".
This phrase, "not able", I do not think it means what you think it means.

I'll try this slowly, in large type:

ROLFE DID GET THE RIGHT ANSWER, SO YOUR INSULT HERE IS NOT ONLY RUDE AND CHILDISH, BUT ALSO AN OUTRIGHT LIE WITH NO SUPPORTING EVIDENCE, AND COUNTER-EVIDENCE ON THE FIRST PAGE OF THE THREAD.

That better?

Okay. Now that that's clear.

She is complaining that you are providing a question targetted to a group of people who's training is in a specific branch of statistics, yet providing that data within a framework that they are not trained in. I do computer work. If someone came up and started asking me programming questions, but used terminology from information theory rather than computer programming, that's rather silly, isn't it? And, since you claim to know all about medical testing and the terms used in it, one can only think that you used different terms intentionally.

You're trying to test a mechanic on engineering, and it just doesn't work that way.

I have never in my life seen such an inflated ego, and for so little reason.

[Wrath of the Swarm]

Translation: "Yes, yes, that's it! The reason I insisted that not enough information was provided was because there was no source for the question! The question didn't have enough information to be answered because there was no source!"

No, I'm not testing a mechanic on engineering. Mechanics are not generally expected to know engineering.

What I'm doing is showing pictures of basic tools to the mechanic and asking "what is this?"

Remarkably, the mechanic somehow managed to identify the screwdriver as a screwdriver, even though the size of the head wasn't specified. Truly, the mechanic is a god among mere mortals for the astounding feat of intuitive leaps.

[Hellbound]

Originally posted by Wrath of the Swarm
What I'm doing is showing pictures of basic tools to the mechanic and asking "what is this?"

Actually, what you're doing is showing the mechanic a wrench. To which, the mechanic replies "That's a crescent wrench", whereupon you proceed to claim the mechanic is an idiot because he obviously can't tell what a wrench is, since he gave you the specific answer instead of a general one. Even that's not a good analogy, but closer to what's happening. I'm sorry, but I just don't see you winning this one. You have yet to show any source whatsoever that presents medical test data using the term accuracy; the term is not used in medical test statistics. Because you used a different term, you're throwing a fit about it just so you don't have to say "My bad, I meant specificity."

You say a mechanic is not expected to know engineering; likewise, a doctor is not expected to know general statistics, only medical ones. Since the term accuracy in reference to tests is, essentially, menaingless (because almost no test ever has a specificity equal to its sensitivity), it is not a part of medical statistics int he sense you use it. You got the terminology wrong, plain and simple.

I'm done here. I get enough practice with these types of arguments with my 2 year old.

[Wrath of the Swarm]

There's no such thing as "medical statistics".

If I had insulted Rolfe for not knowing what 'alpha and beta' referred to, that would be a valid complaint.

But she complained that more specific figures weren't given when they weren't necessary, and insisted that insufficient information had been provided.

Lots of tests have equal alpha and beta values. Medicine tends to develop ones that don't because those tend to be more useful in medical contexts.

Circles are special cases of ellipses. If I had defined a circle according to a radius, and a geometer complained that I hadn't specified two axes and that he couldn't find the area of the shape without knowing the lengths of both axes, that would be the equivalent of what Rolfe did.

Rolfe was wrong. Just admit it and move on.

[Rolfe]

Originally posted by Wrath of the Swarm
There's no such thing as "medical statistics".Medical Statistics (http://www.shef.ac.uk/scharr/spss/).

More Medical Statistics (http://www.medstatsaag.com/).

Even More Medical Statistics (http://www.medstat.medfac.leidenuniv.nl/MS/INDEX.HTML).The goal of Medical Statistics is the application and development of statistical models and designs in medical research. It plays an active collaborative role in the design and analysis of clinical and basic research in the Leiden Medical School. The development of statistical models concerns extensions of established statistical tools like logistic regression, survival analysis and repeated measures models, as well as innovative research on validation of prognostic statistical model and models for complex data such as pedigree data, multi-state follow-up data and random effects models for repeated measurements.

Medical statistics has made important contributions to the field of statistical models for medical data....Certainly, this discipline covers a great deal more than the statistics used to characterise assays in the clinical laboratory, but its existence as a discipline is not really possible to deny.

I could go on all night (http://www.univie.ac.at/medstat/). But why bother.

Rolfe.

[Prester John]

There's no such thing as "medical statistics".

Guess i'd better go shut down the medical stats department then.


edited to add: Hey thats the book i use for stats Rolfe!

[Donks]

Originally posted by Rolfe

More Medical Statistics (http://www.medstatsaag.com/).

Rolfe. [/B]

Just out of curiosity, is that your book?

[pgwenthold]

Originally posted by Wrath of the Swarm
There's no such thing as "medical statistics".

...

Rolfe was wrong. Just admit it and move on.


Wow. I don't think I have ever seen anyone's credibility so completely and utterly demolished as what has occured in the last three posts.

[steve74]

You are, of course, making the assumption that Wrath had some credibility to demolish.

[Rolfe]

Originally posted by Wrath of the Swarm
Translation: "Yes, yes, that's it! The reason I insisted that not enough information was provided was because there was no source for the question! The question didn't have enough information to be answered because there was no source!"Translation: "Yes, that's it! The reason I said accuracy when I meant precision is that I wanted to give these poor dim medical types even more information than they needed to solve the question!"

Wrath, nobody in this discussion had the slightest problem working out your trivial little sum. Your information was sloppily provided, but it was Department of We Know What He Meant.

News flash. I have no difficulty at all with your problem. I didn't even have any difficulty figuring out what you meant to say, even though you didn't say it. I can make that problem sit up and beg, jump through hoops and lie down and die for England. I can make it do so much more than you even imagine it can do, thanks to the wiggle-room left by the other sloppy formulation of the wording.

What I can't do is figure out for a moment why anyone in his right mind, when setting a question specifically aimed at medical professionals, and which was based on a template (nay, "identically worded" to the original study example to start with!) which is widely used in medical education, should deliberately decide to eschew the accepted defined terms of the discipline, adhered to by all other examples of the question we've seen, and instead introduce a term never before encountered in this context.Originally posted by Wrath of the Swarm
"The information wasn't given to me in the form I have been conditioned to expect, ....Originally posted by Wrath of the Swarm
You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it. Why not?

Was it to make it easier? I don't think so. Was it to make it harder? It certainly succeeded in muddying the issue, but again I don't think so. Was it a pathetic attempt to look clever by showing that he knew words not used by the other authors? Probably not.

Anybody at all think it was deliberate? Anybody at all think Wrath sat there, composing his OP, saying to himself, well the question only requires that I state the precision, but I'll make it more interesting and call it accuracy instead, so if they're on the same wavelength as Wrath the Genius, they'll know that not only is the specificity 99%, but also that the sensitivity (which they don't need to know, but never mind, I'm a generous fellow) is also 99%?

And if this was indeed the case, anybody at all think it's either clever or equitable to ignore the terms that would be instantly recognised and understood by the very audience the question was aimed at, and substitute a term which at best was a conundrum in itself?

It certainly wasn't a good way to get the discussion moving in a constructive direction, as I think we can all see.

Wrath, you know perfectly well that you didn't even stop to think what you were saying when you typed that question. None of these unconvincing intellectual back-flips even entered your head. You simply typed the word you were familiar with, rather than the more exact term required, because you hadn't thought it through that assuming equal sensitivity and specificity wasn't something anyone would ever do in this context.

And now you're trying to pretend you meant it all along. Wrath, you're making a prat of yourself.

Rolfe.

[Rolfe]

Originally posted by Donks
Just out of curiosity, is that your book? No! I just Googled on "medical statistics", and clicked on the first couple of the 69,300 links that came up. About this subject that doesn't exist

I had no idea Prester John was familiar with the book, I don't know it at all.

Edited to add: It's the same publisher as my book though.

(I'm not a medical statistician, it's just that the discipline of medical laboratory science requires a particularly in-depth understanding of a small area of medical statistics, that applicable to the characterisation of diagnostic tests. Which, sadly for Wrath, is exactly the area he chose for his attempt to demonstrate how stupid medical professionals are.)

Rolfe.

[drkitten]

Originally posted by Rolfe

News flash. I have no difficulty at all with your problem. I didn't even have any difficulty figuring out what you meant to say, even though you didn't say it. I can make that problem sit up and beg, jump through hoops and lie down and die for England. I can make it do so much more than you even imagine it can do, thanks to the wiggle-room left by the other sloppy formulation of the wording.



Really? I'd be interested to see what else you can do with it. I've always had a sneaky admiration for the way a master statistician can make numbers confess by threatening to torture their assumptions-in-law.

(This isn't a challenge or an attack, just a genuine expression of interest if there's anything really twisted you can do with the problem as stated. In the same vein, I might appreciate Blackbeard's cutlery proficiency.)

[Rolfe]

Originally posted by drkitten
Really? I'd be interested to see what else you can do with it. That's what I wanted to do from the beginning, and started to do on the first page. Unfortunately Wrath's inability to fess up to a minor error in terminology has had the entire discussion on hold for about ten pages while he looks for some sort of fig-leaf to cover his exposed ego.

We may need a new thread. I certainly find it an interesting topic. Especially that in the end it goes full circle and demonstrates that the intuitive assumptions medical personnel make when you don't mess with their minds aren't so far off base, but that when half-assed statisticians start using questions like Wrath's (even if they are better-formulated) to "educate" them, there is a serious danger of substituting reasonable instinctive assumptions with rote-learned counterintuitive reasoning which isn't valid beyond the very limited confines of the original question.

Yes, that's more interesting than getting Wrath to admit he made a simple mistake, but sometimes you have to cross the railway line before you can get to the meadow.

Rolfe.

[Wrath of the Swarm]

But I did say it, and every time you've claimed I didn't is an outright lie. The value I gave you determines both alpha and beta values. No more information was needed.

Think otherwise? Go ahead - show us how you can twist the supplied information.

[Rolfe]

:hb:

Rolfe.

[Deetee]

WotS does have a valid point, its just a pity he made it using poor choice of terminology for his question.

Most doctors would get confused and give the wrong answer. Now in a real-life situation, they might well not make the same error, because there are other factors to consider, and errors may become quite obvious.

But some doctors would muss up.

[Wrath of the Swarm]

It wasn't even a poor choice of terminology. It was a perfectly accurate and straightforward choice of terminology that wasn't what Rolfe had come to expect.

And since we've established that my question really wasn't the same as the one used in the research I found citations for, you still haven't explained why the doctors did so poorly when the question was posed differently.

Well, Rolfe? Are you going to demonstrate how you can twist the statistics without contradicting the facts I'd established?

I established that the test has an accuracy of 99% - defined independently of a sample population - and gave the population incidence.

Go on. Twist them.

[pgwenthold]

Originally posted by Deetee
WotS does have a valid point, its just a pity he made it using poor choice of terminology for his question.

Most doctors would get confused and give the wrong answer. Now in a real-life situation, they might well not make the same error, because there are other factors to consider, and errors may become quite obvious.

But some doctors would muss up.

Hey we know some doctors muss up, because they did muss up (if we believe WotS).

But why? Because they are incompetent? Or because they are making different assumptions about the question as a result of their real-life experience?

Rolfe has shown that real doctors will not necessarily share the assumptions of the question writer, and as such we do not know how much of the mussing is due to doctors lacking cognative skills and how much is due to doctors who apply their experience bias into the assumptions behind the question.

[Wrath of the Swarm]

Most of it seems to be a lack of cognitive skills. When the question is totally rephrased so that the frequencies of all conditions are given, doctors get it right only about 45% of the time.

I think that was mentioned in one of my linked sources... although now I'm not certain I know what they say myself.

Ah, here we are:

And finally, here's the problem on which doctors fare best of all, with 46% - nearly half - arriving at the correct answer:

100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?Studies of clinical reasoning show that most doctors carry out the mental operation of replacing the original 1% probability with the 80% probability that a woman with cancer would get a positive mammography. Similarly, on the pearl-egg problem, most respondents unfamiliar with Bayesian reasoning would probably respond that the probability a blue egg contains a pearl is 30%, or perhaps 20% (the 30% chance of a true positive minus the 10% chance of a false positive). Even if this mental operation seems like a good idea at the time, it makes no sense in terms of the question asked. It's like the experiment in which you ask a second-grader: "If eighteen people get on a bus, and then seven more people get on the bus, how old is the bus driver?" Many second-graders will respond: "Twenty-five." They understand when they're being prompted to carry out a particular mental procedure, but they haven't quite connected the procedure to reality. Similarly, to find the probability that a woman with a positive mammography has breast cancer, it makes no sense whatsoever to replace the original probability that the woman has cancer with the probability that a woman with breast cancer gets a positive mammography. Neither can you subtract the probability of a false positive from the probability of the true positive. These operations are as wildly irrelevant as adding the number of people on the bus to find the age of the bus driver.

[Rolfe]

Originally posted by Wrath of the Swarm
It wasn't even a poor choice of terminology. It was a perfectly accurate and straightforward choice of terminology....

I established that the test has an accuracy of 99% - defined independently of a sample population - and gave the population incidence.

Go on. Twist them. Originally posted by Wrath of the Swarm
But I didn't mean "specificity", I meant accuracy. It's a more powerful concept with strict standards of application.Originally posted by Prester John
Sensitivity - proportion of individuals with the disease who are correctly identified by the test.

Specificity - proportion of individuals without the disease who are correctly identified by the test.

Accuracy is not a term used for describing diagnostic tests. You know, eleven pages, and Prester John nailed it in three lines on page five.

Since "accuracy" is not a term used to describe binary diagnostic tests, one has to assume what-the-hell Wrath means by "accuracy".

This involves assuming that he must have given us the information required to solve the problem. If one assumes that, it's easy. One needs a figure for specificity, one has been given a stray figure of 99%, therefore use that. As there's no way to derive anything other than 99% from the figure of 99%. For the rest of the sums, see Wrath's repeated rationalisations.

However, if we do not assume that we have been given all the information we need, and given that it's obvious Wrath was operating by the seat of his pants and on a poorly-recalled memory of another question while he was formulating his, this is not an unreasonable assumption, we can really do anything we like with it.

Geni went there already.

Rolfe.

[Dragon]

100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?
Wrath - now if you'd posted that question as your OP and then (after the vote) expressed your concern that less than half of the doctors got it right we could have had the debate I assume you wanted.

PS does "alpha"="beta" in the above???? :p

[Rolfe]

Originally posted by Wrath of the Swarm
Most of it seems to be a lack of cognitive skills. Looks as if it takes one to know one.

One little tiny nudge from DeeTee, and Wrath is off where he wanted to go. Posting all the studies he's been saving up about how stupid doctors are. Which was the whole point of him starting the thread.

Maybe Wrath should take the beam of "lack of cognitive skills" out of his own eye before starting on the mote he presumes is present in those of the medical profession.

Rolfe.

[Wrath of the Swarm]

Prester John does not seem to have posted on page five.

Accuracy most certainly is a term used to describe binary tests. It seems that your exhaustive educational expertise just didn't cover that part - perhaps you skipped a chapter? Or twelve?

No, Dragon. Alpha (chance of incorrectly assigning positive to a negative) is about .096, while beta (chance of incorrectly assigning negative to a positive) is about .200.

[Rolfe]

Originally posted by Dragon
Wrath - now if you'd posted that question as your OP and then (after the vote) expressed your concern that less than half of the doctors got it right we could have had the debate I assume you wanted.

PS does "alpha"="beta" in the above???? :p Exactly.

[Probably erroneous inference deleted - I read the question too quickly.]

Rolfe.

[Wrath of the Swarm]

Since the majority of people who take the test fail it, no matter what group they're in (except possibly mathematicians and cognitive psychologists), the plank seems to be in everyone's eye.

Here, Rolfe. Maybe I can help you with that plank once I remove this tiny, tiny splinter in my own.

On the other hand, maybe it's not so small after all, because it seems to have prevented me from seeing where you posted your twisting of the statistics.

C'mon, Rolfe. I wanna see them beg for God and Country like you promised.

[edit] Oh, and it's not one value that's needed. If the test can't be described with an accuracy score, we need to know both how many negatives score positive and how many positives score negative.

[Wrath of the Swarm]

Drat. And to think that I didn't save that post for posterity before I knocked Rolfe's plank out for her.

[Rolfe]

Originally posted by pgwenthold
Wow. I don't think I have ever seen anyone's credibility so completely and utterly demolished as what has occured in the last three posts. Originally posted by steve74
You are, of course, making the assumption that Wrath had some credibility to demolish.Ah, how quickly he forgets.

Rolfe.

[Wrath of the Swarm]

The statistics used in medicine isn't somehow magically different from normal statistics. It's all the same math.

So Rolfe's claims that understanding statistics doesn't prepare a person to understand medical statistics are garbage.

Where's that twisting, Rolfe?

[Wrath of the Swarm]

Again:

Originally posted by Rolfe
Having an understanding of pure statistics doesn't necessarily equip one to understand specialist areas of applied statistics without at least some prior familiarisation with the methods in use and the conventions adopted. Explain to me how it is that a solid background in "medical statistics" leaves a person without an understanding of what the accuracy of a measurement is?

And for that matter, explain to me why you brought up precision in the material that was above that quoted section. That wouldn't apply at all unless we'd performed the test more than once on a particular patient.

[Rolfe]

Originally posted by Wrath of the Swarm
Oh, and it's not one value that's needed. If the test can't be described with an accuracy score, we need to know both how many negatives score positive and how many positives score negative. How many values did you need to derive that artificial "accuracy" score? About four, wasn't it? (And very carefully contrived, or you'd have had no score at all.) But you wrapped them all together and called it accuracy.

In the same way, specificity is defined by calculating:
true negatives / (true negatives + false positives)
but is nevertheless presented as a single descriptive term. Or you can do it by expressing it as false positive rate, it comes to the same thing.

Whichever way you slice it, you only need the specificity to do the problem Wrath originally set. Wrath however used a term implying that he'd factored in the sensitivity too. Without specifying in what way this was achieved. He added unnecessary information to the necessary information, in such a way that it was no longer clear what the necessary information was.

Unless you assumed that it was all a Grand Wrath Plan, and he couldn't possibly have posted a problem without including all the necessary information in it.

Wrath is arrogant, and an idiot. The two together are an unedifying combination. I'm off to do some more work now (you know, these lab tests I make a living from providing, but I don't have a clue how to handle the statistics that describe them....). We'll see what heights his insults have reached when I get back.

"Whore" again? Oh no, that was too good for me, I forgot. Oh well, I'm sure he'll think of something.

Rolfe.

[Wrath of the Swarm]

We don't need any values to calculate that score. It's a hypothetical test - we can define its accuracy to be whatever we like. Or, we can make it so that it doesn't have a true accuracy but has an alpha and beta value.

We'll wait and see how long it takes you to twist the provided statistical information from the first post, Rolfe. I'm sure you'll wow us all.

[Rolfe]

Originally posted by Wrath of the Swarm
explain to me why you brought up precision in the material that was above that quoted section. That wouldn't apply at all unless we'd performed the test more than once on a particular patient. Of course it wouldn't. And it didn't. Go back and read the post properly. You know, like it's a crime to read a post too fast and miss something the first time....

I was talking about something else.

Rolfe.

[Wrath of the Swarm]

Originally posted by Rolfe
Whichever way you slice it, you only need the specificity to do the problem Wrath originally set. Wrath however used a term implying that he'd factored in the sensitivity too. Without specifying in what way this was achieved. He added unnecessary information to the necessary information, in such a way that it was no longer clear what the necessary information was.

Nope. We need to know two points:

1) How many people receive false positives, and
2) How many people receive false negatives.

Some of the people who are actually positive will receive negative results. Therefore, the total population of positive-result people isn't

false positives + everyone who is actually positive

but

false positives + true positives

where (true positives) = (people who are positive)beta.

[Wrath of the Swarm]

Rolfe defines specificity:

"In the same way, specificity is defined by calculating:
true negatives / (true negatives + false positives)"

True negatives are people who don't have the disease and test negative. False positive are people who don't have the disease and test positive.

true negatives + false positives = total people who don't have the disease

That value has nothing whatsoever to do with the question I asked, which was: given that a person tests positive, what are the chances they actually have the disease?

I think Rolfe screwed up her definition of specificity.

[Drooper]

Wow,

I've never seen anybody get so hot and bothered about a pretty simple application of Bayes' Theorem.

[yersinia29]

Wrath you dumbass, specificity = TN/(TN+FP)

Rolfe was right, you are an idiot.

[Wrath of the Swarm]

Nope.

True negatives / real negatives gives the chance that the test will correctly identify a negative as a negative.

That doesn't help us determine whether an identified positive is actually a positive or not.

[yersinia29]

Originally posted by Wrath of the Swarm
Nope.

True negatives / real negatives gives the chance that the test will correctly identify a negative as a negative.

That doesn't help us determine whether an identified positive is actually a positive or not.

You absolute fool.

http://www.childrens-mercy.org/stats/definitions/specificity.htm

The specificity of a test is the probability that the test will be negative among patients who do not have the disease. Specificity is sometimes abbreviated Sp. The formula for specificity is

Sp = TN / (TN + FP)

[Wrath of the Swarm]

Oh yes, I quite agree.

The problem is that Rolfe has claimed we need to know the specificity in order to answer my question. And since that value you mentioned does not permit us to do so... well, Rolfe must have chosen the wrong specificity, yes?

[drkitten]

Originally posted by Wrath of the Swarm
Nope.



Wrath, what are you babbling about, and whom are you trying to contradict? As far as I can tell, Rolfe posted a correct definition of specificity. You contradicted her and then (re)posted a correct definition of specificity. Yersinia pointed out (correctly) that your definition and Rolfe's were equivalent, and now you contradict Yersinia and re-post a definition equivalent to Rolfe's original.

I can't keep the players apart without a scorecard at this point. With whose definition of specificity do you disagree, and what do you think their definition is?

And (for I believe the third time) what was your original point that you were trying to make before this degenerated into playground-bullies-teach-stats-101?

[Wrath of the Swarm]

Anybody at all think it was deliberate? Anybody at all think Wrath sat there, composing his OP, saying to himself, well the question only requires that I state the precision, but I'll make it more interesting and call it accuracy instead, so if they're on the same wavelength as Wrath the Genius, they'll know that not only is the specificity 99%, but also that the sensitivity (which they don't need to know, but never mind, I'm a generous fellow) is also 99%?

The question does not require that I state the precision at all.

Why do we not need to know the sensitivity, again?

[Wrath of the Swarm]

drkitten:

Rolfe has claimed that we need to know the specificity, but not the sensitivity, in order to solve the original problem I gave.

However, specificity (as the term seems to be defined according to her statements and those of yersinia) is just the proportion of actual healthy people who will be given negative results.

With that value, we can also determine the proportion of healthy people who will be given positive results.

But we still can't answer the question unless we know how likely it is that sick people will get positive results. Without that value, we can't determine how likely it is that a person who tests positive also has the condition.

[yersinia29]

Originally posted by Wrath of the Swarm
Oh yes, I quite agree.

The problem is that Rolfe has claimed we need to know the specificity in order to answer my question. And since that value you mentioned does not permit us to do so... well, Rolfe must have chosen the wrong specificity, yes?

There are 4 variables, TP, FN, FP, and TN

If you assume "accuracy" = sensitivity thats enough to calculate TP. sensitivity = TP/(TP + FN), TP + FN = disease positives

If thats ALL you know, then you cant solve the problem.

The question you asked is positive predictive value. PPV = TP/(TP + FP). With knowledge of JUST sensitivity, you know TP but you dont know FP. Knowledge of FP is required to get the PPV.

If on the other hand, you assume "accuracy" = sensitivity AND specificity, then you have enough info to get FP and thus calculate PPV.

So yes, you MUST know the sensitivity AND the specificity to answer the question.

[Wrath of the Swarm]

Consider:

What if no sick person will test positive with this test? (Unrealistic, but conceivable.)

Then no matter what the chance is that a healthy person will test positive (and no matter what the chance is that a healthy person will test negative), a person who tests positive has NO chance of actually having the disease.

Rolfe just claimed that we need to know the specificity, but the sensitivity is unnecessary information.

Hmmm....

[Rolfe]

Originally posted by yersinia29
Wrath you dumbass, specificity = TN/(TN+FP)

Rolfe was right, you are an idiot. Yes, I was right about that. But I've noticed a different erroneous statement I made several times.

You do need a sensitivity value to get the absolutely spot-on positive predictive value. The sensitivity has relatively little influence on the figure compared to the specificity, but it does affect it.

This in fact means that Geni was more right than I was at the start. His assumption is the one you have to make, otherwise it's too simplistic.

Still the same problem, that sensitivity and specificity are independent (or inverse) variables, and whether you need one or both, you still have to make an assumption from the question.

I've been oversimplifying. In this I was wrong. However, the problem of the need to make the assumption before you can do the sums is still there I'm afraid.

Edited to add: I see others were realising this as I was checking my arithmetic.

What Wrath was asking originally was the positive predictive value, which is defined as
True positives / (True positives + False positives)

Rolfe.

[Wrath of the Swarm]

Originally posted by Rolfe
Whichever way you slice it, you only need the specificity to do the problem Wrath originally set. yersinia, you're contradicting an EXPERT! You're obviously an idiot!

[Wrath of the Swarm]

We've been over this before, Rolfe.

If the test can be said to have an accuracy independent of a sample population, then we can derive both the false positive and false negatives rates from that value.

No assumptions are necessary.

[Wrath of the Swarm]

Originally posted by Rolfe
Still the same problem, that sensitivity and specificity are independent (or inverse) variables, and whether you need one or both, you still have to make an assumption from the question. Independent variables cannot be inverse, Rolfe. 'Inverse' refers to a specific relationship between the two variables, and 'independent' means the variables are not related at all.

As it happens, the variables you mentioned are neither inverse nor independent.

Oh yeah, you're an expert, all right. Don't you feel foolish for contradicting this great genius, yersinia?

[yersinia29]

Originally posted by Wrath of the Swarm
We've been over this before, Rolfe.

If the test can be said to have an accuracy independent of a sample population, then we can derive both the false positive and false negatives rates from that value.

No assumptions are necessary.

Wrong.

You assume that "accuracy" = both sensitivity AND specificity.

Most medical tests do NOT have the same sensitivity and specificities, therefore the term "accuracy" is not used very often.

[Rolfe]

Originally posted by Wrath of the Swarm
If the test can be said to have an accuracy independent of a sample population,But you cannot have your definition of accuracy unless you assume that the same percentage of unaffected people test positive as the percentage of affected people who test negative.

In no other case is there a definition possible.

It's still the same assumption.

Rolfe.

[drkitten]

Originally posted by Wrath of the Swarm


If the test can be said to have an accuracy independent of a sample population, then we can derive both the false positive and false negatives rates from that value.



I believe that at this point either of you can claim a draw under the standard rules of Chess, viz "Repetition of Position."

[Rolfe]

Originally posted by Wrath of the Swarm
Independent variables cannot be inverse, Rolfe. 'Inverse' refers to a specific relationship between the two variables, and 'independent' means the variables are not related at all.I said independent OR inverse. There is a tendency for an inverse relationship in many cases.

The only thing left is a direct relationship, which is what they don't have. Which is why the idea of assuming that they are the same is so irrelevant.

Rolfe.

[Wrath of the Swarm]

Ah, but she's repeating an incorrect position.

Since one aspect of the test that was ESTABLISHED is that it has a general accuracy, we know that the two rates are equal.

We don't need to assume this. It follows from a given in the problem.

The point of the problem holds even if this is not a given. If the two rates are different, the seemingly counter-intuitive result still holds, barring truly extraordinary values.

[yersinia29]

Wrath,

In case you missed it, Rolfe already admitted he was wrong about that. I give him/her credit for that.

You on the other hand, despite direct contradiction of your claims, likes to play a shell game and try to weasel your way out of your idiocy.

I can respect somebody who steps up to the plate when they realize an error has been made. I have no tolerance for idiots like yourself who refuse under any conditions to do the same.

[Wrath of the Swarm]

You mean like the way I didn't admit that I'd made an error in thinking that the sources I linked to contained the same exact question as the one I asked?

Gee, I guess I *am* arrogant for not admitting that, admitting it was stupid, and apologizing. I'm a terrible, terrible person, yep I am!

[Wrath of the Swarm]

Originally posted by Rolfe
I said independent OR inverse. There is a tendency for an inverse relationship in many cases.

The only thing left is a direct relationship, which is what they don't have. Which is why the idea of assuming that they are the same is so irrelevant. NO!

There are not only three possible relationships. Alpha tends to increase as beta goes down, and beta tends to increase as alpha goes down, but there're other factors involved. The relationship is not linear - therefore it is not an inverse relationship.

Nor are they independent. The value of one is indeed influenced by the value of the other - it's just not determined.

It is NEITHER independent NOR inverse.

[Rolfe]

Oh, and PS. I just noticed that I wsa right earlier when I thought I spotted a mistake in BillyJoe's definitions. There is one.

He correctly defines the positive and negative predictive values. If you do the sums he says, those are the values you obtain.

However, he has accidentally described the PPV as meaning the same as specificity, and the NPV as meaning the sensitivity, when in fact the PPV is the percentage of all positive results that are correct and the NPV is the percentage of all negative results that are correct. (Both highly dependent on the composition of the population being tested of course).

I saw something wrong, but I only checked that the defined terms were correctly calculated, not that the verbal descriptions were correct.

End of digression, I just thought I'd mention that one while we're at it. It doesn't affect the validity of his calculations at all.

Rolfe.

[Rolfe]

Originally posted by Wrath of the Swarm
The relationship is not linear - therefore it is not an inverse relationship.

Nor are they independent. The value of one is indeed influenced by the value of the other - it's just not determined.

It is NEITHER independent NOR inverse. Here, hang on, I think we are agreeing on this one, no need to shout.

The relationship is not inevitably linear, so yes, it's not correct to describe it as "inverse", baldly, just like that.

But it's not independent either. One does influence the other, in an inverse direction.

This is what I was trying to convey when I said independent (or inverse). Some tests more independent, others with more of an inverse relationship. So we seem to mean the same thing.

Consider an ELISA. You have to decide where your absorbance (OD) cutoff will go. Higher than the cutoff is positive (usually, unless you have a descending reaction which is unusual), lower is negative. You fiddle with this to try to optimise the test. Raise the cutoff and you get fewer false positives - but at the price of more false negatives. Lower the cutoff and the opposite applies.

I think this one is fairly linear actually.

Or a Western Blot, where you are recognising characteristic bands in the gel. Require only two matching bands to call positive, and you'll get too many false positives. Require all four, and you'll get too many false negatives. Settle on three for optimum performance.

This isn't linear, but there's still a qualitative inverse relationship.

And some tests have relatively little relationship.

But in no case is the relationship direct.

For this "accuracy" invention to be valid, you'd need to have tests that vary in exact direct relationship, and the thing is, they don't.

Rolfe.

[steve74]

Originally posted by Wrath of the Swarm
You mean like the way I didn't admit that I'd made an error in thinking that the sources I linked to contained the same exact question as the one I asked?

Gee, I guess I *am* arrogant for not admitting that, admitting it was stupid, and apologizing. I'm a terrible, terrible person, yep I am!

Wrath, you only admitted that when I made your position untenable by pointing out the blatant falsehoods in your posts. You would have continued to peddle those falsehoods had I not called you on them. In fact, the first time I called you on them you lied to try and cover yourself. Here, for those who missed it (and let's face it there is a lot to get through here) is Wrath's lie:

I finally found the sources that duplicated the question (I even pointed them out, remember?).

Only when I called you on this did you back down and admit that you had no sources to back up your question, even though you had earlier claimed:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Wrath, you have lied in this thread. That is plain for all to see. You have accused me of lying, which is false. You owe me an apology and you owe an apology to the people who have posted in this thread for trying to hoodwink them with your lies.

[Wrath of the Swarm]

Originally posted by Rolfe
The relationship is not inevitably linear, so yes, it's not correct to describe it as "inverse", baldly, just like that.

But it's not independent either. One does influence the other, in an inverse direction.

This is what I was trying to convey when I said independent (or inverse). Some tests more independent, others with more of an inverse relationship. So we seem to mean the same thing. The preceding statements, considering in a mathematical light, are utter garbage. They mean nothing.

Two variables cannot be "more independent": either they are, or they aren't. Likewise, they are either linear, or they aren't.

There is a tremendous difference between an inverse relationship and an inverse proportion. Unfortunately, as you never learned either of these concepts (passing by rudimentary mathematics on your way to learning "medical statistics", perhaps?) you have no idea that utter and complete rubbish is spilling out of your mouth.

[Wrath of the Swarm]

In response to steve:

Yes, I was wrong. I admitted this as soon as I realized it was the case. I was sloppy and stupid, and didn't read the sources I'd found carefully enough. They contained much of the information originally presented to me years ago in college courses, but not the one piece of information I was referring to: the formulation of my question.

However, you have made further claims which are mistaken. Furthermore, even when I pointed out to you that they were mistaken, you continued to make them. When you pointed out my error to me, I admitted I'd made a mistake. You lied and repeated the falsehoods.

I am merely a fool. You are a fool and a liar.

[Wrath of the Swarm]

Originally posted by Rolfe
But in no case is the relationship direct.

For this "accuracy" invention to be valid, you'd need to have tests that vary in exact direct relationship, and the thing is, they don't.
No; you don't get it. The two variables do not need to vary in a directly proportional relationship - they merely need to be equal.

In the hypothetical example, alpha happens to equal beta. That's why it's possible to give the test an accuracy - its rate of error is independent of the nature of the population.

How many times have I repeated this? How many times have you ignored it?

You're in way over your head, Rolfe.

[Rolfe]

Originally posted by Wrath of the Swarm
The preceding statements, considering in a mathematical light, are utter garbage. They mean nothing.

Two variables cannot be "more independent": either they are, or they aren't. Likewise, they are either linear, or they aren't.

There is a tremendous difference between an inverse relationship and an inverse proportion.All right, if I've been loosely using terms that mean very specific things to you, I'll endeavour to rephrase. If you will hang off the insults long enough to inform me of the phraseology you find acceptable, I'll endeavour to use it.

I've described two examples where as a rough tendency, efforts of the test developer to improve sensitivity will have an adverse effect on specificity, and vice versa. In one case it's a matter of choosing an absorbance cut-off, in the other it's a matter of choosing the number of gel-band matches that gives optimum performance.

All I can say about the - er, relationship? - between sensitivity and specificity in these cases is that as one improves, the other tends to deteriorate. Without having very exact data about the individual assay in question, no more can be said (and in fact it's only a generality that I'm trying to establish). The one thing that we can say is that an alteration aimed to improve sensitivity will at best leave the specificity unchanged. And vice versa. The one thing that can't happen is that a change in cut-off (or choice of matches) will simultaneously cause an improvement in either value. Most probably there will be some sort of see-saw effect, of variable and unpredictable - er, relationship?.

Now it's taken me quite a lot of words to try to explain the situation without outraging your sense of propriety by misusing a term you want to reserve to a specific definition. So what words would you use to get this point over without typing three paragaphs?

Rolfe.

[Wrath of the Swarm]

Originally posted by Rolfe
All right, if I've been loosely using terms that mean very specific things to you, I'll endeavour to rephrase. These are not personal definitions. They have generally accepted meanings in mathematics - and if we put those meanings into your sentences, we find that they don't make any sense.

If you will hang off the insults long enough to inform me of the phraseology you find acceptable, I'll endeavour to use it. You're an expert, remember? You've written a book? Surely you can use the mathematical terminology correctly, yes? Why haven't you done so, Rolfe? Are you playing with us? Is that it?

Oh, right. You don't have the slightest idea what you're talking about other than the concepts you've learned by rote.

[steve74]

Originally posted by Wrath of the Swarm
In response to steve:

Yes, I was wrong. I admitted this as soon as I realized it was the case. I was sloppy and stupid, and didn't read the sources I'd found carefully enough. They contained much of the information originally presented to me years ago in college courses, but not the one piece of information I was referring to: the formulation of my question.

However, you have made further claims which are mistaken. Furthermore, even when I pointed out to you that they were mistaken, you continued to make them. When you pointed out my error to me, I admitted I'd made a mistake. You lied and repeated the falsehoods.

I am merely a fool. You are a fool and a liar.

Wrath, you really are thick aren't you? I mean mind-numbingly, jaw-droppingly stupid. I have already pointed out to you once in this thread that you do not appear to know the meaning of the word 'liar'. It appears you still don't get it.

All I have done in this thread is post facts and opinions that I believe to be correct. I still believe everything I have posted in this thread is factually correct. So, I ask you once again (you failed to answer last time), how can I be a liar?

You, on the other hand, have blatantly lied. You claimed you had found studies and posted links to them. You did not. Is it really plausible to believe that you had a memory of finding the correct studies and posting links to them? Do you often have false memories like this? If so, I suggest you seek help from a mental health professional.

Wrath, you are either a liar or suffering some sort of mental problem. Which is it?

[ceptimus]

Yes. 99% accurate means exactly what it says:

99 out of every 100 tests give the correct answer.

Period. That's all you need to know. You don't need any alpha or beta in this case - you don't need to know what proportion of the population has the disease to understand this fact.

The anti-Wrath crowd may continue to bluster and obfuscate, but this clear and simple fact will continue to stick out like a very sore thumb.

[geni]

Originally posted by ceptimus
Yes. 99% accurate means exactly what it says:

99 out of every 100 tests give the correct answer.

Period. That's all you need to know. You don't need any alpha or beta in this case - you don't need to know what proportion of the population has the disease to understand this fact.

The anti-Wrath crowd may continue to bluster and obfuscate, but this clear and simple fact will continue to stick out like a very sore thumb.

Yes but you can get that level of correct tests (ok a bit higher) by haveing the test never giving a false posertive but almost always giving a false negative.

[Rolfe]

Originally posted by Wrath of the Swarm
No; you don't get it. The two variables do not need to vary in a directly proportional relationship - they merely need to be equal.Yes, I do get it.

The point is that for this to pertain in any general sense, you'd have to be dealing with tests where an improvement in sensitivity automatically improved specificity along with it, to keep the two the same.

But we are dealing with systems where improvements in sensitivity tend to cause deteriorations in specificity, and vice versa. Which is why the term is meaningless.

You expected an assumption to be made. That's what we're discussing.

Now I admit that I made a mistake. Sensitivity does have a small effect on predictive value. I should have agreed with you when you maintained that the assumption had to be that both values were 99%.

I made a mistake when I presented you with the easy get-out of saying "I just meant specificity". This was wrong. You were correct to keep asserting that the assumption you wanted made was that both values were 99%.

But it's still an assumption which must be made before any attempt can be made to solve the problem. Which is what we were talking about in the first place.

Rolfe.

[Wrath of the Swarm]

No, you can't, because then the accuracy of the test cannot be said to be 99% any more.

Now the equation for accuracy is no longer a constant, but a variable that depends on the particular sample population you put into it. Give the test to a population composed entirely of healthy people, and a certain percentage of the answers will be wrong. Give the test to a population of sick people, and a different percentage of the answers will be wrong. Give the test to a mix, and the percentage will change depending on the relative proportion of sick and healthy people.

[drkitten]

Originally posted by Wrath of the Swarm
No, you can't, because then the accuracy of the test cannot be said to be 99% any more.

Now the equation for accuracy is no longer a constant, but a variable that depends on the particular sample population you put into it. Give the test to a population composed entirely of healthy people, and a certain percentage of the answers will be wrong. Give the test to a population of sick people, and a different percentage of the answers will be wrong. Give the test to a mix, and the percentage will change depending on the relative proportion of sick and healthy people.

For what it's worth, I just checked with the biostats professor here (the guy who brings in most of the department funding through consulting with the local hospitals), and he offered an entirely new definition of "accuracy" in this context :

The accuracy of the test is the number of trials for which the test got the correct answer divided by the total number of trials. (More formally, the true positives plus the true negatives, the sum divided by the total population).

He specifically rejected the idea that "accuracy" could be twisted to mean "both the specificity and the sensitivity."

Rolfe? Ever seen this usage?

[geni]

Ok someone work out the accuracy for the case where for every 100 people there are 1.000101 false negatives per hundred people and no false posertives and for when there are 1.000101 false posertives per hundred and no false negatives.

[Wrath of the Swarm]

If the chance of a false positive isn't the same as the chance of a false negative, though, that value will change depending on the incidence within the test population.

Ignoring statistical variation and looking only at the averages for the moment, we'll find different values for the accuracy if we test populations with different rates of the disease.

Imagine alpha is .2 and beta is .3. Then if we test a population where everyone is negative, the test will have a 20% accuracy. If we test a population where everyone is positive, the test will have a 30% accuracy.

Without making reference to the specific test population, we can't determine the accuracy. It's a variable that depends on an unknown factor.

Again: the only way we can speak about objective, universal accuracies for the test is if alpha equals beta. To simplify everything, I gave such an case in the original question. No assumptions are needed.

[Martin]

Originally posted by Wrath of the Swarm
Again: the only way we can speak about objective, universal accuracies for the test is if alpha equals beta. To simplify everything, I gave such an case in the original question. No assumptions are needed. Apart from the assumption that the 'accuracy' you gave was objective and universal, and not simply correct for the test population you had already identified.

[geni]

Originally posted by Wrath of the Swarm
If the chance of a false positive isn't the same as the chance of a false negative, though, that value will change depending on the incidence within the test population.

But you told us the population so we don't need to worry about that.

[Wrath of the Swarm]

But I gave it for the general population.

Each individual tested is a population of one; either the incidence is zero, or the incidence is one. Either the person doesn't have it, or they do.

The accuracy of the test, in any particular case, then becomes undefined unless we know whether the person has the disease or not - and since the point of the question was determining whether or not the conclusion the doctor reached was correct, I couldn't tell you that.

[Wrath of the Swarm]

"Doctor, how accurate is this test you're giving me?"

"Well, Timmy, that depends. If you actually have the disease, there's an 80% chance the test will detect it and a 20% chance it won't. If you don't have the disease, though, there's a 5% chance the test will detect the disease anyway and a 95% chance it won't."

Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible.

[geni]

Originally posted by Wrath of the Swarm
"Doctor, how accurate is this test you're giving me?"

"Well, Timmy, that depends. If you actually have the disease, there's an 80% chance the test will detect it and a 20% chance it won't. If you don't have the disease, though, there's a 5% chance the test will detect the disease anyway and a 95% chance it won't."

Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible.

No the above is simpler.

[Rolfe]

Originally posted by Wrath of the Swarm
Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible. But you didn't STATE that that's what you were doing. You left it to be assumed. Which was not making things as simple as possible.

Now I've admitted to where I caused unnecessary contortions in the discussion, by erroneously offering you two "assumptions" you might have wanted us to make.

But you wouldn't say, no, there is only one assumption which will get you an answer, so that is the assumption I intended you to make. Because yu won't concede that that was an assumption.

Rolfe.

[ceptimus]

Originally posted by Martin
Apart from the assumption that the 'accuracy' you gave was objective and universal, and not simply correct for the test population you had already identified. Yes. That is what I referred to earlier when I said there was some (slight) room for quibbling. A better wording would have been:

A particular test for a disease always gives results with 99% accuracy.

In the country where I live, one in every thousand people are known to suffer from the disease. Before I took the test, there was no reason to assume I was specially at risk - that is to say I had the usual 1 in 1000 chance that applies in my country.

Yesterday, I took the test, and the test says I have the disease. What are the chances now that I have the disease?

[steve74]

Earlier in the thread, Wrath, you wrote this:

.Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

I've already shown point 1 to be a falsehood, so lets move onto point 2. In the study I quoted and the studies that you linked to, the questions were not phrased using the term accuracy, which you used in your question. Wrath, are you now claiming that that the researchers and statisticians who wrote these questions screwed up? Should they have used the term accuracy? No, they should not have done and neither should you have done. In the context of such a study the term is meaningless, the subjects would need to make an assumption as to what it means. You phrased the question wrongly. You have had plenty of opportunities to admit this and correct your error but you have failed to do so. This is the point you have consistenly failed to get.

By the way, Wrath, you have now called me a liar several times in this thread. Twice, I have asked you how you defend these assertions and twice you have failed to respond. It seems to me likely that this is because you know I haven't lied and only made the assertions in order to draw attention from your own lies

[ceptimus]

Originally posted by geni
Ok someone work out the accuracy for the case where for every 100 people there are 1.000101 false negatives per hundred people and no false posertives and for when there are 1.000101 false posertives per hundred and no false negatives. That situation can't apply to a general population geni. If everyone has the disease, then it is impossible for false positives to occur. Similarly, if no one in a particular population suffers from the disease, then false negatives are impossible.

Note that Wrath's accuracy figure is much more universal - it simply means that the result for one in every 100 tests will be wrong. This figure can work on a population where everyone has the disease, or no one, or any mixture of the two.

[Rolfe]

Originally posted by ceptimus
Yes. That is what I referred to earlier when I said there was some (slight) room for quibbling. A better wording would have been:

A particular test for a disease always gives results with 99% accuracy.

In the country where I live, one in every thousand people are known to suffer from the disease. Before I took the test, there was no reason to assume I was specially at risk - that is to say I had the usual 1 in 1000 chance that appies in my country.

Yesterday, I took the test, and the test says I have the disease. What are the chances now that I have the disease? It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumption.

Also, while you have pinned down the prevelance figure by stating explicitly that you want the 1 in 1000 to be seen as applying to that patient, it's not entirely realistic.

If the overall incidence in the country is 1 in 1000, then the prevalence in people with no observable risk factors is going to be less than 1 in 1000. Because some of these people are going to be excluded from the group because they are showing risk factors or clinical signs.

So if you're going to tell us anything in particular about this patient, such as the fact that he has no observable risk factors, the prevalence figure you really want isn't the prevalence in the entire population of the country but the prevalence in the subsection who are showing no risk factors.

It may seem a slight quibble, but to be absolutley correct here you would be better to give the incidence of disease in clinically unremarkable individulas, since you have told us that we are dealing with a clinically unremarkable individual, or to do what Steve74's test did, and simply say nothing about clinical presentation and state explicitly that you want the problem worked without reference to any signs or symptoms.

I prefer the former approach, because it's more realistic.

Rolfe.

[ceptimus]

Originally posted by Rolfe
It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumption.
No. You are wrong Rolfe. I am beginning to despair of you ever understanding this.

If I state that a test ALWAYS gives results that are 99% accurate, then it FOLLOWS that the specificity and sensitivity MUST both be 99%. There is NO assumption. It is simply mathematically impossible for it to be any other way.

[Wrath of the Swarm]

It's not an assumption, Rolfe. It is a stated property of the test.

It is remarkable how unable you are to grasp even this, the simplest of points. It is not an assumption.

ceptimus: I see your point, but I really don't think that wording was necessary. If I say the test has an accuracy, it can only be a universal one if I'm expressing any meaning at all. Whether I spoke about the general population immediately before doesn't matter, because the value would change for each different population.

So I agree that your wording would have been easier to understand for some, but I still don't think there's a problem with the way I worded things. (I have been wrong before, though, so take this with a grain of salt.)

[Martin]

Originally posted by Rolfe
It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumptionNot this time, no. Ceptimus' test is always 99% accurate, regardless of test population. That means that you'll get 99% accuracy for any subset of the population you choose. Choose the disease-free and disease carrying populations as your subsets, and you'll see that it must be true that specificity = sensitivity = 99% in this case. Wrath didn't specify that his accuracy was independent of the test population he'd set up, so there's some wiggle room there. The version Ceptimus gave seems solid to me.

[Rolfe]

Originally posted by ceptimus
Note that Wrath's accuracy figure is much more universal - it simply means that the result for one in every 100 tests will be wrong. This figure can work on a population where everyone has the disease, or no one, or any mixture of the two. No, it's not universal. It only applies to the very unlikely situation of a test with equal sensitivity and specificity. The fact is that you need to know both figures (sensitivity and specificity), every time that the test you're using doesn't fit this idealised scenario.

Rather than make the reader assume that you must be implying the idealised scenario because you only gave one figure where two were needed, it's still clearer just to say "sensitivity and specificity are both 99%".

You can also dress it all up like they did with the breast cancer question, giving raw data for true positives and false positives, which contains the necessary information within it.

Rolfe.

[Wrath of the Swarm]

If the accuracy hadn't been independent of the population, it would have a different value for different populations, and the value I gave would not only have been utterly useless but technically incorrect as well.

Without a specific testing population, the accuracy of such a test is not defined. (I suppose we could do some calculus and sum up the accuracies across different populations, but that's a lot of work to get a statistic that has little to no value in performing calculations.)

So since I gave it, it must be independent of population.

In hindsight, the question would have been less open to misunderstanding if it had been worded differently - but then we wouldn't have given Rolfe this chance to humiliate herself utterly, so in the end I'm quite pleased.

[Rolfe]

Originally posted by Wrath of the Swarm
In hindsight, the question would have been less open to misunderstanding if it had been worded differently - but then we wouldn't have given Rolfe this chance to humiliate herself utterly, so in the end I'm quite pleased. I'm glad you're pleased, Wrath. But it's still an assumption, certainly the way you worded the question.

And if you were really smart enough to spot the mistake I was making, you could have pointed it out about ten pages ago. But that would have involved opening yourself to the possibility of conceding that your wording was open to misunderstanding, as you have just done.

Edited to add: Since Wrath seems to be taking this post to imply that I believe I'm "humiliated utterly", I'd just like to clarify that I've admitted making a silly error. However, Wrath nevertheless worded his example so that an assumption had to be made, which was the essence of the disagreement in the first place. He has, grudgingly, admitted this, finally.

Yes, I feel silly - because rushing at Wrath the moment I saw the hole he was about to walk into led me to throw away the advantage by not considering what I was doing with the minor ambiguity carefully enough. This is not clever tactics, I agree.

However, the hole is still there. It's just unfortunate that there has been such acrimony that we may never get to see the bottom of it.

Rolfe.

[Wrath of the Swarm]

I've been pointing out your mistakes since this thread started, Rolfe. I called you on that particular one long before the concept finally penetrated your fog of obliviousness.

If you're such an expert, why did you make such a basic mistake in the first place?

Reality check: anything *can* be misinterpreted, but some things are easier for the thickies among us to misunderstand than others.

I should have realized that your almost complete ignorance of actual statistics and your rabid certainty that anything you don't recognize as right must be wrong would lead you to draw the wrong conclusion, thus inducing you to try to wreck and thread and complain about how stupid we all are and how brilliant and educated you are.

Sadly, I didn't, and I made the assumption that the post would be read mainly by intelligent and competent people.

That's the mistaken assumption in this thread, Rolfe: that you wouldn't actually be as stupid as you seemed to be.

Well, I've been wrong before, and I'll probably be wrong again.

[steve74]

Originally posted by Wrath of the Swarm

{snip}

In hindsight, the question would have been less open to misunderstanding if it had been worded differently

Jesus, 10 f*cking pages before you admit the bleeding obvious.

Oh, and Wrath, I have now asked you three times to defend your assertion that I am a liar. Three times you have failed to do so. There has only been one of us lying in this thread - that is you.

[Rolfe]

Originally posted by Wrath of the Swarm
If you're such an expert, why did you make such a basic mistake in the first place?Now, that's a fair question.

Several reasons. Mainly that I was more interested in the point about the choice of prevalence figure, and didn't look closely enough at the relatively minor ambiguity, this accuracy nonsense. At the beginning all I saw was that there was an assumption, and was careless in defining what it was.

Then, I was doing real life, I was more interested in the prevalence and the predictive value discussion anyway, and every time I came on the forum there was somebody calling me a whore. Maybe I don't deal with this so well as I thought. So if that was your intent, well, congratulations.

Congratulations also in succeeding in explaining this "basic mistake" in such clear terms that I so easily understood what you meant. Not.

Congratulations also in admitting that there was an assumption though, so I'm prepared to shake on that.

Rolfe.

[Wrath of the Swarm]

Congratulations on being intelligent enough to pick up on the error right after you made it.

Oh, right. Congratulations on being intelligent enough to pick up on the error when it was pointed out to you.

Oh, right. Congratulations on figuring it out eventually.

Oh, right.

I've figured out what you are. Not a whore, because I can respect an honest whore. Not a dishonest whore, because even that's too good for you.

You are an honest-to-goodness pseudoskeptic. A verifiable and documentable one, who cares only being seen to be right, and so seeks out and attacks the most pathetic creduloids there are for an easy victory.

Congratulations, O Most Educated Expert-Type Person! You have demonstrated the extent of your knowledge and intelligence for all to see!

I think I'll add "but you didn't state the specificity and sensitivity" to "Elton John totally copied REO Speedwagon", "No one has ever refuted me", and "The Iraqi army is throwing off the American aggressors".

Maybe I'll even put it in my sig! What fun!

[yersinia29]

Originally posted by ceptimus
If I state that a test ALWAYS gives results that are 99% accurate, then it FOLLOWS that the specificity and sensitivity MUST both be 99%. There is NO assumption. It is simply mathematically impossible for it to be any other way.

Well sure. But is that really the point?

If I published a medical study with that kind of wording, peer reviewers would ask me to change that language to include the specific terminology of sensitivity and specificity. Not because they COULDNT figure it out, but simply because it results in less confusion if you use specific terms.

Whats "mathematically impossible" and whats good to put in published form are 2 VERY different things.

[Rolfe]

Originally posted by yersinia29
Well sure. But is that really the point?

If I published a medical study with that kind of wording, peer reviewers would ask me to change that language....Strictly, yes, it's not an "assumption" the way Ceptimus puts it. But what he's doing is implicitly limiting his example test to the rare situation of equal sensitivity and specificity. Why?

In this case it's implicit, and you can demonstrate that he's plugged the holes so that you have to take the example that way. Though if you're not coming at it from Ceptimus's angle, it does require quite a bit of thought.

The disadvantage of this approach is that you can't use his example for a more general case. You can't use it except for the very limited situation described. So instead of being able to get on with the example because it mirrors the real-life situation, you have to spend some time working out that there is a covert restriction in place, and realise that it's an artificial example.

Why not just use an example of the general case, where the two variables aren't necessarily the same?

Rolfe.

[geni]

Interesting. There is a second flaw in the question which when taken to it's extream value gives you an answer that isn't on the poll. (the answer btw is 4.9955% assuming I've got the maths right).

[Rolfe]

Originally posted by geni
Interesting. There is a second flaw in the question which when taken to it's extream value gives you an answer that isn't on the poll. (the answer btw is 4.9955% assuming I've got the maths right). I'm done for now. I have to sort out my packing for the weekend (going to Yorkshire for the Bank Holiday Weekend), and I haven't even put the washing up to dry.

Besides, running at what seemed to be a familiar problem too fast in the first place led me to make a silly mistake, so let Wrath at least half way off that particular barb of the hook he's created. I'm now so tired I don't think I'm safe in public to add 2 and 2 until I've had a chance to think about it!

I'm done with taking shortcuts on this example, I want to see the whites of his eyes next time. Which unfortunately may not be before Tuesday.

Rolfe.

[steve74]

Hey Wrath, you big liar, I was just wondering why you've ignored my three requests to defend your assertion that I'm a liar? Is it because you don't even know what the word means? Is it because you're just f*cking rude. Or is it (and I'm betting it is) because you can't defend your assertion, as it's baseless.

Wrath of the Swarm - you are a liar, a boor and the owner of a truly monstrous ego.

[Rolfe]

Originally posted by Wrath of the Swarm
If you're such an expert, why did you make such a basic mistake in the first place?This may be the best question in the latter part of the thread, and I hope I can now answer it truthfully.

Well we all know what Wrath thinks. He's told us so many times. I'm brain-dead. Like all other medical professionals (except I forgot, I'm not a medical professional). He's never going to change that opinion, no matter how explicable the mistake.

However, it is explicable. I was telling the truth about the published papers, the book, the book reviews and the lectures on the subject. And by the way, there weren't any mistakes in these. So why did I make a mistake here? And how come I got the right answer in spite of it?

Remember, I said I used a spreadsheet to derive the answer. I wrote that spreadsheet in about 1996, when I became involved in this particular subject at a detailed level. I had to do so many of these predictive value calculations that it was a lot easier to do it like that, also I wanted to display that graph I showed in my first post, as many of the concepts are a lot easier to get across from a graph. So I became used to doing the calculations using the spreadsheet I was familiar with.

This displayed the difficulty posed by Wrath's incorrect formulation of the question rather differently from the back of the envelope method. Whichever way you look at it, Wrath gave only one parameter (without specifying which), when two were required. Does the number fit one slot only, or is there some way to derive the two different parameters from the single value?

Wrath's constant assertions that his unique use of the word "accuracy" was intended to force the two values to be equal are completely illegitimate. Even if the two values are allowed to be equal to make the problem simpler, they have to be free to be non-equal, because there is nothing constraining them from being non-equal. This is why the term accuracy isn't defined and isn't used in this context - because it is meaningless. There is no way to derive an "accuracy" value for a real-life assay except by a fluke, and the terms employed have to be able to be used with real-life assays or the whole exercise is futile.

When you look at the difficulty from the point of view of the spreadsheet, it is intuitively fairly clear that one of the numbers is much more necessary than the other. Small differences in specificity make a huge difference to the resulting predictive value for this particular problem, while larger differences in sensitivity make relatively little difference. Like this.

<TABLE BORDER="1"><TR><TD ALIGN=CENTER VALIGN=BOTTOM>Variable value</TD><TD ALIGN=CENTER VALIGN=TOP>PPV with
specificity constant</TD><TD ALIGN=CENTER VALIGN=TOP>PPV with
sensitivity constant</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>100%</TD><TD ALIGN=CENTER VALIGN=TOP>9.10%</TD><TD ALIGN=CENTER VALIGN=TOP>100%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>99.5%</TD><TD ALIGN=CENTER VALIGN=TOP>9.06%</TD><TD ALIGN=CENTER VALIGN=TOP>16.54%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>99%</TD><TD ALIGN=CENTER VALIGN=TOP>9.02%</TD><TD ALIGN=CENTER VALIGN=TOP>9.02%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>98%</TD><TD ALIGN=CENTER VALIGN=TOP>8.93%</TD><TD ALIGN=CENTER VALIGN=TOP>4.72%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>97%</TD><TD ALIGN=CENTER VALIGN=TOP>8.85%</TD><TD ALIGN=CENTER VALIGN=TOP>3.20%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>96%</TD><TD ALIGN=CENTER VALIGN=TOP>8.77%</TD><TD ALIGN=CENTER VALIGN=TOP>2.42%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>95%</TD><TD ALIGN=CENTER VALIGN=TOP>8.68%</TD><TD ALIGN=CENTER VALIGN=TOP>1.94%</TD></TR></TABLE>

I certainly didn't think this through at the time. If I had, I wouldn't have made any mistake. I simply saw, quickly, that if specificity was pegged at 99%, then not defining the exact sensitivity within the range of expected values for a good assay didn't really make much difference. In effect, the answer was close-enough to 9 for any value between 95% and 100%. So click the "10%" button and let's get on to the interesting part. (My real mistake was in giving that 9.02% figure so precisely - I didn't stop to check what sensitivity value was in the damn sensitivity box when I increased the number of significant figures visible in the cell to see just how close to 10% we were, and it happened to be 99%. In fact to that number of significant figures the influence of sensitivity is indeed appreciable, if small.)

No doubt Wrath is going to jump all over this. Assumptions all over the place. But remember, assumptions he was generating in his inexplicable desire to force his own assumption, that sensitivity and specificity are equal. As this was such a false assumption, considered as a legitimate method of handling the calculation in general, I went without sufficient thought for the other assumption that would give a meaningful answer, that the figure we had been given was the one we really needed, the specificity.

Wrath will no doubt assert that this proves I'm incapable of coherent thought, and can only plug numbers into my spreadsheet without thinking. However, Wrath, consider. This is the risk you run if you don't use the correct terms to describe the problem you're setting. You may think that the assumption you're trying to force is inevitable, but it may be that someone who sees the problem from a different perspective (in this case, seeing the necessity for using only terms that can be used for all assay conditions, not just a flukey subset seldom encountered in real life) may reject that assumption, and find some other more or less workable one.

So why was I so careless? Because it was the mistake of Wrath's that didn't really matter. It was (or should have been) easy to clarify, and it wasn't a mistake that carried any agenda or affected the result of the problem in any meaningful way. It's still ~10%, whatever. We're not arguing about that. We'd just have liked an unambiguous question.

At the time I made this error, I was already furious. Furious and fed up with Wrath's constant anti-medicine agenda, one plank of which I now understood for the first time, and with the fact that the problem set was clearly worded in such a way as to facilitate the argument that doctors are stupid. Which Wrath indeed lost no time in setting out.

To elaborate. Wrath has been asserting for months now that it is a known fact that doctors can't think. And much of the time he supports this by saying that "doctors can't do Bayesian analysis". Didn't pay much attention. Yes, I understand the statistics I need to understand, but I don't have all the statistician's language. I didn't realise that this particular problem was what he meant. Then I did.

This problem has a number of dangerous angles. For one thing, it can be used as Wrath is trying to use it, to ambush medical personnel into giving the "wrong" answer, even though the same people might in fact make the right assumption if the situation was presented to them in a more familiar context. For another, it is the start of a very dangerous and counterintuitive logic trail, which is often mis-presented in such a way as to undermine the clinician's confidence in an intuitive approach which usually leads to the right course of action, and substitute assumptions which are only valid for the well-patient screening situation, forcing them to be inappropriately applied to the clinical testing situation.

This is the aspect which particularly interests me, and examination of the false logic trails leading from it is the reason I have become especially familiar with the statistics. To the point where I can normally handle them well enough to write books and articles and give lectures about them.

However, in my haste to advance to the interesting part, I made a mistake in the way I described my assumptions stemming from Wrath's minor terminology error. Instead of saying, for this particular problem the exact value of sensitivity (within reason) makes no material difference, therefore we assume you've just given the specificity value, I wrongly, without thinking, said the sensitivity was completely irrelevant. And then I was called sixteen sorts of idiot, a whore and God knows what else. And I don't believe Wrath even realised exactly what I'd done. He was so intent on denying that his wording required any sort of assumption at all, and on finding new and creative insults to throw at me, that he never even looked to see if there was any rationale to what I was saying. Because, you see, I'm a pretend medical type who can't possibly understand anything about all this mathematical stuff, so the correct response is obviously abuse.

By the time the subject was dragged back to the "accuracy" point, I was seeing so much red that I woefully failed to go back and make sure that, however right I might be about Wrath's wording requiring an assumption, my explanation of the assumption I'd hastily made was actually correct. As Wrath was only concerned to defend his position that no assumption was required, I was only concerned to demonstrate that it was. Which it was, as we've finally seen.

All right, that should be a lesson to me. When handling a very familiar problem which you haven't worked through from first principles for several years, don't let anger prevent you from checking your working.

Whether it will be any sort of lesson to Wrath on the danger of making implicit assumptions when formulating from memory a well-known problem designed to annoy people, I very much doubt.

Rolfe.

[Wrath of the Swarm]

Yes, Rolfe. All the researchers who performed studies on this effect were trying to humiliate medical personnel.

Ooh, we're still waiting to see you make the numbers scream? God and country, remember? They'll sit up and beg? Let's see it!

[Rolfe]

Originally posted by Wrath of the Swarm
Ooh, we're still waiting to see you make the numbers scream?Don't mis-quote what I said, please. I said the problem would do so much more than you imagined it would. There's more to the problem than the numbers, although you might not think so.

Later. I don't imagine you will ever calm down, as "angry" seems to be a constant condition with you. However, anger doesn't help anyonte think straight, so I at least need to calm down. And I really am going on holiday in about four hours.

Rolfe.

[Wrath of the Swarm]

Misquote? Oh, I wouldn't dream of it. I pasted it directly from your post into my signature line.

[ceptimus]

Originally posted by geni
Interesting. There is a second flaw in the question which when taken to it's extream value gives you an answer that isn't on the poll. (the answer btw is 4.9955% assuming I've got the maths right). Can you explain how you arrived at this figure geni?

[Paul C. Anagnostopoulos]

Um, Wrath, the quote that you so elegantly include in your sig line says nothing about numbers or statistics.

Edited to delete sarcastic remark.

~~ Paul

[Wrath of the Swarm]

It does in context. That context, however, is too long to put in my signature.

She claimed she could take the orignal problem, as stated, and manipulate it statistically so that it could say things it was never intended to say.

I'd like to see it.

(edit) Changed sig to avoid confusion. How's that?

[geni]

Originally posted by ceptimus
Can you explain how you arrived at this figure geni?

Buy playing around with the the phrase "about 99%" I think that should alow me to play with the second nine but I can't work out how to get a meaningfull answer out of 104% so I just decided to see what happend if I added or subtracted 0.5. Of course this isn't really relivant since the reason WOTS used the word about was because it saved him from haveing to work out the value needed to get excatly 10 (which to honest I cant be bother to work out either)

[ceptimus]

Originally posted by geni


Buy playing around with the the phrase "about 99%" I think that should alow me to play with the second nine but I can't work out how to get a meaningfull answer out of 104% so I just decided to see what happend if I added or subtracted 0.5. Of course this isn't really relivant since the reason WOTS used the word about was because it saved him from haveing to work out the value needed to get excatly 10 (which to honest I cant be bother to work out either) Ahh I see.

I calculate the required test accuracy to give exactly a 10% chance of having the disease, on receiving a positive test as:

111 / 112

which to 6 significant figures is 99.1071%

(but of course, as always, I may be wrong)

[Paul C. Anagnostopoulos]

Wrath said:
Changed sig to avoid confusion. How's that?
Not much better. Even in context, it doesn't appear that Rolfe was talking only about statistics.

I think people can read this thread without needing hints in your sig line. But, it's your sig line.

~~ Paul

[Badly Shaved Monkey]

Originally posted by drkitten


For what it's worth, I just checked with the biostats professor here (the guy who brings in most of the department funding through consulting with the local hospitals), and he offered an entirely new definition of "accuracy" in this context :

The accuracy of the test is the number of trials for which the test got the correct answer divided by the total number of trials. (More formally, the true positives plus the true negatives, the sum divided by the total population).


WotS

Does this look like a reasonably authoritative use of the term 'accuracy'?

If 'accuracy' is said to be 99% does this not allow sensitivity and specificity to vary from 99%? If 'accuracy' = 99% permits specificity to vary from 99% doesn't your original question require us to assume that you meant specificity=99% while paying little attention to what sensitivity may be in order to give the ball-park answer of "10%" in the poll?

Is this correct?

Am I not right in thinking that with an 'accuracy' of 99% both specificity and sensitivity can vary in opposing directions from 98-100% provided their mean is 99%?

[Wrath of the Swarm]

Knowing only the specificity is not enough to solve the problem, despite what Rolfe would have you believe.

If the rates of false positives and false negatives are the same, then we can give the test an overal accuracy.

If they're not the same, though, the accuracy becomes dependent on the population being tested. In other words, instead of the formula you just presented giving the same value in all situations, it will change according to the situation.

"Assumptions" indeed.

[ceptimus]

Originally posted by Badly Shaved Monkey


WotS

Does this look like a reasonably authoritative use of the term 'accuracy'?

If 'accuracy' is said to be 99% does this not allow sensitivity and specificity to vary from 99%? If 'accuracy' = 99% permits specificity to vary from 99% doesn't your original question require us to assume that you meant specificity=99% while paying little attention to what sensitivity may be in order to give the ball-park answer of "10%" in the poll?

Is this correct? I know you asked WotS, but I would like to comment.

The biostat professor's definition of the term is the same as the one WotS has been using throughout. Further, if the same accuracy applies to any population, with any frequency of occurence of the disease, it then (mathematically) follows that the specificity and sensitivity must be the same as the accuracy.

If the accuracy only applies to a specific population, then the specificity and selectivity are related to each other (and to the accuracy, and actual incidence of the disease) by a mathematical function. Puzzle: What is the function?

Edit to add: The wording of 'what the professor said' is a bit vague - it's the number of people tested that matters, not the total population.

[Badly Shaved Monkey]

Hey here's an interesting paragraph that can keep everyone happy for a while.

http://my.execpc.com/~mjstouff/articles/specsen.html

"Bottom line: when someone tells you that a diagnostic test, done properly, is 99% accurate (meaning that both the specificity and the sensitivity = 99%), the actual 'accuracy' of the test will in fact really depend on how common the disease you are testing for is in the population you are testing.

This example clearly points out why diagnostic tests are designed to confirm a diagnosis for rare conditions. They should not be used to go 'fishing' for a diagnosis (which unfortunately happens as an inappropriate use of many diagnostic tests).

In real life, we don't nab 10,000 people randomly and run them through AIDS tests. People who tend to get AIDS tests are people who are at high risk for a variety of factors (lifestyle, occupation, medical condition such as hemophilia, transfusion recipient and the like), so they are in a statistical sense a different 'population' than the general population.

Even so, keep in mind that the test is most certainly NOT 99% 'accurate' in the way all of us think about accuracy, even though both its specificity and sensitivity are 99%. You always must take into account how common the disease you are testing for is in the population you are testing. Many, MANY tests have much lower specificity and sensitivity than the 99% I've used in this example. However, if the condition these tests are trying to diagnose is much more common in the population, then what we think of as 'accuracy' becomes better for a given level of specificity and sensitivity as the prevalence of the disease increases."

The writer goes on to define 'accuracy' as TP/(TP+FP) and gives and example with 'accuracy' of only 92%, yet sensitivity and specificity are both 99%.

It's also inteesrting to note that she puts 'accuracy' in quote marks most of the time to indicate the lability of the concept in contrast to WotS's assertion tht it is a uniquely defined parameter.

This handy definition also contradicts the one that appeared earlier in this thread of (TP + TN)/(TP + TN + FP + FN), which serves to emphasise the arbitrariness of te term.

(Edited to amplify)

[Prester John]

I think this thread has shown why people need to be careful about terminology, and also why for pos/neg tests sensitivity and specificity values are always given, even if they are the same.

The sheer amount of ambiguity over what WoS meant by the use of accuracy (as evidenced by the length of discussion) can be used as an argument as to why it shouldn't be used.

Incidentally, i think that the vast majoritory of the populace is very ignorant about statistics, and i would be unsuprised to find that doctors are included i this group. However it should also be remembered that in diagnosis of patients statistics is not the only factor :)

[Wrath of the Swarm]

I will admit, when I first read Rolfe's post I was so angry about her intentionally spoiling the poll that I didn't even notice her glaring mathematical errors.

Go back and look at what she wrote. She claimed specificity is the percentage of positive results that are correct.

This is wrong. http://www.fpnotebook.com/PRE18.htm

She's also wrong about the definition of sensitivity.

Then she claimed that all we needed to know was the specificity (which is wrong no matter what definition you use, the correct one or the "Rolfe special").

Reality check: this "expert" who supposedly wrote a book on the subject doesn't know the definitions of basic terms. Everything she's said in this thread has been grossly wrong.

And most of you mindlessly repeated her statements! I can go back, read the thread, and see you people repeating statements she'd made as if they were fact!

[Rolfe]

Originally posted by Wrath of the Swarm
Knowing only the specificity is not enough to solve the problem, despite what Rolfe would have you believe.

If the rates of false positives and false negatives are the same, then we can give the test an overal accuracy.1. I acknowledged that was an error. Please see above, I'm not doing it all again.

2. If the rates of false positives and false negatives are the same, then the rates of false positives and false negatives are the same. That's all. It doesn't justify the introduction of a completely foreign term (foreign to binary test discussion, that is) to describe this unusual occurrence. There is a reason why this term isn't used in this context, please see above again, because trying to derive any universally-applicable definition is confusing at best, please see (less far) above.

Yes, assumptions.

Rolfe.

[Badly Shaved Monkey]

Originally posted by ceptimus
The biostat professor's definition of the term is the same as the one WotS has been using throughout.
I don't think that's right, because the prof's definition allowed for sens and spec to vary whereas WotS insists they are both uniquely specified by a single figure for 'accuracy'

It also seems to be the case that there is not, as I have already mooted, any agreed definition of 'accuracy' because the second example I have now posted gives a different, yet reasonable version, and it looks exactly like WotS made up his/her/its version 'on the fly' when setting the question. If that was not the case why do we still lack WotS producing a citation to back up his definition and why do we have two others that differ from it?

Further, if (I think this 'if' is one of Rolfe's criticisms and is illustrated by the example I have just posted from here http://my.execpc.com/~mjstouff/articles/specsen.html ) the same accuracy applies to any population, with any frequency of occurence of the disease, it then (mathematically) follows that the specificity and sensitivity must be the same as the accuracy.

If the accuracy only applies to a specific population, then the specificity and selectivity are related to each other (and to the accuracy, and actual incidence of the disease) by a mathematical function. Puzzle: What is the function?

Edit to add: The wording of 'what the professor said' is a bit vague - it's the number of people tested that matters, not the total population.

[Rolfe]

Originally posted by Wrath of the Swarm
She claimed specificity is the percentage of positive results that are correct.

This is wrong. Yes, it is wrong. That describes the PPV, in fact. And if I said that, I made a unintentional mistake. There are places in here where I was typing too fast, I have noticed. Including a total typo of "precision" when I meant something else.

So go on, I'm off on holiday, go through the thread to find the typos, feel free. I've acknowledged the genuine error.

Wrath acknowledged his too, earlier.

Rolfe.

[Badly Shaved Monkey]

WotS

Will you please address the question of differing definitions of 'accuracy' that I have covered in my last two posts? Thanks.

[Wrath of the Swarm]

It's already been explained to you, BSM.

I explained what 'accuracy' meant several pages back. It's the same one as you got from the professor.

That value can only be defined ahead of time (independently of the sample population) if a certain set of conditions hold. They applied in this case. End of discussion.

[Badly Shaved Monkey]

http://www.rapid-diagnostics.org/accuracy.htm

"Accuracy can be expressed through sensitivity and specificity, positive and negative predictive values, or positive and negative diagnostic likelihood ratios. Each measure of accuracy should be used in combination with its complementary measure"

i.e. 'Accuracy' is the general term for the whole area not a specific parameter within it.


This would represent a third reasonable viewpoint, moving us even further from the idea of 'accuracy' as a uniquely defined numerical parameter that completely specifies specificity and sensitivity.

[Wrath of the Swarm]

They weren't just typos, Rolfe! You repeated them many times, and the people who repeated you did so many times as well!

You presented several arguments with those concepts. Didn't you notice you'd screwed up completely?

You don't understand the concept of accuracy, you don't know how to analyze statistical aspects of the test to reach conclusions, you thought consequences that follow inevitably from the given information were "assumptions"...

You're not an expert at all. You don't know a damn thing about what we've been discussing.

[Badly Shaved Monkey]

Some time ago someone asked why people were bothering with nitpicking over page after page of this thread, isn't it simply that it is an interesting exercise to try to resolve this puzzle of what is really meant by the superficially simple idea of 'accuracy'?

[Badly Shaved Monkey]

http://www.imtech.res.in/raghava/mhcbench/parameter.html

" The term accuracy was defined to provide a single measure of performance. It is defined as the proportion of correctly predicted peptides." Peptide assays is what this page was discussing.

"Accuracy
The proportion of correctly predicted peptides (both binders and non-binders)
((a + d)/(a + b + c + d))*100"

This is the same as the prof's definition.

WotS

Please explain how this definition uniquely specifies sens and spec for a given 'accuracy'

[Wrath of the Swarm]

No, you don't get it.

That definition applies when looking at a particular sample for the particular test. For that sample, the test's accuracy was given by that formula.

Without referencing a specific set of results, that definition is meaningless. The concept of 'accuracy' doesn't apply - except when the chance of error does not depend on the nature of the test population.

In such a case, alpha must equal beta by definition. That is the only time the accuracy of the test can be considered without mentioning a particular sample population.

[Badly Shaved Monkey]

Duplicate

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm

I explained what 'accuracy' meant several pages back.
Not very well it would seem since you claim that your definition uniquely defines sens and spec, whereas, notwithstanding your claim that
It's the same one as you got from the professor. , the prof's one does not, so yours and his can't be the same. Could you please lay them out in parallel as simple equations so you can show us what you mean and how it is the same definition as the professor's even if it differs from the other one I have just cited.

[Wrath of the Swarm]

[edit] Fixed a repeated error where I gave the error instead of the accuracy. Many thanks to those kind enough to point this out.

From a post of mine on page 4:

"Accuracy" is the proportion of correct test responses to total test responses. (Just what the English definition of the word would imply.)

Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the error is 3/10 or .30, and the accuracy is .70.

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Error is .40; accuracy is .60.

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5
wrong answers out of ten, which means accuracy is .65.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an error of .40 and an accuracy of .60, just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall error of .40 and accuracy of .60.

No matter what sample population I give the test, it will always have an accuracy of .60.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm

Without referencing a specific set of results, that definition is meaningless. The concept of 'accuracy' doesn't apply - except when the chance of error does not depend on the nature of the test population. As I have been pointing out, no one else seems to share your idea that 'accuracy' itself is uniquely definable, never mind whether it uniquely defines the parameters sensitivity and specificity, and in the prof's example it does not.

[ceptimus]

As you seem more interested in arguing than providing the formula that answers my puzzle, here it is:

A = (1 - P) * F_p + P * F_n

Where:

A = accuracy
P = proportion of population infected by the disease
F_p = proportion of false positives when only uninfected persons are tested.
F_n = proportion of false negatives when only infected persons are tested.

Note that all these are proportions (in the range 0 to 1) and not percentages. Multiply by 100 to express them as percentages.

But if anyone has learned anything from this thread, it should be that blindly plugging numbers into a formula you don't understand is a bad idea. That and the fact that you shouldn't let emotion enter into your argument, especially if mathematics or science is the the subject being argued over.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
From a post of mine on page 4:



Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the accuracy is 3/10 or .30.

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Accuracy is .40.

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5
wrong answers out of ten.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an accuracy of .40, just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall accuracy of .40.

No matter what sample population I give the test, it will always have an accuracy of .40.


Thanks for finding that.

I see the problem now. You think that by specifiying accuracy to be 99% and giving no other information means that it must mean sens and spec are both 99% or else you would have given us those parameters separately. I'm not sure how we wwere to know that, but I can see that is probably what you thought.

Yes? At last?

[Badly Shaved Monkey]

Originally posted by ceptimus
As you seem more interested in arguing than providing the formula that answers my puzzle, here it is:

A = (1 - P) * F_p + P * F_n

Where:

A = accuracy
P = proportion of population infected by the disease
F_p = proportion of false positives when only uninfected persons are tested.
F_n = proportion of false negatives when only infected persons are tested.

Note that all these are proportions (in the range 0 to 1) and not percentages. Multiply by 100 to express them as percentages.

But if anyone has learned anything from this thread, it should be that blindly plugging numbers into a formula you don't understand is a bad idea. That and the fact that you shouldn't let emotion enter into your argument, especially if mathematics or science are the the subject being argued over.

Forgive me for asking, but what is the provenance of this definition given the part of the problem seems to be a variety of uses of the term 'accuracy'?

[Badly Shaved Monkey]

Anyway, I must leave this for the moment. Patients waiting and blind clinical guesses to be made.

[yersinia29]

BSM,

Wrath makes the mistake of assuming that the only relevant issue here is strict mathematics.

From a strict mathematical sense, "accuracy" can be a substitute for both sensitivity and specificity values.

That does not change the fact that its a poor way to describe the problem.

[Hellbound]

I know I said I was done, but I couldn't pass up this little gem :D

Originally posted by Wrath of the Swarm
Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the accuracy is 3/10 or .30.
Wait a minute. 7 of the responses were accurate, so the accuracy is .30?

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Accuracy is .40.
Again, 6 results were accurate, but the accuracy is .40?

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5 wrong answers out of ten.
So is the accuracy here .65 or .35? Educate me, oh great statistics expert.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an accuracy of .40(psst...60%), just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall accuracy of .40(psst...60%).

No matter what sample population I give the test, it will always have an accuracy of .40.

Psst...60%.

I think you have displayed perfectly your expertise in the field, Wrath. Especially after calling Rolfe to task several times for mistakes she made. That plank must be heavy, no?

[ceptimus]

Originally posted by Badly Shaved Monkey


Forgive me for asking, but what is the provenance of this definition given the part of the problem seems to be a variety of uses of the term 'accuracy'? You are mistaken in your belief that accuracy has a variety of meanings. Of course as an English word, it does have many meanings, as any dictionary will show, but when it is applied to the outcome of tests that give a binary (yes/no) answer, then the meaning is (or should be) totally clear to any mathematician, scientist or engineer.

As I said many pages ago it simply tells you how good the test is - if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong. How could it possibly mean anything else? Can you give an alternative definition that could apply in a mathematical context?

[Prester John]

As I said many pages ago it simply tells you how good the test is - if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong. How could it possibly mean anything else? Can you give an alternative definition that could apply in a mathematical context?

and as such, it has virtually no usefullness for evaluating the utililty of medical binary test results. It does not convey sufficient information. Which is why sensitivity and specificty are used.

[yersinia29]

ceptimus,

I'll say it again. Whats true in a strict mathematical sense, and whats good language to use in a study are 2 very different things.

Like I said, there is no medical test known to man that has equal sensitivity and specificity values. Thats why "accuracy" is never used in regards to describing medical tests.

That doenst change the fact that in a strict mathematical sense accuracy = sensitivity and specificity. But whats relevant in a pure mathematical sense is not the only issue involved here.

[ceptimus]

Originally posted by Prester John


and as such, it has virtually no usefullness for evaluating the utililty of medical binary test results. It does not convey sufficient information. Which is why sensitivity and specificty are used. But it does give all the information necessary when it is unchanging, as I demonstrated above.

'A particular test is always 99% accurate.'

This tells you everything you need to know. Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

To say, as you just did, that it doesn't convey sufficient information is simply wrong.

[yersinia29]

Originally posted by ceptimus
Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

Change "unlikely" to "unprecedented" or "unheard of"

[Wrath of the Swarm]

Oops. How silly of me.

I gave the error instead of the accuracy (which was one minus the value I gave).

My mistake. Of course, I've never claimed to be an expert in anything.

I believe that when the correct terms are subsituted, you'll find that my argument is correct.

[ceptimus]

Originally posted by yersinia29
ceptimus,

I'll say it again. Whats true in a strict mathematical sense, and whats good language to use in a study are 2 very different things.
I agree with you of course. The point is that it was Rolfe and her supporters who began the quibbling. She and they were wrong, and have accepted that. Anyone who brings up the same argument, hasn't read and understood the thread, so we have to keep pointing out the same error over and over.

Now, lest it be misunderstood, I agree that two figures should aways be given for any real world diagnostic test. Even in the unlikely circumstance that both were the same, they should still both be given.

Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.

[Prester John]

Originally posted by ceptimus
But it does give all the information necessary when it is unchanging, as I demonstrated above.

'A particular test is always 99% accurate.'

This tells you everything you need to know. Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

To say, as you just did, that it doesn't convey sufficient information is simply wrong.


You need to know the false positive and false negative rate. Accuracy as defined :

if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong


does not do that except in exceptional circumstances. As these circumstances do not occur in the medical field, the term accuracy as defined by you is not used.

As i said many pages ago, this is a pure v applied argument. Yes under specified circumstances you can define accuracy to encompass sensitivity and specificity. This is not practical in medical statistics and is not done

[geni]

Originally posted by ceptimus
Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.

Well of course you'd say that your the only person who hasn't been wrong yet.

[Wrath of the Swarm]

Originally posted by Prester John
As these circumstances do not occur in the medical field, the term accuracy as defined by you is not used. WRONG!

Those circumstances are rare in medicine, but they can apply. They apply even more often outside of medicine.

[Prester John]

Originally posted by Wrath of the Swarm
WRONG!

Those circumstances are rare in medicine, but they can apply. They apply even more often outside of medicine. [/B]

The first part of my sentence may have been a very slight exageration, but the second is correct. Accuracy is not used to describe binary tests. Lets put is another way, in the 10+ years i have worked in laboratory doing tests, i have not seen accuracy used to describe a binary test result. this is my practical experience, i am not a statistical expert, but neither am i statitically challenged.

I have no comment on the use outside medicine (of the term accuracy), but as the original question concerned a medical test, it was an innapropriate term to use.

[Wrath of the Swarm]

The question is not whether that value is shown on medical test result forms.

In order to understand the concepts of false positive, false negative, true positive, and true negative, the concept of accuracy must be learned first.

If you know those four, you're presumed to know the one.

So... you can't actually find any mathematical errors in the question, can you?

[Paul C. Anagnostopoulos]

Ceptimus said:
Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.
Hmm, sounded like it was trying to be a real-world case to me. A hypothetical example would have been worded without reference to medicine.

Do people really misunderstand statistics all that badly, or was this mostly a giant rat-hole having to do with whether a specific sort of accuracy might or might not be relevant under the ambiguous conditions of the question?

~~ Paul

[Wrath of the Swarm]

The scenario I provided would work just fine in the real world. It would simply be less likely than scenarios where tests don't have universal accuracies.

That doesn't make a bit of difference to the question - if anything, it makes the question easier to answer than it would have been otherwise.

[jj]

Originally posted by Paul C. Anagnostopoulos
Do people really misunderstand statistics all that badly, or was this mostly a giant rat-hole having to do with whether a specific sort of accuracy might or might not be relevant under the ambiguous conditions of the question?

~~ Paul


And Paul sinks the 3-point shot. Paul shoots and scores 3, WOTS pulls a technical and Paul gets a free shot at goal!

[steve74]

Originally posted by Paul C. Anagnostopoulos

Hmm, sounded like it was trying to be a real-world case to me. A hypothetical example would have been worded without reference to medicine.

Paul, you're totally correct, it was was trying to be a real-world case. Wrath even claimed it was worded the same as in a study given to doctors:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

He maintained this lie until I forced him to retract it by demonstrating that he had no evidence to back it up,

[Wrath of the Swarm]

It's not a lie. I couldn't find a study that used that question. It was how the issue was originally presented to me.

It was an invalid claim, since I couldn't support it. I retracted it and admitted I'm an idiot.

You're still a liar, steve, for repeating Rolfe's claims even once you'd been informed they were incorrect. Well, either a liar or a fool, or a lying fool, it's not clear.

[Wrath of the Swarm]

Ceptimus has made clear that the question is a perfectly viable one.

Which reminds me...

[clears throat]

The correct answer to the question was the 10% option. The actual value is very close to 9%.

Congratulations to everyone who was honest enough to put down what they actually thought, even if it turned out to be wrong.

Thank you.

[steve74]

Originally posted by Wrath of the Swarm
It's not a lie. I couldn't find a study that used that question. It was how the issue was originally presented to me.

It was an invalid claim, since I couldn't support it. I retracted it and admitted I'm an idiot.

You're still a liar, steve, for repeating Rolfe's claims even once you'd been informed they were incorrect. Well, either a liar or a fool, or a lying fool, it's not clear.

Ah Wrath, you've finally managed to respond to my posts, at the fifth time of asking. You don't like people pointing out the truth about your lies do you? You're a coward as well as a liar.

I've already had to point out to you twice in this thread that you do not know the meaning of the word liar. Let's try one more time and please pay attention this time. Everything I've posted in this thread had been posted with a belief in its veracity. Therefore, I am not a liar. The same, sadly, cannot be said for you. Once again, you are a liar. I have caught you out in your lies, deal with it.

You are a sad little troll, even the homeopaths think so.

[steve74]

Originally posted by Wrath of the Swarm
Ceptimus has made clear that the question is a perfectly viable one.

{snip]


Oh dear, another factual inaccuracy, Wrath. In the TRSOTTTWND Ceptimus wrote:

You have to make the assumption or it is not solvable. That is why I made it.

You see Wrath you have to make an assumption - a point you have consistently denied. Another point you fail to realize is that in a real study (ie not an imaginary one you dreamt up) there would not be a multiple choice answer (I realize that is the only way to conduct a poll, but that is not the point). In a real study some of the subjects could well decide that the question isn't solvable. I made the assumption and got the correct answer but the only way I could make sure that I had made the correct assumption was to check whether my answer was in the poll options. In a real study that option would not be available.

'Accuracy' is not a defined term in this area. The assumption that needs to be made is 'what the hell does accuracy mean?' You misremembered the question and as a result it was ill phrased. This is why in all the real studies (including the ones in the review articles you linked to) the term 'accuracy' is striking by its absence.

[Wrath of the Swarm]

Conveniently leaving out the later bits where ceptimus is convinced there's no problem.

It's been explained over and over again that presenting a universal accuracy for the test means only one thing.

[steve74]

Originally posted by Wrath of the Swarm
Conveniently leaving out the later bits where ceptimus is convinced there's no problem.

It's been explained over and over again that presenting a universal accuracy for the test means only one thing.

Oh yeah, a really well worded question when even Ceptimus, who's superb at solving puzzles, wasn't convinced at first that no statistical assumption needed to be made. How long do you think they give the subjects to answer these questions (not that any subjects have ever been asked the question you asked?) Five minutes? A day? A week??!!? And I still assert that an assumption of what the term 'accuracy' means would need to be made.

Accuracy is a meaningless term in this case. Meaningless.

Even you admitted:

In hindsight, the question would have been less open to misunderstanding if it had been worded differently

Even though you had earlier claimed:

It wasn't even a poor choice of terminology

Oh dear, Wrath, consistency really isn't your strong point, is it?

I also notice that for the sixth time of asking you've failed to defend your assertion that I'm a liar. Somewhat telling, I feel.

btw Even the homeopaths think you're a complete troll and have deleted your thread. That didn't really work out for you now, did it?

[Wrath of the Swarm]

How is (number of correct responses)/(total number of responses) meaningless?

You don't have a position. You just keep repeating false claims in the hopes that the repetition will make them sound valid.

'Bye.

[steve74]

Originally posted by Wrath of the Swarm
How is (number of correct responses)/(total number of responses) meaningless?

You don't have a position. You just keep repeating false claims in the hopes that the repetition will make them sound valid.

'Bye.

Yeah, see ya Wrath, you run away from the truth. Just in case you are still looking in, I'll try and explain it one more time for you.

Of course (number of correct responses)/(total number of responses) means something. It means, er, the number of correct responses divided by the total number of responses. But accuracy, in this context, does not mean the number of correct responses divided by the total number of responses. In this context, it means nothing.

You never said '(number of correct responses)/(total number of responses)' in your question, you said accuracy. .None of the published studies used this term.

The reason I repeat my (true) claims is that you fail, time and time again, to deal with them. Speaking of which, seven times now you have failed to defend your assertion that I'm a liar. Time and time again you lie, evade questions and contradict yourself. You think that if you evade my questions long enough, I will stop asking them. As with so many other things, you are wrong.

You have no meaningful answers to my questions. You are a sad little troll.


Edited for clarity

[Badly Shaved Monkey]

It also seems to be being conveniently forgotten that the definition of 'accuracy' seems not to be unique, remember the alternative one: TP/(TP+FP)? That seems, to the extent you can trust the internet for anything, to have been put forward by a statistician as a reasonable alternative measure that fulfills the commosense notion of 'accuracy' in the appropriate context.

Remember the other link which said that accuracy is the global subjective interpretation placed on the usual group of parameters: sens, spec, NPV, PPV.

We've seen WotS assert that his definition is the only one, but this is not supported by simple insertion of the words sensitivity, specificity and accuracy into Google to see what others say on the subject.

[Prester John]

Don't know if i missed it but did WoS have anything to say about Huntsmans' post on the previous page of this thread ?

An elementary mistake would surely destroy WoS's own claims to be knowledgable about statistics, using WoS's own logic.


Hmm talking about accuracy, whats this?

http://bell.mma.edu/~jbouch/Glossary/Precision.html

Accuracy and precision, used to desribe test results, but not binary test results!

[Wrath of the Swarm]

I responded on the previous page. Not only made a post about it, but fixed the original message (with a short blurb stating what I'd done).

Furthermore, that site certainly doesn't say that accuracy cannot be used with binary tests. But we don't consider distance from the goal, just whether the result was correct or not.

If you don't know anything about statistics, shouldn't you be keeping your mouth shut? It seems you're more likely to embarass yourself than make any kind of intelligent point.

[Wrath of the Swarm]

Originally posted by Badly Shaved Monkey
It also seems to be being conveniently forgotten that the definition of 'accuracy' seems not to be unique, remember the alternative one: TP/(TP+FP)? That seems, to the extent you can trust the internet for anything, to have been put forward by a statistician as a reasonable alternative measure that fulfills the commosense notion of 'accuracy' in the appropriate context. Except that the relative proportions of true positives and false positives will depend on the specific sample population, so that formula does not give universally applicable results.

We've been over this before.

[Prester John]

If you read what i am saying, you will note that i am not actually arguing numbers with you, i am arguing about their application, which i do know enough about.

Accuracy was the wrong term to use. End of story.

Now you made a mistake, if you apply your own rules you are now ignorant about statistics. Perhaps Huntsman should spend a page of this thread gloating about it. Maybe i should start a poll to confirm that you are ignorant, or would it only show that i am ignorant? Difficult one.

[Wrath of the Swarm]

But accuracy wasn't the wrong term to use. ceptimus established that several pages ago.

The meaning used is the simplest definition of 'accuracy' there is. It's not specialized for certain kinds of tests, or for continuous values - it's rudimentary.

You don't seem to know anything about mathematics, and you're just looking for an error you can use against me, so you pick up one of the arguments that you think sounds plausible and attack me with it.

Keep trying. Maybe one day you'll find one that works! :)

[Prester John]

Originally posted by Wrath of the Swarm
But accuracy wasn't the wrong term to use. ceptimus established that several pages ago.

The meaning used is the simplest definition of 'accuracy' there is. It's not specialized for certain kinds of tests, or for continuous values - it's rudimentary.

You don't seem to know anything about mathematics, and you're just looking for an error you can use against me, so you pick up one of the arguments that you think sounds plausible and attack me with it.

Keep trying. Maybe one day you'll find one that works! :)

It is fortunate then that i am not discussing the maths of the situation isn't it. I understand that in defined circumstances a value, that you have defined as accuracy, can provide you with the sensitivity and specificity of a test. However the term accuracy has another usage in describing medical test results

The question was about a medical test.
The term accuracy is not used to describe this type of medical test.
Thus the term was used wrongly.

You don't seem to know much about practical application of statistics in the real world. Unfortunatly in the real world clear unambiguous information is required, not hypotheticals, so that people such as myself who do not understand the complexities of statistics can actually use statistics to understand the value of results. :con2:

[Wrath of the Swarm]

Oh, really? What other meaning can 'accuracy' have than the degree of error in the results?

And why exactly does this extremely basic mathematical concept not apply to a test merely because it's medical? It's all just math.

[Prester John]

Ok well how about you show me a couple of medical tests which produce a pos/neg answer that are described only using the value of accuracy ?

Also perhaps you can explain why sensitivity and specificity are important for medical tests and the different uses of high sensitivity and high specificity tests. Its not a hard question, and i'm not trying to catch you out, i'm curious as to whether or not you are aware of how the values are used.

[Wrath of the Swarm]

Accuracy in the sense I've used it rarely appears on test results precisely because it generally can't be applied in an absolute sense.

That doesn't mean that the concept is invalidated, or that the term can't be applied at all. It requires knowledge of whether the test is correct, which generally is lacking - OR, that the test's error won't change from situation to situation, which is uncommon.

[Paul C. Anagnostopoulos]

Looks to me as if the various terms are bandied about somewhat cavalierly at times. Note how the title of this paper uses accuracy, but that word is not listed in the keywords:

http://www-rcf.usc.edu/~talonzo/pubs/AlonzoBraunMoskowitz2004.pdf

(Medical) statistics: hard.

~~ Paul

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
Except that the relative proportions of true positives and false positives will depend on the specific sample population, so that formula does not give universally applicable results.
[/B]

Exactly. QED.

[Wrath of the Swarm]

Do you even know what QED means, or do you just know it's something people say at the end of arguments?

Accuracy is not absolutely definable unless the chance of false positives is the same as the chance of false negatives. Since I defined the accuracy of the test absolutely, alpha must equal beta.

So it is demonstrated. Twit.

[Eos of the Eons]

Originally posted by Wrath of the Swarm
Twit.
More name calling. Typical

[Wrath of the Swarm]

Yet my argument is crisp, logical, and valid. Typical.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
Do you even know what QED means, or do you just know it's something people say at the end of arguments?

Accuracy is not absolutely definable unless the chance of false positives is the same as the chance of false negatives. Since I defined the accuracy of the test absolutely, alpha must equal beta.

So it is demonstrated. Twit.

That which was oughten to be demonstrated, in the leaden prose of literal translation.

What was necessary to prove was that the formula does not give universally applicable results. You did not define the sample population adequately so you cannot have your unique 'accuracy'.

"Since I defined the accuracy of the test absolutely, alpha must equal beta."

Nope. Alpha=Beta defines a unique accuracy, but not the other way around.

"So it is demonstrated." Nope. A poor translation, not even close to the right verb tense, but let's not move on to the relative merits of our educational systems!

[Wrath of the Swarm]

No, Monkey-boy. Accuracy is not defined outside of a particular sample population only if alpha is not equal to beta. If alpha equals beta, it's perfectly possible to define the absolute accuracy of the test, since that value will no longer change between different populations.

"So it is demonstrated" is a perfectly adequate definition. In contrast, your argument doesn't even approach coherence.

The person making the basic conceptual error shouldn't brag about the quality of his educational experience. Twit.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
No, Monkey-boy.Can we manage this without the tiresomely juvenile namecalling? Accuracy is not defined outside of a particular sample population only if alpha is not equal to beta. If alpha equals beta, it's perfectly possible to define the absolute accuracy of the test, since that value will no longer change between different populations.

But, if you only give the accuracy and do not define the sample populations then alpha and beta are not constrained to single values. You did not define the sample population adequately

"So it is demonstrated" is a perfectly adequate definition.

No it isn't. Go and look up your verb moods and tenses.

In contrast, your argument doesn't even approach coherence.

The person making the basic conceptual error shouldn't brag about the quality of his educational experience. Twit.

You are truly a shining wit with these hilarious insults.

Please discuss: is a phrase a true Spoonerism if it only works in spoken and not written form.

[Wrath of the Swarm]

I'll stop the puerile name-calling when you cease being a prat.

If it's possible to specify a general accuracy, alpha and beta are set: they're both equal to the general error, which means they're equal to each other. They can take on no other value without contradicting the already-established property of the test.

So by giving the accuracy, I tell you precisely what the alpha and beta are.

This is only about the fifteenth time that particular point has been made, BSM, and not only by me. How many times do we need to repeat it before you understand?

As long as you deserve the names, I'll keep using them. Twit.

[Badly Shaved Monkey]

" To check, he performed a blood test that is known to be about 99% accurate."

Nuff said. You're not going to get it now after all the explaining and it's probably time to stop feeding the troll.

Just remember your Latin homework.

I'm not sure you got the Spooner reference either. So that's another topic to be covered.


By the way folks, doesn't WotS generally imply he/she/it is American? I've not been looking that closely, but that's been my impression. Prat and twit? American or Briton? Child or adult? My reckoning has been sliding down the age groups and East over the Atlantic. Let's turn the thread to more entertaining idle speculation.

[Wrath of the Swarm]

You're right. That is enough said.

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
You're right.

Thank you.

[Wrath of the Swarm]

No - I'm talking about what I said to you. You're clearly not interested in actually thinking about the argument, and you've shown no signs of comprehending any of the points presented to you, so why continue?

Not very good at interpreting semantics, are you?

[Badly Shaved Monkey]

Originally posted by Wrath of the Swarm
No - I'm talking about what I said to you. You're clearly not interested in actually thinking about the argument, and you've shown no signs of comprehending any of the points presented to you, so why continue?

Not very good at interpreting semantics, are you?

Please try to at least pretend you get the jokes. You may feel free to cut and paste the following text if you think you've spotted something that's meant to be humorous and that way you won't stick out too badly from the rest of the class;

"Ha, ha, very funny"

Of, course, you can use it without the square bracket syntax if you really do get the joke.

[Wrath of the Swarm]

It would be difficult to respond in that manner, since all of your posts to this thread seem to have been jokes. Not amusing ones, but jokes nevertheless.

[bpesta22]

Originally posted by Wrath of the Swarm
Let's say that I went for an annual medical checkup, and the doctor wanted to know if I had a particular disease that affects one out of every thousand people. To check, he performed a blood test that is known to be about 99% accurate. The test results came back positive. The doctor concluded that I have the disease.

How likely is it that the diagnosis is correct?

It's best if you don't sit down to work it out. Just give your honest opinion about what you think is likely. If you happen to know the formula that gives the correct answer, feel free to use it.

I haven't read all the replies, but on at least the first page, no one mentions the issue of base rates, which is what this question is all about. Hey, I give this lecture every semester in an HR class.



If a test is 99% accurate, then if 100 people WITH the disease took it, there would be:

99 hits
1 miss

And, if 100 people without the disease took it, there would be

99 correct rejections and
1 false alarm.

But, the practical value of a test depends on the base rate (the % of the population that has what's being tested for).

The optimal base rate is .50. With a base rate of .50, the test would indeed be 99% accurate at identifying who has the disease and who doesn't

As the base rate departs from .50-- in either direction-- the test becomes less useful.

The base rate in this question is so small .001 that the test has almost no practical value (unless people testing positive get called back to take it a second time).

In fact the odds that you have the disease, given that you test positive, would be real small.

Consider a population of 100,000 people.

100 people have the disease, and of those, 99 would test positve.

99,000 don't have the disease, but of those 1% or 990 of them would test positive.

So, with 100,000 people, there would be 990 + 99 = 1089 positive test results.

But, the probability of actually having the disease would be only:

99/1089 = .0909

If one were to redo my example, but instead use a base rate of .50 (half have it, half don't) the probability would be equal to the test's accuracy-- 99%!


Incidentally, when I lecture on this, I think a good example is those counterfeit pen detectors. Even if they are 99% accurate, I'd argue they are worthless as the base rate for counterfeit bills in the money supply has to be real small.

So, far more often than not, when the pen says the bill is counterfiet, it will actually be real!

Sorry if someone else already answered this, but I am quite positive the above is accurate (either that, or I've misled 1000's of students!)

B

[bpesta22]

If one is talking about a test, then accuracy is the test's reliability and validity, plus the base rate (if one wants to determine proportions of hits, misses, false alarms, correct rejections).

Accuracy, with regard to psychometrics, is a well established construct. So, i don't think the original question is vague-- there's only 1 way to interpret a test having 99% accuracy (as per my previous post).

[bpesta22]

Holy crap.....12 pages on this.

Sorry I butted in.

Continue.

:rolleyes:

[Paul C. Anagnostopoulos]

Pesta, where you been the past few days, man?

What is the relationship between reliability/validity and specificity/sensitivity?

~~ Paul

[Radek]

The thread was interesting when it started, now it's just tedious and very repetative. I couldn't be bothered to read the last 4 or 5 pages so please forgive me if this was allready put forward.

In my reading of accuracy and the Dream of a thousand cats WotS should be able to calculate the sensitivity and specitifity used in the latter from the 95,3%.

Thanks Rolfe, I think you taught me a lot.

[Jeff Corey]

Originally posted by Paul C. Anagnostopoulos
Pesta, where you been the past few days, man?

What is the relationship between reliability/validity and specificity/sensitivity?

~~ Paul
I'll jump in on reliabilility/validity.
In psychololgical testing, reliabilty is the repeatability of test scores measured by the test-retest or intertest correlations. A rubber ruler has low reliability.
Validity is the determination of how well a test measures what it it is supposed to measure. There are various kinds: construct, content and predictive.
Validity has another meaning in the design of experiments. Freedom from confounding variables. Reliability is often linked to generality, which has also been termed external validity,
That's why I have avoided posting here before. Terminology may differ between disciplines,
And I use a different example than the first one here.
"If 10 percent of 1000 students use drug X and the test for the drug has a false positive rate of 10 percent, how many people who have no X in their system will fail the test if all 1000 are given the test?"

[bpesta22]

Crap, I hadda open my big mouth.

This stuff can be resolved only by appealing to signal detection theory (this was a method perfected by communication eniginers working for the military in WW II. They used it to help radar detectors tell if that spot on the screen was just noise, or actually enemy planes).

SDT can be used in any situation where two states of reality are possible (you have cancer / you don't), and you have some way-- a test-- of identifying which is which on any trial (i.e., trial being a single patient that you test).

So, as many posted earlier, on any one trial (i.e. patient tested) there are four possibilities:


http://ourworld.cs.com/Bpesta22/sdbox.jpg

The "response" is from the test (positive or negative); the "signa"l is reality / from the patient. God knows (sorry about that) the patient either has the disease or he doesn't

So, this is a detection task. Based on this test, can we detect the presence or absence of disease in our patients?

Detection tasks depend on two issues:

1) Sensitivity (unfortunately, this is not the "sensitivity" people are referring to above). Sensitivity in this example gets at the test's validity. A valid test is more sensitive than an invalid one (in the graph below, the bigger the distance between the means of the two bell curves, the greater the tests sensitivity / validity). If a test had zero validity (or reliability), the two distributions would be right on top of each other.

Finally, sensitivity is often referred to as d' (d prime)

2) Decision Making Criterion. This is up to the tester. At what point (test score) do I conclude the patient has the disease, and at what point (test score) do I assume the person does not have the disease?

This is called Beta, and is represented by the black vertical line in the graph. Any observed test score higher than beta, and the doctor will conclude the patient is positive. Any test score less than beta and the dr. concludes the patient is negative.

http://ourworld.cs.com/Bpesta22/ir2.gif

But, where do you put beta? Ideally, to be unbiased, you should put it exactly at the peak where the two curves intersect (in the graph above, Beta looks like it's about a centimeter left of being optimal.

I'm pretty sure at this point, specificity = sensitivity (as sensitivity is defined by other posters).

But, there are times when you want to be biased one way or the other.

If the detection task is hearing the linebacker husband come home while you are diddling his wife, you might
want to avoid misses at all costs (which will result, though, in making lots of false alarms). So, you would set beta way to the left (so, that any remotely loud noise will force you to get up and investigate).

In this scenario, you will make lots hits, but also lots of false alarms. To the best of my understanding, when beta is non-optimal, you could run into situations where sensitivity (which = hits/hits + fa's) differs from specificity (crs/crs + misses)

If the detection task were deciding whether an accused person is guilty, well, then we have to set beta way to the right (given our real high standard of proof beyond a reasonable doubt).

So, although we lower the incidence of putting innocent people in jail (i.e., false alarms) we raise the incidece on letting guilty go free (the miss)

Now, imagine testing not just one patient but 100's or 1000's. The results can be graphed in an ROC curve, which let you see the validity of the test, and sensitivity / specificity.

But, that's a post for a later date!

B

[Wrath of the Swarm]

Originally posted by Radek
Thanks Rolfe, I think you taught me a lot. Since everything she's said about statistics in this thread is wrong, I can easily believe you've learned a great deal from her.

[Brian the Snail]

Originally posted by bpesta22


I haven't read all the replies, but on at least the first page, no one mentions the issue of base rates, which is what this question is all about. Hey, I give this lecture every semester in an HR class.

If a test is 99% accurate, then if 100 people WITH the disease took it, there would be:

99 hits
1 miss

And, if 100 people without the disease took it, there would be

99 correct rejections and
1 false alarm.

But, the practical value of a test depends on the base rate (the % of the population that has what's being tested for.

Actually, I think Rolfe mentions this on the first page (though she doesn't call them base rates):

The thing is, to get the predictive value of a test (which is what Wrath is asking), you need to know the incidence of the condition in the population representative of the individual being tested. This is obviously higher if that population is "sick people with clinical signs typical of the disease in question". In fact, the relevant figure is the clinical probability that this individual is affected.

Wrath said this was incorrect though.

[Wrath of the Swarm]

Indeed.

First, it's the sick people in the tested sample population that's important for determining the accuracy.

Second, Rolfe is confusing having different sample base rates and performing multiple tests. If people are screened before tests are ever used on them, then the chance that the final conclusion is correct increases dramatically - more than one test is involved. This has nothing whatsoever to do with the accuracy of the test under discussion.

Third, that's just one of the things she got wrong, not the only one.

[Brian the Snail]

Originally posted by Wrath of the Swarm
Indeed.

First, it's the sick people in the tested sample population that's important for determining the accuracy.

Second, Rolfe is confusing having different sample base rates and performing multiple tests. If people are screened before tests are ever used on them, then the chance that the final conclusion is correct increases dramatically - more than one test is involved. This has nothing whatsoever to do with the accuracy of the test under discussion.

I think that was entirely her point. That the prevalence of the disease in the population representative of the patient is the important parameter, and in most situations this will be higher than that of the population as a whole. This affects the final result, making the result of the test more reliable.

Note that here Rolfe is talking about the probabilty that the final conclusion is correct, not about the "accuracy" of the test (whatever that means).

[Wrath of the Swarm]

But that is precisely my point: such a discussion is invalid.

Sorting patients according to symptoms is a test in itself, which results in a certain statistical distribution in the resulting population. Presumably the screening means that more people in the testing population will have the disease.

But that doesn't make any difference to the accuracy of the main test in this case. Moreover, we're dealing with test sample of one (as is always the case), and this person either has the disease or he doesn't. The distribution is discrete instead of continuous.

We could compute how likely it was that the patient was in a particular category, work out how accurate the test would be in such a case, and average out the accuracy across possible states to find an mean accuracy, but that figure wouldn't be very useful at all.

And again: it makes the conclusion drawn more reliable. It doesn't change any aspect of the test itself.

[Brian the Snail]

Originally posted by Wrath of the Swarm
But that is precisely my point: such a discussion is invalid.

Sorting patients according to symptoms is a test in itself, which results in a certain statistical distribution in the resulting population. Presumably the screening means that more people in the testing population will have the disease.

Well, it doesn't neccessarily have be much of a test. For example, if a particular disease is much more prevalent in white males over 50, and the patient fits into this category, then the conclusion drawn from the test will be more reliable than for the general population, no?

But that doesn't make any difference to the accuracy of the main test in this case.

Just to clarify something, in your question the "accuracy" is 99%, yes? If so, then I fail to see how the comment by Rolfe contradicts this. Bear in mind that this was before all the semantic quibbling about whether the specificity and sensitivity are equal, or whatever. What's being talked about here is the probability that the diagnosis is correct.

Moreover, we're dealing with test sample of one (as is always the case), and this person either has the disease or he doesn't. The distribution is discrete instead of continuous.

We could compute how likely it was that the patient was in a particular category, work out how accurate the test would be in such a case, and average out the accuracy across possible states to find an mean accuracy, but that figure wouldn't be very useful at all.

Aren't we talking about the probabilty that the diagnosis is correct? If not, what are we talking about?

And again: it makes the conclusion drawn more reliable. It doesn't change any aspect of the test itself.

Again, Rolfe's comment which you said was incorrect was:

The thing is, to get the predictive value of a test (which is what Wrath is asking), you need to know the incidence of the condition in the population representative of the individual being tested. This is obviously higher if that population is "sick people with clinical signs typical of the disease in question". In fact, the relevant figure is the clinical probability that this individual is affected.

To me, this seems to be saying exactly the same thing as you are saying, if you take "the predictive value of a test" as meaning the probability that the diagnosis is correct. So why did you say it was incorrect?

[Wrath of the Swarm]

Because it's not true if the test's error is blind to the subject condition. If the test is as likely to be wrong when the patient has the disease as not, the distribution in the tested population doesn't matter.

To simplify things, I didn't want to ask about a case where two tests were applied, just one. Using other factors to reach a conclusion about the probability of disease is performing another test.

[Brian the Snail]

Originally posted by Wrath of the Swarm
Because it's not true if the test's error is blind to the subject condition. If the test is as likely to be wrong when the patient has the disease as not, the distribution in the tested population doesn't matter.

Again, I'm refering to the probability that the diagnosis is correct, which is the question you were originally asking (the question to which the answer was 9.02%). Do you agree that this depends upon the incidence of the condition in the population representative of the individual being tested? It seems that from your other replies that you do, but a simple yes or no answer would be clearer.

To simplify things, I didn't want to ask about a case where two tests were applied, just one. Using other factors to reach a conclusion about the probability of disease is performing another test.

Okay, you were just considering the case where one test is used. That's fine for a hypothetical case. However, I think that Rolfe's point was that in clinical situations this is unrealistic. This seems to me to be a perfectly valid point, since it is an important caveat when extrapolating from the hypothetical case, which you presented, to real life.

[Wrath of the Swarm]

In this case, no, it doesn't. Because the test has a specified accuracy.

If there were 9,000 healthy people and 1,000 sick ones, it would give the right answer 99% of the time. If there were 9,990 healthy people and 10 sick ones, it would still be right 99% of the time.

Why is this concept so difficult to understand?

[Brian the Snail]

Originally posted by Wrath of the Swarm
In this case, no, it doesn't. Because the test has a specified accuracy.

If there were 9,000 healthy people and 1,000 sick ones, it would give the right answer 99% of the time. If there were 9,990 healthy people and 10 sick ones, it would still be right 99% of the time.

Why is this concept so difficult to understand?

I understand this concept. As I have repeated several times, I'm talking about the probabilty that the diagnosis is correct. From your original question:

Let's say that I went for an annual medical checkup, and the doctor wanted to know if I had a particular disease that affects one out of every thousand people. To check, he performed a blood test that is known to be about 99% accurate. The test results came back positive. The doctor concluded that I have the disease.

How likely is it that the diagnosis is correct?

The answer to the question in bold, "how likely is it that the diagnosis is correct?" depends upon the incidence of the condition in the population, which in this case is one in a thousand, or 0.001. However, the point that Rolfe was trying to make, which you disagreed with, is that often in real clinical situations this figure will be higher since there will be a screening as well. You can say that this constitutes an extra test, which you've discounted from your scenario, but the fact is that in real life it is the more common situation.

And to clarify again, I'm not arguing with the 99% accuracy figure.

[exarch]

Originally posted by Wrath of the Swarm
Because it's not true if the test's error is blind to the subject condition.Which you didn't explicitely state in your question.

I think a much better question to ask would be: How likely is it the doctor is going to conclude the test is correct? Because you seem to believe that likelyhood is very high, while in reality, he is probably not even going to administer the test without prior symptoms of some sort, or at least base his descision on a number of other factors as well.

So despite being an interesting puzzle question, it has very little to do with reality, which is what you intended to discuss, wasn't it?

[jj]

Originally posted by exarch
Which you didn't explicitely state in your question.

I think a much better question to ask would be: How likely is it the doctor is going to conclude the test is correct? Because you seem to believe that likelyhood is very high, while in reality, he is probably not even going to administer the test without prior symptoms of some sort, or at least base his descision on a number of other factors as well.

So despite being an interesting puzzle question, it has very little to do with reality, which is what you intended to discuss, wasn't it?

Face it, exarch, the original question was left ambiguous in order to allow for the creation of a controversy no matter what the answer.

[Wrath of the Swarm]

Originally posted by Brian the Snail
The answer to the question in bold, "how likely is it that the diagnosis is correct?" depends upon the incidence of the condition in the population, which in this case is one in a thousand, or 0.001. However, the point that Rolfe was trying to make, which you disagreed with, is that often in real clinical situations this figure will be higher since there will be a screening as well. You can say that this constitutes an extra test, which you've discounted from your scenario, but the fact is that in real life it is the more common situation. So what?

In real life, it's unusual for tests to have such a low chance of errors as well. Should we complain about that?

That was not the point I disagreed with.

[Wrath of the Swarm]

Since screening tests frequently are not prefaced by another test, whether formally or informally, and research has shown that doctors aren't very good at understanding how test results actually work, I'd say this scenario has a great deal to do with reality.

[exarch]

Originally posted by Wrath of the Swarm
Since screening tests frequently are not prefaced by another test, whether formally or informally, and research has shown that doctors aren't very good at understanding how test results actually work, I'd say this scenario has a great deal to do with reality.Screening tests are just that, screening. They have a high false positive rate, and, hopefully, a very low false negative rate. And they are often followed by a second, more accurate verification test.

And I would say that not all screening tests are performed in a lab either. Feeling for lumps, spotting a rash, problems with eyesight or balance, they can all be a tell-tale indication of something more serious, which is then checked with x-rays, MRIs, ultra-sound, EEGs, etc...

And I'm pretty sure MRIs are pretty "accurate" at spotting a lot of things, especially if you're looking for them, but it's just not possible to give everyone an MRI every 6 months just to make sure they're not ill. So other tests are employed, with high false positive rates, to make sure people who DO have a lethal affliction might be helped in time. Something which you would probably oppose because of the "unknown dangers" involved :rolleyes:

Basically, you are assuming the doctor will tell anyone who has a lump or a mole that they have cancer.

[Wrath of the Swarm]

No, I am not assuming that.

The point of the question was to illustrate that many people do not understand how to draw conclusions from such tests.

[sickstan]

Rolfe is right. Sensitivity and specificity, and by corollary, true positives and true negative rates cannot be determined by the word "accuracy", defined as (tests = condition) / (tests != condition). Sorry Wrath.

[Wrath of the Swarm]

Yes they can. Accuracy cannot be defined in the absense of a known testing sample unless alpha and beta are equal to accuracy and each other.

If you'd like to claim otherwise, you could try posting an example. I've already demonsrated that, under the given circumstances, no other values for alpha and beta are possible. If you can find a flaw in the argument, or a counterexample, that would defeat my position quite readily.

[yersinia29]

Originally posted by Wrath of the Swarm
research has shown that doctors aren't very good at understanding how test results actually work

Link please