Test scoring

[Luciana]

In Brazil, some positions within the government are highly coveted, and examinations must be prepared for the 100,000+ that will be taking them. Normally, that number is for exams that demand only a high school degree. Those who require specific college degrees have less people trying but are also considerably more difficult.

The easy one is the common multiple choice: 5 options, choose one. 1 point if you get it right. End of story.

The difficult one is a bit more creative. Sentences are offered and you must judge if they're Right or Wrong. But it gets more complicated because if you get an incorrect answer, a point is deducted. There is also the option of leaving the field blank, in which case at least you don't lose points.

Let's say there are 10 sentences. I get 7 right, 2 wrong and 1 blank. Grade: 5, because the 2 wrong ones deduct from the 7 I got right.

Historically, 60% is passing grade for such exams. So if someone gets 6 right and 4 blanks, this is passing grade. My earlier example would mean the person would not have passed. Therefore, there is the wisdom of knowing what you don't know. Never in the history of this exams someone got more than 85% correct, and this guy was way past the median.

What exactly could this thing try and accomplish -only get things more difficult? Because what happens is that risking an answer comes with a penalty. Someone who knows less, but is more prudent, can pass, while someone who gets many questions right does not. Has anyone heard of this? Do you have similar systems in your countries?

[kittynh]

wow! I dont know, but it sounds tough Luciana. You are so brave and strong to tackle this! YOU GO GIRL!!

[drkitten]

What exactly could this thing try and accomplish -only get things more difficult? Because what happens is that risking an answer comes with a penalty. Someone who knows less, but is more prudent, can pass, while someone who gets many questions right does not. Has anyone heard of this? Do you have similar systems in your countries?

This is fairly common in the large-scale standardized tests in the United States (such as the SAT, given as a college entrance examination, or the GRE, given as a graduate study entrance examination). The idea is to discourage random guessing.

[LordoftheLeftHand]

Most tests that I've taken that had a penalty for guessing were scored as follows:

Correct answer= 5 points
Blank= 2 points
Wrong= 0 points

Interestingly enough with this scoring system guessing at a true/false question still pays off slightly.

LLH

[Chaos]

In Public Law, the exam has a multiple choice test - 20 question with 3 answers each, of which anywhere from 0 to 3 are correct.

Checking an answer that ought to be checked: 1 point
Not checking an answer that ought not to be checked: 1 point
Anything else: 0 points

A passing grade is 40 out of 60.

A year ago when I took that test, there was a failure rate of 70-75 percent. :eek: For the record, I scored slghtly below 40 ( I don´t know the exact score any more), and I´m taking the test again this year.

At this exam, just for guessing (or not checking anything) you will probably get a failing grade, but not one so bad that you couldn´t use a (hypothetical) better-than-necessary grade in Private Law (which is a classic questions-and-answers test) to balance the failing grade*.

*We´re graded from 0 to 20 points, 10 being necessary to pass; a grade of 7, 8 or 9 is a passing grade if the average of both Public and Private Law grades is 10 or better. I got 9 in Public Law and 10 in Private Law. :(

[Luciana]

This is fairly common in the large-scale standardized tests in the United States (such as the SAT, given as a college entrance examination, or the GRE, given as a graduate study entrance examination). The idea is to discourage random guessing.

Following Lord's example (oops :D), the system here is:

Right: 1 point
Blank: 0
Wrong: -1 point.

So, in short, 50% right and 50% wrong means zero!

Is that like the ones you're mentioning? The problem I see with this system is that there's a very fine balance between choosing which ones you'll choose to remain blank and which ones you'll risk. If you dare, you're screwed. But if you're too prudent, then you'll kill your chances completely. Then it gets to a point, at least in that small margin between 59-61% (where a large number will fall) that it's a matter not of how much you knew the subject (how many you got right), but how wisely you gambled!

Of course I'm talking of 200+ questions here, and time restraints, so it's not like you can build an equation to estimate probabilities on how well you fared and if you should dare more or less...

[drkitten]

Following Lord's example (oops :D), the system here is:

Right: 1 point
Blank: 0
Wrong: -1 point.

So, in short, 50% right and 50% wrong means zero!

Is that like the ones you're mentioning?

Yup. Our hypothetical gibbon would get a score of zero.

The problem I see with this system is that there's a very fine balance between choosing which ones you'll choose to remain blank and which ones you'll risk. If you dare, you're screwed. But if you're too prudent, then you'll kill your chances completely.

Um, yes. But I think we disagree on whether or not this is as big a problem as you see it being.

First off,... mathematically, you're no worse off guessing than you are leaving an answer blank. Even if you're only 51% confident of your guess, you're still better off guessing than leaving it blank. Our hypothetical gibbon would get a zero whether he took the test or not. So I think it's overstating to say that "if you dare, you're screwed."

Second, there are enough questions on the test that luck is not likely to play a big role in your score; I may be able to guess two or three coin flips successfully, but forty? There are enough questions that "how wisely you gambled" really does become a question of your knowledge of the subject rather than the luck of the draw.

I guess I'm not really seeing the problem here.

[kittynh]

Do they count more than the test scores?

Are personal interviews and education levels also considered?

There is a good book called, "The Examination" about the Chinese practice of official exams for government positions.

[Luciana]

Second, there are enough questions on the test that luck is not likely to play a big role in your score; I may be able to guess two or three coin flips successfully, but forty? There are enough questions that "how wisely you gambled" really does become a question of your knowledge of the subject rather than the luck of the draw.
I guess I'm not really seeing the problem here.

The problem is that you can guess how many correct answers you can give. You do mock tests, you study hard. As you said, with that many questions, it's a matter of knowledge and not luck. So students know beforehand if they can get 50%, 60%, 70% right.

The "luck" part comes with those questions you're not sure about. If you're too prudent, you might pass up chances of scoring right, but also, you will have less wrong answers to nullify your correct ones. But if you're too prudent, you can kill your chances completely. You answer 70%, leave 30% blank. But then you find you commited a few mistakes, a mere 5%. Your score is 60% now, very close to the edge.

If you know a lot, that can boost your confidence... then you start risking a bit too much, get a handful of ones wrong... and you're out.

I'm actually conflicted whether I like this system or not. I only know one thing - you concentrate on the strategy during the test itself, and that's a great anxiety-builder right there. High school was so cool, I'd just be ticking with no worries at all. Good times...

[Luciana]

Do they count more than the test scores?

Are personal interviews and education levels also considered?

No, unless it's for educational fields, then titles, papers, publications will mater. Normally, those with the highest grades will fill up the slots, that's the norm in government positions.

In the position I'm trying for, those who pass the first phase go on for another set of 8 exams: 5 hours each, time enough to write 5 essay questions (600-650 words each). The exams are done over month, every Sat and Sunday. So the exam I'm describing above is just to determine the Top 300 who will go to the other phases.

There is a good book called, "The Examination" about the Chinese practice of official exams for government positions.

No, I've heard of books on Chinese torture. Are they related? :eek: :D

[drkitten]

The problem is that you can guess how many correct answers you can give. You do mock tests, you study hard. As you said, with that many questions, it's a matter of knowledge and not luck. So students know beforehand if they can get 50%, 60%, 70% right.

The "luck" part comes with those questions you're not sure about. If you're too prudent, you might pass up chances of scoring right, but also, you will have less wrong answers to nullify your correct ones. But if you're too prudent, you can kill your chances completely. You answer 70%, leave 30% blank. But then you find you commited a few mistakes, a mere 5%. Your score is 60% now, very close to the edge.

I guess I still don't see the problem.

For any individual question, if you have a greater than 50% chance of guessing right, then it is in your best interest to answer the question. If you think that a given answer is more likely to be correct than incorrect, you should answer it that way, unless you're so out-of-it that you can't even tell a good answer from a bad answer. If your instincts are good, then you'll guess better than chance, which means that your good answers from guessing alone are will more than offset your bad answers, which means that your score will be boosted.

And, if you think about it, "good instincts" are a valuable trait to have, so they're well worth measuring.


If you know a lot, that can boost your confidence... then you start risking a bit too much, get a handful of ones wrong... and you're out.

I don't see this. Even random guessing -- explicitly pulling out a coin and flipping it -- is expected to have no net effect on your score.

The people who will be penalized by "guessing" are the people who guess less accurately than random chance. In other words, the people who not only don't know the answer, but the people who believe, with confidence, that the wrong answer is actually the right one. And those are exactly the people I don't want to hire. I'm cool with someone knowledgable but cautious. I'm cool with someone who can make good guesses under pressure. But the last thing I want is someone who will, confidently and with self-assurance, screw up by the numbers.

"I see dumb people...they're everywhere. They walk around lik everyone else. They don't even know that they're dumb."

[Luciana]

Not checking an answer that ought not to be checked: 1 point


uh??

A passing grade is 40 out of 60.

What's up with grading systems not based on a 1 to 10 or 1 to 100 scale?? Why 40/60 and not 66 minimum out of 100? Much more simple when you look at it! :p

[drkitten]

uh??

Maybe this will help explain (as far as I can read it).

Question: Which of the following are are equal to 4?

A: 2+2
B: 3+1
C: 3-1

"Obviously" answers A and B are correct, and C is incorrect. You get one point for correctly labelling answer A as correct. You get another point for correctly identifying B as correct. You get a third piont for correctly labelling C as incorrect.

If I said that A (and only A) is correct, I would get one point, one for my correct identification of A, one for my correct identification of C (as incorrect), but losing one for misidentifying B.


What's up with grading systems not based on a 1 to 10 or 1 to 100 scale?? Why 40/60 and not 66 minimum out of 100? Much more simple when you look at it! :p

Because there are sixty possible points, so why bother to do a conversion?

[Luciana]

The people who will be penalized by "guessing" are the people who guess less accurately than random chance. In other words, the people who not only don't know the answer, but the people who believe, with confidence, that the wrong answer is actually the right one.

Now I got you. When faced with, say, 30 questions he's not sure about, a good student will weigh in the probabilities - hmm, "I'm almost sure I know this one, I'll risk it", or "no way, what's that about, I'll leave it blank". You're dealing with 10%, 50%, 90% possibilities here. A bad student is always be flipping coins. Over a range of questions, the good students will always have the advantage.

Basically, you're setting up a scale of "not knowing" also.

"I see dumb people...they're everywhere. They walk around lik everyone else. They don't even know that they're dumb."

:D There was a "One Million Reais" show in Brazil. They got their contestants randomly from the millions who sent letters. Some questions were easy, some were quite difficult. I used to wonder why people who were barely literate applied. The majority of those went back home with the $300 minimum after making fools of themselves on TV. Only then I got - they're so dumb they don't know they're dumb...

[Luciana]

If I said that A (and only A) is correct, I would get one point, one for my correct identification of A, one for my correct identification of C (as incorrect), but losing one for misidentifying B.

So it's actually very much like the system I described, that is, one wrong will nullify a correct answer.

Because there are sixty possible points, so why bother to do a conversion?

Why not design a test with 100 points, keeping the proportions? The decimal system is a very intuitive one, and they're "transparent" percentages by themselves.

[drkitten]

Why not design a test with 100 points, keeping the proportions? The decimal system is a very intuitive one, and they're "transparent" percentages by themselves.

Because percentages per se aren't especially useful numbers.

In this context, the passing score of 40/60 is about the maximally intuitive thing you can have. There are sixty questions, and you need to get forty of them correct. What need for percentages?

[drkitten]

:D There was a "One Million Reais" show in Brazil. They got their contestants randomly from the millions who sent letters. Some questions were easy, some were quite difficult. I used to wonder why people who were barely literate applied. The majority of those went back home with the $300 minimum after making fools of themselves on TV. Only then I got - they're so dumb they don't know they're dumb...

Exactly. If I'm picking someone for a job, the one I really want is the one who can do the job. But if I can't get him, I will at least settle for the one who knows he can't do the job, and who calls for help.

He who knows not, and knows not he knows not -- he is a fool, shun him.
He who knows not, and knows he knows not -- he is simple, teach him.
He who knows, and knows not he knows -- he is asleep, waken him.
He who knows, and knows he knows -- he is wise, follow him.

[Luciana]

Because percentages per se aren't especially useful numbers.

In this context, the passing score of 40/60 is about the maximally intuitive thing you can have. There are sixty questions, and you need to get forty of them correct. What need for percentages?

Because the decimal system exists and it simplifies things! Unless you prefer feet and miles over the metrical system, then why not adapt grades to the decimal system also? Percentages also allow you to compare scores across different systems if applicable. Emphasis on applicable, because I know comparisons between two different school systems can be highly subjective. But let's say you need to compare. To do that, you'll have to work the percentages, no matter what. Even if the student needs to compare his scores throughout his academic life, he will have to get down to percentages. But the bottom line is - the simpler, the better!

[drkitten]

Because the decimal system exists and it simplifies things!

Um, no. I mean, yes, the decimal system exists, but it doesn't always simplify things to use it.

Just as an example, if I have a forty volume set of encyclopedias on my shelf, I should number the volumes 1-40; I shouldn't re-normalize them to cover the range 1-100 by making each volume equivalent to 2.5 "virtual volumes" (volume 2.5 -- A-Amer. volume 5 -- Ames-Bal.)

If I'm expecting a shipment of twelve boxes of widgets, and I only get eleven, it's more useful for me to know that I'm missing one box than to know that I'm missing 9.9% of the shipment.

How many shoes do you need to find when you get dressed in the morning? 2, or 10 points worth, with each shoe (left or right) counting for five points?

When you're dealing with discrete countable objects, the most natural and intuitive representation for the count of items is the count itself.

Why not adapt grades to the decimal system also? Percentages also allow you to compare scores across different systems if applicable.

Except it's not applicable,. and making it look like it's applicable can be a very bad thing indeed.

Percentage grades are meaningless. I can set a final examination one year where the pass mark is 80%, and set a different, more difficult, test in the same class the following year where the pass mark is 50%. Trying to compare scores across years is not only meaningless, but outright dangerous -- 70% in year two is a very good student indeed, while in year one it represents outright incompetence.

There is simply no benefit to be gained by rescaling the scores.

[LordoftheLeftHand]

There was a "One Million Reais" show in Brazil. They got their contestants randomly from the millions who sent letters. Some questions were easy, some were quite difficult. I used to wonder why people who were barely literate applied. The majority of those went back home with the $300 minimum after making fools of themselves on TV. Only then I got - they're so dumb they don't know they're dumb...

{derail}
Well I don't know much about the economy in Brazil but:

If you’re having trouble providing for yourself or your family; 300 (units of currency) might be worth making a fool out of yourself.

LLH
{/derail}

[Chaos]

uh??

Say, for example, you have a question that goes:

"Luciana Nery...

A) is a really nice person
B) lives in Sao Paolo
C) loves parrots"

You get...
... 1 point if you check A
... 1 point if you check C
and 1 point if you do not check B


What's up with grading systems not based on a 1 to 10 or 1 to 100 scale?? Why 40/60 and not 66 minimum out of 100? Much more simple when you look at it! :p

There are 20 questions with 3 answers each, for a total of 60 answers; since, if you flip a coin (head = check answer; tails = do not check answer), you will probably get about 30 points, obviously it is not a good idea to give a passing grade for 30 points.
They probably decided not to convert it to a 0-100 scale perhaps, afterwards, they´re going to have to convert it to the 0-20 grade scale anyway. FYI, most other exams are on a 0-90 point scale, because exams go for 90 minutes, so students know that they should not take more than 1 minute per each point that a question is "worth".

[Dogdoctor]

Yeah taking those right minus wrong test can be difficult. When I was at University of Hawaii I had my first right minus wrong test (Graduate Records Exam being the second). I had a genetics class and the final exam was right minus wrong something the instructor was working on and we were the guinea pigs. The exam was supposed to be 3 hours long and he gave us 4 hours during which time no one left the room. I went by his office after I finished all my finals to see what grade I got and the instructor was busy trying to figure out what to do with the results of his test. I asked him how I did on my test and he said pretty good about 70%. A little alarming since I needed all A's to ge into vet school. Then he asked me what grades I got in my other classes. I told him mostly A's which was true and he said ok I am going to make your grade 100% and scale the class from there. I guess the next nearest score was down another 10%. Tough test. Then I took my GRE which is a graduate level test to get into various graduate schools which was right minus wrong. That test I have no excuse for not doing well yet only scored average in the English portion of the test (3 parts, English, math, logic). Probably the reason I scored so low in the English portion was that my English language abilities are actually average. I actually left 20 or so questions unanswered in the English portion and answered every question in the other parts. Right minus wrong discourages guessing a little so they can eliminate those who are good at guessing the answers and look for those who know what they know.

[SixSixSix]

I think that the main reason for deducting points for wrong answers is to discourage guessing. Depending on what sort of position they're after, it may well be that they're interested in determining whether someone is the sort of person who will admit, "I have no idea" (by leaving the question blank) or the sort of person that will pretend to know while actually having no idea (by guessing).

I'm certainly not accusing you of this Luciana! Just trying to reason it through.

[blutoski]

I think that the main reason for deducting points for wrong answers is to discourage guessing. Depending on what sort of position they're after, it may well be that they're interested in determining whether someone is the sort of person who will admit, "I have no idea" (by leaving the question blank) or the sort of person that will pretend to know while actually having no idea (by guessing).

I'm certainly not accusing you of this Luciana! Just trying to reason it through.


This is correct. It's to eliminate score-padding from guessing. Most questions, even if they're out of 5 multiple-choice options, really come down to two likely options (educated guessing). That's a 50% chance of getting the right answer, even if you're not sure.

Consider the student who only knows 30% of the material. If he's guessing on the other 70% with a 50% chance of getting correct answers, his final score would e 65% without a 'wrong guess' penalty.

[Luciana]

There is simply no benefit to be gained by rescaling the scores.

So forget entirely about comparisons. Why should a 0-4 scale be better than a 0-100? It's not, and it's less intuitive.

Some prefer and think in terms in Farenheit instead of Celsius. However, Celsius has the beauty of having a freeze point at 0 degreees and boiling point at 100. Ok, someone will say - oh, only at sea level. Ok, but even then, 0-100 is a nice reference and much more intuitive than Farenheit.

In terms of reward, it's much better to make full points at 100 than 4!

Chaos said:

FYI, most other exams are on a 0-90 point scale, because exams go for 90 minutes, so students know that they should not take more than 1 minute per each point that a question is "worth".

At least there is some logic behind that scaling, even then it's still arbitrary, why not 80 questions/1 point, 10 questions/2 points each? You'd still be true to your time scale. And max would be 100.

[aerosolben]

I thought I'd point out that with most US exams that I'm familiar, the penalties for incorrect answers are set such that random guessing yields the same score as not answering the question. So for 5 question multiple choice, you lose .25 points for an incorrect answer. So even eliminating one answer makes guessing worthwhile (some tests probably scale this up to two questions or so). Losing a whole point seems quite harsh.

[SixSixSix]

Well, the other partner in this relationship is a school teacher, so she knows better than me, but from way back when I was in Oz high school (some time shortly after the Jurassic era) I don't believe the standard Tertiary Entrance Exams deducted points for wrong answers. (Mind you, there wasn't a large percentage of multiple choice either - if I recall correctly).

The only tests I can recall taking that definitely did do something similar were the Westpac Maths Competition and the IBM Science Competitions that they (at least used to) issue to high school students over here.

[Kiless]

Well, the other partner in this relationship is a school teacher, so she knows better than me, but from way back when I was in Oz high school (some time shortly after the Jurassic era) I don't believe the standard Tertiary Entrance Exams deducted points for wrong answers. (Mind you, there wasn't a large percentage of multiple choice either - if I recall correctly).

The only tests I can recall taking that definitely did do something similar were the Westpac Maths Competition and the IBM Science Competitions that they (at least used to) issue to high school students over here.

First paragraph - got that correct. No deductions on the TEE multichoices.

Second paragraph - got that correct. They do deductions, although no points deducted if you just plain don't answer.

[Luciana]

So for 5 question multiple choice, you lose .25 points for an incorrect answer. So even eliminating one answer makes guessing worthwhile (some tests probably scale this up to two questions or so). Losing a whole point seems quite harsh.

Yes, it's harsh, that's why it's only used at college level, as far as I know.

The downside of this method is that adds a lot of tension to the exam. It is disheartening to think that a wrong one can kill one that you applied a lot of thinking to solve.

Also, there's the psychological effect of lowering everybody's grades. While that, in itself, does not favor anyone, it's a bummer to get a low nominal grade if you're not used to that. A true downer.