In all fairness to the woo-woo...

[Wrath of the Swarm]

The concept of "blindly" teaching a neural network is a pretty pointless one. It's not as if it makes any difference if the programmer knows what the right answers are. The networks aren't exactly Clever Hans, after all.

Although it's very odd that this character didn't reserve any data for a test run, which is standard procedure when training neural nets. For all he knows, he trained the network to associate a binary value with a number of specific charts. Veery intresking.

[Jeff Corey]

Originally posted by Wrath of the Swarm
\ Veery intresking. Arte Jonson, "Laugh In"
Where would I find stuff on training neural nets? Is it an anology to operant feedback?

[Wrath of the Swarm]

Is it like operant training? Well, in a very abstract sense, I suppose you could say that.

I recommend the following link to learn basic information about neural nets (including how they're programmed). Neil's Neural Nets (http://vv.carleton.ca/~neil/neural/neuron.html)

[Wrath of the Swarm]

Anyway, having the programmer know what response the test patterns are supposed to generate is pretty harmless. Now, if the same person drew up the horoscopes AND knew which person/profession was supposed to be associated with each one, THAT would be a problem that calls for double blinds.

[Kopji]

It was snowing for a while, but it's stopped now. I must remember to tell the kids to not water the silk flowers.

[Beanbag]

The problem with neural nets is that you're never sure exactly what the net uses to determine the various outcomes. There was a Nova program a few years back that was discussing military uses of machine intelligence, particularly for finding armored vehicles camoflagued in the terrain. A neural net was trained by showing it images with vehicles hidden, and without any hidden vehicles. They were scoring some ridiculously high success rates using scanned photographs, so they mounted the net on an aircraft and flew it over a test range to see if the results held. They didn't -- it was almost a complete flop. So, they went back and examined their "training" images and discovered that the images with the hidden vehicles were all taken on a sunny day, and those without vehicles had been shot on an overcast day. THAT was what the net had glommed onto as the distinguishing characteristic.

Regards;
Beanbag

[Mortenm]

Originally posted by Wrath of the Swarm
Anyway, having the programmer know what response the test patterns are supposed to generate is pretty harmless. Now, if the same person drew up the horoscopes AND knew which person/profession was supposed to be associated with each one, THAT would be a problem that calls for double blinds.

I have had a debate with Eskild (the astrologer in question) regarding his neural network, and he knows the desired results AND he drew up the horoscopes.

[magicflute]

GIGO

[Mortenm]

Originally posted by magicflute
GIGO

Yes, GIGO might be a good description for what is going on, but
perhaps we should explain the meaning of GIGO to the non-computer buffs among us.

From wikipedia.org:

Garbage In, Garbage Out. A reference to the fact that computers, unlike humans, will unquestioningly process the most nonsensical of input data and produce nonsensical output. Of course a properly written program will reject input data that is obviously erroneous but such checking is not always easy to specify and is tedious to write.

GIGO is usually said in response to users who complain that a program did not "do the right thing" when given imperfect input or otherwise mistreated in some way. Also commonly used to describe failures in human decision making due to faulty, incomplete, or imprecise data.

[Wrath of the Swarm]

Originally posted by Mortenm
I have had a debate with Eskild (the astrologer in question) regarding his neural network, and he knows the desired results AND he drew up the horoscopes. Knowing the results isn't a problem. Did he draw the horoscopes blindly, or did he know what category each person was supposed to be in?

[Mortenm]

Originally posted by Wrath of the Swarm
Knowing the results isn't a problem. Did he draw the horoscopes blindly, or did he know what category each person was supposed to be in?

I might not have expressed myself clearly enough. When I wrote that "he knows the desired results", I meant that knows what category each person is supposed to be in, when he makes the horoscopes and corrects the birth time.

[CFLarsen]

Originally posted by Mortenm
I might not have expressed myself clearly enough. When I wrote that "he knows the desired results", I meant that knows what category each person is supposed to be in, when he makes the horoscopes and corrects the birth time.

And he hasn't even tested it with people of unknown category.

[Wrath of the Swarm]

That's not really a problem. If he generates new data for the test without knowing what category the people are supposed to be in, or if someone else performs the calculations, everything will still work out. There will only be a potential conflict if he knows the categorical division AND computes the horoscopes.

It still won't be a very good test of astrology.

[Mortenm]

Originally posted by Wrath of the Swarm
That's not really a problem. If he generates new data for the test without knowing what category the people are supposed to be in, or if someone else performs the calculations, everything will still work out. There will only be a potential conflict if he knows the categorical division AND computes the horoscopes.

I agree, however my point is that since he only has performed training and he puts a lot of faith in his results it is important to consider the possible sources of bias in the training process.

[teddosan]

He HAS to know what the desired result (i.e. the correct classification) is for the training set in order to do the training. You can think of training a neural net like training a dog (though like all analogies this can only be take so far...) If the dog pees on your red carpet, you tell him "bad dog" and if he pees outside on the green grass you tell him "good dog." That's your training set, your red carpet inside and your green grass outside. Eventually the dog stops peeing on your red carpet and you are finished training. Well, here's the problem... Now you think your dog is trained so you take him to your friend's house but your friend has green carpet. And your dog pees on his floor. And you say, "What the heck?! I thought he was house-trained!"
Eskild's problem is that he has never taken his dog to anyone else's house, and when he does it's going to pee on the floor...

[Wrath of the Swarm]

There's no such thing as "bias in the training process". The program can't pick up on signals other than the data that are fed to it - it's not Clever Hans. Obviously the training process is intended to create 'bias' - if it didn't, the network wouldn't have learned anything - but the actions of the programmer aren't a concern. There's nothing he could have done in the programming stage that could compromise the experiment.

[princhester]

I appreciate that Randi doesn't have the time or the inclination to fiddle-arse round with wallies who want to engage in flirtatious chit-chat but never actually get down and do the business.

But in this case Eskild's excuse for not applying was that he didn't want a "split or duel motivation" ie to have the distraction of a potential monetary award.

I was a bit surprised that Randi didn't take the wind out of his sails by just saying: "Well you have to apply in terms of the standard agreement same as everybody else, just so everything's agreed in black and white, but I'm happy to strike out those parts of the agreement that would entitle you to getting the million, if you think that the money would just be a distraction".

That would have taken this guy's excuse away.

Don't get me wrong, I aspire to curmudgeonliness myself, but I don't think that what the world needs is more flimsy excuses for woo-woos, who are ever on the lookout for ways to paint Randi as overly rigid.

[apoger]

> "Well you have to apply in terms of the standard agreement same as everybody else, just so everything's agreed in black and white, but I'm happy to strike out those parts of the agreement that would entitle you to getting the million, if you think that the money would just be a distraction".


The issue (I believe) is that this would set a precedent for allowing claimants extra negotiation on the rules. Many claimants seem fanatically determined to haggle over the rules. Mr. Randi has to draw the line somewhere. Being rigid may seem grumpy, but it is a way of maintaining fairness and clarity, since everyone plays by the same rules.

The gentleman in question could just as easily refused or donated the money... as Randi suggested. I feel it is quite clear which party is being unreasonable.

If Randi had removed the award, do you think this gentleman would have then agreed to the challenge? After watching claimants squirm for years I can predict an easy NO. He was looking for an excuse, obviously since he took a fairly irrational one. Do you think the next excuse would have been any better?

[princhester]

I think there is a major difference between altering the rules in any way that could remove clarity or fairness, and merely altering them such that the claimant receives no money if they win. Given that (as I understand it) quite a few would be claimants use the excuse that "the money would detract from their art", I would have thought that Randi could pretty easily have an alternate set of rules altered in this way at hand and ready to go. I don't think this would open the floodgates to any significant extent.

I don't think for a moment that it would actually stuff up anyone's paranormal abilities to be playing for cash, but nonetheless, that lame excuse seems to be commonly used. And what's more, a common variant seems to be the suggestion that even the temptation of knowing that if the prize is won, the winner would receive the money (albeit that they could then give it away) is enough to detract from the "purity of motive" sometimes said to be necessary for the paranormal to work (how fragile it is!).

This lame dog of an excuse should be shot in the head and put out of its misery, not left to whine noisily.

No I don't think that if Randi had removed the award Eskil would have accepted the challenge. No, I don't think the next excuse offered would have been any better. I think it would have been even more ridiculous. Which is precisely why I think the claimant should have been made to make it.

[deBergerac]

It is a point in itself that the rules are never changed so that everyone will always know what applies. Besides I can imagine what will happen in case Randi agree to have someone do the test without the prise money at stake. We will never stop hearing about the psychic, astrologer, or homeopath that was offered to be tested by Randi without the ability to win the prise. That it was the claimant that wanted to be tested in that way will be forgotten and all would be how Randi was too scared to risk the million.

[princhester]

Originally posted by deBergerac
It is a point in itself that the rules are never changed so that everyone will always know what applies.

Sorry but I really think you are demonstrating precisely the type of mindless inflexibility that allows Randi's woo woo critics to argue that Randi's rules are too rigid to cope with the vagaries of their weird and wonderful alleged "talents".

How hard is it to understand the following: "I will pay you one million dollars if you can pass the test. If you don't want the money, I will just test you anyway, if you like."

Besides I can imagine what will happen in case Randi agree to have someone do the test without the prise money at stake. We will never stop hearing about the psychic, astrologer, or homeopath that was offered to be tested by Randi without the ability to win the prise. That it was the claimant that wanted to be tested in that way will be forgotten and all would be how Randi was too scared to risk the million.

If people are going to lie, they are going to lie. There's nothing you can do about it. People can say Randi cheats, that he won't offer fair tests, they can say anything, and they already do. One more thing that liars say about Randi won't hurt.

But refusing to countenance a harmless change to the usual rules that won't cost Randi anything and won't make a test any less rigorous?

Now that hurts Randi.

[deBergerac]

I do not agree but I do acknowledge that you have a point. To turn down a claimant for something else than a lack of subjecting to scientific rigor is bad. But I still think one does not need to give any more ammunition to those who slander and lie about Randi.
One way around it is perhaps if it is made absolutely clear in the rules that Randi will lose the money if someone wins the prise. It would be possible to be tested without winning the money but the money would still be lost to some charity or other if someone win.

Then if someone claims that they have been offered to take the test without a chance of getting the money it would still be obvious from the rules that it could not be because Randi was afraid of loosing the money.

Of course people would only understand that if they bothered to read the rules, but the important thing is that the money will be gone at the moment someone pass the test.

[Mortenm]

Originally posted by Wrath of the Swarm
There's no such thing as "bias in the training process". The program can't pick up on signals other than the data that are fed to it - it's not Clever Hans. Obviously the training process is intended to create 'bias' - if it didn't, the network wouldn't have learned anything - but the actions of the programmer aren't a concern. There's nothing he could have done in the programming stage that could compromise the experiment.

Normally I would agree with you, but this case is a bit special, because Eskild is in a position to influence the network during the traing process. Let me elaborate a bit:

The bias I was talking about was experimenter bias, which is an issue in this case because Eskild corrects the birth time of the testpersons while knowing whether the person is a musician or a politician.

This will in principle allow him to (consciously or unconsciously) influence the input data before it is given to the network.

As an example we could say that for the musicians he tends to make corrections that places Venus in a certain part of the ecliptica (if that's the right word), and for politicians he tends to make corrections that places Venus in another part.

This will introduce a new feature to the data, which allows the network to separate between the two groups of persons.

[CFLarsen]

Mortenm,

Wait a sec...I thought Eskild would simply enter the birth data, not the astrologically "corrected" birth data.

Are you sure of this?

[Wrath of the Swarm]

I agree that this man might be putting patterns into the horoscopes himself, but the network itself won't have problems with this. There's still no need for double-blinding in the input stage.

The problem would lie entirely with the generation of the horoscopes from the birth data.

[Mortenm]

Originally posted by CFLarsen
Wait a sec...I thought Eskild would simply enter the birth data, not the astrologically "corrected" birth data.

Are you sure of this? [/B]

Yes, I'm pretty sure. Try to read the following quote:

Eskild: Den sidste opdagelse, jeg har gjort, er, at datakvalitet er så afgørende, at programmet i de fleste tilfælde ikke vil komme med det rigtige svar, hvis der er foretaget en forkert korrektion. Korrektionen kan ikke undervurderes.

My translation:

The last discovery that I have made is that data quality is so important, that the program in most cases won't get the correct answer if a wrong correction has been made. The correction can not be underestimated.

Source (http://www.astrologyforum.dk/log/discus/html/messages/1412/1049.html#POST7947) .

You can also try to read the debate between Eskild and myself on Astrology Forum (http://www.astrologyforum.dk/) (in danish).

[Mortenm]

Originally posted by Wrath of the Swarm
I agree that this man might be putting patterns into the horoscopes himself, but the network itself won't have problems with this. There's still no need for double-blinding in the input stage.

The problem would lie entirely with the generation of the horoscopes from the birth data.

I agree, and I hope that we have now removed the misunderstandings, which it seems we have had.

[Wrath of the Swarm]

Indeed.

Upon rereading the letter, it has become clearer to me just what this potential claimant is saying. He suggests that once the network had learned from 150 of the horoscopes, it could assign the remainder with 98% accuracy.

It's not clear (at least to me) whether Randi was criticizing the drawing up of the horoscopes for not being done blindly (which is a perfectly valid complaint) or criticizing the evaluation by the network as not being done blindly (which is not).

[Mortenm]

Originally posted by Wrath of the Swarm
Upon rereading the letter, it has become clearer to me just what this potential claimant is saying. He suggests that once the network had learned from 150 of the horoscopes, it could assign the remainder with 98% accuracy.

I have to disagree. In the letter to Randi Eskild writes:

Eskild: However, there is no guarantee that Astromate will be able to generalize from the training set to new charts, which the network is not familiar with. After finishing training, I will have to run a series of tests with "unfamiliar" data to verify that Astromate is capable of generalisation.

If he had tried putting "the remainder" of the horoscopes through the network, then that would have constituted "unfamiliar" data and the network would have shown an ability to generalise to horoscopes it hadn't seen before. Since Eskild explicitly states that this isn't the case, I think we must conclude that the 98% accuracy was on the 150 horoscopes in the training set.

Furthermore, try to look at this quote:

Eskild: Der er nu gennemført 180 kørsler med Astromate (inklusive kalibrering). Dermed er jeg mere end halvvejs gennem træningssættet på 352 horoskoper. Astromate gættede kun forkert i 5 ud af de 180 tilfælde, dvs. netværket er nu oppe på en nøjagtighed på 97%.

My translation:

There has now been performed 180 runs with Astromate [the neural network] (including calibration). I am thereby more than halfway through the training set of 352 horoscopes. Astromate guessed wrong in 5 of the 180 cases, that is the network has reached an accuracy of 97%.

Source (http://www.astrologyforum.dk/log/discus/html/messages/1412/1041.html#POST16847) (in danish).

I think that this quote (and my discussion with him) supports my interpretation that the accuracy he reports is the accuracy obtained for the input he uses as training data.

I'm not 100% sure I'm correct, and the remark he makes about 180 runs "including calibration" confuses me a bit.

[CFLarsen]

Mortenm,

You translated correctly.

The last discovery that I have made is that data quality is so important, that the program in most cases won't get the correct answer if a wrong correction has been made. The correction can not be underestimated.

So, here is, immediately, an "out" for Eskild: If he fails (and I am being generous here!), all he has to do is claim that the corrections were not right.

Question is, how do we decide if a correction is correct, before the data enters AstroMate? Answer: There is no way that we can decide that, because each correction is entirely up to the astrologer - Eskild, in this case.

Originally posted by Mortenm
Since Eskild explicitly states that this isn't the case, I think we must conclude that the 98% accuracy was on the 150 horoscopes in the training set.

I agree. I understand it the way you do.

I don't like this test one little bit. It seems to me that Eskild gets to write a computer program, which will demonstrate that astrology works. Only, Eskild gets to feed the program with data, and Eskild gets to decide what the data looks like, before it enters the program.

I can do that, too, and win a million bucks.

[Mortenm]

Originally posted by CFLarsen
I can do that, too, and win a million bucks.

WRONG! You won't win the million ;).

[CFLarsen]

Originally posted by Mortenm
WRONG! You won't win the million ;).

Why not? That's what Eskild is trying to do...(OK, he's chickened out on actually applying, but....)

[Mortenm]

Originally posted by CFLarsen
Why not? That's what Eskild is trying to do...(OK, he's chickened out on actually applying, but....)

Trying isn't doing :).

[Wrath of the Swarm]

It's clearly stated that he was using a data set of 176 composers and 176 politicians. After completing a set of 150 (75 from each set), the network demonstrated a 98% accuracy.

What happened to the other 202 sets?

[Mortenm]

Originally posted by Wrath of the Swarm
What happened to the other 202 sets?

As I understand it, he hasn't worked with those yet. He has selected the testpersons for the training set, but he is far from finished with correcting and processing all of them.

[JamesM]

(Warning! Slightly technical and definitely boring neural network discussion follows)

That 98% classification accuracy is a massive warning sign. I must agree with the correspondent in the latest commentary that this almost certainly represents severe overfitting of the model.

By adding more hidden units - the neurons that lie between the inputs and output nodes - to the network, the classification rate will progress happily upwards to 100%. The resulting network is utterly hopeless at generalising to new data, of course.

To avoid this situation, it is standard practice to set aside a small portion of the training data as a monitoring set and make predictions on that during the training phase as hidden units are sequentially added. At some point the prediction results will begin to get worse (indicating overfitting is occuring) and the final model is constructed using the number of units that gave the optimum prediction.

Additionally, a good indicator of the network's predictive ability can be found by cross-validation, i.e. splitting the training data into a small number of groups, making a network using the data from all but one of the groups, and then making a prediction using the left-out group as the test set. This process is repeated, leaving out a different group, until all the groups have been predicted in a blind fashion. The resulting classification error should be reasonably reliable.

My Danish isn't quite what it could be. Are any of these details specified?

[CFLarsen]

Originally posted by Mortenm
As I understand it, he hasn't worked with those yet. He has selected the testpersons for the training set, but he is far from finished with correcting and processing all of them.

Eskild has stated that his program is ready for test in June. You might want to ask him July 1st. :)

[Mortenm]

Originally posted by JamesM
That 98% classification accuracy is a massive warning sign. I must agree with the correspondent in the latest commentary that this almost certainly represents severe overfitting of the model.

I agree, and I have previously tried to point this out to Eskild.

By adding more hidden units - the neurons that lie between the inputs and output nodes - to the network, the classification rate will progress happily upwards to 100%. The resulting network is utterly hopeless at generalising to new data, of course.

He is using a one layer Hopfield network, so there are no hidden neurons.

To avoid this situation, it is standard practice to set aside a small portion of the training data as a monitoring set [...]

My Danish isn't quite what it could be. Are any of these details specified?

It has been some months since I had the debate with Eskild, so I can't remember the details. If I get the time, I'll go back and reread the debate, so that I might answer your question - but don't count on it, because I'm having a lot of overtime at work currently.