Split: The WebBot Project: Can the internet predict the future?

[paximperium]

This a split from this thread:
http://forums.randi.org/showthread.php?t=125890
Where The Professor attempted to claim that some caller in a radio talk show made a 100% super prediction. While being a lie, the talk show did make an interesting claim.

From the talk show transcript:
Approximately 81minutes:
Starts stalking about some stock tracking system(webbot project) that allegedly can predict other things such as 9/11 and the Columbia tragedy.
Quick transcript:
"This program predicted something in January 2008. A big life changing event will occur on October 7th that will make 9/11 pale in comparison. This event will last for 4-6months. This is preceded by something from 9/22-27th a precursor event, likely a stock market crash probably related to the AIG collapse."

I can't find a sane website that specifically states what this WebBot project does and I can't find this project's website(if it has one). All mention of this project belong to survivalist, end of the world, conspiracy theory website only.

One of the best(I use this term very loosely here) is a survivalist website.
http://urbansurvival.com/simplebots.htm
Tries to explain how this thing works:

He described how technology worked. A system of spiders, agents, and wanderers travel the Internet, much like a search engine robot, and look for particular kinds of words. It targets discussion groups, translation sites, and places were regular people post a lot of text.

When a "target word" was found, or something that was lexically similar, the web bots take a small 2048 byte snip of surrounding text and send it to a central collection point. The collected data at times approached 100 GB sample sizes and we could have used terabytes. The collected data was then filtered, using at least 7-layers of linguistic processing in Prolog, which was then reduced to numbers and then a resultant series of scatter chart plots on multiple layers of Intellicad ( http://www.cadinfo.net/icad/icadhis.htm ). Viewed over a period of time, the scatter chart points tended to coalesce into highly concentrated areas. Each dot on the scatter chart might represent one word or several hundred.

To define meanings, words or groups of words have to be reduced to their essence. You know how lowest common denominators work in fractions, right? Well the process is like looking for least common denominators among groups of words.

The core of the technology therefore is to look at how the scatter chart points cluster - condensing into high "dot density" areas which we call "entities" and then dissolving or diffusing over time as the entities change. Do a drill down into a dot and you get a series of phrases...

What becomes obvious when reading about the technology is that it sometimes reads a bit like the I Ching (the Chinese Book of Changes) because the technology doesn't come out and say "go look for a terrorist attack over there" What it does is gives phrases that would be associated with how people talk about an event, or more accurately, how they change their speech to reflect their thought processes because of an event (after).

The web bot technology apparently taps in to an area of preconscious awareness. It's here that you run into the ramifications of Dean Radin's work at the Boundary Institute and the work of the Princeton Global Consciousness Project - both of which Art has talked about on his show.

This thing sounds just like a pattern data mining system. Just like the "Bible Code", this thing find patterns in the Internet in order to make predictions of the future.

Let the discussion begin.

[DevilsAdvocate]

Somehow I just KNEW a thread like this would start.

The webbot project mentioned is Half Past Human. It is a pattern data mining system. But it is not like the "Bible Code".

The website www.halfpasthuman.com doesn’t really display anything publicly. All publicly available information for the project is at www.urbansurvival.com. That site has some deeper explanations of how the bot works, if you take the time to look through everything. It is not organized very well.

My understanding is that it was created to help predict the future value of certain stocks. I can see where this application would be useful. It still seems to be heavily centered around the market and investments.

My understanding is that applying this technology to predicting other things is just for kicks and grins. Even the creators confess that it is usually wrong. But there are some surprising hits. So they do take some predictions seriously. And there may be good reason.

Basically, it takes a key word, searches the Internet, and looks at the words and phrases shortly before and after the keyword. Now you have just a bunch of meaningless words and phrases. So it reduces those down to specific concepts. Adjectives and adverbs are attributed applied to nouns and verbs that, in turn, can be applied to a specific concept. Sort of like a “reverse thesaurus”. Within the data that has been mined, you can pick up dates or date ranges or vague references to time “soon, next week, later, September, end of the year, etc.” and plot out a time frame when the things are supposed to happen. You can also classify a general vibe—good stuff, bad stuff, etc. You can even find good keywords by looking for the most used words.

In the end, the output you get is a concentrated version of the “buzz” on the Internet.

For stock trading, this can be very valuable. Stocks are worth what people are willing to pay for them. You don’t know the value of a stock until it is sold. But before actual trading, the value of the stock is how people feel about the stock, which is probably also what people are saying about the stock in chat room, forums, blogs, newspapers, etc. If you can pick up a “buzz” from the Internet about how people feel about the stock, you can know the value before the stock is actually traded.

It is basically just a breakdown of “here is what people are saying”. Knowing that helps your chances in stock sale, but can it predict the future?

Probably. To some degree. If a lot of people are talking about something with the same type of attitude and referencing the same general timeframes, then they probably know something that is about to happen. So it may well happen.

Imagine playing fantasy baseball. You are looking undervalued players or good players that nobody else knows about. Find the names of the players that are coming up often on the Internet. Now run the bot and get words associated with those players. Run it through filter. Are people saying, in general, the player is good or bad? If good, are they saying now or later or next season? So you can come up with a list of undervalued players who people are saying will be good now.

Will it come true? Maybe. The bot can only tell you that lots of people are making this prediction or talking about a certain concept in a certain way. The bot only works if there are lots of people making certain predictions or talking about certain concepts in a certain way.

Can it predict the future? No. But it can predict what people predict for the future. That helps in investments or stock trading where the value of your stock is based almost solely on what people predict the future value of the stock will be.

But it can be useful as a predictor of other things. It can tell what people are talking about and what their intentions are. If people follow thorough with what they have said, you have, in a way, predicted the future.

I might get to the issue of the “October 7” prediction later (if I haven't already killed this thread with my long post).

[Cuddles]

using at least 7-layers of linguistic processing

They put it through Babelfish 7 times?:)

Even the creators confess that it is usually wrong. But there are some surprising hits.

I don't think I've ever seen a more classic example of counting the hits and ignoring the misses. If it's wrong most of the time and you have no way of telling which times it is actually right, the whole thing is a complete waste of time.

But it can predict what people predict for the future.

Can it? Is there actually any evidence that this is the case?

Edit: Of course, this would also fall victim to the classic problem of economics - any prediction system that actually works will immediately stop working because everyone will try and use it, thereby changing what actually happens and making the predictions useless. The only way to actually have a working system is to make sure no-one else ever finds out about it. It's certainly not impossible to predict stocks but if you see a system for doing so advertised on the internet, you can guarantee that that's not one of them.

[sophia8]

Edit: Of course, this would also fall victim to the classic problem of economics - any prediction system that actually works will immediately stop working because everyone will try and use it, thereby changing what actually happens and making the predictions useless. The only way to actually have a working system is to make sure no-one else ever finds out about it. It's certainly not impossible to predict stocks but if you see a system for doing so advertised on the internet, you can guarantee that that's not one of them.And it becomes a feedback loop, picking up internet chatter about a specific prediction (like this one) because everybody is talking about how there's this amazing super-computer that's made this prediction......

[paximperium]

To summarize, this webbot is basically a gossip and "mood" conglomerator of the internet and its creator's somehow believe this automatically gives them the ability to make real predictions?

Somehow there is a logical hole the size of a hummer I can drive through using that thought process.

[Ivor the Engineer]

I predict a Republican win in November; McCain to become too ill to continue being president; Palin to start a new Cold War 'cause she knows where Russia is.

[geni]

To summarize, this webbot is basically a gossip and "mood" conglomerator of the internet and its creator's somehow believe this automatically gives them the ability to make real predictions?


Its a hopelessly overcomplex way of doing that as well. A far less complex method would be looking at search terms or:

http://stats.grok.se/en/top

(which some ajustment that will tell you what the net in general was thinking about in august).

[hamelekim]

Supposedly it predicted 9/11 and the quake in China. It also predicted a big release on October 7th, which was the day that the Stock market world wide began to crash, and is still in the process of doing so. This language lasts for 5 months, so I'm guessing it will bottom out in 5 months or so? In any case, we will have to wait and see if that is the case.

They also have language that seems to say that an earthquake will occur in the north western US, just off the coast by Seattle/Vancouver around the 12th of December. I guess if that happens we can say that there is something going on with it.

[Cuddles]

Supposedly it predicted 9/11 and the quake in China.

Evidence?

Thought not.

It also predicted a big release on October 7th, which was the day that the Stock market world wide began to crash, and is still in the process of doing so.

The stock market hasn't crashed, so I guess that prediction was wrong. The stock market, along with the economy, of most countries has slumped over several months, but given that several months covers rather a lot of time before 7th October, that doesn't really make it look particularly impressive.

[tyr_13]

Does this mean we can change the predictions by speculating more on specific topics?

Hmmmmmm, I predict that women will start to believe I'm the best lover in existence (and that counts other galaxies).

Now I just have to repeat that about, half a million times, and some people who think this bot means something might consider it to possibly be true. So maybe it isn't worth it.

[xinit]

They also have language that seems to say that an earthquake will occur in the north western US, just off the coast by Seattle/Vancouver around the 12th of December. I guess if that happens we can say that there is something going on with it.

This, taken on its own might be interesting. However, were there other predictions? Let's ignore for the moment that the area is in an active quake zone, and that there are a couple small quakes a year, improving the odds of picking a random date rather significantly.

Maybe there were other predictions buried in the volume of guesses?

1 December: earthquake in Seattle
2 December: earthquake in the Pacific Northwest
3 December: earthquake in Vancouver
4 December: earthquake off the coast of Washington State
5 December: earthquake near Seattle
6 December: earthquake in the western portion of Pacific Northwest
7 December: earthquake near the Rocky Mountains
8 December: earthquake in Redmond, Washington
...

Eventually you have to be right. What are the misses?

[tyr_13]

That would suck major for December if it weren't a crock of steamy fresh ********.

[DevilsAdvocate]

To summarize, this webbot is basically a gossip and "mood" conglomerator of the internet and its creator's somehow believe this automatically gives them the ability to make real predictions?

Somehow there is a logical hole the size of a hummer I can drive through using that thought process.That is basically the gist. Geni’s example of data on Wiki searches is good. But I think the idea is to expand on that to get a context, opinion, evaluation, time frame, significance, etc. of those search items.

You have to take this in context. Keep in mind that the webbot was designed and is used to time value changes of specific stocks. A conglomeration of internet mood and gossip is good for that purpose. The value of a stock is based on the public mood, the public value, of the stock. But if you can detect a “private” mood, you can beat everyone to the trade changes that are coming—you can predict the future.

The webbot was applied to other purposes just for kicks and grins. And it requires a bit of interpretation. Most of the hits claimed are predictions of doom. But they seem to have almost constant predictions of doom.

So is their prediction about 10/07/08 real? Maybe sort of. They don’t expose much specifics about how the webbot works. It seems to take a great deal of interpretation.

There may have been some gossip and mood detected about a crisis around 10/07/08, but considering that this is the only really specific prediction that has (sort of) come true and the people doing the interpretation primarily deal in evaluating the stock market and there was plenty of evidence of a possible stock market crash in the later part of 2008, it isn’t exactly proof of anything.

I think the prediction was more a reasonable guess by stock market experts rather than anything that the webbot predicted.

I’ll mention again that The Professor’s guest, Shots, did not predict anything. He just mentioned the webbot project. Even the webbot did not actually predict anything. The people interpreting that data made the predictions. And they predicted the event as something like 40% military, 40% economic, and 20% “terra”. It was all economic. And it was one of many guesses by economic experts.

Remember that even the creators of this technology say that using it for this purpose is just for fun and that it is usually wrong. However, I do see real potential in this technology. I think it can have some practical uses if done correctly. I think it is a very interesting idea.