Is more capacity always a good thing?

[EHLO]

My computing life started with the Commodore 64 and BBC Micro, and I've followed the growth in computing capacity and storage space with constant amasement.

After trying to filter out all the embarrassing tracks on my iPod (the ones that always seem to pop up when entertaining guests) it occurred to me that within the next 10 years or so we'll reach a point where your average PC or convergent device will be able to store more information than any average person could ever consume in their lifetime.

For example, the typical PC can already store more text than you could read in a lifetime. A Terabyte or so will store more music than you could listen to in a lifetime, and around 500 Terabytes would store more video than you could watch in a lifetime.

I've also noticed that file sharing technologies like bitTorrent have spawned a new breed of consumer who buy and use technology for the sole purpose of "collecting" information - downloading, cataloguing and storing anything and everything they can get their hands on, reagrdless of whether they'll get a chance to use it.

I don't know if this trend is any different to collecting physical media like books, LP's, CD's or DVD's but the sheer capacity that is becoming available is entirely new, and it seems to me that the climax of the consumer IT revolution may be nothing more than everybody having access to their own copy of Ferris Bueller's Day Off, but never getting around to watching it.

Any thoughts?

[Marquis de Carabas]

The technology required to give oneself access to more information than can possibly be consumed has existed at least since the printing of the first library card.

[MRC_Hans]

Now, extra capacity might become useless, but I cannot imagine how it could actually become detrimental (except by bogging you down in garbage, of course).

In fact, I have long suspected that the eventual downfall of Moore's Law may well come not from technological barriers, but from those of usefulness.

Just as an exampl,e I have two stationary computers in my home. One is a 3 year old 2.66GHz machine with 120GgB hard disk. Since I am not playing any leading-edge games, I have absolutely no intention or need to replace, or even upgrade, it in the near future.

My other stationary is an ancient (AD 1998) 400MHz PII, with a measly 8GB disk. While I do consider it obsolete and really only retain it because it runs some games that my grandsons still like, it would really be perfectly adequate for internet browsing, word processing, and spreadsheet work. If I was simply needing it for student work, I would have absolutely no motive to change it.

In other words, there may well come a time when they make yet another fantastic machine, then find that nobody but a few technology freaks and scientists want to buy it. At that point, I think we will see the old split between standard desktop/laptop computers and professional/scientific/workstation computers reappear.

Hans

[EHLO]

The technology required to give oneself access to more information than can possibly be consumed has existed at least since the printing of the first library card.

Agreed, perhaps my point was more related to a natural limit of useful personal storage capacity.

What would be the point of personal multi Terabyte storage if you could never consume all the content (not to mention getting it in the first place).

Or is this like the (possibly made-up) Bill Gates quote that computers would never need more than 640K of RAM. I guess if the capacity is available someone will find a way to use it

[EHLO]

Now, extra capacity might become useless, but I cannot imagine how it could actually become detrimental (except by bogging you down in garbage, of course).


:) That pretty much sums up my iPod problems... It's too easy to stick a whole album on when only a couple of tracks are actually any good.

[Angus McPresley]

For example, the typical PC can already store more text than you could read in a lifetime. A Terabyte or so will store more music than you could listen to in a lifetime, and around 500 Terabytes would store more video than you could watch in a lifetime.
[...]
I don't know if this trend is any different to collecting physical media like books, LP's, CD's or DVD's but the sheer capacity that is becoming available is entirely new, and it seems to me that the climax of the consumer IT revolution may be nothing more than everybody having access to their own copy of Ferris Bueller's Day Off, but never getting around to watching it.


This sounds to me like a sort of cognitive bias -- not in tune with what Charles Stross would refer to as the coming "agalmic" society, and a holdover from outdated thinking in terms of scarcity.

It's not wasteful to not watch or listen to something. What's the harm in having it on your computer, if it's freely available, and you have enough storage?

Personally, I have dreamed of having every song ever produced available to me on my (preferably wearable) computer without having to wait for it to download. If you're thinking, "But you could never listen to it all!" you've missed my point entirely.

[Soapy Sam]

The downside is losing data- not because it's not on there somewhere, but because there was too much data to file efficiently at the time and now you can't remember the filename, or date, or any usable search criterion.
Digital photos are my problem, but I imagine the same may apply to vid or music.
Also, there are too many copies. I actually do take time to label some pics, but I already backed those up on two drives and a dvd, so now have labelled and unlabelled copies. Guess which I lose?

[Marquis de Carabas]

What would be the point of personal multi Terabyte storage if you could never consume all the content (not to mention getting it in the first place).
The point of storage of information need not be consumption. The point is access. I am ignorant of precisely what information I may need or desire to know in the future. Why not store as much of the stuff as I can in an easy to access location, so that when something comes up, I have it at hand?

[EHLO]

This sounds to me like a sort of cognitive bias -- not in tune with what Charles Stross would refer to as the coming "agalmic" society, and a holdover from outdated thinking in terms of scarcity.

I looked up "algamic" and "Carles Stross" and still don't know if I should be offended?

It's not wasteful to not watch or listen to something. What's the harm in having it on your computer, if it's freely available, and you have enough storage?
I agree, it's not wasteful just pointless apart from being able to say something like "Hey, I have every song ever produced available to me on my wearable computer".

Personally, I have dreamed of having every song ever produced available to me on my (preferably wearable) computer without having to wait for it to download. If you're thinking, "But you could never listen to it all!" you've missed my point entirely.

I'm not sure how to differentiate between the "immediate access" to any information we may possibly want, and the personal storage of that information. If I had a personal copy of the entire internet it wouldn't make it more useful.

I'm going to have some dinner and think about it.

[Marquis de Carabas]

I'm not sure how to differentiate between the "immediate access" to any information we may possibly want, and the personal storage of that information. If I had a personal copy of the entire internet it wouldn't make it more useful.

Certainly it would. You would still have access to your personal copy even in situations you could not access the internet itself. Modem blows up? You still have the information. Somebody's server goes down? You still have the information. You are on a camping trip to BFE? You still have the information (hope you have good battery life on your laptop). You become oppressively paranoid about hackers? You still have the information.

[EHLO]

OK, maybe I've been looking at this the wrong way. Network load for sychronisation aside, is the culmination of consumer IT a place where everyone has instant access to every piece of digital content ever produced on their own independent storage device?

Now I'm going to babble a bit…

If we assume a practical limit on the amount of information any individual can consume in their lifetime, can we devise an optimal networking solution to maximise accessibility and minimise network traffic?

Would this not tell us the maximum useful size of a given individuals storage capacity?

As an aside, could I start a rumour that the bitTorrent surge caused by the release of the next series of Battlestar Galactica will crash the internet?

[Rasmus]

Or is this like the (possibly made-up) Bill Gates quote that computers would never need more than 640K of RAM. I guess if the capacity is available someone will find a way to use it

That is one part.

"More film than you could ever watch" assumes that the film has, e.g. a specific resolution.

Maybe it will be possible one day to produce 3D-films that allow the viewer to freely move his viewpoint around. Maybe not, but just imagine how much more data that would take.

The transition from VHS to DVD increased storage capacity. Now movies ship with better resolution and overall quality, multiple languages and comments by the directors, actors and what not.

This is also about what you can chose to use.

Of course, I do think there will be limits of what will ever seem reasonable. But I don't think we're anywhere near yet. (And maybe I am wrong, too. Maybe the future does hold possibilities and reasons for me to simulate several universes from the comfort of my own desk.)

[roger]

The internet contains more information than I can ever consume. Therefore, the internet is too big. Umm, no. :D

Just one example, google earth consumes about 150TB of data (source (http://www.gearthblog.com/blog/archives/2006/09/news_roundup_google.html)). I find google earth to be extremely lacking in data. In the future, I want to be able to zoom down to street level, be able to look around in 3D, actually be able to read street signs, etc. Hundreds of TBs won't begin to cover that kind of storage. This is one tiny application, and I've yet to imagine any kinds of enhancements to it: complicated search algorithms, etc.

Now, we could quibble and argue about networks, bandwidths, etc. If everything is available online, then you don't need the local storage.

We discard, or don't bother to collect so much data today because the storage mechanisms are too cumbersome or don't exist. Imagine a personal recorder surgically embedded in our forehead - 24x7 recording, a 48MB image, 60 fps. That's 90,000 TB in one year. Why on earth record at that detail and frame rate? Why not? Why should we lose all of our data? Why not have access to everything that happened in the past? Why not be able to recall and replay all conversations? This is just a trivial application, the kind of thing we might do once millions of TB becomes throw away memory (like a 8mb chip is these days). We haven't begun to imagine the kinds of things we can do with massive data storage because the options are not yet available to us.

We once couldn't imagine the need for more than 6 computers worldwide. Then, we couldn't imagine why anyone would need a PC. What for, gonna compute missile trajectories in real time? Balance your checkbook on the thing? How absurd, what a waste of time, and think of how difficult that would be compared to just writing the number in the little ledger attached to the checkbook. Writing a letter? Give me a break. You've gotta type it in, and we all know that word processors, if they existed, would encourage bad writing because of how easy it is to edit. Then you gotta generate the punched cards, carry them over to a printer, not dropping them, run them into the printer, get a printout which is cumbersome because the paper is 17" wide, post it, and put it in the mail. Soooooo much easier to just pick up a pen. And then you have to either throw away the card stack, or store it someplace and label it. Good luck remembering what you wrote though, and certainly you aren't going to take hundreds of stacks of cards and run them through the printer to try to find out what you wrote to somebody. Whereas with a letter we can just make a carbon copy with carbon paper and put it in our file, and it is instantly and forever readable. No, we will never use a computer to write letters. Don't be silly.

[tsg]

There's one of those weird laws that I can't find right now that says something to the effect of "there is no filing cabinet so big that it won't eventually be filled." The same goes for computer storage. The more storage you have, the less selective you have to be about what you put in it.

[drkitten]

What would be the point of personal multi Terabyte storage if you could never consume all the content (not to mention getting it in the first place).

Well, one big point is simply storage about storage, or information attached to information. I have a dozen cookbooks on my shelf; I may have more recipes in my house than I will cook in the remainder of my lifetime. But a) I don't know which ones those are, and b) it would be extremely cost-ineffective to try to buy 2/3 of a cookbook because I "know" I will never use the recipe for Sweetbreads Amandine on p. 314.

I have a DVD on my shelf with a soundtrack in English, Spanish, French, and Chinese. I will, in all likelihood, never hear 3/4 of the sounds on that DVD. But it was easier to get a multi-lingual disk (and store the excess information) than a monolingual one.

With enough storage space, the computer can proactively get everything I might want someday, instead of only the stuff that I do want today.

Really, it's like a library card (as has already been mentioned). The librarians make sure that they've got the books people might want, instead of waiting for me to ask for any one specific book.

[drkitten]

Certainly it would. You would still have access to your personal copy even in situations you could not access the internet itself. Modem blows up? You still have the information. Somebody's server goes down? You still have the information. You are on a camping trip to BFE? You still have the information (hope you have good battery life on your laptop). You become oppressively paranoid about hackers? You still have the information.

I know a number of scuba divers that would love to have a fish-identification dataset at twelve meters. Do you know what the wireless reception is like forty feet down?

[Rat]

I have 2.5 TB available on my main machine, and it's more than three-quarters full. There's almost no video stored on there of any sort, either. While I never filled my first PC's hard drive (all ten MB of it), my next one was 1.2GB, and I filled that in no time. It has gone progressively upwards ever since, and I don't see any reason to expect that to change just because I can't define exactly what's going to take it up.

[greymatters]

One thing to remember is that, as storage grows, it's not that we'll have more than we need.

Instead, the extra capacity will encourage new types of files and file formats to be developed, and those will start pushing the capacity limits even further.

Think about it...how many movie files did you save to disk on your Commodore 64's 5" floppy?

For example, maybe future file formats will allow movies to branch in different directions based on user input, or something. The earlier suggestion of a 3-D movie is a similar example.

[SphereGuy]

The technology required to give oneself access to more information than can possibly be consumed has existed at least since the printing of the first library card.


I don't think this is a valid comparison. A library is used by the whole community, each with their own requirments and areas of interest. Likewise for the internet. A personal computer is (as I read in this example) personal, for one person.

[jimlintott]

We'll need lots of capacity when it becomes illegal to delete any files on your computer. (http://www.wired.com/politics/onlinerights/news/2007/08/coupons)

[Jimbo07]

One thing to remember is that, as storage grows, it's not that we'll have more than we need.

Instead, the extra capacity will encourage new types of files and file formats to be developed, and those will start pushing the capacity limits even further.

Think about it...how many movie files did you save to disk on your Commodore 64's 5" floppy?

For example, maybe future file formats will allow movies to branch in different directions based on user input, or something. The earlier suggestion of a 3-D movie is a similar example.

The storage of every current PC on one future one...

The bandwidth of every current network for one future one...

... the applications will be limited only by imagination. Perhaps that's a good question. Can we imagine a situation where out ability outstrips our imagination?

ETA: on bandwidth, I think current technologies (fiber) are already showing a potential bitrate limit and it will be costly to widen the pipes. Storage has and will grow faster.

[Marquis de Carabas]

I don't think this is a valid comparison. A library is used by the whole community, each with their own requirments and areas of interest. Likewise for the internet. A personal computer is (as I read in this example) personal, for one person.
If I had been making a point about availability to multiple users, then it would have been quite invalid. However, I was just illustrating that access to more information than one can consume is not a bad thing, nor is it new.

As to the question of whether one person should have all that information stored locally with only personal access to it, well, why not?

[steve s]

Instead, the extra capacity will encourage new types of files and file formats to be developed, and those will start pushing the capacity limits even further.


Agreed. Back in the early '90s I started getting into 3D graphics as a hobby. Back then I saved files in 8-bit, then eventually 16-bit format to keep the file size down. Of course they looked pretty crappy with all the color banding. Then 24 and 32 bit came along. Wouldn't it be great to save a rendered image with all the 3D info encoded? You could rotate the scene without any lag for re-rendering.

Steve S.

[SphereGuy]

If I had been making a point about availability to multiple users, then it would have been quite invalid. However, I was just illustrating that access to more information than one can consume is not a bad thing, nor is it new.

As to the question of whether one person should have all that information stored locally with only personal access to it, well, why not?

I see what you're saying. Technically, we all have access to anything on earth all the time. It's just a matter of getting to it.

I agree with the "why not?", after all it's not like if I have every book ever written on my hard drive that I'm depriving the world of books. It just may have been a wasted effort to save them all in the first place if I only read 2 of them.

On a side note, when I got my first Mac, a Performa 450, it had a 100MB hard drive and the guy who sold it to me told me I'd never fill it up.

[EHLO]

Think about it...how many movie files did you save to disk on your Commodore 64's 5" floppy?


5" Floppy... luxury... I was stuck with the cassette drive!

Maybe my concern is purely a management issue - the inherent cost associated with collecting and looking after such huge amounts of information. This is a separate issue from, say, streaming information on-demand from some remote source.

Bah. I don't really know what my problem is - a side effect of the horror of discovering Bonnie Tyler on my iPod.

[webfusion]

Anyone here wanna buy my venerable mac powerbook 1400 with 800MB Hard Drive? (it is filled with only 350MB of files and applications, including the 15.2 MB MacOS)

[Terry]

What would be the point of personal multi Terabyte storage if you could never consume all the content (not to mention getting it in the first place)

having more than you can possibly consume would still be useful if you had a convenient way of extracting what you needed at any particular moment. Which is why Google is a successful company, I guess.

[tsg]

We'll need lots of capacity when it becomes illegal to delete any files on your computer. (http://www.wired.com/politics/onlinerights/news/2007/08/coupons)

I like the comparison to DVD Jon and DeCSS. They really don't get it.

[portlandatheist]

Great topic, it got me thinking:
1) Audio resolution has long ago reached a maximum of sorts. I think that a CD has a resolution of 24 bits and for all practical purposes, that’s good enough. Now you can spend thousands of dollars on better methods of reproducing that signal (good speakers, good receiver) but there isn’t much need to improve the information resolution itself. It won’t be long before Hi-Def video reaches a limit where, even though better resolution and better DPI is possible, it won’t improve the overall experience much because the human eye won’t be able to discern the difference. There’s still a ways to go but it won’t be that long before there will be no need to upgrade your video card anymore and where video files reach a max in terms of MB/second of video. I’m sure there will be something to upgrade from HD-DVD and Blue Ray in the future but soon the only upgrade necessary will be reduction in size/usability than improvement in video resolution quality
2) I think bandwidth is a much more important factor than storage capacity. If there was a video library of every movie ever made available on the net, there wouldn’t be much need to store it locally. You could also store your music and videos remotely. The $100 laptop goal of laptop.org (for every child a laptop) the computer doesn’t even have a hard drive and it will be easy to create a remote hard drive space for people using such laptops.
3) We can’t imagine what storage demands we’ll have in the future. It may become common practice to store our entire personal genome on our computers as well as that of other people and other species. As we add storage capacity, we find ways to use it. Still, a lot of this information could be centrally stored.
4) Breakthroughs in technology will be software and our ability to discern meaningful information from the massive amounts of data we accumulate

There, that’s my crystal ball for ya

[Cuddles]

It depends what you plan on doing with the capacity. If all you are doing is storing music, text and video for personal use then we have pretty much reached the point where we don't need any more. But what about all the other uses?

What about databases with people's information? No-one will ever want to read through it, but you need it all there so that when you need one specific file you can get it, even though most of them will never be read. What about games? They use far more space than just words and pictures. The more space available, the more the graphics, AI, area covered, general realism and so on will be improved. What about technical data? A fairly coarse measurement of vibration in a machine can take a terabyte for just a few days worth of data. No-one could ever hope to read all the data we have collected, but it can still be analysed on a computer and provide useful results.

If you only consider a few limited uses then we had all the capacity we needed back in the 60s. If you consider everything computer storage is used for now, there is no end in sight to how much more we could use.

[Rasmus]

Since you mention AI for games: Think *that* would ever turn out to be a possibility for computers. How much storage might a genuine AI need just to hold a conversation with a user.

Just how much storage (and processing speed) might it take to enable us to design a first, primitive AI system?

[Cuddles]

I think true AI would actually take less storage than trying to fake it. It's the difference between knowing how to work something out and just memorising the answers. A program with the rules of language would take up less space than a list of all possible responses. I think intelligence would take a surprisingly small amount of space, it's giving it the memory to be able to learn and interact meaningfully with the world that would need more.

[Rasmus]

I think true AI would actually take less storage than trying to fake it. It's the difference between knowing how to work something out and just memorising the answers.

I am by no means an expert on this ...

... but even if you didn't "memorize all the answers" I would assume something like AI could easily start out with loads and loads of meta-information stored on language alone. (I don't know if a reasonable measure exists for the amount of information even a 4 year old posesses just to function.)

A program with the rules of language would take up less space than a list of all possible responses.

Yes, because all the actual rules of a language fit on a few pages of paper. But that has very little to do with ai, doesn't it?

I think intelligence would take a surprisingly small amount of space, it's giving it the memory to be able to learn and interact meaningfully with the world that would need more.


Well, yes. That I can agree with. I was thinking of an ai trained to a degree where one could interact with it, rahter than a blank one, so to speak, that would still need ot learn "everything".

[Ryokan]

...it occurred to me that within the next 10 years or so we'll reach a point where your average PC or convergent device will be able to store more information than any average person could ever consume in their lifetime.

That's what the salesman said when I bought a pc with a 240Mb harddrive in the very early 90's. I would never be able to fill it up.

Hah!

[Saliva]

Lots of capacity just makes me disorganize the whole disk structure. I have a total of 500 gigabytes of hard disk space, and even with 2 operating systems, a mega collection of games, and lots of music, it still won't give me a single error. As for it doing bad stuff, the 'locate' command is very popular in my shell-history.

Anyway, you might not be able to watch 500 TB of
standard defenition DVD compressed movies in your lifetime. But you could always upscale them to 4000 x 3000 resolution, 8.2 surround sound, and no compression! You might have a chance then...