Opening a HUGE text file

[pgwenthold]

OK, my calculation didn't crash, but generated a warning message in the output file. Again, and again, and again...repeat probably a million times.

Fortunately, or not, the problem wasn't fatal, and the calculation finished. However, I am left with a 730 MB output file filled mostly with the repeated warning, and I need to get to the bottom to see the result. Way too big for most editors. I thought I might be able to open it in a browser, and tried Netscape and Firefox, but neither could pull it off.

So my question is, is there a way to open only the last, say, 10K of the file? Because I think the answer would be there.

It's a 48 hour calc and uses big resources, so it would be easier if I could open the result now. I could do it again and bump up some allocation to eliminate most of the warnings, but if I could figure this out now it would save a lot of hassle.

Any suggestions?

[roger]

tail.

No, I'm not being insulting, you want the tail utility. I have no idea what OS you are using, but GNU probably supports it. Just google tail utility along with your OS.

[CynicalSkeptic]

If it's winders, there's a freeware program called mtail that is the same thing.

[Wowbagger]

A hex editor might be adequate.

[ddt]

Try vim (http://www.vim.org/). AFAIK, it has no limits of the size of the file. Back in the days of 486 computers, I opened files of several MB's with it, no problems (well, that was on Linux of course).

[ddt]

OK, my calculation didn't crash, but generated a warning message in the output file. Again, and again, and again...repeat probably a million times.

On another tangent, I'd like to ask about this, as a Unix man and C programmer.

The Unix standard says that each program "out of the box" has two output channels: stdout for the regular output, and stderr for all error messages. I can also redirect them separately from the command line:
$ myprogram > outputfile 2> errorfile
The C standard also demands a program should have these two output channels "out of the box", as do all programming languages derived from or inspired by C, or any other programming language that was originally conceived on Unix.

On a Unix box, I can expect any program (if it's written by a competent programmer of course) to behave like this.

How's that on Windows?

[roger]

stdout and stderr both go to the command prompt window.

source (http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true).

[jsiv]

Yes, Windows works the same way, with the same syntax (and where they go is entirely dependent on the programmer, but by default they go to the console).

[pgwenthold]

If it's winders, there's a freeware program called mtail that is the same thing.

Yeah, I should have mentioned. I am trying to open it in Windows XP

[pgwenthold]

If it's winders, there's a freeware program called mtail that is the same thing.

mtail worked. Thanks.

(the secret was that it just loaded the last 50K)

[ddt]

stdout and stderr both go to the command prompt window.

source (http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true).
That's standard on Unix too. But they're separate channels, so you can redirect them separately.

Yes, Windows works the same way, with the same syntax (and where they go is entirely dependent on the programmer, but by default they go to the console).

Yep, see this snippet from the above MS documentation page:
You can use Cmd.exe to run a program and redirect any of the first 10 handles for the program. To specify which handle you want to use, type the number of the handle before the redirection operator.

and the rest of that page describes exactly the same things you can do with I/O channels as in a Unix (Bourne based) shell.

ETA: thanks roger and jsiv for the answers!

[jsiv]

Yeah, I should have mentioned. I am trying to open it in Windows XP
I use TextPad (http://www.textpad.com/download/), and I just tried to open a 750MB large ISO file in it as text, and it worked fine. Took about half a minute, but it worked fine. There's a free unrestricted evaluation version, so you could give that a try.

Not easy to scroll through millions of lines though. One click of the scroll bar and you move 200,000 lines down. :D

[nathan]

tail.

No, I'm not being insulting, you want the tail utility. I have no idea what OS you are using, but GNU probably supports it. Just google tail utility along with your OS.

tail -50 | more
is something I use quite often.

[Paul C. Anagnostopoulos]

Well kiss my grits. I run 4NT for my DOS box. I've never heard of tail before, but there it is. Cool!

~~ Paul

[grmcdorman]

Re 4NT: A free version of 4NT - command line only - is available, from www.jpsoft.com. I use it and like it, although because 4DOS/4NT/TCC commands implemented similar things to what was later in the Windows command line (cmd.exe), there are a few command that take slightly different options or syntax to do the same thing (e.g. FOR command syntax to process file contents instead of file names).

[/derail]

[Hatchet]

tail -50 | more
is something I use quite often.Heathen!!!
Wake up and start using the one true pager.
tail -50 | less

Cheers.

[ddt]

Wake up and start using the one true pager.
tail -50 | less

less is more :D

[nathan]

Heathen!!!
Wake up and start using the one true pager.
tail -50 | less

Cheers.

Thrp!

nathan@ivanova:17>ls -l $(which more less)
32 -rwxr-xr-x 1 root root 30 Sep 25 2008 /bin/more*
124 -rwxr-xr-x 1 root root 119 Feb 2 2008 /usr/bin/less*

Why page in 124 blocks when 32 will do?

Why mount /usr when / will do?

and more importantly: Why retrain my fingers?

[ddt]

Thrp!

nathan@ivanova:17>ls -l $(which more less)
32 -rwxr-xr-x 1 root root 30 Sep 25 2008 /bin/more*
124 -rwxr-xr-x 1 root root 119 Feb 2 2008 /usr/bin/less*

Why page in 124 blocks when 32 will do?
You recently used man anyway, which happens to use less on your system, so it's already paged in anyway.


Why mount /usr when / will do?
Duuh, you've already mounted /usr, you're working multi-user.


and more importantly: Why retrain my fingers?
alias more='less'

[nathan]

Duuh, you've already mounted /usr, you're working multi-user.

/usr isn't even a seprate partition (in this case) -- I was just messin with ya.

alias more='less'

But, that'd slow my interactive shell startup time!

I knew someone would point these out -- You may collect your gold star.

[ddt]

/usr isn't even a seprate partition (in this case) -- I was just messin with ya.
Bad practice! If you're afraid to run out of partitions, use LVM!


But, that'd slow my interactive shell startup time!
Then here's an alternative:
# ln -s /usr/bin/less /usr/local/bin/more
and now you fixed it for every user on the system :D


I knew someone would point these out -- You may collect your gold star.
Where do go to collect it?

[deep]

Try vim (http://www.vim.org/). AFAIK, it has no limits of the size of the file. Back in the days of 486 computers, I opened files of several MB's with it, no problems (well, that was on Linux of course).


No self-imposed limit; however, if vim is compiled/running on a 32-bit system, it won't be able to open anything over 2GB.