How to download a list of files?

[AK-Dave]

I am trying to find a program that will let me specify a list of image files (either from a text file list or a list that was cut-and-pasted from a text file) for download. For example, I would like to be able to make a list of files like the following and download them without having to manually right-click and save each one. Some of the file names are static (do not change on a daily basis and can be re-used, as long as they are saved in a different folder each time the download list is executed). For example:

(Last 5 hours from weather.com – Download would have to be executed every 5 hours to collect all the data in for 24-hour period)
http://image.weather.com/looper/archive/aksat_720x486/1L.jpg (http://image.weather.com/looper/archive/aksat_720x486/1L.jpg)
http://image.weather.com/looper/archive/aksat_720x486/2L.jpg (http://image.weather.com/looper/archive/aksat_720x486/2L.jpg)
http://image.weather.com/looper/archive/aksat_720x486/3L.jpg (http://image.weather.com/looper/archive/aksat_720x486/3L.jpg?1202858391661)
http://image.weather.com/looper/archive/aksat_720x486/4L.jpg (http://image.weather.com/looper/archive/aksat_720x486/4L.jpg)
http://image.weather.com/looper/archive/aksat_720x486/5L.jpg (http://image.weather.com/looper/archive/aksat_720x486/5L.jpg)

or

(Last 24 hours from the GOES-West loops, in 1 hour increments)
http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/0.jpg (http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/0.jpg)
http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/1.jpg (http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/1.jpg)
http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/2.jpg (http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/2.jpg)
http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/3.jpg (http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/3.jpg)
…
http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/23.jpg (http://www.orbit.nesdis.noaa.gov/smcd/opdb/aviation/loops/realtime/gifs/pac_wv/23.jpg)

or

(Current Surface Map; current 24 and 36-hour significant weather maps, updated every 6 hours (or 12, I can’t remember right now))
http://aawu.arh.noaa.gov/fcstgraphics/sfc.gif (http://aawu.arh.noaa.gov/fcstgraphics/sfc.gif)
http://aawu.arh.noaa.gov/fcstgraphics/sigWx24.gif (http://aawu.arh.noaa.gov/fcstgraphics/sigWx24.gif)
http://aawu.arh.noaa.gov/fcstgraphics/sigWx36.gif (http://aawu.arh.noaa.gov/fcstgraphics/sigWx36.gif)

Some of the file names are sequential date-time groups, such as from the Japan Meteorological Agency. I can write a simple script to generate the lists. For example:

(Last 24 hours from the MTSAT-1R, in ½ hour increments)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120500-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120500-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120530-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120530-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120600-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120600-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120630-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120630-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120700-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120700-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120730-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120730-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120800-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120800-00.png)
http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120830-00.png (http://www.jma.go.jp/en/gms/imgs/3/infrared/1/200802120830-00.png)


Any suggestions? I would prefer a freeware application, and it could be for Mac OSX or WinXP.

Thanks,

- David

[Yalius]

Run Firefox; install an add-on called DownThemAll; create the list of links like you just did, and invoke DownThemAll to, well, download them all.

[Agular]

Here's another for Firefox, just for images, Unlinker:

http://twinturbo.org/unlinker/

http://i231.photobucket.com/albums/ee105/Agularis/unlinker.jpg

[69dodge]

wget

[richardm]

wget

... sounds like a better choice to me, since you can include it in your script and call it in Windows Scheduler or whatever the Mac equivalent is at your required interval. Having said that I've never used wget myself, so "sounds like" is the operative phrase.

[ddt]

wget... sounds like a better choice to me, since you can include it in your script and call it in Windows Scheduler or whatever the Mac equivalent is at your required interval. Having said that I've never used wget myself, so "sounds like" is the operative phrase.
I'd recommend using wget too. It has a wealth of options to control what you want to download. For having a file with a list of URL's:

wget -i urllistfile

Another nice option is -r for recursive retrieving. Giving it one URL to download, it will then download every other URL pointed at from that one, and so on. You can restrict, of course, the depth you will go to, or that it will only download URL's from the same website. It will also put them nicely in the same directory structure as the URL's indicate.

The Mac equivalent of Windows Scheduler is of course true and tried Unix cron.

[richardm]

The Mac equivalent of Windows Scheduler is of course true and tried Unix cron.

Oh yes, I suppose it would be :D

[Terry]

wget or curl get my vote too

[AK-Dave]

Thanks for the replies. Right now, I'm leaning towards using wget for the actual downloading and using AutoIt (http://www.autoitscript.com/) to do the scripting.

- David

[Paul C. Anagnostopoulos]

Can't you just enter this command into your command window?

copy "http://image.weather.com/looper/archive/aksat_720x486/1L.jpg" .


~~ Paul