Wget Command Lines, Various presets Options

[djellison]

Hopefully, between akuo's help and Emily's advice re wgetgui, I might be able to create BAT's that will automatically download only specific image 'sets'

So - this thread is for the contents of little BAT's that download something specific, for instance, All MRD Navcam frames and so on.

If you've managed to make a wget command line that works for a particular mission / dataset - post it in here so that we dont end up downloading the whole interweb

Doug

[elakdawalla]

Well, for starters, I asked Myche McAuley at the PDS how to go about downloading Galileo data. Since he is An Authority, I'll post his full message to me here...

The Galileo collection will be a bit frustrating so, for now, you'll have to persevere with some ugly file paths.  The Galileo collection is slated for migration to the new, cleaner, directory structure in the near future.  The commands below will then change only to reflect this new directory structure.  For now, though, these instructions should get you going.

To get to the Galileo collection, try the following:

1. use your browser to look at this ugly directory: http://pds-imaging.jpl.nasa.gov/data/
You'll see a lot of go-*... directories.  Those are data set IDs and are an artifact that only a PDS data archivist could love.  We at the Imaging Node are moving away from having these be something that a user would have to know.  Unfortunately, they're in the way for now.

2. Locate a Data Set ID that is interesting to you (this could be for no other reason than it contains a CD volume you're interested in). E.g.:
http://pds-imaging.jpl.nasa.gov/data/go-v_e-ssi-2-redr-v1.0/
This data set ID refers to the Venus and Earth data.  Inside the above URL you should see 5 CD volumes.  Let's say you want the contents of go_0004.  To get that volume in its entirety, try this:

wget -np -r -erobots=off \ http://pds-imaging.jpl.nasa.gov/data/go-v_...r-v1.0/go_0004/

and stand back.    (The -np option is critical.  If you don't use it, you'll start downloading _everything_ we have!  )

This will retrieve the contents of the go_0004 CD-ROM volume to your current local directory in a subdirectory named:
pds-imaging.jpl.nasa.gov/data/go-v_e-ssi-2-redr-v1.0/go_0004/

There are some command-line switches to wget that will not create the pds-imaging.jpl.nasa.gov/data/go-v_e-ssi-2-redr-v1.0 elements of the destination directory.  E-mail me if you'd like help in obtaining these.

Maybe if we develop some questions here I can go back to him with a list of questions all at once. He is delighted that there are people Out There interested in the data and he wants to help.

--Emily

[lyford]

For those on mac, SimpleWget seems to be a good GUI wrapper if you don't want to visit the command line. (You will still need to install wget as OSX releases since 10.2 include curl instead)

And for the not so l33t haxx0rs amongst us, (*ahem*) here is a link to a binary download of wget so you don't have to compile your fine self.

Something tells me this thread is going to generate quite a few hard drive purchases....

I have also had success with WebGrabber on the Galileo PDS site listed above.

[djellison]

These need to be put in a folder with wget, and tha folder needs to be wherever you want it to sprout all the folders. But these all work. I'll edit this in a minute to let you know if they're working w.r.t not downloading already aquired files.

If we were really clever - we'd end up with a zip which contains all of these ready to go. The zip can be sponsored by Maxtor (my 250 gig external is petitioning for divorce on grounds of an extra marital affair with this - http://www.komplett.co.uk/k/ki.asp?sku=314941&cks=PRL - tempting )

I have them getting MRD's - they're the engineering camera version of RAD's as far as I can tell - if anyone knows better, let me know

Doug

Attached File(s)

 oppy_hazcams.bat ( 104bytes ) Number of downloads: 480
 oppy_navcams.bat ( 104bytes ) Number of downloads: 403
 spirit_hazcams.bat ( 104bytes ) Number of downloads: 413
 spirit_navcams.bat ( 104bytes ) Number of downloads: 426

 

[djellison]

Having done the above - then the attached might be useful...

I had the mer1ho_0xxx , 2ho, 2no and 1no folders on the root of a drive called H

If you unzip the attached into the root of H - you will find img2png.exe ( hope you dont mind me repackaging it with these little bat's Bjorn - img2png is from here - http://www.mmedia.is/bjj/utils/img2png/ ) - imgbat.bat (my bat to run img2png on a load of files, and then tuck the raw img's into a 'raw' folder to keep things tidy.

AND - four bats - one named after each of the folders above.

Now all this HAS to run in the root, and HAS to be on a drive called H:. If you can dump it all onto an external drive and set that to H: ( control panel, admin tools, computer management, disk management ) - or any drive ( that isnt C: obviously) then do it - otherwise you'll have to open each bat and search and replace every h: for a whatever: you're stuff is on.

THEN

Just run them - it'll plough thru the downloaded data - copying my bat and img2png into each folder, running it, then backing up and doing the next folder.

I'm sure there is a more glamorous way of doing it - but I did the brain-dead way of having a massive long bat that will simply do this for 1500 sols worth of data. 1500 was big enough to not have to worry about rewriting it in the near future, but no so big that the BAT file ends up too large.

I've checked, and as long as you have those four folders on the root of H - it works beautifully. Expect about an hour for a ho data set, and about two hours for a no data set. - Since about 1030am, I've done two HO's and one N ( and that's in 3 hrs 26)

Hope these are usefull. Combined with the wget, and the bats above - this setup will let you download and process about 30,000 images.

Just as a double check - you need to have the highlighted files in the root of a drive called H to run them as they are.



Doug

Attached thumbnail(s)

 

Attached File(s)

 batch.zip ( 437.12K ) Number of downloads: 434

 

[djellison]

Now - here's the thing - can we modify those command lines to ONLY do sols 450+ ?

Doug

[elakdawalla]

Resurrecting a really ancient thread here, I thought I'd post a couple Wget command lines I've been using....

To get a complete Cassini (or any other) volume on the PDS Imaging or Rings nodes, turns out you have to be a little bit obnoxious and tell Wget to ignore the robots.txt file. For example:

wget -r -np -e robots=off http://pds-rings.seti.org/volumes/COISS_0011/

will get you the volume of calibration data that Bjorn's IMG2PNG software wants to have on hand for image calibration.

Also, I've been querying Bjorn's database to find all images of a moon where Cassini was within a certain distance. This is complicated slightly by the fact that optical navigation images have a target of SKY instead of the actual moon target. This query spits out URLs for all the images of Iapetus taken close enough that Iapetus is at least 118 pixels across (1 pixel per degree at the equator).

SELECT 'http://pdsimg.jpl.nasa.gov/data/cassini/cassini_orbiter/' & LCase(coiss.VOLUME_ID) & '/' & coiss.FILE_SPECIFICATION_NAME AS URL
FROM coiss
WHERE (((coiss.TARGET_DISTANCE)<2082200) AND ((coiss.TARGET_NAME)='IAPETUS')) OR (((coiss.TARGET_DISTANCE)<2082200) AND ((coiss.TARGET_NAME)='SKY') AND ((coiss.TARGET_DESC)='IAPETUS'))
ORDER BY coiss.IMAGE_TIME;

Then I just export those query results to a text file, call it iapetus.txt, and do

wget -i iapetus.txt

Piece of cake.

--Emily

[djellison]

Hmm - anyone know why those wget bat's I wrote previously would no longer work?

Doug