Hopefully, between akuo's help and Emily's advice re wgetgui, I might be able to create BAT's that will automatically download only specific image 'sets'

So - this thread is for the contents of little BAT's that download something specific, for instance, All MRD Navcam frames and so on.

If you've managed to make a wget command line that works for a particular mission / dataset - post it in here so that we dont end up downloading the whole interweb

Doug

Well, for starters, I asked Myche McAuley at the PDS how to go about downloading Galileo data. Since he is An Authority, I'll post his full message to me here...


Maybe if we develop some questions here I can go back to him with a list of questions all at once. He is delighted that there are people Out There interested in the data and he wants to help.

--Emily

For those on mac, SimpleWget seems to be a good GUI wrapper if you don't want to visit the command line. (You will still need to install wget as OSX releases since 10.2 include curl instead)

And for the not so l33t haxx0rs amongst us, (*ahem*) here is a link to a binary download of wget so you don't have to compile your fine self.

Something tells me this thread is going to generate quite a few hard drive purchases....

I have also had success with WebGrabber on the Galileo PDS site listed above.

These need to be put in a folder with wget, and tha folder needs to be wherever you want it to sprout all the folders. But these all work. I'll edit this in a minute to let you know if they're working w.r.t not downloading already aquired files.

If we were really clever - we'd end up with a zip which contains all of these ready to go. The zip can be sponsored by Maxtor (my 250 gig external is petitioning for divorce on grounds of an extra marital affair with this - http://www.komplett.co.uk/k/ki.asp?sku=314941&cks=PRL - tempting )

I have them getting MRD's - they're the engineering camera version of RAD's as far as I can tell - if anyone knows better, let me know

Doug

Having done the above - then the attached might be useful...

I had the mer1ho_0xxx , 2ho, 2no and 1no folders on the root of a drive called H

If you unzip the attached into the root of H - you will find img2png.exe ( hope you dont mind me repackaging it with these little bat's Bjorn - img2png is from here - http://www.mmedia.is/bjj/utils/img2png/ ) - imgbat.bat (my bat to run img2png on a load of files, and then tuck the raw img's into a 'raw' folder to keep things tidy.

AND - four bats - one named after each of the folders above.

Now all this HAS to run in the root, and HAS to be on a drive called H:. If you can dump it all onto an external drive and set that to H: ( control panel, admin tools, computer management, disk management ) - or any drive ( that isnt C: obviously) then do it - otherwise you'll have to open each bat and search and replace every h: for a whatever: you're stuff is on.

THEN

Just run them - it'll plough thru the downloaded data - copying my bat and img2png into each folder, running it, then backing up and doing the next folder.

I'm sure there is a more glamorous way of doing it - but I did the brain-dead way of having a massive long bat that will simply do this for 1500 sols worth of data. 1500 was big enough to not have to worry about rewriting it in the near future, but no so big that the BAT file ends up too large.

I've checked, and as long as you have those four folders on the root of H - it works beautifully. Expect about an hour for a ho data set, and about two hours for a no data set. - Since about 1030am, I've done two HO's and one N ( and that's in 3 hrs 26)

Hope these are usefull. Combined with the wget, and the bats above - this setup will let you download and process about 30,000 images.

Just as a double check - you need to have the highlighted files in the root of a drive called H to run them as they are.



Doug

Now - here's the thing - can we modify those command lines to ONLY do sols 450+ ?

Doug

Resurrecting a really ancient thread here, I thought I'd post a couple Wget command lines I've been using....

To get a complete Cassini (or any other) volume on the PDS Imaging or Rings nodes, turns out you have to be a little bit obnoxious and tell Wget to ignore the robots.txt file. For example:

wget -r -np -e robots=off http://pds-rings.seti.org/volumes/COISS_0011/

will get you the volume of calibration data that Bjorn's IMG2PNG software wants to have on hand for image calibration.

Also, I've been querying Bjorn's database to find all images of a moon where Cassini was within a certain distance. This is complicated slightly by the fact that optical navigation images have a target of SKY instead of the actual moon target. This query spits out URLs for all the images of Iapetus taken close enough that Iapetus is at least 118 pixels across (1 pixel per degree at the equator).

SELECT 'http://pdsimg.jpl.nasa.gov/data/cassini/cassini_orbiter/' & LCase(coiss.VOLUME_ID) & '/' & coiss.FILE_SPECIFICATION_NAME AS URL
FROM coiss
WHERE (((coiss.TARGET_DISTANCE)<2082200) AND ((coiss.TARGET_NAME)='IAPETUS')) OR (((coiss.TARGET_DISTANCE)<2082200) AND ((coiss.TARGET_NAME)='SKY') AND ((coiss.TARGET_DESC)='IAPETUS'))
ORDER BY coiss.IMAGE_TIME;

Then I just export those query results to a text file, call it iapetus.txt, and do

wget -i iapetus.txt

Piece of cake.

--Emily

Hmm - anyone know why those wget bat's I wrote previously would no longer work?

Doug