Curiosity Image Retrieval Tools, scripts and software |
Curiosity Image Retrieval Tools, scripts and software |
Aug 11 2012, 09:18 AM
Post
#1
|
|
Member Group: Members Posts: 270 Joined: 29-December 04 From: NLA0: Member No.: 133 |
I wrote a shell script that allows you to download all images from a given sol.:
http://paranoid.dechengst.nl/files/MSLget.sh There are two rm commands in the script to clean up the tempdir. If you're afraid my script screws up your system replace them with rm -i commands. If you want any features added please let me know and I'll see what I can do. EDIT1: Added leading zeros in the directory names for easy sorting after a request for it on IRC. EDIT2: I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox. EDIT3: As RoverDriver pointed out you need to have wget on your system to use this script. The script now handles this gracefully. -------------------- PDP, VAX and Alpha fanatic ; HP-Compaq is the Satan! ; Let us pray daily while facing Maynard! ; Life starts at 150 km/h ;
|
|
|
Aug 11 2012, 01:32 PM
Post
#2
|
|
Member Group: Admin Posts: 976 Joined: 29-September 06 From: Pasadena, CA - USA Member No.: 1200 |
... I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox. I looked over and this should work also on Macs. Your script relies upon wget being installed on your system which is not always the case. That is the only dependency I see. DEChengst, this is not a criticism, just trying to be helpful to anyone who might try using your script and found a problem. Wget for Mac can be found here and it is a cool tool that you should get (no pun intended) anyway. I use it all the time to download large stuff, like HiRISE images, DTMs. Paolo -------------------- Disclaimer: all opinions, ideas and information included here are my own,and should not be intended to represent opinion or policy of my employer.
|
|
|
Aug 11 2012, 01:34 PM
Post
#3
|
|
Member Group: Admin Posts: 976 Joined: 29-September 06 From: Pasadena, CA - USA Member No.: 1200 |
As an example of using wget to download the HiRISE images, here's my script that downloads the non-map projecte, the map projected and label files to your current working directory:
CODE #!/bin/csh echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/download/PDS/EXTRAS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.QLOOK.JP2\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/download/PDS/EXTRAS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.NOMAP.JP2\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/PDS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.LBL\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh Paolo -------------------- Disclaimer: all opinions, ideas and information included here are my own,and should not be intended to represent opinion or policy of my employer.
|
|
|
Aug 11 2012, 01:52 PM
Post
#4
|
||
Junior Member Group: Members Posts: 36 Joined: 7-November 04 From: Sweden Member No.: 108 |
I've written a small too to help me download the "RAW" Curiosity images from the mars.jpl.nasa.gov site. Originally, I wrote it for my own use, but since some people here might find it useful I thought I should make it public.
The system requirement to run it is .NET 4.0. It's important to note that this is just a quick and dirty tool I whipped up this morning, so the code quality isn't exactly great. There's not a lot of fault tolerance in the code, for example. Another thing to keep in mind is that it's dependant on the current appearance of the mars.jpl.nasa.gov site. If that changes significantly, the tool will stop working. If that happens, and there's interest, I will hopefully have time to update the tool to make it work again. Anyway, here it is, Curiosity RAW Get. It works for me up to Sol 38 data. EDIT: Since I don't have a web site of my own, I had initially intended to provide public support for the tool on this board, releasing new version here and making sure this post always pointed to the latest version of the software, and handling support in the followup posts. After all posts related to software were merged into one thread, that is unfortunately no longer possible, so we'll go to plan B instead: The version included in this post might in a few days or weeks get old. To get new versions or support, contact me off this board through email instead (My email address is included in the About dialog of the program.). CHANGELOG: 0.16.38 - 2012-09-14 Added subframe support. 0.15.12 - 2012-08-18 Added thumbnail browsing mode. Added installer. 0.14.10 - 2012-08-16 Added support for the Sol 10 version of the Curiosity web site. Enabled threaded downloading. 0.13.3 - 2012-08-11 Switched from Microsoft's HTML parser to the HtmlAgilityPack parser. 0.12.3 - 2012-08-11 Fixed a problem where the Microsoft HTML parser didn't return parent element for some elements. 0.11.3 - 2012-08-11 Fixed version dependency on the Microsoft HTML parser. 0.10.3 - 2012-08-11 First release.
Attached File(s)
|
|
|
||
Aug 11 2012, 02:11 PM
Post
#5
|
|
Member Group: Members Posts: 691 Joined: 21-December 07 From: Clatskanie, Oregon Member No.: 3988 |
For windows users, I've been using Star Downloader Pro. Its a link leecher that Scans and follows links on webpages based on what filtered file extention you input. Then it filters and downloads them. In this case .jpg files. There is a 30 day free trial to try it out, works like a charm for me, so I bought it.
http://www.stardownloader.com/ |
|
|
Aug 11 2012, 02:26 PM
Post
#6
|
|
Newbie Group: Members Posts: 7 Joined: 13-April 09 Member No.: 4730 |
I'm also working in a phyton script that automate big parts of the downlading process, but as I started teaching myself programming the first week of july it won't be so automatic and will require input from the user, at least until I improve my programming skills. The good part is that anyone will be able to use it as python runs in Windows, Linux and Mac.
|
|
|
Aug 11 2012, 02:42 PM
Post
#7
|
|
Senior Member Group: Members Posts: 1669 Joined: 5-March 05 From: Boulder, CO Member No.: 184 |
Also a shell script using wget initially mentioned in another thread. Several related posts appear right afterward - might be nice to consolidate in this thread.
MOD NOTE: Good idea- done! Sounds good - the script is now post #1 by DEChengst in this thread. -------------------- Steve [ my home page and planetary maps page ]
|
|
|
Aug 11 2012, 02:49 PM
Post
#8
|
|
Member Group: Members Posts: 198 Joined: 2-March 05 From: Richmond, VA USA Member No.: 181 |
For windows users, I've been using Star Downloader Pro. I would also suggest the DownloadThemAll plugin for Firefox for those who wish. http://www.downthemall.net/ It is not at powerful as wget, but it has served me well for six to seven years now. -- Pertinax |
|
|
Aug 11 2012, 03:16 PM
Post
#9
|
|
Junior Member Group: Members Posts: 36 Joined: 7-November 04 From: Sweden Member No.: 108 |
Hm... actually... The reason why I posted my tool in a new thread was so that the link to it could be in the first post (and thus easily updated when future versions are released), and then use the rest of the thread for support and change logs (compare this to how other message boards [for example doom9] dealing with software releases work).
Is there any chance we could break this out again into one thread per tool? As it is now, I fear it won't be obvious to anyone using the tool when a new version is released. In fact, the first post in the thread called "Curiosity RAW Get" isn't even about the tool called "Curiosity RAW Get" anymore. MOD EDIT: Thread title changed to reflect a more general theme. Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage. |
|
|
Aug 11 2012, 04:09 PM
Post
#10
|
|
Senior Member Group: Members Posts: 2228 Joined: 1-December 04 From: Marble Falls, Texas, USA Member No.: 116 |
...the first version of Curiosity RAW Get. I installed it, but when I try to run it, nothing appears to happen. Actually, not nothing. I get an info window with some disclaimers, but that is it. I have .NET 4.0 running under win7. I tried double-clicking the icon and also running it from a command line. Do you have any idea what I may be doing wrong? -------------------- ...Tom
I'm not a Space Fan, I'm a Space Exploration Enthusiast. |
|
|
Aug 11 2012, 04:20 PM
Post
#11
|
|
Junior Member Group: Members Posts: 36 Joined: 7-November 04 From: Sweden Member No.: 108 |
QUOTE Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage. Okey, that's too bad. I'm afraid supporting the tool in the middle of a generic software thread is too much work for me to manage, instead, so I'll take support off this board instead. I'll update the post to reflect that. |
|
|
Aug 11 2012, 05:00 PM
Post
#12
|
|
Senior Member Group: Members Posts: 2228 Joined: 1-December 04 From: Marble Falls, Texas, USA Member No.: 116 |
It seems that it might be useful to create a subforum under MSL similar to the "Tech, General, and Imagery" section under MER. That would seem to allow for a little more flexibility with regard to specialized topics like specific software applications.
-------------------- ...Tom
I'm not a Space Fan, I'm a Space Exploration Enthusiast. |
|
|
Aug 11 2012, 07:40 PM
Post
#13
|
|
Junior Member Group: Members Posts: 43 Joined: 7-August 12 From: The Netherlands Member No.: 6493 |
I'm building a webportal for easier access to the raw images, but I'm highly time-constrained, so it's going slowly.
But the first step is there: I've put an automatically up-to-date JSON listing of all RAW images online, as a more convenient listing for (web)developers or as input for wget scripts. (With the lack of folder listings in the raw-images CDN, I've done the same HTML parsing as all of you:) You can find the JSON list at: http://msl-raw-images.appspot.com/landing There are a couple of parameters: counts -> Only list image counts per sol/camera. cam -> Only list specific camera sol -> Only list specific sol Examples: http://msl-raw-images.appspot.com/landing?...m=MD&counts (will list the count of descent images: 1522) http://msl-raw-images.appspot.com/landing?sol=1&cam=NRA (will list all right Navcam images for sol 1) The site will automatically update every 30 minutes, so expect a ~15 minutes delay compared to the original site. Hope this portal will be usefull already. I'm planning on making a webpage similar to the MER raw images as soon as possible. (hopefully within a day or two) Greetings, Ludo. |
|
|
Aug 11 2012, 08:12 PM
Post
#14
|
|
Administrator Group: Admin Posts: 5172 Joined: 4-August 05 From: Pasadena, CA, USA, Earth Member No.: 454 |
On previous missions when it's been tough to parse raw images (Phoenix being a major example) I've posted easier-to-browse indexes on the Society's site. But I'm no programmer; I always did this manually. There appear to be lots of different solutions brewing here. I would LOVE it if when one of these gets fleshed out, it could be dropped in to a subsite on the Planetary Society website. With lots of praise and credit to whomever developed the solution. I can handle making it play nicely with our site, but I can't do automated image retrieval/indexing.
-------------------- My website - My Patreon - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
|
|
|
Aug 11 2012, 08:31 PM
Post
#15
|
|
Junior Member Group: Members Posts: 36 Joined: 7-November 04 From: Sweden Member No.: 108 |
I think it'll be fairly easy to get this "fleshed out". I see mainly two problems:
|
|
|
Lo-Fi Version | Time is now: 27th September 2024 - 05:49 AM |
RULES AND GUIDELINES Please read the Forum Rules and Guidelines before posting. IMAGE COPYRIGHT |
OPINIONS AND MODERATION Opinions expressed on UnmannedSpaceflight.com are those of the individual posters and do not necessarily reflect the opinions of UnmannedSpaceflight.com or The Planetary Society. The all-volunteer UnmannedSpaceflight.com moderation team is wholly independent of The Planetary Society. The Planetary Society has no influence over decisions made by the UnmannedSpaceflight.com moderators. |
SUPPORT THE FORUM Unmannedspaceflight.com is funded by the Planetary Society. Please consider supporting our work and many other projects by donating to the Society or becoming a member. |