IPB

Welcome Guest ( Log In | Register )

25 Pages V   1 2 3 > »   
Reply to this topicStart new topic
Curiosity Image Retrieval Tools, scripts and software
DEChengst
post Aug 11 2012, 09:18 AM
Post #1


Member
***

Group: Members
Posts: 270
Joined: 29-December 04
From: NLA0:
Member No.: 133



I wrote a shell script that allows you to download all images from a given sol.:

http://paranoid.dechengst.nl/files/MSLget.sh

There are two rm commands in the script to clean up the tempdir. If you're afraid my script screws up your system replace them with rm -i commands. If you want any features added please let me know and I'll see what I can do.

EDIT1:

Added leading zeros in the directory names for easy sorting after a request for it on IRC.

EDIT2:

I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox.

EDIT3:

As RoverDriver pointed out you need to have wget on your system to use this script. The script now handles this gracefully.


--------------------
PDP, VAX and Alpha fanatic ; HP-Compaq is the Satan! ; Let us pray daily while facing Maynard! ; Life starts at 150 km/h ;
Go to the top of the page
 
+Quote Post
RoverDriver
post Aug 11 2012, 01:32 PM
Post #2


Member
***

Group: Admin
Posts: 976
Joined: 29-September 06
From: Pasadena, CA - USA
Member No.: 1200



QUOTE (DEChengst @ Aug 11 2012, 02:18 AM) *
...
I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox.


I looked over and this should work also on Macs. Your script relies upon wget being installed on your system which is not always the case. That is the only dependency I see. DEChengst, this is not a criticism, just trying to be helpful to anyone who might try using your script and found a problem. Wget for Mac can be found here and it is a cool tool that you should get (no pun intended) anyway. I use it all the time to download large stuff, like HiRISE images, DTMs.

Paolo


--------------------
Disclaimer: all opinions, ideas and information included here are my own,and should not be intended to represent opinion or policy of my employer.
Go to the top of the page
 
+Quote Post
RoverDriver
post Aug 11 2012, 01:34 PM
Post #3


Member
***

Group: Admin
Posts: 976
Joined: 29-September 06
From: Pasadena, CA - USA
Member No.: 1200



As an example of using wget to download the HiRISE images, here's my script that downloads the non-map projecte, the map projected and label files to your current working directory:

CODE
#!/bin/csh
echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/download/PDS/EXTRAS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.QLOOK.JP2\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh

echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/download/PDS/EXTRAS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.NOMAP.JP2\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh

echo $1| tr "_" " "|awk '{printf "wget http://hirise-pds.lpl.arizona.edu/PDS/RDR/%s/ORB_%04d00_%04d99/%s_%06d_%04d/%s_%06d_%04d_RED.LBL\n", $1,$2/100,$2/100,$1,$2,$3,$1,$2,$3}' | /bin/sh


Paolo


--------------------
Disclaimer: all opinions, ideas and information included here are my own,and should not be intended to represent opinion or policy of my employer.
Go to the top of the page
 
+Quote Post
M_Welander
post Aug 11 2012, 01:52 PM
Post #4


Junior Member
**

Group: Members
Posts: 36
Joined: 7-November 04
From: Sweden
Member No.: 108



I've written a small too to help me download the "RAW" Curiosity images from the mars.jpl.nasa.gov site. Originally, I wrote it for my own use, but since some people here might find it useful I thought I should make it public.

The system requirement to run it is .NET 4.0.

It's important to note that this is just a quick and dirty tool I whipped up this morning, so the code quality isn't exactly great. There's not a lot of fault tolerance in the code, for example.
Another thing to keep in mind is that it's dependant on the current appearance of the mars.jpl.nasa.gov site. If that changes significantly, the tool will stop working. If that happens, and there's interest, I will hopefully have time to update the tool to make it work again.

Anyway, here it is, Curiosity RAW Get. It works for me up to Sol 38 data.
Attached Image


EDIT:
Since I don't have a web site of my own, I had initially intended to provide public support for the tool on this board, releasing new version here and making sure this post always pointed to the latest version of the software, and handling support in the followup posts.
After all posts related to software were merged into one thread, that is unfortunately no longer possible, so we'll go to plan B instead:
The version included in this post might in a few days or weeks get old. To get new versions or support, contact me off this board through email instead (My email address is included in the About dialog of the program.).

CHANGELOG:
0.16.38 - 2012-09-14
Added subframe support.

0.15.12 - 2012-08-18
Added thumbnail browsing mode.
Added installer.

0.14.10 - 2012-08-16
Added support for the Sol 10 version of the Curiosity web site.
Enabled threaded downloading.

0.13.3 - 2012-08-11
Switched from Microsoft's HTML parser to the HtmlAgilityPack parser.

0.12.3 - 2012-08-11
Fixed a problem where the Microsoft HTML parser didn't return parent element for some elements.

0.11.3 - 2012-08-11
Fixed version dependency on the Microsoft HTML parser.

0.10.3 - 2012-08-11
First release.
Attached File(s)
Attached File  CuriosityRAWGet_0.16.38.zip ( 122.25K ) Number of downloads: 889
 
Go to the top of the page
 
+Quote Post
James Sorenson
post Aug 11 2012, 02:11 PM
Post #5


Member
***

Group: Members
Posts: 691
Joined: 21-December 07
From: Clatskanie, Oregon
Member No.: 3988



For windows users, I've been using Star Downloader Pro. Its a link leecher that Scans and follows links on webpages based on what filtered file extention you input. Then it filters and downloads them. In this case .jpg files. There is a 30 day free trial to try it out, works like a charm for me, so I bought it.

http://www.stardownloader.com/
Go to the top of the page
 
+Quote Post
Marc
post Aug 11 2012, 02:26 PM
Post #6


Newbie
*

Group: Members
Posts: 7
Joined: 13-April 09
Member No.: 4730



I'm also working in a phyton script that automate big parts of the downlading process, but as I started teaching myself programming the first week of july it won't be so automatic and will require input from the user, at least until I improve my programming skills. The good part is that anyone will be able to use it as python runs in Windows, Linux and Mac.
Go to the top of the page
 
+Quote Post
scalbers
post Aug 11 2012, 02:42 PM
Post #7


Senior Member
****

Group: Members
Posts: 1621
Joined: 5-March 05
From: Boulder, CO
Member No.: 184



Also a shell script using wget initially mentioned in another thread. Several related posts appear right afterward - might be nice to consolidate in this thread.

MOD NOTE: Good idea- done!

Sounds good - the script is now post #1 by DEChengst in this thread.


--------------------
Steve [ my home page and planetary maps page ]
Go to the top of the page
 
+Quote Post
Pertinax
post Aug 11 2012, 02:49 PM
Post #8


Member
***

Group: Members
Posts: 198
Joined: 2-March 05
From: Richmond, VA USA
Member No.: 181



QUOTE (James Sorenson @ Aug 11 2012, 10:11 AM) *
For windows users, I've been using Star Downloader Pro.


I would also suggest the DownloadThemAll plugin for Firefox for those who wish. http://www.downthemall.net/

It is not at powerful as wget, but it has served me well for six to seven years now.


-- Pertinax
Go to the top of the page
 
+Quote Post
M_Welander
post Aug 11 2012, 03:16 PM
Post #9


Junior Member
**

Group: Members
Posts: 36
Joined: 7-November 04
From: Sweden
Member No.: 108



Hm... actually... The reason why I posted my tool in a new thread was so that the link to it could be in the first post (and thus easily updated when future versions are released), and then use the rest of the thread for support and change logs (compare this to how other message boards [for example doom9] dealing with software releases work).

Is there any chance we could break this out again into one thread per tool? As it is now, I fear it won't be obvious to anyone using the tool when a new version is released. In fact, the first post in the thread called "Curiosity RAW Get" isn't even about the tool called "Curiosity RAW Get" anymore.

MOD EDIT: Thread title changed to reflect a more general theme. Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage.
Go to the top of the page
 
+Quote Post
CosmicRocker
post Aug 11 2012, 04:09 PM
Post #10


Senior Member
****

Group: Members
Posts: 2228
Joined: 1-December 04
From: Marble Falls, Texas, USA
Member No.: 116



QUOTE (M_Welander @ Aug 11 2012, 08:52 AM) *
...the first version of Curiosity RAW Get.

I installed it, but when I try to run it, nothing appears to happen. Actually, not nothing. I get an info window with some disclaimers, but that is it. I have .NET 4.0 running under win7. I tried double-clicking the icon and also running it from a command line.

Do you have any idea what I may be doing wrong?


--------------------
...Tom

I'm not a Space Fan, I'm a Space Exploration Enthusiast.
Go to the top of the page
 
+Quote Post
M_Welander
post Aug 11 2012, 04:20 PM
Post #11


Junior Member
**

Group: Members
Posts: 36
Joined: 7-November 04
From: Sweden
Member No.: 108



QUOTE
Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage.


Okey, that's too bad. I'm afraid supporting the tool in the middle of a generic software thread is too much work for me to manage, instead, so I'll take support off this board instead. I'll update the post to reflect that.
Go to the top of the page
 
+Quote Post
CosmicRocker
post Aug 11 2012, 05:00 PM
Post #12


Senior Member
****

Group: Members
Posts: 2228
Joined: 1-December 04
From: Marble Falls, Texas, USA
Member No.: 116



It seems that it might be useful to create a subforum under MSL similar to the "Tech, General, and Imagery" section under MER. That would seem to allow for a little more flexibility with regard to specialized topics like specific software applications.


--------------------
...Tom

I'm not a Space Fan, I'm a Space Exploration Enthusiast.
Go to the top of the page
 
+Quote Post
Stellingwerff
post Aug 11 2012, 07:40 PM
Post #13


Junior Member
**

Group: Members
Posts: 43
Joined: 7-August 12
From: The Netherlands
Member No.: 6493



I'm building a webportal for easier access to the raw images, but I'm highly time-constrained, so it's going slowly.
But the first step is there:

I've put an automatically up-to-date JSON listing of all RAW images online, as a more convenient listing for (web)developers or as input for wget scripts.
(With the lack of folder listings in the raw-images CDN, I've done the same HTML parsing as all of you:)

You can find the JSON list at: http://msl-raw-images.appspot.com/landing
There are a couple of parameters:

counts -> Only list image counts per sol/camera.
cam -> Only list specific camera
sol -> Only list specific sol

Examples:
http://msl-raw-images.appspot.com/landing?...m=MD&counts (will list the count of descent images: 1522)
http://msl-raw-images.appspot.com/landing?sol=1&cam=NRA (will list all right Navcam images for sol 1)

The site will automatically update every 30 minutes, so expect a ~15 minutes delay compared to the original site.
Hope this portal will be usefull already. I'm planning on making a webpage similar to the MER raw images as soon as possible. (hopefully within a day or two)

Greetings,
Ludo.
Go to the top of the page
 
+Quote Post
elakdawalla
post Aug 11 2012, 08:12 PM
Post #14


Administrator
****

Group: Admin
Posts: 5172
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



On previous missions when it's been tough to parse raw images (Phoenix being a major example) I've posted easier-to-browse indexes on the Society's site. But I'm no programmer; I always did this manually. There appear to be lots of different solutions brewing here. I would LOVE it if when one of these gets fleshed out, it could be dropped in to a subsite on the Planetary Society website. With lots of praise and credit to whomever developed the solution. I can handle making it play nicely with our site, but I can't do automated image retrieval/indexing.


--------------------
My website - My Patreon - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
M_Welander
post Aug 11 2012, 08:31 PM
Post #15


Junior Member
**

Group: Members
Posts: 36
Joined: 7-November 04
From: Sweden
Member No.: 108



I think it'll be fairly easy to get this "fleshed out". I see mainly two problems:
  • First, that the tools need to be able to run on a variety of strange and wonderful systems, which means a lot of compatibility testing to start with. It's not difficult, it just takes time, and for a while the tools will embarrass their programmers with really, really stupid bugs. wink.gif
  • Second, and most important, the tools will need to be constantly updated to reflect any changes in the web sites from which they gather data.
Go to the top of the page
 
+Quote Post

25 Pages V   1 2 3 > » 
Reply to this topicStart new topic

 



RSS Lo-Fi Version Time is now: 28th March 2024 - 08:25 AM
RULES AND GUIDELINES
Please read the Forum Rules and Guidelines before posting.

IMAGE COPYRIGHT
Images posted on UnmannedSpaceflight.com may be copyrighted. Do not reproduce without permission. Read here for further information on space images and copyright.

OPINIONS AND MODERATION
Opinions expressed on UnmannedSpaceflight.com are those of the individual posters and do not necessarily reflect the opinions of UnmannedSpaceflight.com or The Planetary Society. The all-volunteer UnmannedSpaceflight.com moderation team is wholly independent of The Planetary Society. The Planetary Society has no influence over decisions made by the UnmannedSpaceflight.com moderators.
SUPPORT THE FORUM
Unmannedspaceflight.com is funded by the Planetary Society. Please consider supporting our work and many other projects by donating to the Society or becoming a member.