Curiosity Image Retrieval Tools, scripts and software Options

[jmknapp]

OK, lots of good suggestions, I'll get to most of them as I have time. For now I just made a fewchanges--

1) image width and height is now given instead of file size
2) added a sortable column for sol number
3) the top level domain curiositymsl.com now goes to the table so there's no need to add the table/view part

I'm trying to keep it simple from an ongoing maintenance angle, so it just checks the JPL website every 15 minutes for new images and just adds them to the db as it finds them. Therefore, the "official" thumbnails get treated just like any other image. Weeding out the database manually is a non-starter, but maybe some automated way would work, like if a full-frame image comes in, change the file name to the thumb version and nuke the latter if it's is in the db. But then the next time it checks for updates it would have to know not to flag thumbs as new files. So rather than deleting, maybe the db needs a "visibility" flag for each file. And so on...

[Stellingwerff]

Hi all,

In the meantime my listing has also evolved, a first listing can be found at:
http://msl-raw-images.appspot.com/lists.html

Lots to do and some work-arounds to get it faster online, with the most noticeable the "spartan" camera selection. (currently you need to list all camera's in a comma seperated list, empty list leads to all cameras selected!)

I think it's somewhat usefull by now. Stay tuned for many new features in the coming week(s). Yes, I am planning on picking up many of the tips you've mentioned to Joe and integrate those into my version as well.

Greetings,
Ludo.

[elakdawalla]

Oh, cool. Hopefully you two guys' solutions will be divergent, so each will have its distinct uses.

[walfy]

In the meantime my listing has also evolved, a first listing can be found at:
http://msl-raw-images.appspot.com/lists.html

This is very nice! Thank you. In my limited knowledge of all the possibilities out there to make easy the downloading process, I found your page very easy to set the parameters of which files I wanted, then along with the DownThemAll! FireFox add-on, I could download all the images to any folder on my hard drive in one fell swoop. A great thing about DownThemAll! is that it's freeware and also can be set to skip over files already downloaded. So that's nice for checking for updates.

For example, I wanted all the full-res files from only Sol 3 to download them in their own folder, and setting it up was very easily for that (had to make sure I kept clicking the "show all" button below the list as it resets itself after each setting of filters and goes back to showing only a partial list of images). Later I will see if I can separate them out by camera. (Can't believe how many cameras there are!)

Thanks again.

[algorimancer]

The AlgorimancerPG (APG) rangefinder utility is updated for Curiosity,
http://clarkandersen.com/RangeFinder.htm

I think I have the installer problem fixed....

It turns-out the installer problem is not quite fixed. The executable link installed under Start/Programs Menu works, but can't find the associated DLL file. However, if you navigate to the c:/Program Files/AlgorimancerPG directory (or wherever you opted to install the application), double-clicking on AlgorimancerPG.exe from that location works fine.

This was my first attempt at having the installer put a link in the Programs Menu area, and clearly I have not mastered the trick. I'm guessing that I need to somehow provide a direct link to the DLL, perhaps through the registry, or perhaps simply have a copy of the DLL installed in the Programs Menu area; I'll have that sorted in the next few days. Meanwhile, the rangefinder application is useable, but you might want to map your own shortcut to the executable.

Just at the moment, only the navcams from Sol 2 are really of interest -- this will of course change once driving commences. One problem with the MSL image naming system is that left/right image pairs tend to be separated by rather a lot when sorted alphabetically -- perhaps this could be compensated for in the posted image browser tools to make it easier to pair them together? It would take some minimally creative sorting.

For those new to the APG utility, be sure to grab only the full-frame navcam image pairs, as it doesn't work with thumbnails or subsampled images.

[jmknapp]

Hey, good news and bad news, the good outweighing the bad. It seems that just a little while ago JPL revamped their raw image website. They fixed the issue where full size "thumbnails" were being loaded. They also flag the latest images ("The 5 new images for Sol 3 are marked with a red *") as well as images grouped by camera.

The bad news is that with the changes in their html, page-scraping for my app is broken for the time being & I turned off updates until that can be fixed. Always a danger with hacks like page-scraping!

Anyway, the main site is on move.

[ElkGroveDan]

It seems that just a little while ago JPL revamped their raw image website.

This is very atypical for JPL, normally they are right on top of things I would think this should have been all hashed out six months ago. Anyone know who is responsible for the new website?

[djellison]

Mars EPO - headed by Michelle Viotti

[Stellingwerff]

Hey, good news and bad news, the good outweighing the bad. It seems that just a little while ago JPL revamped their raw image website. They fixed the issue where full size "thumbnails" were being loaded.

Anyway, the main site is on move.

Good! Although following a moving target is challenging, any progress on their side is excellent. Happily they decided in this "round" to keep their old "sol-based" page largely similar, so my HTML loading still works correctly. (Largely similar doesn't mean that they didn't hasitate to put another 30.000+ empty lines in the HTML!)

Let's see where they are moving towards, I hope they are reading the user-feedback on this thread as usefull input:)

Greetings,
Ludo.

[jcims]

Does anyone in here know if any metadata is available for the images? E.g. elevation, heading, exposure, location, etc? I thought this was available for the MER raw images a while ago, but I couldn't find anything lately.

[elakdawalla]

It's a big improvement! Now we just need the metadata that'll help all you mosaic-assemblers (*cough*ahem*Mike*) automate the building of panoramas.

But I'm ready for one of you guys to write us an auto-raw-downloader that'll replicate at least that function and maybe the slideshow mode, if not the panorama mode, of MMB

We ran into a similar weird whitespace problem on the Planetary Society website about 9 years ago. We noticed slow page download times and I realized our home page (which was then coded in Dreamweaver, I think, but I stubbornly insisted on hand-coding) had bloated to the horrifying size of 900k with huge amounts of whitespace. A little searching led me to a known issue where there was some problem that happened with transferring text files between unix systems and either Macs or PCs (I don't remember which) where each line feed was replaced with a CR-LF going one way, and then when the file was downloaded again the CR and LF were each replaced with a line feed, resulting in a doubling of the amount of white space each time the file was uploaded and re-downloaded. Since there were several different people who might edit the home page in a given day, who downloaded the current one and re-uploaded it each time, exponential growth happened rapidly.

[Stellingwerff]

Just a quick update notice:

http://msl-raw-images.appspot.com/lists.html

has been updated with the following changes:

• Filter settings are now kept over page reloads
• "Show All" is now persistent, will not fall back to paging unless "Show less" is clicked
• A camera multiselect is added, empty means "all cameras"

[jmknapp]

Happily they decided in this "round" to keep their old "sol-based" page largely similar, so my HTML loading still works correctly.

My parsing was a bit more fragile--it's fixed now though and updating again curiositymsl.com. I also made a change to hide official thumbnail images where a full-frame image has also come in. Turns out that's about 10% of the total images (right now 1,998 total images).

Nice app!--Google appengine, right? I guess the only thing my app adds is the ability to sort by release date and time taken. BTW, if you're interested in adding that, you can use curl to get the timestamps of the image files on the JPL site, and the time taken on Mars can be scraped from the web pages or with some of the cameras extracted from the image file name (in ephemeris seconds). Maybe some future update on the JPL site will include more metadata and in a more robust form as Emily, Paolo & others have wished for. I imagine it's a bigger problem getting accurate position and pointing info from rovers than with missions such as Cassini. At least in the latter case, Mr. Kepler is largely in control of the positioning without the added complications of terrain, slip and dead reckoning.

[vk3ukf]

Hi all,

this doc might help help those looking to decode the image file names a little bit.

It's regarding the sclk breakdown.

http://naif.jpl.nasa.gov/pub/naif/MSL/kernels/sclk/msl.tsc

I've been looking for the full MSL EDR SIS file name definition for ages.

I saw further back there, a fellow (rtphokie), piecing it together, thanks for what you've figured out so far.

The MER version,

<scid><inst><sclk><prod><site><pos><seq><eye><filt><who><ver><ext>

(above from, http://pds-imaging.jpl.nasa.gov/documentat...v4_9-25-07.pdf)

and this is a great straight forward page for the MER EDR SIS,

http://marsrover.nasa.gov/gallery/edr_filename_key.html.

For the Phoenix lander PHX EDR SIS, this the pdf to read,

http://pds-geosciences.wustl.edu/missions/...x/phxCamSis.pdf

MPF and Viking links escape me at the moment.
They were a bit odd by today's standards, short file names and part of the sclk string was the storage directories name.

If anyone knows if MSL EDR SIS is online somewhere yet, love to read it.

Any further pointers greatly appreciated.

Perhaps it will appear at the Analysts Notebook, early next year, hopefully before then.


Kevin. Werribee.

[Stellingwerff]

Nice app!--Google appengine, right?

Thanks for the compliment! Yes, it's a GAE based server, scraping the HTML, building a graph database of the images and then providing this data through a JSON API. (I hope to create some more APIs, like RSS and notifications)

The frontend is HTML5 stuff, taking the JSON and using pure.js (see: http://beebole.com/pure/) to generate the listing, including filtering and ordering.

The applications source-code is available at: https://github.com/ludost/msl-raw-images

The graph database I use (and created) is open-source, see: https://github.com/almende/memo-nodes, one day it will be good enough for broader use...
I've chosen to use my graph db in-between to give this hobby project also some "use" for my work. I also hope to put some more structure in there for determining "stereo" images, some statistics (graphs), etc.

I guess the only thing my app adds is the ability to sort by release date and time taken. BTW, if you're interested in adding that, you can use curl to get the timestamps of the image files on the JPL site, and the time taken on Mars can be scraped from the web pages or with some of the cameras extracted from the image file name (in ephemeris seconds).

I'm looking into getting some timeinfo in the application. I just need to modify my HTML scraper to work somewhat multi-line:) I did notice that there is a whole world out there for time and location interpretation(SPICE @NAIF). There is however no real JAVA implementation of that toolset. (JNI won't work on GAE) I guess the more MMB-like, advanced features will be hard to achieve on GAE:(

[Edit: If somebody does know a Java implementation that can work with NAIF data, please let me know! Hack, why not a JavaScript implementation:) ]

Greetings,
Ludo.