Curiosity Image Retrieval Tools, scripts and software Options

[Stellingwerff]

Early morning release:

• Better IE support
• Support subframe images of non-Malin cams
• Create wget script from export
• Minor tweaks

http://msl-raw-images.appspot.com/lists.html

PS. Curiosity really needs a "nickname", like Oppy. How about "MC"? Any suggestions?

[Edit] PPS. As you've noticed I'm not much of a graphical designer:) I could use some help with creating a nice banner/header for my listing, various styling tips and maybe a nice, non-intrusive background. Anybody here up for the challenge of helping me a step forward on that subject?

[elakdawalla]

Well, I know what I've been leaning toward as a nickname for Curiosity. But I don't feel like the nickname's been earned yet. I need to wait to see some drives and have this monstrous thing turn into a moving machine.

As for graphic design, honestly, this is one place where less design is better! No chartjunk!

[Stellingwerff]

Completely agree, less is way more. That's why it is such a challenge!

[elakdawalla]

Ludo, another feature request. In the "Type" column, it would be great to have the option "remove thumbnails" instead of or in addition to "full only." Lots of images will get returned that are "downscaled" and will never be returned at full resolution.

[Stellingwerff]

I've a new version ready for your enjoyment, would have been sooner had Google Appengine cooperated with the deployment....

Anyway, changes:

• Make image types a multiselect (Thanks Emily for the heads-up)
• Minor fixes
• Make the page better show-up in Google searches (e.g. added author info)

http://msl-raw-images.appspot.com/lists.html

[Edit] Might be obvious, but please make sure you refresh the entire page instead of getting cached versions. (ctrl-F5 in IE, shift-F5 in FF, etc.)

Teaser: I've been working on the next "killer" feature, custom lists with meta-data (e.g. description). At first these lists will be local in the browser for personal usage, but soon these lists will be shareable to others. This allows the site to be used for documenting lists of images, like Paolo suggested. So, keep a close watch to the site (and this thread:), nice stuff coming up in a couple of days.

Grz,
Ludo.

[walfy]

Teaser: I've been working on the next "killer" feature,...

That would be fantastic! Thanks for your efforts. Your page is great, already makes for easy sorting and downloading with FireFox add-on.

[elakdawalla]

Split Curiosity nicknaming posts to its own thread in chit-chat section.

[Stellingwerff]

Nice round image milestone: 4000! Only many 10s of thousands to go:)
Really shows the scalability goals of any listing tool....

[jmknapp]

Yes, particularly when having to depend on scraping web pages--would be very nice if JPL could provide JSON-style queries on their images. As it is, as the sols rack up & given that an image for any of the sols might belatedly come in on any day, to be thorough the individual page for each sol needs to be checked.

I'm thinking it might be good to only check, say, the last 15 sols or so on a frequent basis, and scan all the sols once a day or something like that.

[Stellingwerff]

I haven't checked yet, but how about parsing the first page (with their per camera, per sol listing) for marked sols? I currently check that page already for the total count, should be relatively straight forward to obtain a workset from it, too.

It would be particullarly welcome if we could skip sol 0, as it is quite large due to the Descent images....

[Edit] Yep, checked it: any line ending on '<span class="OrangeRawMark">Sol' preceeds a line with sol number. Not sure how long JPL keeps images "new", as sol 17s images are currently unmarked....

[jmknapp]

Re: checking the first page for the "OrangeRawMark" tag--that's a good idea, although one more thing to depend on not changing. I'm thinking I need to detect a "parse error" condition and send myself a text message if/when the raw website design changes and breaks the scraper.

[Stellingwerff]

Nope, the frontpage trick seems not trustworthy as the current situation is not correct: There are at least 8 new navCam images in the last day (after the currently marked sol0 and sol13 images) which are not marked in orange.....

[jmknapp]

A couple recent changes to http://curiositymsl.com

- added a persistent option to display time in local time zone
- keeps track of last visit & tags new images with a small DSN icon

[Stellingwerff]

Hi All,

Proud to present: Local lists of images! From now on it's possible to select a range of images and save them to personal lists. Use them to keep track of specific situations, sources for a particular panorama, etc. Head over to http://msl-raw-images.appspot.com/lists.html (refresh your caches on loading (ctrl-F5/shift-F5)) and look at the new feature!

That's the good news, now the "caveats":
This is very early work and I have loads of stuff I like to add to it. Especially I like to provide serverside storage of lists, to provide two services: better protection against losing lists (They are currently stored in your webbrowser, not the safest place) and more importantly: The ability to share lists with other people. That will be the next step I'll work on in the coming week and I do plan on keeping the current lists compatable with the sharing features. So lists you create today can be shared tomorrow, no worries there:)

Please use the lists and give me feedback on them, especially on the "user-experience". If anything strikes you as "annoying", "plain-wrong", "pretty nice", "I haven't a clue what this should do", etc. don't hessitate to drop me a message. I did put a small help button in there as well, just to explain the icons!

You might have noticed that I did some effective performance optimizations over the last few days, I am now quite happy about the response times (and the closely related financial back-lash from Google AppEngine usage).

Greetings,
Ludo.

[Edit] I forgot a major caveat: currently the meta information of a list (name, description, url) can't be editted. You only have the option to create a new list, possible based on the old lists images. Editting existing meta info will be implemented in a near-future version.

[Pando]

A couple recent changes to http://curiositymsl.com

Hi Joe, awesome tools there. Many thanks!

I'm still using your older list script (which works great), however the uppercase JPG file extension doesn't seem to work with JPL website (HTTP 403 Forbidden error). I hope it wouldn't be too much for you to change this to lowercase .jpg ...
http://curiositymsl.com/cgi-bin/getfiles.cgi?sol=17