As the doctor said, "stop doing that!"

Actually, that counts as a bug--I just de-linked that field. Thanks--good testing.

So far it's not too overtaxed I think, since JPL (or rather Amazon) handles most of the bandwidth (images download), so all it has to do is serve up a php page and make a database call for each page view. It's possible I guess that the mysql database might bog down. So far it's about 15,000 page views per day according to Google analytics. I'd think about Google appengine like Ludo uses with his app if necessary, although their database is a little weird as I recall, not a relational database.

of course its tempting to ask for more features. and i used to be a QA guy so i cant resist so in case you ever tire of looking at all those beautiful MSL images and want to get back to coding heres something: i was tempted to ask for a selector for 'full size' vs 'thumbs' (or whatever other options come down) in order to nix all those preliminary downsamples and focus on the final hi-res versions. [EDIT: i overlooked the 'type' column which could better sort thes images by full frame (F or E), cropped (D) or thumbnail (T, I)]

I see you've nicely put the WxH info in there as separate columns which seems to accomplish the same thing so i can sort of pseudo-zoetrope the mardi images by scrolling the browser window to see them play out as a movie. however it looks like theres a lot of breaks still in the mardi hi-res images, but i think they just arent sorting perfectly using my approach, 0000MD9999000442E1_DXXX is followed by 0000MD9999000599E1_DXXXusing this approach, but 0000MD9999000443E1_DXXX does exist in hi-res its just sorting elsewhere using this method, so i guess we cant stack 'order by' priorities like as if it were an sql query.

I was about to do this very site on a long weekend I'm having, but you basically beat me to it. Nice work Mr Knapp.

As a kind of braindump, here's a bunch of ideas I had for the site. Take 'em, and run with 'em, or leave 'em - all yours:

• I think the main one was just that I was going to make it more jQuery-y - changes to the view settings would be updated instantly, a click on a thumbnail wouldn't be a full page hit. Eventually I would've make it like GMail - no page hits besides the first, just AJAX updates.
• A variety of view modes. The list, pretty much like you have it. But also a contact sheet mode - just a big grid of pictures. It would have the metainfo come up on hover. Also an Imgur- or Flickr-like slideshow mode. Each view mode needs to present photo metainfo slightly differently.
• Links back to the JPL page.
• Cookied view settings
• Binding the JPL-released thumbs types with the subframes/fulls into one line (assuming JPL leaves enough info to do that), then hiding the lesser-sized shot but presenting the option to see it.
• Optional columns (again, cookied) so people can customize their view a bit
• Somehow allow links to full panorama versions, if a shot is a part of a panorama (user-edits? user submissions?)
• Looooong term I was thinking like, user tags and a search of the meta info, maybe even comments and/or votes (again, Imgur/Mlkshk as a guide star here).
• And like a few folks were saying above - this is an ideal spot to collate other metainfo about shots. If JPL ever releases pointing info, that color-map some of the more science-y folks want, etc, this would be an ideal spot to put it.
• Also, I had a few thoughts on how to politely spider JPL's site (using HEAD HTTP requests, checking the "Last Updated" text, etc) but I guess they don't care that much.

This is all a ton of work, no way I'd do this all in a weekend. But it's the kind of thing I think about.

And we're back!

Man, someone said something in the beginning of this thread about "programmers making embarrissing mistakes". Well, mine was depending on too experimental, self-made database. Scalability issues abound. Anyway, after locking myself up for four days, the site is back online and reacting quick enough. (max waiting time should be below 15s for a full Google Appengine spin-up delay, normal reload delay of about 1-5 seconds.) Normal services have resumed by now.

Check: http://msl-raw-images.appspot.com/lists.html

Hopefully I'm back to feature request implementation by now.

Greetings,
Ludo.

Seems to be pretty snappy. I think it only took a few seconds the first time--maybe someone else had already spun-up the appengine. I looked at the json features. People coming up with auto-downloaders could really use that.

I'd have to look up jquery to really get a handle on that... instant updating sounds good. The only thing that would give me pause is if every change in view causes another database query. Like right now anyway to get results for a single camera you have to uncheck all the others. If each click results in a query...


Now that sounds fancy and very cool.


One problem I could see with that is, right now when you go to the homepage you automatically see what's new on all the cameras. If it remembered the last quirky search you did then you might be annoyed if you just wanted to see what's new.


Can definitely see the value in that.


That'd be killer. For example, to be able to retrieve all images pertaining to "N165."



If metadata ever becomes available, either through the JPL web site or maybe through the SPICE kernels, I definitely intend to add it in. Imagine being able to click on an image link and switch to a Google Mars view of where the image was taken and even with the same look angle.

Ok, back to feature development:

Newest version changes:

• Select range of images and open a text-only url listing for exporting purposes.
  ->first step for Emily's "killer feature" request:)
  ->selector can use gmail style range selections!
• Mark new images with a red star.
• Now works correctly in IE

For the next version I like to know which commandline options most of you use for 'wget' retrieval of images. I'm planning on adding a configurable "prefix" for the text export so the data can be pasted to a shell and executed directly.

http://msl-raw-images.appspot.com/lists.html

Hi Joe - thanks for reading over my scattered notes. Let me fill in some of the details.

First and foremost: I don't know what else you're running on this VM or host you have, nor your traffic, but, if my guess is right - don't worry too much about the database. Databases are built to handle this kind of thing, in general. I've done far, FAR worse, many times. (Both on my own and with someone's else money at stake, heh.) The worst thing that could happen is that you'd have to add an index or two somewhere down the road. It's no big deal, especially with only 4 or 5 digits (<100000) of pictures. Feel free to email/PM if you find your database is slowing things down too much.

I didn't explain myself very well on "cookied view settings". Apologies for the confusion. I was referring to selectively saving particular settings that related to view preferences. Not the whole kit and kaboodle. So, currently, I'd think about throwing the items per page, thumbnail size, and maybe the camera selections into a cookie. I would definitely not save in a cookie which page you were on. Also potentially interesting as a "sticky" setting is the optional columns.


BTW I had another idea about a feature - this probably applies equally well to Ludo's site. It'd be great if the site remembered the last time you came. It could mark pictures posted since then as "new", or draw a line in the listing. Sophisticated "new" markers can get tricky. But luckily, this can be simple. Simply set the cookie at time of the current page hit, but only if there's no cookie yet, or if last cookied time is more than 30 minutes ago.

Hi,

Thanks for the tip, I'll put it on the todo list. (somewhere on the top, as it's relatively easy to implement)

Nowadays I would not store stuff in cookies, with the advance of HTML5 capabilities I would definitely put stuff only in localStorage, works much cleaner and simpeler.

Greetings,
Ludo.

Yup. HTML5's storage only works on fairly modern (post-2010) browsers, which, sadly, a lot of my users don't have. It's pretty danged nifty, though. Count me jealous.

Session storage is actually how I normally prefer to roll these days, but that depends heavily on your web library having tools for it built-in, or building it out yourself in your database. And it helps to have a user login or at least a permanent cookie to attach to. It gets complicated quickly.

Cookies are a lowest-common-denominator, plain and simple. They're stupidly primitive, but tried-and-true, easy, and widely applicable.

Any of the above would work, under appropriate assumptions. It's nice to have options.

I have been using this feature to great effect today. Thank you for implementing it! For wget, the most common modifier I use is -P to set a target folder for the downloaded images.

Interesting thought. What's the idea of the 30-minute guard time? I guess you don't want to leave unflagged items that may have come in during the last session?

It's easy to mark your last visit now - just visit the latest image link (whether it's interesting or not), and next time look for the first visited colour link.

One odd thing I noticed first this morning, Joe - the site no longer displays vertically the full page. It now gives you a narrow window and vertical scroll bar:

Since other people haven't complained, I suspect my archaic browser may be to blame...

Whoa, sorry about that. I just tried it with my version of Firefox & it seems OK. Wish I could reproduce that. One change I made yesterday was to re-add the "lag" column. I think that's interesting after all to see which images show up very quickly versus which ones take longer. MAybe it's a glimpse into the priorities of the investigation. Anyway, maybe the columns no longer fit horizontally which causes your window to scroll?

BTW, I made a couple other changes per maschnitz's suggestions: the "items per page" and "thumbnail size" options are now cookied and persistent.

Sorta. It's more that, if you didn't have a guard similar to that, the "new" markers would all disappear upon your second page hit in the session. You don't want to reset the "new" time too often.

30 minutes is basically the industry standard for "you must've left and came back, by now". There's a long involved story behind sessioning in HTTP. But the short summary is it turns out that it's impossible to tell when someone's left your site, short of them telling you by logging out. It's because HTTP is a sessionless protocol. So you get to make grand assumptions off thin data, instead.

If you want to get really fancy, what you do is, keep two cookies for this (or two pieces of data in one cookie). You keep the last time there was any page hit, and the last time you considered everything "new". Then, you only reset the "new" time if it's been more than 30 minutes since the last page hit. That is, you only reset the new time if the person hasn't touched your site for 30 minutes. It's a little more accurate. But in the larger scheme of things, it's still kind of a cheat.