Printable Version of Topic

Posted by: DEChengst Aug 11 2012, 09:18 AM

I wrote a shell script that allows you to download all images from a given sol.:

http://paranoid.dechengst.nl/files/MSLget.sh

There are two rm commands in the script to clean up the tempdir. If you're afraid my script screws up your system replace them with rm -i commands. If you want any features added please let me know and I'll see what I can do.

EDIT1:

Added leading zeros in the directory names for easy sorting after a request for it on IRC.

EDIT2:

I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox.

EDIT3:

As RoverDriver pointed out you need to have wget on your system to use this script. The script now handles this gracefully.

Posted by: RoverDriver Aug 11 2012, 01:32 PM

QUOTE (DEChengst @ Aug 11 2012, 02:18 AM)

Posted by: RoverDriver Aug 11 2012, 01:34 PM

As an example of using wget to download the HiRISE images, here's my script that downloads the non-map projecte, the map projected and label files to your current working directory:

Posted by: M_Welander Aug 11 2012, 01:52 PM

I've written a small too to help me download the "RAW" Curiosity images from the mars.jpl.nasa.gov site. Originally, I wrote it for my own use, but since some people here might find it useful I thought I should make it public.

The system requirement to run it is .NET 4.0.

It's important to note that this is just a quick and dirty tool I whipped up this morning, so the code quality isn't exactly great. There's not a lot of fault tolerance in the code, for example.
Another thing to keep in mind is that it's dependant on the current appearance of the mars.jpl.nasa.gov site. If that changes significantly, the tool will stop working. If that happens, and there's interest, I will hopefully have time to update the tool to make it work again.

Anyway, here it is, Curiosity RAW Get. It works for me up to Sol 38 data.

 CuriosityRAWGet_0.16.38.zip ( 122.25K ) : 891

Posted by: James Sorenson Aug 11 2012, 02:11 PM

For windows users, I've been using Star Downloader Pro. Its a link leecher that Scans and follows links on webpages based on what filtered file extention you input. Then it filters and downloads them. In this case .jpg files. There is a 30 day free trial to try it out, works like a charm for me, so I bought it.

http://www.stardownloader.com/

Posted by: Marc Aug 11 2012, 02:26 PM

I'm also working in a phyton script that automate big parts of the downlading process, but as I started teaching myself programming the first week of july it won't be so automatic and will require input from the user, at least until I improve my programming skills. The good part is that anyone will be able to use it as python runs in Windows, Linux and Mac.

Posted by: scalbers Aug 11 2012, 02:42 PM

Also a shell script using wget initially mentioned in another thread. Several related posts appear right afterward - might be nice to consolidate in this thread.

MOD NOTE: Good idea- done!

Sounds good - the script is now post #1 by DEChengst in this thread.

Posted by: Pertinax Aug 11 2012, 02:49 PM

QUOTE (James Sorenson @ Aug 11 2012, 10:11 AM)

Posted by: M_Welander Aug 11 2012, 03:16 PM

Hm... actually... The reason why I posted my tool in a new thread was so that the link to it could be in the first post (and thus easily updated when future versions are released), and then use the rest of the thread for support and change logs (compare this to how other message boards [for example doom9] dealing with software releases work).

Is there any chance we could break this out again into one thread per tool? As it is now, I fear it won't be obvious to anyone using the tool when a new version is released. In fact, the first post in the thread called "Curiosity RAW Get" isn't even about the tool called "Curiosity RAW Get" anymore.

MOD EDIT: Thread title changed to reflect a more general theme. Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage.

Posted by: CosmicRocker Aug 11 2012, 04:09 PM

QUOTE (M_Welander @ Aug 11 2012, 08:52 AM)

Posted by: M_Welander Aug 11 2012, 04:20 PM

Posted by: CosmicRocker Aug 11 2012, 05:00 PM

It seems that it might be useful to create a subforum under MSL similar to the "Tech, General, and Imagery" section under MER. That would seem to allow for a little more flexibility with regard to specialized topics like specific software applications.

Posted by: Stellingwerff Aug 11 2012, 07:40 PM

I'm building a webportal for easier access to the raw images, but I'm highly time-constrained, so it's going slowly.
But the first step is there:

I've put an automatically up-to-date JSON listing of all RAW images online, as a more convenient listing for (web)developers or as input for wget scripts.
(With the lack of folder listings in the raw-images CDN, I've done the same HTML parsing as all of you:)

You can find the JSON list at: http://msl-raw-images.appspot.com/landing
There are a couple of parameters:

counts -> Only list image counts per sol/camera.
cam -> Only list specific camera
sol -> Only list specific sol

Examples:
http://msl-raw-images.appspot.com/landing?sol=0&cam=MD&counts (will list the count of descent images: 1522)
http://msl-raw-images.appspot.com/landing?sol=1&cam=NRA (will list all right Navcam images for sol 1)

The site will automatically update every 30 minutes, so expect a ~15 minutes delay compared to the original site.
Hope this portal will be usefull already. I'm planning on making a webpage similar to the MER raw images as soon as possible. (hopefully within a day or two)

Greetings,
Ludo.

Posted by: elakdawalla Aug 11 2012, 08:12 PM

On previous missions when it's been tough to parse raw images (Phoenix being a major example) I've posted easier-to-browse indexes on the Society's site. But I'm no programmer; I always did this manually. There appear to be lots of different solutions brewing here. I would LOVE it if when one of these gets fleshed out, it could be dropped in to a subsite on the Planetary Society website. With lots of praise and credit to whomever developed the solution. I can handle making it play nicely with our site, but I can't do automated image retrieval/indexing.

Posted by: M_Welander Aug 11 2012, 08:31 PM

I think it'll be fairly easy to get this "fleshed out". I see mainly two problems:

• First, that the tools need to be able to run on a variety of strange and wonderful systems, which means a lot of compatibility testing to start with. It's not difficult, it just takes time, and for a while the tools will embarrass their programmers with really, really stupid bugs.
• Second, and most important, the tools will need to be constantly updated to reflect any changes in the web sites from which they gather data.

Posted by: jmknapp Aug 11 2012, 08:41 PM

To simply get a list of all the raw images for a given sol, I put up a web widget--access it with wget as follows, for example for sol 4:

wget -O sol4files.txt http://curiositymsl.com/cgi-bin/getfiles.cgi?sol=4

I.e., just set the sol=N parameter and you get a text file back with all the raw images currently on the MSL public web site

For a really basic web interface to same, see:

http://curiositymsl.com/rawlist

Posted by: elakdawalla Aug 11 2012, 10:28 PM

Thanks, this is great! Note that with Navcam and Hazcam images it's indexing thumbnail versions of the raws (*-thm.jpg) whereas with the MSSS cameras the raws are linked directly.

Posted by: jmknapp Aug 12 2012, 12:51 AM

QUOTE (elakdawalla @ Aug 11 2012, 05:28 PM)

Posted by: walfy Aug 12 2012, 04:37 AM

Any chance that Midnight Mars Browser can be modified by the code wizards to do the trick for quick MSL downloads? I love that program! It sends photos of each camera to their own folders, even folders for separates sols. Very nice way to keep track. (Crossing fingers).

Posted by: elakdawalla Aug 12 2012, 04:47 AM

Believe me, Mike knows very well we want that, and no amount of pestering will make it more likely to happen (or happen any faster), though compliments are probably always welcome. As with pretty much all of you guys, that isn't his day job. We just have to wait and hope he can find time somehow!

Posted by: elakdawalla Aug 12 2012, 04:49 AM

QUOTE (jmknapp @ Aug 11 2012, 01:41 PM)

Posted by: M_Welander Aug 12 2012, 05:37 AM

QUOTE (walfy @ Aug 12 2012, 06:37 AM)

Posted by: rtphokie Aug 12 2012, 08:15 PM

Anyone had any luck decoding the filenames on the raw images?

SSSSIICCCCCCCCCCT1_DXXX

SSSS = sol number (good to see 4 digits here)
II = is the instrument (ML = Mastcam left)
CCCCCCCCCC = ????, is this a serial number? It doen't appear to be a time (see below)
T = image size I or E
1_DXXXX = ????

Looking at a couple of examples:
0003ML0000139000I1_DXXX taken at 2012-08-09 06:01:57 UTC
0003ML0000134000I1_DXXX taken at 2012-08-09 05:59:34 UTC

Those timestamps are 2 min 23 seconds apart. the numbers in the filename do not seem to refer to time, Earth or Mars.


Also, some of the mastcam images ending in I1_DXXX also have higher resolution versions ending in E1_DXXX but aren't listed on the mars.jpl.nasa.gov site. Are they listed somewhere else?

Posted by: jmknapp Aug 13 2012, 12:40 AM

Noticed a little bug on the JPL raw images web site--for full frame MAHLI images they show a thumbnail with a link to the full frame image, but it turns out the thumbnail IMG SRC is actually the full frame image. The html code sets the thumbnail width to 160 so it looks like a thumbnail, but it's a waste of bandwidth as all the full frame MAHLI images for a given sol will be downloaded when just looking at the thumbnail page.

Here's an example (MAHLI images with 'E' in the 17th character of the filename are full-frame, 'I' for thumbnail):

Posted by: mcaplinger Aug 13 2012, 12:56 AM

QUOTE (jmknapp @ Aug 12 2012, 05:40 PM)

Posted by: jmknapp Aug 13 2012, 01:43 AM

QUOTE (mcaplinger @ Aug 12 2012, 08:56 PM)

Posted by: elakdawalla Aug 13 2012, 01:45 AM

I reported the same to Veronica McGregor a few days ago (who is not in charge, but obviously can talk to whomever is) and she said that they officially do not care about bandwidth since everything is stored in the cloud. Now, that's fine for them but it'd be better for the user if they didn't have to download so much data. But I don't want to hassle them about this detail since it's the least of our problems with accessing data. If I'm going to bug them I want to bug them about things like getting metadata (filter information, pointing information etc) and faster distribution of the raws hopefully by automating things. As of the time I left JPL, as far as anybody knew it was a manual process. Priorities! Other people have been begging Mike Howard to make a version of MMB for Curiosity. Even if he wanted to, he couldn't right now, because they're not posting enough metadata.

Posted by: mcaplinger Aug 13 2012, 02:18 AM

QUOTE (elakdawalla @ Aug 12 2012, 06:45 PM)

Posted by: Astro0 Aug 13 2012, 03:02 AM

Let's remember that the mission is not obliged to give us access to these image at all. At least until the PDS release.
It's early days in the mission and the priority is the mission.

All of the so-called "issues" with getting quick and easy access to the images will sort itself out in due course.

Remember that we are privileged to get access to this data, but they are not obligated to provide it.
The fact that they do provide it at all is highly appreciated and while frustrating to some that it's not quicker and easier, we need to remember that their people and resources are not unlimited. Give them time.

UMSF is a great place to provide useful suggestions and ways to access the imagery.
It is of no value to complain about it.

Posted by: mcaplinger Aug 13 2012, 03:54 AM

QUOTE (Astro0 @ Aug 12 2012, 08:02 PM)

Posted by: CosmicRocker Aug 13 2012, 04:06 AM

QUOTE (M_Welander @ Aug 11 2012, 07:52 AM)

Posted by: djellison Aug 13 2012, 05:03 AM

QUOTE (mcaplinger @ Aug 12 2012, 05:56 PM)

Posted by: jmknapp Aug 13 2012, 10:28 AM

QUOTE (mcaplinger @ Aug 12 2012, 10:54 PM)

Posted by: M_Welander Aug 13 2012, 10:54 AM

QUOTE (elakdawalla @ Aug 13 2012, 03:45 AM)

Posted by: algorimancer Aug 13 2012, 02:28 PM

I have my AlgorimancerPG rangefinder/photogrammetry tool updated to work with the MSL navcam images -- I hope to post it to the website tonight (it's not there yet). I'll announce the availability here when I do (this seems the closest topic match in the MSL section).

One thing it is lacking is the correct sol/time calculation; it will display a value, but it's not going to be a correct value. If someone could clarify how to get the sol/time from the image title, that would be much appreciated.

Another thing lacking is camera orientation. For MER, this could be obtained from the Pancam Tracking Database. I'm not aware of any such database as yet for MSL.

Support for the Mastcam's higher resolution cameras may or may not happen for MSL, as this will require rather more time than I happen to have at the moment. If there's someone with experience programming with Microsoft Visual C++ (Visual Studio 2010 professional) who would like to take-on the project, particularly someone familiar with imagery and vectors and quaternion transformations, I would be happy to share the code. This may take anywhere from a few days to a couple of weeks of programming time. Considering that we have high resolution images and topography from orbiters already, I'm not sure there would be much value in adding Mastcam capability, but it is an option.

Posted by: mcaplinger Aug 13 2012, 03:14 PM

QUOTE (algorimancer @ Aug 13 2012, 07:28 AM)

Posted by: ugordan Aug 13 2012, 03:41 PM

QUOTE (mcaplinger @ Aug 13 2012, 05:14 PM)

Posted by: jmknapp Aug 13 2012, 03:44 PM

QUOTE (mcaplinger @ Aug 13 2012, 10:14 AM)

Posted by: mcaplinger Aug 13 2012, 04:18 PM

QUOTE (jmknapp @ Aug 13 2012, 07:44 AM)

Posted by: algorimancer Aug 13 2012, 04:44 PM

QUOTE (mcaplinger @ Aug 13 2012, 09:14 AM)

Posted by: jmknapp Aug 13 2012, 06:07 PM

QUOTE (mcaplinger @ Aug 13 2012, 11:18 AM)

Posted by: mcaplinger Aug 13 2012, 06:28 PM

QUOTE (jmknapp @ Aug 13 2012, 10:07 AM)

Posted by: jmknapp Aug 14 2012, 12:17 AM

Trying to develop a handier web interface to the raw images, here's a first go if anyone would like to help beta test it:

http://curiositymsl.com/table/view

It shows the raw images in table form, sortable by date taken, release date, etc. by clicking on the column headers. The default sort, by release date, is the handiest I think, since it shows the most recent uploads to the JPL site. It's hard otherwise to keep track of the newest files because they can come in under any sol.

Open to suggestions as to how it could be more useful. One thing that comes to mind is to be able to filter by camera, because with all the images in one big list, there are a lot of pages, mostly due to the large number of MARDI images.

Posted by: elakdawalla Aug 14 2012, 12:40 AM

First: AWESOME!

It would be great to filter by camera.

I think image dimensions would be more useful than file size.

I'm amused by the "lag" column, though it's not exactly fair, since it folds in both transmission lag (when did it arrive on Earth?) and raw posting lag.

I'd prefer smaller thumbnails and more images per page, but don't know if that opinion is shared.

This takes more logic, but: it would be great if it did not show *both* thumbnails *and* fulls of the same image. It would be lovely if it could just show the highest-resolution one returned to Earth so far.

Posted by: Oersted Aug 14 2012, 12:44 AM

Works great on a Samsung Galaxy Nexus smartphone, for what its worth... -Excellent job!

Posted by: lyford Aug 14 2012, 12:58 AM

Allow me to say OMAGERD THANK YOU

Posted by: RoverDriver Aug 14 2012, 01:18 AM

QUOTE (jmknapp @ Aug 13 2012, 05:17 PM)

Posted by: jmknapp Aug 14 2012, 01:20 AM

QUOTE (elakdawalla @ Aug 13 2012, 07:40 PM)

Posted by: jmknapp Aug 14 2012, 01:46 AM

QUOTE (RoverDriver @ Aug 13 2012, 08:18 PM)

Posted by: ChrisC Aug 14 2012, 02:12 AM

Fantastic, Joe!

My one suggestion has to do with the curiositymsl.com home page. Please either A) link to the basic index page from there, or configure your site to automatically forward to the index page. That way I just have to remember "curiositymsl.com" when I get inspired to check for new images, no matter what computer I'm at.

Of course, I can navigate to UMSF, find this thread, click to the right page, and then click the link, but ...

Posted by: volcanopele Aug 14 2012, 02:29 AM

QUOTE (Oersted @ Aug 13 2012, 05:44 PM)

Posted by: mcaplinger Aug 14 2012, 02:33 AM

QUOTE (jmknapp @ Aug 13 2012, 05:20 PM)

Posted by: RoverDriver Aug 14 2012, 02:36 AM

QUOTE (jmknapp @ Aug 13 2012, 06:46 PM)

Posted by: Cargo Cult Aug 14 2012, 06:52 AM

Evening, all -

It's definitely not a fantastically useful image-listing tool like those above, but in keeping with the scraping-NASA-sites-to-present-data-in-more-versatile-ways theme, I've created a supremely unofficial RSS feed for the http://mars.jpl.nasa.gov/blogs/ blog.

It's at https://hylobatidae.org/misc/msl_blog.php - I haven't had a chance to test it in Google Reader, but it seems to work okay in Safari and Vienna for Mac OS X. It'll automatically check for updates every hour. Let me know if you have any issues!

(It's a horrible hack, derived from my equally unofficial RSS feed for Astronomy Picture of the Day: https://hylobatidae.org/misc/apod.php. They were built for my own use, but I hope people find them in some way helpful. Now, back to lurking on this fascinating forum! ;-) )

Posted by: Explorer1 Aug 14 2012, 07:52 AM

It works on my end; good job!

It's always a little strange when totally unrelated hobbies collide...

Posted by: algorimancer Aug 14 2012, 03:14 PM

The AlgorimancerPG (APG) rangefinder utility is updated for Curiosity,
http://clarkandersen.com/RangeFinder.htm

I think I have the installer problem fixed, but don't happen to have another computer to test it on, so if someone would like to download/install/test it on a stereo pair of MSL navcam images, it would be appreciated.

The update only adds capability to do distances/photogrammetry with MSL's navigation camera images, and lacks any information about orientation since that database has not been made available as yet. I'm not too worked-up about adding support for the higher resolution Mastcam images, as it would take a lot of programming effort and I'm not clear on what real benefit it would offer -- sure, it would allow rangefinding out to much greater distances, but we have Hirise images already for that sort of thing. I may be susceptible to persuasion, given a good motivation.


Incidentally, loving the table view (http://curiositymsl.com/table/view).

Posted by: fredk Aug 14 2012, 03:30 PM

Your site is already looking fantastic, Joe. I've got a few suggestions based on years of MER jpl and exploratorium use.

MER jpl is nice in that it sorts by sol, which is crucial when trying to find older images. Exploratorium is great because it sorts by arrival/release time, which is crucial when you're regularly looking at the latest images. So definitely the ability to sort either by sol or release time would make this the single best source of MSL images.

I agree with Emily that thumbs should be tossed out when superceeded by higher resolution images (though perhaps the official thumbs from Mars could be used as the image thumbs on your site, if the size is right?). It's odd that the MSL jpl site doesn't do that.

Another crucial feature of both MER jpl and exploratorium interfaces is that they indicate visited image links (ie visited links change colour). This makes it so much easier to catch up with images once a day or whenever you like, without having to remember which pics you've already looked at, especially when you sort by sols. One suggestion would be a border around the image thumb that changes from unvisited to visited colour, like with the MER jpl site.

Posted by: stevesliva Aug 14 2012, 03:53 PM

QUOTE (Cargo Cult @ Aug 14 2012, 01:52 AM)

Posted by: mhoward Aug 14 2012, 04:31 PM

QUOTE (fredk @ Aug 14 2012, 09:30 AM)

Posted by: elakdawalla Aug 14 2012, 04:44 PM

MESSENGER has a feed of sorts -- through its iPhone app. I get a notification every time they post a new image. I actually like their iOS gallery browser better than their Web image browser, which I find pretty clunky.

Posted by: jmknapp Aug 14 2012, 07:07 PM

OK, lots of good suggestions, I'll get to most of them as I have time. For now I just made a fewchanges--

1) image width and height is now given instead of file size
2) added a sortable column for sol number
3) the top level domain http://curiositymsl.com now goes to the table so there's no need to add the table/view part

I'm trying to keep it simple from an ongoing maintenance angle, so it just checks the JPL website every 15 minutes for new images and just adds them to the db as it finds them. Therefore, the "official" thumbnails get treated just like any other image. Weeding out the database manually is a non-starter, but maybe some automated way would work, like if a full-frame image comes in, change the file name to the thumb version and nuke the latter if it's is in the db. But then the next time it checks for updates it would have to know not to flag thumbs as new files. So rather than deleting, maybe the db needs a "visibility" flag for each file. And so on...

Posted by: Stellingwerff Aug 14 2012, 07:41 PM

Hi all,

In the meantime my listing has also evolved, a first listing can be found at:
http://msl-raw-images.appspot.com/lists.html

Lots to do and some work-arounds to get it faster online, with the most noticeable the "spartan" camera selection. (currently you need to list all camera's in a comma seperated list, empty list leads to all cameras selected!)

I think it's somewhat usefull by now. Stay tuned for many new features in the coming week(s). Yes, I am planning on picking up many of the tips you've mentioned to Joe and integrate those into my version as well.

Greetings,
Ludo.

Posted by: elakdawalla Aug 14 2012, 08:44 PM

Oh, cool. Hopefully you two guys' solutions will be divergent, so each will have its distinct uses.

Posted by: walfy Aug 14 2012, 09:16 PM

QUOTE (Stellingwerff @ Aug 14 2012, 11:41 AM)

Posted by: algorimancer Aug 15 2012, 12:11 AM

QUOTE (algorimancer @ Aug 14 2012, 10:14 AM)

Posted by: jmknapp Aug 15 2012, 02:18 AM

Hey, good news and bad news, the good outweighing the bad. It seems that just a little while ago JPL revamped their raw image website. They fixed the issue where full size "thumbnails" were being loaded. They also flag the latest images ("The 5 new images for Sol 3 are marked with a red *") as well as images grouped by camera.

The bad news is that with the changes in their html, page-scraping for my app is broken for the time being & I turned off updates until that can be fixed. Always a danger with hacks like page-scraping!

Anyway, the main site is on move.

Posted by: ElkGroveDan Aug 15 2012, 02:46 AM

QUOTE (jmknapp @ Aug 14 2012, 07:18 PM)

Posted by: djellison Aug 15 2012, 03:02 AM

Mars EPO - headed by Michelle Viotti

Posted by: Stellingwerff Aug 15 2012, 03:46 AM

QUOTE (jmknapp @ Aug 15 2012, 04:18 AM)

Posted by: jcims Aug 15 2012, 05:39 AM

Does anyone in here know if any metadata is available for the images? E.g. elevation, heading, exposure, location, etc? I thought this was available for the MER raw images a while ago, but I couldn't find anything lately.

Posted by: elakdawalla Aug 15 2012, 05:41 AM

It's a big improvement! Now we just need the metadata that'll help all you mosaic-assemblers (*cough*ahem*Mike*) automate the building of panoramas.

But I'm ready for one of you guys to write us an auto-raw-downloader that'll replicate at least that function and maybe the slideshow mode, if not the panorama mode, of MMB

We ran into a similar weird whitespace problem on the Planetary Society website about 9 years ago. We noticed slow page download times and I realized our home page (which was then coded in Dreamweaver, I think, but I stubbornly insisted on hand-coding) had bloated to the horrifying size of 900k with huge amounts of whitespace. A little searching led me to a known issue where there was some problem that happened with transferring text files between unix systems and either Macs or PCs (I don't remember which) where each line feed was replaced with a CR-LF going one way, and then when the file was downloaded again the CR and LF were each replaced with a line feed, resulting in a doubling of the amount of white space each time the file was uploaded and re-downloaded. Since there were several different people who might edit the home page in a given day, who downloaded the current one and re-uploaded it each time, exponential growth happened rapidly.

Posted by: Stellingwerff Aug 15 2012, 07:24 AM

Just a quick update notice:

http://msl-raw-images.appspot.com/lists.html

has been updated with the following changes:

• Filter settings are now kept over page reloads
• "Show All" is now persistent, will not fall back to paging unless "Show less" is clicked
• A camera multiselect is added, empty means "all cameras"

Posted by: jmknapp Aug 15 2012, 11:14 AM

QUOTE (Stellingwerff @ Aug 14 2012, 11:46 PM)

Posted by: vk3ukf Aug 15 2012, 11:39 AM

Hi all,

this doc might help help those looking to decode the image file names a little bit.

It's regarding the sclk breakdown.

http://naif.jpl.nasa.gov/pub/naif/MSL/kernels/sclk/msl.tsc

I've been looking for the full MSL EDR SIS file name definition for ages.

I saw further back there, a fellow (rtphokie), piecing it together, thanks for what you've figured out so far.

The MER version,

<scid><inst><sclk><prod><site><pos><seq><eye><filt><who><ver><ext>

(above from, http://pds-imaging.jpl.nasa.gov/documentation/MER_camsis_v4_9-25-07.pdf)

and this is a great straight forward page for the MER EDR SIS,

http://marsrover.nasa.gov/gallery/edr_filename_key.html.

For the Phoenix lander PHX EDR SIS, this the pdf to read,

http://pds-geosciences.wustl.edu/missions/phoenix/phxCamSis.pdf

MPF and Viking links escape me at the moment.
They were a bit odd by today's standards, short file names and part of the sclk string was the storage directories name.

If anyone knows if MSL EDR SIS is online somewhere yet, love to read it.

Any further pointers greatly appreciated.

Perhaps it will appear at the Analysts Notebook, early next year, hopefully before then.


Kevin. Werribee.

Posted by: Stellingwerff Aug 15 2012, 12:03 PM

QUOTE (jmknapp @ Aug 15 2012, 01:14 PM)

QUOTE (jmknapp @ Aug 15 2012, 01:14 PM)

Posted by: jmknapp Aug 15 2012, 12:25 PM

QUOTE (Stellingwerff @ Aug 15 2012, 08:03 AM)

Posted by: Stellingwerff Aug 15 2012, 08:14 PM

Hi all,

Another update, starting to get there:)

• Added the date/time from the JPL website, sorting on that column (=stereo images closer together)
• Reload button
• Social/sharing widget
• Styling (background can be switched off!)

Posted by: elakdawalla Aug 15 2012, 10:05 PM

This is awesome. Since you asked, the "killer app" would be the ability to tick checkboxes next to images of interest and then click one button to download them all. Not sure how that would work since you're not actually hosting the images but it's what I love best about the PDS Rings Node's OPUS search tool. Next best thing would be to tick checkboxes and click one button that would spit out a text listing of the full URLs to the selected images, which I could then get with wget.

I never run out of feature ideas Good thing I'm not actually a programmer, I'd only produce bloatware that never got to market.

Posted by: RoverDriver Aug 15 2012, 11:32 PM

If I may suggest something, I think it would be critical to also have a database linking specific features and observations to specific documentation images. You have no idea of how many hours I have spent on MER trying to track images pertaining certain events because I did not have them indexed. With a larger science package and longer expected mission duration it will be a nightmare to keep everything organized on MSL.

Things I had been asked to trace were:

1) when was the first/last time some activity happened
2) how many timed did we do some activity
3) when did a specific event/anomaly happened
4) when was the anomaly resolved or a workaround put in place
5) find where and when did we cross a specific type of terrain
6) do we have images of a specific feature/location

These sort of things and you can extrapolate similar queries from the science perspective. Unfortunately, I'm not really a database guy so I do not have specific suggestions on how to handle this, but the sooner we start keeping track of things the better we are off. Even if later on we need to transition to other methods or architectures I think it is best
to start doing something now. What do you guys (and gals) think?

Paolo

Posted by: Reckless Aug 15 2012, 11:53 PM

This retrieval system is great but for me the links to B & W images (nav cams etc) don't work I get an error message XML has no style information.
any idea if I've got my settings wrong.

Roy F

Posted by: jmknapp Aug 16 2012, 12:31 AM

I added an RSS feed for the latest released images. Click on the icon at the top of the image listing

http://curiositymsl.com

Posted by: algorimancer Aug 16 2012, 12:58 AM

QUOTE (Stellingwerff @ Aug 15 2012, 03:14 PM)

Posted by: algorimancer Aug 16 2012, 01:08 AM

QUOTE (algorimancer @ Aug 14 2012, 07:11 PM)

Posted by: jmknapp Aug 16 2012, 09:21 AM

QUOTE (RoverDriver @ Aug 15 2012, 06:32 PM)

Posted by: MahFL Aug 16 2012, 10:04 AM

My impression was they gave the task to "scientists" who at that point did not have much to do.............
As for inconsistancy, thats why you have a Chief Project Scientist, he has the final say.....

Posted by: RoverDriver Aug 16 2012, 10:56 AM

QUOTE (MahFL @ Aug 16 2012, 03:04 AM)

Posted by: Stellingwerff Aug 16 2012, 02:41 PM

Somebody will have to talk with the MSL website people, their Sol=0 page is now >150.000 lines long and still growing. (>1.5MB text by now) I'm running into severe problems parsing the document in an automated system.....

Anyway, Sol 10 pictures have arrived, including chem-cam images!

grz,
Ludo.

Posted by: elakdawalla Aug 16 2012, 04:03 PM

QUOTE (jmknapp @ Aug 16 2012, 01:21 AM)

Posted by: maschnitz Aug 16 2012, 07:47 PM

Emily, if you're in contact with Michelle, could you ask her to put the time in to the Last Updated notice on the raws page? This one: "Last Updated: 08/16/2012 UTC". It's a little hard to tell when new images dropped. You have to memorize the count of images last time, instead.

I figure it's more constructive for one person to deliver requests than all of us to pile on the poor lady. Of course, don't let me nominate you for this if you don't want to do it.

Posted by: mhoward Aug 16 2012, 08:05 PM

QUOTE (elakdawalla @ Aug 16 2012, 10:03 AM)

Posted by: imipak Aug 16 2012, 08:17 PM

QUOTE (jmknapp @ Aug 16 2012, 10:21 AM)

Posted by: elakdawalla Aug 16 2012, 08:34 PM

QUOTE (mhoward @ Aug 16 2012, 01:05 PM)

Posted by: Stellingwerff Aug 16 2012, 08:41 PM

Hi all,

In part due to the huge sol=0 page (currently 250.000+ lines) and in part through bugs/issues I created, my app currently is "intermitted".... I'm working on it!

The good news is: As soon as I have the performance issues fixed, I'll have image release dates in the app as well, including sorting. Which means you can easily see what's new.

Grz,
Ludo.

Posted by: Stellingwerff Aug 16 2012, 11:22 PM

Ok, definitely not happy with the performance (and degradation thereof if something goes wrong) but at least it works again.

New features:

• New image filter (<24 hours ago released)
• Show and sort by release date
• Show date in local timezone or UTC

Posted by: 3d_mars Aug 17 2012, 03:59 AM

First post. Here's a small script I wrote to automatically process left and right full frame Navcam image pairs into SBS stereoscopic images. Thanks go to jmknapp for his image list service, http://curiositymsl.com. I originally intended to automate the full process, from downloading the images to creating the final SBS output, but then I realized that I may want to process the images a bit before the final step. I use the free http://stereo.jpn.org/eng/stphmkr/ to create the final images. It's a windows app which is why I chose to write this in http://www.autoitscript.com/site/autoit/ (also free).

BTW, the images are PHENOMENAL when viewed with the http://store.sony.com/webapp/wcs/stores/servlet/ProductDisplay?catalogId=10551&storeId=10151&langId=-1&partNumber=HMZT1. I purchased this hardware a few months ago in anticipation of this mission (and other 3D applications), and I've been extremely pleased with the quality of the high contrast dual OLED displays, which present a ghost-free stereo image.

Posted by: jmknapp Aug 17 2012, 01:36 PM

QUOTE (3d_mars @ Aug 16 2012, 10:59 PM)

Posted by: jmknapp Aug 17 2012, 01:41 PM

Some updates to my http://curiositymsl.com:

1) results selectable by camera (MASTCAM,NAVCAM,CHEMCAM,HAZCAM,MAHLI,MARDI)
2) number of items per page selectable (20, 50, 100)
3) thumbnail size selectable (small,large)
4) removed "lag" column
5) added columns for sol number and LMST of observation

Note that all parameters selected appear in the url, so if you have a particular kind of search you like, you can just bookmark it to return to it.

Probably about it for now. I might put in a selection for sol number. Happy raw image harvesting!

Posted by: fredk Aug 17 2012, 02:07 PM

QUOTE (jmknapp @ Aug 17 2012, 02:36 PM)

Posted by: 3d_mars Aug 17 2012, 02:40 PM

QUOTE (jmknapp @ Aug 17 2012, 05:36 AM)

Posted by: fredk Aug 17 2012, 02:41 PM

QUOTE (jmknapp @ Aug 17 2012, 02:41 PM)

Posted by: ChrisC Aug 17 2012, 03:15 PM

QUOTE (jmknapp @ Aug 17 2012, 09:41 AM)

Posted by: mcaplinger Aug 17 2012, 03:21 PM

QUOTE (3d_mars @ Aug 17 2012, 07:40 AM)

Posted by: elakdawalla Aug 17 2012, 04:41 PM

http://www.unmannedspaceflight.com/index.php?showtopic=7419&hl=; I hope people will share their experiences/recommendations over there.

Posted by: jmknapp Aug 17 2012, 06:34 PM

QUOTE (fredk @ Aug 17 2012, 09:41 AM)

Posted by: atomoid Aug 18 2012, 12:09 AM

nice work! this is a very handy and fun browser to poke around with, i fear if the word gets out it will be overloaded much too soon!!
i was wondering what sorting by 'thumbnail' could possibly mean.
of course it means: "Unknown column 'thumbnail' in 'order clause'"

Posted by: jmknapp Aug 18 2012, 12:38 AM

QUOTE (atomoid @ Aug 17 2012, 08:09 PM)

nice work! this is a very handy and fun browser to poke around with, i fear if the word gets out it will be overloaded much too soon!!
i was wondering what sorting by 'thumbnail' could possibly mean.
of course it means: "Unknown column 'thumbnail' in 'order clause'"

Posted by: atomoid Aug 18 2012, 01:32 AM

of course its tempting to ask for more features. and i used to be a QA guy so i cant resist so in case you ever tire of looking at all those beautiful MSL images and want to get back to coding heres something: i was tempted to ask for a selector for 'full size' vs 'thumbs' (or whatever other options come down) in order to nix all those preliminary downsamples and focus on the final hi-res versions. [EDIT: i overlooked the 'type' column which could better sort thes images by full frame (F or E), cropped (D) or thumbnail (T, I)]

I see you've nicely put the WxH info in there as separate columns which seems to accomplish the same thing so i can sort of pseudo-zoetrope the mardi images by scrolling the browser window to see them play out as a movie. however it looks like theres a lot of breaks still in the mardi hi-res images, but i think they just arent sorting perfectly using my approach, 0000MD9999000442E1_DXXX is followed by 0000MD9999000599E1_DXXXusing this approach, but 0000MD9999000443E1_DXXX does exist in hi-res its just sorting elsewhere using this method, so i guess we cant stack 'order by' priorities like as if it were an sql query.

Posted by: maschnitz Aug 18 2012, 02:01 AM

I was about to do this very site on a long weekend I'm having, but you basically beat me to it. Nice work Mr Knapp.

As a kind of braindump, here's a bunch of ideas I had for the site. Take 'em, and run with 'em, or leave 'em - all yours:

• I think the main one was just that I was going to make it more jQuery-y - changes to the view settings would be updated instantly, a click on a thumbnail wouldn't be a full page hit. Eventually I would've make it like GMail - no page hits besides the first, just AJAX updates.
• A variety of view modes. The list, pretty much like you have it. But also a contact sheet mode - just a big grid of pictures. It would have the metainfo come up on hover. Also an Imgur- or Flickr-like slideshow mode. Each view mode needs to present photo metainfo slightly differently.
• Links back to the JPL page.
• Cookied view settings
• Binding the JPL-released thumbs types with the subframes/fulls into one line (assuming JPL leaves enough info to do that), then hiding the lesser-sized shot but presenting the option to see it.
• Optional columns (again, cookied) so people can customize their view a bit
• Somehow allow links to full panorama versions, if a shot is a part of a panorama (user-edits? user submissions?)
• Looooong term I was thinking like, user tags and a search of the meta info, maybe even comments and/or votes (again, Imgur/Mlkshk as a guide star here).
• And like a few folks were saying above - this is an ideal spot to collate other metainfo about shots. If JPL ever releases pointing info, that color-map some of the more science-y folks want, etc, this would be an ideal spot to put it.
• Also, I had a few thoughts on how to politely spider JPL's site (using HEAD HTTP requests, checking the "Last Updated" text, etc) but I guess they don't care that much.

Posted by: Stellingwerff Aug 19 2012, 08:40 PM

And we're back!

Man, someone said something in the beginning of this thread about "programmers making embarrissing mistakes". Well, mine was depending on too experimental, self-made database. Scalability issues abound. Anyway, after locking myself up for four days, the site is back online and reacting quick enough. (max waiting time should be below 15s for a full Google Appengine spin-up delay, normal reload delay of about 1-5 seconds.) Normal services have resumed by now.

Check: http://msl-raw-images.appspot.com/lists.html

Hopefully I'm back to feature request implementation by now.

Greetings,
Ludo.

Posted by: jmknapp Aug 20 2012, 12:16 AM

QUOTE (Stellingwerff @ Aug 19 2012, 03:40 PM)

Posted by: jmknapp Aug 20 2012, 12:38 AM

QUOTE (maschnitz @ Aug 17 2012, 09:01 PM)

Posted by: Stellingwerff Aug 20 2012, 11:53 AM

Ok, back to feature development:

Newest version changes:

• Select range of images and open a text-only url listing for exporting purposes.
  ->first step for Emily's "killer feature" request:)
  ->selector can use gmail style range selections!
• Mark new images with a red star.
• Now works correctly in IE

Posted by: maschnitz Aug 20 2012, 07:14 PM

Hi Joe - thanks for reading over my scattered notes. Let me fill in some of the details.

First and foremost: I don't know what else you're running on this VM or host you have, nor your traffic, but, if my guess is right - don't worry too much about the database. Databases are built to handle this kind of thing, in general. I've done far, FAR worse, many times. (Both on my own and with someone's else money at stake, heh.) The worst thing that could happen is that you'd have to add an index or two somewhere down the road. It's no big deal, especially with only 4 or 5 digits (<100000) of pictures. Feel free to email/PM if you find your database is slowing things down too much.

I didn't explain myself very well on "cookied view settings". Apologies for the confusion. I was referring to selectively saving particular settings that related to view preferences. Not the whole kit and kaboodle. So, currently, I'd think about throwing the items per page, thumbnail size, and maybe the camera selections into a cookie. I would definitely not save in a cookie which page you were on. Also potentially interesting as a "sticky" setting is the optional columns.


BTW I had another idea about a feature - this probably applies equally well to Ludo's site. It'd be great if the site remembered the last time you came. It could mark pictures posted since then as "new", or draw a line in the listing. Sophisticated "new" markers can get tricky. But luckily, this can be simple. Simply set the cookie at time of the current page hit, but only if there's no cookie yet, or if last cookied time is more than 30 minutes ago.

Posted by: Stellingwerff Aug 20 2012, 09:53 PM

Hi,

Thanks for the tip, I'll put it on the todo list. (somewhere on the top, as it's relatively easy to implement)

Nowadays I would not store stuff in cookies, with the advance of HTML5 capabilities I would definitely put stuff only in localStorage, works much cleaner and simpeler.

Greetings,
Ludo.

Posted by: maschnitz Aug 20 2012, 10:09 PM

Yup. HTML5's storage only works on fairly modern (post-2010) browsers, which, sadly, a lot of my users don't have. It's pretty danged nifty, though. Count me jealous.

Session storage is actually how I normally prefer to roll these days, but that depends heavily on your web library having tools for it built-in, or building it out yourself in your database. And it helps to have a user login or at least a permanent cookie to attach to. It gets complicated quickly.

Cookies are a lowest-common-denominator, plain and simple. They're stupidly primitive, but tried-and-true, easy, and widely applicable.

Any of the above would work, under appropriate assumptions. It's nice to have options.

Posted by: elakdawalla Aug 21 2012, 07:41 PM

QUOTE (Stellingwerff @ Aug 20 2012, 04:53 AM)

Posted by: jmknapp Aug 22 2012, 12:52 AM

QUOTE (maschnitz @ Aug 20 2012, 03:14 PM)

Posted by: fredk Aug 22 2012, 01:02 AM

It's easy to mark your last visit now - just visit the latest image link (whether it's interesting or not), and next time look for the first visited colour link.

One odd thing I noticed first this morning, Joe - the site no longer displays vertically the full page. It now gives you a narrow window and vertical scroll bar:

Posted by: jmknapp Aug 22 2012, 01:22 AM

Whoa, sorry about that. I just tried it with my version of Firefox & it seems OK. Wish I could reproduce that. One change I made yesterday was to re-add the "lag" column. I think that's interesting after all to see which images show up very quickly versus which ones take longer. MAybe it's a glimpse into the priorities of the investigation. Anyway, maybe the columns no longer fit horizontally which causes your window to scroll?

BTW, I made a couple other changes per maschnitz's suggestions: the "items per page" and "thumbnail size" options are now cookied and persistent.

Posted by: maschnitz Aug 22 2012, 03:31 AM

QUOTE (jmknapp @ Aug 21 2012, 05:52 PM)

Posted by: Stellingwerff Aug 22 2012, 05:02 AM

Early morning release:

• Better IE support
• Support subframe images of non-Malin cams
• Create wget script from export
• Minor tweaks

Posted by: elakdawalla Aug 22 2012, 05:26 AM

Well, I know what I've been leaning toward as a nickname for Curiosity. But I don't feel like the nickname's been earned yet. I need to wait to see some drives and have this monstrous thing turn into a moving machine.

As for graphic design, honestly, this is one place where less design is better! No http://en.wikipedia.org/wiki/Chartjunk!

Posted by: Stellingwerff Aug 22 2012, 05:36 AM

Completely agree, less is way more. That's why it is such a challenge!

Posted by: elakdawalla Aug 22 2012, 08:01 PM

Ludo, another feature request. In the "Type" column, it would be great to have the option "remove thumbnails" instead of or in addition to "full only." Lots of images will get returned that are "downscaled" and will never be returned at full resolution.

Posted by: Stellingwerff Aug 22 2012, 09:35 PM

I've a new version ready for your enjoyment, would have been sooner had Google Appengine cooperated with the deployment....

Anyway, changes:

• Make image types a multiselect (Thanks Emily for the heads-up)
• Minor fixes
• Make the page better show-up in Google searches (e.g. added author info)

Posted by: walfy Aug 22 2012, 09:48 PM

QUOTE (Stellingwerff @ Aug 22 2012, 01:35 PM)

Posted by: elakdawalla Aug 23 2012, 05:20 PM

Split Curiosity nicknaming posts to http://www.unmannedspaceflight.com/index.php?showtopic=7427&hl=.

Posted by: Stellingwerff Aug 24 2012, 08:28 AM

Nice round image milestone: 4000! Only many 10s of thousands to go:)
Really shows the scalability goals of any listing tool....

Posted by: jmknapp Aug 24 2012, 10:41 AM

Yes, particularly when having to depend on scraping web pages--would be very nice if JPL could provide JSON-style queries on their images. As it is, as the sols rack up & given that an image for any of the sols might belatedly come in on any day, to be thorough the individual page for each sol needs to be checked.

I'm thinking it might be good to only check, say, the last 15 sols or so on a frequent basis, and scan all the sols once a day or something like that.

Posted by: Stellingwerff Aug 24 2012, 11:02 AM

I haven't checked yet, but how about parsing the first page (with their per camera, per sol listing) for marked sols? I currently check that page already for the total count, should be relatively straight forward to obtain a workset from it, too.

It would be particullarly welcome if we could skip sol 0, as it is quite large due to the Descent images....

[Edit] Yep, checked it: any line ending on '<span class="OrangeRawMark">Sol' preceeds a line with sol number. Not sure how long JPL keeps images "new", as sol 17s images are currently unmarked....

Posted by: jmknapp Aug 24 2012, 11:47 AM

Re: checking the first page for the "OrangeRawMark" tag--that's a good idea, although one more thing to depend on not changing. I'm thinking I need to detect a "parse error" condition and send myself a text message if/when the raw website design changes and breaks the scraper.

Posted by: Stellingwerff Aug 24 2012, 03:06 PM

Nope, the frontpage trick seems not trustworthy as the current situation is not correct: There are at least 8 new navCam images in the last day (after the currently marked sol0 and sol13 images) which are not marked in orange.....

Posted by: jmknapp Aug 25 2012, 05:42 PM

A couple recent changes to http://curiositymsl.com

- added a persistent option to display time in local time zone
- keeps track of last visit & tags new images with a small DSN icon

Posted by: Stellingwerff Aug 26 2012, 06:58 PM

Hi All,

Proud to present: Local lists of images! From now on it's possible to select a range of images and save them to personal lists. Use them to keep track of specific situations, sources for a particular panorama, etc. Head over to http://msl-raw-images.appspot.com/lists.html (refresh your caches on loading (ctrl-F5/shift-F5)) and look at the new feature!

That's the good news, now the "caveats":
This is very early work and I have loads of stuff I like to add to it. Especially I like to provide serverside storage of lists, to provide two services: better protection against losing lists (They are currently stored in your webbrowser, not the safest place) and more importantly: The ability to share lists with other people. That will be the next step I'll work on in the coming week and I do plan on keeping the current lists compatable with the sharing features. So lists you create today can be shared tomorrow, no worries there:)

Please use the lists and give me feedback on them, especially on the "user-experience". If anything strikes you as "annoying", "plain-wrong", "pretty nice", "I haven't a clue what this should do", etc. don't hessitate to drop me a message. I did put a small help button in there as well, just to explain the icons!

You might have noticed that I did some effective performance optimizations over the last few days, I am now quite happy about the response times (and the closely related financial back-lash from Google AppEngine usage).

Greetings,
Ludo.

[Edit] I forgot a major caveat: currently the meta information of a list (name, description, url) can't be editted. You only have the option to create a new list, possible based on the old lists images. Editting existing meta info will be implemented in a near-future version.

Posted by: Pando Aug 26 2012, 07:18 PM

QUOTE (jmknapp @ Aug 25 2012, 10:42 AM)

Posted by: Stellingwerff Aug 26 2012, 07:20 PM

They are not very consistent with the casing of the suffix. I actually ended up checking both upper and lower case while doing the head retrieval. That way I doublecheck (and repair) any incorrect filename.

Posted by: jmknapp Aug 27 2012, 12:11 AM

QUOTE (Pando @ Aug 26 2012, 02:18 PM)

Posted by: 3d_mars Aug 27 2012, 03:32 AM

Here's version 2.0 of my Navcam download tool, which simplifies the process of downloading multiple left/right stereo image pairs and creating stereoscopic images. This version includes a GUI and supports anaglyph as well as SBS stereoscopic 3D images, a horizontal image offset, a function to find the latest sol number, and a few other changes and additions. A windows executable is attached and the source is below. Here are the links to http://stereo.jpn.org/eng/stphmkr/ and http://www.autoitscript.com/site/autoit/. Thanks again jmknapp for http://curiositymsl.com.

After downloading the image pairs and creating the script file, run StereoPhoto Maker, select File, Multi conversion from list, select the script file, the output image type, and then click the Convert Files button.

http://imgur.com/7Hzgj

 navcam_download.zip ( 312.94K ) : 280

Posted by: Pando Aug 27 2012, 03:54 AM

QUOTE (jmknapp @ Aug 26 2012, 04:11 PM)

Posted by: Stellingwerff Aug 28 2012, 09:48 PM

Hi all,

I've done another update to http://msl-raw-images.appspot.com/lists.html

• New, cleaner layout
• New images are announced on Twitter: http://www.twitter.com/MSLRawImages
• Fixed the export function
• Many serverside enhancements for faster HTML scraping, now checking every 5 minutes.

Posted by: Stellingwerff Sep 4 2012, 08:25 PM

Hi all,

Remind me to never ever say anything about backwards compatibility again.... Big sorry for anybody who has made lists on my site in the last few weeks, I've repeatedly broken them:(
I will really try to not do that again...

To "force" me to not do so, let me share a list with you all as an example:
http://msl-raw-images.appspot.com/lists.html?subscribe=C55FA0C8-2660-0001-F75B-1C00EE731309

It subscribes you to a list I've made of most full images that show Curiosity herself. The list could for example be used to make a nice self portrait. I plan on keeping the list updated, so expect it to grow in the coming period when new images become available.

Ok, the customary feature change report:

• There is now a gallery button, which opens the current image listing in a gallery viewer for viewing convenience.
• Maybe you've missed in in the earlier release notes: Do follow MSLRawImages on twitter, as it will report on new images becoming available.
• Lists can now be shared through an URL link.

• Making list meta info (description, url, etc.) editable
• Making shared lists searchable. (now the only way to subscribe to lists is through the link, like I do above)

Posted by: Stellingwerff Sep 12 2012, 11:35 AM

MAHLI is sending a set of new images with a slightly different filenames: I now see filetypes R,U,S & T, besides the regular E & I. Anyone knows what each letter stands for? I'll do an analysis myself later this afternoon if nobody has a list available somewhere. Quick Google didn't produce a result (yet).

Greetings,
Ludo.

Posted by: mcaplinger Sep 12 2012, 02:24 PM

QUOTE (Stellingwerff @ Sep 12 2012, 04:35 AM)

Posted by: fredk Sep 20 2012, 05:23 PM

Joe - I'm seeing all the latest images on your site listed as "sol 0" - is there a chance something's broken?

Posted by: Joffan Sep 20 2012, 06:54 PM

QUOTE (fredk @ Sep 20 2012, 10:23 AM)

Posted by: jmknapp Sep 20 2012, 07:00 PM

Thanks for the heads up--I'll check it out. I'm guessing that either the NASA website html changed or these recent photos don't have a timestamp for some reason.

Jan-1-2000 is when NASA ephemeris time currently starts (et=0) so indeed it's a null capture date.

Posted by: centsworth_II Sep 20 2012, 07:07 PM

Y2K strikes at last!

Posted by: jmknapp Sep 20 2012, 07:44 PM

QUOTE (centsworth_II @ Sep 20 2012, 02:07 PM)

Posted by: Explorer1 Sep 21 2012, 05:04 AM

Will this include streamlining the page select at the bottom, Joe? The list of page numbers covers nearly the whole browser screen at the bottom, and that's in just over a month of imagery! I know I can set the items per page to 100, but that will just delay the inevitable as it grows and grows. For a mission that is going to last years, there must be a more permanent solution.

Posted by: jmknapp Sep 21 2012, 12:25 PM

Here's another go at an alternative interface to the raw images:

http://curiositymsl.com/search

Try searches like:

mastcam -- all mastcam images
mastcam sol39 -- all mastcam images on sol 39
mr large -- all large mastcam right images
navcam -thumbnail -small -- all navcam images excluding thumbnails and smalls
mardi -sol0 -- all mardi images excluding sol 0
sunrise -- all images taken at sunrise

Mousing over the thumbnails brings up a preview. Scrolling to the bottom of the page loads more items if available.

Click on 'help' for more info.

Posted by: Reckless Sep 21 2012, 01:47 PM

Brilliant
chemcam large is good too
and many thanks

Roy

Posted by: fredk Sep 21 2012, 02:04 PM

This looks great, Joe! One suggestion: could you specify what the various sizes mean on the help page - maybe in a table?

Posted by: jmknapp Sep 21 2012, 02:31 PM

QUOTE (fredk @ Sep 21 2012, 09:04 AM)

QUOTE (Reckless @ Sep 21 2012, 08:47 AM)

Posted by: Ant103 Sep 21 2012, 02:52 PM

Wow ! Great great job there !

Will it be possible to display the result in grid form ?

Posted by: jmknapp Sep 21 2012, 06:05 PM

QUOTE (Ant103 @ Sep 21 2012, 09:52 AM)

Posted by: Ant103 Sep 21 2012, 06:28 PM

Hmm, maybe keeping some meta-info like the the filename and Sol. And by hovering, display additional informations. And one click to access the full version.

Posted by: elakdawalla Sep 25 2012, 05:48 PM

Joe, this is really promising. I have a couple of requests if you'd be willing to entertain them:

- I agree with Ant103's request for a grid view of thumbnails. Right now the only site that gives you that is *shudder* JPL's. Only meta-info I'd need displayed with that are sol, camera abbreviation, and LMST (ideally using a 24-hr clock, thus avoiding "AM" and "PM"); I'd stick filename in the alt text so it'd be displayed on mouseover.

- Not very many results are displayed; in order to see all results I have to do a *lot* of clicking or wheel-scrolling. Too much of that kills my wrists :| I'm not sure how to fix this (though a grid display of lots of thumbnails would help).

- The homepage mirroring Google is cute, but I'm wondering if you could just put the help text on the homepage, or at least some suggested searches. I keep having to go to help to remind myself of syntax.

Posted by: jmknapp Sep 25 2012, 06:56 PM

QUOTE (elakdawalla @ Sep 25 2012, 01:48 PM)

Posted by: EdTruthan Sep 25 2012, 08:22 PM

I second the Wow's! Joe the search page is just stellar. Cannot thank you enough. MSL living just got much easier. If you're ever putzing around with adding improvements it'd be a nice feature too if the results for a given image could somehow include a "flag" of some kind indicating whether a corresponding left or right match were available (especially helpful with Navcam). Knowing what's obtainable (or not) for anaglyphing would then be a breeze. Thanks again for the great job.

Posted by: jmknapp Sep 25 2012, 11:33 PM

QUOTE (EdTruthan @ Sep 25 2012, 04:22 PM)

Posted by: atomoid Sep 26 2012, 12:47 AM

it would be interesting to know how the firmware on MSL incorporates metadata for each image, and if so, it could be accessible for public use. im guessing for engineering purposes they do keep track of certain data per-image such as metrics on exactly where the camera is pointed and be able to sync it to all other metrics recorded by MSL. maybe its all recorded but not necessarily linked and has to be mined and refined if needed for anything.

Tapping into such a matrix of all that metadata could be used to generate movies by querying images that point in a particular direction (like a view of looking out the left window as driving along) or apply logic against a coordinate map to extrapolate images where the camera happened to get a certain coordinate in view (such as 'Jake' or some distant yardang).

Posted by: CosmicRocker Sep 26 2012, 02:16 AM

I'm trying to learn which method is the most efficient for downloading MSL imagery. I know some of you are using wget for downloading, but that seems not to be a convenient option for me on a Windows system. I have to admit that, not being a programming type, writing and testing scripts of one sort or another is not very appealing to me. However, if the potential rewards are great enough I might be willing to give it a try.

Currently I am using the Windows application http://www.unmannedspaceflight.com/index.php?s=&showtopic=7408&view=findpost&p=188174, which was made available at the beginning of this thread. I find it very convenient because:

1. It has an intuitive graphical user interface.
2. It allows the user to filter on camera type, left/right side, and various image characteristics.
3. It downloads the images selected according to the criteria above,
4. saves the files into a logical directory structure,
5. and it maintains some kind of image index so it knows which images you have already downloaded.

Posted by: jmknapp Sep 26 2012, 10:14 AM

True, my web page doesn't enable bulk downloading. With Ludo's app you can select images and export to a list for later use with wget.

As for MSL metadata, I gather that the rover tries to maintain a reasonably accurate idea of where it is in the "topo" ftp://naif.jpl.nasa.gov/pub/naif/MSL/kernels/fk/msl.tf and local level defined at the landing site (X north, Y east, Z up from the landing site), but with wheel slippage and other uncertainties, it's all subject to revision. I wonder if part of the daily drill is to tell the rover "oh, by the way, you're actually over here a little."

The camera frames (X horizontal, Y vertical, Z forward) have a calibrated relationship ultimately to the rover frame (X forward, Y right, Z down). Ideally, one should be able to take an XY position of a pixel in a camera frame and a time of observation and get an absolute position and direction vector in whatever coordinate system is convenient (J2000, MARS_IAU, MSL_TOPO, etc.).

Seems reasonable that the rover EXIF info so to speak would include an initial estimate, but who knows? I think they'll publish this kind of data on the NAIF web site eventually. In the meantime, for us it's by guess and by golly.

Posted by: ronald Sep 27 2012, 07:33 AM

I'm getting used to Stellingwerffs http://msl-raw-images.appspot.com/lists.html.

Especially useful is the generated list of files you want to download. It is for wget only which is fine on my linux machine but at work we use OSX which comes with curl instead of wget. So here is the code line for using curl with a list of urls:

Posted by: RoverDriver Sep 27 2012, 04:01 PM

QUOTE (ronald @ Sep 27 2012, 12:33 AM)

Posted by: Ant103 Sep 27 2012, 04:07 PM

curl is fine too.

Just an example, to how to download a bunch of frame, from "url/folder/0001.jpeg" to "url/folder/0010.jpeg" :

cd destination
curl -O url/folder/00[01-10].jpeg

Works like a charm .

Posted by: ronald Sep 27 2012, 07:04 PM

QUOTE (RoverDriver @ Sep 27 2012, 06:01 PM)

Posted by: charborob Sep 28 2012, 01:38 PM

I've been trying to use Joe's "Curiosity search engine" today, and a search for "sol51" turns up no results, even though navcam images are posted on the MSL raw images pages. Also, a search for "sol50" doesn't list the mastcam images that I see on the raw images page. Is there something I am doing wrong?

Posted by: Stellingwerff Sep 28 2012, 01:48 PM

QUOTE (ronald @ Sep 27 2012, 09:33 AM)

Posted by: fredk Sep 28 2012, 05:14 PM

QUOTE (charborob @ Sep 28 2012, 01:38 PM)

Posted by: Stellingwerff Sep 28 2012, 08:58 PM

As far as I can tell no changes on the JPL site. So we'll just have to wait for Joe to look at it.

My listing site is currently up to date, and includes an early preview of a mosaic/grid thumbnail view. Very early work, so probably not useful yet.
See: http://msl-raw-images.appspot.com/listing.html

Posted by: jmknapp Sep 28 2012, 11:38 PM

QUOTE (fredk @ Sep 28 2012, 12:14 PM)

Posted by: Stellingwerff Oct 6 2012, 09:46 AM

Hi all,

Time for a status update on the http://msl-raw-images.appspot.com site:

The site has run in a degraded mode for a couple of weeks, due to deeply rooted scalability issues. Finally I've found some time to get to the problem and fix it. Since a couple of days the service is fully up and running, including twitter updates @MSLRawImages, running on the 5 minutes update cycle I like it to be on. I like to thank all the visitors that have sticked with me during this time.
Now I'm back to developing features and layout, starting with a couple of cleanups:

• Site is now hosted on the "/" of the URL, instead on lists.html. Makes for a much more convenient URL:)
• Separated date column in "taken","lag" and "Released" (taking lessons from Joe, thanks!)
• Added preference option to select font-size
• Moved UTC v. Localtime selection to preferences menu.
• Moved gallery and mosaic/grid button to icon menu

Posted by: maschnitz Oct 6 2012, 06:15 PM

Nice work, Ludo. Yeah, I suspect the number of pictures have gotten into the "limit/offset and index or die" stage of database usage (16,000+ listed on MSL Raws, sounds about right).

Joe, as promised, here's some database tuning advice. Right now, page hits are taking roughly 10 seconds. This is probably because the select query is returning all 16,000 rows, then tossing most and just displaying the 50 its got, or (less likely) the query takes 10 seconds to sort the results the right way. The latter is a little trickier than the former to fix.

The former is pretty easy. MySQL and PostgreSQL both support "LIMIT" and "OFFSET" clauses, which will restrict the data transferred between the database and the webserver. So, for example, "SELECT ... WHERE .... LIMIT 50 OFFSET 100". Most quality web libraries also allow you to "page" database results. You might be surprised just how expensive shipping data out of a database really is. Databases are much happier doing a lot of crunching on their end, then returning tiny little results.

The latter problem, getting the results sorted quickly, is simple in theory. Add indexes. You can get an order of 100x speedup with database indexes, with only a small cost of insertion and update speed. The tricky part is getting the right indexes. Databases tend to ignore indexes unless they're exactly right. I can go into how to do that if you're interested.

Posted by: jmknapp Oct 6 2012, 09:31 PM

maschnitz, are you referring to the original page at http://curiositymsl.com? If so, the queries are already limited like so, for example:

SELECT * FROM msl WHERE visible=1 ORDER BY etreleased desc LIMIT 1,100

So the slowdown is coming from somewhere else. I'm getting about 3-4 seconds for a 100 results/page query. I think the bulk of the delay is loading the thumbnails. On the road right now, but I can look at it more closely later. I think there's a linux app for profiling mysql that I can look into.

The newer website at http://curiostymsl.com/search uses a much more complex query although it also is limited (to 50 results/query). BTW, both sites use jquery per your suggestion!

Posted by: maschnitz Oct 6 2012, 09:49 PM

Yeah, the root page. Hm. OK. I checked the root page with Firebug - the initial page hit took 4 seconds; the thumbnails as a whole took 2 seconds. There's not a lot you can about the thumbnails besides hosting them yourself.

4 seconds could be better. The index page I think is starting to suffer under all the pictures in the result set, then. Probably will just get worse. Nice work with the limit.

I strongly suspect, then, that you need a few indexes. The way to confirm this is to try running the front-page SQL query by hand against the database; then run it again without the order-by clause. If I'm right, the query will run a LOT faster without the order-by.

So, something you could try is to create an index. Something like "create index etrel_idx on msl (etreleased)". See if that speeds it up. You might need one of those for each of the columns you can sort by on the front page. (Sometimes that's not enough, though - long story.)

Posted by: jmknapp Oct 6 2012, 11:08 PM

I'm not indexing on etreleased or ettaken (maybe the most likely search fields), but even so the queries take only 50 msec or so for 100 items. I find that the time taken to load a page pretty much scales with the number of items displayed (20, 50 or 100). I.e., with a cleared cache, about 2 seconds for 20 items and 9 seconds for 100 items. My guess right now is that besides loading the thumbnails (which can be 10KB each) most of the remaining time is taken up converting the etreleased and ettaken times to a string date (they're stored in the database as ephemeris time, seconds since 1/1/2000). The php script does exec calls to external programs to do these conversions. At least, the time taken doesn't scale with the number of items in the database, but is limited to the number of items in the result set, 20, 50 or 100. As it is, I guess a way to speed it up would be to store the string dates in the database.

Posted by: maschnitz Oct 7 2012, 12:11 AM

Very odd. 4 seconds is an eternity for a computer. Nothing you listed should take more than a second, in theory.

The images in the web page I see are coming direct from JPL's site, after the index page loads in the browser. The four seconds I'm talking about is the amount of time it takes for the index page to load. Are you loading the images from JPL inside PHP, too? (Maybe to get the width and height? If so, that'd be a great thing to stash in the database, just to avoid talking to JPL during your PHP script. Talking to JPL directly from PHP is gonna take some time.)

Of the things you listed, the calls out to convert the dates are the most suspect.

You should be able to do that from within PHP. Here's a trick you can use. Ephemeris time is quite similar to UNIX time - epoch seconds since 1/1/2000, vs. midnight 1/1/1970. You can convert from ephemeris time to UNIX time just by adding 946684800 (the number of seconds between 1970 and 2000). Then PHP has a couple of mechanisms for making a UNIX time a usable date object - eg under PHP 5.3+, you can use:

$unixtime = $ephemeris + 946684800;
$date = new DateTime('@' . $unixtime);

If this does the right thing, this will be a lot faster than shelling out.

Posted by: jmknapp Oct 7 2012, 10:51 AM

QUOTE (maschnitz @ Oct 6 2012, 08:11 PM)

Posted by: elakdawalla Oct 7 2012, 04:50 PM

Just a comment to say, carry on, guys -- I'm learning lots from this discussion! And the continuous tweaks and improvements to the image retrieval tools are great

Posted by: maschnitz Oct 7 2012, 05:21 PM

Excellent, glad that's helping. Here's a tool to convert UTC time to UNIX time that should help: http://www.onlineconversion.com/unix_time.htm

Posted by: jmknapp Oct 7 2012, 07:01 PM

OK, made the et->utc conversion change & it's running quite a bit faster--thanks again!

http://curiositymsl.com

Now to add autoscrolling and listing by sol number....

Posted by: jmknapp Oct 13 2012, 11:21 PM

I updated the http://curiositymsl.com site last night with some new features--unfortunately there was a problem causing the site to be down for 8 hours while I slept--sorry for any inconvenience.

The new features (hopefully they will be considered an improvement or at least not detrimental) are:

1) page links (next, prev, etc.) eliminated in favor of infinite scrolling
2) search order selection moved from column headers to drop-down menus
3) images now listable by sol (dropdown menu)
4) format of image meta info changed
5) some meta info shown in image preview

Grid view is still on the to-do list. I had something that worked fairly well, but with the infinite scrolling the page gets bogged down when, say, a thousand thumbnails are shown, which is pretty easy to get to in grid view.

Posted by: EdTruthan Oct 14 2012, 01:23 AM

List by Sol with infinite scroll! (hey that rhymes...) Perfecto! I know I speak for the masses when I say infinite thanks.

Posted by: Joffan Oct 16 2012, 04:42 AM

Joe, if you're taking requests, I'm generally not interested in images below about 300 pixels is either dimension.... can I filter out of the list?

Posted by: jmknapp Oct 16 2012, 07:00 PM

QUOTE (Joffan @ Oct 16 2012, 12:42 AM)

Posted by: ronald Oct 16 2012, 07:24 PM

Joes and Ludos pages make life so much easier! Amazing, fantastic, Thank you both!

Posted by: Paul Fjeld Oct 17 2012, 09:47 PM

Joe, is there a way to list by LMST as you could when you clicked on the column header? I was using it to compare images at the same time of day (for color and shadow). Other than that, I love the changes!

Posted by: jmknapp Oct 17 2012, 10:34 PM

QUOTE (Paul Fjeld @ Oct 17 2012, 05:47 PM)

Posted by: Joffan Oct 18 2012, 04:09 AM

QUOTE (jmknapp @ Oct 16 2012, 01:00 PM)

Posted by: jmknapp Oct 18 2012, 01:02 PM

OK, I put in a selection to sort by local Mars time, but I don't know how useful that is, because it requires a lot of scrolling to get through the list. Also, the sorting is weird because LMST is stored as a string.

I think an upcoming feature I'm working on--user tagging of images--will be more useful in this regard, as images will be marked with tags such as "afternoon," "sunrise," "night," "midday," etc. & you can search on those tags to get images taken at roughly the same time of day. Should be a few days before that's ready for "beta" though.

Posted by: Paul Fjeld Oct 18 2012, 09:06 PM

I see what you mean. In your older implementation you could pick the page number and jump to roughly where the time of day would be. I love the infinite scroll but it would be nice to jump where you need to go if that is deep in the database. Pickle!

I like the idea of sorting on morning, noon, or etc. Thanks!

Posted by: jmknapp Oct 25 2012, 02:55 PM

OK, finally have a web site for tagging images ready to beta test:

http://marstag.com

It's much like the http://curiositymsl.com web site except searches are filtered by tags rather than menu selections.

To add your own tags it's necessary to set up an account (pretty painless). That allows a modicum level of control and also allows you to do things like delete your own tags and also create private tags that are viewable only by you if desired. Ludo has suggested I use OpenID for this but I'm not too familiar with that yet, so that may come later.The login link takes you to the account creation page.

It should be an interesting experiment in crowdsourcing, assuming that it's developed enough to get used. As always, suggestions and bug reports encouraged.

As an example of what can be done with tagging, here's a search for images that I tagged with 'jake':

http://marstag.com/?q=+jake&thumbSize=large&tzsel=local&orderby=ettaken&sortdir=asc

Give it a try!

Posted by: elakdawalla Oct 25 2012, 03:52 PM

I tried to establish a login, but after half an hour I haven't received the confirmation email (it's not in the spam folder either). Dunno if that's a problem for everybody or just me.

I suggest you establish a controlled or at least strongly suggested vocabulary, or at the very least a way to list tags currently in use. Certain things that get imaged a lot -- the chemin inlet port, marsdial, etc -- should somehow get tagged identically.

It would be great if there were a suggested style of tag that would help collect the images taken as part of a single panorama together. For MER I'd propose the use of a combination of sol number and sequence ID; I haven't paid close enough attention to MSL image file naming to know if there's an obvious way to do this, or if we would need to develop our own naming convention, e.g. "MLsol17panSW" or whatever. We're handicapped by not having the names of targets available as we do with MER...

If this gets used, it will represent a considerable amount of effort, so I would want to ensure that it will eventually get mapped to the PDS data as it becomes available.

Posted by: jmknapp Oct 25 2012, 05:32 PM

I heard from Ed Truthan that the confirmation email ended up in his spam folder, so looks like that is a likely place to check!

To avoid that, add curiositymsl@gmail.com to your spam filter's approved list & it should be OK. I'll put a note about that on the signup page.

Posted by: jmknapp Oct 25 2012, 05:57 PM

QUOTE (elakdawalla @ Oct 25 2012, 10:52 AM)

Posted by: CosmicRocker Oct 26 2012, 03:49 AM

Your site keeps getting better and better. I love it. My registration proceeded very quickly and I had no problem with spam issues.

Posted by: maschnitz Oct 26 2012, 06:14 AM

Just an FYI, Joe - the software lingo that magically opens up results on Google for that tag feature is https://www.google.com/search?q=php%20tag%20autocomplete#hl=en&safe=off&spell=1&q=tag+autocomplete&sa=X&ei=YiiKUOXbHISu8ATw64HQBQ&ved=0CB4QBSgA&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.&fp=5fc5e948d96e0b0c&bpcl=35466521&biw=1194&bih=1306.

JQuery has https://github.com/aehlke/tag-it on the browser side. I imagine the PHP server side has some how-tos out there, too. But it can be "work", and I can definitely understand being hesitant. And thanks again for the site, I use it every day.

Posted by: jmknapp Oct 26 2012, 02:29 PM

Alright, I changed the tag entry page so that it uses a jquery plugin for tag suggestion and auto-completion. Pretty slick!

I put in a list of terms to start, mostly gleaned from the literature, but I'm not too familiar with geological terms, so any suggestions welcome. I'm kind of in awe of the lingo and analysis tossed around here by the geology types.

Here's the initial list:

http://marstag.com/suggestedtags.txt

Posted by: fredk Oct 27 2012, 04:01 PM

Hi Joe - a question about your site http://curiositymsl.com/table/view/. I noticed that the http://mars.jpl.nasa.gov/msl/multimedia/raw/?s=25&camera=MAST%5F that was returned recently didn't show up on your site (yet). Do I remember right that you ignore older released images? How far back do you go? If it means too many http requests (or whatever) to check all of the sols every 15 minutes, can you check the older sols less frequently (even once per day)?

Posted by: jmknapp Oct 27 2012, 06:35 PM

I go back 60 sols so it should have been in the bunch. The sol 25 MR images seem to be there now,, yes? Not sure why there might have been a delay.

Good idea about occasionally going further back, maybe once a day. It does take a while to scrape and process all those pages though, and, knock on wood, this mission might last a thousand sols. There has to be a statute of limitations somewhere!

Kinda funny thinking about a file sitting there on the rover for months waiting for the "call."

Posted by: fredk Oct 27 2012, 08:22 PM

OK, I see the problem. My browser isn't handling the auto loading properly when I scroll to the bottom of the page. You can see in this screenshot a discontinuity of three days, between images released Oct 27th and 24th:

Posted by: fredk Oct 29 2012, 10:09 PM

QUOTE (jmknapp @ Oct 27 2012, 07:35 PM)

Posted by: jmknapp Oct 30 2012, 09:48 AM

I scrape every camera/sol page regardless of the "new" flags--I think Ludo determined at one point that the flags are not reliable.

Posted by: ronald Nov 17 2012, 08:10 PM

Neither Joes nor Ludos sites do work since they changed something over at the Nasa raw images web page. Browsing for images there is really unpleasing and sometimes confusing.

HELP!

Posted by: mhoward Nov 17 2012, 09:53 PM

Yes, the view-by-sol functionality on the raw images site is basically broken as of today. If you view images by sol and save the page data, the saved file won't include any image data. I think it probably has something to do with the sol selector functionality (maybe that's new?).

Also, the page for sol 100 currently isn't even showing all sol 100 images. Oddly, http://mars.jpl.nasa.gov/msl/multimedia/raw/?s=100 doesn't show all the images, but adding http://mars.jpl.nasa.gov/msl/multimedia/raw/?s=100&camera= seems to work correctly. At the moment.

So, the simplest workaround for those of us trying to get data from these pages seems to be to view by instrument and sol, or add "&camera=" to your query.

Posted by: jmknapp Nov 17 2012, 10:49 PM

Yep, looks like they've made some changes, jquerying things up quite a bit, and the usual scrape urls by sol don't work. E.g., it used to be that scraping this url:

wget -O xyzzy http://mars.jpl.nasa.gov/msl/multimedia/raw/?s=100

...would get you a file xyzzy with all the sol 100 images in the html, but now it doesn't, you just see the jquery command.

Anyway, going through the code that is retrieved using the above, I see that this is the magic jquery incantation to get the files:

$("#listImagesContent").load("../../admin/modules/multimedia/module/inc_ListImages_Raw.cfm?s=100");

Accordingly, this wget command works as a replacement:

wget -O xyzzy http://mars.jpl.nasa.gov/msl/admin/modules/multimedia/module/inc_ListImages_Raw.cfm?s=100

Bottom line, I think it's working again. The detail above is for the benefit of Ludo or others trying to scrape the raw image pages.

Posted by: ronald Nov 17 2012, 11:06 PM

BIG thanks!

Posted by: mhoward Nov 17 2012, 11:19 PM

Again: ?s=100 isn't returning all the images. ?s=100&camera= is.

Just FYI.

Posted by: fredk Nov 18 2012, 03:27 AM

Thanks a lot for quickly getting your site working again, Joe! It's such a nice way to browse the new images.

Posted by: Stellingwerff Nov 18 2012, 09:22 PM

Hi All,

My site has been fixed as well. This did show some lack of robustness which I'll fix in the coming period.

Greetings,
Ludo.

Posted by: Stellingwerff Nov 19 2012, 08:34 AM

Aj, my "trick" to find the newest sol doesn't work correctly at this time, seems like a bug on the JPL side. Basically I'm calling:

http://mars.jpl.nasa.gov/msl/admin/modules/multimedia/module/inc_ListImages_Raw.cfm?camera=&s= (used to be http://mars.jpl.nasa.gov/msl/multimedia/raw?s=)

which defaulted to the newest sol, but that link is still only showing sol=101 images, i.s.o. the 102 images.....

Looking for a new trick.... (Maybe going back to scraping the html of the front page to get the highest sol number?)
@jknapp: what are you currently using?

Stay tuned!

[Edit: As we speak it returned to the expected behavior, maybe a caching issue?, still need to make this more robust, tho]
[Edit 2: @MidnightMartian, Thanks a lot, yes you are very right about needing the camera parameter, they seems to have added a filter removing downsamples/thumbnail images for which a full exists]

Greetings,
Ludo.

Posted by: mhoward Nov 25 2012, 08:37 PM

Heads up: the previously mentioned trick of using inc_ListImages_Raw.cfm seems to no longer be working; at least, at the moment, it's not returning the latest full frame sol 107 Mastcam images, instead still showing some thumbnail versions. I'm not going to look into why, I'm just going back to the regular website for now. (Using inc_ListImages_Raw.cfm was nice and fast, but otherwise not essential... I hope.)

Posted by: jmknapp Nov 25 2012, 10:02 PM

QUOTE (mhoward @ Nov 25 2012, 03:37 PM)

Posted by: kemcab2012 Nov 26 2012, 07:29 PM

Not sure if this is well known or not, but looking at the source for the page, I see that each individual sol link on the main page does the following when clicked:
-change the class of the button to selected from non-selected
-Make a call via jQuery to retrieve the contents of inc_ListImages_Raw.cfm using the specified GET parameters (which in that case is just "s=")
-Load the contents of the response from the above call into the div tag with the listImagesContent id (which is the main content screen on the main site).

Also, the cfm file is in another directory from the main site. While the thumbnail images use absolute urls to acquire the images, the raw, full-sized images are accessed via relative links, and are therefore unavailable when a user clicks on the cfm link directly.

I also made two requests to the cfm file - one with "camera=" and the other without. I then saved the source of each request. When I ran a diff between the two files, the one called via "s=" had an extra line with the following contents:
<div class="TotalRawImagesNote" >Total for Sol 107: 373 Images</div><br clear="right">

I hope the above information helps.

Posted by: iMPREPREX Nov 26 2012, 08:30 PM

I really liked the Curiosity RAW GET program.

Can that program be updated to work with the current configuration on the NASA/JPL site?

I love your site too, Stellingwerff. I just don't know my wget and all of that, so I can't mass-download

I know I should learn...

Hate being an amateur, but I love making panoramas!

Posted by: fredk Nov 30 2012, 10:47 PM

Joe, maybe something's changed again on the jpl site? http://curiositymsl.com/table/view/ isn't showing some new sol 113 images...

Posted by: jmknapp Dec 6 2012, 10:37 AM

QUOTE (fredk @ Nov 30 2012, 05:47 PM)

Posted by: fredk Dec 6 2012, 03:18 PM

Pointing info - that's fantastic!

Are you going to let us in on where you get that?

Posted by: Ant103 Dec 6 2012, 03:30 PM

Huh, metadata infos, that's great

But, how did you get these infos ?

Posted by: jmknapp Dec 6 2012, 03:59 PM

The pointing info comes from SPICE kernels used with the NASA NAIF toolkit CSPICE. Post-landing MSL kernels haven't been put on the main NAIF ftp site to date because "their production does not yet contain all telemetry data that we need, resulting in CKs with many gaps and all kernels lacking actual data during motion and appendage articulation periods." CKs are the camera pointing kernels. However, the team does put out some form of MSL surface kernels twice per sol that they use for rough planning and operations I guess. That's what I used to get the pointing info.

It's interesting that Google Mars now shows the current traverse as a path on the map & I notice that it's exactly what I get from the MSL SPK kernels (those kernels have the position information), so evidently Google is tapped into that source.

Posted by: mhoward Dec 6 2012, 04:37 PM

So does that mean that the current data isn't on the public FTP site? Because I still don't see it there.

Posted by: mcaplinger Dec 6 2012, 05:10 PM

QUOTE (mhoward @ Dec 6 2012, 09:37 AM)

Posted by: mhoward Dec 6 2012, 05:31 PM

Thanks. I'd missed that one. It looks likely.

Posted by: jmknapp Dec 6 2012, 05:52 PM

That's it. I sent an email to a NAIF guy to see if it was OK to post the url, which it probably is, but now that's a moot point!

There's no readme file in that directory so here's a description of the files for anyone interested:

http://curiositymsl.com/mslkernels.txt

Posted by: mcaplinger Dec 6 2012, 06:46 PM

QUOTE (jmknapp @ Dec 6 2012, 10:52 AM)

Posted by: jmknapp Dec 6 2012, 07:00 PM

QUOTE (mcaplinger @ Dec 6 2012, 01:46 PM)

Posted by: mhoward Dec 6 2012, 07:18 PM

Well, it's public info for the moment anyway. But even in plain sight it's probably still pretty obscure.

Thanks for the example, Joe. Guess I know what I'll be doing today.

Posted by: elakdawalla Dec 6 2012, 09:44 PM

I just exchanged email with Michelle Viotti. She said that they are running a couple of test projects, one of them with Ames (which explains the Google Mars part), with the goal of putting the metadata out via JPL's website publicly in the future. No time frame given.

Posted by: jmknapp Dec 7 2012, 11:05 AM

QUOTE (elakdawalla @ Dec 6 2012, 04:44 PM)

Posted by: mhoward Dec 8 2012, 08:49 PM

Yep - that'll work.

Posted by: jmknapp Dec 8 2012, 09:05 PM

QUOTE (mhoward @ Dec 8 2012, 03:49 PM)

Posted by: mhoward Dec 8 2012, 09:53 PM

I'm getting good pointing data for Navcam as far as I've seen so far, but I'm taking a slightly different approach. What I need is the rover-relative camera mast az,el, and the rover orientation quaternion; that's what I've got for MER and it makes sense for what I'm doing. So, my approach so far is:

1) Calculate the ephemeris time from the UTC time stamp from the raw images page using str2et_c.
2) Calculate the encoded SCLK from the ephemeris time using sce2c_c.
3) Get the camera mast az,el in two steps using ckgp_c:

// FRAME_MSL_RSM_AZ = -76202
ckgp_c(-76202, esclk, 0, "MSL_RSM_ZERO_AZ", cmat, &clkout, &found);
// ... pull out the azimuth in a way similar to your code example ...
// then get the elevation
// FRAME_MSL_RSM_EL = -76204
ckgp_c(-76204, esclk, 0, "MSL_RSM_ZERO_EL", cmat, &clkout, &found);
// ... pull out the elevation, and there you go.

There's probably a simpler way to do it, but I don't know how, I just started learning CSPICE yesterday.

I'm getting those rover az,el values from msl_surf_rsm_tlmenc_spanned.bc. I have no idea if that's the correct file, but it seems to be working for the moment. (Edit: some time later, it turned out that msl_surf_rsm_tlmres_spanned.bc is better for what I'm doing. msl_surf_rsm_tlmenc_spanned.bc does not seem to be accurate enough when it comes to exact placement of M-100 images.)

Actually the problem I'm having is with ChemCam. I get wrong values there, and no idea why. For starters, the time stamps on ChemCam images on the raw images page don't seem to match up with the rover clock value in the filenames, so... who knows. That's a problem for another day. Now I need to figure out how to pull out the orientation and position information.

Sorry if that doesn't help, but at least it indicates that there is good data along the line somewhere.

Posted by: mcaplinger Dec 8 2012, 10:29 PM

QUOTE (mhoward @ Dec 8 2012, 02:53 PM)

Posted by: mhoward Dec 8 2012, 10:49 PM

QUOTE (mcaplinger @ Dec 8 2012, 04:29 PM)

Posted by: jmknapp Dec 9 2012, 01:29 AM

QUOTE (mhoward @ Dec 8 2012, 04:53 PM)

Posted by: fredk Dec 9 2012, 02:33 AM

Oh man, that's cool, Joe! Does this mean you've put our resident map makers out of work?

Edit: comparing with Phil's map, there are obvious offsets, though maybe those are due to translating rover location coordinates into positions on the orbital image. That would mean understanding the image projection.

Still, you've really outdone yourself here!

Posted by: jmknapp Dec 9 2012, 01:10 PM

QUOTE (fredk @ Dec 8 2012, 09:33 PM)

Posted by: jmknapp Dec 17 2012, 02:55 AM

OK, another feature update related to mapping--the image listing on http://curiositymsl.com now has a "map it" link associated with each image. Clicking on it brings up a page with a map showing the rover position on the left hand side, and the associated image on the right. If camera pointing information is available (currently only for MASCAM and HAZCAM images), an arrow on the map indicates the direction.

Here's an example of the "map it" page for an image without pointing info, http://curiositymsl.com/imgpoint.php?name=NLA_408954917EDR_S0051576NCAM00535M_

...and with pointing info, http://curiositymsl.com/imgpoint.php?name=0106ML0681003000E1_DXXX

Posted by: elakdawalla Dec 17 2012, 03:29 AM

Wow. That is awesome.

Posted by: walfy Dec 17 2012, 03:48 AM

Way beyond my wildest dreams. What a wonderful reference! Usually had to work my brain to guess which way I was looking. Fantastic coding wizardry! Deepest thanks.

Posted by: Poolio Dec 17 2012, 06:18 AM

Amazing. I have often struggled to map images to terrain. With this, each location on the map has an immediate sense of place, making the stops on the journey that much more vivid and tangible. I think I have a new favorite website. Thank you for this.

Posted by: CosmicRocker Dec 17 2012, 06:23 AM

QUOTE (jmknapp @ Dec 16 2012, 08:55 PM)

Posted by: Stu Dec 17 2012, 09:01 AM

That is fanTASTIC, thank you! Really brings Curiosity's mission to life.

Posted by: Explorer1 Dec 17 2012, 09:12 AM

That's amazing Joe, bookmarked!
Presumably the map will get extended as the rover moves beyond the borders?

Posted by: jmknapp Dec 17 2012, 02:10 PM

I just wanted to get a better sense of the context myself, particularly a side-by-side view of map and image--that's really the sweet spot, absent a totally immersive 3-d visualization only a few years away down the road surely!

The main thing bugging me now is that for some reason I'm not getting accurate NAVCAM pointing info from the NAIF data files, don't know if it's user error or what. With a 45 degree field of view the NAVCAMs really give a good context. I keep getting that the NAVCAMs are pointed about 40 degrees down and the azimuth isn't right either. It's weird because the calculation for the MASTCAMs is very similar and they are all fixed to the mast pointing more or less in the same direction.

Knowing the CHEMCAM pointing would also be very useful as it's usually very hard to figure out what rock it's looking at. But that data comes out squirrelly too at the moment.

I can add more map tiles to expand the area covered, but as of now unless Curiosity heads deep into Yellowknife Bay to the east, I think it's due to turn around and head more or less southwest, so I think the current map will be good for months.

As for clicking on the track and getting some sort of display of images from on/near that location, I guess Mars geeks think alike, because I was thinking the same thing. There are a few edge cases to consider. Like how about if the rover moves 1cm--is that a new location? The maximum scale of the map is 0.25 meters per pixel.

One funny thing I noticed: the sequences of NAVCAM thumbnails seen occasionally are taken when the rover is moving, or at least in the middle of the traverse, Also, haven't looked at the extremely fine detail yet (maybe it isn't even totally valid for these preliminary files) but the track seems to indicate that the rover travels in a fine-scale zig-zag pattern occasionally.

Posted by: Greenish Dec 17 2012, 03:32 PM

Joe, thanks again for your continued improvements and contributions. It's a great tool and really helps makes the place come alive. For the true amateurs like me it really helps interpretation too since I spend less time wondering which way is up.

Regarding the suggestion to expand the map, I think having even a lower-res larger area map behind the current one would be useful for identifying far-off features in horizon images. But since you've provided pointing and location data that's no longer needed to triangulate position.... and most of the time is spent looking down at the local area anyway.

What I'd find somewhat useful, and perhaps simpler to implement than reverse-map-lookup is a graphical interpretation of the field-of-view - i.e. a wedge around the arrow that's 5.1 deg wide for MR, 16.8 wide for ML etc. I know that depends on camera pitch, whether the whole image width is used, etc. But really it's all icing on the cake at this point.

Posted by: fredk Dec 17 2012, 03:35 PM

Let me add to the chorus - this is insanely cool.

About clicking on the map to get a list of images taken near there, that strikes me as not extremely useful, since we can simply read the corresponding sol(s) from a route map and view images from that sol(s). Maybe the idea is that this would automatically select the sol range if it's more than one sol? Also, as you point out, (as with MER) there are mid-drive navcam images that could be found this way.

But here's what would strike me as really useful: Click on two points on the map. The first one selects the rover position, and the second defines an azimuth. (So the 2nd defines only a unit vector, not a distance.) Then you could return a list of images from that rover location, looking (within some preset +/- azimuth range) towards the selected azimuth.

In principle this could be extended to image elevation as well, ie you could somehow specify an elevation and then the returned images point near that elevation (and previously specified azimuth). Though it's not clear how you'd specify that...

Posted by: mhoward Dec 17 2012, 03:54 PM

Joe: Just to make sure you're aware, in the engineering rawids characters 19-21 is the site counter and 22-25 is the drive counter. It works similar to MER; you've probably seen the relevant papers. You can determine which locations are significant by counting the number of images in whatever way you like. Occasionally you'll have non-driving locations that are close enough together you might want to combine them, but not often.

Posted by: jmknapp Dec 18 2012, 01:24 AM

Thanks for the tips and suggestions. No, I wasn't aware of the siteid/driveid thing.

Greenish's suggestions are good and have the virtue of being pretty simple to implement. The map now ahows an FOV wedge when the pointing info is available. The wedge extends out far irrespective of the elevation, so the actual range of an image needs to be taken into consideration/guesstimated based on the given elevation and terrain.

Also, I added a layer with a mosaic of the NW quadrant of Gale Crater from the MRO CTX camera. Note that it doesn't appear until you zoom out pretty far. It's slightly misregistered--I'll tweak that eventually.

Posted by: atomoid Dec 18 2012, 03:22 AM

That's absolutely amazing! i hope they pay you enough!
a glutton might whine for even more tasty morsels, so ill try:
filter by: bearing, elevation, SOL range,

regarding the "clicking on the map to get a list of images taken near there", id see the usefulness of clicking on the map itself rather to find images that have that point in their view, and filter as desired to reduce the hit list, especially if an elevation map could be used in the plot to eliminate images where the point is out of view behind a rise. would be nice to have an easy way to find images of various features from various angles, near or far.

Posted by: stewjack Dec 18 2012, 03:51 AM

This new fangled concept of being confident of the direction that you are looking out from the mast towards -
is going to take some getting used to! At least as long as there is no pointing information associated with the
Navcams I can claim that it was because I was looking at the Navcams, that I became disoriented. Otherwise -
I wouldn't have ANY excuse!

Posted by: jmknapp Dec 18 2012, 02:48 PM

QUOTE (stewjack @ Dec 17 2012, 10:51 PM)

Posted by: stewjack Dec 18 2012, 08:22 PM

QUOTE (jmknapp @ Dec 18 2012, 10:48 AM)

Posted by: maschnitz Dec 18 2012, 08:34 PM

Oh, Joe, doing your own site, for free - no apologies or excuses needed. Stuff happens. We're lucky to have your work.

And, great work lately. Keep it up!

Posted by: atomoid Dec 19 2012, 01:10 AM

just a tiny gotcha:
it seems the 'SOL' setting snaps back to 'all' whenever i select any of the MASTCAM NAVCAM CHEMCAM HAZCAM MARDI MAHLI boxes.
or maybe its just a quirk in Chrome, i haven't checked others..

you do realize Joe, that yourself and others are constructing a Mars geeks' paradise ?!

Posted by: eoincampbell Dec 19 2012, 01:12 AM

Thanks to Joe for the "go-to-site" for Curiosity images. Sitting-in through it's development here is awesome btw.......

Posted by: jmknapp Dec 19 2012, 06:36 PM

The reason the sol list snaps back to all is that only sols that have images, given the current selection of cameras, appear in the list. When you uncheck a camera it could very well happen that there aren't any images for the given sol. So in all cases it resets to all and recalculates the list of sols that have images. Granted, maybe it could be done a little more user-friendly and it annoys me sometimes too. Until I change that, the thing to do is make the camera selections first and then select sol, order, size, etc. at will.

As for getting a set of images in more or less the same direction and location as a given image, yes, that sounds very useful!--but tricky. It would be great for figuring out CHEMCAM targets. I hate to think of a clunky interface where you have to specify az/el and location bounds, database-query style. What I'm thinking is maybe a more immediate interface where you simply click on a pixel in a displayed image, the software figures out the azimuth, elevation and location of the camera at that pixel, and then gets the set of all images from the same location that contain that az/el vector. The latter step is actually not too hard in SPICE--you just rotate the vector from the topo frame to the camera frame and check whether it's in the field of view. Maybe the returned images could each have a marker at the given direction.

A tricky part is that many images are not full size. Some are just scaled down versions which wouldn't be a problem, but others are sub-framed and it's not clear what part of the frame they're taken from. Maybe in most cases the center of the sub frame corresponds to the center of the full frame? If some false positives and negatives are tolerated maybe it wouldn't be half bad.

Posted by: jekbradbury Dec 20 2012, 05:58 AM

With Navcam images, one thing you might be able to do is to overlay on the map (in a different color) clickable wedges for other Navcam images produced at the same location. If two panorama rows were taken at the same site at different elevations, only those wedges corresponding to images at the same elevation should probably be displayed.

Posted by: Airbag Dec 29 2012, 01:52 PM

jmknapp's curiositymsl.com is not working for me today - it goes to a generic RoadRunner "what are you looking for?" web site instead. Is anybody else having the same problem? Ludo's "MSL Curiosity - Raw Images Listing" has not shown any updates for a few days now. I may <gasp> have to go to directly the NASA site instead!

Airbag

Posted by: Tesheiner Dec 29 2012, 02:07 PM

Nope. It's working for me.

Posted by: jmknapp Dec 29 2012, 02:12 PM

QUOTE (Tesheiner @ Dec 29 2012, 09:07 AM)

Posted by: Airbag Dec 29 2012, 02:21 PM

Thanks!

Airbag

Posted by: Stellingwerff Dec 29 2012, 08:16 PM

QUOTE (Airbag @ Dec 29 2012, 02:52 PM)

Ludo's "MSL Curiosity - Raw Images Listing" has not shown any updates for a few days now. I may <gasp> have to go to directly the NASA site instead!

Posted by: jmknapp Dec 30 2012, 07:32 PM

Thinking about different ways to torture the SPICE camera-pointing data, here are a couple of charts showing the mast pointing during the course of one sol, sampled every second:

Posted by: mhoward Jan 7 2013, 08:03 PM

For Mac and (to a limited extent) iOS users, I'm making some alpha software available today. 'Alpha' means an early development version. Please note that this isn't finished (in fact it still has a long, long way to go), but I figure a few of the diehard enthusiasts here might want to know about it and follow along as it continues to evolve:

http://www.midnightplanets.com/alpha/

For everyone, my website now includes recent image updates, for Opportunity as well as Curiosity. (And since we have metadata for Opportunity, you can see observation descriptions when you click on an image.)

http://www.midnightplanets.com

- Mike

Posted by: phase4 Jan 7 2013, 09:04 PM

QUOTE (mhoward @ Jan 7 2013, 09:03 PM)

Posted by: eoincampbell Jan 8 2013, 01:21 AM

Great, MacMini loves it! Thanks.

Posted by: mhoward Jan 14 2013, 01:03 AM

I'm not usually going to post about updates, but some of you might enjoy http://twitter.com/PAL9000MP, which I think is working now. You can also get text message notifications sent to your cellphone if you set that up in Twitter.

Posted by: pospa Jan 14 2013, 03:59 PM

Dear Joe, I like your CuriosityMSL image browser a lot and would like to suggest a small improvement to yout consideration.
Could you add one more column "Total distance traveled" into the Drive Log page? This value would show cumulative distance traveled since landing for every sol.
Thx

Posted by: jmknapp Jan 14 2013, 08:23 PM

Good idea--Greenish was thinking that rover azimuth (before and after the drive) would be good, as well as elevation change. Another item of interest would be the rover tilt (roll, pitch). I'll add all those columns if it will all fit horizontally in a reasonable space.

Posted by: jmknapp Jan 16 2013, 11:09 AM

OK, I updated the rover http://curiositylog.com to include more data. Also, thanks to Ludo for getting me straightened out with MAHLI pointing, so now the MAHLI images have azimuth and elevation info.

Although it'd be nice to be able to take a CHEMCAM or MAHLI image and automatically identify the location in a MASTCAM or NAVCAM image, turns out there's a problem, which I think is due to parallax between the camera locations. That is, if you take, say, a CHEMCAM pointing direction and try to find a MASTCAM image that contains that direction, things don't quite line up. It gets you in the ballpark, but you have to visually search to find the corresponding spot. Parallax effect? That would be especially pronounced for MAHLI, at the end of the robotic arm. I think that a 3D model of the scene is required to really do this automatically.

Posted by: elakdawalla Jan 16 2013, 04:34 PM

Earlier in the mission, ChemCam images were almost always taken with a left Navcam for context -- but I guess once they got confident in ChemCam pointing they stopped doing that. I'm a little surprised that left Navcam pointing is different enough that geometry doesn't match between left Navcam and ChemCam -- but I'm not too good at imagining spherical coordinates, so it's also not surprising that I'm surprised!

Posted by: elakdawalla Jan 16 2013, 04:36 PM

I just want to make a more general comment on this thread that I love the proliferation of different ways to display these data. I have no favorites -- I find jmknapp's useful for some circumstances, mhoward's for others, stellingwerff's for others. The addition of Twitter feeds of drive data and image download data is super. Keep the great work coming

Posted by: jmknapp Jan 16 2013, 05:09 PM

Here's an example--using the "http://curiositymsl.com/gallery/cc157.html" CHEMCAM image and correlating it with a left NAVCAM image:

Posted by: mhoward Jan 16 2013, 05:12 PM

I'm seeing basically the same thing at the moment. Was just about to look into what's going on.

Posted by: Gerald Jan 16 2013, 07:08 PM

QUOTE (jmknapp @ Jan 16 2013, 12:09 PM)

Posted by: fredk Jan 16 2013, 07:36 PM

The size and direction of the offset in your navcam-chemcam example, Joe, are crudely what you'd expect from parallax. For a 1.9 metre navcam height and target elevation of about -33 degrees, the target is about 3.5 metres from the cameras. A L-R navcam separation of 42.4 cm gives a NLA-chemcam separation of about 16 cm. Trig gives an expected parallax of 2.2 degrees. Your green and red circles are separated by about 3.0 degrees.

Not only is the separation not perfect (though distance uncertainties will contribute to that), but more seriously the offset direction is not good. Chemcam is not vertically above NLA, it is at an "elevation" of roughly 45 degrees, towards the centre of the mast. So the chemcam target bearing should be to the lower left of the navcam bearing, regardless of target position. But your red circle is directly below your green.

Can you repeat this with more chemcam/navcam pairs? Is the chemcam bearing consistently offset, in angular distance and more importantly direction, from the navcam bearing, or is there some random scatter? That should be a clue...

Posted by: mcaplinger Jan 16 2013, 07:55 PM

QUOTE (jmknapp @ Jan 16 2013, 04:09 AM)

Posted by: jmknapp Jan 17 2013, 01:23 AM

Wow, some interesting ideas there, particularly that the position of MAHLI is often very close to the object, so it's a 3D probe... that's one point in space anyway. As for being repeatable, there may be an issue there--never discount the possibility of programming errors or SPICE inaccuracies, not necessarily in that order. Here's another example that might indicate something else going on:

Take this CHEMCAM shot of a calibration target: http://curiositymsl.com/imgpoint.php?name=CR0_411084181EDR_F0051954CCAM02153M_

Here's the result of a http://curiositymsl.com/imgpoint.php?name=NLA_411431900EDR_F0051954NCAM07504M_ on a NAVCAM image:

Posted by: pospa Jan 17 2013, 12:09 PM

QUOTE (jmknapp @ Jan 16 2013, 12:09 PM)

Posted by: Gerald Jan 17 2013, 12:19 PM

If you are ready for some math, the focus-stacked MAHLI case most likely becomes feasible in a determinstic and reproducible way:
For this I have to assume, that SPICE data are reliable; I'm optimistic, that this assumption mostly holds, because otherwise the MAHLI close-ups weren't possible.

To understand the geometry of the imaging sufficiently for a match of two images, we have to think at least 7-dimensional (degrees of freedom): Three dimensions (x,y,z) for the position of the camera relative to a frame (coordinate system), three rotational dimensions (heading, pitch, yaw), and at least one dimension for the aperture.
To understand the relative positions and rotations in space, the first six are needed. These six dimensions describe the transformation from one frame to a second. The inverted transformation describes the transformation from the second to the first frame.
In the case of two cameras we have to look at three frames (in the most simple case): Frame of camera 1, frame of camera 2, and frame of e.g. MSL.
To transform frame cam 1 to frame cam 2, we have to take the inverted transformation MSL->cam1 and multiply (matrix multiplication provided by software library) to this the transformation MSL->cam2.

Now take MAHLI as cam1. From focus stacking we (hopefully) know the focal length focus distance for each pixel. The focal length focus distance in many cases will be very narrow at the distance of the 3D-point of the object from the MAHLI frame origin. By the pixel position within the MAHLI image and the MAHLI aperture, we get the missing two dimensions of the 3D-point within the MAHLI frame.
Now apply the MAHLI->cam2 transformation to the 3D-point, and you get the 3D-position within the cam2 frame. By the cam2 aperture you may project the 3D-position into the image plane of cam2 and get the pixel pos (Check for division by zero). Now range check.

Doing this for the four corner points of the MAHLI image you get the endpoints of a morph from MAHLI to cam2.

Hopefully the discrepancy between the red and the green circle will almost vanish with an accurate calculation based on all degrees of freedom. Probably the parallaxe is just the distance between the (z?-)axes of the two camera frames, if pointings are identical; also check apertures.

Posted by: jmknapp Jan 18 2013, 01:41 AM

I think I follow that--taking aperture to be field of view? So would the MAHLI distance be derived from the height map that they make from multiple images/focal planes?

Posted by: Gerald Jan 18 2013, 11:19 AM

That's exactly what I meant.
I'm looking forward to your results. Thanks for investing your time for doing that! I think, we all will enjoy your work very much.

Posted by: mcaplinger Jan 18 2013, 03:13 PM

QUOTE (jmknapp @ Jan 17 2013, 06:41 PM)

Posted by: jmknapp Jan 18 2013, 10:21 PM

QUOTE (mcaplinger @ Jan 18 2013, 10:13 AM)

Posted by: Gerald Jan 19 2013, 12:50 AM

Thanks for this info! So we have to look for a way to do a depth calibration. This won't be quite trivial.
There might be a way to reconstruct/guess the focussing heuristics; e.g. a hyperbolic interpolation of the type D = C + A / N, for N = N_min..N_max, and some constant distances A and C.
If the heuristics cannot be retrieved in a straightforward way, we'll need a somewhat fuzzy optical technique.
One idea for that might be the simulation of the optical blur of single (lo-res) images from the sharp focus-stacked image. Via difference image and a chi-square method (minimizing the sum of the squares of the difference pixel values of the examined region) and constraints known of the focussing heuristics we could try to find out the most likely focus distances.
That's just a first idea and has to be elaborated in more detail, if necessary.

Posted by: mcaplinger Jan 19 2013, 02:49 AM

QUOTE (Gerald @ Jan 18 2013, 05:50 PM)

Posted by: Gerald Jan 19 2013, 12:36 PM

Thanks, that's perfect! All infos we need.
And thanks for your hard and excellent work!

EDIT: For technical data see http://www.researchgate.net/publication/221673906_Curiositys_Mars_Hand_Lens_Imager_(MAHLI)_Investigation, especially section 6.5 on p. 25f, and Fig. 27 on p. 102 of the submitted version.

Posted by: Greenish Jan 19 2013, 12:37 PM

Won't be too long now, according to http://pds-imaging.jpl.nasa.gov/ , "The first MSL release for EDRs is expected on February 27, 2013; the first release for RDRs is expected on March, 20, 2013."

Posted by: Deimos Jan 19 2013, 07:47 PM

QUOTE (jmknapp @ Jan 17 2013, 02:23 AM)

Posted by: jmknapp Jan 20 2013, 02:33 AM

Good to know! Thanks, deimos.

Posted by: Gerald Apr 5 2013, 02:16 PM

Explanations to http://mars.jpl.nasa.gov/msl/multimedia/raw/?rawid=0230MH0246001001C0_DXXX contain the following data points of a motor count - cm-distance - map for MAHLI with opened dust cover:

Posted by: jmknapp Apr 5 2013, 02:56 PM

Wow, that was a geeky-informative note they added to that image. Which motor does the count refer to?

Posted by: mcaplinger Apr 5 2013, 03:07 PM

QUOTE (jmknapp @ Apr 5 2013, 07:56 AM)

Posted by: Gerald Apr 5 2013, 03:15 PM

It refers to the "focus motor count position" of the MAHLI camera. The distance may be useful for a 3d localization of objects we thought about earlier.
There are also provided micrometers per pixel, but less accurate. I'll see, whether I can retrieve an acceptable formula, that interpolates this second value. It may be interesting for a scaling of the MAHLI images.

All we need is the appropriate motor count. Joe, do you know how to access to it?

Posted by: Gerald Apr 5 2013, 05:20 PM

Slightly improved hyperbola formula for MAHLI distance, and formula for microns per pixel, based on http://mars.jpl.nasa.gov/msl/multimedia/raw/?rawid=0230MH0246001001C0_DXXX:

Posted by: jmknapp Apr 5 2013, 07:37 PM

Like most metadata not in the PDS, I wouldn't know how to get the MAHLI information, but maybe it will be included in the MAHLI PDS release scheduled for June 6 ("release 2").

Posted by: Gerald Apr 5 2013, 11:59 PM

This is a link to an arbitrary MAHLI image: http://mars.jpl.nasa.gov/msl/multimedia/raw/?rawid=0129MH0154002000E1_DXXX&s=129.
The adress is composed of the fixed string "http://mars.jpl.nasa.gov/msl/multimedia/raw/?rawid=", the name of the jpg-file without extension, here "0129MH0154002000E1_DXXX", the constant string "&s=" and the sol number, here "129".
The referenced website containes the constant text "When this image was obtained, the focus motor count position was " plus the motor count (here "15148") plus ".". There are no html tags in that part of the html source.
By scanning (searching) for the fixed text, then reading the number until the "." it should be possible to read out the motor count.
What do you think?

Posted by: jmknapp Apr 6 2013, 12:24 AM

Huh, didn't know they had that for every MAHLI image--seems like that info could easily be scraped and converted to cm as you say. I may try that for my website. Just did a quick check & there have been 4224 MAHLI images to date, so that's a fair bit of scrapage.