What to do after the closure of Midnight Mars Browser? Options

[nogal]

Those thumbnails are not part of the JSON feeds

Another challenge! By the way, I couldn't reproduce the bug you've mentioned. It worked for all the days you pointed out and a few extra on either end of the interval. Perhaps it is related to the slowness of the loading? I'll make it more obvious in the tool when it is loading. Thanks
Fernando

[fredk]

I've been parsing the MSL json files into html and thought I'd make the pages public in case anyone finds them useful. My pages are static html so there's no selecting cameras, changing sort orders, autoload scrolling, etc. It's inspired more by the linear style of the old curiosityrover.com than MMB, Fernando's page, or www.mars-browser.co.uk/curiosity.

The info and pics are straight from the json files and the mars.nasa.gov/msl-raw-images server with two exceptions. First I calculate the pointing according to mcaplinger's suggestion in this post. I haven't tested this much so let me know if you notice any problems.

The second is that I deBayer the Bayered mastcam frames automatically as they arrive. Due to the chroma subsampling I do you shouldn't take the colours too literally, and there are lots of crosshatch artifacts, so these images are only meant to give a rough sense of the scene.

On my site there's the "latest images" page located here: http://lcdm.ca/msl/. This checks for updates every half hour and adds any new images to the top, chronologically by time taken. After updates I use this by scrolling and paging down until I see image links change to the visited colour, which makes it very easy to keep up and not miss any new images, even if they're from old sols.

There are also individual sol pages, eg: http://lcdm.ca/msl/2924/ (going back to sol 2714). These are always ordered by time taken.

More details on this page.

If anyone has any suggestions hopefully I'll have time to implement them (within the limitations of static html), so let me know.

[PaulH51]

Viewed on my Mobile phone and impressed Looking forward to seeing it in action on my home PC when my covid travel restriction order is supposed to be lifted in 11 days 😊

[fredk]

Thanks, Paul. Sorry my site's not small-screen friendly. Maybe now that we're moving again on Mars that'll relieve your earthbound status a bit?

[nogal]

... thought I'd make the pages public in case anyone finds them useful.

Thank you for sharing this. Fast, straight and simple! Definitely useful. Cheers
Fernando

[Ant103]

This.

Is.

AWESOME !

It's fast, it's clear, filenames are immediately readable, and it's easy to navigate. And you even debayer the pictures o_o

[Phil Stooke]

Yes, I really like this.

Phil

[fredk]

Thanks, guys. I hope to improve the deBayering still - treat the edges better and maybe remove the glitches that appear in each frame.

[nogal]

Here is an improved (I hope!) version of my MSL Image Viewer. Still work in progress so, for the time being, I'll continue to distribute it here. Having been on vacation last week I only had time to implement part of your suggestions - thank you all for those.

To install, download the zip file to your favourite location and unzip it. You should see the following 5 files:

• MSLImageViewer.html - the main page and the only one you should call;
• MSLImageDisplay.html - called from the main page to display a single image in large format, with associated information ;
• MSLImageUrlList.html - called from the main page to display a list of URLs of the currently selected (visible) images ;
• MSLImageViewer.css - Cascading Style Sheets for the project ;
• MSLImageViewer.js - JavaScript code for the project.

I made a general review of the code to make it more robust and changed the layout a bit (it is stable now).

Opening the MSLImageDisplay.html page automatically loads the images (in small format) of the most recent sol. Below the header, on the left, there is a bar with several sections: a sol navigator, sol statistics, image filters, and option selectors. To its right lies the images area.

The sol navigator has:

• two buttons to move to the previous or the next sol that has images - sols with zero images are skipped over. "First sol" and "Last sol" stoppers are dynamically generated. Buttons are accordingly disabled.

• an input field to directly go to any sol - if the sol does not exist or has no images a message is issued and the sol does not change.

Below the navigator there is a panel with a few statistics for the sol.

Next comes the image filters panel. Those are grouped into two categories: cameras (mastcam, chemcam, hazard, etc.) and image types (full, subframe, etc.). Only the entries that exist on the sol are shown. It is now possible to select one or more entries in each category. Entries in the same category are ORed and the two categories and ANDed. For instance, select all Front Haz Right and Left cams but only for full frame images. "Show all" and "Hide all" buttons per category provide shortcuts.

These filters are "sticky". When you move from sol to sol only the categories that were previously selected will be shown - until you change them. But note that all the images for the sol will still be loaded. Having selected (or deselected) a category that does exist on the new sol does not cause an error.

Entries are selected by toggle buttons. A green or red rim indicates whether the filter is active or not.
The option selectors panel currently contains just one button. The "List URLs (of visible images)" button opens a new page with the list of full URLs of the images that are currently visible. The page has a "Copy to the clipboard" button.
The page's right side contains the sol's images. Place the mouse over an image to reveal the image's full URL.

Clicking on an image opens it in a new page and shows the associated information. Images are shown at their natural size, if they fit the screen, otherwise they will be reduced to fit. From this page you can load the previous or the next image. Any number of such pages can be simultaneously open, allowing, for instance, to compare the images.

In addition to the suggestions I did not have the time to implement I have other ideas. For instance, highlighting the recent images.
But what is recent? 24h? 48h? 1 week? Please tell me your opinion. It is very easy to highlight recently added images for a specific sol. But finding all recently added images implies going through the json files for all sols. This takes time and uses network and should perhaps be implemented as an on-demand process (this is the drawback of a client-side, versus server-side, implementation).

Your feedback is requested and most welcome. Sorry for the long post.
Fernando
EDIT: the attachment has been removed. A new version, supporting more functions, is available. See post #56

[fredk]

I've improved the deBayering on my site http://lcdm.ca/msl/. I now handle the edges of the frame better, so colours are good right to the edges. Also I use a simple method to remove the "hot column" in the ML images. This is a bright column (column 1070, a green-blue column, in the full-resolution frame) that runs down most of the ML images. I simply replace that column of pixels with the average of the neighbouring green-blue columns before deBayering. Due to the jpegging in the public Bayered images, the removal isn't perfect and you may notice some residual "scarring" within an 8-pixel jpeg block of the hot column. Both these changes apply to new images, not old ones still on my server.

I've also added lists of deBayered images at the top of each individual sol page. These can be used to download all at once, with a browser plugin or, for example, with

CODE

wget -Fi http://lcdm.ca/msl/2935/ML.html

if you can use wget (or some similar command with curl). Again, these lists only include new deBayered images.

[nogal]

I am releasing a few additional functions for the MSL Image Browser. In a separate post I'll discuss some issues which have influenced some decisions and prevented implementing some of the suggestions - I'm very grateful for those.

Recent images are now highlighted by a red border. Right now - and this is very easy to change - "recent" means the image has been added in the last 72 hours. I had to choose a criteria. It could have been "added since my last visit" but this implies storing the date and time of "my last visit" in a special area of the browser (using a standard - and safe - HTML 5 API). Please let me know if you're interested in having this option. A row of buttons for each sol with "recent" images (according to the above criteria) appears just below the page banner. Click them to go the the respective sol.

On the main page three new buttons (on the left panel) allow invoking three different views of a sol's image set:

• Grid view - just the images (reduced in size, if necessary), side by side (the layout already present in previous releases)
• List view - one image per row (reduced in size, if necessary) with a few basic data points
• Recent - only the sol's recent images in a grid layout

The Grid and List views are influenced by the filter selection buttons whereas the Recent view is not and always shows the recent images, regardless of the instrument or image type.
To speed-up the download it is now done in groups of up to 250 images each. This number was chosen because, right now, 54.3% of all sols have 250 images or less, 86.9% have 500 images or less, 95.6% have 750 or less, and 96.2% have 1000 images or less. Thus, the vast majority of sols requires very few clicks. The use of -thm thumbnails is discussed in another post. While there are more images to download a "Show more" button appears at the bottom of the images set.

I also added a "Go to Top" button at the bottom of the image set, for ease of use.

One consequence is that if one is looking for all the images of a certain category then all the sol's images must be downloaded. The list and grid views are independently loaded and scrolled; it is possible to have downloaded all images in one view but not on the other but, since the images are downloaded, showing them in a view is very fast. There is a new counter in the sol's statistics showing how many images have been downloaded (which changes with the view, naturally).

No changes were done to the URL list page.

The image view page now has a "magnifier" capability. This is intended to help better see details of the bigger images. Click the button (it is a toggle) and a small lens is positioned on the upper left corner of the image. The enlarged contents of the lens are shown on an overlay at the page top left (below the banner). The enlarged area is a 6x6 of the lens. Please let me know if you are interested in other settings or being able to set it as required.
Note: this does not increase the image's resolution, just enlarges the image, possibly blurring it. As such, the magnifier is not made available for small images.

You can download the image, in the original size, by right-clicking on it (the same is valid for the main page views).
Thank you for using the tool. Please let me know your thoughts, any requests, or bugs you may find.

Fernando

 MSL_Image_Viewer_2020NOV11.zip ( 17.11K ) Number of downloads: 135

[nogal]

What some people are beginning to figure out is how useful the Sequence name is.... ( in a file like NRB_655824982EDR_S0822188NCAM00595M_.JPG the SeqID is the NCAM00595 part ) - being able to filter by SeqID for a given Sol would also be really good

Doug,
I want to implement this next. But I found the json manifest also points to files with a different file name structure, for instance
... /msss/02938/mhli/2938MH0007060001004000I01_DXXX.jpg
I found a document describing the structure of the file name in your example. Where can I find a description of these file name's structure (some parts are obvious)? Do image sequences also appear on these files?

Any information is much appreciated.

Fernando

[nogal]

On the other hand, I believe all images hosted at mars.nasa.gov/msl-raw-images are paired with thumbs generated on the ground - for that navcam the thumb is https://mars.nasa.gov/msl-raw-images/proj/m...00595M_-thm.jpg. It's these latter thumbs that should be used to speed loading of sites like this.

I found that not all images in the json manifests have a -thm counter part (sol 0 gives thousands of errors, for instance). Because there is no way to find whether a file exists on the server without trying to download it, one is then left with the need to recover from the 403 error (when the -thm file does not exist) by loading the full size file... which may also not exist, which needs another recovery routine. This is more overhead.

This is, in essence, why I opted for downloading images in relatively small groups.

I also found the json manifests do not list the exact same files the official site shows. For instance, for sol 0 (my nemesis!) json lists 3270 images - of which fewer than 1000 exist on the given url - but the official site mentions 2198.

Any information on this subject is very much appreciated.

Fernando

[fredk]

Where can I find a description of these file name's structure (some parts are obvious)? Do image sequences also appear on these files?

I think MAHLI follows the file naming convention for mastcam, which is described in this paper.

[fredk]

I found that not all images in the json manifests have a -thm counter part (sol 0 gives thousands of errors, for instance).

I see what you mean. I only go back a couple of hundred sols and the -thm images always exist in these recent sols. But I can imagine that things are more sporadic in the older sols.

About image counts, I find agreement between the public jpeg site and json, at least back a couple hundred sols. For sol 0, json v1 gives 3720 images, and the public jpeg site lists 2198 full-res images + 1520 thumbs for a total of 3718. That's very close, but the jpeg site seems to be missing two images!