Perseverance Imagery, technical discussion of processing, cameras, etc. Options

[Greenish]

Putting this here, for reference as the payload of the JSON feed link for Perseverence raws (see source in other thread here)
https://mars.nasa.gov/rss/api/?feed=raw_ima...;&extended=
Looks like there's a ton of good data in addition to just the (PNG! Bayer color separated!) images

This is for the first image shown on the page at the moment.

CODE

"images":
    [
        {
        "extended":
            {
            "mastAz":"UNK",
            "mastEl":"UNK",
            "sclk":"667129493.453",
            "scaleFactor":"4",
            "xyz":"(0.0,0.0,0.0)",
            "subframeRect":"(1,1,1280,960)",
            "dimension":"(1280,960)"
            },
        "sol":2,
        "attitude":"(0.415617,-0.00408664,-0.00947025,0.909481)",
        "image_files":
            {
            "medium":"https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00002/ids/edr/browse/rcam/RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01_800.jpg",
            "small":"https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00002/ids/edr/browse/rcam/RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01_320.jpg",
            "full_res":"https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00002/ids/edr/browse/rcam/RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01.png",
            "large":"https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00002/ids/edr/browse/rcam/RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01_1200.jpg"
            },
        "imageid":"RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01",
        "camera":
            {
            "filter_name":"UNK",
            "camera_vector":"(-0.7838279435884001,0.600143487448691,0.15950407306054173)",
            "camera_model_component_list":"2.0;0.0;(46.176,2.97867,720.521);(-0.701049,0.00940617,0.713051);(8.39e-06,0.0168764,-0.00743155);(-0.00878744,-0.00869157,-0.00676256);(-1.05782,-0.466472,-0.724517);(-0.702572,0.0113481,0.711523);(-448.981,-528.002,453.359)",
            "camera_position":"(-1.05782,-0.466472,-0.724517)",
            "instrument":"REAR_HAZCAM_RIGHT",
            "camera_model_type":"CAHVORE"
            },
        "caption":"NASA's Mars Perseverance rover acquired this image of the area in back of it using its onboard Rear Right Hazard Avoidance Camera. \n\n This image was acquired on Feb. 21, 2021 (Sol 2) at the local mean solar time of 15:37:11.",
        "sample_type":"Full",
        "date_taken_mars":"Sol-00002M15:37:11.985",
        "credit":"NASA/JPL-Caltech",
        "date_taken_utc":"2021-02-21T02:16:26Z",
        "json_link":"https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020&feedtype=json&id=RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01",
        "link":"https://mars.nasa.gov/mars2020/multimedia/raw-images/?id=RRB_0002_0667129492_604ECM_N0010052AUT_04096_00_2I3J01",
        "drive":"52",
        "title":"Mars Perseverance Sol 2: Rear Right Hazard Avoidance Camera (Hazcam)",
        "site":1,
        "date_received":"2021-02-21T23:12:58Z"
        },

(with syntax color):


Here's hoping that one of you skilled characters can make good use...

[phase4]

A small Perseverance update for Marslife, implementing the camera pointing data is a bit troublesome.
It suffers from a hit-and-miss accuracy.

[Greenish]

So cool!

Maybe they're still refining which way is up... lots of local coordinate frames to sort out and probably need some time (and the sun shots it looks like they just took, and some 3d processing of the local area, and some radio ranging) to get absolute position & orientation info.

The instrument position details will certainly be different vs MSL. And in my skim of the data it looked to me like they re-jiggered some of the pointing data formats, did you see that too? Not sure which parts you're using.

[djellison]

For the less technically minded among us...Ryan Kinnet has put up a page that grabs a listing with links to the PNG files that you can then use any browser-plugin-batch-downloader with

https://twitter.com/rover_18/status/1364309922167488512

I tried Firefox with 'DownloadThemAll' and it worked perfectly.

Meanwhile THIS GUY has python code to also grab the data
https://twitter.com/kevinmgill/status/1364311336000258048

[ugordan]

Speaking of raw images, are these weird colorations due to something wrong with the pipeline or the cameras themselves (I'm thinking and hoping it's the former)?
Examples attached, one is a NavCam-L from yesterday (seems to have been pulled since), the other an RDC frame.

Attached thumbnail(s)

 

[mcaplinger]

Speaking of raw images, are these weird colorations due to something wrong with the pipeline...

Yes.

[ugordan]

Yes.

That's reassuring to hear. One could sort of expect this from (well, almost) off-the shelf commercial EDL cameras, but the navcams are on a whole other level.

New raw images just trickling in.

Yep, looks like they started download on the full quality, Bayered RDC camera views.

On a side note, I found it interesting how the color balance between RDC and DDC was so different. Yes, different detector resolutions, but same vendor.

[phase4]

Yes, I hope this erronous positioning will improve when the rover is calibrated properly. Otherwise it is my code to blame...
I use the rover attitude, mastAz and mastEl values for camerapointing, I don't know if that has changed much since Curiosity.

[nprev]

Hey, all. This thread is for the imagewizards among us & will focus on the abundance of data Mars 2020 will provide. Please post your products, methods, and tips, and use this thread to share information & likewise learn from others.

[bdshort]

Okay, I hope this is the right forum for this question - one of the things I really like about both rovers having the ability to take stereo imagery is the ability to see stuff in 3D. Is there a utility out there that lets us view images in 3D using a VR headset? I've briefly searched around but haven't seen much. Anaglyphs are cool and everything but it leaves a bit to be desired.

[Cargo Cult]

Okay, I hope this is the right forum for this question - one of the things I really like about both rovers having the ability to take stereo imagery is the ability to see stuff in 3D. Is there a utility out there that lets us view images in 3D using a VR headset? I've briefly searched around but haven't seen much. Anaglyphs are cool and everything but it leaves a bit to be desired.

Stereo imagery viewed in a VR headset is a bit underwhelming - reconstructing geometry using photogrammetry to create a fully 3D representation of a landscape is much more interesting.

I did this with some Curiosity imagery a few years ago, with fascinating results. If you have a SteamVR capable VR headset, you can have a look here: https://steamcommunity.com/sharedfiles/file...s/?id=928142301 - I typed up some fairly detailed notes in the description which will broadly apply to Perseverance imagery.

(Full disclosure: I work for Valve, creators of SteamVR. The Mars stuff was a fun personal project which turned into something a bit larger...)

Getting right back on topic for this thread - some notes on photogrammetry involving Perseverance imagery!

Some decent camera parameters* to start with in Agisoft Metashape (formerly PhotoScan):

Navcam
Camera type: Fisheye
Pixel size (mm): 0.0255 x 0.0255 (for 1280x960 images)
Focal length (mm) 19.1

Hazcam
Camera type: Fisheye
Pixel size (mm): 0.0255 x 0.0255 (for 1280x960 images)
Focal length (mm) 14

Using separate calibration profiles for left and right cameras may make sense - stuff worked better for Curiosity's navcams when I did this. (They're beautifully hand-made one-off scientific instruments, after all.) Metashape will further refine camera parameters once given these reasonable starting points.

It's all looking like really exciting data to play around with - navcam imagery in high-resolution, full colour after a bit of processing. I'm starting to figure out debayering stuff - I'm sure that this thread will come in great use!



* derived from The Mars 2020 Engineering Cameras and Microphone on the Perseverance Rover: A Next-Generation Imaging System for Mars Exploration
: Table 2, Perseverance Navcam, Hazcam, and Cachecam characteristics

[bdshort]

Great! Thanks for the answer. I have an Index, so I'll check that out!

Edit: Just tried it - that's awesome! Gives a great sense of scale to the rover, and I liked all the point of interest spots. I'm surprised there isn't more stuff out there like this, it would be a fantastic educational tool and it's just fun to stand or sit around in for awhile.

Stereo imagery viewed in a VR headset is a bit underwhelming - reconstructing geometry using photogrammetry to create a fully 3D representation of a landscape is much more interesting.

I did this with some Curiosity imagery a few years ago, with fascinating results. If you have a SteamVR capable VR headset, you can have a look here: https://steamcommunity.com/sharedfiles/file...s/?id=928142301 - I typed up some fairly detailed notes in the description which will broadly apply to Perseverance imagery.

(Full disclosure: I work for Valve, creators of SteamVR. The Mars stuff was a fun personal project which turned into something a bit larger...)

Getting right back on topic for this thread - some notes on photogrammetry involving Perseverance imagery!

Some decent camera parameters* to start with in Agisoft Metashape (formerly PhotoScan):

Navcam
Camera type: Fisheye
Pixel size (mm): 0.0255 x 0.0255 (for 1280x960 images)
Focal length (mm) 19.1

Hazcam
Camera type: Fisheye
Pixel size (mm): 0.0255 x 0.0255 (for 1280x960 images)
Focal length (mm) 14

Using separate calibration profiles for left and right cameras may make sense - stuff worked better for Curiosity's navcams when I did this. (They're beautifully hand-made one-off scientific instruments, after all.) Metashape will further refine camera parameters once given these reasonable starting points.

It's all looking like really exciting data to play around with - navcam imagery in high-resolution, full colour after a bit of processing. I'm starting to figure out debayering stuff - I'm sure that this thread will come in great use!



* derived from The Mars 2020 Engineering Cameras and Microphone on the Perseverance Rover: A Next-Generation Imaging System for Mars Exploration
: Table 2, Perseverance Navcam, Hazcam, and Cachecam characteristics

[scalbers]

There are what i assume to be quaternions (?) regarding attitude in the search/query api. see https://mars.nasa.gov/rss/api/?feed=raw_ima...04096_034085J01 for example. There are altazimuth compass bearings and elevation for the regular people as well, which could be used to project the images to a compass ball according to the metadata, rather than manually overlapping and tie-pointing the images. I do not know of software or libraries that can help with this, if you do please tell.

Regarding the API "quaternion" coordinate info with the raw images, a LINUX program 'jq' is good for manipulating these. While this is just a starting point it would be interesting to see how an automated program might work with putting together a mosaic. I would just need some time to hook up some of my Fortran code with some scripts.

CargoCult - can your SteamVR creation be viewed somehow with a regular computer? With Curiosity, sittingduck (YouTube link below) had some really nice videos moving through a 3D landscape that might be interesting to see in VR. Sittingduck back in 2016 had used a Blender Plug-In obtained from phase4.

https://www.youtube.com/watch?v=7zW9yISB01Y...eature=youtu.be

[fredk]

About image pointing, the algorithm I use for MSL, based on this post, seems to work fine for Percy, with the "rover_attitude" quaternion field replaced with "attitude". So for the "shiny rock" image:
https://mars.nasa.gov/mars2020-raw-images/p...6_034085J01.png
I get elevation,azimuth = 1.2, 251.5 degrees, which looks about right. That presumably is for the centre of the FOV, which happens to correspond almost exactly with the shiny rock.

[djellison]

It's not great - but this is my agisoft metashape results with the Navcam images so far
https://sketchfab.com/3d-models/m2020-landi...7215aa7db2fb0c8