MSL Images & Cameras, technical discussions of images, image processing and cameras Options

[CosmicRocker]

I'm still trying to figure out a number of things about the new images we are trying to work with. Assuming others are likewise trying to learn, I thought I would open this thread to create a place for such discussions.

I'd like to start out with a comment about raw image contrast. There have been several postings in the main threads about whether or not the MSL raw images have been stretched like those from the MER missions. I am certainly no expert on this, but it looks to me as if the MSL images have not been stretched at all. I haven't tried to analyze all of the image types, but the hazcams and navcams have pixel brightness histograms that are very different from their MER counterparts.

This attached image compares MER and MSL navcams along with their luminosity histograms.


The MSL images clearly are not using the entire, available range of brightness values, whereas the MER raws do. For this reason, the MSL raw images can usually be nicely enhanced by simply stretching the distribution of brightness across the full 256 value range.

[Herobrine]

If you're happy I'm happy, but it seems like an awful lot of work for something with not a lot of practical utility to me.

I'm mostly interested in it for the prospect of removing some of the effects of the lighting conditions that existed when the image was acquired. I'm doing work with generating a color, 3-D environment from mountains of MSL data. I'd like to be able to light the scene myself so I can change the time of day, rather than being stuck with the lighting conditions that existed at the time an image was acquired, conditions that will not be consistent when combining data from different times.

And there are all kinds of secondary effects from the brightness distribution of the sky, and in the near field, light reflecting off the rover, that I think you're ignoring. Though I'm not sure how large those effects are.

There are a lot of factors I'm ignoring. I'm guessing (read "hoping") that their effects are smaller than the error introduced by the less-than-ideal spatial data so I'm okay with ignoring them for now. For me, close is better than nowhere.

I have to confess that I glazed over a bit while reading the discussion of what happens next in, e.g., http://arxiv.org/abs/1403.4234 -- and a lot of the described MER processing is unneeded for MSL, or at least different.

I don't know if I've missed that paper until now or if I came across it and glazed over as well and moved on, hoping for one specific to MSL. Either way, thank you for linking it; it does look like it contains the information I need.

Sorry, I haven't followed your procedure in complete detail, but if you've assumed the same mean albedo for the two frames to begin with, then the fact that the brightnesses of the two final frames in your post agree so well doesn't sound surprizing.

I only use the "local mean albedo" when calculating the irradiance contribution reflected by the surrounding terrain. This affects how much light my calculations think vertical/slanted surfaces received. Increasing that value has the effect of making those vertical/slanted surfaces darker in my final image.

Interesting to see this type of analysis. One possible refinement to the consideration of "albedo" is to express it as a reflectance. The so-called Bidirectional Reflectance Distribution Function (BRDF), or Anisotropic Reflectance Factor (ARF) is a way to see how the reflectance varies depends on the view angle and where this is compared to the sun.

One of the things I was hoping I might eventually be able to do with the output of this process was actually to try to generate a rough approximation of the BRDF of different points on the surface, if I had enough imagery containing that spot at different times of day. If I could come up with something even remotely close, I think it would enhance the realism of a realtime 3-D rendered scene if I used BRDFs generated by crunching data from multiple observations (rocks would look more rocky, dust more dusty, etc.). I don't know if I'll ever get to that point, though. I tend to work slowly.


I might as well just explain exactly what I did to make those images.
What I've written isn't exactly in a sharable, easily reusable state, but here's the basic process that produces that result. I'll use degrees for angles here.

CODE

First:
Define a constant for solar irradiance at 1 AU: { s_1AU = 1371 (W * m^-2) }
Define a unit vector for the local zenith in the site frame: { ZENITH = (0, 0, -1) }
Define a function (called "f()" below) of optical depth and zenith angle, that yields a normalized net irradiance factor. For my purposes, I wrote a 2-dimensional lookup table and a function that interpolates the values. My lookup table can be found at http://mc.herobrinesarmy.com/msl/normalized_net_irradiance_factor.csv. The first row lists solar zenith angle in degrees; the first column lists optical depths.

Then, per frame:
Define a local mean albedo (used to calculate the ground-reflected contribution to irradiance): { A = 0.2 }
Define the optical depth: { OD = 0.5 }
Read in MXY data (use this to exclude pixels containing rover structure)
Read in UVW data (called "normals" below).
Read in RAD data (called "values" below).
Read in metadata from LBL as follows: {
rover_quaternion = LBL>ROVER_COORD_SYSTEM_PARMS>ORIGIN_ROTATION_QUATERNION
solar_elevation = LBL>SITE_DERIVED_GEOMETRY_PARMS>SOLAR_ELEVATION
solar_azimuth = LBL>SITE_DERIVED_GEOMETRY_PARMS>SOLAR_AZIMUTH
radiance_offset = LBL>DERIVED_IMAGE_PARMS>MSL:RADIANCE_OFFSET
radiance_scaling_factor = LBL>DERIVED_IMAGE_PARMS>MSL:RADIANCE_SCALING_FACTOR
start_time = LBL>START_TIME
}
Calculate Julian date for start_time (too complicated to include here)
Calculate heliocentric distance for Mars in AU (called "r" below) using Julian date (too complicated to include here)
Calculate solar irradiance incident at top of Martian atmosphere: { s_toa = S_1AU / (r^2) }
Create unit vector toward Sun in site frame: {
sun_x = cos(-solar_elevation) * sin(solar_azimuth)
sun_y = cos(-solar_elevation) * cos(solar_azimuth)
sun_z = sin(-solar_elevation)
v_sun = (sun_x, sun_y, sun_z)
}
Calculate the solar zenith angle: { solar_zenith = 90 - solar_elevation }
Fetch net irradiance factor: { net = f(OD, solar_zenith) }

Then, per pixel:
Calculate the angle between the surface normal and the Sun: { a_s = normals(x,y).angleTo(v_sun) } Clamp it to no more than 90 degrees.
Calculate the angle between the surface normal and the zenith: { a_z = normals(x,y).angleTo(ZENITH) } Clamp it to no more than 180 degrees.
Calculate direct irradiance: { s_dir = s_toa * cos(a_s) * exp(-OD / cos(solar_zenith)) }
Calculate diffuse irradiance: { s_dif = s_toa * cos(solar_zenith) * cos^2(a_z / 2) * (net - exp(-OD / cos(solar_zenith)) }
Calculate surface-reflected irradiance: { s_gnd = s_toa * cos(solar_zenith) * A * sin^2(a_z / 2) * net }
Calculate total irradiance: { s = s_dir + s_dif + s_gnd }

Calculate observed radiance { observed = values(x,y) * radiance_scaling_factor + radiance_offset }
Calculate scaled albedo { result = observed / s }

I won't describe how to take that result and derive actual albedo (0-1) because I doubt I did it correctly. I get reasonable values, but that could just as easily be luck.
The "result" value should be in a scale that is the same from frame to frame, so call that "brightness calibrated" if you like.
I'll say once more that there are many factors this process doesn't account for and it is far from perfect. In practice, it has proved useful for me; that's all I can say with any confidence. I'll also repeat that I had almost no idea what I was doing when I implemented this stuff, so if any of it looks wrong, it probably is.