MSL data in the PDS and the Analyst's Notebook, Working with the archived science & engineering data Options

[Phil Stooke]

"February 27, 2013. MSL Release 1, part 1, Sols 0-89.

The first release of MSL data takes place in two parts.

Part 1, February 27, 2013, includes raw data products (EDRs) acquired on Sols 0 through 89, August 6 through November 5, 2012, for these instruments: APXS, ChemCam, DAN, Hazcam, Navcam, and REMS, along with SPICE data.

Part 2, March 20, 2013, will include the derived data products (RDRs) for Sols 0 though 89 for the APXS, ChemCam, DAN, Hazcam, Navcam, and REMS instruments, along with both the EDRs and RDRs for the CheMin and RAD instruments, and the RDRs for the SAM instrument.

Release 1 does not include data from the MAHLI, MARDI, or Mastcam instruments. These instrument teams have not yet delivered data products to PDS.

Some documents in the MSL archives are awaiting clearance by JPL Document Review and/or the JPL Import/Export Control Office. They will be posted online as soon as clearance has been received, and announced on this web site."


Phil

[Phil Stooke]

"Mars Science Laboratory (MSL) #1 On HOLD by the MSL Project

February 26, 2013: The first MSL EDRs (Sols 0-89) for Hazcam and Navcam is pending. "



Still not quite there yet! - not in the planetary image atlas anyway. But at Geosciences, ChemCam images are in.

Phil

[renee]

MSL Release 1 Announcement:

The NASA Planetary Data System announces the first release of data from the Mars Science Laboratory (MSL) mission, covering data acquired on Sol 0 through Sol 89, August 6 through November 5, 2012.

This release takes place in two parts. Part 1, February 27, 2013, is the release of raw data sets from the following instruments:

Alpha Particle X-ray Spectrometer (APXS)
Chemistry & Micro-Imaging (ChemCam)
Dynamic Albedo of Neutrons (DAN)
Hazard Avoidance Cameras (Hazcam)
Navigation Cameras (Navcam)
Rover Environmental Monitoring Station (REMS)
Spacecraft, Planet, Instrument, Pointing C-Matrix, and Event kernels (SPICE)

Part 2, March 20, 2013, will include derived data sets for the above instruments, along with these additional data sets:

Chemistry and Mineralogy (CheMin) raw and derived data
Radiation Assessment Detector (RAD) raw and derived data
Sample Analysis at Mars (SAM) derived data

Release 1 does not include data from the MAHLI, MARDI, or Mastcam instruments.

Links to all MSL data sets may be found on the PDS Geosciences Node web site http://pds-geosciences.wustl.edu/missions/msl/. The data may also be reached from the main PDS home page, http://pds.nasa.gov/. MSL data are archived at the PDS Atmospheres, Planetary Plasma Interactions (PPI), Geosciences, Imaging, and Navigation and Ancillary Information Facility (NAIF) Nodes.

PDS offers two services for searching the MSL archives:

The Planetary Image Atlas at the Imaging Node allows selection of MSL image data by specific search criteria.
http://pds-imaging.jpl.nasa.gov/search.

The MSL Analyst's Notebook at the Geosciences Node allows searching and downloading of all MSL data in the context of mission events.
http://an.rsl.wustl.edu/msl. The Analyst's Notebook will be available starting February 28, 2013.

Some documents in the MSL archives are awaiting clearance by JPL Document Review and/or the JPL Import/Export Control Office. They will be posted online as soon as clearance has been received, and announced on the web site http://pds-geosciences.wustl.edu/missions/msl.

To receive email announcements of future releases of MSL data, please sign up on the PDS Subscription Service at http://pds.jpl.nasa.gov/tools/subscription_service/top.cfm.

The PDS Team

[Phil Stooke]

Yes, folks, now the nav and haz images are in the Planetary Image Atlas!

Phil

[jmknapp]

The recent PDS release included some NAIF files for sols 0-89. Not sure how absolutely accurate they are yet, but here's an animation based on them, from sol 34, I think the first time the robotic arm was used extensively (MAHLI took some images from underneath the rover):

http://www.youtube.com/watch?v=-kxF6Fo-EOg

[elakdawalla]

Oh that is cool. You can see them imaging the cal target and pointing APXS at its cal target toward the end, I think.

[Phil Stooke]

Nice! And I see that the rock Jake Matijevic was originally called Musk Ox... I'm collecting some new feature names.

Phil

[fredk]

Joe, could you compare an actual mahli image that shows some of the rover with a render from the model mahli at the corresponding time? I'm curious how accurate the rendered view of the rover will be. Of course you may see differences due to the optical distortions in the (real) mahli system.

[jmknapp]

Well, I don't have a way as yet to automatically set up the animation software to show the view of a particular camera, but as a manual test take this MAHLI shot:

0034MH0067001000E1_DXXX 10SEP2012 01:36:28 UT.

Positioning the RA per the NAIF files at that moment gives this:



A wire frame sketch of the model "movie" camera is shown above, which I positioned by hand to more or less match the MAHLI orientation. Here's another angle:



OK, so, drum roll, here's how the scene is rendered by that camera:



Not too far off! I used the MAHLI focal length of 21mm and f/9.8. The aspect ratio is different.

[fredk]

Nice job! Did you spin the model wheels manually so the morse cutouts matched the real positions?

[djellison]

So as of now, img2png + MSL Navcam IMG's = no joy.

I'd imagine Bjorn's already on it

[Airbag]

I'm just lost in the flood of information available there; the Mission Manager and SOWG documents alone could keep my busy for a long time.

Question - is there a way to download (say) all the .LBL files for a specific instrument for a range of sols all at once instead of selecting them one by one manually?

Airbag

[elakdawalla]

You can do it via FTP, right? http://geo.pds.nasa.gov/dataserv/anonftp.html

[Airbag]

Well, only to some extent it seems - for instance, when I go to ftp://pds-geosciences.wustl.edu/msl/ (or ftp://geo.pds.nasa.gov/msl/) there are no Navcam entries there while they are available via the Analyst's Notebook. Am I looking in the wrong place?

Airbag

[elakdawalla]

I usually don't use FTP; I download the INDEX.TAB file in the INDEX folder and use that info to build lists of things that I want that I then use wget to grab. I haven't tried this for the MSL data, but there's got to be a way.

[Airbag]

An INDEX.TAB file would work (for instance I see one for APXS) , but there is no NAVCAM subdirectory at the MSL top level. I guess that data has not been added to the FTP area yet.

Airbag

[Greenish]

Question - is there a way to download (say) all the .LBL files for a specific instrument for a range of sols all at once instead of selecting them one by one manually?
Airbag

Not sure about LBL specifically but you can easily grab all or some of the data files for a specific instrument for a range of sols using the search tab. Enter the sols of interest, select the instrument (say navcam) and select the data types (e.g. EDR) if you only want certain data. Hit submit. Then you can use the checkboxes to add whole sols or expand the tree to add individual items to your cart. When you checkout you can get the whole thing zipped or ask for a webpage of links, which includes the data and LBL files separately. You can save all of those links or I know some download managers (i.e. DownloadThemAll or Flashgot) can then grab all the .LBL files. It also looks like in future it may provide an option for a FTP script to be generated.

I did see somewhere a wget option in a corner somewhere but haven't been able to find it again.

[kemcab2012]

Interesting. A quick look reveals that the headers on some of the NAVCAM .IMG files don't start with the standard LBLSIZE or PDS_VERSION_ID metadata labels like most VICAR/PDS formatted files do. These start with ODL_VERSION_ID, with LBLSIZE popping up 26k later in the file.

Looks like they updated the PDS imaging tools back in 2011 for this. I'll need to update my own software as well.

[Airbag]

Ah yes, thanks, the Atlas Search has exactly what I am looking for, and the wget script creation option lets me grep out just the .LBL files if needed later.

[Greenish]

The MSL .IMG files do seem to be viewable with the gimp by installing the PDS plugin (http://registry.gimp.org/node/1627 or http://areo.info/gimp/).

[Airbag]

This is the kind of thing I was looking for (just out of, well, curiosity...); some sample engineering data. In this example, the NAVCAM left instrument and CCD temperatures at the time each image was taken, for a range of sols. No big surprises here - the more the instrument is used, the warmer it gets - but it (or at least the sensor) cools down very quickly again when not in use. As for any good CCD camera, the CCD temperature is kept as low as possible. It is interesting to see how much colder the instrument was later in the day on sol 45 as compared to the more typical early afternoon images.

The data came from the .LBL files associated with each NAVCAM EDS product. Well, it's a hobby!



Airbag

[JohnVV]

NAVCAM .IMG files don't start with the standard LBLSIZE or PDS_VERSION_ID metadata labels like most VICAR/PDS formatted files do.

Mind you if you use the wget option
the labels might NOT work
there is a known bug , some of the closing "end sections" are missing and some vicar data is included
-- NOT every one but a lot are affected --

downloading the zip file might be better for use in ISIS3

The MSL .IMG files do seem to be viewable with the gimp by installing the PDS plugin

that is an ancient plugin for gimp2.2

you might want to use "gdal_translate " in GDAL or FWTools3
or Bjorn's "img2png"

the old gimp plugin outputs to a 8 bit INDEXED image
just like "nasaview"

[Greenish]

I was careful to say you could see the images, not necessarily use them for anything special:)
Seriously, though, thanks for the heads-up. So far as the plugin, there is a recent version 2.8 file so it is at least kept up to the Gimp releases... And since img2png didn't work at all I thought some folks might find it useful in the interim. The images are at least not contrast stretched like the jpgs. Since the readme specifically doesn't mention the rovers, I was pleasantly surprised it worked at all.
I didn't know about the gdal tool, will check that out at some point.

[kwan3217]

The EDL spice kernels are not up yet. Perhaps those are the files that are held up in ITAR?

Bummer, that and MEDLI are the interesting things to me on this mission.

[Gerald]

I wonder, whether these visualized sol 80 Boom 1 Wind counter data

based on this Sol 80 REMS file and the associated TAB-File indicate a vortex event.

EDIT: I think, I've got it: There is a gap of 15 seconds of recording after row 19099 (timestamp 404574363), which I didn't notice before. So the peak will probably be an artifact.

[Phil Stooke]

This is a comparison of the route map in the Analyst's Notebook (top, a set of screen shots) with the latest HiRISE image of tracks and my UMSF route map. My LPSC poster is a comparison like this for Spirit and Opportunity maps (to show the difference between localization using MOC, early in the mission, and HiRISE now). So I thought I would do this comparison with the new map in PDS.

Very interesting comparison. First, look at the right side. The rock Jake Matijevic can be seen in HiRISE and my map as a black dot (arrowed by the sol 45 marker at right edge, bottom map). On the PDS map it's offset to the north. Similarly a crater between sols 40 and 41 is offset to the north in the PDS map. Actually what's happening is, the PDS map has its traverse offset to the south a few meters relative to the HiRISE base. That persists along its whole length... except between sols 29 and 38. There, the PDS traverse is shifted north to exactly the right place, causing an offset at each end of that segment which clearly does not exist in the HiRISE tracks - it must be based on a different localization result.

(PS the similarity between my map and the HiRISE is not a coincidence, I edited my map to fit HiRISE)

Phil

[Gerald]

Based on REMS data:
"REFERENCE_COORD_SYSTEM_INDEX = 3" in "PLANET_DAY_NUMBER = 29" (subdirectory for Sol 28)
"REFERENCE_COORD_SYSTEM_INDEX = 4" in "PLANET_DAY_NUMBER = 30" (subdirectory for Sol 29)
May be, the offset has something to do with the switch from Coord System 3 to 4.

[Phil Stooke]

Yes, I expect so - the basic idea being that dead reckoning from the start of the previous coordinate system took them off track a bit, but then a new localization at the start of the new site coordinate system put them back where they should be. A new error crept in again later.

Phil

[Gerald]

Next error near Sol 58? There seems to be the switch from Frame 4 to 5.
A second mere coincidence should be unlikely.

[Phil Stooke]

No, the offset at sol 29 is 'corrected' by a second offset at sol 38, and after that the routes stay together pretty closely. There's just that one strange departure from the proper path. So it may have been caused initially by a change in coordinate systems, but most changes are fairly seamless.

Phil

[jmknapp]

They released some more sol 0-89 data to the PDS today:

The NASA Planetary Data System announces Part 2 of the first release of data from the Mars Science Laboratory (MSL) mission, covering data acquired on Sol 0 through Sol 89, August 6 through November 5, 2012. Part 2 consists of derived data products. Release 1, Part 1, took place on February 27, 2013, and consisted of raw data products.

Part 2 consists of derived data sets from the following instruments:
Alpha Particle X-ray Spectrometer (APXS)
Chemistry and Mineralogy (CheMin)
Hazard Avoidance Cameras (Hazcam)
Navigation Cameras (Navcam)
Rover Environmental Monitoring Station (REMS)
Sample Analysis at Mars (SAM)

Release of the following data sets has been delayed. The data sets will be released as soon as they are made available to PDS.

Chemistry & Micro-Imaging (ChemCam) derived data
Chemistry and Mineralogy (CheMin) raw data
Dynamic Albedo of Neutrons (DAN) derived data
Radiation Assessment Detector (RAD) raw and derived data

Release 1 does not include data from the MAHLI, MARDI, or Mastcam instruments.

Links to all MSL data sets may be found on the PDS Geosciences Node web site http://pds-geosciences.wustl.edu/missions/msl/. The data may also be reached from the main PDS home page, http://pds.nasa.gov/. MSL data are archived at the PDS Atmospheres, Planetary Plasma Interactions (PPI), Geosciences, Imaging, and Navigation and Ancillary Information Facility (NAIF) Nodes.

PDS offers two services for searching the MSL archives:
The Planetary Image Atlas at the Imaging Node allows selection of MSL image data by specific search criteria.
http://pds-imaging.jpl.nasa.gov/search.

The MSL Analyst's Notebook at the Geosciences Node allows searching and downloading of all MSL data in the context of mission events.
http://an.rsl.wustl.edu/msl.

Some documents in the MSL archives are awaiting clearance by JPL Document Review and/or the JPL Import/Export Control Office. They will be posted online as soon as clearance has been received, and announced on the web site http://pds-geosciences.wustl.edu/missions/msl.

To receive email announcements of future releases of MSL data, please sign up on the PDS Subscription Service at http://pds.jpl.nasa.gov/tools/subscription_service/top.cfm.

The PDS Team

Mailto: pds_operator@jpl.nasa.gov

[PaulH51]

They released some more sol 0-89 data to the PDS today:

Thanks Joe,

I am going through the PDS learning curve, and learning a little, but still have a lot to learn. Can I ask if you have found any "REMS Ground Temperature" info in the PDS? I saw your posts in another thread regarding 'REMS Air Temperature' Thought it best to post here this time.

[Gerald]

The ground temperature may most likely be provided by some of the thermopiles. With thermopiles you can measure temperature from a distance.

[jmknapp]

Haven't looked at it closely myself yet, but I think the column definitions are in this file:

http://atmos.nmsu.edu/PDS/data/mslrem_1001/LABEL/ENVRDR1.FMT

...and as Gerald says, look for the thermopile data, such as:

COLUMN_NUMBER = 17
NAME = "BRIGHTNESS_TEMP_A"
UNIT = "KELVIN"
FORMAT = "F7.2"
DESCRIPTION = "Brightness temperature of the GTS Thermopile A
(band 8-14 um)"
DATA_TYPE = ASCII_REAL
START_BYTE = 187
BYTES = 7

So that looks like it might even be calibrated to temperature (K), although it's brightness temperature which I gather is not the same as "real" ground temperature ("kinetic temperature") from this reference:

Description of the REMS Ground Temperature Sensor aboard MSL NASA mission to Mars

[PaulH51]

...and as Gerald says, look for the thermopile data, such as:

Many thanks Gerald and Joe

[jmknapp]

OK, here's a taste of some calibrated REMS data, the brightness temperature of ground temperature sensor A during sol 89:



The sensor looks at the infrared in the band 8-14 um. Looks pretty noisy, particularly at night. Note that brightness temperature is different than actual temperature. There must be some conversion formula?

The data file for that is:

RME_405347393RNV00890000000_______P1.TAB


Generally the files with the temperature readings have "RNV" in the file name.

The column definitions are here:

ENVRDR1.FMT

[mcaplinger]

Note that brightness temperature is different than actual temperature. There must be some conversion formula?

You need to know, or assume, the emissivity of the surface in order to do that conversion. The brightness temperature is the temperature that a black body of the observed IR radiance would have.

[jmknapp]

That would explain how it's a bit high.

According to the reference above:

The use of two different bands to measure ground temperature allows the estimation of the emissivity of the surface by means of colour pyrometry algorithms.

...so the ground temperature reading is evidently a function of the thermopile A & B readings.

[Gerald]

You'll be close to a solution by dividing the two values, because (by gray-assumption) the emissivity cancels out, see http://en.wikipedia.org/wiki/Pyrometer.

Then by applying Planck's, or for simplicity, Wien's law you may get close to the absolute temperature, see http://en.wikipedia.org/wiki/Wien_approximation.

My rough approximative ad-hoc idea is, to divide the values of Wien's (or better Planck's) law for the respective mean measured wavelengths for any given fixed temperature, and compare it with the measured quotient.
More accurately, Planck's curve for a fixed temperature multiplied by the sensivity of the respective thermopile (as a function of wavelength) has to be integrated over the wavelength.

The underlying principle is, that the quotient of the emissions of a black or grey body at two fixed wavelengths is temperature-dependent, and provided by Planck's law. Restricted to an appropriate temperature interval the quotient determines temperature uniquely.

Some adjustment might be necessary due to IR absorption by CO2 or dust coating. This may be a second step.

EDIT: By the two values I meant the sensor values as defined in raw data, B1_IR_OUT_2 and B1_IR_OUT_3, rover-induced fluctuations removed, and calibrated for power or radiance. It's not easy, nevertheless.

[jmknapp]

I had a feeling calculus might be involved--a more direct route would be to get the paper submitted to the EGU by Gomez-Elvira et al, maybe they said exactly what they do.

Failing that, wouldn't the brightness temperatures in effect already be the result of integration or other analysis of the raw data? I.e., a sample at a particular time says that the ground looks like a gray body radiating at 265K for the A band (8-14 um) and 273K for the B band (16-20 um). Maybe the conversion to actual temperature from those data points isn't too involved.

P.S. I'm trying to track down this reference cited by the authors:

http://www.ncbi.nlm.nih.gov/pubmed/16642125

[mwolff]

If you are after ground temperature, then the Zorzano reference will not really help. They are using measurements at two different temperatures (and two channels) to reduce the uncertainty associated with the instrument calibration. They are interested in the time series of temperature differences because this is the more important quantity in sensible heat flux characterization. As mentioned by M. Caplinger above, you need to know the emissivity of the material to get absolute temperature. The observed radiance is equal to the emissivity times the Planck function (ignoring atmospheric effects); calculus not needed if you believe your assessments of radiance and emissivity. The hard part, and the subject of the Zorzano paper, is dealing with getting radiance accurately from the instrument measurements. If you want to send me a message, I am sure that one can find you a copy of that paper.

[jmknapp]

The thing is, the emissivity can't be determined a priori, right?

Thanks for the offer, but I was able to get a copy of the paper. By taking the measurements at two different temperatures (in practice, two different times of day) and two wavelength bands it looks like they were able to get within maybe 6K on the individual measurements and less than 1K for the difference.

[Gerald]

The paper can be read online here.
Now I understand better how the thermopiles work internally.
But I didn't find an answer, how to estimate the emissivity; there is mentioned an estimate of somewhere between 0.9 and 1.0, depending on the composition of the ground.
A 10% uncertainty means up to about 27K near 0°C.
The influence of reflected sun light is estimated to be below 0.5%.

As quotient of the Planck function for Kelvin temperature T, and two different wavelengths frequencies nu and mu, I got
(nu/mu)^3 x exp(4.8 x 10^-11 x [(mu-nu)/T] s K),
s = second, K = Kelvin, 4.8 x 10^-11 is Planck's constant h divided by Boltzmann's constant k.
(no errors assumed)
This avoids emissivity by cancelling out, if it is assumed to be the same for both wavelength frequency bands.

By taking 11 um for mean wavelength thermopile A and 18 um for mean wavelength thermopile B, one could try to avoid calculus in a first attempt.

Using the brightness temperatures instead of the radiance sounds good, especially because they will already be calibrated. I'll need to think a bit, how to do it exactly.

EDIT: The above formula just shows, how emissivity cancels out, but to be useful, integrals of the Planck function over the respective wavelengths have to be divided, with the same cancellation effect. Hard to avoid calculus or a numerical solution at the end.

With the very rough assumptions and the formula above, and without calculus, I get
T = 509 K / ln (0.228 q), with q = I(18 um) / I(11 um).
Still to solve is, how to calculate I(18 um) and I(11 um) from the brightness temperatures.

P.S.: In the meanwhile I found the calculus version here. (The paper uses the wavelength version of Planck's law.)

[mwolff]

The thing is, the emissivity can't be determined a priori, right?

Not in the way you are suggesting. With wavelength resolution and broad enough coverage, IR remote sensing has "exploited" the Christiansen frequency to have a region where the emissivity was ~1. You are probably aware of the TES products, but just in case:
(http://tes.asu.edu/products/). This approach is much more applicable from orbit, unfortunately.

[jmknapp]

Just going by some comments in the Zorzano paper.

Let us perform two measurements... with two sensors operating at different wavelengths... Standard two-color pyrometric techniques use the ratio...

All the geometrical dependencies cancel out, but any wavelength-dependent factor remains. (For instance, the medium absorption, sensor response, and target emissivity may differ from one wavelength to the other.)

Also, they say regarding the Planck equation:

This is true for a perfect blackbody radiator, but in the general case the radiance must be multiplied by the emissivity. The emissivity is a wavelength-dependent quantity 

that depends mostly on the material’s exact composition as well as the surface roughness, angle of observation, surface oxidation and contamination, particle size, level of compaction, etc. These factors are difficult to be known accurately a priori...

The user must know the emissivity to get the correct temperature value; thus this technique is not useful to explore unknown targets. In ratio pyrometry intensities are measured simultaneously at two different wavelengths and divided. The resulting representative equation is solved for temperature assuming that the spectral emissivities in both ranges are equal and are canceled in the division. This method works only if the emissivity is the same at both wavelengths, but, again, it is not useful for a general case.

They don't talk about Christiansen frquency but that sounds interesting. Thanks for the link to the TES maps.

[Gerald]

Planck's function isn't linear, and it's dependent of the absolute temperature. So it should be possible to infere the absolute temperature from the brightness temperatures at two known wavelengths and two different temperatures...
Under ideal conditions, in theory.
Before doing theory in detail, I first checked the data to find out, whether it can work in practice.
The intermediate answer is: Not in a straightforward way.
Emissivity seems to change with time, and in a different way for the two frequency bands A and B.
The first reason I could isolate is warm-up. Here some statistical analysis of 18 warm-ups on sol 89 for brightness temperature A, based on this Sol 89 REMS RDR file:


The first diagram shows brightness temperature A for the first 100 seconds of 18 warm-ups, adjusted by average and stretched by respective standard deviations.
The second diagram averages over these 18 curves.
The third diagram smoothes the latter curve by averaging over a window of 10 seconds.
It shows a significant average increase of brightness temperature A during warm-up.

So a first step to get better data, will be to apply the last column (column 63) of the referenced table, indicating warm-up, not just to pressure data as described in this FMT file, but also to brightness temperatures.

EDIT: Some number-crunching later: Radiance at 250K and 11um seems to be much less (about factor 5) sensitive to emissivity near 1 than to absolute temperature (use Planck function with 250K at 11um as brightness temperature, and divide it by the value at 255K as absolute temperature; the result is the corresponding emissivity). So an emissivity of 0.9 will lead to an absolute temperature 5K above brightness temperature (near 250K). I overestimated emissivity in the beginning. Although I didn't yet investigate, how the emissivity quotient influences measurement of absolute temperature by dividing radiances of two wavelength bands (two-colour pyrometer).

[Gerald]

Here the simplified theoretical principle for the two-colour pyrometer to calculate absolute temperature and the emissivities at two wavelengths from brightness temperatures, as far as I could infere it from Planck's law:

Input data: Take the (measured) brightness temperatures Tb_11 and Tb_18 at two frequencies nu_11 = c/11um and nu_18 = c/18um.

Algorithm:
Apply Planck's function to retrieve measured radiances I_11 := I(nu_11, Tb_11) and I_18 := I(nu_18, Tb_18).
The graph of the according Planck functions is shown in the left of the two diagrams:


Now start an iteration with initial emissivities E_11 = E_18 = 1:
Calculate a modified radiance quotient by Q := (I_11 / E_11) / (I_18 / E_18).

Calculate absolute temperature T_abs := T_abs(Q) for a given radiance quotient Q by applying the inverse function of the quotient I(nu_11,T)/I(nu_18,T) of the Planck functions (restricted to an appropriate temperature interval).
The graph of I(nu_11,T)/I(nu_18,T) is shown in the right of the two diagrams above.

Apply the Planck function to T_abs, to get the black body radiances I(nu_11, T_abs) and I(nu_18, T_abs) at T_abs for nu_11 and nu_18.
Calculate E_11 := I_11/I(nu_11, T_abs) and E_18 := I_18/I(nu_18, T_abs) to get better approximatives for the emissivities.
Repeat the iteration until T_abs changes less than a given epsilon; abort with an error, if it doesn't converge after an upper bound of iterations.

If the iteration converges, we get absolute temperature, and emissivities at both wavelengths.

[jmknapp]

Have you tried this on the time series yet? In one of the press conferences they had a chart of ground temperature vs. time that could be used as a rough check. Another issue would be how best to deal with the noisy raw data.

On the "warm up" effect--what does that refer to exactly? Like when the readings are taken after an idle time of an hour or so? Looks like they normally take readings every second for a few minutes every hour, with occasional longer stretches.

[Gerald]

I'll answer the easy questions first:
Most of the noise can be averaged away by appropriate low-pass filters, if it is purely statistical, because temperature changes are slow; the simplest way to do it, is just averaging over a few dozens of samples; a more advanced way is using appropriate regression curves.
Example for simple averaging over 40 samples:

Again based on this Sol 89 REMS RDR file.

About warm-up I know just that little info from the FMT-file; it referes to pressure data:

OBJECT = COLUMN
COLUMN_NUMBER = 63
NAME = "PS_CONFIDENCE_LEVEL"
DESCRIPTION = "String representing the confidence level for the
pressure sensor; 0 = bad; 1 = good;
- Byte 0: Warm up effect (only for barocap 1)
0 = warm up effect; 1 = no warm up effect;
- Byte 1: Shadow effect; 0 = shadow effect;
1 = no shadow effect;"
DATA_TYPE = CHARACTER
START_BYTE = 568
BYTES = 8
END_OBJECT = COLUMN

Warm-up seems to take 180 seconds at the beginning of each series of measurements, at least on sol 89.

[Gerald]

Have you tried this on the time series yet? In one of the press conferences they had a chart of ground temperature vs. time that could be used as a rough check.

The simplified answer is: No, not yet, but a good idea for a better calibration.

A little more detailed: I also looked at the UV data. They are less noisy. I tried to get a 5-point absorption spectrum of atmospheric trace gases, that form or degrade after sunrise. For this I had to do a very detailed analysis. Doing this, I also found a jiggering caused by truncation of numeric values, and indications for superposed sinus oscillations with a period of about 300 to 350 seconds of unknown origin, might be due to my way of data analysis, may be due to data analysis done by the REMS team, or due to technical properties of the sensors, like a combination of inductivity and capacity.
Similar systematic errors may also occur for brightness temperature data. Accurate results for absoute temperatures can only be obtained, after those systematic effects are annihilated. Therefore I was focussed on that.
The only thing I had been doing, was dividing the Sol 80 EDR raw thermopile A/B data to see, whether they can be used as two-band pyrometer data, and it looked good, but calibration wasn't available to me.
It will take me a while to elaborate things in more detail.

[Gerald]

Here the sol 89 results, based on the two-color pyrometer algorithm described above, and the Sol 89 REMS RDR data, without correction of systematic errors, no bugs assumed:

The green curve in the top diagram describes the inferred absolute Kelvin temperature by record (not by time), averaged (all weights set to 1) over 40 samples.
Absolute temperature was restricted to a range between 100 and 400K before averaging.
The bottom diagram describes the inferred emissivities, also averaged over 40 samples.
Emissivities were restricted to a range between 1e-20 and 100 before averaging.

The algorithm leaves one degree of freedom, e.g. the emissivity quotient. It is set to 1 in this run, meaning grey body assumed.

The absolute emissivity values look rather confusing to me, because I expected them to be between 0.9 and 1.0, and constant over time.
On the other hand, absolute temperature values look rather reasonable for less noisy regions (sufficiently high brightness temperatures).

Any of my attempts to get more reasonable-looking emissivities by adjusting emissivity quotients, absolute value of one of the emissivities, or wavelengths failed,
because other values became less reasonable.

Some calibration of the absolute temperature is possible by adjusting the difference of the wavelengths.

[Gerald]

In one of the press conferences they had a chart of ground temperature vs. time that could be used as a rough check.

The best approximation of the chart of the press conference I could get, was just using Brightness Temperature A of this Sol 10 REMS RDR Tab File, discarding 180s warm-up at the beginning of each measurement, and averaging:

Compare it with http://photojournal.jpl.nasa.gov/jpeg/PIA16081.jpg.

[PaulH51]

Gentlemen,

Not sure if you are aware, but the Spanish REMS team (Ashima Research) have "processed" the 1st 90 days of the REMS data using the public NASA PDS data sets.

Their page provides NetCDF versions of the first 90 sols of REMS measurements created from data available in public PDS archive as of March 20th, 2013. Each file is a direct conversion from the PDS archive of the corresponding file type (except for the ADR files, which is now split into two files containing the correction and geometry data, respectively. 'UNK' data values are converted to masked data, -9e36 for floating point, -2**15 for integer.

Each variable in the PDS dataset is converted to a NetCDF variable depending on the type stored in the PDS label .CHARACTER variables (strings) are converted to character arrays, ASCII_INTEGER to 32 bit integers, ASCII_REAL to 32 bit floats. In addition a "Time" dimension is added by dividing the TIMESTAMP data by 86400 to give the number of days since noon January 1, 2000. The Local Mean Solar Time (LMST) stored in the PDS tables is stored as a 18 character string, and split into sol, hour, minute, and second, with the latter containing the milliseconds component. Where available, the unit and description fields for each column have been converted to NetCDF variable attributes.

I hope that this will assist in resolving some of the issues you have encountered, but will no doubt raise further questions and additional debate

LINK to page containing the document download links : http://www.marsclimatecenter.com/rems.html

LINK to Blog post that alerted me of this data : http://marsweather.com/first-90-sols-of-re...cessed-versions

[Gerald]

Thanks! This may help more people to access the PDS data, and contribute to the debate.
I personally have been learning how to work with PDS data directly, and how to read and process them with spreadsheet or self-written software.

I think, the underlying question is, how to interprete the REMS data correctly. Several of those questions have been answered during the recent EGU conference, e.g. the origin of the UV oscillations I mentioned before; they seem to be attributed to gravity waves, not to systematic errors as I've been suspecting as likely, surprising news (to me)! Or rover movements leading to changes of the measured area, less surprising, but important to consider.
Difficult to understand are Brightness Temperature B data; jumps in the time series of e.g. UV data still have to be explained explicitely, afaik, most likely recalibrations, imho.

[Gerald]

Here one of the charts that shows UV oscillations, which I was assigning to my lack of appropriate data analysis:

It's still based on this Sol 89 REMS RDR file.
The first diagram describes each of the UV curves normalized by its average and divided by the UV ABC value.
The second diagram contains delta curves, obtained by averaging values of 150 seconds and subtracting this from the average of the next 150 seconds of the first diagram.

My original intention was to find a shift of absorption for different UV bands, that might indicate a short-term change of composition of the Martian atmosphere.
(Btw.: A shift between the red curve (UVA) and the green curve (UVB) - the two steeply ascending curves in the first diagram - looks obvious.)

The fine jiggering is likely due to numeric errors.
The next-longer oscillations have been the mystery. Can anyone else duplicate similar results? Are they due to errors or due to real physical processes like gravity waves?

[PaulH51]

I think, the underlying question is, how to interpret the REMS data correctly....

I was sort of hoping that the "Processed Data" would have provided more insight to the well documented issues you and Joe have been debating, I have been following your discussion closely, sadly I do not have the knowledge to contribute. However, I have been learning much as it progresses... So lets hope the provision of the "Processed" REMS data does bring more people into this debate.

Separately I was also hoping that the chart detailing the ground temperature (min / max) for the first 200 sols released at that EGU conference would have helped you and Joe to test / prove some of your theories / assumptions re the temperature of a target by understanding the brightness temperatures.

I will continue to monitor this debate and hopefully learn a lot more along the way...

Thanks

[Gerald]

Thanks for your interest!
There are ways to go deeper into analysis.
First: The emissivity values obtained thus far are most likely nonsensical.
During the conference they told, that kinetic temperature should be at most three Kelvin above brightness temperature. That's exactly my opinion. And it means, that emissivities for both IR wavelength bands should be between 0.9 and 1.0, and rather similar to each other, as stated earlier. Taken this as a basis we get a contradiction to the results above. The logical consequence is, that at least one of the assumptions was wrong.
One assumption was a two-color assumption instead of a two-band assumption in order to avoid calculus.
An other (implicite) assumption was, that noisy temperature data are allowed to be averaged to get the actual mean temperature; this is valid only for symmetric noise distribution.

Both assumptions will most likely to be replaced by more appropriate ones. Dropping the first one might lead to a spectrum combined of the spectra of the two IR sensors and the wavelength-dependent albedo/emissivity properties of the soil.
Dropping the second one may lead to a function, which transforms Gaussian noise to the actual noise. The idea to get this function may be the calculation of the higher momenta of the noise distribution, and infer from this the first few Taylor coefficients of the transformation function; I'll have to check, whether it works in this case. The result may then be used to average appropriately.

There will remain rover- and sun-induced effects that have to be isolated.
I'll explain details based on actual data, as soon as I succeed with one step. It's far from easy, and I've other jobs as well, so it may take a little time.
Everyone is invited to contribute improved solutions.

[Gerald]

A small step towards a better assessment of Brightness Temperature B:
Take a (cartesian) coordinate system with two axes, one for Brightness Temperature A, one for Brightness Temperature B.
Now take the Sol 89 data and add a point for each record of the data at the respective (Brightness Temperature A / Brightness Temperature B ) position.
Tile the coordiante system parallel to the axes. Count the points in each rectangle and take that count as the respective position on a third axes.
The result is a sequence of histograms. They may be drawn in the following way as "slices":

Looking just at one such "slice" (for a fixed Brightness Temperature A interval) at a time, and repeating this for the first 20 slices may result in an animated gif like this one, as a different way to represent the same data:

(Link to the gif)
The headline indicates the considered Brightness Temperature A interval. The horizontal axis describes Brightness Temperature B. The vertical axis shows the number of entries near the respective Brightness Temperature B.
(I allowed neighbouring slices to overlap.)
The diagrams show clearly, that for low Brightness Temperature A there are two peaks for Brightness Temperature B. That's not at all a Gaussian distribution. So the concern about averaging correctly is justified based on actual data.

[Gerald]

The official statement regarding the noise of thermopiles B and C according to the PDS file REMS RDR Data Set Reference Information is

Data from thermopiles B and C of the Ground Temperature Sensor are included but are too noisy to be considered useful.

I'm not yet quite at that point, because the noise seems to be structured; it might be possible - besides appropriate averaging - to exploit this structure by analysing a sufficiently large set of data.
Examples:
1. The double peak of the Brightness Temperature B noise distribution seems to indicate temperatures below 200K. Dividing the areas of the two peaks may lead to a temperature estimate.
2. Standard deviation and skewness of the noise distribution seem to be correlated to temperature.

For all REMS RDR data:

These data are not corrected from several factors such as external heat sources or shadows, so confidence level codes shall be revised carefully before using them.

[Gerald]

Here a regression polyline (consisting of 20 Kelvin line fragments) using a least squares method for Sol 89 Brightness Temperature B vs. Brightness Temperature A:

It doesn't fully exploit all statistical properties, but it may be useful from a practical point of view as a map from Brightness Temperature B data to an adjusted value comparable to Brightness Temperature A:

The first diagram shows the Sol 89 Brightness Temperature B, as provided by the PDS data.
The second plot shows the adjusted B data (in red) together with A data (in black).
The third diagram shows the two Brightness Temperatures averaged over a 40 record window, respectively. Averaging adjusted B-temperatures should be sufficiently appropriate.

EDIT: Replaced graphics after fixing inaccuracies of the least squares algorithm.

[Gerald]

This diagram shows qualitatively, how REMS UV RDR data are influenced by Masthead movements:

Data are rescaled to fit into one diagram.
Sol 89 Masthead movements can be found in this ADR file, explained in this FMT file.
UV data are contained in the same file as temperature data. They are explained in this FMT file.

So, when analyzing UV data, Masthead movements can be used as a filter to skip data invalidated by those movements.

[Gerald]

There are ways to further reduce the discrepancy between Sol 89 Brightness Temperature A and B by applying time-dependent calibrations:
Take the central part of the Sol 89 time series of the adjusted version of Brightness Temperature B, average it by a window of 100 records, and subtract Brightness Temperature A, averaged in the same way. The result shows the remaining discrepancy of the two temperature series. The goal is, to reduce this discrepancy to noise by applying simple adjustments.
The following two calibration steps lead to an improved consistency:
Step one: Subtract an appropriate biquadratic parabola for a time interval.
Step two: Subtract an appropriate exponential function for a time interval:

The applied functions aren't the best possible, but they may be sufficient to show the desired effect.

Discussion: A quadratic parabola instead of a biquadratic one didn't work. But using squared sine or cosine instead of the parabola may yield similar good results.
The exponential function is adjusted to the averaging window of 100 records; for other averaging windows either its parameters need to be adjusted to the way of averaging, or an appropriate exponential function itself has to be averaged over the window.

Interpretation: The Stefan–Boltzmann law is a temperature-related physical law that contains a 4th power. There might be a connection to the biquadratic parabola.
Newton's law of cooling follows an exponential law. There might take place a heat transfer, which explains the exponential function.

At the right wing of the central part of the Sol 89 Brightness B time series there are still discrepancies, which need to be explained in a different way. I didn't yet find an evident coincidence with other Sol 89 causes, like masthead movements applicable to UV.

[Gerald]

The REMS data don't comprise the same observation period each Sol.
So, here a comparison of 3000 seconds of Sol 82 with according 3000 seconds of Sol 89:


The Masthead wasn't in motion, and up to 0.01 degrees at the same pointing during both observation periods:
MASTHEAD_AZIMUTH: -179,00 degrees, MASTHEAD_ELEVATION: 43,00 (Sol 82) resp. 43,01 (Sol 89) degrees.
SOLAR_AZIMUTH and SOLAR_ELEVATION, relative to the rover coordinate system, were the only obviously changing parameters.
This is a hint, that the displaced peeks (Sol 82: 15:29 - 15:41 LMST, Sol 89: 15:43 - 15:52 LMST) should have been induced by the sun position relative to the rover.
The peak isn't there (14:00 - 16:00 LMST) in Sol 77 REMS RDR data. At that Sol the Masthead was at a different pointing. This indicates, that the peak was caused by the shadow of the Masthead.
At Sol 72 the peak occured (14:14 - 14:20 LMST), with Masthead pointing the same as Sol 82.

May be, Joe will be able to add more evidence by looking from Boom 1 towards the sun in a 3d simulation.

I didn't find any further obvious kind of systematic discrepancy between Sol 89 Brightness Temperatures A and B.

[Gerald]

A summarized list of ChemCam targets of the first 90 sols, calibration targets excluded:
 msl_ccam_obs_targets_noCal_Sols_10_88.txt ( 1K ) Number of downloads: 746

It consists of target/sol-pairs in a syntax usual for csv files; it's an excerpt of the file msl_ccam_obs.csv in this PDS subdirectory.
Images of targets and of spectra can be found in this PDS subdirectory.

[Gerald]

This is a "blink" gif between some raw Sol 88 ChemCam image and its processed PDS version, converted from a tif in this PDS subdirectory:

Link to the GIF.

[Gerald]

A new summarized list of ChemCam targets, first 180 sols; calibration target, sky and drill head observations have been removed:
 msl_ccam_observations_Sol_Target_TargetType_summary__sols_10_to_176.txt ( 3.52K ) Number of downloads: 716

This time it consists of target/target type/sol triples, again as comma separated textfile, ready for spreadsheet import.
It's again an excerpt of the file msl_ccam_obs.csv in this PDS directory, the syntax of which has been extended a bit. This makes it easier to filter for target types.

[Phil Stooke]

"MAHLI, MARDI, and Mastcam data are still undergoing PDS peer review by the PDS Imaging Node. The data are in lien resolution. When the major liens have been resolved the data will be posted on the Imaging Node web site (link below)."

You'll have to wait a wee bit longer for these goodies, however.

Phil

[mcaplinger]

You'll have to wait a wee bit longer for these goodies, however.

From http://pds-imaging.jpl.nasa.gov/ "The Imaging Node is pleased to announce the release of Mastcam, MAHLI, and MARDI EDRs and RDRs for Sols 0-179."

http://pds-imaging.jpl.nasa.gov/data/msl/M...R_RDR_DPSIS.PDF would be a good document to read first.

[mcaplinger]

http://pds-imaging.jpl.nasa.gov/data/msl/M...R_RDR_DPSIS.PDF would be a good document to read first.

BTW, the simplest way to use the archive products would be to stick with the EDRs, which for all JPEG compressed images (the vast majority) just consist of the JPEG data as we received it from Mars, with a simple header containing critical metadata at the beginning. I'd use only that metadata to get the SCLK, exposure time, filter, and other image parameters (ignoring the detached label), and continue to use the SPICE data for all the pointing information. IMHO getting the other metadata from the MMM archive products isn't going to be any better than using SPICE, but will likely be harder, especially if you've already written SPICE code.

Depending on what you're trying to do, doing your own processing might be as easy as using the RDRs, but that's a determination you have to make for yourself.
All of the information we use to make RDRs should be on the volumes.

[CosmicRocker]

.... The data are in lien resolution. ...

Phil: What does that mean?

[mcaplinger]

What does [the data are in lien resolution] mean?

It's legal jargon that PDS uses (inappropriately, IMHO) to indicate that there are issues associated with the dataset that they are awaiting correction of. In this instance I think these were all addressed in the volume errata files. Since they put the datasets online it doesn't really matter.

[Ant103]

Yay ! Finally the Mastcam and MaHLI images released in the PDS .

BUT, I have a big problem now. I'm using ImageJ and the PDS Reader plugin to open the IMG files in order to convert them into png files. This works for MER files, and Phoenix files. But with MSL, no. I have a message saying me : "This doesn't appear to be a PDS file". So, I do not know what else to do, because I'm on Mac OS 10.5, and there is no support from developpers for this -only 6 years old- OS. Many years ago, I tried a Gimp plugin to open IMG files, it did worked on OS 10.3 (and also Windows), but not on OS 10.5. For IMG2PNG (developped by Björn Jonsson), I can forget it, this is an executable, and I will not run any virtual machine in order to run it (by guessing it's even work). And dual boot, no, be serious! (I don't even have the space on my hard drive to do that, and the Windows installer).
Upgrade to OS 10.7 is not in my projects for now. I will certainly upgrade to OS 10.6 because I just want to run the latest versions of both Gimp, Blender, Inkscape and Hugin.

So, what can I do to open these precious MSL IMG files ?

[mcaplinger]

So, what can I do to open these precious MSL IMG files ?

The only program these were tested with, AFAIK, was NASAView: http://pds.jpl.nasa.gov/tools/release/software_download.cfm

I had nothing to do with the software development, but I'll try to look at this when I get a chance. But any solution I come up with will likely be based on the Python Imaging Library.

[mcaplinger]

But with MSL, no. I have a message saying me : "This doesn't appear to be a PDS file".

These files have detached labels; the IMG file has no label information. I don't know how the ImageJ plugin works, but possibly you need to open the LBL file rather than the image file?

For example, I just looked at http://pds-imaging.jpl.nasa.gov/data/msl/M...959E02_DRCL.IMG and LBL and this is a normal, 8-bit, band sequential RGB image with 1646 samples and 1198 rows (don't ask me where those image dimensions came from, something to do with removing some of the dark pixel columns or something? I had nothing to do with it

[Ant103]

This is weird. I can't open also Navcam IMG files with my plugin. But I thought it was the same Navcam as on MER. And so they could be open the same way. Can't tell what seems to be the problem…

[mcaplinger]

But I thought it was the same Navcam as on MER. And so they could be open the same way. Can't tell what seems to be the problem…

Looking at the source at http://rsbweb.nih.gov/ij/plugins/download/PDS_Reader.java , it appears that the plugin (from 2001) requires there to be a SFDU label in the data product. These were required a long time ago, but aren't any longer. "The PDS does not require Standard Formatted Data Unit (SFDU) labels on individual products." Seems like some updating of the plugin is required.

[Ant103]

Okay :/

This is sad because this plugin was simple, opened the files in 16 bits, and with ImageJ it was cool to export it in png with 16 bits.

I guess, I'll have to find an other way to handle MSL data (maybe try this NASAView software, but as far as I can recall, I've already tried it and found it not very convenient to use, and it's an online tool I believe, so exit the reading of local files).

Edit : I've downloaded NASAView, and yes, it can open local files (I mistooken it with an other software). So now, the challlenge is to apply the right color scheme. Because I'm opening an LBL files, and I have a window that suggest me to enter value for each R, G and B "layer" of the Bayer matrix (I think).

Edit 2 : Opening Mastcam. Check. Find the good values for the LBL file. Not quite. Save the result as a Jpeg file. Not very good. It leads to a greyscale images with vertical bands all over the image. And with Navcam images, the Jpeg export lead to a stretched picture. Not the thing I'm looking for when working with IMG files…

Edit 3 (and this will be the last) : I'm reading on http://pdssbn.astro.umd.edu/tools/software.shtml this line about NASAView : "The program is not intended to be used for saving and intensive scientific analysis of the data.". As well I'm not a scientist, I quite like an high quality imagery. So, NASAView is not the droidsoftware I'm looking for.

Edith : An interesting reeading here about a modifyed version of the ImageJ plugin. http://www.marsroverblog.com/discuss-20354...sicspage14.html It seems to working well with MSL IMG files . I managed to download the .java source file, and copy / paste the new code. I succeed to compile it. I will try with Navcam files.

[mcaplinger]

This is sad because this plugin was simple, opened the files in 16 bits, and with ImageJ it was cool to export it in png with 16 bits.

Someone who knows how to compile Java in a platform-independent manner could fix the plugin in about 5 minutes, I suspect.

Attached is a simple Python script that reads these files and saves them out as PNGs, but if you don't have Python and PIL installed on your system, that won't do you much good. Also, PIL can't save a 16-bit-per-channel RGB image, so the 16-bit forms of the RDRs would have to be managed as separate channels. For this version, I convert 16-bit data back to 8 bits.
 pdspildet.txt ( 1.56K ) Number of downloads: 580


Of course, I hasten to point out that if you want to use the RDRs, saving them out as JPEGs is losing information, since they were JPEGs to begin with as transmitted from Mars.

[Ant103]

Okay, this new PDS Reader plugin for ImageJ run like a charm .

Here a Sol 2 Navcam image.

Attached thumbnail(s)

 

[mhoward]

Thanks for all the tips on this. One thing I'm wondering is if any files are duplicated between, for example, the MSLMST_0001 and MSLMST_0002 directories, and if so, are the contents of the files the same or different. (I'll probably find out the hard way before too long.)

[PaulH51]

Okay, this new PDS Reader plugin for ImageJ run like a charm .
Here a Sol 2 Navcam image.

Looks crisp... Can you point us to the original 'raw' file name for this image (prior to calibration)? Or if that is not possible then its current file name in PDS.

TIA

[mcaplinger]

One thing I'm wondering is if any files are duplicated between, for example, the MSLMST_0001 and MSLMST_0002 directories, and if so, are the contents of the files the same or different.

There are certainly instances where we have transmitted the same image with different compression types or quality levels (for example, the MARDI EDL images were initially transmitted as JPEGS of one quality, then a subset were retransmitted with higher quality, and eventually as lossless versions.) Also, I wouldn't be surprised if some files got archived in partial form (data missing at the end) and then later in complete form, although those arguably should have been weeded out in the archiving process. If you see anything that looks confusing, let me know by PM and I'll pass anything I don't understand on.

[Ant103]

Yes Paul

http://pds-imaging.jpl.nasa.gov/data/msl/M...AUT_04096M1.IMG

[fredk]

Edith : An interesting reeading here about a modifyed version of the ImageJ plugin. http://www.marsroverblog.com/discuss-20354...sicspage14.html It seems to working well with MSL IMG files . I managed to download the .java source file, and copy / paste the new code. I succeed to compile it. I will try with Navcam files.

Thanks for the tip, Ant. I get an error message when I compile the new PDS reader:

CODE

C:\Program Files\ImageJ\plugins\PDS_READER_NEW.java:10: Public class PDS_Reader_New must be defined in a file called "PDS_Reader_New.java".
public class PDS_Reader_New extends ImagePlus implements PlugIn {
             ^
1 error

Did you have this problem?

Edit: Fixed. The filename needed to be "PDS_Reader_New.java", with mixed upper/lower case. Works great now.

And thanks to Horton, if you still read here...

[Gerald]

A list of APXS targets, first 180 sols:
 apxs_summary_targets_sol_0_179.txt ( 4.01K ) Number of downloads: 502


CSV-file (for spread sheet software) containing according weight percent (rwp) data:
 apa_rwp_Sol_0_179_comma.txt ( 8.3K ) Number of downloads: 695


Graphical synopsis:


The above data and graphics are derived from data provided by this PDS subdirectory.

[Greenish]

Nice, Gerald. Here is MAVOR, the one from Sol 158 that is the obvious outlier in terms of way more SO3 content. Distances are from the daily uplink notes.
context 25cm http://mars.jpl.nasa.gov/msl-raw-images/ms...1000E1_DXXX.jpg
closeup 2cm (focus merged) http://mars.jpl.nasa.gov/msl-raw-images/ms...0006R0_DXXX.jpg

(edit, fix link to full image not thumbnail)

[Gerald]

Thanks, Greenish!

Mavor APXS data are consistent with most of the sulfur being bound to calcium sulfate. But there seems to be a small excess of sulfur:

CODE

28.8  (g SO3/100g) / 80 (g SO3/mol) = 0.360 mol / 100g,
18.95 (g CaO/100g) / 56 (g CaO/mol) = 0.338 mol / 100g,
excess of SO3:                        0.022 mol / 100g.

PDS data source in file apa_411516167rwp01580051954_______p1.csv in this PDS subdirectory.

[Gerald]

For convenience, here the PDF version of the MSL mission summaries for sols 0 to 179:
 MSL_Mission_summaries_sol_0_179.pdf ( 55.73K ) Number of downloads: 943

It can be obtained via Analyst's Notebook:
Follow the link "Historical overview", then "Download as PDF".
Print the page, redirected to saving as PDF file.

[Gerald]

The elemental composition determined by APXS analysis is derived from APXS spectra.

Two examples of APXS spectra:


They show the rsp files apa_401575435rsp00460042100_______p1.csv (this PDS directory for sol 46 APXS RDRs) and apa_411516167rsp01580051954_______p1.csv (this PDS directory for sol 158 APXS RDRs) as diagrams.

Transforming the x-ray photon energy (in electron volt) via
Z = sqrt(eV / (0.75*13.6057))+1
simplifies the identification of the K-alpha spectral line with the atomic number of the respective chemical element:

(See Wikipedia about Rydberg constant, Lyman-alpha line, and Moseley's law)

Note the peaks at K-alpha for atomic numbers 16 (sulfur) and 20 (calcium) in the Mavor spectrum.

Although some of the x-ray emissions aren't K-alpha.

A comparison of Ekwir_1 (29896 seconds counting time) with Snake river (1330 seconds counting time) shows, how counting time influences data quality (logarithmic scale for the counts):

[Gerald]

Here an annotated version of the Ekwir_1 APXS spectrum:

I've been trying to find the most plausible x-ray emission lines the above spectrum is composed of without using sophisticated optimization software.
It may provide a rough idea of how APXS data can be analysed. Besides the K-alpha spectral line the K-beta line(s) occur, particularly for calcium (Ca) and iron (Fe). For the peaks between 4.6 and 5.0 keV I didn't find a fully plausible explanation; the peak near 22.2 keV is close to K-alpha of silver, and occurs also in pure SiO2 calibration data.
For Rayleigh and Compton backscattered x-rays plutonium emission lines are relevant, because curium used by APXS decays (partly) to excited plutonium, which releases x-ray photons when returning to the ground state. The Rayleigh component occurs close to the originally emitted photon energy, the associated Compton spectrum is in an energy range a little below.

A summary of APXS as used by MSL can be found in the MSL science corner.
Detailed data about X-ray transition energies can be queried via NIST.
Technical and physical background about X-ray spectroscopy can be found in this IAEA paper.
This LPSC 2008 paper provides an idea of how light elements can be inferred by comparing Rayleigh with corresponding Compton backscattering when the preliminary elementary composition is known.
The recommended document for APXS calibration is behind the paywall.

[jmknapp]

Here's the UV irradiance data from the PDS displayed in graphical form by sol:

http://curiosityrover.com/rems/uv.php

There are some weird jumps and scale changes in the data from time to time--it's from the ENVRDR files. I think that the MODRDR files are supposed to have corrected data, but from what I can see the UV data isn't included there yet.

The max in the broad band (bandABC, 200-400nm) looks to be about 20 watts/m^2, whatever that translates to in sunburn factor. I that it's a lot, particularly in the "erythemal" band (approximately bands A&B), where 250 mW/m^2 on Earth is considered a UV index of 10.

[elakdawalla]

One does not simply....find UV data in MODRDR.

(Sorry, couldn't help myself)

[jmknapp]

It is a barren wasteland, riddled with fire and ash and dust, the very air you breathe is a poisonous fume.

[Gerald]

There are some weird jumps and scale changes in the data from time to time--it's from the ENVRDR files.

Corrected a processing error in UV sensor in ENVRDRs, in which when the Solar Zenith Angle was between 20 and 30 degrees UV radiation was processed as diffuse radiation instead of as direct radiation.

cited from Errata.
So some of the REMS calibration errors seem to have been fixed in the meanwhile. Not yet all as it looks, and more fixes may be provided in the August 30 release.

... And really terrible are UV C to E. There isn't enough oxygen/ozone in the Martian atmosphere to protect the surface.

(after , not just me)

[jmknapp]

Interesting. I changed the data files to use release 2--things look a lot better. Thanks, Gerald.

BTW, the UV index as calculated on Earth would (roughly) be in excess of 100, where 11+ is "extreme risk of harm from unprotected sun exposure."

[Gerald]

PDS data of Sols 180 to 269 are going to get online, e.g. REMS data.

Edit: WUStL also online now.

[Gerald]

PRELIMINARY(!) CheMin analysis results for John Klein, PDS file sorted by abundancy:

MINERAL,PERCENT,ERROR
ANDESINE, 43.8, 3.6
PIGEONITE, 12.7, 4
AUGITE, 8.5, 3.4
MAGNETITE, 7, 1.9
ORTHOPYROXENE, 6.8, 3.1
FORSTERITE, 5.1, 3.3
ANHYDRITE, 3.3, 2.1
ALBITE, 2.7, 1.3
AKAGANEITE, 2.4, 1.4
BASSANITE, 2, 0.9
SANIDINE, 1.7, 1.8
PYRRHOTITE, 1.5, 1.5
HEMATITE, 1.2, 0.9
QUARTZ, 0.6, 0.7
PYRITE, 0.4, 0.5
HALITE, 0.2, 0.3

"... ~28 +/- 15 weight% of this sample consists of X-ray amorphous material ..."
Link to akaganeite.

[Gerald]

Mission summary update, sols 167 to 269 (site 6, accessible via Analyst's Notebook) :
 Mission_summaries_sol_167_269.pdf ( 31K ) Number of downloads: 596

Details about conjunction activities:

Sol 236-260 : Mission Manager

Activities

The pre-built conjunction plan for Sols 236-261 includes background environmental science with the REMS, RAD and DAN instruments. No commanding is planned during this period but the team continues to assess S/C health via a daily direct-to-earth X-band beep as well as basic telemetry relayed via Odyssey.

[elakdawalla]

The mission summaries are now available through sol 359:

 MSL_Historical_Overview.pdf ( 218.92K ) Number of downloads: 852


The images are not in the Analyst's Notebook yet -- I'm told by someone at Wash U that they are being ingested into the system now but that it'll be a day or two before the database update is complete. You can, however, download all the images from the Imaging Node of the PDS. (Wash U gets them from the Imaging Node.)

[xflare]

I still cant find any Mastcam images.