Printable Version of Topic

Posted by: ugordan Nov 9 2007, 12:59 AM

I've made some progress with the IR channel on VIMS, have managed to do partial calibration of the cubes. It's still buggy, but at least it's something. Here's a couple of rough results, all using R=5um, G=2um, B=1.26um.

In case of Saturn the green and blue channels were scaled up to match the intense thermal radiation. The cubes were flatfielded, quantum efficiency corrected and divided by solar spectrum, though I don't know if all the steps were carried out correctly. Dark current subtraction remains to be done - it's possibly the reason why the lower left Titan insets have banding in the red channel. I'm actually pretty satisfied with the way these two turned out, they are reminiscent of official VIMS surface composites, including the "blue" material.

Hopefully I'll be able to put something out to convert cubes to PNGs before long. A better solution would be a GUI but that's the tedious part for me.

Posted by: djellison Nov 9 2007, 07:40 AM

Ahh - congratulations - to be honest, command line based makes a lot of sense as then we can write nice long batchfiles to process a whole load of it

Doug

Posted by: ugordan Nov 9 2007, 08:02 AM

True, but there are issues with that approach - waiting for say 352 images to be written (and it might be pretty slow saving as PNGs due to ZLIB, I'll have to see yet), only to find out the channels you wanted are either severely underexposed, overexposed, etc.

Rest assured, though, a command line version is coming first.

Posted by: remcook Nov 9 2007, 10:20 AM

That looks very impressive! I think PDS also has some complicated calibration software with it. Did you use that, or did you make your own?

Posted by: ugordan Nov 9 2007, 11:33 AM

I'm "reverse-engineering" that software and the available documentation. The software included is a mix of C, Perl and IDL and not particularly straightforward. The code is cryptic enough for me to have trouble deciphering it so I'm not very certain I'm doing everything right. I've found some peculiarities, for example with VIS flatfields which (apparently from what I've seen) need to be multiplied with the cube, not divided out as usual (and as is in fact the case with the IR portion of the instrument). Things like that make your job a little harder and makes you wonder if you're supposed to be multiplying or dividing at certain other points in the pipeline as well.

For a reference on the current state of agreement between these Titan results and official VIMS team calibrated products, see Jason W. Barnes' paper: http://c3po.barnesos.net/publications/papers/Titan.spectral.diversity.pdf

Pretty good on my part, but there's room for improvement. Jason's first maps using the same wavelengths actually seem to use several cca. 5 um wavelengths summed up to reduce noise in that channel (those last few wavelengths are particularly noisy unless long exposures are used). Other than that the colors agree pretty well with slight differences possibly attributed to mentioned dark current/background subtraction missing here.

There is one major step in calibrating VIMS cubes my code will never do: geometric reprojection. There are all sorts of distortions introduced into the cubes including spacecraft motion, slew maneuvers and jitters that can and often do mess up the end result. To sort this out spice kernels with accurate s/c pointing knowledge are used. That's way beyond my league and all this is bound to be for recreational purposes only - pretty pictures and stuff.

Posted by: elakdawalla Nov 9 2007, 06:20 PM

ugordan, this is awesome. I particularly liked the crescent view of Titan with surface features visible -- tremendous. Command-line is good enough for me!

Out of curiosity, do you have any idea how the total number of VIMS cubes compares to the total number of ISS images? I am thinking it would be very helpful to put together a browse page, showing either this color combo (which is great) or even just a single grayscale image, maybe the 1-micron one, from each...

--Emily

Posted by: ugordan Nov 11 2007, 12:39 AM

Preliminary, test release of QUB2PNG which dumps the entire cube to PNG files.
It expects the following files to be located in a directory named "calib" which should be in the same directory as QUB2PNG.EXE:

vis_flatfield.qub
vis_hires_flatfield.qub
vis_lowres_dark_model.tab
vis_hires_dark_model.tab
vis_perf.qub
ir_flatfield.qub
ir_hires_flatfield.qub
ir_perf.qub
solar.qub

These calibration files can be located on every VIMS PDS DVD, for example http://pdsimg.jpl.nasa.gov/data/cassini/cassini_orbiter/covims_0015/calib/

At the moment there are a number of limitations to the program - it doesn't correctly work on spectrally summed cubes nor ones where either VIS or IR part of the instrument is turned off. It also doesn't yet work with IR sampling mode "UNDER" a.k.a. NYQUIST mode.

When searching the PDS you want cubes where the mode is IMAGE, not LINE or OCCULTATION.
By default, running the program with just the name of the VIMS cube it will dump the 352 wavelengths into the same folder - not a very good idea, read on.
The second parameter is a scaling factor, useful for tweaking the brightness of various targets and managing underexposure/overexposure. Note that due to significant detector quantum efficiency variation over the spectral range, raw data that was saturated at certain wavelengths doesn't necessarily appear saturated (maxed-out brightness) in the output PNG data.
The third parameter is the output directory.
An example: qub2png v1356762057_2.qub 10 output\

The resulting filenames are as follows:
v1356762057_2 - [072] 0.87863.png

Where 072 is the channel number, 0-95 is VIS channel, 96-351 is IR channel. 0.87863 is the wavelength in microns. In general you do not want to mix the VIS and IR channel data in your processings, they aren't compatible currently (different scale factors, not entirely calibrated), have slightly different boresights, FOVs and distortions depending on mode. You'll notice the last few VIS channels and the first few IR channels overlap.

VIS HI-RES mode is 3x spatial resolution of NORMAL mode on both axes, IR HI-RES mode is 2x spatial resolution in the horizontal axis while the vertical axis is the same as in NORMAL so it "squashes" the images.
VIS and IR NORMAL modes are almost identical in FOV, but the boresights are slightly shifted a pixel or two in both axes.

Looking forward to any feedback.

 QUB2PNGv01.zip ( 43.96K ) : 662

Posted by: ugordan Nov 11 2007, 01:02 AM

In the meantime, a few more "results":


Airglow in Titan's atmosphere extending to huge altitudes. Note the "reddish" cloud just peeking over the limb at 1 o'clock in the surface view. Another shot of the airglow, 3.316 um is mapped to red, 3.283 um to green and 3.213 um to blue:



Jupiter with Europa (seen in natural color composite) and the green splotch in IR at the left edge is Ganymede.

Finally, a scan through the 256 IR wavelengths in a Quicktime movie, just for fun:
 titan_vims.mov ( 534.98K ) : 643

Posted by: ElkGroveDan Nov 11 2007, 03:08 AM

Wow Gordan. This stuff is amazing. That atmospheric Titan image is an APOD candidate.

Posted by: nprev Nov 11 2007, 02:50 PM

Beautiful work, Gordan; really awe-inspiring, actually!

Posted by: ugordan Nov 11 2007, 03:44 PM

New version is up, improved IR background subtraction (as per official SW), results are pretty good.

 QUB2PNGv02.zip ( 45.7K ) : 607


I'm all ears for any glaring bugs.

Compare these two composites with previous ones:

The crescent Titan view is actually the same as in this official http://wwwvims.lpl.arizona.edu/whatsnew.html (first image). Obviously the colors don't exactly match, but to do that I think I'd need exact channels used in the mosaic since even adjacent channels can have significantly varying brightnesses.


Left: R=4.97 um, G=4.55 um, B=3.31 um. The stack on the right now looks very close to the official result I linked to in my second post, the map on page 8.

Finally, two more Quicktime IR scans. Click on images:
http://www.fileden.com/files/2007/9/14/1431389/v1477468808_1.mov http://www.fileden.com/files/2007/9/14/1431389/v1514284191_1.mov

Posted by: nprev Nov 11 2007, 05:48 PM

Wow... ...just don't know what to say, man. Those are amazing vistas, obviously of scientific importance as well.

Posted by: Bjorn Jonsson Nov 12 2007, 12:49 AM

This is awesome, I'll definitely try this out once I have downloaded some VIMS data in a day or two. An idea: If that feature is not already present, it would be great to be able to generate true color images using data from all of the VIS channels. This should result in more accurate color than the ISS data (as has been discussed in different threads somewhere).

This calibration stuff is remarkable complex. I'm still discovering bugs in my ISS calibration code which is reverse engineered from CISSCAL and some of the calibration documents. It would be interesting to compare our results there.

Posted by: ugordan Nov 12 2007, 07:46 AM

That feature was the original reason I wrote a tool for processing VIMS cubes. For some obscure reason, though, I get different results depending on whether I use CISSCAL's solar spectrum (originally what I did) or the VIMS solar spectrum cube. Using the CISSCAL one I can get the usual whitepoint for gray bodies (Enceladus, Mimas, Tethys...), while if I divide with the cube I can simply not manage to do that, it either turns yellow or red, depending on whether I divide or multiply in the VIS channel in certain steps (I mentioned this before).

As a consequence I removed that part that generates "true" color temporarily until I sort this out. The VIS channel is bad enough in that you need an additional dark current model to not get ugly results. As it currently stands the VIS dark current removal is lacking, usually requiring manual tweaking to either not retain residuals or over-subtract, each resulting in color shifts - the former towards blue color the latter toward green.

Posted by: remcook Nov 14 2007, 10:08 AM

I assume it would be relatively straightforward to repeat the recent analysis of Adamkovics et al. from telescope data and show the morning drizzle. As far as I can see it's just subtracting images between 2.060-2.070 um (an average might be good enough, maybe) from the same image between 2.027-2.037 um times 0.72 (or some factor close to that depending on the exact shape of the filter used). Although, as is common with Science and Nature papers, it is not quite clear (to me) what they did exactly.

Posted by: ugordan Nov 14 2007, 07:42 PM

I don't have access to that paper to really know what they did. Even if that were the case, I wonder if a repeat would be straightforward as I get the impression they were looking at very specific wavelengths on the edge of a methane window and whether or not VIMS' discrete wavelengths would fit the bill. The closest channels are 166 at 2.03424 microns and 168 at 2.06757 microns.

I did try some funky ratios between spectral windows, below are some composites. The left-hand sides show R=5 microns, G=2.7 microns, B=2 microns. The right sides are all ratio composites:
R=5 microns/2 microns
G=5 microns/2.7 microns
B=2.78 microns/2.7 microns & contrast stretched.

The upper image is one of the longest IR exposures, while the lower one is more typical of distant shots so it's more noisy.

In the 5/2 ratio image the contrast between dark equatorial stuff and the rest is practically lost. Tui Regio and Hotei Arcus consistently stand out. Intriguing.

Things at Titan at small scales really start to look strange if you take a ratio of 2.7/2.78 microns. Quivira, for example, looks completely different:
http://i108.photobucket.com/albums/n15/ugordan/quivira_ratio.png

Top left: normal 2 micron reflectance
Top right: 2.7/2.78 micron ratio, heavily contrast stretched
Bottom left composite: R=ratio, B=2 micron, G=( R+B )/2
Bottom right composite: Classic 5 micron (red), 2 micron (green), 1.26 micron (blue, slightly overexposed in raw data)

Posted by: remcook Nov 15 2007, 10:58 AM

They used filters from ground-based telescopes, so a (small) range of wavelengths convolved. The paper itself doesn't give much more info than i've just given I'm afraid. Would be cool if it was possible with VIMS too, since the spatial resolution is much better and there are lots of observations.

Posted by: Bjorn Jonsson Jan 10 2008, 10:45 PM

This is slightly off-topic but has anyone been able to download the TIFFs on the January 1, 2008 VIMS PDS data volumes? I'm getting "430 Forbidden" when using Wget and errors too when using IE or Firefox. I get the impression that something is wrong on PDS' end. I was able to download everything else.

The TIFFs are highly useful for figuring out which QUBs you want to convert to PNGs.

Posted by: ugordan Jan 10 2008, 10:59 PM

Yeah, the TIFFs don't work for me either, it does seem like a problem with file access permissions on their end.

P.S. If you're using QUB2PNG for the conversion be warned that the way HIRES IR cubes are flatfielded is wrong in the latest version. I fixed it in the meantime, but never got around to posting it here due to apparent lack of interest. I may post an updated version some time soon.

Posted by: elakdawalla Jan 11 2008, 12:08 AM

Rest assured, ugordan, it's not lack of interest, just lack of time. What you're doing with QUB2PNG is really awesome. Have you contacted any members of the VIMS team to tell them about it?

--Emily

Posted by: Bjorn Jonsson Jan 11 2008, 01:28 AM

A similar situation in my case - lack of time so only very rudimentary testing so far but this will soon change.

I've been spending huge amounts of time making DEMs of Saturn's satellites but fortunately the "development phase" is finally over. Stay tuned for "Iapetus the Movie" .

Posted by: ugordan Jan 11 2008, 07:29 PM

Emily, I'm curious - why do you think the VIMS team would want to know about this?

Anyway, here's the version that does flatfielding correctly (as far as I can tell):
 QUB2PNGv03.zip ( 45.78K ) : 714


P.S. Bjorn, that movie better be released soon or we'll start to get impatient now that we're know what's been cooking

Posted by: djellison Jun 21 2010, 05:40 PM

>2 year old thread time warp.

Version 1 and 2 run great on my machine. Version 3 crashes immediately. XP64 machine. Just doing a straight qub2png filename.qub without any other details. Anything I can do to help in identifying and fixing. PLUS - did that suggestion of automatically spitting out a true color image get any traction?

Posted by: ugordan Jun 21 2010, 06:43 PM

Hmmm, did v3 ever work for you properly? Does it crash with every *qub file? Do you get any printouts whatsoever when you run it manually in a command line box, both with and without a filename?

Posted by: djellison Jun 21 2010, 09:05 PM

I don't think I ever tried V3 before today - so I don't know. Every QUB I had tried crashed it, including one's that worked fine in V2 and V1.

But - inexplicably - I restart my machine at lunch, and it seems to work now. Which is v good news.

That just leaves part two - the true color interpretation.

Posted by: ugordan Jul 4 2010, 04:51 PM

Here's a standalone version of the tool for "true" color output only as they use slightly different processing pipelines. Put the two *.tab files into that calib subdirectory where qub2png.exe already resides. Usage is pretty much the same, filename and then an optional scale factor and gamma factor. Both factors default to 1.0 if unspecified. Output is a single 16 bit RGB PNG file.

 QUB2RGBv01.zip ( 89.93K ) : 590

Gamma corresponds to the power function mapping linear I/F data into the nonlinear sRGB colorspace. The correct value should be 2.2 for proper contrast representation, although this is rarely used with spacecraft imagery. CICLOPS seem to use around 1.33 gamma for their releases. If you leave gamma at a default setting, the contrast and saturation will be too high if your viewing/editing software assumes sRGB gamma of 2.2 (which by and large they all do by default).
For myself, I created a custom color space in Photoshop for ISS and VIMS calibrated imagery that has gamma set to 1.0 so Photoshop converts to sRGB on the fly, but you may want to skip that hassle and hence this option.

Word of warning: many of the cubes have at least some saturated visual channels, causing pink hues to be output. This is not real, but is an artifact of the saturation, quantum efficiency of the CCD at certain wavelengths and the way color is calculated. Here are two such cases, one severe and one less noticeable:

Posted by: djellison Jul 10 2010, 05:42 AM

RESULT

http://twitpic.com/23vggd

Posted by: mhoward Jul 10 2010, 02:26 PM

I love that. It gives a dizzying sense of actually being there.

Posted by: machi Apr 28 2011, 09:54 AM

My first larger work with VIMS spectral qubes. Thank you Ugordan for your excellent program qub2png.
Animation of southern hemisphere of Saturn from distance ~1.1 mil. km - http://www.youtube.com/watch?v=O49WZlfi3ic.
Further particulars in youtube' info.

Posted by: ugordan Apr 28 2011, 09:52 PM

Nice! I do have one question, though - if that's the *southern* hemisphere, isn't Saturn rotating the wrong way round?

Posted by: machi Apr 29 2011, 09:36 AM

I suppose, that it's southern hemisphere, because maximum latitude visible in animation is ~ -30 (information from index.tab).
You are right (and you are actually first who draw attention to this), Saturn is rotating probably in wrong way. I realized that, after I uploaded video to youtube and youtube is not sophisticated about changing parameters of uploaded video .
For now, I prepared this corrected version: