Hello,
I'm a bit new here - I recently started on a project which I thought would be fun, to assemble movies of all the Voyager flybys. Of course, with 70k+ images, it would have to be automated as much as possible, and the results would be fairly crude. But there's enough information to make some rough black and white and color movies with one segment per target, and eventually combine them all into one movie.
The main task is getting the images properly centered, since the cameras don't point right at the target - there are still some jitters at the moment, so it needs some more stabilization. But in any case, here are some slightly bumpy rides along with Voyager - these are from versions 0.3-0.34 -
Ariel (Uranus)
https://www.youtube.com/watch?v=HeGgH34v8R4
Uranus
https://www.youtube.com/watch?v=0RLGadmvc40
Neptune
https://www.youtube.com/watch?v=HlU2_NOgoew
Assembling a color movie is a matter of combining the images that seem to go together (since it's not explicit in the data) - this is a first attempt at that. There are frames that lack certain channels, so colors are pretty off at points - later it could borrow a nearby image when a channel is missing.
Neptune, colorized
https://www.youtube.com/watch?v=c8O2BKqM0Qc
The goal is to generate the movies with as little manual annotation as possible - there are parts that need it though, like handling images with multiple targets, slowing down the movie at certain points, and turning off the centering at closest approach.
Anyway, just wanted to share what I've got so far - hopefully one day it will be able to make a nice, stable movie of all the flybys, with some appropriate music. Once the code is a little more stable I'll post the project on GitHub - it's written in Python with OpenCV, SciPy, and IMG2PNG.
-Brian
Great work Brian, and welcome to the board!
It's fascinating to compare the recent Juno approach movie with the Uranus flyby; strange seeing moons going in a visible arc around rather than passing in front and behind their planet as usual. I have a sudden hankering for a followup mission to one (or both!) of these ice giants.
Thank you! Yes, I love how Uranus is tilted like that. The moons are a bit clearer in the actual video - YouTube seems to have downgraded it to 360p, so I might put the source mp4s up on a server, when they get more stabilized.
I do hope we go back to those outer planets soon - they each need an orbiter. Maybe one of these will make it out there someday - at least I haven't heard anything about them being cancelled yet -
https://en.wikipedia.org/wiki/Uranus_orbiter_and_probe
https://en.wikipedia.org/wiki/MUSE_(spacecraft)
I just started looking at the Voyager 1 Jupiter flyby and there are some amazing images - it might be interesting to try crowdsourcing some composite and mosaic stitching into the movies, though Juno might make them obsolete pretty soon. I'll see if I can put together a rough color movie though - the automatic colorization works fairly well on Jupiter and the moons.
I love Voyager movies! People complain about the quality of Voyager images, but they really aren't that bad, especially when freshly processed, and I think they look even better animated. For me, the leftover fuzziness and other imperfections add to the warm, nostalgic charm, and what's almost as cool as a spacecraft carrying a record with Chuck Berry on it to the stars is a spacecraft using an old, 70s TV camera out at the orbit of Neptune. Besides, the Voyager images of the Jovian system, and especially Uranus and Neptune's, are still some of the best available.
If I could offer one suggestion, it would be to think about uploading movies using a single filter and exposure. Sometimes the alternating filter/exposure kind of movies give me eye strain. Really cool stuff you're doing!
I always thought it would be really cool if a crowdsourced project came about that divided all the Voyager images into their respective sequences, animated them in all color combinations available for each sequence while also animating single filter versions, and then made simulated movies out of the mosaics, showing the changing perspective of the target over the course of the mosaic's creation. Then all the movies could be arranged in chronological order and slapped on a Spacecraft Films-style DVD file. Each alternate color or single filter version could be accessed through alternate DVD angles - I've been able to do this myself.
I don't think the Voyager images are ever going to be obsolete. It's a great data set to play around with if you're just starting to get good, like me. Also, even when their scientific usefulness is dried up, they will always be of interest to people who want to see what a particular spacecraft saw at a particular time. That's the big reason I like Voyager movies. You get a unique view from a fantastic spacecraft during a great period in space exploration.
Yeah, some of the mosaics I've seen on this site are really amazing. I still remember the slightly fuzzy images of Io, Europe, and Ganymede I got in the mid 80s - those old NASA 8x10's - and those knocked my socks off.
I initially wanted to make the movies with the RAW images because I like the imperfections also, but it turned out to be too difficult to detect the target across all the different image backgrounds, so I just switched to the calibrated images. It would be possible to write out the target x,y values from the CALIB images to a file but that was just adding more complexity - maybe later.
And I'd thought about limiting some movies to a single filter also, but I hadn't considered the exposure time - maybe you could adjust the brightness depending on that value also. Or try to align the brightnesses across all the different filters in some way.
But yeah, in looking at these Jupiter images, there are tons of 2x2 and 3x3 color mosaics that would be really beautiful - it would definitely need to be crowdsourced in order to incorporate those into a movie. They need to be manually aligned, except for places where there's enough detail for some image stitching algorithms to help. But with the right set up you could have some .csv files describing the alignments which could be hosted on a GitHub project, then people could work on different chunks. Might take a while, depending on how many people were interested. But it would make for really cool HD movies or DVD with the larger mosaics.
And I guess there would be areas of color missing from some of the frames, but you could incorporate prior images into a composite or mosaic also - maybe that would look alright in most cases. As it is now, the colorizing routines manage to group the ogb images together into the different segments of the mosaics, but they're all out of alignment, and of course not put into mosaics.
And it's a good idea to make all the color combos available - I've never tried a disc that had alternate angles. Do you mean you were able to make a DVD that did that?
Yes, the Voyagers were really amazing - and all the planning that went into targeting all these different objects and guiding the craft through the orbits correctly is pretty awesome. And yeah, I think the image quality is fantastic.
Here are some color Uranus movies - eventually I'd like to merge the narrow and wide angle images to make a more complete movie - the wide angle version has some nice shots of the moons in orbit and the rings. These are version 0.36 - they need more stabilization and filling in of missing color channels.
Uranus narrow angle
https://www.youtube.com/watch?v=DReYpMJ98G0
Uranus wide angle
https://www.youtube.com/watch?v=Bnvzx7xNNrI
This time YouTube posted the wide angle movie in HD, so you can see the moons a little more clearly.
---
So far I've only looked at Voyager 1 at Jupiter, and Voyager 2 at Neptune and Uranus. The Jupiter flyby has some nice movies of the moons that will need some work, especially manually aligning the color closeups, e.g. of Ganymede - right now it just slows down at closest approach and shows all the images in b&w.
Edit: Not sure about merging the narrow and wide angle movies - it would sort of work for Uranus because it's so featureless, but not any other target.
This is *right* in my wheelhouse, as I've been tinkering with various Voyager data-sets for a few years now, looking for suitable sequences to be turned into movies. To date I've worked on:
These are great - I had come across some of yours here and on YouTube, but hadn't seen the Saturn or Neptune movies. The Jupiter and Saturn ones are really well done - after dealing with these images for a while I can appreciate the amount of work that went into them.
Those comments on your first video though...
I finally got some of the stabilization working, so here's the complete Uranus system movie with all the moons, but (seizure warning) at close approach to Uranus it flashes red and blue a lot. It's a bit long at 5 minutes, so it could be sped up a bit. This is version 0.40 (following the version of the program). I put the narrow angle at the start and the wide angle at the end - the wide angle has the view of the moons orbiting Uranus and the rings - the Uranus sections are in color and the moons in black and white - https://www.youtube.com/watch?v=JuL3hSbp7Yc.
There are still uncentered frames here and there that I'll need to correct manually, and I'd like to make color moon images and some mosaics, including the closeups of the planet, and the rings.
The stabilization code didn't work on the Jupiter movies, unfortunately - there are 11 parameters controlling the centering and stabilization routines, and it's a bit like a game of whack-a-mole - you fix them for one section and it messes up another one. I need to set up a test harness so I can get a good set of parameters.
I've got the centering and stabilization working fairly well now - here's a clip of the Voyager 1 Io approach (still working on the colors) -
http://imgur.com/a/8YWlS
The reseau removal marks from the CALIB images look like volcanoes going off all over the surface - those will need to be infilled from the prior frame where possible.
It uses a Hough circle detector (or a blob detector if that fails) then ECC maximization to a disc of the expected target size. It's fairly robust to noise and missing information - e.g. here are a couple of troublesome frames with the expected target size in yellow (based on the SPICE data) -
http://imgur.com/a/YTIlv
http://imgur.com/a/zy90J
One thing I'm not sure how to do is adjust the brightness of the images, as the CALIB images are so dark - at the moment I'm just doing histogram stretching on them, but that might be causing some of the brightness variations and odd color combinations.
---
The complete pipeline for the movies will look something like this -
Download - download archives from PDS
Unzip - decompress archive volumes
Convert - convert RAW imgs to pngs
Rotate - rotate 180
Flatfield - subtract good flatfields
Denoise - identify/eliminate noise where possible
Adjust - brighten
Center - center and stabilize images - first pass
Composite - align and combine channels using auto+manual info
Mosaic - align and combine images using auto+manual info
Dereseau - remove reseau marks cleanly - infill from prior frame where possible
Inpaint - fill in missing areas with pixels from prior frame
Undistort - geometric correction 800x800->1000x1000
Stabilize - find optimal solution - remove minor jitters
Annotate - add optional caption information
Clips - combine images into movies, one per target
Movies - combine clips into movies, add music
I was initially thinking it would need a crowdsourced type project to clean up the images, but I've come across 10 or so different types of noise that might be able to be removed or corrected automatically - then the inpainting step would try to fill in any gaps. Not sure how well it will work yet though.
The parts that would need to be crowdsourced are mainly the composites and mosaics, as there are so many of them, but some approximate solutions could be found using image alignment, then users could tweak the alignments.
There's no step in there yet for target spinning corrections, or position geometry corrections, or rotation problems (e.g. Voyager 1 Jupiter wide angle approach wobbles back and forth).
I get the feeling a complete movie might be an hour long or so by the end, unless some things are cut down, like the Jupiter cloud tops. But I guess you could produce a couple of movies - a short one with the highlights and a long one with everything, so I'll try doing that.
Here is a rough draft for the Voyager 1 at Jupiter movie, including the moons, with a bit of music - composite channels are not aligned yet -
https://www.youtube.com/watch?v=i38gzr6j5q4
I only put about 10-15% of the Jupiter cloud images in there - there are tons of them - at the end I'll probably make a complete version and a highlights version. Eventually could try to normalize the colors, or leave them as they are...
The idea was that this would be an open source movie, controlled through .csv files on GitHub, and it might work out that way, depending on how much work is involved in the editing. So far it's a fair bit of work, as I don't have good tools set up for it yet. It also depends on how well the automatic color and geometric alignment work - ideally the only editing would be controlling the framerates, what images to pause on, mosaic positionings, music timing, etc.
I'd like to publish the Jupiter movies before Juno sends back its cloud pictures - won't be able to get the mosaicing part done by then though.
Also open to suggestions for music - I'd like something more bassy for Jupiter - maybe piano should be more for Saturn...
----
Anyway, the project is up here, if anyone is interested - it can also be used to just manage downloading and looking through the images. Not sure how well the install instructions will work - I think I covered everything though - it's on Windows with Python - it uses Bjorn's img2png for the image conversion -
https://github.com/bburns/PyVoyager
Anyone is welcome to help out if you want to do some image processing - the code isn't completely stable yet, but it's fairly close - parts might be broken. Some things still to do -
- Align composites - took a stab at this with ECC Maximization but results were varied - need to tune parameters or find other algorithm
- Denoising - will try to automate this - detect noise and black it out, then inpaint where possible
- Mosaicing/geometry - since the camera orientation is not known, will need to take a guess as to where to project the image onto the globe or 2d map, account for planet rotation/time, and try to fit all the puzzle pieces together with some kind of geometric image stitching. Then can take a snapshot of the globe from some viewing location/angle for a mosaic (won't be great).
- Colorization - use lowres color data to fill in the higher res bw images - will need geometry information from previous step
- Arrange movie to combine wide and narrow angle segments - would like to do away with the separate sections for each, if possible.
- Fix hot pixels interfering with brightness levels
- Stabilize the Jupiter rotation segment - align to previous and next images
- Do Voyager 2 at Jupiter - I still haven't looked at it yet...
- Output C kernel data!
I haven't tried any of this on the Saturn images yet - the rings will break things...
I managed to get the color channels lining up properly for the closeup images, at least in most of the cases.
It works by using feature detection and matching using ORB in OpenCV, with RANSAC to eliminate outliers from a least-squares fit model for the translation (which amounts to the translation tx, ty between images being an average of the feature movements).
In more detail, 'interesting' features are detected using ORB (basically points with high variation) in one image, and matched with their corresponding point in another image. This is done for dozens-hundreds of interest points - they are each described with a feature vector (describing their color, the region around them, etc.), also obtained by ORB, then matched up with their corresponding point in the other image by a brute-force search. The RANSAC algorithm is used to throw out outliers, which would otherwise throw off the determined average translation.
If this approach fails to find a good translation (e.g. due to lack of enough corresponding points), it will fall back on ECC Maximization to try to align the images.
Here is an image showing what the feature-matching process looks like (the circles are feature points and the lines indicate matches), and the resulting combined image with contrast enhanced (part of the alignment process).
http://imgur.com/tOyS31F
http://imgur.com/9k5Brn4
It works pretty well for vague looking cloud patterns as well, though they look fairly good even if they're not exactly aligned.
----
The next part would be to try projecting these onto a cylindrical map and aligning them there as a mosaic - then you'd be able to apply some lower resolution color information to supply the missing channel (since most of the closeups of the clouds are just 2 color). Then each image would be a complete color image. This should work well on Io also, and the other moons.
I still kind of like the psychedelic colorings of Jupiter's clouds though, and I won't have time to do all that geometrical projection stuff before Juno's images arrive, so I might just leave those as they are for now and work on the rest of the movie editing.
Also, I don't know how to balance the colors yet - I've read a lot of posts on here about it and it sounds pretty complicated, or requires fudge factors. Maybe there's a way to get the average spectrum to line up with what's expected for a target and use that to adjust the channel weights?
Do properly calibrated Voyager images have a correct, intended brightness, or just a correct brightness relative to other Voyager images? The way I've been processing them into color has involved contrast stretching the brightest image and applying an identical stretch to the other, relatively dimmer images of the same target. When I apply the same stretch to Titan as I would to Saturn, Titan appears very dim, as it should be relative to Saturn.
Also, is there a correct way to balance the colors in a false color image? UV images of Jupiter are pretty dim relative to the visible light images... do those UV images have to be brightened?
I've just been constrast stretching every image as bright as it can go, which I'm sure is wrong... the colors of the composites are often odd looking, e.g. Io (though the approach and rotation movies of Jupiter, Uranus, and Neptune look okay, I think). Good idea though to apply the same stretch to every image, wish I had thought of that.
At the moment the channel weights would need to be tweaked manually, so there needs to be a more automatable way to balance the colors. I was thinking of having a reference image for each target with good colors, then adjusting the channel weights to try to approximate the correct spectrum. This would be nice for Neptune, e.g., since then you could colorize the movie to match the nicest blue images.
But I don't know about the UV filter - those images are pretty dim. I guess it depends on what you want it to look like?
And the other problem with brightening images is the areas in the corners that are sometimes whited out, or other noise in the image - they throw the contrast stretching off. I'd like to automate ignoring those hot pixels somehow also as there are a lot of those images scattered around. Something about there being a big gap between the brightest pixels and the mass of the rest of the histogram - will have to fiddle around with that.
You'll always need fudge factors and color channel mixing to get realistic color. It should be possible to 'standardize' these for at least all images of Jupiter and possibly for all of the Jovian system images (I'm working on this). Io is an exception - it's impossible to get good color for Io from Voyager images due to Io's weird spectrum and Voyager's lack of a red filter.
Automatically contrast stretching individual images without taking adjacent images (or images taken of the same area with a different filter) into account is usually a bad idea.
no kidding about Io
my map is a bit too white
the usgs's is too yellow
Here is a kind of preview of what Juno might see - Voyager 1's Jupiter encounter with the clouds, including lightning at the end! (3mins)
https://www.youtube.com/watch?v=lYUgU-Bc1_w
The color channel alignment is working fairly well, but the colors aren't balanced yet - will be working on that for a while.
And the Jupiter rotation sequence is still a bit jittery - might need special handling - the routine seems to work okay in other places.
Just an update - I've been switching things over to use IMQ files instead of the newer IMG/LBL files, so I can use ISIS with them (so can colorize all the images).
Some issues I've run into with ISIS -
IMG/LBL files not handled
https://isis.astrogeology.usgs.gov/fixit/issues/4345
cam2map doesn't handle distant targets
https://isis.astrogeology.usgs.gov/fixit/issues/4376
Some IMQ files throw error unable to initialize camera model
https://isis.astrogeology.usgs.gov/fixit/issues/4352
Some IMQ files throw error invalid time string due to UNKNOWN image time
https://isis.astrogeology.usgs.gov/fixit/issues/4414
I'm still working on getting distant targets to work with ISIS - ie when a target doesn't take up the whole image frame, and the pointing is off as with Voyager, the different ISIS camera alignment programs won't work. At the moment I can shift the image so the target appears where the pointing kernel expects it to be and then run cam2map correctly, but this would end up losing information as the target fills more of the frame. So I'm trying to rotate the camera matrix with a new ISIS program so it will point correctly at the target, but so far haven't been able to get it to work - it winds up pointing way out of the image frame.
Another problem is the amount of time needed to run cam2map, which projects from an image to a cylindrical map, using attached SPICE data - for a 1600x800 map it takes about 40 seconds on my i3 laptop - if I were to run this on every image it would take 73000*40/60/60/24 = 34 days. I looked through the code and it seems to be written more for clarity than speed, which is fine - I do need a faster computer though. I'd initially tried doing the projections in Python with OpenCV using the remap function, and at least for the simple case of a distant spherical target it would project to a 1600x800 map in 15 seconds - which would take a total of 13 days. I think cam2map is also doing corrections for the Voyager camera distortions though.
So I was thinking of cheating a bit and using predefined cylindrical maps for the missing info - it wouldn't work with Jupiter or Neptune due to cloud movements etc, but maybe it would work for things like Io, Europa and Triton. For Jupiter maybe I could run cam2map - for Voyager 1 and 2 it would be on the order of 30k images - with a faster computer using all cores maybe it wouldn't be too bad.
Still have the issue ahead of correcting the camera pointing for closeup images, though as far as I understand it the ISIS program jigsaw should handle those cases. Then I'd be able to pull back the missing info from the map and colorize the images - fortunately map2cam runs quickly. Not sure how fast jigsaw runs though... I'm guessing it's slow - it uses iteration to solve for the camera matrix.
And also need to handle the different map scales - e.g. at the highest resolution the map of Jupiter would be enormous, so will need to just focus on the relevant regions.
Was also thinking of using the cylindrical maps to animate the flybys, e.g. simulating the view as if Voyager was taking pictures of each target the whole time - would slow things down so the terrain would go by slowly underneath. Might be good for making a short movie of all the flybys.
Edit:
Aaaand another error - some IMQ files seem to be corrupted?
voy2isis throws **USER ERROR** Unable to set PDS file.
https://isis.astrogeology.usgs.gov/fixit/issues/4421
Powered by Invision Power Board (http://www.invisionboard.com)
© Invision Power Services (http://www.invisionpower.com)