High-Res DEMs from single HiRISE images, First results of new "Shape from Shading" algorithm Options

[Nirgal]

Hi all,

Here the long overdue continuation of the "Alien Landscapes" series. This time based on 3D
DEMs generated with "Shape from Shading" from single HiRISE images.
Enjoy

Click on Images for larger version.

Detail views from PSP_002172_1410 (large gully system)





Detail view of Gullies from PSP_001376_1675





Detail of gully system in PSP_002022_1455



Dune Views from PSP_004339_1890





Detail from PSP_001834_1605





Here is some background info on the making of the images:

"Shape from Shading" (SFS) i.e. the possibility to extract shape information from a single image has always been a fascinating topic for me.
Now I found the time to implement a prototype for a new SFS algorithm based on some ideas that I've been thinking about for a long time.
The problem with existing SFS approaches (see here for a survey is that they either tend to over-smooth the details (due to the regularization constraint) or suffer from excessive noise in the high-frequency components of the reconstructed surface. Another problem is the large demand on CPU ressources which would make them very challenging to apply to large scale input data, such as HiRISE orbiter images.

So for a long time I was rather sceptical as to the potential of SFS and it was my impression that Methods based on multiple images (stereo) must be far superior to single-image SFS.

However, after a long time of experimenting, combining existing approaches with some new ideas, I got the following quite promising first results that I'd like to share:

All of the images were generated from a single HiRISE image (no depth information was used from stereo or laser altimeter data).
Also, no texturing or additional coloring/shading was applied when rendering the surface.
Every detail visible is real 3D down to the pixel-level...
For rendering I used a very simple model based on lambertian reflection with gouraud shading.

The resolution of the images is still moderate: that is downsampled details crops in the order of 0.5-1 Megapixels.
However, despite the heavy math machinery that drives the core of the algoritm (several systems of equations with millions of unknowns)
the processing time is still moderate (about 15 Minutes per med-res image, using about 2 Gigs main mem) such that the application to full-res HiRISE images should be possible

The following image shows an example to illustrate the general principle (click to enlarge).


On the left hand side the 2D input image (simple noisy JPEG from the Web with unknwon light source direction). On the right hand side shows the recovered 3D surface re-lighted under a different light source direction.
Note that one problem of the current implementation of the algorithm is it's vulnerability to notable distortions in the low frequency components (i.e. large scale variations) of the generated surface. However I'm confident that this can be overcome by an improved version or by adding the large-scale depth information from stereo-based DEMs or altimeter data (MOLA) where available.

[vikingmars]

Bernhard, just an idea...
Because this 3D HI-Res DEM rendere is a huge work of yours that is to be acknowledged, what if you share your software to UMSF members versus a financial participation (or a donation) ?

[Nirgal]

Bernhard, just an idea...
Because this 3D HI-Res DEM rendere is a huge work of yours that is to be acknowledged, what if you share your software to UMSF members versus a financial participation (or a donation) ?

Thank you all so much for all the nice words and suggestions !

At this time, the project is unfortunately not yet ready to be officially released in any publicly useable way for several reasons. First, it is still in a very early stage of development and I have just begun to explore the possibilities and there are a lot of ideas for improvement that I would like to try out, many bugs to be fixed etc.
Also, there are technical and license related obstacles : for example I am using a large collection of self written and third-party C/C++ APIs and libraries that I developed and collected over the past 15 years.
While some of them are open source, there are also commercial or closed source libraries that would have to be completely re-written first. (this task alone would be a several-years project I'm afraid)
And most importantly, there is no such thing like a GUI or comfortable user interface. The whole project is more like an experimenting platform consisting of a complex & ugly command line interface, a cryptic shell language and about 2000 tuning parameters ...
Also, I am doing this all in my spare time only, as a means of recreation and relaxation ...
And for me it is simply much more fun to explore and implement interesting algorithms without taking care to write clean and robust user interfaces, doing customer support or the like (this is what I have to do in my profession as software developer all the time

Finally I view my software projects mainly as tools, i.e. the means for creating atmospheric images and it is those images that are the real "publishable end product" of the whole process. The algorithms and software are just the "painters brush and easel" or the photographer's camera so to speak.

So at the very least I will continue to publish those images on a regular basis ... maybe I even find the time to setup a Web Gallery or Blog for that purpose.

[Bjorn Jonsson]

As should already be obvious I'm very interested in this so there are some things I'm curious about. From the examples posted it looks as if something like this might revolutionize the quality and resolution of my stereo derived DEMs where the low frequencey stuff is already reasonably accurate. I have already figured out how I would combine stereo DEMs and SFS DEMs.

Also, there are technical and license related obstacles : for example I am using a large collection of self written and third-party C/C++ APIs and libraries that I developed and collected over the past 15 years.
While some of them are open source, there are also commercial or closed source libraries that would have to be completely re-written first. (this task alone would be a several-years project I'm afraid)

Is this (commercial/closed source libraries) a problem because it's explicitly forbidden to distribute software based on them or is the problem licence costs or stuff like that? Vikingmars' idea (donations) is interesting and the reason I ask. There's nothing that says the software must be freeware if you make it available one day (and obviously that wouldn't work at all if library licence costs are an issue).

And most importantly, there is no such thing like a GUI or comfortable user interface. The whole project is more like an experimenting platform consisting of a complex & ugly command line interface, a cryptic shell language and about 2000 tuning parameters ...

How important this is depends on the 'type' of users. Almost all of my software is command line utilities, including the stereo software - doing a nice GUI is a waste of precious time ;-). For lots of users this might be a problem though.

Also, I am doing this all in my spare time only, as a means of recreation and relaxation ...

Now this is something familiar (and lack of time is too ;-)).

I should add that despite the questions/comments above I'm not surprised you are not interested in immediately releasing this. My stereo software is in a similar state as far is this is concerned - it works but there are some bugs and quirks I know of. For example it currently works only for images from framing cameras - using pushbroom source images would probably result in distorted DEMs (I haven't tried it though).