MARCI Images, ISIS pointers/advice/help much appreciated and desired! Options

[Drkskywxlt]

As part of my research, I'm starting to learn ISIS and display MARCI imagery. I've quickly learned that MARCI imagery is pretty hard to work with since it's a push-frame camera, which a few people claim is the reason there's a relative lack of papers published using its data. I tried to run through the process for an entire pole-to-pole MARCI swath and after processing cam2map for nearly 48hrs, the Linux box booted me off. That's on 8GB of RAM, BTW. After that, I started cropping the images to much smaller chunks. Below is my first recognizably Martian image that I've been able to produce (I've only been at this for a few days). This is from the southern mid-latitude highlands. Some things I don't understand...why is the left part of the frame somewhat washed out/bright? The red/yellow bands around the edge are areas where there apparently is no data in the green or blue channels...not sure why. My processing steps are:

marci2isis
spiceinit
marcical
crop
cam2map
automos

I need to figure out how to really speed this up. I want to look at complete MARCI global mosaics over at least 1 martian year if not the entire MRO mission. At this rate, that would take years . Apparently an ISIS update is coming out in the near future which will help work with LROC WAC images. LROC WAC is also pushframe, so maybe that will help with MARCI imagery as well?

Thanks in advance!

Attached thumbnail(s)

 

[stevesliva]

What requires all those cores? You were talking RAM before.

Just curious... memory hogging is far more common than core hogging.

[Drkskywxlt]

What requires all those cores? You were talking RAM before.

Just curious... memory hogging is far more common than core hogging.

To make that last image, took about 1.5 hours from start to finish and I used two different 8GB of RAM processors to complete. And that's only a tiny fraction of 1 full MARCI pole-to-pole swath! So, doing 2000 or 4000 of those images at once would let me do several days of MARCI images all at once in the same time.

Apparently, there's no systems where I am that have much more RAM than 8GB. So, you just need lots of cores doing the same thing to speed it up. ISIS is apparently not set up for true parallel-computing yet, although I guess there's been talk to make it so.

[stevesliva]

So, doing 2000 or 4000 of those images at once would let me do several days of MARCI images all at once in the same time.

Apparently, there's no systems where I am that have much more RAM than 8GB. So, you just need lots of cores doing the same thing to speed it up. ISIS is apparently not set up for true parallel-computing yet, although I guess there's been talk to make it so.

Oh, batch processing with the batch parallelized. My company's perfectly set up for that sort of thing, which is why I was curious. I can get a few hundred cores on boxes with 256GB of memory, but I'd probably get fired if I did it for images. But like you said, truly SMP or well-threaded applications are more rare. So we just pool lots of machines are throw individual jobs at them.

Interestingly enough, the World Community Grid is set up to do the same sort of thing, albeit with much smaller boxes:
http://www.worldcommunitygrid.org/about_us/viewAboutUs.do

Not sure an RFP for planetary science image processing would fly there... but I would gladly donate you my laptop's 2 cores and 2 GB RAM each night if there were an easy way to set up a client.

[Drkskywxlt]

Not sure an RFP for planetary science image processing would fly there... but I would gladly donate you my laptop's 2 cores and 2 GB RAM each night if there were an easy way to set up a client.

Very kind, thanks!