Help - Search - Members - Calendar
Full Version: When the JPG is just too big
Unmanned Spaceflight.com > Mars & Missions > Orbiters > MRO 2005
Pages: 1, 2
djellison
Like most - I'm now HiRISE obsessed - so I'm often finding myself trying to load images that are 10-20k on a side.

The large Victoria JPG - the 25k x 34k 'red' filter image I found would not load in Photoshop, Imageready or Gimp - and others have cited similar problems...I've found an answer..

www.imagemagick.org ( and for me - specifically http://www.imagemagick.org/download/binari...windows-dll.exe )

I installed that and simply doing convert red.jpg red.png - gave me a 500mb PNG of the 200 odd MB JPG which Photoshop loaded very quickly indeed.

I've tried comparing some of the smaller PNG's and JPG's from the HiRISE team and there is very very little difference - for the sanity of the HiRISE server, your own hard drive and bandwidth useage in general, my plan is to use the JPG's where possible, and if they're too big for photoshop, use Imagemagick to convert to PNG, and in some cases simple convert a subset of the image to PNG ( i.e. from pixel x,y to pixel a,b only )

Doug
helvick
Dang. smile.gif
Considering that I've spent so much time playing around with IM I'm a bit disgusted that I didn't think it would be up to the task so I didn't even try. smile.gif

Nice catch Doug.
Nix
How much ram are you using?

Nico
RNeuhaus
I have tried to open an 210 MB JPEG from the Hiroc by Photoshop, MS Imaging and WEB Browser without any success. All of them, the error msg is: It cannot be displayed, because it contains errors.

How were yours?

Rodolfo
djellison
I have 2 Gig - and I was getting errors in everything execpt just double clicking on the JPG in windows and getting the preview window.

Doug
Nix
I'm trying to download the file to see for myself. I've tried downloading it several times but something's fishy with my connection from time to time huh.gif

Nico
RNeuhaus
QUOTE (Nix @ Oct 9 2006, 02:16 PM) *
I'm trying to download the file to see for myself. I've tried downloading it several times but something's fishy with my connection from time to time huh.gif

Nico

You can solve it by using the one of the FireFox extensions which manages the resume after any communications failure. "DownThemAll!". During the downloading process, the communications has cut many times but the wonderfull product was able to resume until its completion.

Rodolfo
RNeuhaus
QUOTE (djellison @ Oct 9 2006, 02:02 PM) *
I have 2 Gig - and I was getting errors in everything execpt just double clicking on the JPG in windows and getting the preview window.

Doug

The same to me with 1 GB RAM. Maybe, the image is corrupted or it might have corrupted by a long and interrupted downloading process.

Hence, the MRO is becoming a new imaging wave by forcing us to upgrade the CPU, HDisk and RAM computer components. blink.gif

Rodolfo
djellison
No - the image file is fine -imagemagik can handle it, as can windows all on its own - just the larger apps can not manage to open it. When photoshop says it is corrupt - it is basically lying.
MarkL
At first blush, IM is a bit geeky (it is command line based after all). If you find something that works great, please post the command line you used here and a sample of the result. Thanks.
Nix
Thanks for the tip Rodolfo smile.gif

I'm using 4 gig (3.6 recognized) and I get this message in PS(see attachment below)

A very basic python script to convert to .png;

Just install python (via http://www.python.org) and PIL (Python Imaging Library). The file you choose to use in the script must be in the same directory as the script itself..

download by 'save target as'. -> http://www.awalkonmars.com/jpg2png.py

Nico

edit; perhaps a bit geeky too...
helvick
Here's some discussion on using ImageMagick to process HiRise IMG's

For my part I've thrown this at the monster:
convert TRA_000873_1780_RED.jpg -crop 4096x4096 +repage Victoria-%d.png
Just to see if it does give me 50 or so images. I don't expect it to finish anytime soon as I'm running it on my laptop.

I think it might be smarter to kick my girlfriend off the desktop with the Raptor as ImageMagick doesn't seem to care so much about RAM but really seems to like having a fast hard drive.
djellison
Assuming the girlfriend is typical of internet usage for females - she's using Ebay and emailing 'men are useless because....' emails to other like minded females..

She doesn't need the raptor smile.gif

Doug
David
QUOTE (djellison @ Oct 9 2006, 10:11 PM) *
Assuming the girlfriend is typical of internet usage for females - she's using Ebay and emailing 'men are useless because....' emails to other like minded females..


I understand that most of the people who post to this board are something over 90% male, and of course it's Doug's board and he can post whatever he likes. But I feel obliged to note that a remark like that is likely to make women who do read this board feel somewhat unwelcome. sad.gif
helvick
QUOTE (djellison @ Oct 9 2006, 11:11 PM) *
She doesn't need the raptor smile.gif

She would disagree, I think, being the one who generally does the hard core image processing.

Anyway for the curious this exercise took slightly more than 2.5 hours to complete on an IBM T43 (1.8Ghz Pentium M (Dothan), 1GB RAM, 5400rpm laptop drive). I'm going to have to run it on the Raptor equipped desktop now to see how much more suitable that is for this sort of thing.

For those that might be interested I was intrigued to see that the temporary files that ImageMagick created to deal with this were quite frankly enormous. The initial step appears to be that the jpg is expanded out into an internal flat image format that uses 10 bytes per pixel - this results in an 8.7GB "image" file that is then subdivided into slices of the requested size and those are finally re-encoded into the requested format (PNG) in this case. This one command used more than 17GB of temp file space and I think somewhere around 30GB of overall disk I/O. That's a serious test of your disk subsystem.
djellison
David - My other half is member #2 of this place and found it funny, and with respect, most of the women I know who use the internet would have the intelligence to take what I said in the way it was intended.

No offense intended and I find it very VERY hard to believe that any of the women who I know visit this place would find it anything other than mild chuckle fodder.
Doug
Tesheiner
QUOTE (djellison @ Oct 9 2006, 07:17 PM) *
The large Victoria JPG - the 25k x 34k 'red' filter image I found would not load in Photoshop, Imageready or Gimp - and others have cited similar problems...I've found an answer..


I talked before (on another thread) about my problems with IrfanView and Serif-PhotoPlus. But actually the only "problem" with IrfanView is that it takes some time to load the jpg, once the monster is loaded the tool *can* handle it.
Indian3000
Paint Shop Pro 7 work really fast with this huge JPG biggrin.gif, but Paint Shop Pro X are very slow sad.gif
NoVi
Mac dual G5 1.8Ghz, 2 Gb internal memory and Photoshop CS works great smile.gif
ElkGroveDan
This discussion kind of reminds me of when Windows 95 was first released. Everyone everywhere was looking to add memory. The used computer parts store where I shopped in those days was always out. I can remember lining up somewhere that was advertising 4Meg for $100. I couldn't believe how cheap memory was getting!

Fast-forward to last week and here we all are looking to add performance to our machines with the release of the HiRISE images. Good thing it's a small universe of Mars addicts. Who knows what this would do to the electronic components market if someone in every household was downloading the images. I added another Gig for about $60, and then dropped a spec sheet for a new system on my wife's desk for her Christmas shopping.
CosmicRocker
For what it's worth, nothing I had currently installed could open it, other than Paint Shop Pro 7. Photoshop 6 barfed on it, as did several other programs. I have 2 gigs of memory, but it seemed to only require about 3/4 gig with Paint Shop. I don't understand why it should need much more than 200 megs, but I don't care now that I can finally see it.

Once I got it up though, Whoah! That was even better than the large, colored version. Now, I have a whole new set of things to look at.
helvick
QUOTE (CosmicRocker @ Oct 11 2006, 06:44 AM) *
I don't understand why it should need much more than 200 megs, but I don't care now that I can finally see it.

It's an 870Megapixel image - the jpg representation of it is compressed. In order to display it or navigate around it the program has to uncompress at the very least the component you are viewing. For a grayscale image like this an app could get away with 1byte/pixel but most apps will store it using at least 4bytes/pixel and as I noted earlier ImageMagick's internal image working format appears to need 10bytes per pixel. You don't have to keep all of that in RAM but if you can it will be much,much faster.
edstrick
ACDSee Classic (strongly recommended as a image brouser, it's version 2.43 or so) and ACDSee 5.0 open the file just fine in my machine with (I think) 1/2 gig memory, but when you try to zoom into it stepwise, when you hit the memory limit, it goes black-screen and won't let you back out one step.

Classic may have an absolute file size limit... there's things I can't zoom on all the way in it that ACDSee 5 can. I don't like the later ACDSee's beyond Classic as they're like 200 blade swiss army knives.

(Win 9x and XP etc)
Oersted
Exactly: ACDSee was best around 2.4.

Later versions actually anti-alias the pixels without giving you a no-thanks option! - And then there are the gazillion unwanted features and extremely cluttered interface... yuck. Bloatware indeed.
tuvas
I myself bought an extra gig of RAM for the images, but don't worry, it won't be too long that we'll all have to do that. Soon the HiRISE team will make it a bit easier.
edstrick
Yeah.. they'll stitch adjacent images together in 2 x 2 and 4 x 4 arrays and we'll have multigigapixel maps.
tuvas
QUOTE (edstrick @ Oct 17 2006, 04:46 AM) *
Yeah.. they'll stitch adjacent images together in 2 x 2 and 4 x 4 arrays and we'll have multigigapixel maps.


Actually, something somewhat better than that, but it's fairly new technology, and as such, well, it's still coming... It'll work over JPIP, the JPEG 2000 Internet Protocol.
DEChengst
QUOTE (tuvas @ Oct 16 2006, 07:36 PM) *
I myself bought an extra gig of RAM for the images, but don't worry, it won't be too long that we'll all have to do that.


Are you sure about that ? Because no matter how the image is compressed for viewing your computer still has to decompress it to 3 bytes per pixel. Ofcourse smart software could only decompress the part of the image you're looking at, but that may give a big performance hit while scrolling through the image. Anyone knows how actual image viewers handle this ?
Tesheiner
Remember that color is only available on the center part of the images; the BIG 25k x 34k images are B&W only.

In any case I've just opened the picture covering Victoria Crater (B&W) using IrfanView and it took about one minute to load. Once loaded, it was using about 850MB of memory and scrolling was smooth.

Edited: This machine has a Pentium IV @ 2.4GHz and 1GB RAM.
DEChengst
You're correct about the B&W images. Those only take one byte per pixel. Uncompressed an 1 byte per pixel, 25k x 34k image would take up 850 MB of memory. So it seems that Irfanview decompresses the entire image into memory before displaying it.

I don't really like webbased viewers like Zoomify. Ofcourse it saves on memory and bandwith, but nothing beats the ease of use of downloading the entire image, so you can scroll and zoom it with a good image viewer on your own PC.
djellison
Anyone figured out an Imagmagik command line to take one of the huge jpgs and just slice it into say 2000 x 2000 pixel JPG chunks?

Doug
tuvas
HiView will only download the part of the image you are looking at. It is expected to be able to run on a PC with dial-up connection speeds reasonably fast. There is a program that is capable of doing this, look at Kakadu, download the program called kdu_show there, and try out their demo applications. HiView will use the same protocol as this to view images, and be about the same speed. It will allow the full 14 bits to be viewable by anyone with an internet connection, the data will be reversable, ei, no loss in compression. It will be quite neat:-)
helvick
QUOTE (djellison @ Oct 17 2006, 04:56 PM) *
Anyone figured out an Imagmagik command line to take one of the huge jpgs and just slice it into say 2000 x 2000 pixel JPG chunks?

You're not reading the thread Doug. smile.gif I posted this earlier:
convert TRA_000873_1780_RED.jpg -crop 4096x4096 +repage Victoria-%d.png
Obviously you want to change the 4096x4096 with 2000x2000. Took more than an hour on all of my systems, some tweaking of performance switches might help and running on the Raptor just about doubled the speed when compared to running on a 5400rpm drive. No surprise there.
Oh and remember to have enough space - the full Victoria image needed a peak of ~19GB working space for that command.
Sunspot
QUOTE (tuvas @ Oct 17 2006, 05:22 PM) *
HiView will only download the part of the image you are looking at. It is expected to be able to run on a PC with dial-up connection speeds reasonably fast.



What Operating Systems will this HiView program work with? smile.gif
tuvas
QUOTE (Sunspot @ Oct 17 2006, 10:23 AM) *
What Operating Systems will this HiView program work with? smile.gif


All, it's a Java-based application. So... It's not an applet, mind you, but an application, so it'll run on any system that supports Java, which is essentially anything.
RichardLeis
I cannot wait for HiView. We use ISIS qview for visual image validation and my eyes have gone cross and my patience has gone lost trying to pan through the observations. Thank goodness they are beautiful! When I have gone blind I will always remember the beauty...
dilo
Sorry if slightly OT, but this should be the new image size record (8.6 gigapixel!)
JTN
(from the PSP thread, since this is clearly the better place for image viewing nargery)
QUOTE (tuvas @ Oct 24 2006, 12:35 AM) *
QUOTE (Sunspot @ Oct 24 2006, 12:32 AM) *
Will the big black and white image of Victoria/Erebus be going up on the "zoomify" website too?

Don't know, we actually don't run that, JPL does. We're working on HiView, which will ultimately solve the problem.

I hope that someone will continue to provide new images via Zoomify even once HiView is up and running. For casual browsing from random clients, you're not going to beat Flash.

Does anyone know if JPL plan to continue providing this service?
Sunspot
QUOTE (JTN @ Nov 25 2006, 04:06 PM) *
(from the PSP thread, since this is clearly the better place for image viewing nargery)

Don't know, we actually don't run that, JPL does. We're working on HiView, which will ultimately solve the problem.
I hope that someone will continue to provide new images via Zoomify even once HiView is up and running. For casual browsing from random clients, you're not going to beat Flash.

Does anyone know if JPL plan to continue providing this service?


We still don't know what this HiView software will do, but we really need something like the zoomify feature, it's just not practical or even possible to view many of the HiRISE images otherwise. Some of the images are nearly 800MB, not many people are going to be able to download that and even fewer own a computer capable of displaying it.
JTN
QUOTE (Sunspot @ Nov 25 2006, 04:55 PM) *
We still don't know what this HiView software will do, but we really need something like the zoomify feature, it's just not practical or even possible to view many of the HiRISE images otherwise. Some of the images are nearly 800MB, not many people are going to be able to download that and even fewer own a computer capable of displaying it.

I think tuvas et al have made it clear that HiView will support selective download à la Zoomify, using JPIP. It also sounds like HiView, unlike Zoomify, will allow you to get every last drop out of the data (full 14-bit depth and so on), and thus will probably be the viewer of choice for enthusiasts (if RichardLeis' salivation is anything to go by wink.gif ).

However, the necessity of downloading and installing a Java application (and in many cases also Java itself) will deter casual viewing, e.g. by the wider public. IME, Flash is pretty ubiquitous now, and all you have to do for Zoomify is visit a web page; Java installations are not ubiquitous, and a separate application is not as immediately convenient.

(I admit a personal bias: I don't have an up-to-date Java installation, I'm unsure whether I could even get one for my fairly old OS, and I'm unsure whether HiView will run well on my slightly old hardware, whereas I know Zoomify is usable for me.)

So I'd like Zoomify as a cheap-and-cheerful alternative.
RichardLeis
From http://www.aware.com/products/compression/jpip.html

QUOTE
"JPIP is a client/server communication protocol defined in Part 9 of the JPEG2000 suite of standards, officially entitled “Interactivity Tools, APIs and Protocols.” JPIP enables a server to transmit only those portions of a JPEG2000 image that are applicable to the client’s needs. In conjunction with either HTTP or UDP protocols, JPIP enables the client to access metadata or other contents from the image file. This capability results in a vast improvement in bandwidth efficiency and speed when performing some very important and valuable image viewing tasks in a client/server environment, while reducing the storage and processing requirements of the client. The larger the images—and the more constrained the bandwidth between the client and server—the greater the benefit of JPIP."


Zoomify and LizardTech's ExpressView are temporary solutions while work continues on HiView. I do not know much about HiView (Java, browser plug-in?) or continued Zoomify support.

Our standard and PDS release products will be in JPEG2000 format (plus an attached meta label) and Zoomify does not currently support JPEG2000 - it uses a high resolution JPEG instead.

Everything related to images, captions, JPEG2000, and HiView is currently in flux because of new hires, a new website design, new software testing, and continued development. We'll try to keep everyone posted as best we can smile.gif
Sunspot
Anyway you could get the images that are perhaps more interesting to the general public such as the Large/fullsize BW Victoria Crater images and eventual image of the Spirit landing site onto the Zoomify page? biggrin.gif biggrin.gif
MarkL
QUOTE (RichardLeis @ Nov 26 2006, 07:43 PM) *
We'll try to keep everyone posted as best we can smile.gif

We are all waiting with bated breath Richard!
RichardLeis
QUOTE (Sunspot @ Nov 27 2006, 03:31 AM) *
Anyway you could get the images that are perhaps more interesting to the general public such as the Large/fullsize BW Victoria Crater images and eventual image of the Spirit landing site onto the Zoomify page? biggrin.gif biggrin.gif


Unfortunately, I don't think Ames is Zoomifying any more images. That was only during transition imaging to put together something quickly. Now that we are in PSP, priorities are getting HiView and HiWeb released and trying to speed up our captioning process. We are not holding back images because we are heartless smile.gif The torrent of images released should start soon.
djellison
In the meantime - get OpenEV
http://openev.sourceforge.net/

Suggested by Trent Hare via Emily - and I must admit - it makes viewing HiRISE images feasable on this machine (1.7G Centrino with 2Gig of Ram)....whereas before it was virtually impossible.

Doug
Borek
Does it support JPEG2000? I tried to use binary from http://fwtools.maptools.org/, but I only get core dump when I try to open JP2 picture.

Borek
djellison
QUOTE (Borek @ Nov 30 2006, 11:11 AM) *
Does it support JPEG2000?


Yes - get one of these - http://openev.sourceforge.net/index.php?page=download and try again.

Doug
Borek
Hmm, something is wrong:
ERROR 4: `/home/borek/PSP_001414_1780_RED.jp2' not recognised as a supported file format.
MarkL
I installed OpenEV. No success viewing the JP2s. The version I found was 1.8 from 2004. The UI is very sparse and you cannot drag and drop. Download directory repeatedly defaults to the program's home directory each time you open a new file. It doesn't seem to understand the role of the Desktop folder in WinXP. I had to run a CMD batch file to get it to start and it seems to be written in Python. All of which forgivable if it worked.

If anyone has had a better experience, please let me know how you got it working. Other image viewer suggestions welcome. Irfanview chokes on any JP2 over 200MB on my machine. Many THanks.
djellison
Ahh - I had that same problem borek - did you actually let it install fully....I was getting that, then when i closed everything else that I was running - there was a further install that had to happen that I did not see and did not have an entry on my taskbar. Once I'd finished that - it opened them up no problem at all.

Doug
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2013 Invision Power Services, Inc.