IPB

Welcome Guest ( Log In | Register )

5 Pages V  < 1 2 3 4 > »   
Reply to this topicStart new topic
Voyager 2 Saturn Revisited, Still a lot to be processed and reprocessed
Bjorn Jonsson
post Jun 4 2007, 04:27 PM
Post #16


IMG to PNG GOD
****

Group: Moderator
Posts: 1482
Joined: 19-February 04
From: Near fire and ice
Member No.: 38



QUOTE (elakdawalla @ Jun 4 2007, 04:17 PM) *
I tried going to that Zip archives page but still have the same problem with wget, an "Abnormal program termination." It looks like the files have negative file sizes!

--Emily

I managed to dig up an URL I couldn't remember when I posted the previous message:

http://pdssbn.astro.umd.edu/nodehtml/large_files.html

This was associated with the (now removed) NH data and has lots of useful info.

I'm starting to suspect these problems might be associated with the fact that these particular files (34-38) are bigger than 2.1 GB (the Voyager 1 files you successfully downloaded are not).

EDIT: Despite this I managed to download volumes 34-38 but I wasn't using Wget.
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 4 2007, 04:32 PM
Post #17


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



It could be that large file sizes are creating a problem somewhere. However, I know that wget works to download large files; it's what I used to download the New Horizons LORRI data file (nearly 6 GB!) so if there is a large-file-size problem I don't believe it's on my end. That page you pointed to on the Small Bodies Node says: "We have ensured that we are serving our large files through a specially-built HTTP server that contains large file support, so that we can at least guarantee that we can pump the files out." I wonder if the Rings Node HTTP server has been configured for large file support.

What did you use to download the files?

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
Bjorn Jonsson
post Jun 4 2007, 04:39 PM
Post #18


IMG to PNG GOD
****

Group: Moderator
Posts: 1482
Joined: 19-February 04
From: Near fire and ice
Member No.: 38



I used GetRight for most/all of them. I don't remember exactly what I did but if I remember correctly I downloaded ZIP files and not TAR/GZ files - I vaguely remember hacking the URL. The files got a bit corrupt but WinZIP still managed to unzip them - about 5 images per file got lost and I then downloaded these images manually. In contrast, if you get a corrupt GZ/TAR files you lose lots of images, probably all of them.

I strongly suspect this to be a problem with the Rings Node HTTP server large file support.
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 4 2007, 08:15 PM
Post #19


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



Bjorn, your suspicion turns out to be right on target. I sent a followup email and I've now heard back from Mark Showalter at the Rings Node, who says they hadn't recognized there was a problem with serving files of this size before, and that they will either try to implement the solution used by the Small Bodies node, or start splitting the archives into smaller pieces to stay under the 2 GB limit.

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
brellis
post Jun 4 2007, 08:47 PM
Post #20


Member
***

Group: Members
Posts: 700
Joined: 9-February 07
Member No.: 1700



QUOTE (elakdawalla @ Jun 4 2007, 01:15 PM) *
... they will either try to implement the solution used by the Small Bodies node, or start splitting the archives into smaller pieces to stay under the 2 GB limit.

--Emily


Ironic that handling these old images would strain the size capacity of 21st C bundling tongue.gif
Go to the top of the page
 
+Quote Post
Bjorn Jonsson
post Jun 5 2007, 12:45 AM
Post #21


IMG to PNG GOD
****

Group: Moderator
Posts: 1482
Joined: 19-February 04
From: Near fire and ice
Member No.: 38



QUOTE (elakdawalla @ Jun 4 2007, 08:15 PM) *
Bjorn, your suspicion turns out to be right on target. I sent a followup email and I've now heard back from Mark Showalter at the Rings Node, who says they hadn't recognized there was a problem with serving files of this size before, and that they will either try to implement the solution used by the Small Bodies node, or start splitting the archives into smaller pieces to stay under the 2 GB limit.

Great - should make these files much easier to download. In addition to the corruption I have mentioned this problem also manifests itself in an inability to resume a failed download - a feature you really want when downloading monster files like these. I look forward to this getting fixed.
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 21 2007, 08:58 PM
Post #22


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



I just heard again from Mark Showalter, who says they've updated their Web server, which should have solved the problem. I'll post again when I've had a chance to try it out. Bjorn -- have at it.

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 26 2007, 04:26 PM
Post #23


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



All right, it's all downloaded, and I've now posted all the rings and moons images. As for Saturn -- there's way too many to post. I'll have to consider what to do with those. I'll probably post a subset of them at some point, but I think I may wait until they've finished going through peer review. Apparently there's still some pretty serious problems with their dark current subtraction, which results in a goodly fraction of the images being badly calibrated, so they have to work on that.

http://planetary.org/explore/topics/voyager/rawdata.html

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
Guest_AlexBlackwell_*
post Jun 26 2007, 06:26 PM
Post #24





Guests






I just wanted to say that your space imaging effort is a pretty cool service to the public.
Go to the top of the page
 
+Quote Post
ChrisP
post Jun 26 2007, 10:51 PM
Post #25


Newbie
*

Group: Members
Posts: 19
Joined: 17-June 07
From: Near Southampton, UK
Member No.: 2430



One minor nit on your new raw images page Emily, the links to Mimas and Tethys are broken. sad.gif

The 1kx1k PNG images are very dark in any program that can't handle and display 16bit greyscale images. Now I think I understand why this was done, but wonder if a better balance couldn't be found between visibility and bit depth. The original dynamic range was from -32768 to 32767. How does IMG2PNG convert this to a 16 bit greyscale anyway?

Cheers,
Chris.
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 26 2007, 11:27 PM
Post #26


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



Woops. I fixed the Mimas and Tethys links.

The problem with the 16 bit images is not that your software can't handle their display, it's that the pixels all have values that cluster toward the low end of the 16-bit numbers, for the most part. I think that there was some command I could have issued to IMG2PNG to make it multiply all images by a constant, but I didn't do that. Some of the images -- the ones with bad dark current subtraction, I think -- wouldn't have worked with that option anyway.

There is free software you can use that can adjust the contrast of these images to reveal more detail. I've used and liked ImageJ; others here have recommended Gimp.

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post
ChrisP
post Jun 26 2007, 11:54 PM
Post #27


Newbie
*

Group: Members
Posts: 19
Joined: 17-June 07
From: Near Southampton, UK
Member No.: 2430



Thanks for fixing the links.

The GIMP doesn't handle 16 bit colour planes (yet?) and the image I looked at using GIMP had a maximum greyscale value of 64 out of 256. I guess passing the images through a series of netpbm filters could fix them to be 8bit. Would you consider adding netpbm to your new image programs page? It really is so useful.

Cheers,
Chris.
Go to the top of the page
 
+Quote Post
Bjorn Jonsson
post Jun 27 2007, 12:45 AM
Post #28


IMG to PNG GOD
****

Group: Moderator
Posts: 1482
Joined: 19-February 04
From: Near fire and ice
Member No.: 38



QUOTE (elakdawalla @ Jun 26 2007, 04:26 PM) *
All right, it's all downloaded, and I've now posted all the rings and moons images. As for Saturn -- there's way too many to post. I'll have to consider what to do with those. I'll probably post a subset of them at some point, but I think I may wait until they've finished going through peer review. Apparently there's still some pretty serious problems with their dark current subtraction, which results in a goodly fraction of the images being badly calibrated, so they have to work on that.

I now have downloaded every calibrated Voyager image from the PDS Rings Node smile.gif.

I don't know if it's because of dark current subtraction but I've noticed that some of the calibrated images look significantly more noisy than the raw versions and sometimes low contrast details get lost (or almost lost). This is especially true of something like the hi-res narrow angle Voyager 1 images of the unlit side of the rings (the Cassini division is a nice example). These images were dark in their original versions so maybe this is hard to avoid but I still feel that this could probably be improved a bit.

The space imaging pages should be highly useful to a lot of people interested in this stuff. I even found links to stuff I had forgotten about.

Not multiplying the intensity values with a constant when converting the images with IMG2PNG makes a lot of sense in this case in my opinion because it results in image files containing data that is identical to the original files. The resulting images are very dark but can easily be brightened.
Go to the top of the page
 
+Quote Post
tedstryk
post Jun 27 2007, 01:13 AM
Post #29


Interplanetary Dumpster Diver
****

Group: Moderator
Posts: 4214
Joined: 17-February 04
From: Powell, TN
Member No.: 33



QUOTE (Bjorn Jonsson @ Jun 27 2007, 12:45 AM) *
I now have downloaded every calibrated Voyager image from the PDS Rings Node smile.gif.

I don't know if it's because of dark current subtraction but I've noticed that some of the calibrated images look significantly more noisy than the raw versions and sometimes low contrast details get lost (or almost lost). This is especially true of something like the hi-res narrow angle Voyager 1 images of the unlit side of the rings (the Cassini division is a nice example). These images were dark in their original versions so maybe this is hard to avoid but I still feel that this could probably be improved a bit.

The space imaging pages should be highly useful to a lot of people interested in this stuff. I even found links to stuff I had forgotten about.

Not multiplying the intensity values with a constant when converting the images with IMG2PNG makes a lot of sense in this case in my opinion because it results in image files containing data that is identical to the original files. The resulting images are very dark but can easily be brightened.

I have had some real problems with background noise when trying to deal with underexposed images. The calibrated images are usually better, but many are noisier than the ones I calibrate myself.


--------------------
Go to the top of the page
 
+Quote Post
elakdawalla
post Jun 27 2007, 02:02 AM
Post #30


Bloggette par Excellence
****

Group: Admin
Posts: 4414
Joined: 4-August 05
From: Pasadena, CA, USA, Earth
Member No.: 454



Hi Chris, I'd be happy to add instructions regarding netpbm to the page -- if someone will compose some instructions describing how to use it. (Hint, hint.)

Yes, if you convert directly from 16-bit to 8-bit without adjusting contrast first, you'll find only pixel values up to (at most) 64. It's better to stretch the images while they're 16-bit, before converting to 8-bit, then you have more dynamic range to work with.

Bjorn, if there aren't links to stuff you haven't forgotten about on the space imaging pages (hope you can follow the multiple negatives there), I'd appreciate suggestions for more links!

--Emily


--------------------
My blog - @elakdawalla on Twitter - Please support unmannedspaceflight.com by donating here.
Go to the top of the page
 
+Quote Post

5 Pages V  < 1 2 3 4 > » 
Reply to this topicStart new topic

 



RSS Lo-Fi Version Time is now: 31st August 2014 - 06:15 AM
RULES AND GUIDELINES
Please read the Forum Rules and Guidelines before posting.

IMAGE COPYRIGHT
Images posted on UnmannedSpaceflight.com may be copyrighted. Do not reproduce without permission. Read here for further information on space images and copyright.

OPINIONS AND MODERATION
Opinions expressed on UnmannedSpaceflight.com are those of the individual posters and do not necessarily reflect the opinions of UnmannedSpaceflight.com or The Planetary Society. The all-volunteer UnmannedSpaceflight.com moderation team is wholly independent of The Planetary Society. The Planetary Society has no influence over decisions made by the UnmannedSpaceflight.com moderators.
SUPPORT THE FORUM
Unmannedspaceflight.com is a project of the Planetary Society and is funded by donations from visitors and members. Help keep this forum up and running by contributing here.