IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
PDS data format?
Yadgar
post Jan 21 2015, 02:49 AM
Post #1


Newbie
*

Group: Members
Posts: 14
Joined: 6-April 11
From: Cologne, Germany
Member No.: 5951



Hi(gh)!

I would like to generate POV-Ray meshes from PDS altimetry data files (for example the global Vesta file from Dawn Public Data) - but I have no clue what data format they contain... obviously, its binary, simple cylindrical projection, based on a 255 x 255 x 255 kms sphere (Vesta), 32 bits per data point. But what data format - int or float? Signed or unsigned? Big or little endian? Where do I get detailed information on the format of these PDF files?

See you in Khyberspace!

Yadgar
Go to the top of the page
 
+Quote Post
djellison
post Jan 21 2015, 02:56 AM
Post #2


Founder
****

Group: Chairman
Posts: 14431
Joined: 8-February 04
Member No.: 1



Probably better to link to a web-page rather than directly to a 500+,mb .pds file.
http://dawndata.igpp.ucla.edu/tw.jsp?secti...Ms/DLR_HAMO_DTM

Simply opening the file in a text editor reveals.....
CODE
PDS_VERSION_ID                    = PDS3

/* FILE DATA ELEMENTS */

RECORD_TYPE                       = FIXED_LENGTH
RECORD_BYTES                      = 69124
FILE_RECORDS                      = 8644
LABEL_RECORDS                     = 2

/* POINTERS TO DATA OBJECTS */

^IMAGE_HEADER                     = 3
^IMAGE                            = 4

/* DATA OBJECT DEFINITIONS */

OBJECT                            = IMAGE
  INTERCHANGE_FORMAT              = BINARY
  LINES                           = 8641
  LINE_SAMPLES                    = 17281
  SAMPLE_TYPE                     = IEEE_REAL
  SAMPLE_BITS                     = 32
  BANDS                           = 1
  BAND_STORAGE_TYPE               = BAND_SEQUENTIAL
  MINIMUM                         = -21527
  MAXIMUM                         = 19005
END_OBJECT                        = IMAGE

/* MAP OBJECT DEFINITIONS */

OBJECT                            = IMAGE_MAP_PROJECTION
  A_AXIS_RADIUS                   = 255.0
  B_AXIS_RADIUS                   = 255.0
  C_AXIS_RADIUS                   = 255.0
  CENTER_LATITUDE                 = 0.0
  CENTER_LONGITUDE                = 180.0
  COORDINATE_SYSTEM_NAME          = PLANETOCENTRIC
  EASTERNMOST_LONGITUDE           = 359.9999991462263
  LINE_PROJECTION_OFFSET          = 4320.0
  MAP_PROJECTION_TYPE             = SIMPLE_CYLINDRICAL
  MAP_RESOLUTION                  = 48.00000026515721
  MAP_SCALE                       = 0.09272061600000001 <km/pixel>
  MAXIMUM_LATITUDE                = 90.0
  MINIMUM_LATITUDE                = -90.0
  POSITIVE_LONGITUDE_DIRECTION    = EAST
  SAMPLE_PROJECTION_OFFSET        = 8640.0
  WESTERNMOST_LONGITUDE           = 0.0
END_OBJECT                        = IMAGE_MAP_PROJECTION

/* MISCELLANEOUS */

TARGET_NAME                       = VESTA
PRODUCER_INSTITUTION_NAME         = "German Aerospace Center (DLR)"
PRODUCT_CREATION_TIME             = 2013-05-24T13:27:30.000
MISSING_CONSTANT                  = -32768
DESCRIPTION                       = "DTM: Height [m] equals DN. Vertical
                                     reference: Height above
                                     sphere/ellipsoid (as defined by map
                                     axis)."

/* IMAGE HEADER DATA ELEMENTS */

OBJECT                         = IMAGE_HEADER
HEADER_TYPE                   = VICAR2
INTERCHANGE_FORMAT            = ASCII
BYTES                         = 69124
^DESCRIPTION                  = "VICAR2.TXT"
END_OBJECT                     = IMAGE_HEADER
END



That data is enough to open it as a raw file in Photoshop, Gimp or ImageJ etc. and then use it as a displacement map on a mesh.
Go to the top of the page
 
+Quote Post
Yadgar
post Jan 21 2015, 03:54 AM
Post #3


Newbie
*

Group: Members
Posts: 14
Joined: 6-April 11
From: Cologne, Germany
Member No.: 5951



QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
INTERCHANGE_FORMAT = BINARY


O.k., it's binary...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
LINES = 8641
LINE_SAMPLES = 17281


...17281 by 8641 sample points...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
SAMPLE_BITS = 32


...4 bytes per sample, which would be either standard-precision float or int...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
MINIMUM = -21527
MAXIMUM = 19005


Obviously, the sample values range from -21527 to 19005 metres relative to the rotation ellipsoid - which means that they are in fact integer! But why do they use signed int rather than signed short int? And which bit order - big oder small endian?

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
A_AXIS_RADIUS = 255.0
B_AXIS_RADIUS = 255.0
C_AXIS_RADIUS = 255.0


...the underlying ellipsoid, here a perfect sphere...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
MAP_PROJECTION_TYPE = SIMPLE_CYLINDRICAL


...so spacing of points can be calculated from the x and y dimensions of the data array...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
SAMPLE_PROJECTION_OFFSET = 8640.0


Is this a value to be added to or subtracted from the sample values?

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
MISSING_CONSTANT = -32768


Missing data code...

QUOTE (djellison @ Jan 21 2015, 03:56 AM) *
That data is enough to open it as a raw file in Photoshop, Gimp or ImageJ etc. and then use it as a displacement map on a mesh.


When I try to open it, I just get a strange noisy thing which, when zoomed in, appears like vague rastered interference shapes...
perhaps I better should try writing a C++ program which extracts the sample data and stores them as an uncompressed TGA image file!

See you in Khyberspace!

Yadgar
Go to the top of the page
 
+Quote Post
djellison
post Jan 21 2015, 05:40 AM
Post #4


Founder
****

Group: Chairman
Posts: 14431
Joined: 8-February 04
Member No.: 1



You're probably not culling the header before reading the image. I just tried it and it loads fine in Photoshop as a 32 bit file when you tell it the header is 69124 bytes. One needs to apply a stretch, obviously, to get an image with visible contrast. Moreover you may be confusing the line and the sample. Lines = height. Samples = width.
Go to the top of the page
 
+Quote Post
mcaplinger
post Jan 21 2015, 05:59 AM
Post #5


Senior Member
****

Group: Members
Posts: 2504
Joined: 13-September 05
Member No.: 497



QUOTE (Yadgar @ Jan 20 2015, 08:54 PM) *
...4 bytes per sample, which would be either standard-precision float or int...
Obviously, the sample values range from -21527 to 19005 metres relative to the rotation ellipsoid - which means that they are in fact integer!

Nope. It says SAMPLE_TYPE = IEEE_REAL, that's a single-precision float.


--------------------
Disclaimer: This post is based on public information only. Any opinions are my own.
Go to the top of the page
 
+Quote Post
Bjorn Jonsson
post Jan 21 2015, 11:05 AM
Post #6


IMG to PNG GOD
****

Group: Moderator
Posts: 2250
Joined: 19-February 04
From: Near fire and ice
Member No.: 38



It should be possible to use IMG2PNG to convert this to a 16 bit PNG image if that helps. See http://www.mmedia.is/bjj/utils/img2png/index.html
Go to the top of the page
 
+Quote Post
Yadgar
post Jan 27 2015, 03:45 AM
Post #7


Newbie
*

Group: Members
Posts: 14
Joined: 6-April 11
From: Cologne, Germany
Member No.: 5951



Hi(gh)!

QUOTE (mcaplinger @ Jan 21 2015, 06:59 AM) *
Nope. It says SAMPLE_TYPE = IEEE_REAL, that's a single-precision float.


I started with a small C++ program (using g++ under Linux) to at least get a sense of the data stored in global.pds - but the data I got were far beyond the limits given in the header!

Here is the code:

#include <cstdlib>
#include <fstream>
#include <iostream>

using namespace std;

int main()
{
ifstream Rawdata;
Rawdata.open("global.pds", ios::binary);

if (!Rawdata)
{
cerr << "File cannot be read!\n";
exit(-1);
}

float h;
int i=0;

while (Rawdata.read((char*)&h, sizeof(h)))
{
i++;
if (i > 2585) // skipping the ASCII header
cout << i << "\t" << h << endl;
}
Rawdata.close();
}

and this is an example of what it read from global.pds:

592609 1.08726e-38
592610 1.08726e-38
592611 -1.07807e-38
592612 -1.07807e-38
592613 -1.07807e-38
592614 -1.07807e-38
592615 1.07807e-38
592616 1.07807e-38
592617 -1.06889e-38
592618 -1.06889e-38
592619 1.06889e-38
592620 1.06889e-38
592621 -1.05971e-38
592622 -1.05971e-38
592623 1.05971e-38
592624 1.05971e-38
592625 -1.05052e-38
592626 -1.05052e-38
592627 1.05052e-38
592628 1.05052e-38
592629 -1.04134e-38
592630 -1.04134e-38
592631 1.04134e-38
592632 1.04134e-38

Clueless,

Yadgar
Go to the top of the page
 
+Quote Post
mcaplinger
post Jan 27 2015, 07:09 AM
Post #8


Senior Member
****

Group: Members
Posts: 2504
Joined: 13-September 05
Member No.: 497



Most likely a byte-ordering problem. PDS files are usually in network order. I'd take Bjorn's advice and not reinvent the wheel.


--------------------
Disclaimer: This post is based on public information only. Any opinions are my own.
Go to the top of the page
 
+Quote Post
Yadgar
post Jan 27 2015, 09:03 PM
Post #9


Newbie
*

Group: Members
Posts: 14
Joined: 6-April 11
From: Cologne, Germany
Member No.: 5951



QUOTE (mcaplinger @ Jan 27 2015, 08:09 AM) *
Most likely a byte-ordering problem. PDS files are usually in network order. I'd take Bjorn's advice and not reinvent the wheel.


I tried a small routine programmed by Gregor Brandt (http://stackoverflow.com/questions/2782725/converting-float-values-from-big-endian-to-little-endian):

float ReverseFloat( const float inFloat )
{
float retVal;
char *floatToConvert = ( char* ) & inFloat;
char *returnFloat = ( char* ) & retVal;

// swap the bytes into a temporary buffer
returnFloat[0] = floatToConvert[3];
returnFloat[1] = floatToConvert[2];
returnFloat[2] = floatToConvert[1];
returnFloat[3] = floatToConvert[0];

return retVal;
}

- and I got believable values, such as:

9681062 229914
9681063 229918
9681064 229924
9681065 229928
9681066 229934
9681067 229938

Obviously, the underlying basic unit is metres rather than kilometres...

See you in Khyberspace!

Yadgar
Go to the top of the page
 
+Quote Post
mcaplinger
post Jan 27 2015, 09:32 PM
Post #10


Senior Member
****

Group: Members
Posts: 2504
Joined: 13-September 05
Member No.: 497



QUOTE (Yadgar @ Jan 27 2015, 02:03 PM) *
Obviously, the underlying basic unit is metres rather than kilometres...

Like it says in the header: "DTM: Height [m] equals DN."


--------------------
Disclaimer: This post is based on public information only. Any opinions are my own.
Go to the top of the page
 
+Quote Post
siravan
post Jan 28 2015, 12:42 AM
Post #11


Member
***

Group: Members
Posts: 128
Joined: 10-December 06
From: Atlanta
Member No.: 1472



If you really need to do it in C/C++, I can send you the source of a converter I wrote near 10 years ago that can handle it and converts it to FITS format. But this is generally not the optimal solution and it is better to use img2png (although as far as I know, Bjorn has yet to port it to Linux) or just read the raw data using GIMP or ImageJ as pointed above.
Go to the top of the page
 
+Quote Post
Yadgar
post Jan 29 2015, 08:40 AM
Post #12


Newbie
*

Group: Members
Posts: 14
Joined: 6-April 11
From: Cologne, Germany
Member No.: 5951



Hi(gh)!

QUOTE (siravan @ Jan 28 2015, 01:42 AM) *
If you really need to do it in C/C++, I can send you the source of a converter I wrote near 10 years ago that can handle it and converts it to FITS format. But this is generally not the optimal solution and it is better to use img2png (although as far as I know, Bjorn has yet to port it to Linux) or just read the raw data using GIMP or ImageJ as pointed above.


I intend to generate an ASCII POV-Ray mesh2 script from the PDS data - with my current amount of RAM (16 GiB) I will need to drastically scale down the data resolution, such as picking only each tenth data point from each tenth row, to get 1.5 million rather than 150 million vertices... up to now, it seems to work well, but nevertheless, thanks for your readiness to help!

See you in Khyberspace!

Yadgar
Go to the top of the page
 
+Quote Post
JohnVV
post Apr 7 2015, 07:39 AM
Post #13


Member
***

Group: Members
Posts: 890
Joined: 18-November 08
Member No.: 4489



Are you still having issues ?

for the DEMS like the mola and lola the pds img files are 100% raw
no attached header
they have a detached lbl for the header

the dawn vesta data is on the older dawn site
see the post from a few days back .
http://www.unmannedspaceflight.com/index.p...st&p=219315

for NON dem files the header can be static or dynamic
as in it might be the SAME length for most in a mission or change with EACH image

most img 's in pds hav a vicar header and a detached pds lable
but not all
some have a pds attached header

if you are not using say isis3 or some other program that reads the pds format
Use a HEX editor to read the plain text header on the binary image

some text editors can even read the plain text
gedit and kate can so i would expect on windows that "SciTE " should


for future reference the link in post 2 has a isis3 cub file
"gdal_translate" has no problem converting it to a GEO tiff ( Photoshop should be able to open that )
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic

 



RSS Lo-Fi Version Time is now: 29th March 2024 - 02:23 PM
RULES AND GUIDELINES
Please read the Forum Rules and Guidelines before posting.

IMAGE COPYRIGHT
Images posted on UnmannedSpaceflight.com may be copyrighted. Do not reproduce without permission. Read here for further information on space images and copyright.

OPINIONS AND MODERATION
Opinions expressed on UnmannedSpaceflight.com are those of the individual posters and do not necessarily reflect the opinions of UnmannedSpaceflight.com or The Planetary Society. The all-volunteer UnmannedSpaceflight.com moderation team is wholly independent of The Planetary Society. The Planetary Society has no influence over decisions made by the UnmannedSpaceflight.com moderators.
SUPPORT THE FORUM
Unmannedspaceflight.com is funded by the Planetary Society. Please consider supporting our work and many other projects by donating to the Society or becoming a member.