IPB

Welcome Guest ( Log In | Register )

25 Pages V  « < 8 9 10 11 12 > »   
Reply to this topicStart new topic
Curiosity Image Retrieval Tools, scripts and software
Stellingwerff
post Aug 26 2012, 07:20 PM
Post #136


Junior Member
**

Group: Members
Posts: 43
Joined: 7-August 12
From: The Netherlands
Member No.: 6493



They are not very consistent with the casing of the suffix. I actually ended up checking both upper and lower case while doing the head retrieval. That way I doublecheck (and repair) any incorrect filename.
Go to the top of the page
 
+Quote Post
jmknapp
post Aug 27 2012, 12:11 AM
Post #137


Senior Member
****

Group: Members
Posts: 1465
Joined: 9-February 04
From: Columbus OH USA
Member No.: 13



QUOTE (Pando @ Aug 26 2012, 02:18 PM) *
I hope it wouldn't be too much for you to change this to lowercase .jpg ...


Try it now Pando--I think it should handle either the JPG or jpg extension now, as long as it's consistent within each sol's directory for a given camera.


--------------------
Go to the top of the page
 
+Quote Post
3d_mars
post Aug 27 2012, 03:32 AM
Post #138


Newbie
*

Group: Members
Posts: 11
Joined: 12-August 12
Member No.: 6550



Here's version 2.0 of my Navcam download tool, which simplifies the process of downloading multiple left/right stereo image pairs and creating stereoscopic images. This version includes a GUI and supports anaglyph as well as SBS stereoscopic 3D images, a horizontal image offset, a function to find the latest sol number, and a few other changes and additions. A windows executable is attached and the source is below. Here are the links to StereoPhoto Maker and AutoIt. Thanks again jmknapp for http://curiositymsl.com.

After downloading the image pairs and creating the script file, run StereoPhoto Maker, select File, Multi conversion from list, select the script file, the output image type, and then click the Convert Files button.

Screenshot

CODE

; This script processes all full frame MSL Navcam images using jmknapp's http://curiositymsl.com, downloads only full frame LR pairs (if not downloaded already) and creates a StereoPhoto Maker script file
; version 2.0, 20120826
; UMSF: 3d_mars

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Outfile=navcam_download.exe
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
#include <file.au3>
#include <array.au3>

#Region ### START Koda GUI section ### Form=navcam_sbs.kxf
$Form_Navcam = GUICreate("MSL Navcam Image Download and Script Generator", 569, 400, 368, 376)
$SetFont = GUISetFont(9)

$Label_Title = GUICtrlCreateLabel("This app downloads MSL Navcam full frame stereo pairs and exports a StereoPhoto Maker script for generating stereoscopic images. NOTE: Only matching left/right image pairs will be downloaded.", 24, 8, 521, 55)

$Label_Input_Sol_Number_Text= GUICtrlCreateLabel("Input Sol", 24, 72, 86, 17, $SS_CENTERIMAGE)
$Input_Sol = GUICtrlCreateInput("1", 90, 71, 105, 21)
$Button_Prev_Sol = GUICtrlCreateButton("Prev Sol", 235, 69, 65, 25)
$Button_Next_Sol = GUICtrlCreateButton("Next Sol", 300, 69, 65, 25)
$Button_GetLatestSol = GUICtrlCreateButton("Get Latest Sol", 385, 69, 105, 25)

$Checkbox_Create_Scriptfile = GUICtrlCreateCheckbox("Create StereoPhoto Maker script file", 24, 100, 270, 17)
GUICtrlSetState($Checkbox_Create_Scriptfile, $GUI_CHECKED)
$Checkbox_download_images = GUICtrlCreateCheckbox("Download Images", 24, 123, 130, 17)
GUICtrlSetState($Checkbox_download_images, $GUI_CHECKED)

$Input_Image_Path = GUICtrlCreateInput("Path to save images", 24, 150, 420, 21)
GUICtrlSetTip($Input_Image_Path, "Input path to save images and script file")
$Button_Browse = GUICtrlCreateButton("Browse...", 450, 148, 81, 25)

$Label_hOffset = GUICtrlCreateLabel("Horizontal image offset", 24, 185, 160, 17)
$Input_hOffset = GUICtrlCreateInput("-160", 180, 185, 86, 20)

$Button_Start = GUICtrlCreateButton("Start", 24, 224, 65, 25)
$Progressbar = GUICtrlCreateProgress(24, 264, 249, 17, $WS_BORDER)

;GUICtrlCreateLabel("text", left, top, width, height)
$Label_Total_Image_Text = GUICtrlCreateLabel("Total Images:", 29, 304, 93, 17)
$Label_Total_Image_Count = GUICtrlCreateLabel("0", 139, 304, 50, 17)
$Label_Total_New_Image_Text = GUICtrlCreateLabel("Total New Images:", 29, 328, 107, 17)
$Label_Total_New_Image_Count = GUICtrlCreateLabel("0", 139, 328, 50, 17)

$Label_Status_Text = GUICtrlCreateLabel("Status:", 29, 368, 37, 17)
$Label_Status = GUICtrlCreateLabel(" ", 71, 368, 400, 17, $SS_SUNKEN)


GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

; read settings from ini
Global $sIniFile = @ScriptDir & "\navcam_sbs.ini"
;IniRead(filename,section,key,default)
GUICtrlSetData($Input_Sol, IniRead($sIniFile, "Sol", 0, "1"))
GUICtrlSetData($Input_Image_Path, IniRead($sIniFile, "Image_Path", 1, "Input save path"))
GUICtrlSetData($Input_hOffset, IniRead($sIniFile, "hOffset", 2, "-160"))

While 1
$nMsg = GUIGetMsg()
Switch $nMsg
Case $GUI_EVENT_CLOSE
_Shutdown()
Exit
Case $Button_GetLatestSol
GetLatestSol()
Case $Button_Next_Sol
Next_Sol()
Case $Button_Prev_Sol
Prev_Sol()
Case $Button_Browse
_Browse()
Case $Button_Start
_Start()
EndSwitch
WEnd

Func GetLatestSol()

$jplsite_file = @TempDir & "\jplsite.txt"

;dump site into file
$return = InetGet("http://mars.jpl.nasa.gov/msl/multimedia/raw/?s=", $jplsite_file, 1, 0)

;read file into array
Local $jplsite
If Not _FileReadToArray($jplsite_file, $jplsite) Then
MsgBox(4096, "Error", " Error importing from file to Array error:" & @error)
Exit
EndIf

While 1
;search array for latest sol number
$index = _ArraySearch($jplsite, "Total for Sol",0,0,0,1)

If $index = -1 Then
MsgBox(0,"","no match for sol number on JPL site")
ExitLoop
EndIf

$jplsite_foundline = $jplsite[$index]

ExitLoop

WEnd


$solstrt = StringInStr($jplsite_foundline,"Total for Sol")
$solend = StringInStr($jplsite_foundline,": ")

$sol_latest = StringMid($jplsite_foundline,$solstrt+14,$solend-$solstrt-14)

GUICtrlSetData($Input_Sol,$sol_latest)

EndFunc
Func Next_Sol()
$sol_num=GUICtrlRead($Input_Sol,1)
$sol_num += 1
GUICtrlSetData($Input_Sol,$sol_num)
EndFunc
Func Prev_Sol()
$sol_num=GUICtrlRead($Input_Sol,1)
$sol_num -= 1
if $sol_num < 1 then $sol_num = 1
GUICtrlSetData($Input_Sol,$sol_num)
EndFunc
Func _Browse()
$sol=GUICtrlRead($Input_Sol,1)
;choose path and file name for images and generated StereoPhoto Maker script file
Local $MyDocsFolder = "::{450D8FBA-AD25-11D0-98A8-0800361B1103}"
Local $StereoPhoto_script_file = FileSaveDialog("Choose a path and filename for images and image list:", $MyDocsFolder, "Text Files (*.txt)", 2, "navcam_sol_" & $sol & ".txt")
; option 2 = dialog remains until valid path/file selected
GUICtrlSetData($Input_Image_Path,$StereoPhoto_script_file)
EndFunc
Func _Start()

; start a loop so the function can be aborted at any point
Do

GUICtrlSetData($Label_Total_Image_Count,0)
GUICtrlSetData($Label_Total_New_Image_Count,0)
GUICtrlSetData($progressbar, 0)

$sol=GUICtrlRead($Input_Sol,1)
$hOffset=GUICtrlRead($Input_hOffset,1)

; create temp filename for listing all URLs for a given sol
; example: c:\users\username\appdata\local\temp\URLs_sol_1.txt
$URL_list_file=@TempDir & "\URLs_sol_" & $sol & ".txt"

; download image URL list to text file
; InetGet option syntax: URL, filename, 1=force reload, 0=wait for download
GUICtrlSetData($Label_Status,"Download URLs")
InetGet("http://curiositymsl.com/cgi-bin/getfiles.cgi?sol=" & $sol, $URL_list_file,1,0)

; if no URLs found then abort
$URL_list_size=FileGetSize($URL_list_file)
If $URL_list_size = 0 Then
GUICtrlSetData($Label_Status,"No URLs found")
ExitLoop
EndIf

; read URL list and import to array
; note: $allURLs[0] contains the number of records read into the array.
GUICtrlSetData($Label_Status,"Read URLs into array")
Local $allURLs
If Not _FileReadToArray($URL_list_file, $allURLs) Then
MsgBox(4096, "Error", " Error importing from file to Array error:" & @error)
;Exit
EndIf

; search for Navcam full frame (*EDR_F*) (exclude thumbnails) into new array $allframes which contains only image URLs we care about
; Navcam full frame example URLs:
; http://mars.jpl.nasa.gov/msl-raw-images/pr...AUT_04096M_.JPG
; http://mars.jpl.nasa.gov/msl-raw-images/pr...AUT_04096M_.JPG

; declare large array instead of resizing array multiple times, uses very little RAM
Local $allframes[9999]=[0]
GUICtrlSetData($Label_Status,"Search for Navcam full frame images")
$x = 1
For $URL in $allURLs
;check for Navcam full frame
If StringInStr($URL,"EDR_F") Then
$allframes[$x]=$URL
$x += 1
EndIf
Next

;record total number of frames
$allframes[0]=$x - 1

; read allframes to find matching left/right pairs
; insert into new arrays $leftframe_Nav, $rightframe_Nav

Local $leftframe_Nav[9999]=[0]
Local $rightframe_Nav[9999]=[0]
Local $leftframe_counter = 0
Local $rightframe_counter = 0

GUICtrlSetData($Label_Status,"Search for Left and Right frames")
For $x = 1 to $allframes[0]
If StringInStr($allframes[$x],"NLA_") Then
;left frame found, store in $leftframe_Nav array
$leftframe_counter += 1
$leftframe_Nav[$leftframe_counter]=$allframes[$x]
$leftframe_Nav[0]=$leftframe_counter
ElseIf StringInStr($allframes[$x],"NRA_") Then
;right frame found, store in $rightframe_Nav array
$rightframe_counter += 1
$rightframe_Nav[$rightframe_counter]=$allframes[$x]
$rightframe_Nav[0]=$rightframe_counter
EndIf
Next

; if no left full frames then abort and exit
If $leftframe_counter == 0 Then
;MsgBox(0, "", "No frames found")
GUICtrlSetData($Label_Status,"No full frames found")
EndIf


; using lists of left and right frames, find matches and save output
Local $right_matching_frame_number[9999]=[0]
Local $StereoPhoto_script_lines[9999]=[0]
Local $URL_left_temp, $left_image_file_name, $URL_right_temp, $right_image_file_name, $left_image_full_path, $right_image_full_path

GUICtrlSetData($Label_Status,"Find Left/Right frame matches")
For $x = 1 to $leftframe_Nav[0]
;extract frame number
$leftframe_image_number=StringMid($leftframe_Nav[$x],StringInStr($leftframe_Nav[$x],"NLA_")+4,9)
; returns -1 if no match found
$right_matching_frame_number[$x] = _ArraySearch($rightframe_Nav, $leftframe_image_number, 0, 0, 0, 1)

; update progress bar
Local $Percent = ($x / $leftframe_Nav[0] ) * 100
GUICtrlSetData($progressbar, $Percent)

; download left and right images
If $right_matching_frame_number[$x] <> -1 Then

; increment total image counter (+2 because left and right frames found)
GUICtrlSetData($Label_Total_Image_Count,GUICtrlRead($Label_Total_Image_Count,1) + 2)

; find the file names
$URL_left_temp = StringSplit($leftframe_Nav[$x], "/")
$left_image_file_name = $URL_left_temp[UBound($URL_left_temp)-1]
$URL_right_temp = StringSplit($rightframe_Nav[$right_matching_frame_number[$x]], "/")
$right_image_file_name = $URL_right_temp[UBound($URL_right_temp)-1]

; create full path names
$left_image_full_path=@WorkingDir & "\" & $left_image_file_name
$right_image_full_path=@WorkingDir & "\" & $right_image_file_name

; build array for StereoPhoto Maker script file
$StereoPhoto_script_lines[$x]="21,0,0," & $hOffset & ",0," & $left_image_full_path & "," & $right_image_full_path & ",,"

; download the images
; if the file already exists don't download it again
If Not FileExists($left_image_full_path) Then
; increment total new image counter
GUICtrlSetData($Label_Total_New_Image_Count,GUICtrlRead($Label_Total_New_Image_Count,1) + 1)
If GUICtrlRead($Checkbox_download_images) = $GUI_CHECKED Then InetGet($leftframe_Nav[$x],$left_image_full_path,1,0)
EndIf
If Not FileExists($right_image_full_path) Then
; increment total new image counter
GUICtrlSetData($Label_Total_New_Image_Count,GUICtrlRead($Label_Total_New_Image_Count,1) + 1)
If GUICtrlRead($Checkbox_download_images) = $GUI_CHECKED Then InetGet($rightframe_Nav[$right_matching_frame_number[$x]],$right_image_full_path,1,0)
EndIf

EndIf

Next

GUICtrlSetData($Label_Status,"Processing complete")

If GUICtrlRead($Checkbox_Create_Scriptfile) = $GUI_CHECKED Then
GUICtrlSetData($Label_Status,"Save script file")

$StereoPhoto_script_file=GUICtrlRead($Input_Image_Path,1)
; write StereoPhoto Maker script file
Local $hFile = FileOpen($StereoPhoto_script_file, 2) ; 2 = erase previous
_FileWriteFromArray($StereoPhoto_script_file, $StereoPhoto_script_lines, 1) ; 1 = index to start
FileClose($hFile)
GUICtrlSetData($Label_Status,"Script file saved")
EndIf

ExitLoop
; end function loop
Until 1
; open StereoPhoto Maker, File menu, multi conversion from list...
EndFunc
Func _Shutdown()
; save settings to ini
IniWrite($sIniFile, "Sol", 0, GUICtrlRead($Input_Sol))
IniWrite($sIniFile, "Image_Path", 1, GUICtrlRead($Input_Image_Path))
IniWrite($sIniFile, "hOffset", 2, GUICtrlRead($Input_hOffset))
EndFunc


Attached File(s)
Attached File  navcam_download.zip ( 312.94K ) Number of downloads: 280
 
Go to the top of the page
 
+Quote Post
Pando
post Aug 27 2012, 03:54 AM
Post #139


The Insider
***

Group: Members
Posts: 669
Joined: 3-May 04
Member No.: 73



QUOTE (jmknapp @ Aug 26 2012, 04:11 PM) *
Try it now Pando--I think it should handle either the JPG or jpg extension now, as long as it's consistent within each sol's directory for a given camera.


My gosh, thanks Joe! This works really well. I didn't realize they are mixing the uppercase/lowercase file extensions.
Go to the top of the page
 
+Quote Post
Stellingwerff
post Aug 28 2012, 09:48 PM
Post #140


Junior Member
**

Group: Members
Posts: 43
Joined: 7-August 12
From: The Netherlands
Member No.: 6493



Hi all,

I've done another update to http://msl-raw-images.appspot.com/lists.html
  • New, cleaner layout
  • New images are announced on Twitter: http://www.twitter.com/MSLRawImages
  • Fixed the export function
  • Many serverside enhancements for faster HTML scraping, now checking every 5 minutes.


I'm also working on an email notification, besides the twitter one. I didn't provide the needed management functions yet, to subscribe, unsubscribe, etc. But if any of you guys or girls would like to get an email notification/report on new images, please drop me a message. I'll arrange the setup then.

Greetings,
Ludo.
Go to the top of the page
 
+Quote Post
Stellingwerff
post Sep 4 2012, 08:25 PM
Post #141


Junior Member
**

Group: Members
Posts: 43
Joined: 7-August 12
From: The Netherlands
Member No.: 6493



Hi all,

Remind me to never ever say anything about backwards compatibility again.... Big sorry for anybody who has made lists on my site in the last few weeks, I've repeatedly broken them:(
I will really try to not do that again...

To "force" me to not do so, let me share a list with you all as an example:
http://msl-raw-images.appspot.com/lists.ht...5B-1C00EE731309

It subscribes you to a list I've made of most full images that show Curiosity herself. The list could for example be used to make a nice self portrait. I plan on keeping the list updated, so expect it to grow in the coming period when new images become available.

Ok, the customary feature change report:
  • There is now a gallery button, which opens the current image listing in a gallery viewer for viewing convenience.
  • Maybe you've missed in in the earlier release notes: Do follow MSLRawImages on twitter, as it will report on new images becoming available.
  • Lists can now be shared through an URL link.


Still on my todo:
  • Making list meta info (description, url, etc.) editable
  • Making shared lists searchable. (now the only way to subscribe to lists is through the link, like I do above)


Greetings,
Ludo.
Go to the top of the page
 
+Quote Post
Stellingwerff
post Sep 12 2012, 11:35 AM
Post #142


Junior Member
**

Group: Members
Posts: 43
Joined: 7-August 12
From: The Netherlands
Member No.: 6493



MAHLI is sending a set of new images with a slightly different filenames: I now see filetypes R,U,S & T, besides the regular E & I. Anyone knows what each letter stands for? I'll do an analysis myself later this afternoon if nobody has a list available somewhere. Quick Google didn't produce a result (yet).

Greetings,
Ludo.
Go to the top of the page
 
+Quote Post
mcaplinger
post Sep 12 2012, 02:24 PM
Post #143


Senior Member
****

Group: Members
Posts: 2511
Joined: 13-September 05
Member No.: 497



QUOTE (Stellingwerff @ Sep 12 2012, 04:35 AM) *
MAHLI is sending a set of new images with a slightly different filenames...

I believe these are associated with focus stacking products. See 7.6 Focus Stack Acquisition and Merge in the MAHLI paper. Not sure which is which but it should be fairly obvious.


--------------------
Disclaimer: This post is based on public information only. Any opinions are my own.
Go to the top of the page
 
+Quote Post
fredk
post Sep 20 2012, 05:23 PM
Post #144


Senior Member
****

Group: Members
Posts: 4246
Joined: 17-January 05
Member No.: 152



Joe - I'm seeing all the latest images on your site listed as "sol 0" - is there a chance something's broken?
Go to the top of the page
 
+Quote Post
Joffan
post Sep 20 2012, 06:54 PM
Post #145


Member
***

Group: Members
Posts: 178
Joined: 13-September 05
Member No.: 498



QUOTE (fredk @ Sep 20 2012, 10:23 AM) *
Joe - I'm seeing all the latest images on your site listed as "sol 0" - is there a chance something's broken?

It's worse than that - they'e all dated from 1-Jan 2000, which I assume is a null capture date.
Go to the top of the page
 
+Quote Post
jmknapp
post Sep 20 2012, 07:00 PM
Post #146


Senior Member
****

Group: Members
Posts: 1465
Joined: 9-February 04
From: Columbus OH USA
Member No.: 13



Thanks for the heads up--I'll check it out. I'm guessing that either the NASA website html changed or these recent photos don't have a timestamp for some reason.

Jan-1-2000 is when NASA ephemeris time currently starts (et=0) so indeed it's a null capture date.


--------------------
Go to the top of the page
 
+Quote Post
centsworth_II
post Sep 20 2012, 07:07 PM
Post #147


Senior Member
****

Group: Members
Posts: 2173
Joined: 28-December 04
From: Florida, USA
Member No.: 132



Y2K strikes at last!
Go to the top of the page
 
+Quote Post
jmknapp
post Sep 20 2012, 07:44 PM
Post #148


Senior Member
****

Group: Members
Posts: 1465
Joined: 9-February 04
From: Columbus OH USA
Member No.: 13



QUOTE (centsworth_II @ Sep 20 2012, 02:07 PM) *
Y2K strikes at last!


Ha!--in this case it was option [c] operator error. I've been working on an alternative interface to the database (more like a google search) and in the machinations managed to move a script file from where it was needed. Anyway, should be OK now.


--------------------
Go to the top of the page
 
+Quote Post
Explorer1
post Sep 21 2012, 05:04 AM
Post #149


Senior Member
****

Group: Members
Posts: 2079
Joined: 13-February 10
From: Ontario
Member No.: 5221



Will this include streamlining the page select at the bottom, Joe? The list of page numbers covers nearly the whole browser screen at the bottom, and that's in just over a month of imagery! I know I can set the items per page to 100, but that will just delay the inevitable as it grows and grows. For a mission that is going to last years, there must be a more permanent solution.
Go to the top of the page
 
+Quote Post
jmknapp
post Sep 21 2012, 12:25 PM
Post #150


Senior Member
****

Group: Members
Posts: 1465
Joined: 9-February 04
From: Columbus OH USA
Member No.: 13



Here's another go at an alternative interface to the raw images:

http://curiositymsl.com/search

Try searches like:

mastcam -- all mastcam images
mastcam sol39 -- all mastcam images on sol 39
mr large -- all large mastcam right images
navcam -thumbnail -small -- all navcam images excluding thumbnails and smalls
mardi -sol0 -- all mardi images excluding sol 0
sunrise -- all images taken at sunrise

Mousing over the thumbnails brings up a preview. Scrolling to the bottom of the page loads more items if available.

Click on 'help' for more info.


--------------------
Go to the top of the page
 
+Quote Post

25 Pages V  « < 8 9 10 11 12 > » 
Reply to this topicStart new topic

 



RSS Lo-Fi Version Time is now: 20th April 2024 - 02:57 AM
RULES AND GUIDELINES
Please read the Forum Rules and Guidelines before posting.

IMAGE COPYRIGHT
Images posted on UnmannedSpaceflight.com may be copyrighted. Do not reproduce without permission. Read here for further information on space images and copyright.

OPINIONS AND MODERATION
Opinions expressed on UnmannedSpaceflight.com are those of the individual posters and do not necessarily reflect the opinions of UnmannedSpaceflight.com or The Planetary Society. The all-volunteer UnmannedSpaceflight.com moderation team is wholly independent of The Planetary Society. The Planetary Society has no influence over decisions made by the UnmannedSpaceflight.com moderators.
SUPPORT THE FORUM
Unmannedspaceflight.com is funded by the Planetary Society. Please consider supporting our work and many other projects by donating to the Society or becoming a member.