Printable Version of Topic

Posted by: Analyst Nov 8 2006, 11:50 AM

Did nobody notice this:

http://www.jpl.nasa.gov/news/features.cfm?feature=1226

Ten year after launch, there is some trouble with a solar array motor and a comm problem probably resulting from this and entering safe mode. Nothing dramatic yet, but something to follow closely.

There are other things than MRO and MER

Analyst

Posted by: PhilCo126 Nov 9 2006, 07:52 PM

Well losing it wouldn't be good for the already loaded communications relays ...
http://spaceflightnow.com/news/n0611/08mgs/

Posted by: tuvas Nov 9 2006, 07:59 PM

QUOTE (PhilCo126 @ Nov 9 2006, 12:52 PM)

Well losing it wouldn't be good for the already loaded communications relays ...
http://spaceflightnow.com/news/n0611/08mgs/

Posted by: ElkGroveDan Nov 9 2006, 09:07 PM

QUOTE (tuvas @ Nov 9 2006, 11:59 AM)

Posted by: tedstryk Nov 9 2006, 09:41 PM

It would be a real loss to have this happen now. With MARCI taking over the role that MOC's wide angle global images have filled for so long, it would be nice to have some overlapping coverage. Not to mention having TES and passive MOLA coverage overlapping MRO. And, as for MOC high resolution, the amount of the planet covered by MOC so far and what HIRISE can hope to cover is a small percentage, so more would be very helpful. I hope the mission continues until it either breaks down (lets hope it hasn't) or has instrument failures that render it useless.

Posted by: Sunspot Nov 9 2006, 11:44 PM

http://space.newscientist.com/article/dn10498-nasa-struggles-to-contact-lost-mars-probe.html

From reading this it seems that currently all communications have been lost and they don't know if it is in safe mode at all.

Interestingly they might try and observe MGS with MRO

Posted by: djellison Nov 9 2006, 11:55 PM

Well - it would be a very cool picture from HiRISE....but I hope we don't have to see it and that they can get MGS back online.

Doug

Posted by: lyford Nov 10 2006, 01:20 AM

Or this could all be a plot to get an http://www.nasa.gov/vision/universe/solarsystem/mgs-images.html of MGS....
I agree with Doug, but almost hope that we get contact established just after the imaging:
"Never mind on that HiRiSE shot... what? Already taken? Just put it over there on the front page of the NASA portal then..."

Posted by: tuvas Nov 10 2006, 01:47 AM

I'm impressed with this news source, after hearing about it, I looked in the archives to see if we had been contacted about the possibility, it hasn't been for very long that it has been the case... But, HiRISE has been contacted in some kind of official capacity to perhaps image MGS, all I can say is, it would be quite a trick... Not only for it's dificultly, but also it's timing. Still it would be cool;-) We're already the first to photograph a rover on another planet that's confirmed beyond anyone's doubt, why not add a satellite to the mix (Which I do recall has been done).

Posted by: elakdawalla Nov 10 2006, 02:29 AM

Indeed it has been done -- one of the many landmark accomplishments of MGS.
http://www.msss.com/mars_images/moc/2005/05/19/

--Emily

Posted by: ElkGroveDan Nov 10 2006, 04:04 AM

QUOTE (tuvas @ Nov 9 2006, 05:47 PM)

Posted by: tuvas Nov 10 2006, 05:54 AM

QUOTE (ElkGroveDan @ Nov 9 2006, 09:04 PM)

Posted by: Analyst Nov 10 2006, 07:42 AM

Better see a mission end this way than by a shortage of funding. Even better to see it going on.

Analyst

Posted by: slinted Nov 10 2006, 11:30 AM

To head off any freak-out about the potential impact on the MER relays, keep in mind that MGS is doing only minor relay duty compared to Mars Odyssey. I don't know if more recent data has been published, but as of January 05, only 7% of the rover data came down through MGS ( see http://sunset.usc.edu/gsaw/gsaw2005/s2/wilklow.pdf)

Posted by: djellison Nov 10 2006, 12:37 PM

As I understand it - relays are JUST Odyssey now and have been for a long time...i.e. MGS stopped doing relay before the first solar conjunction.

Meanwhile from New Scientist

"If the spacecraft does not receive commands from Earth for seven days in a row, it is programmed to stop whatever it is doing and try to transmit a signal to Earth using its high gain antenna. This could happen at about 0014 GMT on Friday (1614 PST on Thursday), so NASA will be listening for a signal from MGS's high gain antenna at that time."

http://space.jpl.nasa.gov/cgi-bin/wspace?tbody=399&vbody=499&month=11&day=10&year=2006&hour=00&minute=15&rfov=2&fovmul=-1&bfov=50&porbs=1&showsc=1

That would have been a Canberra pass.

Doug

Posted by: volcanopele Nov 10 2006, 04:05 PM

QUOTE (djellison @ Nov 10 2006, 05:37 AM)

Posted by: djellison Nov 10 2006, 04:09 PM

Oh boy.....and we thought Jason hated Mars before......

Doug

Posted by: ugordan Nov 10 2006, 04:10 PM

Please don't tell me the Cassini periapsis data is being trashed for MGS contingency operations!

Posted by: djellison Nov 10 2006, 04:13 PM

I'm sure somewhere someone's thinking "well - that makes us even for when Cassini took our DSN time when it had a problem"



Doug

Posted by: nprev Nov 10 2006, 04:15 PM

So no news from the Canberra pass?

Posted by: volcanopele Nov 10 2006, 04:23 PM

QUOTE (ugordan @ Nov 10 2006, 09:10 AM)

Posted by: Sunspot Nov 10 2006, 04:33 PM

Maybe that's why there hasn't been any new images from Opportunity for about 48 hours too.

Posted by: djellison Nov 10 2006, 04:48 PM

I'm not sure this should be impacting other Mars ops (it may be though...perhaps for AM DFE Uplinks?)

MRO has been busy getting uplinks from the ground :

Posted by: ugordan Nov 10 2006, 04:57 PM

QUOTE (volcanopele @ Nov 10 2006, 05:23 PM)

Posted by: volcanopele Nov 10 2006, 05:25 PM

QUOTE (ugordan @ Nov 10 2006, 09:57 AM)

Posted by: Analyst Nov 10 2006, 05:33 PM

If I remember correctly they can communicate with more than one spacecraft at Mars using the same DSN dish because the antenna beam covers the whole Mars disk and beyond.

As for Cassini: It is a normal procedure that a spacecraft emergency gets priority over normal (Cassini) operations. The same about primary vs. extended missions and manned vs. unmanned missions. It's the way it should be. If they have exhausted their options, MGS is declared lost, not before.

Analyst

Posted by: nprev Nov 10 2006, 05:36 PM

QUOTE (volcanopele @ Nov 10 2006, 09:25 AM)

Yes.

All is well. Though still, giant laser...MGS...

Posted by: ugordan Nov 10 2006, 05:41 PM

QUOTE (djellison @ Nov 10 2006, 05:13 PM)

QUOTE (Analyst @ Nov 10 2006, 06:33 PM)

Posted by: volcanopele Nov 10 2006, 05:50 PM

yeah, there are contingency plans (woohoo for that). And I hope MGS is recovered, sooner rather than later. I guess it just annoying, that's all.

Posted by: djellison Nov 10 2006, 07:58 PM

Posted by: Analyst Nov 10 2006, 08:41 PM

SWTB - Solar Array Wing tip brake?
SR - Switch ...??

Posted by: djellison Nov 10 2006, 08:45 PM

Just for fun, probably highly inaccurate...if they DO do HiRISE imaging...I've seen figures of 100km smacked around which would be approx 10cm/pixel....so here's a couple of simulated views using ye-olde VRML model that's online if you google for it.

Doug

Posted by: akuo Nov 10 2006, 09:18 PM

You have to remember, MGS might be out of HiRise's focus at 100km. Though the focus is adjustable, I don't know if its this much adjustable, and whether they would do a major adjust right at the beginning of the PSP.

Should still look pretty good.

Posted by: djellison Nov 10 2006, 10:57 PM

Two more of whatever those things were...followed by two SCGNT :

Posted by: tuvas Nov 10 2006, 11:38 PM

Doug,

Where do you get that kind of info? Just curious...

Anyways, unless something drastic happens, it's looking more likely like JPL will have to take more drastic options to try and find MGS, feel free to use your imagination. Also note that everything public about the possibility of MRO photographic MGS mentions MRO, no specific instruments were mentioned.

Posted by: jamescanvin Nov 10 2006, 11:50 PM

Have we gone past the 7 days out of contact yet?

It still seems likely to me that MGS is happily pointing her arrays at the sun and waiting for 7 days before attempting to contact Earth through the HGA, as she is programmed to do in safe mode.

Posted by: djellison Nov 10 2006, 11:53 PM

http://mgsw3.jpl.nasa.gov/seq/MGS/rad/MGSradiation.log_nohdr

Doug

Posted by: Bubbinski Nov 11 2006, 06:03 AM

Does this "radiation log" mean MGS is still alive?? Have they regained contact? Let's hope so!

Posted by: djellison Nov 11 2006, 07:51 AM

It doesn't really tell us anything to be honest - it's guess work at best, but it's a list of the sequences being sent TO the spacecraft, not ones being recieved from it....and the same sequence being sent many times over would be suggestive (I would have thought) of an unresponsive spacecraft.

Lots more commands radiated overnight, including....

Reset a sun-angle timer to a different time.
that's looped a few times, and then

Turn on telementry modulation
Switch that to 10bps
then that gets looped a couple of times

They've radiated 50 commands in the last 24 hours.

Doug

Posted by: Analyst Nov 11 2006, 08:45 AM

It's only guessing, and I guess they have not got telemetry back because almost every block of commands ends with "Turn on TWTA". Without a working TWTA you can't get telemetry.

Analyst

Posted by: djellison Nov 11 2006, 01:58 PM

Well - we've had this loop of 5 commands

Posted by: Analyst Nov 11 2006, 05:02 PM

There are small variations in this 5 command pattern lately.

Could it be that MGS is in a command loss routine, switching (by itself) between different hardware components with the goal to deselect a failed one and doing this until one of the commands sent here again and again comes through?

Analyst

Posted by: Bubbinski Nov 11 2006, 06:35 PM

Yikes...thanks for the answers. Guess we'll be waiting for the Hirise picture of MGS to tell us more. If this truly is the end, then let's salute MGS for a decade of service over Mars and a job well done.

Posted by: djellison Nov 11 2006, 06:36 PM

My best efforts in understanding what the sequences are is limited - but there are specific, multiple references to some of them...

CXSPG1 and 2 are "Set the modulation index for 10 bps data rate" on both Box Sides XSU1 and 2.
The XSU's are cross strap units for routing telemetry to the SSR's or Telecoms.

TCM1MN and TCM2MN turns telemetry modulation on for MOT 1 and 2
The MOT's are Mars Orbiter Transponders

STRPAN turns on the TWTA.

To try and put it all in to context - attached is an extract from http://mars.jpl.nasa.gov/mgs/pdf/SE012V1.PDF

I wish someone from the MGS team would do a HiBlog type effort The money just isn't around for good outreach with the older missions I guess.

Doug

Posted by: mcaplinger Nov 12 2006, 03:16 AM

QUOTE (djellison @ Nov 11 2006, 11:36 AM)

I wish someone from the MGS team would do a HiBlog type effort The money just isn't around for good outreach with the older missions I guess.

Posted by: ElkGroveDan Nov 12 2006, 03:58 AM

Thanks for weighing in Mike. Anything you can share with us as events transpire would be appreciated. Even just a quick posting of an emoticon when you get some news.

Posted by: climber Nov 12 2006, 10:46 AM

I think I get the point from Mike and had not doubt they try the maximum.
Nevertheless, I'd like also to point out how badly needed are Doug's updates. Once you're on UMSF, you realise that you're not alone to consider Spacecrafts as an entity you're familiar with. They're not "persons" but they're no longer inert objects, they definitively have a kind of soul.
So, when something go wrong, you WANT to know and since the Internet exist you can actualy try to look for news...and soon you realise that the best place is here, at UMSF. This is because all of us share the same passion for the adventure/exploration AND because we know that some of us are ABLE to find the place where the information is.
OK, we only talk to MGS with no reply so far and, as Mike says, it's either power positive or not, but knowing what JPL does is badly needed. Just think about how we'll be the day when the first rover will not show anymore. Are you gona give up the first day or switch to the other one and try to forget? I don't think so, and I'm sure the new trait called "xxx doesn't respond anymore" will have more hit everyday than all the other traits of UMSF and this will be untill JPL Officialy give up.
Thanks again Doug for your dedication and thanks to Mike to give us your insider view. I back you guys.

Posted by: djellison Nov 12 2006, 10:59 AM

Well - reading the logs from overnight - there were about four hours of a different commanding sequence which included two references to the HGA (I can't see the sequences defined anywhere unfortunately) but since then - it's been the same CXSPG1, CXSPG2,TCM1MN,TCM2MN, STRPAN. If I had to guess a diagnosis - I'd say that they tried something different (perhaps based on predicts of spacecraft attitude) , had no results, and then resorted to the baseline sequencing.

Thanks for the input Mike - as with all such things the longer we know nothing the worse the situation seems.

Doug

Posted by: diane Nov 12 2006, 02:01 PM

Keeping in mind that SOHO was out of service for more than a year, let's keep in mind that many things are still possible.

http://sohowww.nascom.nasa.gov/operations/Recovery/

Posted by: djellison Nov 12 2006, 06:14 PM

Well - no updates to the log for 14 hours....most likely reason being that the script that adds things to the log is either broken, or intentionally turned off.

Doug

Posted by: Sunspot Nov 12 2006, 06:27 PM

Maybe they don't want people to know what they're doing

Posted by: djellison Nov 12 2006, 06:37 PM

Well - I found http://mgsw3.jpl.nasa.gov/seq/ which includes lots of similar info just by googling for "UHF Relay" - I can't believe that JPL would unintentionally allow that information to be out and about and so easy to find - indeed some parts are marked as being removed due to ITAR......but then it wouldn't be the first time that JPL would have let something be online that shouldn't have been. It wouldn't be beyond belief to have JPL stop updates going to that site because they didn't want people ( i.e. me ) reading it.

Doug

Posted by: nprev Nov 12 2006, 10:44 PM

Just out of curiosity, what is MGS' consumable status? Reason I ask is that I hope it isn't burning fuel trying to reacquire Earth if that turns out not to be the appropriate course of action.

Posted by: djellison Nov 12 2006, 11:04 PM

http://www.unmannedspaceflight.com/lofiversion/index.php/t902.html

"As of 05-153 (06/02/05) MGS fuel consumption is 3.3 g/day, with 9.15 kg of usable fuel remaining. At this consumption rate, the usable fuel will support operations into 2013."


Doug

Posted by: nprev Nov 12 2006, 11:25 PM

Much better than I'd hoped....thanks, Doug!

Posted by: Norm Hartnett Nov 12 2006, 11:45 PM

QUOTE (diane @ Nov 12 2006, 06:01 AM)

Posted by: mcaplinger Nov 13 2006, 01:03 AM

QUOTE (djellison @ Nov 12 2006, 04:04 PM)

Posted by: nprev Nov 13 2006, 01:54 AM

Hmm...okay, thanks very much for the elucidation, M.

Is it possible that the array is stuck in such a fashion that it's interfering with the HGA in between eclipse periods? I assume that MGS maintains inertial lock with respect to Mars nadir duing normal ops. If this scenario was true, then we might catch a break in a week or two due to the relative orbital motion between Earth & Mars (maybe longer...I think that we're in the middle of that long, slow distant "catch-up" phase between the two orbits).

Posted by: mcaplinger Nov 13 2006, 03:01 AM

QUOTE (nprev @ Nov 12 2006, 06:54 PM)

Posted by: nprev Nov 13 2006, 07:04 AM

I see. That sounds like a potentially much more serious situation, then; hopefully, it's entirely notional.

Posted by: djellison Nov 13 2006, 06:38 PM

Well - reading the logs for MODY and MRO - I think they've turned off the updates to the webpage as they have no radiated files since before the last MGS log entry

Doug

Posted by: Sunspot Nov 14 2006, 01:47 AM

Still no contact yet

http://space.newscientist.com/article/dn10522-fleet-of-probes-enlisted-to-contact-silent-mars-orbiter.html

Late on Wednesday, MRO will try to determine MGS's location by taking a picture with a low resolution camera. Using this information, MRO will take another image of MGS on Friday using its High Resolution Imaging Experiment (HiRISE) – the most powerful camera ever sent to Mars.

The HiRISE image should be detailed enough to determine how MGS is oriented in space and how its solar arrays are positioned.

Posted by: nprev Nov 14 2006, 02:02 AM

Whoa! Now there's a powerful means of hopefully resolving the problem...the pics should eliminate many possible failure modes. Thanks for the update, Sunspot.

Posted by: Stu Nov 14 2006, 06:46 AM

Typical. Frakking typical!

For years we've been getting sensational pictures from Mars Global Surveyor, breathtaking images of martian features, dust-storms, the whole planet itself. We've seen gullies, craters, valleys and gorges in more detail than ever before. The probe just reached its tenth anniversary, an amazing achievement...

All that with hardly a nod of the head from the BBC.

But now, following the usual "bad news from space is the only news worth covering" guidelines, this morning, Kate Silverton - bless her cute, punky hair, elfin face and trendy square framed glasses... okay, I'm a fan... - is telling everyone on Breakfast TV about the "ailing probe", trotting out the usual "lost in space" cliches.

Not happy.

Posted by: paxdan Nov 14 2006, 08:47 AM

QUOTE (Stu @ Nov 14 2006, 06:46 AM)

Posted by: Borek Nov 14 2006, 11:45 AM

QUOTE (Stu @ Nov 14 2006, 07:46 AM)

Typical. Frakking typical!

For years we've been getting sensational pictures from Mars Global Surveyor, breathtaking images of martian features, dust-storms, the whole planet itself. We've seen gullies, craters, valleys and gorges in more detail than ever before. The probe just reached its tenth anniversary, an amazing achievement...

All that with hardly a nod of the head from the BBC.

Posted by: Rakhir Nov 14 2006, 01:54 PM

QUOTE (Sunspot @ Nov 14 2006, 04:47 AM)

Posted by: Stu Nov 14 2006, 02:31 PM

QUOTE (Borek @ Nov 14 2006, 11:45 AM)

Posted by: Sunspot Nov 14 2006, 02:37 PM

If they do manage to recover it, I doubt it will make the news at all.

Posted by: djellison Nov 14 2006, 02:39 PM

Calm down boys....calm down.


Anyway - back to the actual topic in hand...

Odyssey or MRO to MGS UHF comms - I've not thought of that but it should be possible - however I imagine that the signal strength pattern from the UHF antenna would require a serious spacecraft manouver for it to work.

Doug

Posted by: Analyst Nov 14 2006, 03:14 PM

QUOTE (djellison @ Nov 14 2006, 03:39 PM)

Posted by: djellison Nov 14 2006, 03:34 PM

The thinking behind it is probably like this :

We are sending X-Band signals, but we do not know if it is getting them as we see no evidence in return

Most likely option - the vehicle is incapacitated and/or not recieving the signals.
Less likely - the vehicle is recieving them but for some reason can not transmit on X-Band

In the second case - if we uplink a command to transmit on UHF which Spirit/Opportunity might hear, then we will have evidence of life onboard MGS.

Basically, you start at the top of the fault tree and cross out all the different failure options until you've crossed them all out....at which point you go out, have a drink, toast MGS, and then move on.

Anyone who followed the MPL post-landing story will have been through this...it's painful, as you get to the increasingly unlikely failure modes, but on which you end up pinning more and more of your hopes.

Doug

Posted by: jaredGalen Nov 14 2006, 03:38 PM

Using the UHF, is it to try and send something from the rovers to MGS and then try and relay via Odyssey or use the rovers to try and listen in to see if the MGS is transmitting anything via UHF and then relay the results via Odyssey?

I read an article that mentioned it this morning but can's see it now

Edit: Ah, never mind

Posted by: djellison Nov 14 2006, 04:02 PM

No - we'd have to use the DSN to command MGS to turn on it's UHF transmitter during a rover flyover...and then the rovers would report back to see if it worked....I MAY be wrong but I don't believe that MGS can be commanded via UHF.

Also - Stanford might get out their UHF antenna and see if a UHF carrier from MGS can be seen here on the ground....sometimes it gets a signal from Mars Orbiters, sometimes it doesn't.

Doug

Posted by: Analyst Nov 14 2006, 04:07 PM

QUOTE (djellison @ Nov 14 2006, 04:34 PM)

Posted by: tuvas Nov 14 2006, 04:12 PM

Let me just pipe in and say from what I've been able to understand, the space.com article is the far more likely approach to what will happen. http://www.space.com/missionlaunches/061114_mgs_mro.html . It seems to cover the main concerns better than the other articles posted by other news sources.

Posted by: ElkGroveDan Nov 14 2006, 04:14 PM

QUOTE (djellison @ Nov 14 2006, 07:34 AM)

Posted by: Marz Nov 14 2006, 04:33 PM

QUOTE (ElkGroveDan @ Nov 14 2006, 10:14 AM)

Posted by: Stu Nov 14 2006, 04:37 PM

QUOTE (Marz @ Nov 14 2006, 04:33 PM)

Yeah, like with Beagle...

Posted by: djellison Nov 14 2006, 04:52 PM

QUOTE (Marz @ Nov 14 2006, 04:33 PM)

Posted by: elakdawalla Nov 14 2006, 05:05 PM

See also my notes from the Bob Preston talk at last May's OPAG meeting:
http://www.planetary.org/blog/article/00000570/

Upgrading infrastructure isn't a sexy project, but deep space communications really do represent a bottleneck, and the lack of upgrades to the DSN is harming our ability to make the most of our deep space assets. It's time for those arrays of 12-meter antennas to be built -- that would give the DSN so much more flexibility.

--Emily

Posted by: djellison Nov 14 2006, 05:19 PM

"Spitzer, which trails Earth in its orbit, now needs the 70-meter dishes."

I did NOT know that - wow....that's a serious stretch on resources - I'd forgotten about that write up - it reads pretty much the same as the one I saw at IAC really.

Doug

Posted by: odave Nov 14 2006, 05:19 PM

While I'm very concerned about MGS, I'm not feeling the anxiety or anguish like I did with MPL and Beagle. To use a human analogy, while then death of anyone is tragic, the death of a small child seems especially so considering the loss of all that potential - a whole life never realized. If MGS has passed, I'll be sad and lift my glass to her accomplishments. I've got my fingers crossed, though!

Posted by: Sunspot Nov 14 2006, 05:53 PM

http://www.space.com/missionlaunches/061114_mgs_mro.html

There’s a feeling that maybe the well-used MGS felt it was time to sign off.

On the 10th anniversary of MGS in space—November 7—that’s the same day that MRO cranked up its primary science tasks.

“It really seems like there’s some fate involved in this,” Sidney said. “MGS knew it was time to retire.”


awwwwwww

Posted by: MarkL Nov 14 2006, 07:25 PM

It's like losing an old friend. I remember watching the aerobraking page way back when and following MGS' progress. It's the mission that really "cracked open" Mars for us to explore in my view. And many of us were able to actually tell the thing what to take photos of which was phenomenal. Lets hope it can be recovered as MOC is still a terrific instrument. It does, though seem like handing off the baton, given the timing.

Posted by: tuvas Nov 14 2006, 09:13 PM

I should also add in that if this process can be accomplished, it will be extremely difficult, and thus there isn't a fixed day assigned to make the attempt to photograph MGS, only that the attempt should be made reasonably soon, but not to disturb any critical science/ Phoenix endevours.

Posted by: Decepticon Nov 14 2006, 09:49 PM

I was hoping they would use the last part of this funded mission to explore PHOBOS.

Posted by: mcaplinger Nov 14 2006, 10:24 PM

QUOTE (tuvas @ Nov 14 2006, 01:13 PM)

Posted by: ugordan Nov 14 2006, 10:44 PM

One thing Emily pointed out in her blog is that the s/c has been out of contact for 9 days. Its orbit is not exactly nailed down anymore. HiRISE attempts to image the predicted locations might be futile, resulting in huge amounts of data of empty space. I'm actually pretty skeptical they'll be able to pull it off easily.

Posted by: climber Nov 14 2006, 11:07 PM

If I understood it right they're doing a kinda long exposure and hope that MGS will show somewhere on the picture and then they can deduce where it is and shoot in the rigth direction a few days later. Did I get it right?

Posted by: mcaplinger Nov 14 2006, 11:09 PM

QUOTE (ugordan @ Nov 14 2006, 02:44 PM)

Posted by: djellison Nov 14 2006, 11:21 PM

"We'll use HiRISE on Friday"
"We'll use CTX on Wednesday then HiRISE on Friday"
"We'll use a long exposure with HiRISE on Wednesday then a targetted observation on Friday"

The usual suspect media outlets have all reported one or more of those.....so it's been a bit confusing for the layperson.

Doug

Posted by: mcaplinger Nov 14 2006, 11:44 PM

QUOTE (djellison @ Nov 14 2006, 03:21 PM)

Posted by: nprev Nov 15 2006, 12:06 AM

QUOTE (djellison @ Nov 14 2006, 08:52 AM)

Posted by: mcaplinger Nov 15 2006, 12:31 AM

QUOTE (nprev @ Nov 14 2006, 04:06 PM)

Posted by: nprev Nov 15 2006, 01:18 AM

Good questions. I'm not sure if it would be feasible with RF unless 1) we can successfully develop & deploy very large collapsable antennae (the Galileo experience was instructive), 2) advanced DSP on the receiving end of all terminals involved could compensate for much lower transmitter power outputs, and 3) flight-qualified ultra-stable transmitters with very fine output frequency resolution could be developed.

This idea would work better with lasers; something like an MTO for at least each of the inner planets would provide the necessary link between active exploration missions & the new network. This network would be used almost exclusively for data return & housekeeping, freeing up the DSN for critical activities such as resolving the current MGS anomaly, early mission support, and tracking during cruise.

EDIT: Apologies if anyone saw a smiley instead of 2) above; this was not intended as a shot against Galileo, the message board just interpreted by original use of a letter plus a parenthesis as a smiley.

Posted by: RichardLeis Nov 15 2006, 02:30 AM

QUOTE (mcaplinger @ Nov 14 2006, 03:24 PM)

Posted by: RichardLeis Nov 15 2006, 02:36 AM

QUOTE (djellison @ Nov 14 2006, 04:21 PM)

Posted by: mcaplinger Nov 15 2006, 03:47 AM

QUOTE (RichardLeis @ Nov 14 2006, 06:30 PM)

Posted by: tuvas Nov 15 2006, 04:11 AM

QUOTE (mcaplinger @ Nov 14 2006, 08:47 PM)

Posted by: edstrick Nov 15 2006, 10:02 AM

Unless Global Surveyor's done a lot of attitude gas jetting, it's on-orbit location should be pretty well known. I don't, however, have a clear idea of how the along-orbit uncertainty spreads with time. What's not known at all and is the object of investigation is MGS's ATTITUDE.

Of course, if they spot a Klingon bird of Prey next to it.......

Now if we could just get something other than random noise (and fullsome self-praise) from the remote-viewing pseudo-psychics....(sigh)

Posted by: ugordan Nov 15 2006, 10:13 AM

The along-track uncertainty is likely the principal uncertainty here. The exact orbit need only be slightly higher/lower for significant timing differences to be accumulated on a time scale of days. These would manifest as a change in when the spacecraft passes a certain point in its orbit, with respect to the (practically identical) reference orbit. Given the speed these things move, even a couple of seconds worth of timing error means they move quite far along track. As to how much of a contributor the possible thruster firings are (ideally, they should not be one at all) or how significant the atmospheric friction is in lowering the orbit is open to question.

Posted by: edstrick Nov 15 2006, 10:21 AM

Assuming no net thruster firings, it's a relatively simple <and probably documented somewhere in published mission navigation papers, like AAS Advances in Navigation, or AAS Advances in Communication and Control type volumes in engineering libraries>, how the along track knowledge degrades with time following a normal orbit solution.

Posted by: djellison Nov 15 2006, 10:25 AM

When I started this place a couple of years back - I never thought I'd see a debate between MOC/CTX and HiRISE people over the relative difficulty of photographing spacecraft - what a very very sureal thing to read over ones cornflakes

Doug

Posted by: ugordan Nov 15 2006, 10:44 AM

A very back-of-the envelope calculation, could be very wrong, but still I think it's illustrative:
Assuming a 118 minute orbit, 3800 km orbital radius, 3.37 km/s orbital velocity here. That amounts to 122 orbits in 10 days. Let's suppose we change the orbital radius by just 100 meters, from 3800 km to 3800.1. Given the ratio of orbital periods T1/T2=SQRT(R1^3/R2^3), that gives me around 0.9999605 T ratio. Multiply (1- 0.9999605) by 118 minutes, that amounts to a difference of about 0.28 seconds per orbit.
Now, 122 * 0.28 s * 3.37 km/s = 115 kilometers in along-track drift. Not exactly peanuts. It's not getting any smaller as time passes, either. Note how even a small, 100 meter radial change in orbit radius results in an 3 orders of magnitude larger change in along-track position.

Posted by: monitorlizard Nov 15 2006, 01:42 PM

To change the subject just a bit, if MGS were not able to orient its solar panels toward the Sun, wouldn't that mean that it couldn't recharge its batteries, which would leave the spacecraft electrically dead (i.e., totally dead)? In fact, isn't that what killed the Phobos-1 spacecraft (though a different root cause of losing solar panel orientation)?

This is a worst case scenario, I know, but we seem to be running out of other scenarios.

Posted by: djellison Nov 15 2006, 02:35 PM

Yup - what you describe is the process of remaining power positive.

The moment you have less power than you require for basic survival....then you are running a very dangerous game. I imagine a Mars orbiter would get very cold very quickly without power whilst in eclipse - Once you have a spacecraft that has basically 'browned out' - you have no means to maintain attitude, and nasty things can start to happen such as frozen prop etc.

The only story I know of a recovery from that sort of problem was with SOHO - and that was a very very lenghty procedure that began with something as crazy as bouncing radar signals off the spacecraft with the Areciebo dish to identify its orientation and spin rate.

In some respects - imaging MGS with an one or more MRO instruments is a martian equiv of that - identify the physical config of the spacecraft - or even better - do it twice to identify what, if any, attitude control it is exhibiting.

Doug

Posted by: lyford Nov 15 2006, 04:29 PM

QUOTE (djellison @ Nov 15 2006, 02:25 AM)

I never thought I'd see a debate between MOC/CTX and HiRISE people over the relative difficulty of photographing spacecraft...

Posted by: tty Nov 15 2006, 07:45 PM

QUOTE (ugordan @ Nov 15 2006, 11:44 AM)

Posted by: djellison Nov 15 2006, 08:49 PM

All the camears in question are push broom cameras - they require the motion of the planet below them to build up the 'length' of an image.

You can get creative with their operation when it comes to tracking down MGS though

Doug

Posted by: djellison Nov 16 2006, 12:43 AM

MRO has been been commanded...

2006-319T18:20:35 ri3500 r01st_mgs_image_319.abs

Sent about 6 hrs ago. Can't tell which instrument it is ( although CTX as 'pin down' and HiRISE as a follow up would make the most sense )

Meanwhile, many repeated commands to MGS - Tuesady and most of Wednesday seemed packed full of more explicit transmitter commands - turning on heaters etc etc - but repeated many times - cycling between 'sides' so one presumes still an 'in the dark' set of commands...just trying all options.


Doug

Posted by: elakdawalla Nov 16 2006, 02:37 AM

According to MGS Project Manager Tom Thorpe, they're actually using the navigation camera. I got a lot of info from him this afternoon.

http://planetary.org/news/2006/1115_Mars_Global_Surveyor_Falls_Silent_All.html

--Emily

Posted by: tuvas Nov 16 2006, 03:16 AM

QUOTE (elakdawalla @ Nov 15 2006, 07:37 PM)

Posted by: dvandorn Nov 16 2006, 03:35 AM

Just a mild suggestion to the moderators -- perhaps the subtitle of this thread is, um, inappropriate? At least until we know whether or not we'll get MGS back, it seems mighty incorrect to be discussing these rescue efforts in a thread subtitled "still going strong"...

-the other Doug

Posted by: jamescanvin Nov 16 2006, 03:43 AM

Indeed. Done.

I was considering changing the title anyway considering we don't actually know what 'mode' MGS may be in by now. I hadn't noticed the subtitle, good call oDoug.

Posted by: Stu Nov 16 2006, 06:39 AM

(To the tune of Pink Floyd's "Comfortably Numb")

Hello.
Is there anybody out there?
Just bleep if you can hear us.
Is there anyone home?

Come on now, MGS.
We're worried by your silence.
Maybe we can ease your pain,
Get you taking pics again.

Relax.
We're trying hard to find you.
You've been quiet for a week now:
Can you tell us where it hurts?

There is no pain, you are receding.
A distant blue star on my horizon.
You are only coming through in waves.
Your lips move but I cant hear what youre sayin.
When I first arrived I had a fever.
My gyros felt just like balloons.
Now I got that feeling once again.
I cant explain, you would not understand.
This is not how I am.
I have become comfortably numb...

Posted by: djellison Nov 16 2006, 08:08 AM

Thanks for the heads up Emily - nice write up....I had forgotten about the old Nav camera...we never saw much from that (well, one image) - I hope they dump it's entire results onto the PDS...might make a nice movie

Doug

Posted by: tanjent Nov 16 2006, 02:22 PM

The solar panel trouble indications that marked the last communications with MGS - were they traceable to the same joint that was stressed during aerobraking many years ago? My brief keyword search did not turn up any mention of a connection but it seems like this should be in the minds of the diagnosticians as they try to figure out what went wrong. Perhaps at this stage it would not actually make the problem any easier to solve though.
Peter

Posted by: mcaplinger Nov 16 2006, 02:33 PM

QUOTE (tuvas @ Nov 15 2006, 07:16 PM)

Posted by: CosmicRocker Nov 17 2006, 07:20 AM

QUOTE (djellison @ Nov 15 2006, 04:25 AM)

... - what a very very sureal thing to read over ones cornflakes ...

Posted by: nprev Nov 17 2006, 07:42 AM

Doug's quote would make an excellent subtitle for UMSF.com!

Seriously, best wishes for MGS; she will someday be greatly missed, but please not yet. I remember the excitement way back when of finally being able to really see Mars again after so many, many years...in fact, to see the latest & greatest daily via the then-newfangled Internet. My ex eventually became quite annoyed with my constant calls of "Honey, come here, you gotta see this!!!" as the spectacular images just kept on coming...

Posted by: djellison Nov 17 2006, 09:20 PM

Posted by: monitorlizard Nov 18 2006, 12:52 AM

Just speculation, but I always thought that "eph_mars" in a command was an ephemeris update to give
the best possible coordinate data to the spacecraft.

Posted by: djellison Nov 18 2006, 07:14 AM

Well - I thought that, but why would the spacecraft need ephemeris for itself or another spacecraft? That's what WE need so we can tell it what to do and when - unless it's a far more intelligent spacecraft than I thought and they're just telling it the orbital params of MGS and letting it work everything out itself.. (which I really really doubt)

Doug

Posted by: tuvas Nov 18 2006, 07:43 AM

MRO is quite a smart spacecraft actually, it's the first one to target the location and not the time. But my guess would be that MRO took two pictures to blink them and see if anything moves (I'm assuming the command log was from MRO). This is just my guess, no inside info.

Posted by: Zvezdichko Nov 18 2006, 08:21 AM

Hello.

This is my first post in the forum. I'm Svetlio from Bulgaria.

Today MRO should take pictures of MGS - any news? We know that today MRO will use its navigation camera,and if MGS is detected, MRO will make another set of picture with its HiRISE camera to see the orientation of its solar panels.

Posted by: stevelu Nov 18 2006, 07:04 PM

I assume everybody's so quiet because nobody knows anything yet.

We're just waiting for some word from the folks who do know (who are understandably a bit busy right now and have a lot on their minds besides this board).

I imagine that when there is news, this'll be one of the first places it appears.

Posted by: Stu Nov 18 2006, 10:38 PM

Welcome, Svetlio, nice to have you onboard the good ship UMSF

Posted by: djellison Nov 20 2006, 08:16 AM

Posted by: Sunspot Nov 20 2006, 10:10 AM

Think we're going to see any of these MGS images? I'm a little surprised that were now 2 weeks into the science mission and we haven't seen anything at all yet.

Posted by: Stu Nov 20 2006, 10:29 AM

I know what you mean. No new Oppy Pancams since the 10th, no new MGS pics... I fear a crack team from ESA has taken over Mars Operations and is forcing them to work to their image release rules, i.e release b****r all...

Posted by: djellison Nov 20 2006, 10:35 AM

You mean HiRISE images.

I would have thought the efforts would have been going into finishing up the viewing software that has been mentioned elsewhere instead of doing the rush-job on releasing images that we saw during the transition phase. However - the MGS imaging efforts have probably disruted things at HiRISE HQ, and indeed to some extent the command and reception of MRO data in general via DSN congestion.

Give them month or two to get into the flow of the primary science phase, get the viewing software sorted..THEN - you will have more images that time.

It takes 6 to 12 months for an MGS MOC image to be released (if the PDS release is done on time) .... to start moaning about HiRISE releases at this time would be a long long way from reasonable ( not to mention off topic for this thread)

Doug

Posted by: Sunspot Nov 20 2006, 10:41 AM

Yes, but we haven't seen anything new in nearly 5 weeks. Which seems to contradict what was being said at the press conferences.

Posted by: djellison Nov 20 2006, 11:23 AM

QUOTE (Sunspot @ Nov 20 2006, 10:41 AM)

Posted by: tuvas Nov 20 2006, 01:51 PM

With respect to HiRISE images, well, there should be a nice batch released sometime soon, it should be this week... As for the why, well, just see the HiBlog. I think it's actually harder to release orbiter pictures, due to needing the SPICE kernels, which are only released once a week. After that, we just had to sit back and wait. But, hopefully that time is over now, so we should be releasing pictures soon.

As to why nothing about MGS has been released, well, my guess would be because there's nothing to release. The process isn't an easy one, and MRO/NASA will want to make sure they get it as exactly right as possible, thus it might take some time. Remember Victoria Crater, it was released only a few days after the image was taken, and I would well imagine the same type of thing will occur with MGS. As to when that will occur, well, my guess would be ASAP. Note however that nothing here is based on inside info, this is purely coming from my brain/public sources.

Posted by: MarkL Nov 20 2006, 05:47 PM

I find it hard to gripe too much. The firehose is about to open! We'll have way more than we can handle once MRO settles in and as Mars works toward opposition. The lull in the action is kind of nice actually. And if HiRise snaps a pic of MGS, wouldn't that be something for the ages? So it's all good far as I can tell 'cept for poor old Global Surveyor. MGS phone home please.

Posted by: Sunspot Nov 20 2006, 07:50 PM

Nov. 20, 2006

Erica Hupp/Dwayne Brown
Headquarters, Washington
202-358-1237/1726

Guy Webster
Jet Propulsion Laboratory, Pasadena, Calif.
818-354-6278

MEDIA ADVISORY: M06-179

NASA PROVIDES MARS GLOBAL SURVEYOR UPDATE

NASA will hold a media teleconference at 1 p.m. EST, Tuesday, Nov. 21,
to discuss the status and science accomplishments of the Mars Global
Surveyor. The 10-year old spacecraft is the oldest of five NASA
spacecraft currently active at the red planet.

Reporters must call the NASA Jet Propulsion Laboratory, Pasadena,
Calif., media relations office at 818-354-5011 for participation
information. Images supporting the briefing will be posted online at
the start of the briefing at:

http://www.nasa.gov/mission_pages/mgs/20061121.html

Briefing participants:
--Michael Meyer, Lead Scientist, Mars Explorations Program, NASA
Headquarters
--Fuk Li, Mars Program Manager, Jet Propulsion Laboratory (JPL),
Pasadena, Calif.

Supporting participants:
--Tom Thorpe, Project Manager, Mars Global Surveyor, JPL, Pasadena,
Calif.
-- Phil Christensen, Principal Investigator, Thermal Emission
Spectrometer, Arizona State Univ, Tempe
-- Michael Malin, Principal Investigator, Mars Orbiter Camera, Malin
Space Science Systems, San Diego

Audio of the event will be available on the Internet at:

http://www.nasa.gov/newsaudio

Posted by: tedstryk Nov 20 2006, 08:34 PM

"NASA will hold a media teleconference at 1 p.m. EST, Tuesday, Nov. 21,
to discuss the status and science accomplishments of the Mars Global
Surveyor."

That has a bad ring to it...sounds funerary.

Posted by: Marz Nov 20 2006, 08:39 PM

QUOTE (Sunspot @ Nov 20 2006, 01:50 PM)

Posted by: elakdawalla Nov 20 2006, 09:33 PM

So much for taking time off for Thanksgiving Why do spacecraft events always seem to happen around major holidays? "Science accomplishments" does sound a bit obituary-like. I'll have to watch & report.

--Emily

Posted by: djellison Nov 20 2006, 09:39 PM

No Alfred McEwen would be suggestive that there's no HiRISE image to be unveiled - but they might still have something.

I have to go to a distant relatives funeral tomorrow...then come home and listen to this....I think I'll leave the suit on just in case.

Doug

Posted by: lyford Nov 20 2006, 09:56 PM

Well, it does say "The 10-year old spacecraft is the oldest of five NASA spacecraft currently active at the red planet. "
But maybe that's reading too much in it from my end.

Posted by: tuvas Nov 20 2006, 10:09 PM

QUOTE (lyford @ Nov 20 2006, 02:56 PM)

Well, it does say "The 10-year old spacecraft is the oldest of five NASA spacecraft currently active at the red planet. "
But maybe that's reading too much in it from my end.

Posted by: Stu Nov 20 2006, 11:02 PM

QUOTE (MarkL @ Nov 20 2006, 05:47 PM)

Posted by: infocat13 Nov 21 2006, 01:59 AM

I remember the MER images of stars and martian moons from the surface, would enough of MGS show on such am image to better navigate the images from MRO ?

Posted by: volcanopele Nov 21 2006, 05:28 AM

QUOTE (infocat13 @ Nov 20 2006, 06:59 PM)

Posted by: Norm Hartnett Nov 21 2006, 05:27 PM

Apparently the wide angle shots showed something.

http://www.rockymountainnews.com/drmn/local/article/0,1299,DRMN_15_5158669,00.html

Keeping my fingers crossed.

Posted by: CosmicRocker Nov 21 2006, 05:59 PM

This brief video file description suggests MGS is dead. ...item #1...

ftp://ftp.hq.nasa.gov/pub/pao/tv-advisory/nasa-tv.txt

Posted by: djellison Nov 21 2006, 06:04 PM

-Dr. Michael Meyer
"We may have lost a dear old friend and teacher."
blah blah - what the instruments have done - blah blah...

Fuk LI
Activities in last two weeks.

On Noc 2nd 3.35 pm Pacific - telem showed more than 50 difficulties to move an array, automatically tried to switch to redundent systems at that time - 5.27pm should have come back, but no communication. In last two weeks - no communication with the spacecraft in a normal fashion. Only time we thought we might have heard from it - late 5th - early 6th - 4 partial orbits showing potential carrier only signal. After that - not heard a thing. Sent 800 command files - none of that has been succesfull.

Tried to use MRO - not easy - our knowledge of MGS location is not good. Do not know exact orientation so don't know how bright it will be. MRO just starting primary science phase and wanted to do everything to make sure we don't impose undue risk on MRO. Last friday used Star Camera. yesterday HiRISE and CTX to image a region where we thing MGS could be. Preliminary analysis has not yielded ANY sighting of the spacecraft. Today and tomorrow - send messages to MGS to turn on its UHF - have Opportunity listen and relay any info via Odyssey.

Not exhausted everything yet...."we believe the prospect of recover is not looking very good at all"

Still ongoing - but that's the 'meat' of it.

Doug

Posted by: Sunspot Nov 21 2006, 06:07 PM

Is the briefing over?

EDIT....OK I found it now

Posted by: MizarKey Nov 21 2006, 06:09 PM

I'm so mad! Our network at work is offlimits to RealPlayer...I cannot listen in, I'll have to wait to see if it comes out in Windows Media format.

Posted by: Norm Hartnett Nov 21 2006, 06:19 PM

News Release for ongoing briefing

http://www.nasa.gov/mission_pages/mgs/mgs-20061121.html

Posted by: djellison Nov 21 2006, 06:34 PM

Well - it's a bit odd that they have not found anything in the images...but they were not specific as to how much of the area in which they expect MGS would be in ( a one minute window in its orbit, roughly ) they have covered via CTX or the ONC. If they've imaged most of it, it would worrying that they have seen nothing. If they've only imaged a bit, it's not so worrying yet.

Doug

Posted by: Analyst Nov 21 2006, 06:54 PM

MGS has been the first Delta II launch I saw live on CNN. I still have it on tape somewhere.

If it's really gone, it's very sad, but contrary to the Viking days we have new spacecraft already at Mars and in the making.

Good bye MGS. It's the end of the beginning of the second exploration of Mars.

Analyst

Posted by: djellison Nov 21 2006, 06:55 PM

Last question was about how many images have been taken in the search - 750 Star camera images, 1 CTX image, 1 HiRISE image taken.

Doug

Posted by: Tom Tamlyn Nov 21 2006, 06:59 PM

Phone number for archived press conference: 866-513-1230. Unfortunately I didn't catch the international call-in number.

[Edit: the tape will be available for 1 week]

I thought it was a good press conference, with decent questions and responsive answers from well-informed panelists.


TTT

Posted by: djellison Nov 21 2006, 07:02 PM

Veronica Mcgregor ( Manager of the Media Relations Office at JPL ) has been kind enough to email me and let me know that the conference will be available for a week at....

TOLL FREE FROM WITHIN THE U.S.: 866-513-1230
INTERNATIONAL TOLL: 203-369-1973

Doug

Posted by: ElkGroveDan Nov 21 2006, 07:22 PM

I was pleased to hear that they are continuing to pursue their options with an upbeat attitue. I'm an insufferable optimist and in light of some of the "magic" that these folks have performed in the recent past, I am still not writing off MGS yet.

Posted by: Mariner9 Nov 21 2006, 07:39 PM

I don't wish to sound the pessimist, but it just seems to me unlikely that MGS is going to show up again. If they missed photographing it my understanding was that likely means there have been some thruster firings since loosing contact. Possibly the spacecraft was trying to orient itself or recover in some way.

My gut says that the vehicle is very possibly out of fuel and out of power.

It was a tremendous run. As people have pointed out, it totally changed how we saw Mars. I told many friends at the time that Pathfinder was great public relations, and a good engineering exercise, but MGS was where the real breakthroughs were going to come from.

If the only thing that mission accomplished had been a global topographic map from the laser altimiter, it would have been worth it. As it was, we got so much more.

Godspeed, MGS.

Posted by: diane Nov 21 2006, 07:48 PM

QUOTE (Mariner9 @ Nov 21 2006, 02:39 PM)

Posted by: Tom Tamlyn Nov 21 2006, 07:59 PM

QUOTE (Mariner9 @ Nov 21 2006, 02:39 PM)

Posted by: djellison Nov 21 2006, 08:21 PM

Operating normally, it's got fuel through to beyond 2010. Operating in various safe modes, 1 to 2 years.


As for figures..... well... until the second Shuttle SAR mission, MOLA had produced a better elevation map of Mars than we had of Earth.... it took more than 600,000,000 altitiude readings of Mars before the laser failed and it operated as a passive radiometer.

TES has taken dust loading, temperature and mineralogical readings for 5 martian years - collecting more than 206 million spectra.

MOC...well...wide angle and narrow angle added up - probably 250,000 images, the entire planet, every day, in red and blue, for weather monitoring with the wide angle - and then probably more than a million sq km of Mars covered at better than 4m/pixel resolution.

Can't find anything more specific for the Magnetometer and Electron Reflectometer.

Doug

Posted by: Myran Nov 21 2006, 09:30 PM

Yes I concur with Tom and djellison, attitude fuel should not be one issue. But energy are a major one, MGS cant operate on one solar panel only, assuming that the one with the problem now are misaligned.
If both are, well it seems quite unlikely but as pointed out on this forum already, Ulysses had a nasty energy situation and in addition its fuel frozen solid and it was still possible to retrieve it. So its still an open question, though the odds for one retrieval have gone up now.

Posted by: djellison Nov 21 2006, 09:33 PM

QUOTE (Myran @ Nov 21 2006, 09:30 PM)

Posted by: belleraphon1 Nov 22 2006, 12:44 AM

Well...

what can I say if MGS is finished!? I have such affection for that gal/machine.

I remember the long wait to get back to Mars after Viking and the awful consternation when Mars Observer bit the cosmic dust in 1993. That year I had been laid off from my job with two children to support.

Then, the wonderful launches of MGS and Pathfinder in 1996. And the horror of losing the Russian Mars 96 probes. That year I started working for one of the NASA field centers as a programmer for business apps.
Not the fun stuff but paid the bills and kept my kids fed.

I agree, that Pathfinder was just that, a pathfinder for MER (and wonderful in it's own sense to be ON THE GROUND AGAIN). But MGS was the gravy....... and anyone who wanders the MOC gallery cannot but be impressed artistically and scientifically with that incredible contribution to enlarging Mars for us all. MGS has truly introduced a NEW Mars.... and I will ALWAYS hold her in great esteem.

Thank you MGS and sleep well if sleep you must.

Craig

Posted by: John Flushing Nov 22 2006, 12:49 AM

http://today.reuters.com/news/articlenews.aspx?type=scienceNews&storyID=2006-11-21T214012Z_01_N20307841_RTRUKOC_0_US-SPACE-MARS.xml

Posted by: odave Nov 22 2006, 01:57 AM

Here's a groaner from the above Reuters article:

One chance remained on Tuesday to recover the probe, which has been programmed to transmit a signal to NASA's robotic geology station, Opportunity, located near Mars' equator.

Now, I suppose one could call Spirit a "station" at Low Ridge Haven before she moved, but Opportunity has hardly been stationary

I know, I know, mainstream media.....yadda yadda yadda....

Posted by: tedstryk Nov 22 2006, 02:02 AM

Not to mention "Mars Global Surveyor's cameras were the first to record topographic features suggesting flowing water on Mars,"

Mariner 9? Vikings 1 and 2? Mars 5?

Posted by: Jeff7 Nov 22 2006, 03:34 AM

I wonder what kind of a run MRO is going to have? With its greater bandwidth and resolution, think of how much data it could return in 10 years.

Posted by: tuvas Nov 22 2006, 06:42 AM

It's sure to be alot. MRO returned more data in it's first picture at Mars then Galileo during it's entire mission. It's sure to be an exciting time!

As for MGS, well, it's sad to see it go, but not all hope is lost (Just most of it), and it's certainly served it's purpose well. I've personally been looking at the HiRISE pictures, it's alot of work to carefully look for something out of the ordinary... There's still a glimmer of hope that there is something near the noise level that is MGS. FYI, the pictures weren't of the entire 2 minute uncertainty window that MGS currently has, only a few seconds of it, based off of two canadates from the nav. camera. Still, they are among the largest HiRISE images taken... Cheer up though, there might still be some exciting news from HiRISE in the short term future. The release of pictures was delayed in part due to the press conference today, and with the holiday in the US, but it will be coming shortly. MRO in many ways is an upgraded MGS, it will allow for photographing the entire weather patterns at Mars, and for high-resolution pictures anywhere, just like MGS, only it's spacial and spectral resolution will be better. That's not to say that MGS couldn't have done something more, only that it served it's purpose to the day that it was replaced, and not a day longer. Sad it is there couldn't be a bit more overlap, to compare the data sets a bit more, but it's something that will take MRO quite some time, to build up the great reputation that MGS left for us.

Posted by: Myran Nov 22 2006, 07:59 AM

Posted by: ugordan Nov 22 2006, 08:12 AM

QUOTE (tuvas @ Nov 22 2006, 07:42 AM)

QUOTE (Myran @ Nov 22 2006, 08:59 AM)

Drat! Always told myself I shouldnt post late at night.

Posted by: tuvas Nov 22 2006, 01:12 PM

QUOTE (ugordan @ Nov 22 2006, 01:12 AM)

If anything is an unfair comparison, that is it. Galileo with its crippled HGA and many times weaker signal since it was so much further away at Jupiter can't be compared to MRO. Comparisons with previous or current Mars missions do have sense on the other hand.
I find that posting early in the morning is much worse. And here I am...

Posted by: Marz Nov 22 2006, 05:07 PM

QUOTE (tedstryk @ Nov 21 2006, 08:02 PM)

Posted by: lyford Nov 22 2006, 07:43 PM

QUOTE (Marz @ Nov 22 2006, 09:07 AM)

Posted by: mcaplinger Nov 23 2006, 12:40 AM

Could we move the discussion unrelated to MGS to some other location?

Posted by: mhoward Nov 23 2006, 12:51 AM

Moved the MRO discussion to http://www.unmannedspaceflight.com/index.php?showtopic=3502.

Posted by: tasp Nov 23 2006, 03:16 AM

Ooooog,


don't get me started on the Galileo high gain antenna . . . .

Posted by: infocat13 Nov 23 2006, 08:07 AM

QUOTE (volcanopele @ Nov 21 2006, 01:28 AM)

Posted by: Zvezdichko Nov 24 2006, 07:23 AM

Opportunity did not detect any signal from MGS on Wednesday.

Posted by: djellison Nov 24 2006, 07:30 AM

I wasn't hopefull of that one - that would have required a power positive vehicle with multiple failures on the X-Band sdie which given the symptoms before the loss would seem unlikely.

Doug

Posted by: Zvezdichko Nov 24 2006, 07:50 AM

I was hopeful. A signal was received on November 5, after the eclipse which means that the failed panel isn't that badly positioned. The spacecraft should be power positive... Opoortunity should be able to pick a weak signal in that case.
hmmm, that's strange.

Posted by: Zvezdichko Nov 27 2006, 04:20 PM

I'm sorry for the double post ( I could just edit the previous message ), but more bad news is coming...

The star tracker of MRO has detected some light points that are not in the star catalog. Prelimitary results show that this could be pieces from Mars Global Surveyor, but they may also be gamma rays that impacted the quality of the images.

Posted by: djellison Nov 27 2006, 05:29 PM

from http://www.livescience.com/blogs/2006/11/27/mars-global-surveyor-mums-the-word/

Doug

Posted by: Sunspot Nov 27 2006, 05:57 PM

Maybe the panel broke off somehow... weird

Posted by: odave Nov 27 2006, 06:15 PM

Would the cracked panel yoke have given way?

Posted by: 4th rock from the sun Nov 27 2006, 06:59 PM

Hum... perhaps some small impact on the damaged panel? Or is my imagination too wild?
Anyway... a spacecraft doesn't just disappear!

Posted by: djellison Nov 27 2006, 07:31 PM

Ominous reminders of the CoNTour accident.

Doug

Posted by: volcanopele Nov 27 2006, 08:51 PM

My first thought was what Sunspot was thinking... if the star tracker did see chunks of MGS, maybe that solar panel broke off. Not sure how realistic that is though...

Posted by: djellison Nov 27 2006, 09:07 PM

It would be odd for an array to have become detached wouldn't it...even if a solar array gimble went rogue and thrashed around flat out for days ( which I can't imagine being possible ) it wouldn't rattle anything loose would it?

If there's MGS 'bits' drifting around, perhaps a more likely situation was a low power situation and some sort of rapid outgassing ( I hate to use the word explosion ) through fuel freezing or something like that? A burst prop line chucking a few mylar blankets around would be a little more plausable I would have thought.

Doug

Posted by: Sunspot Nov 27 2006, 09:08 PM

But IF MGS had been in the field of view of the star tracker cameras, are the MGS team sure it would have been visible for certain? If its not visible in the images, why is it so far from it's predicted position? Or, if it is still fairly close to its original orbit why wasn't it seen in the images? - All very strange

It's a shame the rovers are on reduced winter power, perhaps they could have attempted to image MGS as it flew over - if that were technically possible of course.

Posted by: djellison Nov 27 2006, 09:16 PM

MGS orbits at early-afternoon as far as the ground goes. Odyssey is a couple of hours later and thus sometimes visible at dusk, but it's just not possible for MGS.

Doug

Posted by: elakdawalla Nov 27 2006, 10:03 PM

If Leonard David is referring to what was said in the press conference last week, I'd've called it something even less strong than speculation. Here's what I wrote down that Jim Erickson said: "We believe we would be looking for either the intact spacecraft, but we did not discount that there might be other pieces out there, so we were willing to consider multiple hits. We adopted strategies of looking along the MGS track, and let the possible locations of MGS drift through the fields of view of our instruments...." I interpreted that to mean that while they had no reason to believe there were multiple pieces, they made sure that their observations would cover that possibility.

--Emily

Posted by: tedstryk Nov 28 2006, 01:00 AM

MGS had solar panel gimbal problems during the early part of the mission, which required the beta supplement orbits. Perhaps that mechanism finally snapped (or at least ceased to function).

Posted by: Analyst Nov 28 2006, 07:38 AM

If I remember correctly the reason for the beta supplement orbits has been with the HGA, not the solar array. The solar array caused slower/longer aerobreaking.

Analyst

Posted by: odave Nov 28 2006, 11:44 AM

That's the cracked yoke I was wondering about. Here's an excerpt from the http://www.jpl.nasa.gov/releases/97/mgsplan.html

"The investigation of the unexpected motion of the unlatched panel led us to identify a secondary source of damage in the yoke, a piece of structure that connects the solar panel to the spacecraft," Cunningham said. "This secondary source of damage was a result of the failure of the damper arm that jammed in the panel's hinge joint shortly after launch when the solar panels were initially deployed."

Mechanical stress analysis tests suggest that the yoke -- a triangular, aluminum honeycomb material sandwiched between two sheets of graphite epoxy -- probably fractured on one surface. The analysis further suggests that the fractured surface, with increased pressure on the panel during aerobraking, began to pull away from the aluminum honeycomb beneath it.

MGS made it through the rest of aerobraking just fine. If the yoke did finally break off, what kind of stresses would have made that happen during normal operations?

Posted by: Rakhir Nov 28 2006, 04:05 PM

Europe joins hunt for missing Mars probe
http://space.newscientist.com/article/dn10669-europe-joins-hunt-for-missing-mars-probe.html

The radio beam from MGS's antennas is about 60° wide, giving only about a 1 in 6 chance of it reaching Opportunity, since MGS's orientation is unknown.

Spirit, will try to detect the beacon, too. But Spirit will not have enough power to spare for this task until a few weeks from now, Thorpe says.

"We've asked the Mars Express people to take an image of MGS with their High Resolution Stereo Camera," Thorpe says, adding that the Mars Express HRSC team had agreed to make the attempt. The earliest opportunity is on 7 December 2006, when the two spacecraft should come within 400 kilometres of each other.

MRO is too busy to continue hunting for MGS. But if MEX can locate MGS, a case could be made for a second imaging attempt with MRO.

NASA is beaming commands to MGS from Earth every day in the hope of reviving it.

The two points of light in MRO images were in two very different orbits, so it's pretty unlikely that both could have come from the spacecraft.

Posted by: elakdawalla Nov 28 2006, 06:45 PM

I thought I'd point out that Thorpe gave me an update yesterday...

http://www.planetary.org/blog/article/00000778/

Also, he told me earlier that the problem panel is indeed the same one that caused the lengthy aerobraking period. However, he said it was a different part of the panel that was reporting problems this time.

--Emily

Posted by: AlexBlackwell Dec 5 2006, 08:06 PM

Ten posts dealing with http://www.nasa.gov/home/hqnews/2006/dec/HQ_M06186_Mars_Briefing.html were moved to a http://www.unmannedspaceflight.com/index.php?showtopic=3559&hl= in the same (MGS) forum.

Posted by: AlexBlackwell Dec 13 2006, 12:46 AM

http://www.livescience.com/blogs/2006/12/12/mars-global-surveyor-still-silent-yet-hope-remains/
Leonard David
LiveScience.com Blog
December 12, 2006

Posted by: Zvezdichko Dec 13 2006, 06:52 PM

Any news regarding the images of HRSC?

Posted by: tuvas Dec 15 2006, 06:27 AM

I just found out that the MRO star tracker successfully imaged Odyssey, so there's still a chance that it could image MGS. If it manages to successfully image MGS, then HiRISE will follow suit.

Posted by: ustrax Dec 16 2006, 06:37 PM

QUOTE (elakdawalla @ Nov 28 2006, 06:45 PM)

Posted by: jaredGalen Dec 20 2006, 06:37 PM

Anyone have more details on this?? MGS possibly seen, perhaps tumbling?

http://www.livescience.com/blogs/author/leonarddavid

Posted by: tuvas Dec 20 2006, 06:46 PM

QUOTE (jaredGalen @ Dec 20 2006, 11:37 AM)

Posted by: elakdawalla Dec 20 2006, 06:49 PM

A tumbling spacecraft is an out-of-control spacecraft.

--Emily

Posted by: djellison Dec 20 2006, 06:52 PM

Well - tumbling would suggest ACS failure - and ACS failure (for whatever reason) would almost certainly mean a spacecraft that was not power positive - and MC's words on that were that unless it was power positive, it was good-night Mr Bond.

But - conversely - SOHO was found to be tumbling back when it played truent for 6 months - and was saved ( even from a point when all its fuel had frozen solid ).

However - I think MGS is probably lost - I just hope that HiRISE can get a picture of MGS...even if it's a hard sequence to get right, and a bandwidth whore ( black should compress really well ) - I think it's an image we would all want to see (and one that would make it into the main stream press as well )

Doug

Posted by: tuvas Dec 20 2006, 06:58 PM

QUOTE (elakdawalla @ Dec 20 2006, 11:49 AM)

A tumbling spacecraft is an out-of-control spacecraft.

--Emily

Posted by: djellison Dec 20 2006, 07:16 PM

Problem is - you could have tumbling in an axis that had zero power - or full power...but more likely somewhere inbetween. Call it an average of Half Power. But that half power is over half the orbit - so you never have a charged battery going into eclipse, and always come out with it flattened. Then, if you don't get a good enough angle on the arrays next time around - you end up with a cold spacecraft, a flat battery, and that's when you have actual failures of the hardware required to keep things under control.

Depending on what sort of orientation it is in and what that spin is like - it may well be that after a few months - everything rotates around the sun to give the old girl enough power to wake up and perhaps tell us what's wrong...but it would be a long shot.

Doug

Posted by: Zvezdichko Dec 21 2006, 08:31 PM

That's pretty strange. We had contact on 5th November which means that the batteries were charged... that probably means that ... we don't have zero power. So we could still have a chance...

Posted by: gpurcell Jan 9 2007, 04:31 PM

From NasaWatch's LiveBlog on MEPAG

"We think that failure that a software load we sent up in June of last year was the cause. This software tried to synch up two flight processors. Two addresses were incorrect - two memory addresses were over written. As the geometry evolved. We drove the arrays against a hard stop and the spacecraft went into safe mode. The radiator for the battery pointed at the sun, the temperature went up, and battery failed. But this should be treated as preliminary."

If true, a sad end to a magnificent mission.

http://www.nasawatch.com/archives/2007/01/mepag_meeting_i.html

Posted by: PhilCo126 Jan 9 2007, 06:46 PM

NASA still didn't announce an official " RIP MGS " ... correct?

Posted by: AlexBlackwell Jan 10 2007, 06:20 PM

QUOTE (gpurcell @ Jan 9 2007, 06:31 AM)

Posted by: Sunspot Jan 10 2007, 07:27 PM

Oh Dear............... human error

Posted by: Bob Shaw Jan 10 2007, 09:37 PM

QUOTE (Sunspot @ Jan 10 2007, 07:27 PM)

Posted by: AlexBlackwell Jan 10 2007, 09:39 PM

QUOTE (Bob Shaw @ Jan 10 2007, 11:37 AM)

Posted by: AlexBlackwell Jan 10 2007, 10:07 PM

http://www.jpl.nasa.gov/news/news.cfm?release=2007-004
NASA/JPL
January 10, 2007

Posted by: Lorne Ipsum Jan 10 2007, 11:24 PM

More accurately, it was a parameter upload error (somewhat similar to the error that killed one of the Viking landers). I'm in the process of writing up a blog post to explain, should be up later tonight / early Thursday...

Lorne
http://geekcounterpoint.net

Posted by: Zvezdichko Jan 11 2007, 02:26 PM

Space agencies more like miles not meters *crash*

It seems that faulty software has doomed more than half of Mars spacecraft.
Firstly it was Viking ( bad antenna positioning )
Secondly it was Mars Climate Orbiter
Thirdly it was Mars Polar Lander and these spurious signals.
I'm not counting Phobos Spacecraft...
I don't know why it's always software. We almost lost Spirit three years ago...
It really makes me sad.

Posted by: odave Jan 11 2007, 03:10 PM

It's the nature of the software beast. I'm a software engineer at an industrial robotics company, and I've been told by a mechanical guy that he detests software because you can't see it or measure it. To him, it's black magic that can fail for no apparent reason. Code can be hideously complex, and even if you test the snot out of it before deploying, it seems like there's always one oddball set of circumstances or sequence that nobody even dreamed of encountering that happens almost immediately (and usually to your most important and sensitive customer )

I've encountered unexpected memory address overwrites in our stuff, and they don't always show up during testing for a variety of reasons. It may be that the test cases didn't create a situation where the corrupted memory was accessed, or that the data that was written to the wrong memory locations is benign at the time of the test. I would assume that JPL's testing is much more rigorous than ours since the stakes are so much higher, but it's really hard and time/fund consuming to test for absolutely everything. So yes, a sad end for MGS if this was the case, but hopefully they can learn from it and improve the testing process.

Posted by: Zvezdichko Jan 11 2007, 03:15 PM

A little offtopic but...
I'm very concerned about the future and Phoenix. I don't see how spurious signals could be avoided. We have two successful landers ( Viking 1&2 ), and one failure ( MPL ). Actually, we don't know the exact reason for the failure ( for both MPL and MGS ), this is just a likely scenario.
Any news on the latest december attempt with HiRiSe? ESA said that they ( may ) have detected a tumbling MGS?

Posted by: djellison Jan 11 2007, 03:20 PM

5 landers..V1, V2, MPF, MERA, MERB - all used radar.

Doug

Posted by: ugordan Jan 11 2007, 03:22 PM

One also has to consider that it's software, not hardware that actually "thinks" for the spacecraft. Hardware processors, as complex they may be, have straightforward instruction sets and architecture that can be tested pretty well (though remember that Pentium bug years ago...). Processors are dumb pieces of electronics that expect to be told what to do. Software is what makes the thing "tick" and it's vastly more complex than what is essentially a state machine and a powerful calculator. Were you to develop a processor that did all the thinking by itself, it'd still be bugged because it was designed by humans. Complex tasks mean complex things might happen. They might not always be what you expect. You expect and hope they be, you can test the hell out of the system, but there are always gremlins hiding somewhere. You can't test everything; remember: even test cases are created by humans!

Posted by: Zvezdichko Jan 11 2007, 03:29 PM

I have some information about processors on spacecraft. The statement however is not a processor failure, but overheating of the batteries ( which means death of a spacecraft ).
The previous statement of a tumbling spacecraft could mean at least two things. The spacecraft has lost control after overheating. Or after problems with the solar panel we had improper turn-over of the spacecraft.
Am I right? ( just trying to guess)

Posted by: mcaplinger Jan 11 2007, 03:41 PM

QUOTE (Zvezdichko @ Jan 11 2007, 06:26 AM)

Posted by: djellison Jan 11 2007, 03:41 PM

Some sort of software/commanding problem caused...

A bad attitude which heated up the battery radiator which caused...

Battery failure which caused...

Loss of vehicle, as I understand it so far.

Posted by: Littlebit Jan 11 2007, 05:00 PM

QUOTE (mcaplinger @ Jan 11 2007, 08:41 AM)

Posted by: lastof7 Jan 11 2007, 05:14 PM

I feel for the MGS software team if it turns out to not be a direct software issue. It's understandable that NASA management and the public want to know as quickly as possible the root cause of a fault, but, having experienced similar situations, it's painful to see headlines like http://space.com/news/070110_mgs_softwareglitch.html before we have a definitive answer. Unfortunately, it may be too late to correct the impressions that have been made if it is a parameter issue or something of that nature.

Posted by: climber Jan 11 2007, 07:25 PM

It's a wellknown fact that the majority of car accidents occur on the road you know the better. You can call that statistics or lack of concentration. The longer a mission goes, the more likely an human error will occur. I'm just amazed how long the Voyagers have flown, it'll be 30 years this year.
To the software people : we back you guys, habit is a bad thing and people only remember your failures. We just CAN't fly without you.

Posted by: tedstryk Jan 11 2007, 10:47 PM

QUOTE (Littlebit @ Jan 11 2007, 05:00 PM)

Another English to Metric conversion?

Given the shear number of human interactions with the MGS, it is a extraordinary accomplishment of the MGS team to have kept the ball in the air this long. In a way it is like playing Tetres: No matter how great you are, the result of every mission without a firm time-line will be a failure of some sort, usually human...no matter how super human the effort:)

I hope they will be candid and timely in providing a detailed description of the failure and the lessons learned. Knowing the reason that an un-timed mission failed is one more mission success.

Posted by: Lorne Ipsum Jan 12 2007, 12:30 AM

Gang,

This might help explain things a bit:

http://geekcounterpoint.net/files/GC052B.html

Lorne

Posted by: climber Jan 12 2007, 12:52 AM

QUOTE (Lorne Ipsum @ Jan 12 2007, 01:30 AM)

Posted by: nprev Jan 12 2007, 12:55 AM

Absolutely superb & highly educational analysis, Lorne; thank you VERY much!

The bottom line is that many parts of this read exactly like every aircraft accident report I've ever read: there is always a chain of events that increases unknowns and ultimately leads the entire system (including the human element) into an uncontrollable situation with basically unpredictable, often undesirable outcomes.

I sure hope that the MGS software team member(s) involved near the end don't feel too bad; they shouldn't. Aside from the brilliant performance of the spacecraft that vastly exceeded all reasonable dreams before launch, complex systemic failures just plain happen. They seem to be an inevitable feature of the Universe, and I'm sure that the mathematics of chaos theory could easily prove this.

Posted by: Lorne Ipsum Jan 12 2007, 03:16 AM

nprev & climber,

Thanks -- glad you liked the writeup! I'm with you -- hopefully the poor guy at the bottom of the totem pole doesn't get beat up too severely over this (he'll be reliving it for the rest of his life anyway).

I've worked mission ops for old spacecraft with static memory maps before, and I remember how we ALWAYS got paranoid whenever we did parameter updates. When push comes to shove, the fact that a mistake like this could go unnoticed for months says there's a bad process being followed (or a good one not being followed) somewhere. Hopefully the review board can come up with some lessons that can be applied to more modern architectures.

Lorne

Posted by: stevesliva Jan 12 2007, 03:56 AM

Thanks for the extremely informative blog!

I am unsure about one thing though. Towards the end, in "the spark that lit the fire," you do not mention when MGS was switched back to SCP-1. Was this part of the safe mode? Or had it already been transitioned back to SCP-1? And if the transition was an intentional switch back, I tend to agree that at least some process should have caught the bad parm upload. (ie a comparison of the two memories) But if the switch back was a result of a safing event before the SCP-1 memory repair was fully verified, well, that just sucks but is less faultworthy.

Posted by: helvick Jan 12 2007, 10:19 AM

Lorne - superb write up, one of the best bits of reporting on spacecraft ops I've ever come across. Any chance you're available to help the BBC out as they seem to be in need of a major quality control overhaul at the moment?

Posted by: edstrick Jan 12 2007, 11:45 AM

Viking's case involved a thrown-together set of people from the disbanded engineering and software team. VL1 was on an automatic "eternal" mission that was hopefully not going to require any further commanding ever. They were trying to salvage or extend the mission by uploading battery conditioning commands as the battery started to show similar problems to the VL2 batteries that killed that lander's operations.

Note that the Magellan Venus radar mapper mission was nearly lost early on due to a high-lethality interrupt handling error that could send the computer essentially into runaway crashes. They finally "trapped" the error when the ground duplicate test system did a interupt fault and crashed while full diagnostic info was available.

I'm deeply unhappy with trusting in software driven "safe modes", preferring that the spacecraft be able to fall back into an ultimate nearly lobotomized mechanical safe mode. Remember, Pioneers 10 and 11 never had a software problem, never rebooted, never crashed. No computers. All the way beyond Pluto on direct commands (except for sequencer stored commands for midcourse maneuvers).

I'm also deeply unhappy with spacecraft inside of Jupiter's orbit that do not have essentially 100% omnidirectional coverage with low data rate omni-antennas. We nearly lost the ability to command Mariner 10 when it was being stabilized in a drifting roll mode and it rolled into a null in the receiving antenna pattern shortly before the third Mercury encounter. We also had problems with Magellan getting into nearly communication-unable attitudes during one or more of it's computer crash crises. You really want to get 8 bits/second telemetry as long as a spacecraft has power and live command decoding circuits, and the ability to send 1 bit/second commands.

Posted by: Guido Feb 13 2007, 01:42 PM

QUOTE (Lorne Ipsum @ Jan 12 2007, 01:30 AM)

Posted by: PhilHorzempa Feb 15 2007, 10:39 PM

QUOTE (Lorne Ipsum @ Jan 11 2007, 08:30 PM)

Posted by: Littlebit Feb 16 2007, 03:37 PM

Somewhere - but I cannot find where - I read Lorne's scenario was not likely to be correct. This may be why the article was pulled, which is too bad, because it was a very good description of MGS era computer systems.

In any case, it will be disappointing if yet another 'successful mission unplanned ending' investigation is kept under wraps.

Posted by: PhilHorzempa Feb 16 2007, 05:46 PM

Lorne,

Please let us at UMSF know what happened to your
MGS software article on Geek Counterpoint. It was
an excellent presentation and helped a lot of us who
may not know software as well as you, but are technically
informed enough to comprehend the issues.

If there are questions as to whether this is what really
ended the MGS mission, then please consider re-posting
an edited version of the article that omits that conclusion.
It was fascinating to catch this glimpse into a crucial
aspect of unmanned exploration. As I believe someone
else has already said, our robot explorers do exactly what
we tell them. The unfortunate thing is that sometimes
we don't realize what we have told them.


Another Phil

Posted by: elakdawalla Apr 13 2007, 03:44 PM

The preliminary report is out, and it sounds like what Lorne described.
Here's the report:
http://www.nasa.gov/pdf/174244main_mgs_white_paper_20070413.pdf

--Emily

Posted by: djellison Apr 13 2007, 06:59 PM

I hope someone asks what the projected remaining on-orbit lifespan of the spacecraft was before it went awol - that tells us the true value of the loss really.

(And guess who got in with the first question - a great one about orientation...nice one ESL - I hope you can manage a trademark timeline of events to break it all down )

Damn - I missed the last 5 minutes.

Doug

Posted by: elakdawalla Apr 13 2007, 09:57 PM

I've now posted a story on the review board report.

http://planetary.org/news/2007/0413_Human_and_Spacecraft_Errors_Together.html

--Emily

Posted by: brellis Apr 14 2007, 03:26 AM

QUOTE (elakdawalla @ Apr 13 2007, 02:57 PM)

Posted by: nprev Apr 14 2007, 03:43 AM

An absolutely classic 'chain of mistakes/events' scenario, all too familiar from aircraft accident accounts. Excellent reporting, Emily, and thanks!

There are indeed many lessons to be learned here. The main one is that configuration control is an imperative. Two different groups should never have been responsible for maintaining identical spacecraft software-driven bus functions; that's inviting disaster right there.

Posted by: helvick Apr 14 2007, 08:59 AM

I hope http://geekcounterpoint.netwill re-instate his analysis now too - my recall of the article was that it was fundamentally correct and his explanation of the challenges involved in the "simple" day to day management of MGS systems was enlightening.

Posted by: edstrick Apr 14 2007, 10:19 AM

There is a real need for a computer controlled spacecraft to be able to declare "utter dire emergency" and nearly lobotimize itself, switch to a hopefully nearly bulletproof safety control system and safe itself. There's an increasingly long list of lost, nearly lost, and compromized missions where vehicles couldn't properly safemode (Magellan's computer system crashes and NEAR's pre-orbit-insertion burn screwup at Eros) etc.

Pioneer Jupiter missions never had a computer crash and safemode emergency EVER... (no computer)... The missions were done entirely by direct ground command except for turn and burn stored commands in a sequencer for midcourse maneuvers.

Posted by: MarsIsImportant Apr 14 2007, 12:29 PM

It seems to me that part of the solution is they need to redefine what safe mode is. What they thought was safe mode was actually self-destruct mode. ...Of course, I understand that it's not quite that simple.

Posted by: mcaplinger Apr 14 2007, 02:35 PM

QUOTE (edstrick @ Apr 14 2007, 03:19 AM)

Posted by: elakdawalla Apr 14 2007, 05:44 PM

I totally agree, Mike. One of the questions I wasn't able to get an answer to, which I would have liked to include in the article, was: how many times did MGS encounter a fault, enter safe mode, and recover successfully because its fault protection worked? Its 10 years were made possible by lots of "lessons learned" from previous missions, and its demise, though sad, does give designers insight into a whole 'nother set of potential faults that they can now plan for, and help make sure it never happens to another mission.

Until robots really do become intelligent, I fear it's much more likely for a long-lived mission to fail unexpectedly due to some bizarre chain of unforseen events that human programmers just didn't plan for, than for the mission to fail for purely mechanical reasons. It seems to me that we now make plans to end missions before they fail for mechanical reasons, and deorbit them or take some other such protective action. But you just can't plan for every possible human error. You just have to try to plan for everything that's remotely likely. They just didn't plan for this particular bizarre string of events.

--Emily

Posted by: mcaplinger Apr 14 2007, 08:52 PM

QUOTE (elakdawalla @ Apr 14 2007, 10:44 AM)

Posted by: nprev Apr 15 2007, 12:36 AM

Emily, "bizarre strings of events" are almost always how mishaps occur in aviation & probably in every other field of endeavour as well. Good systems engineering strives to minimize design features that might induce single-point and at least some chained failures, but ultimately in the real world external systemic influences add many layers of complexity (and often thousands of variables) that can never be completely controlled. This is concisely and quite accurately summarized in pop culture as "**** happens", of course...

I am convinced that this is a fundamental heuristic of the Universe, and unfortunately probability implies that the most unlikely chain of events will someday occur to induce an uncontrollable amount of entropy into any given system, thus making its future behavior impossible to predict with accuracy. The MGS ground team did nothing fundamentally wrong; in fact, despite the prima facie tone of my previous post, I meant no criticism of them at all. Lessons learned to realize small single-point improvements is all we can do; entropy will always win in the end, despite our best efforts.

Posted by: brellis Apr 15 2007, 02:54 AM

nprev said:

"I am convinced that this is a fundamental heuristic of the Universe, and unfortunately probability implies that the most unlikely chain of events will someday occur to induce an uncontrollable amount of entropy into any given system, thus making its future behavior impossible to predict with accuracy. The MGS ground team did nothing fundamentally wrong; in fact, despite the prima facie tone of my previous post, I meant no criticism of them at all. Lessons learned to realize small single-point improvements is all we can do; entropy will always win in the end, despite our best efforts."

I have an abiding and long-lived personal fascination with the concept of entropy, because I've been friends with the http://losangelesentropy.com/ Humans trying to define laws of nature become enforcers of those laws. "The system as defined is now perfect, and it will work indefinitely into the future" is a frustrating mental block in the human effort to define the universe and reform it in our own image.

If entropy killed MGS, it was just tiny particles of entropy. Entropy on a grander scale would turn MGS, Mars and the entire human endeavour into some kind of mush, would it not?

Posted by: ElkGroveDan Apr 15 2007, 03:15 AM

QUOTE (brellis @ Apr 14 2007, 06:54 PM)

Posted by: nprev Apr 15 2007, 03:39 AM

To continue the nihilism: a cold vacuum of slowly decaying protons amidst a sea of barely energized leptons several trillion years from now...

No matter. We all do the best we can, and the MGS team performed WAY beyond any initial expectations. (How inspiring, how refreshing, to see such magnificent dedication, brilliance, and innovation to make this mission last so long, yes? This is the spirit of humanity at its very best.) Each and every one of them should get a medal as far as I'm concerned for making truly significant contributions to human knowledge and exploration. I envy them the private satisfaction they each must feel for doing something that really meant a great deal, not just now but a thousand years from now...

Posted by: brellis Apr 15 2007, 05:43 AM

QUOTE (nprev @ Apr 14 2007, 08:39 PM)

Posted by: cndwrld Apr 16 2007, 08:48 AM

I read the anomaly report that NASA put out, and I can follow what happened based upon similar experience. I've screwed up enough while at the console to understand what happened.

From some of the comments here, people may have a very different picture of how the usual (old) spacecraft console software works as compared to the reality. I'll try to make a few points, and maybe people can tell me if I'm missing anything.

In my experience, the old console software is not very high tech. The projects are always run by hardware guys, who don't know much about software. And the pressure in the programs, before launch, are almost always hardware driven. So, you end up with software which is not exactly state of the art, being used to control hardware which often IS state of the art.

Most of the old ground software I've used is very manual. So for instance, I can understand exactly how the report's errors occurred. There is a command prompt that asks you for the value of the parameter you want to change, and the memory address in RAM. You type it in. You don't change the redundant side's values at the same time, until you know it worked on the primary. So later, another guy repeats it for the redundant side. He types it in. But he types it in different than the first guy. Error 1.

Later, you do a memory dump. These were generally crude tools that spit out pages and pages of hard copy, in hex, with very little technology to help you make sense of it. It is fingers moving over the page, finding values in two places that match (or don't). But as usual, they found the problem. Good job, team.

Now they do it all again. Run the memory update program, enter the addresses of the parameters by hand, then enter the correct value by hand. But in this case, the parameter was entered correctly this time, but they typed in the address wrong. Error 2. I've done that. And it isn't pretty. Generally, the old console software won't catch it. It will do whatever you tell it to do, and put whatever you want in any location. There are no limit checks, no graphical displays to show you in what location these parameters are actually going. There's nothing to back you up. So you should have people double check what you're doing. But into your fourth mission extension, it may not seem that important. Your computers have been shoved into some corner to make way for more important things, you have people working part time whose real focus is on other things, none of your managers are paying attention to the mission anymore, so nothing you do on it seems like it is going to exactly help your career. The edge, you could say, is missing, and inevitably things happen. Usually recoverable, sometimes not.

You are not working with changes in operating systems. No one changes the operating system once it is launched, unless you do a patch to fix a serious problem. You do everything in your power to forget you have an operating system. You just work with parameters, whenever possible. And you change parameters by making direct writes of numbers to specified memory locations, all of which are entered manually at a prompt. Type in either a bad address, or a bad parameter value, and if it goes through unnoticed, you have a time bomb in your RAM. A memory location where the parameter should be between X and Y, and you just put in a value of Z = 3Y.

A lot of these problems with the ground systems are now fixed, with missions starting off with much better ground systems than the older missions had. MGS launched in 1996; the ground system software was locked down at least six months or a year before that. The software was probably based on designs from the early nineties. As with everything else, the ground software has changed a lot between 1992 and 2007. And it has changed because of exactly the kinds of errors that got made on MGS. But since no program every spends budget to improve the working software of old missions, things like this can happen.

As for safe modes, keep in mind that any spacecraft safe mode is designed to handle a single fault. No one even attempts two-fault solutions, because anything beyound single-fault planning gives you an almost infinite number of possibilities to plan against, which cannot be done on the budget you have. And when you enter safe mode, the flight code uses defined parameters in RAM. You can have a perfectly lovely safe mode definition, but if the parameters have been corrupted, all bets are off; anything can happen. If you think that using some sort of safe mode that is absolutely hard coded would be safer, I would disagree. Things are learned after launch, often very very disturbing things. Having the flexibility to alter the parameters is much safer than not. And this flexibility allows you to tailor the safe mode to things like failed hardware, which cannot be planned for in advance.

There is talk about how the lessons learned from this will include periodic end-to-end reviews, looking into how the manned program does things, and ways to keep the operators fresh and enthused. Well, end-to-end reviews that will actually be detailed enough to catch parameter discrepancies are long, detailed, require experts who are working on current programs with tight deadlines and budgets, and require money to fund them. The human spaceflight side has a lot more money for these things, because lives are at stake. Unmanned missions get their fourth extension based on the fact that they promise to spend almost no money at all, otherwise the spacecraft would have been shut down and hurtled into the planet. These are the sorts of things managers say at times like this, but when it comes down to funding them, count me as quite sceptical. New missions will take priority for the cash. And sometimes, that is the right decision.

There are people out there who know a lot more about the MGS specifics than I do. If I'm way off, let me know. But this was my take on the whole thing, for what its worth.

Posted by: nprev Apr 16 2007, 02:56 PM

Interesting and valuable insight from someone who's been there, cndwrld...thank you!

There's a lot of trade space between flexibility & foolproofing in human/machine interface, but in your general examples it sure sounds like the bias is sometimes set too far to the former. Setting up a table of parameters in MS Access or something for each of the redundant databases & then continuously comparing them for equality (and flagging fields that don't match) doesn't seem too hard or expensive to build.

Foolproof? No, nothing really is. I'm sure that many if not most SV operators do something exactly like this, and bad things still happen.

Posted by: dvandorn Apr 16 2007, 03:37 PM

That was an excellent summary, Don -- it demonstrates what I've been saying all along, that the limitations on most every human endeavor have more to do with financial and schedule pressures than they do with the limits of our technology or imagination.

Now, as we all know, there are a lot of ways to automate the processes you discuss. Heck, back in Gemini days, more than 40 years ago, command loads to the Agena target vehicles were sent up pretty much exactly as you describe, here. But even back then, they had an automatic comparator that would check the command load as sent against the command load as received by the Agena. Only when that comparator failed did they end up digging through printouts of the command loads to verify that the load was properly received.

Now, that's not exactly the same as comparing an actual command load to a desired command load, but its similar in process. And thus, the technology to error-check a lot of this stuff has been around for a long time.

As you have so effectively pointed out, the ground support stuff is usually designed (or used off-the-shelf) to do its job, bare-bones, no extras. Error trapping is almost non-existent.

And lest anyone think that this is just an issue with ESA's efforts, recall that an average command load to the MERs requires most of an individual's workday to prepare -- seven or eight hours. We all know it's *possible* to create error-trapping front-end software for such things that would allow a rover driver to tell the front-end interface: "We want to drive 20 meters in this specific direction, take the following image series, and then prepare for an overnight Odyssey pass." It's very possible to set it up so that creating and radiating the appropriate command series would take the rover driver 10 or 15 minutes, and the front-end would ensure that all commands sent to the spacecraft would be safe and properly executable. Why isn't it done like that? Probably because it would have cost too much in time and money to develop such a front-end system in the first place...

-the other Doug