MGS in Trouble, Formerly: MGS in safe mode Options

[gpurcell]

From NasaWatch's LiveBlog on MEPAG

"We think that failure that a software load we sent up in June of last year was the cause. This software tried to synch up two flight processors. Two addresses were incorrect - two memory addresses were over written. As the geometry evolved. We drove the arrays against a hard stop and the spacecraft went into safe mode. The radiator for the battery pointed at the sun, the temperature went up, and battery failed. But this should be treated as preliminary."

If true, a sad end to a magnificent mission.

http://www.nasawatch.com/archives/2007/01/..._meeting_i.html

[Guest_PhilCo126_*]

NASA still didn't announce an official " RIP MGS " ... correct?

[Guest_AlexBlackwell_*]

If true, a sad end to a magnificent mission.

There are couple of other words that I might use in addition to "sad."

[Guest_Sunspot_*]

Oh Dear............... human error

[Bob Shaw]

Oh Dear............... human error

It sounds like 'safe mode' wasn't. The whole idea of safe mode is that the spacecraft gets a breathing space while the humans look at the spaghetti code, but in this instance safe was not the word. The real question to me isn't so much why some commands were wrongly written, as why the lifeboat had a hole in it.

MGS was a fine old bird! Better to go out in action than simply to be switched off because of budget pressures.


Bob Shaw

[Guest_AlexBlackwell_*]

It sounds like 'safe mode' wasn't. The whole idea of safe mode is that the spacecraft gets a breathing space while the humans look at the spaghetti code, but in this instance safe was not the word. The real question to me isn't so much why some commands were wrongly written, as why the lifeboat had a hole in it.

Admittedly, there is a dearth of details available, but assuming there is a failure review report, I'd be interested to see how something like this wasn't caught in the testbed.

[Guest_AlexBlackwell_*]

Panel Will Study Mars Global Surveyor Events
NASA/JPL
January 10, 2007

[Lorne Ipsum]

More accurately, it was a parameter upload error (somewhat similar to the error that killed one of the Viking landers). I'm in the process of writing up a blog post to explain, should be up later tonight / early Thursday...

Lorne
Geek Counterpoint

[Guest_Zvezdichko_*]

Space agencies more like miles not meters *crash*

It seems that faulty software has doomed more than half of Mars spacecraft.
Firstly it was Viking ( bad antenna positioning )
Secondly it was Mars Climate Orbiter
Thirdly it was Mars Polar Lander and these spurious signals.
I'm not counting Phobos Spacecraft...
I don't know why it's always software. We almost lost Spirit three years ago...
It really makes me sad.

[odave]

It's the nature of the software beast. I'm a software engineer at an industrial robotics company, and I've been told by a mechanical guy that he detests software because you can't see it or measure it. To him, it's black magic that can fail for no apparent reason. Code can be hideously complex, and even if you test the snot out of it before deploying, it seems like there's always one oddball set of circumstances or sequence that nobody even dreamed of encountering that happens almost immediately (and usually to your most important and sensitive customer )

I've encountered unexpected memory address overwrites in our stuff, and they don't always show up during testing for a variety of reasons. It may be that the test cases didn't create a situation where the corrupted memory was accessed, or that the data that was written to the wrong memory locations is benign at the time of the test. I would assume that JPL's testing is much more rigorous than ours since the stakes are so much higher, but it's really hard and time/fund consuming to test for absolutely everything. So yes, a sad end for MGS if this was the case, but hopefully they can learn from it and improve the testing process.

[Guest_Zvezdichko_*]

A little offtopic but...
I'm very concerned about the future and Phoenix. I don't see how spurious signals could be avoided. We have two successful landers ( Viking 1&2 ), and one failure ( MPL ). Actually, we don't know the exact reason for the failure ( for both MPL and MGS ), this is just a likely scenario.
Any news on the latest december attempt with HiRiSe? ESA said that they ( may ) have detected a tumbling MGS?

[djellison]

5 landers..V1, V2, MPF, MERA, MERB - all used radar.

Doug

[ugordan]

One also has to consider that it's software, not hardware that actually "thinks" for the spacecraft. Hardware processors, as complex they may be, have straightforward instruction sets and architecture that can be tested pretty well (though remember that Pentium bug years ago...). Processors are dumb pieces of electronics that expect to be told what to do. Software is what makes the thing "tick" and it's vastly more complex than what is essentially a state machine and a powerful calculator. Were you to develop a processor that did all the thinking by itself, it'd still be bugged because it was designed by humans. Complex tasks mean complex things might happen. They might not always be what you expect. You expect and hope they be, you can test the hell out of the system, but there are always gremlins hiding somewhere. You can't test everything; remember: even test cases are created by humans!

[Guest_Zvezdichko_*]

I have some information about processors on spacecraft. The statement however is not a processor failure, but overheating of the batteries ( which means death of a spacecraft ).
The previous statement of a tumbling spacecraft could mean at least two things. The spacecraft has lost control after overheating. Or after problems with the solar panel we had improper turn-over of the spacecraft.
Am I right? ( just trying to guess)

[mcaplinger]

It seems that faulty software has doomed more than half of Mars spacecraft.

The VL1 and MCO cases are not what I would call software faults. In the Viking case, ground controllers commanded things with raw memory writes instead of a higher level command protocol, and they inadvertently wrote into the wrong locations. You could argue that they should have had better software, but the software they did have was working as it was supposed to -- it was operator error. The MCO loss was more a process problem, stimulated by a simple calculation error. Nor is the MPL failure a pure software error -- it was a miscommunication between hardware and software design. Of your examples, only the Spirit flash anomaly was what I would call a pure software error, and it was recoverable via other software.

I can't discuss the MGS failure because unlike some other people on this forum, I was too straightforward in my choice of user name and can't speak anonymously