I just stumbled across this presentation on the Web...some of this stuff I knew about, but not all of it. In retrospect I can't believe how calm Mark always seemed, on the surface, as the rovers were approaching EDL.
Mars Exploration Rovers: The Ugly Truth Behind the Pretty Pictures (PDF format)

--Emily

And the truth behind the 'cleaning events' is revealed at last!

Bob Shaw

Nice "stuff" Emily....Verry dramatic...


UUUUUUHHHHAAAAHAHAHAHHAHAHAA!!!!

Thanks for sharing that weblink ... Just Plain Lucky (JPL)

Great stuff, Em!
JPL has the basis here of a 'SLAM! BAM! POW!' comic book series that would appeal to kids of all ages. Just imagine Steve with a big red S on his shirt.

Hi Emily,
Mark's presentation only captured some of the many trials and tribulations that personified MER development. (the static images in his presentation are the first frames of movies). Some other engineering nightmares that Mark left off include:

-Plans for re-use of the Athena pancam mast assembly from the aborted MSP'01 lander falters when the parachute canister grows (down) onto the stowed mast fforcing a complete redesign.
-Mis-routing the Pancam and hazcam's wires that crossed the mast rotary joint causes weird pictures
-Rover standup testing on the EDU rover fails. Need new lub job and a lot of margin tap dancing around the lifting screw.
-The power supply on the Rover's (can't remember which - I think MER-2) mobility potentiometers (measures angles) shorts out at low temp. somewhere. on the rover. somewhere. never found out where.
-Some of the lander petal actuator's magnetic brakes fail to engage, petals droop when open. Hunt for good ones.
-cable cutters (part of the electicity and knives not mixing issue). where do I start?? Two months to lift off, found a nifty way to take the rover batteries off line. automatically. during EDL. when the backshell cables were cut, short circuit through the cutter back into the ground support equipment battery safety input control. -Also found that we may have blown the "ballast" pyro circuit resistor (before fuse was blown). No time to fix, we need to prove that all of the pyro's had high residual impedance - but we had to find them first! After searching everywhere they were found in a bag in a closet at the cape. Not a moment too soon.
-Discovering weeks before landing that the lander battery could not deliver enough current quickly enough as the cruising rover turned away from the sun before entry. We would have lost Spirit had we not done an in-flight test of the lander batteries.
-On the fuse issues, I called the flight system & project managers in the middle of the night when they were at headquarters (they were there to tell the NASA AA andmistriator that it was OK to launch). I told them that I found three problems that would scrub both launches. I was a mess. Fortunately I was wrong on one and found work-arounds on the others.
- many more.


Considering all of these sleepless nights before launch and landing, the longevity of these beasties is down right spooky, if not ironic.

Cheers,
Rob Manning (Adler's co-conspirator and co-author of that ugly presentation.)

***********
Comments made here are the express opinion of the author and do not represent the views of JPL, Caltech nor NASA.

I like to think it's more likely the Universe's way of paying you and everyone there back for sticking with the project when times were tough and everyone and everything seemed against you. You built a fine pair of machines there, by keeping going and going like martian Duracell bunnies they're just rewarding you for having such faith in them.

We all know that one day we'll log on here and read the news we've all been dreading - that one of the rovers has died - and that will be a sad, sad day. But until then, I'm going to relish and soak up every image sent back, mentally walking alongside Oppy during the day as she trundles south towards Victoria, and sitting beside dusty Spirit at dusk, keeping her company as the Sun sets and she faces another lonely, brutally cold night on her own in the shadow of the Columbias.

These rovers have changed our view of Mars, and, if we're honest, our lives too in some ways. We've become part of the adventure, part of the mission itself. They've introduced us to wonderful new friends, through forums like this, and educated us about geology, meteorology, image processing, history, geography, Spongebob and Daffy Duck and so many other fascinating things along the way.

Guess what I'm trying to say is, well... thanks.

Rob:

Yes, thanks for the further insights - these really *are* great little rovers! Please keep up the good work!

Bob Shaw

I'm more familiar with software development than hardware, but in my limited experience it seems like heroic efforts are always required to produce a product of quality - or even just a product without glaring errors! The MER project seems to have required more heroism than most, though. I attribute the rovers' phenomenal longevity to the fact that engineers like Rob really cared about what they were doing and sweated and fretted over the details, for which I thank them. Thanks, Rob!

While I appreciate the comments (a lot), please, do not thank me! It was a huge team effort - we all put in a good deal of sweat and angst into these. I worked very long hours on Pathfinder - including nearly every weekend, but MER beat MPF hands down in the level and scope of work. On MPF it was all about simply making it work for our own pride's sake - there were very low expectations outside of our team. For MER we were on a mission to restore public confidence in our ability to do hard things; land on Mars and to take a huge step in performing surface science. I felt very badly about constantly asking my team to come to work on holidays and midnights, but it had to be done.

Fortunately, like you, most of us are passonate about exploration and we have a deep curiousity that drives us. Plus ... we are paid to do this by taxpayers. We have an obligation to do the best we can. While oftentimes we forget, occasionally (like when I read this forum) we get reminded that it is a privilege to be allowed to do this kind of work.

Your enthusiasm does not hurt either! Thank you!

Cheers,
Rob

***********
Comments made here are the express opinion of the author and do not represent the views of JPL, Caltech nor NASA.

When you do the maths, 888 sols is essentially the mission baseline.

Thanks for popping in Rob......now go make Skycrane work

Doug

To make thing as perfect as possible is a passion. The report will serve a very good base for learned lessons plus ones of the post EDL learned as the good input for the next project such as MSL. Lots of creativity to do things as simple as possible as the key for the success of the mission.

Again thanks Rob for sharing us your experiences.

Rodolfo
P.D. Corrected

I find the candor of these comments very refreshing; there should be more of this. I think the engineering on MSL would be better served if it was more widely acknowledged that a lot of MER involved such close calls. In some quarters there's an awful lot of "we have to do it this way because that's how we did it on MER" and frankly, some of these things (the pyro controller would be a good example) are not worthy of emulation.

The line between success (e.g. MER) and failure (e.g. MPL) is really a lot thinner than many people would claim it is.

You are correct Mike.
Success is only won by those who understand failure. A healthy respect for and a deep understanding of the errors we and our predisessors have made on past missions must inform our decisions today if we are to continue to succeed. While there are a few new faces, I believe the bulk of the MSL team does understand past failures and near failures including those encountered on MER. They know all these stories and many more besides. They were there.

I have been party to (or personally responsible for) many design decisions that I have come to realize only later that were either wrong or not right enough. (Actually you might call Adler's and my arguments to build MER based on MPF heritage in under 3 years a huge error.) All design decisions have to balance risk; the risk of getting too big to fly (mass and volume), the risk that it will be too hard to test, slow to build, too costly, or that it might fail or cause collateral failures. Some of my decisions that others thought were too risky resulted in lower risk in the end. The risk balancing process is very difficult and is deeply shaded with personal experience. One person's view on how the risk balance swings may be drastically different that another's. The trick is to immerse oneself in how the decisions made today will affect the future and what future alternatives are available to recover. A good engineer (or manager) might be willing to take some risk today to avoid others in the future especiailly if there are viable aces in the pocket to bail him or her out later if the wheels fall off. The "make better" paradigm is a great one if the risk balances out. Sometimes it does, other times not.

I am one of those who beleives that the MSL design has hedged a lot (but not all) of the risks that we faced on MER. But these systems can ALWAYS be made better. The devil dwells in the details.

Take Care,
Rob

***********
Comments made here are the express opinion of the author and do not represent the views of JPL, Caltech nor NASA.

Great relation Doug. Do you mind if I share with the gang?
(leads me to think that bp meds for the EDL gang should be in the offing)

-Rob

Please do

I did these way before I set up this place, mainly in reference to Beagle 2 ( as I'm only 5 minutes from where Beagle 2's 'PAW' was put together ).....but I'm sure you'll get a kick out of them as well......apologies to Scott Adams

Doug

Very good.
The MSL gang will resonate (then explode).

thank you, Doug.

-Rob

PS we thought about it in '94, but we just could not get ourselves to descope Sojourner; I mean where else could we go for mass & volume after that ? What good is half an airbag or half a parachute?

Rob, something slightly off topic but with all this near and far rim disputes can't avoid to ask something as a jazz fan...
Are you still on the Yardbird side or have you finally seen the light and pass to our brave Coltrane team?...

Really, do I have to choose teams? Bird in the am, trane in the pm, and what about Ornette in the evening to keep the sleepies away? there is but one jazzer's team.


As for rim teams, may the best win, all I ask is that we stay away from sand surfing. We need to get a good rimshot before the ground gives way.

-Rob

No, we really don't...

But I'm a real Coltrane admirer...
Listening to him and Miles "walkin'" side by side show us there are no teams on this smooth ground...

Emily,

thank you for this excellent posting. I heard about some of this stuff, but its mostly new and highly informative.

Hi Rob,

thank you for the detailed followup info and candor.

and your recent Viking conference presentation was fantastic and also filled with lots of realism for MSL, especially the sky crane.

ken

Oh my, this is fascinating, it's a miracle that everything converged just in time for a successful landing.
It's funny thing, that none of that was publicly presented, as far as I know.

Borek

You'll see quite a lot of it in Steves book (When are you doing yours Rob?..don't tell me you hadn't thought of it ) but it wasnt really very public at the time. Airbags and chutes, yes...the deep inner stuff, not really.

Doug

The NOVA special covered some of the prelaunch design headaches that made it seem a very long, stressfull push to get everything together. I wondered if this was overdramatized, but after seeing Mark's slides, I think it was white-washed.

Ok, let's all send happy-thought vibes to the Phoenix team. I'm sure they need it now:
http://uanews.org/cgi-bin/WebObjects/UANew...ArticleID=12747

Hopefully since Phoenix is smaller, it doesn't have as many design headaches...
... happy thoughts ... ... happy thoughts ...