Sol 22 anomaly, File system problem Options

[elakdawalla]

Today's press release from the Phoenix mission contained the following nugget of information:

Newly planned science activities will resume no earlier than Sol 24 as engineers look into how the spacecraft is handling larger than expected amounts of data.

This sounded alarming, and immediately brought the very scary Spirit sol 18 anomaly to mind. (That anomaly, in brief, had to do with too many files being kept in flash memory, which resulted in Spirit descending into a cycle of continuous reboots that might, if not stopped, have depleted the batteries and killed the rover within a day or two. Through heroic efforts Spirit was recovered and obviously returned to perfect health.)

I requested an interview with someone from JPL and am happy to say I just got a call from Barry Goldstein. I'm copying here the entire text of what he said to me. I will be blogging this but am wondering if someone here could help expand a bit on the business about APIDs (Application Process Identifiers) and what part they play in an operating system. I started off by asking for more detail on the problem, and for him to compare and contrast with Spirit Sol 18.

When the anomaly happened with Spirit, we lost communication. We never lost communication or control of the vehicle here. It's quite different. On Spirit we had a file management problem that ran amok.

What happened was, at one of the downlinks on sol 22, the engineering housekeeping data was being looked at by the spacecraft team. And they noticed one of the APIDs for a housekeeping data packet, which is normally generated only one to three times every time we do an uplink, was generated 45,000 times. It was a surprise, to say the least. And the reaction of the team was, the obvious which was concern about why the heck did this happen, and the other issue was we were concerned about two things. One, since the APID priority for this data type was very high, would it starve out any of the science data from being saved overnight because it's now so large? And the resolution of that it turned out, yes indeed, it was that large, and we ended up losing very low priority science data from sol 22. But the scientists are not at all concerned about that. The second concern we had yesterday was, we had a restriction on the amount of time it takes for the spacecraft to boot. I can't remember the total value but it's over 60 seconds. If it doesn't boot within a certain amount of time, it will reset and then eventually go over to the B side (it's block redundant, unlike MER). The reason we were concerned is that this data structure, now which is huge because of these 45,000 blocks, it has to pull that out of the flash as part of the boot process. And so we were concerned it would take too long and therefore it would side-swap. So we took some emergency action last night, and I'm happy to say we got the uplinks in due to the following things. Number one, we updated the priority of that APID such that it will restrict the amount of that data type to be saved in flash. Second thing we did is we lost science operations on sol 23. Third thing we did is up the priority of the downlink of that data structure that we generated so often so that we could retrieve what we have so it could help us diagnose the problem. The current state of the spacecraft is as follows.

We have the data down, we have the spacecraft under control, we have the size of the file system in control such that we're no longer worried abou tthe size of the file system growing and keeping us from booting appropriately. The second thing is, the only restriction we put on science activity for sol 24, which the science team is planning right now, is that they can't save the data to the flash because we want to keep the flash small, we don't want this thing to eat us alive. So what the team is doing now is planning sol 24. However, there's a little paradox here. Because we were in this anomalous state, we requested and received a bunch of contingency passes from MRO and Odyssey. So what ends up happening is we told the science team you can do whatever you want, because the only thing we are worried about was flash, we just are not going to save it to flash when we turn off. And we then told them we have all these passes. So as it turns out, what the science team is planning is the most data-rich sol we've had to date, because we have all these extra passes. I was joking with Peter that he should pray for these things more often because he gets more data.

{What other kind of memory is there besides flash?} We execute out of RAM, and every time we turn the vehicle off to save power at night, charge the batteries, we save off the critical data structures which include this file system with the telemetry that has not been marked as received on the ground. And that's what really ate our lunch is the saving of this to the flash. We ran out of room in the flash and that's what caused them to lose the science data, which was low priority. And then it's the time it takes to read it out of flash and get it down on the ground.

{What's generating all these APIDs?} We have a suspect, and I'd prefer not to go into a lot of detail, but the suspect has to do with the packet counter number for each of the packets that are stored. It's been less than 24 hours so I'd like to let the team get a chance to look at this and analyze it completely. At this point it's our prime suspect but that doesn't necessarily mean it will pan out.

Even though we have had this anomaly, the vehicle is under control. We lost a sol of operations, because when this occurred we stopped the uplink for that sol. We have the vehicle under control, we understand the problem, we don't know the root cause, but we've taken preventive measures to make sure it's still functional without risking a problem.

It's much less scary {than Spirit sol 18} but I'll feel a lot better when we know exactly what's going on. All these things are scary to one degree or another. I'd rather have this problem though; not hearing from a vehicle is disconcerting.

--Emily

[jmjawors]

That little blurb caught my eye as well. Thanks for following up on it.

Afraid I can't help with the APIDs, though.

[climber]

An info from AW&ST, june 9th, Craig Covault page 35 "thirty days of the Phoenix 90-day mission are planed as lander "down days" when primary sampling or other science commands will be disrupted by relay difficulties". I understand that this means Phoenix can do unplanned observations.
It's not the case described by Emily & Barry but I guess there is some room for some issues to show up and still be ok within the 90 sols.

[mcaplinger]

I... am wondering if someone here could help expand a bit on the business about APIDs (Application Process Identifiers) and what part they play in an operating system.

From http://mars.jpl.nasa.gov/MPF/nasa/pipfaq.html -- this is old Pathfinder data, but the general concept is the same.

How is data acquired by the spacecraft stored in the central computer and prioritized for return to Earth?

Answer:

All Pathfinder downlink data are packetized and assigned to APID ("Application Process IDentifier") queues, from which data is downlinked in FIFO (first in first out) order. The packetization and assignment takes place immediately as a result of execution of commands which acquire data -- separate commands to packetize and enqueue the data are not used. APIDs can be configured as rings, where old data is overlayed, or as queues, where new data is rejected if the size limit is exceeded. APIDs are identified by both name and number (0-42). Specific data formats are permanently assigned to specific classes of queues. For instance, one queue is for rover health data, and only data of that format can be assigned to it. There are multiple queues for IMP image data. Any IMP image can be assigned to any of these IMP queues, and within limits, the assignment is negotiable.

Within a single downlink session, APIDs are prioritized according to a two-dimensional priority matrix called a DPT ("Downlink Priority Table"). The DPT structure is used to make sure the most important data gets in the front of the downlink stream, regardless of when it is acquired, with the proviso that downlink from individual queues is FIFO. In the DPT, APIDs can be assigned to completely override others in priority (ie, completely prevent other APIDs from getting any downlink so long as any data is left in the higher priority APID), or they can be assigned to share a priority level on a percentage-of-bits basis.

Different downlink sessions can be governed by different DPTs, and within limits, the DPT organization is negotiable.

It is not possible to reorder packets within the queue, nor is it possible to move data packets between queues. Data packets can be deleted from the front of the queue up to a commanded time (the time when they were acquired) or by specific packet number at any point in the queue. It is not currently possible to delete specific data reliably from the center of a queue, but further study could mitigate this problem.

[helvick]

Emily,

My understanding is that each spacecraft [telemetry?] function that generates engineering data has an associated APID and at each data point that that particular function samples whatever it is looking at it generates a data packet (which is stored in CCSDS format or something close to that) that includes the APID and an incremented counter in its CCSDS packet header. CCSDS packets also have a priority level associated with them that is also included in the header - that priority level is used to indicate which packets should "win" out in cases where resource contention arise during relay.

The situation that they describe sounds to me as if some telemetry function is returning far more data packets of a high priority level than expected and the sheer volume of those data packets (45,000 at maybe 1Kbyte per packet would chew up 45Mbyte of storage for example) was filling up available data storage. The spacecrafts operating system responds to that by deleting lower priority packets already present if necessary in order to save the packets that it believes are more important.

Given that they clearly know the APID they must know what spacecraft function is generating the packets - that doesn't necessarily mean that the root cause is obvious but it sounds to me that this should be easier to get to the bottom of than the Spirit Sol 18 problem.

The above is mostly speculative but I got some hints from here.

[jekbradbury]

Why is nobody looking on the bright side? We get more science and more images than ever before on sol 24, and we miss only one sol of data (hopefully).

[Deimos]

The current problem is within an engineering APID, 40. The APID structure and use is similar to MPF, but there are some interesting nuances. There are also more APIDs for both engineering and science (SSI has 13). Flash size motivated the science team to ask for more APIDs. There is a downlink priority table (DPT) in use on any given downlink. There is also a nighttime priority table that is used when saving to flash. Higher priority APIDs get saved first, lower ones may not make it. So APIDs map a 2-D space: How urgent is it that we get the data soon? How important is it that we never lose the data?

You can imagine a few kinds of data. An image of the dig we just did may be key for the next planning cycle, thus it has to be high on the DPT. But, if we somehow didn't get it (think electra), we could as easily reaquire it as save it overnight. So maybe it is low on the NPT (in practice, this specific example tends to be high in both). Or a RAC image of a sample in the scoop just before be deliver that sample: you may not need that image to plan the next sol, but you can never take it again. So, high in the NPT, maybe not in the DPT. A TEGA or WCL run ends up being very high in the NPT; they may also be high in the DPT if, for example, a follow-on TEGA ramp is desired the next sol. And, many things are not urgent and can also be redone. An image of some rock several meters away: if it falls out of flash, just take the picture again. So, for every product generated, a decision has to be made on both urgency and the need to save the data--then APIDs are assigned.

In strategic planning, the data is that is neither urgent nor critical to save to flash (especially SSI_LOW) has gotten the nickname "red-shirt" data, and is always vulnerable to loss in the event of even minor problems. Actually we've only lost it a few times though.

A further complication is "sent" data. If the data were specifically for tactical planning, you could treat it as "fire and forget". If the data is a TEGA bake, you cannot. What if the data are lost in transmission and need resending? Thus, the most important "sent" data trumps the least important unsent data (the red-shirts) when saving to flash.

And just when you thought I'd be out of further complications ... what if we could use MRO to get an extra 30-40 Mb of data? But, what if we new there was a larger risk of losing that data compared to the (now normal) ODY passes? You want to take and send the extra data; but you cannot afford to send urgent or critical data the risky way. Send in the red-shirts. So sometimes the lower priority stuff comes down at 2 PM (Mars time) while the more urgent stuff waits until 4 PM (and the first ODY pass).

How close a resemblance does this bear to MER file management? Well, just about none. There are files ... they are managed ... that's about it. Actually, on MER files are managed, on PHX APIDs are managed. MER data have priorities that can be dynamically reassigned (as opposed to moving whole APIDs around) and do not use APIDs for prioritization. MER priorities are for both saving and downlink, and MER is managed to avoid most "auto-deletes" when there is more data than flash. Phoenix cannot be managed that way, since we usually have more downlink in a sol than flash capacity, before even worrying about sent data that needs protecting.

[ugordan]

In strategic planning, the data is that is neither urgent nor critical to save to flash (especially SSI_LOW) has gotten the nickname "red-shirt" data, and is always vulnerable to loss in the event of even minor problems.

I don't suppose "red-shirt" is a Star Trek reference?

Thanks for the detailed explanation, Mark. I was wondering what the "None of the above" comment for sol 23 was as well as that tidbit from the press release.

[ElkGroveDan]

Good catch Gordan. More on red shirts here.

[MahFL]

I loved the series Startrek, you always waited with anticipation for when the person in the redshirt was going to die.

[Cargo Cult]

How close a resemblance does this bear to MER file management? Well, just about none. There are files ... they are managed ... that's about it. Actually, on MER files are managed, on PHX APIDs are managed.

Out of (somewhat nerdy) interest, which operating system (if any) is Phoenix running? I'm sure I read an article somewhere about it being something other than VxWorks as used by the rovers, but I can't remember what exactly it was.

(For everyone else, there's an interesting article here about Spirit's problems - essentially the number of files on flash grew to require more memory than the filesystem module could allocate, forcing the system to reboot, only to try to mount that filesystem again...)

I had a weird sense of mental inversion last night, where Phoenix and friends stopped being space probes with computers inside them, to being computers with space probes built around them. All my laptop asks is - can it go to Mars too? ;-)

[PaulM]

Out of (somewhat nerdy) interest, which operating system (if any) is Phoenix running? I'm sure I read an article somewhere about it being something other than VxWorks as used by the rovers, but I can't remember what exactly it was.

(For everyone else, there's an interesting article here about Spirit's problems - essentially the number of files on flash grew to require more memory than the filesystem module could allocate, forcing the system to reboot, only to try to mount that filesystem again...)

I had a weird sense of mental inversion last night, where Phoenix and friends stopped being space probes with computers inside them, to being computers with space probes built around them. All my laptop asks is - can it go to Mars too? ;-)

MCAPLINGER very kindly gave me the following link about 2 weeks ago:

http://www.klabs.org/richcontent/MemoryCon...irit_mishap.htm

This article confirms that VxWorks is used by the Mars Rovers and also includes a link to the following more complete description of the MER flash problem:

http://www.klabs.org/mapld04/presentations..._costello_s.ppt

One thing that did suprise me when reading this article is that the Mars Rovers depends so much upon linked lists residing in heap memory. This is because RAM in Space borne microprocessors is very suceptible to Single Event Upsets (SEU). Perhaps the 25 MHz RAD6000 Power PC in MER is less suceptible to SEUs than most microprocessors flown in space?

Single Event Upsets are explained here:

http://en.wikipedia.org/wiki/Single_event_upset

[mcaplinger]

Out of (somewhat nerdy) interest, which operating system (if any) is Phoenix running? I'm sure I read an article somewhere about it being something other than VxWorks as used by the rovers, but I can't remember what exactly it was.

Google is your friend. http://blogs.windriver.com/deliman/2008/05...ou-watch-i.html confirms that Phoenix uses VxWorks 5.2.

As for SEUs, the RAD6000 is not very subject to SEUs: http://www.baesystems.com/BAEProd/groups/p..._eis_sfrwre.pdf says 7.4e-10 errors/bit-day in 90% worst-case GEO. Of course, each system costs about a million dollars IIRC.

[hendric]

The current problem is within an engineering APID, 40. The APID structure and use is similar to MPF, but there are some interesting nuances.

Let me see if I can translate this. You have a desk with multiple outboxes (APIDS). They are stacked one on top of another, with the top of the stack getting sent out first. The boxes are different sizes. Messages are put into the outboxes at the end of each day, depending on their future usefulness - RAC scoop image is not useful now, but priceless, so goes into a big bottom box. Panorama images of next worksite are useful now, but not priceless, so goes into a smaller top box; if the top box is already full, no big, just take another picture tomorrow. Now, when it comes time to send the stuff out, it makes sense to send the stuff needed tomorrow by a reliable sender, so it can be deleted sooner. The not-useful-now-but-priceless stuff can be sent be the (potentially! Don't want to offend any MRO telecoms people ) less reliable sender, to be deleted once it has been acknowledged as received. On top of these are engineering data, mixed in with their own priority levels. Solar panel output is a need-to-know-tomorrow-but-dump-if-necessary data type, while temperature level is probably a not-real-important-but-keep-for-later data type.

So someone was stuffing APID 40 with data, 45,000 times. (I would agree that it's probably obvious who is stuffing the APID with data, and that someone is staying up nights trying to figure out why it's happening.) Meanwhile, any other data sharing that APID can be moved to point to another APID, and that APID's size limited.

Just seat-of-my-pants guessing, someone left a debug message on, and forgot to disable it in the flight software. Maybe something that triggers every 30 seconds, starting about 5 days ago from the 45000. Perhaps there is a thread checking the TEGA door every 30 seconds and within the packets there is a message:

0x43256432: TEGA door #1 did not deploy fully?

I jest, only because I have been there!

[glennwsmith]

Hendric, you're idea of a debug message as the cause of the anomaly is an excellent guess. We programmers have all been there. But it's one thing for a debug message to be scrolling harmlessly across a CRT screen, and quite another for them to be piling up as strings in flash memory!