Built to last: Computer systems that simply cannot fail

Space shuttle Endeavour's flight deck

Share This article

A failing computer system can be a problem in some settings, but it’s catastrophic in others. No one likes when their computer crashes out while in the middle of an overnight render, but the cost of that delay is nothing compared to a failure in a mission-critical environment, such as in a hospital, on a satellite, or, in the case of Curiosity, on another planet entirely. These extraordinary settings have led to the creation of specialized computer systems that are designed to work in high-demand situations, often for years on end. They are, not surprisingly, some of the most interesting computer systems out there.


Hospital equipment, first and foremost, must be immune to fluctuations on mains power. Static transfer switches can switch power to backup batteries within a single cycle of a 60Hz mains power failure, and provide seamless emergency power for the few seconds it takes for the generators to be started. Inductive spikes from switching compressor motors need to be filtered or blocked with isolation transformers as well. But these devices do not protect against glitches occurring at more localized levels, which can reset instruments, often freezing or restarting them in an unpredictable state.

Operating Room -- lots of computers!

Precision robots which assist surgeons during operations are large integrated systems often having several computers managing high speed FPGAs (field-programmable gate arrays) and DSPs (digital signal processors). The Da Vinci surgical system has 7 degrees of freedom and at least as many servo controllers. As a result of many subsystems having different supply requirements, the overall system ends up being designed with tight voltage supply specs and therefore needs sufficient buffering against fluctuations that exceed a few percent.

An even more insidious threat to healthcare/medical tablet computers and Windows PC-based computer systems found in hospital equipment, is infection with conventional malware. Due to strict regulations, manufacturers frequently cannot allow OS security patches or updates leaving many computers vulnerable, with the result that many infected instruments run slower, and many others are shelved altogether.


In space

Computers in spacecraft have their own unique set of challenges. In addition to requiring real-time low latency control for maneuvering and communication, spacecraft also need to be hardened against the effects of cosmic rays and other forms of radiation. Shielding is simple and effective but is prohibitive because of its weight. Therefore efforts have been focused on making the chips themselves resistant to radiation.

FPGAs are frequently found in spacecraft due to their speed and computational efficiency in performing tasks like fast Fourier transforms and beam-forming in communications. They do not require code like a microcontroller, instead they write a particular computation directly into their logic gates. FPGAs based on SRAM can be reconfigured at will, however the same technology which makes them possible also makes them vulnerable to radiation. A charge deposited on a control structure like a transistor can induce it to momentarily change state. If it is part of a persistent circuit like a flip-flop or RAM cell the change can be permanent. One-time programmable FPGAs based on anti-fuse technology are much more resistant but can be over ten times the initial cost.

Mars Exploration Rover Curiosity

The microprocessors and memory in spacecraft also need to be able to withstand radiation. The RAD750 PowerPC in Curiosity’s flight computer was designed to survive for 15 years before intervention is required from Earth. These chips are slower than some newer systems but NASA has been using them successfully for some time and likes to stay with what works. Curiosity’s computers run a real-time operating system, VxWorks, which it has used on two previous rovers. The VxWorks microkernel is reportedly better optimized for minimal interrupt and thread switching latency than the the monolithic RTLinux kernel, although it is not as fast overall.

A final consideration for failure resistant computers in spacecraft would be that if they do encounter significant trouble, they should have provision either to be remotely rebooted or some mechanism to be rebooted on-board. With proper consideration to input power fluctuations, interrupt latency (or jitter, variation over time), and external insults such as radiation, computer systems can be designed to be arbitrarily robust.

DARPA's Crusher autonomous vehicle, powered by QNX

The examples above represent but a small portion of the range of mission critical systems in service everyday. Other real-time operating systems like QNX, now owned by RIM, can be found in drones and military vehicles like the Crusher tank (pictured above). Recently the security of the Boeing 747 engine control system running Solaris was questioned; engineers on the ground apparently could access control systems for re-tuning en-route. In this case, secure protocols like SSH were not compatible with the parts of the existing software, and insecure Telnet was still being used.

As the tentacles of internet more intimately weave into military and civilian infrastructures, new concerns will present themselves, and increasing vigilance will be required to keep computer systems fail-safe.

[Image credit]


Post a Comment

  • John Pombrio

    Nice article! Sebastian and I were talking about this with comments on Curiosity. The shielding on the printed circuit boards is a slab of metal, no wonder they are trying to get away from that.
    As you have mentioned, not only is the hardware need to be protected but the software as well. A bad line of code can literally crash a Mars lander. For aircraft and medical devices, changing code takes a writ from GOD.

    • http://www.mrseb.co.uk/ Sebastian Anthony

      Let’s just not talk about the time they got their imperial and metric units mixed up…

      • Tony R.

        Does any engineer worth his salt still use Imperial units in new designs?

      • Tony R.


        • http://www.mrseb.co.uk/ Sebastian Anthony
          • http://pulse.yahoo.com/_XMS2RNVZI2WJXS5TI7PQTXWZUE Magic Rat

            …… and what about British Whitworth?

        • http://twitter.com/jhewitt123 John Hewitt

          I would say perhaps 7/8 of all machinists in America still prefer the inch. You may find a metric dial indicator floating around, but metric 6″ dial calipers are not very common in American shops. Why? Fractions are exact and they scale in an intuitive way within the mind’s eye at the scales that are real to the hand. Would European machinists ever say, for example, 1 – 1/4 centimeters? I wouldn’t know. When you actually reach into the tool crib looking for a replacement turning tool insert, it is not too difficult to discriminate between inserts with 1/32 tool nose radius with those of 1/64 or even 1/128 nose radius. You certainly can find a metric tap and make metric threads. But if an engineer puts a hole callout in the CAD drawing that specifies a 6.9mm tap drill to be used for an 8mm x 1.25mm bolt hole, the machinist will probably be using a 17/64 drill.


    Apple fans bragging that they can leave their computers turned on forever in 3.. 2.. 1..

    • Fredo

      But I can leave it on for like… a whole day! =P

    • some_guy_said

      I have a PC that froze once that I remember, possibly twice, over 5 years on winxp.

      took me 15 minutes to troubleshoot because I assumed the wireless mouse wasn’t connecting/out of power.

      Upgraded it last summer, guts and win7. No freezing yet.

      Granted, I set up my hardware and software to be highly optimized for malware resistance and lean performance. But then again, I bought my gf a stock HP with a quad AMD Phenom a few years back, and haven’t had trouble with that one either – Other than the initial setup – getting it to play nice with the XP computer on network/sharing/media settings.


        That thing about Macs being on forever and ever and ever was from the (mid to late) PPC era, when Apple computers were the best Professional machines ever.

        • some_guy_said

          They would be trumpeting stability like the 35 year old who still spends all his time reminiscing about his glory days as a highschool football player. lol.

          Having alternately built/bought computers since about 2001, I would say that the hardware/software stability really hit a few years after xp launched – drivers and hardware started working effortlessly with it around ~ ’03 or so – So my personal experience would in fact support your claim.


            I remember when XP was first launched. VERY unstable, full of bugs, lousy driver support and people were running it with 64MB or 128MB of RAM. Was so crappy at that point people were saying that XP means “Xtra Problems,” Then Service Pack 1 came and RAM became affordable enough to bump the RAM to 512MB which was just enough for it to run decently along with MS Office.

  • Franc Schiphorst

    If you look at the measures they take to get the rover software safe it’s astounding. Tricks are not allowed no recursion as they have over 130 processes running in parallel. Code scanning logfile analysis etc etc
    And it survived the landing :)

  • http://www.facebook.com/profile.php?id=749773855 John Wood

    Computer, meet my friend thermite. Computer dies. /story

  • Ron007

    {snip} An even more insidious threat to
    healthcare/medical tablet computers and PC-based computer
    systems found in hospital equipment, is… {/snip}

    WINDOWS, USB Thumb drives and the internet connections !