NASA’s new rocket would be the most powerful ever. But it’s the software that has some officials worried. – The Washington Post

When it launches, NASA’s Space Launch System rocket, a towering 322-foot behemoth — taller than the Statue of Liberty — would be the most powerful rocket ever flown, eclipsing both the Saturn V that flew astronauts to the moon and SpaceX’s Falcon Heavy, which has launched commercial and national security satellites as well as founder Elon Musk’s Tesla Roadster on a trip to Mars.

But as NASA moves toward the SLS’s first flight, putting the Orion spacecraft in orbit around the moon, it’s not the rocket’s engines that concern officials but the software that will control everything the rocket does, from setting its trajectory to opening individual valves to open and close.

Computing power has become as critical to rockets as the brute force that lifts them out of Earth’s atmosphere, especially rockets like the SLS, which is really an amalgamation of parts built by a variety of manufacturers: Boeing builds the rocket’s “core stage,” the main part of the vehicle. Lockheed Martin builds the Orion spacecraft. Aerojet Rocketdyne and Northrop Grumman are responsible for the RS-25 engines and the side boosters, respectively. And the United Launch Alliance handles the upper stage.

All of those components need to work together for a mission to be successful. But NASA’s Aerospace Safety Advisory Panel (ASAP) recently said it was concerned about the disjointed way the complicated system was being developed and tested.

At an ASAP meeting last month, Paul Hill, a member of the panel and a former flight and mission operations director at the agency, said the “panel has great concern about the end-to-end integrated test capability and plans, especially for flight software.”

Instead of one comprehensive avionics and software test to mimic flight, he said, there is “instead multiple and separate labs; emulators and simulations are being used to test subsets of the software.”

“As much as possible, flight systems should be developed for success, with the goal to test like you fly. In the same way that NASA’s operations teams train the way you fly and fly the way you train,” Hill said.

Also troubling to the safety panel was that NASA and its contractors appeared not to have taken “advantage of the lessons learned” from the botched flight last year of Boeing’s Starliner spacecraft, which suffered a pair of software errors that prevented it from docking with the International Space Station as planned and forced controllers to cut the mission short.

NASA has since said that it did a poor job of overseeing Boeing on the Starliner program, and has since vowed to have more rigorous reviews of its work, especially its software testing.

The SLS software concerns are the latest red flags for a program that has struggled to overcome a series of cost overruns and setbacks. A slew of government watchdog reports over the years have painted a troubling picture of mismanagement.

Three years ago, the NASA Inspector General reported in an audit that NASA had spent more than $15 billion on SLS, the Orion spacecraft and their associated ground systems between 2012 and 2016. It estimated the total would reach $23 billion.

The report chided Boeing, the main contractor, which it said “consistently underestimated the scope of the work to be performed and thus the size and skills of the workforce required.”

NASA says now that the program is finally on track, with the vehicle undergoing a series of rigorous tests known as the “Green Run” at the Stennis Space Center in Mississippi that will culminate with a “hot fire” — the ignition and eight-minute burn of its engines scheduled for later this year.

Then it would be moved to the Kennedy Space Center in Florida, ahead of its first launch, currently scheduled for late 2021. NASA administrator Jim Bridenstine said “all of the elements that we need for a successful 2024 moon landing are underway as part of the agency’s Artemis program. And we’re moving rapidly to achieve that goal” — a dramatic White House-ordered acceleration of the original timetable that foresaw a moon landing in 2028.

For that deadline to be achieved, however, the flight software has to work perfectly. The first test is expected to come late next year, when the SLS would fly for the first time in the Artemis I mission, putting the Orion spacecraft without any crews on board in orbit around the moon

“When it all comes down to it, flight software is the functional integration piece of the rocket,” Dan Mitchell, NASA’s senior technical leader for SLS avionics and software engineering, said in an interview. “The rocket doesn’t fly without flight software. The software commands all the valves and the engines. It takes reasons of all the parameters inside the vehicle, the navigation and position information and uses all that information to control the fight.”

There was perhaps no better illustration of the significant role software plays in space flight, and how flaws in the coding can have severe consequences, than Starliner’s test flight.

Shortly after it reached orbit, the spacecraft, which had no astronauts on board, ran into trouble because the spacecraft’s flight computers were 11 hours off. With the spacecraft thinking it was at an entirely different point in the mission, it attempted to correct its course, burning precious fuel and forcing controllers to end the mission early without completing the main goal: docking with the International Space Station. Controllers later found another software problem that could have caused the service module to collide with the crew capsule after separation, potentially endangering astronauts, if any had been on board.

Boeing was able to diagnose the problem, send up a software fix and ultimately bring the spacecraft down safely. Later, Boeing said its testing of the software was deeply flawed, allowing the two problems to go undetected in the spacecraft’s one million lines of code. It was an admission reminiscent of the software problems that plagued its 737 Max airplane, which suffered two crashes that killed 346 people combined and remains grounded worldwide.

Boeing officials have said that during the test flight, the Starliner was pulling its time from the rocket. During testing, officials were mainly focused on making sure the two vehicles were communicating correctly, but cut short the test so that it never uncovered that the spacecraft was reading the wrong time.

If the test had continued, “we would have caught it,” John Mulholland said earlier this year, when he was the Starliner program manager for Boeing. He’s since transferred to Boeing’s space station program.

During the software test for the service module separation, Boeing didn’t use the actual hardware but rather an “emulator,” a computer system designed to mimic the service module. The problem was the emulator had the wrong thruster configuration programmed in at the time of the test, Mulholland said.

NASA officials in charge of the SLS program said they are confident the testing protocols for the SLS rocket and Orion spacecraft are far more robust. For starters, the program is set up differently. Boeing owns and operates the Starliner spacecraft and uses it to perform a service for NASA — namely flying its astronauts to the space station.

On the SLS program, by contrast, NASA owns and will operate the rocket, and is responsible for all the integrated testing.

Mitchell, the NASA senior technical leader, said the SLS team took the Starliner mishap “to heart.” As a result, they spent four days testing the various interfaces between the SLS and Orion, he said. “We methodically walked through requirement by requirement. … It was a very, very detailed and fruitful interaction that we had across all the interfaces,” he said.

The review turned up one issue with how the rocket’s second stage interpreted data from the first stage, he said, but that “has been determined to be a benign issue” that doesn’t require any modifications at this time.

NASA pushed back on the safety panel’s findings, saying in a statement that “all software, hardware, and combination for every phase of the Artemis I mission is thoroughly tested and evaluated to ensure that it meets NASA’s strict safety requirements and is fully qualified for human spaceflight.”

The agency and its contractors are “conducting integrated end-to-end testing for the software, hardware, avionics and integrated systems needed to fly Artemis missions,” it said.

Once the vehicle is moved to the Kennedy Space Center, testing will continue with a “countdown demonstration and wet dress rehearsal [by fueling the rocket] with the rocket, spacecraft, and ground systems prior to the Artemis I launch.”

Speaking to reporters in October, John Shannon, a Boeing vice president who oversees the SLS program, said the core stage holds “the brains” of the rocket, the avionics, flight computers and “the systems to control the vehicle.”

But he said the company’s portion of software development and testing was limited to what’s called the “stage controller,” or “ground software that commands the vehicle itself.”

Shannon said the systems have been “completed, tested in integration facilities at [NASA’s] Marshall Space Flight Center. We’ve had independent verification and validation on it to show that it works well with the flight software and the stand controller software. And it’s all all ready to go.”