Each of SpaceX's monthly launches of 60 internet-beaming Starlink satellites carries 4,000 stripped-back Linux computers, SpaceX software engineers have revealed.
SpaceX engineers disclosed the detail in a Reddit Ask Me Anything (AMA) session over the weekend.
Elon Musk's rocket company last week launched 60 more internet-beaming satellites into space on a reused Falcon 9 rocket, bringing its total count to around 480 and moving it closer to the 800 it needs to provide moderate coverage over the US.
The company plans to launch a public beta of the satellite internet service by the end of the northern hemisphere's summer and has won approval to deploy one million end-user terminals in the US.
SpaceX recently applied to the Federal Communications Commission to launch 30,000 second-generation satellites over and above the 12,000 that had already been approved. Assuming the second-generation satellites carry the same number of Linux computers, it would mean SpaceX plans to send at least two million Linux computers into space in the next few years.
It also means that it's now sent 32,000 Linux computers to space for the existing constellation.
"The constellation has more than 30,000 Linux nodes (and more than 6,000 microcontrollers) in space right now," wrote Matt Monson, SpaceX's director of Starlink software.
"And because we share a lot of our Linux platform infrastructure with Falcon and Dragon, they get the benefit of our more than 180 vehicle-years of on-orbit test time."
The AMA was also hosted by Jeff Dexter, who runs flight software and cybersecurity at SpaceX; Josh Sulkin, the software design lead for Crew Dragon, the spacecraft that took NASA astronauts Bob Behnken and Doug Hurley to the International Space Station last week; Wendy Shimata, who manages the Dragon software team; John Dietrick, the software development lead for Demo-2; and Sofian Hnaide, who works on the Crew Displays software for Demo-2.
Sulkin revealed that the Linux computers run with the PREEMPT_RT patch, which is developed by a Red Hat engineer as part of an initiative to make Linux a real-time operating system.
Sulkin noted that SpaceX doesn't use a third-party Linux distribution and that it has several custom drivers to interface with its hardware, which forms a distributed computer system.
Hnaide noted that the team uses an in-house reactive library for Chromium interface.
There's also been some cross-development between the interface that NASA astronauts Behnken and Hurley are using and Starlink satellites.
"The tech from the crew displays (especially the map and alerts) formed the basis of our UI for the first couple of Starlink satellites (Tintin)," said Starlink's Monson.
Tintin A and B were the first two test Starlink satellites put in orbit on the back of a Falcon 9 rocket in February 2018.
"It's grown a ton since then, but it was awesome to see Bob and Doug using something that somehow felt familiar to us too."
SpaceX cybersecurity boss Dexter said any students aspiring to work at the company should make Linux a central part of their education.
"Get your CS degree (or something) similar," said Dexter. "Spend time to really make sure you know how things work – engineers who do well at SpaceX are meticulous in their understanding of how their code works, how the network works, how Linux works, how the hardware works, etc. Get real-world experience building things and solving hard problems, either through hobby projects or in internships (at SpaceX)."
As the Starlink satellite broadband service moves from a test to operational phase in coming months, security is going to become a critical issue for SpaceX. Moran offered some insight into the issues that SpaceX engineers are looking out for, from the satellites to gateways and the 'UFO-on-a-stick' end-user terminals located at households.
"We designed the system to use end-to-end encryption for our users' data, to make breaking into a satellite or gateway less useful to an attacker who wants to intercept communications," wrote Moran.
"Every piece of hardware in our system (satellites, gateways, user terminals) is designed to only run software signed by us, so that even if an attacker breaks in, they won't be able to gain a permanent foothold.
"And then we harden the insides of the system (including services in our data centers) to make it harder for an exploited vulnerability in one area to be leveraged somewhere else. We're continuing to work hard to ensure our overall system is properly hardened, and still have a lot of work ahead of us (we're hiring), but it's something we take very seriously."
And while the tech industry is using machine learning (ML) to automate almost everything, it hasn't reached SpaceX's Dragon and Falcon spacecraft.
"Dragon and Falcon do not use any ML tech, but that's not to say things like this aren't in SpaceX's future," wrote Dietrick.
The SpaceX engineers didn't divulge what computing hardware (CPUs and GPUs) the company uses on its spacecraft, but it's likely the computers aren't powerful enough to support ML applications.
As noted by Vaughan-Nichols, the International Space Station (ISS) runs on 1988-vintage 20MHz Intel 80386SX CPUs that need to be hardened against radiation from cosmic rays in space.