Key details are emerging about how Air Force managers are working to pull into line the substantially delayed and over budget program to build a new GPS ground system.
Key details are emerging about how Air Force managers are working to pull into line the substantially delayed and over budget program to build a new GPS ground system.
From the outside the task appears monumental. In the spring of 2016 the Senate Armed Services Committee killed funding for the GPS Next Generation Operational Control System (GPS OCX) because of cost overruns. Lawmakers contended the program had exceeded its budget by 50 percent — that is, that there had been a critical contract breach as defined by the Nunn-McCurdy Act. The Department of Defense (DoD) did indeed declare a Nunn-McCurdy breach on June 30, 2016, putting the program on the path to automatic cancellation unless the Secretary of Defense certified OCX was essential and that DoD had a plan for getting the program back on track.
On October 12, 2016, one day before the deadline, the program was certified by Frank Kendall, the under secretary of defense for acquisition, technology, and logistics on behalf of then-Secretary Ashton Carter. In his statement Kendall assured lawmakers that the "remaining costs for the restructured OCX program are reasonable." That pledge appeared overly optimistic given that the cost for OCX, originally expected to be around $1.5 billion, had jumped at that point to around $5.5 billion.
But something had happened just 10 days earlier that gave credence to DoD’s positive outlook — on October 3, OCX moved to the cloud.
Amazon Delivers
The cloud is the vast computing and information storage capability made possible by arrays of networked computer servers located at data centers or “server farms" and accessed through the internet. The suggestion to move OCX to the cloud came from the Pentagon’s relatively new Defense Digital Service (DDS). DDS is "a team of self-described nerds who come in on short stints from companies such as Google, Amazon, and Netflix to work on problems impacting DoD," the Pentagon explained in a statement earlier this year.
In the case of OCX, the GPS Directorate is using Amazon Web Services (AWS) to build and run multiple virtual OCX environments — what amounts to a plethora of software test tracks running at the same time.
"We can rapidly build up this OCX representative virtual environment to do the integration and test," said Col. Steve Whitney, the director of the GPS Directorate. "We can stand up more of them, simultaneously, so that we can have multiple people go in at once."
There is enough capacity to stand up individual test tracks for each and every developer, Whitney told Inside GNSS. Coders can load and test software with results coming back in hours, not days. "That rapid access basically allows us an unconstrained number of environments for engineers to integrate, test and code," he said. They can rapidly identify and fix issues before they become full blown bigger issues, he added, checking their work daily.
"The traditional way the Department (of Defense) does software," Whitney said, "has always been this massive waterfall approach where we design it, and then we go through this massive coding period and then we go into an integration period and then at the end of that integration you have a success or not. But that takes months to get through. What we’re talking about doing, what the cloud enables us to do, is to do that in a much more rapid, almost on a daily basis. And it allows us to do it in multiple strings. So if Coder Number 1 has his work done he can stand up his own cloud environment to be able to test out what he’s doing and Coder Number 2 can do it for his as well. So it saves us a lot in terms of duplication. We’re not constrained by resources … and it also provides us the opportunity to quickly get those results back and check it out that night or by the next morning."
The biggest disadvantage of the approach is that the software environment is, indeed, virtual. When the time comes to integrate the system into the actual, new OCX hardware there will be an integration period then full-up qualification testing and full-up acceptance testing, said Whitney.
Working across the internet also raises security concerns, a particular worry for OCX, which has a cybersecurity mandate at its core. AWS announced September 17 of this year that it had achieved a Provisional Authorization (PA) by the Defense Information Systems Agency (DISA) for Impact Level (IL) 5 workloads.
"To enable extremely high security levels for our customers, AWS employs a robust set of security technologies and practices, including encryption and access control features that exceed DoD security requirements," the company said in a statement.
AWS said the OCX program required more than 200 dedicated hosts running upwards of 1,000 individual virtual machines. Each virtual machine needed at least eight vCPUs and 32GB RAM. "When the Air Force looked at other cloud providers," said AWS, "none of them were able to immediately handle the compute scale while also meeting the DOD CC SRG IL5 requirements. Not only did AWS meet the Air Force’s needs," the company wrote, "but the Air Force also experienced a 30 percent cost savings for storage costs."
The AWS statement underscores one of the key advantages of doing development in the cloud — it’s rented space. DoD is not buying hardware or building and maintaining a server farm of its own to do the development and testing. The virtual environments "can be built up and torn down and there’s no residuals," said Whitney. There’s no need to update the computers every three or four years, he said, and "I don’t have to invest massive amounts of money in separate hardware strings."
The experience working this way is also useful, said Whitney, because ultimately, OCX will operate in its own cloud. The hardware prime contractor that Raytheon is delivering will enable GPS managers to create and run OCX as a cloud operation — a virtualized capability that gives the system a lot of its security mechanisms.
"This is where it kind of gets a little bit different," said Whitney. "Our operational system is resident inside of our system that we have, our servers — those set of servers will form our own OCX cloud. We’ll be able to stand up environments inside of there." Having its own set of hardware enables the Directorate to control the boundary spaces of its cloud, Whitney said.
Operating in a cloud enables the Directorate to create virtualized environments inside of its servers," he said, "and if we run into issues with something running on them from a security perspective, it’s easy to tear it down and start over." When a set of software patches change or the security settings need to be adjusted, "it’s easy to stand up a new environment with new settings and not have to go through a massive deployment effort."
Whitney said he foresees the Directorate continuing its work on the AWS cloud as it tests new technology to incorporate into the soon-to-be announced next tranche of GPS III spacecraft.
"Our intent is to keep using AWS for both our on-ramp testing," he said, "and we’re looking at what potential benefits it may have as to the sustainment effort once (OCX) has been operationally deployed."
