It All Comes Down to Data
The system designers from the Extreme Attitude Team discuss data from some of the many maneuvers conducted by the high-speed acrobatic airplane.
Once the jet-fighter fumes dispersed, the Florida dust settled, and the engineers returned to Hexagon| NovAtel headquarters in Calgary, go time may have finished but crunch time was just getting underway. What would the data show after being run through the company’s many iterations of post-processing software? What would they learn, how would they learn it, and most importantly, how would it inform product road mapping and other developments? How could it benefit customers in a wide range of user communities?
We posed these questions to an Extreme Attitude triumvirate of Gordon Heidinger, segment manager, automotive and safety-critical systems; Sam Kiley-Kubik, geomatics engineer; and Kurt Bahan, manager, sensor fusion. Kiley-Kubik had actually ridden in one of the Aero L-39 Albatros test flights and was eager to share what he’d found in the earliest data runs through post processing.
“The first thing we did with this data,” he said, “once we got it back in the office is select one of the higher-grade IMUs, and that changes based on which of the flights we selected, since we had different equipment on different ones. Those all get run through our post-processing software, which is the commercially available Inertial Explorer that can do processing forward and backward in time. And because you’re doing this after the fact, it can get a much better trajectory out of it than the real time receivers could ever hope to.”
Running the data forward and backward through the software can help cope with GNSS outages, such as those that occur when the plane rolls over in the air and the GNSS antenna is obscured from the sky. During this outage, errors in the inertial solution will start accumulating over time, with an exponential upward growth; this is seen running the data forward. If the same procedure is conducted in reverse in post processing, starting at the back and moving to the front of the file, one again encounters the accumulating error. Taking the two trajectories and combining them, one can in effect meet in the middle of the outage and essentially average the two runs out to obtain a position error that is much lower than it would have been in real time.
“Using that forward-looking ability,” Kiley-Kubik continued, “we generate truths in position velocity and attitude for all that and then we take it back and compare it to all of the test units for each flight. So we’re mostly looking at position-velocity-attitude differences.”
Although still in the early stages of what could become months of data crunching, the team reported interesting findings but nothing particularly surprising. All of the units on all the flights (see Figures 1 and 2 for one example) performed well. Turning to a methodical maneuver-by-maneuver, Kiley-Kubik and colleagues found a few curiosities they want to look at more closely, particularly in the extreme attitude positions where the plane pitched up or down, close to 90 degrees.
“The other thing we’re doing with this data,” he added, “is passing it through several different firmware versions [of Inertial Explorer]. We have the ability to take the recorded data and process it again, through sort of different revisions of the code, and see how the changes have affected it. The code we flew was a release from this past summer.
“We were very curious also to test it out on sort of our main development branches and see if those changes are in fact helping. And it looks like they are.”
As to how that improvement is measured, that’s primarily done with the position, velocity and attitude error statistics. Especially as the plane rolls and turns over during its maneuvers, it’s obstructing GNSS antennas. “So you get a lot of error growth and it’s just how you best constrain that. Particularly with the way our installation is done in the aircraft, the antennas, with the exception of the SPAN Passenger mount, are fairly obstructed by the aircraft’s seats and body around them. So they don’t have the greatest view of the sky either.”
The error is constrained using tightly coupled INS and GNSS. The instruments are taking singular observations from GNSS satellites, in addition to the position as a whole, and using those through the Kalman filter updates to estimate the error in the initial solution.
Although the team’s equipment and software afford them the ability to perform various degrees of coupling, so far they have not experimented with comparing loosely versus tightly coupled sensors. It’s in their plans, but in the offing. The SPAN set-up is closer to a deeply coupled system.
Software Revisions
“With our post-process control,” Bahan said, “we’re able to get continuous absolute error estimates for position in three dimensions, velocity in three dimensions and attitude in three dimensions. Bahan also mentioned that typically for ground vehicles the most interesting maneuvers are in 2D, with the third z-axis dimension pretty stable and uninteresting. Not the case on the L-39; the third dimension is just as dynamic as the 2D.
“We focus in on those particular areas, and then compare performance across the [software] revision it was run on and future development branches that we have and we were testing. We’ll hyper-focus on certain areas to see if there’s performance improvements. We can make further firmware changes and run it again and see if we can improve it. And that can be done basically across all the dynamics of the test flight that we have collected over the four flights.”
By hyper-focus he means zoom in on it to examine the beginning of the airplane rolling or the line-of-sight to the satellites slowly rolling out of sight. That would be one case to study closely, perform a data crop and do the error analysis.
Inertial Explorer software can overlay the satellite usage to visualize how many satellites are actually being used in the positioning algorithms at that point, and track it through the maneuver. That gives a better idea as to the expectation of the error growth and if the IMU’s performance is worse than expected or desired—or, as it turned out in some instances, better than its spec figure.
This type of data isn’t easy to obtain—customers who use such vehicles aren’t able to share information—and will be used for continual future development, Bahan said, making it important to have multiple sensors with different layers of performance. That’s why the team chose IMUs across the company’s portfolio in different price/performance grades.
The more aviation-minded members of the Extreme Attitude Team designed the test flights (see Figures 3 and 4) to give as large a variability in dynamics and maneuvers as possible. That way, they add the collected data to their regression or continuous monitoring system, which runs on all the firmware changes that NovAtel makes.
In the process of continued product improvement, they can make a small algorithm change, run it before and after, and see the impacts and effects that change has on particular use cases or applications. This adds to their understanding of performance from an iterative level as they develop their code. Future feature development benefits as well.
Truth Talk
“What we’re sharing here is just raw trajectories: how the aircraft flew,” Bahan said. “It doesn’t really give anything away in terms of performance unless we overlaid them and one of them diverged and blew up. What we really look at is comparing latitude, longitude, height; North, East, up; velocity; and roll, pitch, azimuth. The full spectrum of positioning, which is what our customers expect.
“Different customers want better performance in the position domain. They may be less concerned about attitude or they may be less concerned about velocity. But we have requirements and specifications for all of our IMUs, that they should be performing within. Now, that is going to depend on the dynamics and the environment that you’re testing.
The statistics on NovAtel spec sheets come from land-vehicle dynamic environments, which don’t necessarily translate to an aircraft, which lose potential updates. For example, a vehicle can stop and the positioning unit can perform a zero velocity update to help constrain error growth during a GNSS outage. That’s not possible in an aircraft.
“So it becomes a more challenging environment,” Bahan said, “and that usually relates to some level of worst performance. What that is, is harder to say because we’ve never had this data with an IMU alongside of it at the quality level to perform a proper truth analysis.
“So, we will be looking at all of these trajectories in individual three-axes-per-PVA situations and ensuring that our solution is meeting expectations.”
NovAtel continually evolves its SPAN firmware, with developers working every day with micro changes, trying to tweak and improve, adding new features all the time, new constellations, new signals, all of which have downstream impacts into position. The company must be capable of evaluating all those changes at an individual level, ensuring specific algorithm changes don’t cause a degradation in performance.
“Having this data gives us an incredible value of being able to do that at an individual changeless level. Every time any developer, not just one within my team but one of the upstream teams, puts some type of change in, that would flow down through our system. On top of just the individual micro-change level, we also do bigger feature development. There’s potential there to make specific improvements to particular use cases.
Right Down to the Real Nitty Gritty
Kiley-Kubik began putting up plots on a shared display screen and zoomed into one section of the Immelmann-Split S-dynamic weaves section of flight, keying in on the Immelmann (Figures 5 and 6). This maneuver, also known as a roll-off-the-top, was traditionally used to re-engage in combat, transitioning from one attack vector to another. The pilot climbs steeply into a backward half-roll, emerging in an inverted position flying level at a higher altitude in the opposite direction; then the pilot rotates the aircraft to fly right side up.
“One of the things that happens there is your 3D Euler angles that we’re used to using for reporting pitch, roll and yaw sort of loses a lot of meaning. Your azimuth axis and your roll axis are essentially the same. And so if you look at it in terms of the roll and azimuth error, the two thick lines here, azimuth in cyan and roll in the deeper blue. As you go through that, the very top of that maneuver [just before second 251600], you get what shows up, at least on these plots, as an error.
“It’s really to do with the way this attitude is being represented. When you’re in this condition, what actually matters is the difference of these two parameters rather than their raw numbers by themselves. But all of the IMUs through this area show that behavior like this where it looks like the error’s getting away.”
He then moved the cursor one hundred seconds later, just prior to the 251700 mark on the plot, where the same thing occurs on a different maneuver, the Split S (Figure 7). In the Split S, essentially the opposite of an Immelmann and used to disengage from combat, the pilot inverts the plane, dives steeply backward and downward in a half-loop, and emerges in level flight in the opposite direction.
“You felt that personally too, right?” Heidinger interjected. “So you know.”
“Yeah, well, I thought we did a Split S,” Kiley-Kubik replied. “After watching the video, I think my brain is lying to me.”
“This is why you need our equipment, right?” Heidinger said. “That’s the story right there: don’t trust your brain, trust the GNSS receivers.”
All laughed, then Kiley-Kubik continued. “The other thing would be that if you look at the flight as a whole, once we start doing these super high-rate maneuvers or once the plane starts inverting, you can see an almost immediate decrease in your GNSS position accuracy. Which is both a symptom of losing your GNSS observations (blue in Figure 8) as you go over the top and the fact that we’re losing the TerraStar corrections (green). Where we had fairly low and consistent error estimates before, now all of a sudden we’re sort of falling back to a single-point type position which has a lot more noise in it. Then once the maneuvering calms down here toward the end again, we return to consistent estimates.”
To round out the data and maneuver sampling, he turned next to the Stall maneuver (Figure 9).
“You’re pitching up through 50–60 degrees and the aircraft is losing all of its horizontal speed. It’s essentially down to 27 meters per second, that’s like highway speed: Less than 100 kilometers an hour (60 mph). That’s his sort of horizontal velocity. And then you see, if you look at the altitude plot as vertical velocity, you’re just doing like a full freefall here, coasting on almost zero G, which is just a complete hoot to do when you’re in the aircraft. That’s probably the most roller coaster type maneuver.
“If you watch the video, I’m busy looking at the instruments watching what’s happening and then the pilot tells me to look out the window, and I look off to the left side. And we’re pitching up at that 60-degree angle. And I had no idea, my body had not processed that we were into that turn.”
Kiley-Kubik isn’t sure if he lost consciousness, but he knows he came close during the Immelmann toward the end of the flight.
Data, Specs and Applications
We came back to Earth, so to speak, talking about performance specifications. How do people analyze the quality or the appropriateness of one type of positioning solution versus another? What insight does this newly gained data bring to standard IMU specs?
Bahan took up the conversation to answer, “Every customer has a different use case with different potential requirements. Certain applications require a very accurate attitude solution. You’re doing LiDAR, photogrammetry, small angles in your attitude error are going to translate to big position errors in the solutions. Whereas other customers just want to navigate a car in a road. It’s less important.
“Every IMU that we support has a different smattering of performance. In general, they all fall in line from a cost and overarching performance point of view. As the price increases, you get better attitude performance, better outage performance, better velocity performance, less noise and more reliability in your outages.
“We want to understand the position errors [of each grade of IMU]. We can post process that in RTK, PPP, single point so we can get all that information continually offline. We want to know velocity error, because that’s important in this type of application. We want to know the attitude error. We had vibration sensors on the aircraft that allow us a separate type of control on high vibration potentials that might be going on in the aircraft. We can look at the individual IMU raw data and cross-compare what vibration and frequencies of motion it’s observing compared to the separate higher bandwidth sensor.
With this data, NovAtel customers, current and future, can understand the spec, the performance characteristics, and the capabilities that are enabled by SPAN and the various SPAN solutions.
The L-39 data provides the ability to run it through incremental algorithm changes and to compare trajectories, compare specific position-velocity-attitude, down to just looking at only the roll if necessary, to ensure new algorithm changes are making the work better.
Kiley-Kubik: “The biggest long-term benefit is having this in our library of datasets. It’s a flight or a trajectory and sort of attitude configuration we don’t see very much. It’s very useful to have that library to evaluate future changes against and to see if there’s any improvements we need to drive based on that, although I haven’t seen any particular yet.”
“I get excited,” Heidinger said, “when I know what we’ve done—especially being on the ground and observing the data. I really look forward to future customer discussions. Our confidence from a sales and business development point of view has literally been elevated. Our solutions have proven they can handle extreme conditions like this, and we know they’ll work on a multitude of other applications.”
Not such an extreme attitude afterall.
