Q. What are the fundamentals of an effective GNSS test plan?
A. One aspect of GNSS development that engineers often find challenging is the lack of common testing standards and procedures. This can make life difficult for the engineer tasked with constructing a test plan for a new GNSS-enabled system. How much testing is proportionate, at which stages of development? What are the key performance parameters to measure? What apparatus is best suited to the application, and what are the appropriate pass/fail criteria?
Q. What are the fundamentals of an effective GNSS test plan?
A. One aspect of GNSS development that engineers often find challenging is the lack of common testing standards and procedures. This can make life difficult for the engineer tasked with constructing a test plan for a new GNSS-enabled system. How much testing is proportionate, at which stages of development? What are the key performance parameters to measure? What apparatus is best suited to the application, and what are the appropriate pass/fail criteria?
The Problem
Many different factors influence the performance of a GNSS-enabled system — from its internal circuitry to the signal environment where it is used. The role of the test engineer is to characterize the performance of the system to ensure that it conforms to the manufacturer’s quality standards and meets the expectations of the end-user. To do that, most GNSS test regimes measure the receiver’s performance in several fundamental areas: time to first fix, receiver sensitivity, positioning accuracy, re-acquisition time, and robustness to threats.
While all of these parameters are essential to measure, their importance may differ from application to application. For example, some applications need to find a position quickly or to work with low signal levels, while for others, users will care more about absolute accuracy. Time to first fix (TTFF) and re-acquisition time may be extremely important for automotive navigation but less of a priority in applications such as surveying, where position accuracy is likely to be the primary concern.
In a precision timing application, the receiver may not need a rapid acquisition but may require a high level of tracking sensitivity, as well as the ability to generate a system warning if high levels of radio frequency (RF) interference are present, or a signal anomaly or change in receiver position is detected.
Five Fundamental Tests
Depending on the intended end-user application, engineers may choose to test these receiver performance parameters in basic conditions first, such as different orientations, motion speeds, urban or open country environments, and then add further conditions as required, for example, different pressures, temperatures, rates of acceleration, or satellite constellations in use. Let’s take a closer look at these five key tests.
1. Time to first fix (TTFF) for cold, warm and hot starts. On start-up, how quickly does the receiver need to begin tracking satellites and outputting useful information? At the most basic level, testing can be as straightforward as deducing how long the receiver typically takes to achieve a satellite fix and, critically, report an acceptable position and/or time calculation.
However, this test should also take account of how much data the chipset already has about its likely location and time when it is used; pre-existing almanac, ephemeris and approximate location data can have a significant effect on the TTFF result.
Testing should therefore consider cold, warm, and hot starts — giving greatest weight to the state that most closely resembles the device’s typical use. For a more detailed overview of TTFF start-up states, see Table 1 (see photo at the top of this article for all Figures and Tables).
To quickly simulate cold start conditions, engineers with GNSS simulators often create test scenarios that virtually place a receiver at opposite sides of the world, then alternate between them — thus contradicting the stored data in the receiver under test without needing to delete it each time.
2. Receiver sensitivity for both acquisition and tracking. Once satellite lock is achieved, it can usually be maintained well below the threshold power level for signal acquisition; so, engineers typically test both what strength of GNSS signal the receiver requires to acquire a fix and how well it can retain it when confronted with obstacles.
If the device needs to establish its first position where signals are weak — for example, indoors — then acquisition sensitivity is likely to be a key parameter. The most basic test establishes the minimum signal power level needed to obtain a position, velocity, and time solution, by attenuating to significantly below the nominal power level for the constellation, before gradually increasing the strength until an accurate position fix is achieved.
Tracking sensitivity is likely to be a valuable test for GNSS devices that will need to continue to produce acceptable results when signal strength varies — most commonly, when the device is on the move. The test is effectively the same as for acquisition sensitivity, only performed in reverse. Signal level begins slightly above the acquisition threshold and is gradually attenuated until the receiver can no longer track the code and carrier phase and provide an accurate, reliable position fix.
3. Time and/or positioning accuracy. Measuring the accuracy of the receiver is essential, and the question of acceptable performance is most often answered by the application. For some uses, position accuracy within several meters is perfectly adequate, whereas for others millimeter-level accuracy is an absolute requirement.
The test itself involves plotting the observed output from the receiver against known truth data in a controlled signal environment.
As GNSS positioning estimates are stochastic in nature, the test must be repeated several times and an overall probability calculation performed to determine the error radius. Figure 1 shows typical test results when comparing two receivers. For some applications, horizontal accuracy is sufficient; others will need to allow for spherical error in three dimensions.
For most applications, testing will need to confirm acceptable performance both when the receiver is stationary and when it is in motion. (Again, the nature of the movement is governed by the device’s likely use.)
4. Re-acquisition time following a loss of satellite signal. Consider how frustrating it would be if an in-vehicle navigation system took several minutes to recompute its position every time you drove through a tunnel. Time to re-acquisition may be critical to user satisfaction and can be easily tested by simulating a temporary loss of signal.
Testing in laboratory conditions involves using the simulator’s on/off commands to simulate a temporary signal outage, then observing receiver outputs to record the interval between the signal’s reintroduction and the receiver reestablishing an acceptable position reading.
Importantly, satellite geometry plays a critical role in time to reacquisition performance; so, when comparing components and designs it is essential to ensure this parameter is exactly the same for each test — whether using a simulator or a record-and-playback system (more on this later).
5. Robustness when faced with real-world threats. As users become more reliant on accurate positioning, navigation, and timing, measuring the receiver’s resilience when stressed by real-world threats — such as jamming, spoofing, solar weather, and segment errors — is becoming as fundamental as any other GNSS performance test.
Engineers are increasingly required to evaluate the impact of signal disruption, whether accidental (for example, through ionospheric scintillation or interference from nearby RF equipment) or in the form of deliberate jamming or spoofing. Tests can determine how the receiver performs as the interference becomes stronger and at what stage to generate user warnings.
The first step in resilience testing is a risk assessment to determine the probability that systems or equipment will encounter one of the identified GNSS threat vectors in day-to-day operations and, if so, what the nature of the threat(s) or vulnerabilities may be. First characterizing the RF interference environment using commercially available test tools may be appropriate. Then, the system’s response to the threats and vulnerabilities identified in the risk assessment can then be measured under laboratory conditions.
Today, we can simulate a variety of GNSS jamming scenarios with real interference waveforms captured from jammers or simulate different types of spoofing attack, with key system parameters being continuously monitored. This vulnerability audit of the system enables test engineers to make informed decisions about proportionate and cost-effective mitigation techniques and integrity-flag mechanisms.
Testing from R&D to Manufacturing
Engineers designing a GNSS test plan will want to consider how testing should progress as the product moves from research and development, through integration and verification, to manufacturing. For the R&D phase of a project, detailed controlled measurements (sometimes known as “parametric” testing) of both nominal conditions and error or extreme states are essential. By the time a product reaches the production line, simpler functional testing is often all that is needed.
The objective of R&D testing is to evaluate, compare, and adjust the system’s response in a wide variety of signal states. This requires that a series of very repeatable tests must be undertaken. The performance of the receiver should be tested against the relevant GNSS interface control document or interface specification. Alongside fundamental and application-specific tests, R&D testing should include evaluating the performance and accuracy of the receiver’s 1 pulse per second (1PPS) output, multi-GNSS performance, and the comparative effectiveness of any mitigation mechanisms against real-world threats.
Integration testing has different requirements. The chipset or module has to be selected for integration, its performance evaluated, and the complete system tested for suitability for production — while simultaneously minimizing the time to market.
Integrators often find it challenging to recreate a chipset’s standalone performance in the context of the whole system, a process that involves allowing for RF losses and interference sources in the system circuitry itself. Where possible, sharing test scenarios with the chip manufacturer can help to establish common performance standards, speed approvals and testing, and reduce time to market.
Once integration testing has produced a baseline specification defining acceptable system performance, this can be used as the basis for “go/no-go” tests required in the production phase, allowing product performance to be confirmed without delaying the start of manufacturing any more than necessary.
However, establishing “go/no-go” tests for GNSS is itself problematic. Depending on its intended use, the receiver may need to be tested not just in an “ideal” situation but also under a wide variety of different conditions, as observed performance depends to a large extent on the GNSS system itself.
Many GNSS engineers therefore construct “go/no-go” production test schedules that consider best and worst-case satellite geometries, as well as other sources of error such as multipath, less than ideal atmospheric conditions (including scintillation if the receiver is to be deployed anywhere near the equator), and response to RF interference.
Establishing a Test Solution
When deciding upon a signal source for testing, live signals from satellites overhead are rarely suitable for a professionally conducted test. With constantly changing geometry and unpredictable atmospheric factors to consider, such “live sky” signals fail the engineer on almost every level: they lack knowledge of the true received signal parameters thus making results difficult to interpret, present an insufficient range of likely use cases, and make it impossible to truly compare like with like.
Instead, most GNSS engineers seek a reliable, controllable, and repeatable signal source. Most usually, this is either a simulator or an RF record-and-playback system. A simulator emulates one or more GNSS constellations and provides a replica RF signal to the receiver.
Simulators enable users to set up test cases with fine control over the test conditions. Record and playback systems sample real-world RF conditions that are then stored digitally and available for subsequent regeneration back to RF. Record-and-playback systems are good at capturing the richness of real-world conditions, whereas simulators are good at enabling control and flexibility. A rigorous test regime will often include aspects of both approaches.
Table 2 summarizes the key differences between a simulator, a record-and-playback system, and live sky testing.
Ideally, a signal source for GNSS testing should allow fine, accurate control over the signal — for example, enabling detailed attenuation to pinpoint a receiver’s tracking threshold. It should also have the flexibility to create scenarios that can both represent likely real-world uses and push the receiver to the limits of its performance.
The signal environment created should be complete and faithful to real life, exposing the device to nuanced, realistic signals in all their complexity and detail. The tests must be repeatable, and the whole apparatus should be fast to use, enabling the rapid, like-for-like iteration required to adequately compare performance and to ensure a statistically meaningful test. For this reason, a degree of automation can be invaluable.
Figure 2 shows a typical variables and sources of impairment to consider in a GNSS test setup.
Testing the Testing Apparatus
One final set of tests that is easily overlooked is to confirm the accuracy of the testing equipment itself. Variations in the performance of the apparatus can introduce systemic errors into the results; so, regular recalibration and certification of such equipment is an important safeguard.
