Thursday, January 25, 2018

Speed Deltas

I finally got a couple of the new Delta XT-50 fins for speedsurfing in shallow water from Maui Ultra Fins. But we did not have any wind the last 2 days, so that's not what the post is about - instead, it's another post about GPS tests.

I had to pick up a sail that Bob had repaired at the other end of Corpus Christi, so I put three GW-60 watches on the dashboard, and added two phones with USB GPS dongles. I analyzed the results by exporting the speed results from GPSResults, importing them into a spreadsheet, and comparing the top 10 individual speeds in various categories. . Here is what that looked like for the 10 second category:
That's a lot of numbers, so let's look at the differences instead. The next table shows the differences for each number relative to the average of the numbers I got for 2 GW-60s run at 5 Hz, the current "gold standard":
The three rows at the bottom show the average difference for each device, the standard deviation of the differences, and the maximum absolute observed difference in the top 10 speeds.

So far, so good. The maximum observed differences are below 0.1 knots, the typical differences even smaller - that great accuracy. The numbers for several other categories looked similar, but for the 500m category, one of the top-10 results showed much larger differences:
For the 9th value, the dongles reported speeds that were 0.2-0.4 knots higher than what the GW-60s reported - that's a cause for concern. Here are the 500m speeds:
The numbers that appear to be the outliers are highlighted in red. Note, however, that this is based on the assumption that the GW-60 numbers are correct. I examined the traces for the 500m region in question, and observed something peculiar at the start:
The top part of the graph shows doppler speed, the lower part shows accuracy estimates (SDoP). The GW-60s are shown in blue and red, the USB dongles in green and magenta. The GW-60s have much higher error estimates in this region:

The "+/-" numbers are about 2x higher for the GW-60s, indicating much lower accuracy. That the GW-60 numbers are inaccurate can also be seen by looking at the numbers: from points 5453 to 5455, the second GW-60 says I accelerated from 12.4 knots to 17.2 knots in 0.4 seconds. I can assure you that my van does not accelerate like that!

Here is a map view of this start region:
You can see a few things in the image. The "high error" region was when I took a left turn off the highway, and had to drive under the highway, which blocked the GPS signal. Apparently, the GW-60 was more affected by this than the USB dongle; this is in line with previous observations that the watch is more sensitive towards poor GPS reception. You can also see that this 500 m run has low speeds both at the beginning and the end. Because of this, the "understated" speed from the GW-60s at the beginning has a large effect on the average speed over 500 meters, since the region has to be extended by including more low-speed data at the end.

The bottom line from this test: overall, the results from the USB dongles are very similar to the GW-60s, with observed differences below 0.1 knots for most categories. In the one case where the differences were substantially larger, there are multiple indications that the USB dongles gave more accurate speeds than the GW-60, probably because the u-blox chip and antenna in the dongle handle situations of poor GPS signal strength better than the GW-60. This example also underscores the usefulness of accuracy estimates (SDoP values), although it also indicates that the currently used thresholds of 3.0 to 4.0 may be too high.

Saturday, January 20, 2018

Dongle Update

We tried to escape the cold weather on Cape Cod, and drove three days to Texas. But after one decent day of sailing, temperatures dropped to below freezing, and stayed at low for the rest of the week.  Too low to go sailing .. maybe we're too spoiled already. But at least it gave me time to play around with the GPS dongles some more.

The first goal was to use two dongles at the same time so that I could compare the results - any difference is definite noise. I was able to record from two dongles on one phone with a USB hub, but the hub itself introduced some electronic interference which degraded the signal. So a second phone was needed.

Finding a cheap Android phone with USB OTG support was not easy. The first two phones I tried, which had been described as having OTG on some web sites, did not work. I finally found a Moto E4 for $40 at Best Buy that worked, and did a few test drives. After reading that the Bavarian Speedkini contest now allows the use of phones with GPSLogit or Windsport Tracker, I decided to record at 1 Hz, so that I could compare the phone GPS and the USB dongle to the "gold standard" GW-60. Here is a speed track from the latest test drive:

The graph shows tracks from 2 GW-60 phones in blue and red - but since the data are almost identical, only the red track is visible most of the time. The tracks for the two phones (Samsung Galaxy J3 and Moto E4) and for the two GPS dongles look pretty much the same.

A typical way of comparing GPS units is to simply compare the speed results, for example for 2 second and 10 second speeds. That is intuitive, but it has the disadvantage that only a very small subset of the data is used; everything that's not part of the top-speed regions is effectively ignored. To answer the more interesting question about how close the data are in general, I wrote a little program that calculates the differences between the tracks over all points above a minimum speed of 15 knots. It gives the average of the absolute differences, which is a good indicator of average reproducibility.  Here are the results for this test:

  • Average absolute differences between 2 tracks:
    • GW-60: 0.035 knots
    • Phone GPS: 0.051 knots
    • USB dongle: 0.025 knots
All these differences are quite small - so far, so good. How about the largest differences?
  • Maximum observed differences between 2 tracks:
    • GW-60: 1.07 knots
    • Phone GPS: 0.796 knots
    • USB dongle: 0.318 knots
Let's look at these areas - first the GW-60 watches:
Next the phones:
Finally, the USB dongles:
Visual examination of the traces confirmed what the numbers indicated: the phone traces have noticeable more diverging areas than the USB dongles, and the GW-60 is somewhere in the middle. The fact that the USB dongles and not the GW-60s had the most consistent data is somewhat surprising, considering that the GW-60 data are actually averages build from higher-frequency data, while the dongle data are (presumably!) single-point measurements. Theoretically, the averages data should have about 2-fold less noise than single-point measurements.

Comparing two identical units only gives us a lower limit of errors in the data, since it measures only truly random noise. Comparing the different GPS types to each other is more interesting, and did indeed give larger numbers for observed differences. However, a closer look revealed that there are some systematic error sources that need to be addressed first - take a look at the image below for an example:

The graph shows a comparison of a GW-60 track in blue and a USB dongle track in red. The GW-60 track is time-shifted relative to the dongle track - it's about 0.6 to 0.8 seconds behind. The likely cause for this particular shift is that the GW-60 reports averages of the last five 5-Hz measurements; the shift is not present when comparing data recorded at 5 Hz. However, I have noticed similar time shifts when comparing tracks from different phones; in this case, the underlying cause is probably different (e.g. the GPS chips in the phones take measurements at different times).

For a thorough comparison of data from different devices, the observed time shifts would need to be corrected. That is algorithmically quite simple, but will require a few hours of programming and testing, so it may be a while before I get to it. However, some of the data can already be analyzed at least qualitatively, as shown in the graph below:

The speeds shown are from the GPS chip in the Moto E4 (blue) and two USB dongles (green and red). In this region, the phone GPS was quite inaccurate. The two values at and before the vertical line were about 1 knot too low, and soon followed by 2 values that were about 1/2 knot too high, followed by another down-up cycle. If this artifact had occurred in the  top-speed region, the measured 2-second speed might have been 1 knot too low, or 0.5 knots to high.

The data above are just from one 15-minute test drive, with about 800 data points used for the analysis, so this is just anecdotal evidence. It's definitely way too early to claim that the USB dongle has better accuracy than the GW-60, that would require a lot more tests. However, it is once again an encouraging sign, and the observations are in agreement with what I have seen in other tests. Hopefully, we'll get some wind tomorrow for a windsurfing test!