Tuesday, February 13, 2018

The Disappointment

No, I'm not bitching about the weather in Texas. Sure, it was near freezing yesterday, and about 15ºF colder than on Cape Cod .. but on a couple of days, we'll have 70 degrees here, and Cape Cod will be cold.

This is another GPS post. One about some major frustration - it took me 10 days to get ready to write about hit. This is about the results from a windsurfing test, comparing the USB GPS dongle to the GW-60. Here are the first numbers I looked at (from GPSResults):
The numbers are for 2 seconds, with GW-60 on the left and the dongle on the right. The dongle was 1.45 knots higher than the GW-60 .. very suspicious! A quick look at the tracks showed that the dongle was clearly wrong:
The dongle track is blue, the GW-60 dark green. The dongle has a clear artifact peak at the "top speed" region that is about 5 knots too high!

It did not end there. When I compared the results for the nautical mile, things also looked bad:
Again, the dongle gave substantially higher numbers - about 0.9 knots for the nautical mile!

Why did I suddenly get such bad results, after all my driving tests had indicated that the USB dongle would likely be more accurate than the GW-60?

The first hint came from the tracks that GPSLogit had recorded for the phone GPS. For several runs, the phone GPS tracks (in red below) also showed large artifacts:
The phone had slipped on my armband several times during the session; with the phone and USB dongle pointing down instead of up, and my upper arm and body blocking the GPS signal, the data quality suffered .. a lot. I verified this in the next two sessions, as I reported in my two previous posts. With the phone properly fixed so that it could not slide down, I got much more accurate, artifact-free results in two sessions.

I finally went back and gave the "bad" data a closer lock. The nautical mile results looked suspicious, so I looked at the numbers in GPS Action Replay. First the GW-60:
Next the USB dongle:
Finally, the phone:
All the numbers are very close - within 0.1 knots of each other, and within 0.03 knots between the GW-60 and the USB dongle! An examination of the data showed that, once again, missing data points and GPSResult's approach to "filtering" were the cause. At the beginning of the fastest nautical mile run, the GW-60 skipped quite a number of data points; instead of recording a point every 0.2 seconds, it recorded at irregular intervals for about 50 seconds: mostly one point every second, but also at 0.6, 0.8 second, and 3 second intervals. Here's a screen shot of the track point table from the start of the nautical mile run:
Since GPSResults will generally ignore sections with missing data points, it picked a slower section of the track as the "fastest" nautical mile. So this time, the observed differences in the nautical mile results were due to (a) the GW-60 missing data points, and (b) GPSResults not using this section. The effect was that the top speed for the nautical mile was understated by 0.9 knots because the GW-60 dropped points. The GPS dongle and the phone both gave the more accurate results!

This was quite different to the 2-second results, which are much more prone to distortions from artifacts (this is even more so for the "top speed", which is very often too high by 0.5-1 knots!). Here's a look at the data from the USB dongle in this region:
Note that eight of the ten points have error estimates above 1 knot, and 5 points have error estimates above 1.2 knots. Almost a year ago, when I looked at artifacts in GW-60 tracks, I had come to the conclusion that 2 second results with an average SDoP above 1.4 should not be considered for GW-60 data, since they are quite likely to be highly inaccurate. It is safe to assume that the u-blox GPS chip inside the USB dongle calculates the "+/-" numbers a bit differently than the GPS chip inside the GW-60; it appears that a slightly lower threshold may be necessary. When applying a SDoP filter with a 1.2 knot threshold in GPSResults to the dongle data from this session, the artifact peak is filtered out, and the top speed numbers for the dongle are very close to the GW-60 numbers.

One problem with accuracy filtering is that GPS Action Replay currently does not support it; another issue is that it can easily be turned off in GPSResults. However, there is a simple solution for the USB dongle recording software: it could simply set apply the filter before saving the data to a file, for example by setting the speed in the affected region to 0. The app could then also alert the user about the issue with a message on the screen and audio feedback, which could serve as a little reminder to check if the phone and dongle are positioned correctly. I'll look into this, but first I plan to collect more data with the USB dongle while windsurfing, to see if such a filter would possibly through out "good" data.

Friday, February 9, 2018

More GPS Comparison Data

Below are more results from comparing three GPS devices: the Locosys GW-60 watch, an Android phone (Moto E4) with GPS Logit, and the USB GPS dongle I had been testing. These results are from a Kona One session 2 days ago, as reported by GPSResults:
No surprise for the 2 and 10 second results. All differences for the phone are well within the +/- ranges for the watch, with maximum differences of 0.2 knots for 2 seconds, and 0.09 knots for 10 seconds. Numbers for the phone GPS are similar, but slightly higher. Note that for this session, the GW-60 watch and the GPSLogit data were recorded at 1 Hz, while the USB dongle recorded at 5 Hz.

The results for 500 m are similar, except that the phone GPS now does as well as the dongle. For the nautical mile, the results reported by GPSResults differ a lot for the phone. This is because GPSResults filtered out single data points in the middle of a few one-mile runs:
The sixth point is shown in brackets, meaning GPSResults filtered it. Since the GPSLogit data do not have accuracy estimates, and both acceleration and HDoP around this point is low, we have to assume that GPSResults considers data from 7 GPS satellites as insufficient (unfortunately, the Mac version of GPSResults does not show a setting for minimum number of satellites). Unchecking the "On" checkbox for all filters leads to results that are more similar to the watch and dongle:
However, the last 3 nautical mile runs for the phone show larger differences even without filters. This, however, appears to be an artifact of the analysis: the entire session was only about 20 nautical miles long, with just 12.9 nautical miles at speeds above 12 knots. Looking at the sections in GPS Action Replay shows that the speeds over the aligned nautical mile regions only differ by 0.01-0.05 knots; the reported larger changes are caused by slightly different regions being picked for the different units by the analysis programs. For the GPSLogit tracks, GPSResults was able to "squeeze in" a extra nautical mile run of 13.49 knots, while the same region between previously picked (higher speed) nautical miles was just a tad shorter than 1 nautical mile in the data from the two other devices. For speedsurfing competitions, this kind of artificial difference is entirely irrelevant; it only shows that some care has to be taken when analyzing the data.

Wednesday, February 7, 2018

GPS Dongle Test Results

We finally got some wind to test the USB GPS dongles while windsurfing, and the initial results were frustrating. In the driving tests, both the USB dongle and the phone GPS (through GPSLogit) had given results that always were very close to the results from the"gold standard" GW-60 GPS watch. But in two successive windsurf sessions, the results differed quite a bit more, and the tracks showed some disturbing artifacts. Here is an example of a particular bad section in GPSLogit data:
The GW-60 speeds are shown in blue, the phone GPS data are shown in read. Over the selected region, which is a bit more than 1 km long, the phone gave an average speed that was 0.81 knots lower than the GW-60 speed. The cause is obvious: more than 10 times during this run, the phone GPS speed showed a sudden drop by several knots that often lasted 2-4 seconds.

What was puzzling about this was that in other runs of the same session, the phone data did not show any "sudden drop" artifacts; instead, they mirrored the GW-60 data quite closely, as shown in the image below:
What was going on? When I wrote about the negative results on the Seabreeze forum, several experts posted useful hints. The GPS antennas in the USB dongle and the phone are quite small, and therefore sensitive to poor reception. In the driving tests, the antenna orientation was always near-optimal and essentially fixed. But during my windsurfing tests, the waterproof bag on the armband I used often slipped from the top of my upper arm to the underside or the inside of my arm, which effectively block a lot of the GPS signal. At the root of the problem was the kind of armband I used, which allowed the bag quite a bit of movement along the armband. Other armbands, like the more expensive Aquapac, don't allow this kind of movement.

After having spent more than $100 for Android phones and USB dongles, and after getting very good results in driving tests, I was not quite ready to give up. Instead, I attacked a piece of velcro to the backside of my waterproof bag, which keeps the bag from slipping on the armband, and the phone and USB dongle always looking upwards towards the GPS satellites. I did two more tests - yesterday on a Kona One for a couple of hours, and a shorter session today on a speedboard in confused chop and light rain. In both sessions, the speed results from the phone GPS and the USB dongle were much closer to the GW-60 data, and the tracks did not show weird artifacts. Here are some data from today's session, analyzed by GPSResults:
The 2-second results are most prone to distortions from "noise" in the data, but for all 8 runs with speeds above 20 knots, the numbers for the phone and USB dongle are within about 0.2 knots of the GW-60 numbers: well within the estimated accuracy of +/- 0.3 knots. On average, the absolute difference is just 0.11 knots.

For 10 seconds and longer runs, the expectation is that the numbers should be even closer, since distortions from random noise should largely cancel each other out. Here are the numbers for 10 seconds:
For the USB dongle, all numbers are within 0.1 knots of the GW-60 numbers, and also within the estimated accuracy window - great! The differences for the phone GPS, as recorded by GPSLogit, are slightly larger, which may be partially due to the slower measuring rate (1 Hz vs. 5 Hz for GW-60 and USB dongle).

The 500 m data look similar:
 Again, all the dongle data are within the (rather small) error range of the GW-60 results. The same is true for 6 of the 8 GPSLogit results, but 2 are slightly outside of the error range.

When I looked at the results for the nautical mile, I was in for a surprise:
GPSResults gave 4 nautical mile results, the fastest at 18.6 knots, for the USB dongle; but for the watch, it gave only 2 results, the fastest at  only 17.8 knots. A closer look at the data revealed the cause:
My nautical mile runs all had a jibe in the middle, which usually was wet (I still have to learn how to jibe the 72 l speedboard in chop). In the middle of the fastest nm run, this caused the GW-60 to drop three points, probably because the watch was under water. I wore the watch on my wrist, while the phone and USB dongle were on my upper arm, and therefore may not have even been under water. With default settings, GPSResults will not include any region with any missing data points in any speed results up to the nautical mile, so it did not report the fastest nautical mile run for the watch. Similarly, it dropped the 4th-fastest run, and selected a different (and slower) region for the second-fastest run it reported.

Fortunately, GPS Action Replay is not as strict about missing data points, and gave results that included the regions with missing points:
Again, the results are very close to each other, with a slight advantage going to the USB dongle.

These results indicate that the USB dongle and the phone GPS in the Moto E4 can give very accurate results while windsurfing, if it is ensured that the phone is facing up, and cannot slip down. The dongle seem to be slightly more accurate, although the data set presented here is very small; additional tests would be needed to see if the observed difference holds up. The more important issue, however, is that the u-blox based USB dongle provides accuracy estimates, which can be used to automatically identify artifacts caused by poor reception.

More tests to come .. whenever the wind comes back. Today was wicked cold for Texas standards, with air temperatures dropping below 10ºC during our session. We almost could have stayed on Cape Cod!

Thursday, February 1, 2018

South Padre Island

What do you do if you've got a new board but it's not windy in Corpus Christi?
You drive to South Padre Island! Well, at least that's what we did last Sunday, when the wind forecast for SPI was about 5 knots better than for Corpus Christi. Nina really wanted to try her brand-new Skate 86! As you may have guessed from Nina's expression, the trip was a big success - two days of wind, flat and shallow water, and even a freestyler showing Spocks, Funnels, and Switch Konos to keep Nina motivated (perhaps I should rather say "even more motivated").

Our first stop was at a kite boarding shop to get a pass for the SPI North Flats - an area a couple of miles from town that the local windsurf and kiteboard association has leased. Here's a picture:
It looks flat because it is flat - the water is about hip-deep for at least 3 miles out (and probably also three miles or more to the left and right). A perfect training ground for the OBX-Wind long distance race! On Sunday, Nina took out her Skate, but on Monday morning, she joined for a bit of slalom practice, and to improve our monthly numbers on the GPS Team Challenge. I had already tried to improve my personal best in the 1 hour category on Sunday, but a broken harness line after 55 minutes stopped that (neither did it help that I got a bit disoriented and ended up a mile upwind of where I had started). But on Monday, things went smoothly, and we ended the session with three personal bests - 1 hour for both of us, and nautical mile for Nina. The session also got our team into the top-10 rankings for the month of January in the 1 hour and distance categories. Nice!

After finishing the 1 hour, I tried to go out on my 72 l speed board, but just then, the wind took a little break: meter readings dropped from 18 mph averages down to 15. Even considering that the meter reads low, that was too little for the small board, and I switched to using my freestyle gear for the first time here in Texas. Using a light 5.6 m freestyle sail after the 7.8 m 3-cam freerace sail felt so easy!

After a much-needed break day, we were back on the water yesterday. I rented a Goya Bolt 115 (nice board!), Nina had a tuning and jibe practice session on the Falcon 99 / Racing Blade 7.0 combo. Perhaps I'm biased, but I think she looked good on the water:

There are a couple more things I want to talk about, including the new "Do it yourself" kit from Ianovated that lets you convert any wetsuit into a "heated tubes" wetsuit for cold weather windsurfing - but the forecast for tomorrow predicts great NE wind in the morning, so I'll leave you with one more picture of Nina - this time from the "I got my Skate!" celebration at a local brewery:

Thursday, January 25, 2018

Speed Deltas

I finally got a couple of the new Delta XT-50 fins for speedsurfing in shallow water from Maui Ultra Fins. But we did not have any wind the last 2 days, so that's not what the post is about - instead, it's another post about GPS tests.

I had to pick up a sail that Bob had repaired at the other end of Corpus Christi, so I put three GW-60 watches on the dashboard, and added two phones with USB GPS dongles. I analyzed the results by exporting the speed results from GPSResults, importing them into a spreadsheet, and comparing the top 10 individual speeds in various categories. . Here is what that looked like for the 10 second category:
That's a lot of numbers, so let's look at the differences instead. The next table shows the differences for each number relative to the average of the numbers I got for 2 GW-60s run at 5 Hz, the current "gold standard":
The three rows at the bottom show the average difference for each device, the standard deviation of the differences, and the maximum absolute observed difference in the top 10 speeds.

So far, so good. The maximum observed differences are below 0.1 knots, the typical differences even smaller - that great accuracy. The numbers for several other categories looked similar, but for the 500m category, one of the top-10 results showed much larger differences:
For the 9th value, the dongles reported speeds that were 0.2-0.4 knots higher than what the GW-60s reported - that's a cause for concern. Here are the 500m speeds:
The numbers that appear to be the outliers are highlighted in red. Note, however, that this is based on the assumption that the GW-60 numbers are correct. I examined the traces for the 500m region in question, and observed something peculiar at the start:
The top part of the graph shows doppler speed, the lower part shows accuracy estimates (SDoP). The GW-60s are shown in blue and red, the USB dongles in green and magenta. The GW-60s have much higher error estimates in this region:

The "+/-" numbers are about 2x higher for the GW-60s, indicating much lower accuracy. That the GW-60 numbers are inaccurate can also be seen by looking at the numbers: from points 5453 to 5455, the second GW-60 says I accelerated from 12.4 knots to 17.2 knots in 0.4 seconds. I can assure you that my van does not accelerate like that!

Here is a map view of this start region:
You can see a few things in the image. The "high error" region was when I took a left turn off the highway, and had to drive under the highway, which blocked the GPS signal. Apparently, the GW-60 was more affected by this than the USB dongle; this is in line with previous observations that the watch is more sensitive towards poor GPS reception. You can also see that this 500 m run has low speeds both at the beginning and the end. Because of this, the "understated" speed from the GW-60s at the beginning has a large effect on the average speed over 500 meters, since the region has to be extended by including more low-speed data at the end.

The bottom line from this test: overall, the results from the USB dongles are very similar to the GW-60s, with observed differences below 0.1 knots for most categories. In the one case where the differences were substantially larger, there are multiple indications that the USB dongles gave more accurate speeds than the GW-60, probably because the u-blox chip and antenna in the dongle handle situations of poor GPS signal strength better than the GW-60. This example also underscores the usefulness of accuracy estimates (SDoP values), although it also indicates that the currently used thresholds of 3.0 to 4.0 may be too high.

Saturday, January 20, 2018

Dongle Update

We tried to escape the cold weather on Cape Cod, and drove three days to Texas. But after one decent day of sailing, temperatures dropped to below freezing, and stayed at low for the rest of the week.  Too low to go sailing .. maybe we're too spoiled already. But at least it gave me time to play around with the GPS dongles some more.

The first goal was to use two dongles at the same time so that I could compare the results - any difference is definite noise. I was able to record from two dongles on one phone with a USB hub, but the hub itself introduced some electronic interference which degraded the signal. So a second phone was needed.

Finding a cheap Android phone with USB OTG support was not easy. The first two phones I tried, which had been described as having OTG on some web sites, did not work. I finally found a Moto E4 for $40 at Best Buy that worked, and did a few test drives. After reading that the Bavarian Speedkini contest now allows the use of phones with GPSLogit or Windsport Tracker, I decided to record at 1 Hz, so that I could compare the phone GPS and the USB dongle to the "gold standard" GW-60. Here is a speed track from the latest test drive:

The graph shows tracks from 2 GW-60 phones in blue and red - but since the data are almost identical, only the red track is visible most of the time. The tracks for the two phones (Samsung Galaxy J3 and Moto E4) and for the two GPS dongles look pretty much the same.

A typical way of comparing GPS units is to simply compare the speed results, for example for 2 second and 10 second speeds. That is intuitive, but it has the disadvantage that only a very small subset of the data is used; everything that's not part of the top-speed regions is effectively ignored. To answer the more interesting question about how close the data are in general, I wrote a little program that calculates the differences between the tracks over all points above a minimum speed of 15 knots. It gives the average of the absolute differences, which is a good indicator of average reproducibility.  Here are the results for this test:

  • Average absolute differences between 2 tracks:
    • GW-60: 0.035 knots
    • Phone GPS: 0.051 knots
    • USB dongle: 0.025 knots
All these differences are quite small - so far, so good. How about the largest differences?
  • Maximum observed differences between 2 tracks:
    • GW-60: 1.07 knots
    • Phone GPS: 0.796 knots
    • USB dongle: 0.318 knots
Let's look at these areas - first the GW-60 watches:
Next the phones:
Finally, the USB dongles:
Visual examination of the traces confirmed what the numbers indicated: the phone traces have noticeable more diverging areas than the USB dongles, and the GW-60 is somewhere in the middle. The fact that the USB dongles and not the GW-60s had the most consistent data is somewhat surprising, considering that the GW-60 data are actually averages build from higher-frequency data, while the dongle data are (presumably!) single-point measurements. Theoretically, the averages data should have about 2-fold less noise than single-point measurements.

Comparing two identical units only gives us a lower limit of errors in the data, since it measures only truly random noise. Comparing the different GPS types to each other is more interesting, and did indeed give larger numbers for observed differences. However, a closer look revealed that there are some systematic error sources that need to be addressed first - take a look at the image below for an example:

The graph shows a comparison of a GW-60 track in blue and a USB dongle track in red. The GW-60 track is time-shifted relative to the dongle track - it's about 0.6 to 0.8 seconds behind. The likely cause for this particular shift is that the GW-60 reports averages of the last five 5-Hz measurements; the shift is not present when comparing data recorded at 5 Hz. However, I have noticed similar time shifts when comparing tracks from different phones; in this case, the underlying cause is probably different (e.g. the GPS chips in the phones take measurements at different times).

For a thorough comparison of data from different devices, the observed time shifts would need to be corrected. That is algorithmically quite simple, but will require a few hours of programming and testing, so it may be a while before I get to it. However, some of the data can already be analyzed at least qualitatively, as shown in the graph below:

The speeds shown are from the GPS chip in the Moto E4 (blue) and two USB dongles (green and red). In this region, the phone GPS was quite inaccurate. The two values at and before the vertical line were about 1 knot too low, and soon followed by 2 values that were about 1/2 knot too high, followed by another down-up cycle. If this artifact had occurred in the  top-speed region, the measured 2-second speed might have been 1 knot too low, or 0.5 knots to high.

The data above are just from one 15-minute test drive, with about 800 data points used for the analysis, so this is just anecdotal evidence. It's definitely way too early to claim that the USB dongle has better accuracy than the GW-60, that would require a lot more tests. However, it is once again an encouraging sign, and the observations are in agreement with what I have seen in other tests. Hopefully, we'll get some wind tomorrow for a windsurfing test!