Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round

Fitness trackers and smartwatches have somewhat merged over the past 18 months. Having discovered that the niche for wearables is just way too tight, manufacturers started bonding features from the two platforms, ultimately ending with devices that function pretty similarly. Sure, dedicated fitness trackers have a rubber band, a more rugged look, a smaller screen, and possibly no touch functionality, while smartwatches will rock leather or nylon bands, have big screens, and look more high-tech. But at the end of the day, feature-wise, both categories will usually have a time display, an HR sensor, step-counter, vibration and alarms for calls and texts, and either have their own GPS or use your phone's GPS.

And, for those of us that wear them, we do rely on our trackers / watches to let us know how much physical activity we've been getting, how well our heart is doing, and how many calories we've burned today. But there's this small caveat slinking through the back of our minds, reminding us that there are not enough scientific tests out there to tell us if any of our wearables' readings are correct or actually mean something. So, that expensive piece of gear may actually be giving us all the wrong data.

The Multidisciplinary Digital Publishing Institute (MDPI) — an academic publisher that produces 178 diverse electronic journals — published a study, which pits 7 popular smartwatches / fitness trackers against each other and measures their error margins. The point of the examination is more to create a framework for wearable device evaluation, but it also shows us some interesting results.

The tested devices were the Apple Watch, Samsung Gear S2, Microsoft Band, Basis Peak, Fitbit Surge, Mio Alpha 2, and PulseOn. They were strapped to 60 different volunteers, 29 male and 31 female, with an age range between 27 and 49 years old. In order to provide an accurate reference point in the experiment, the subjects were also continuously monitored with telemetry and indirect calorimetry.

Each subject went through 4 stages — sitting, walking, running, and cycling — and wearable readings were evaluated for each of these actions. Interestingly enough, all devices were most accurate in the cycling stage, while they were most off in the walking stage — the Basis Peak most prominently went way off the mark on its walking reads. Another piece of information, which we found curious — reading errors were higher in males with higher body mass index and / or darker skin tones.

Now, here's the interesting part — error margins and reading accuracy. As far as heart rate readings go, all devices were pretty accurate, with the Apple Watch having the best error margin (a median of 2.0%), and the Samsung Gear S2 being the worst (a median of 6.8%). But when it came to energy expenditure (calories burned), all the wearables were pretty off — none had an error below the 20% mark, and the PulseOn was a staggering 92.6% off with its stats.

The study concluded that the Apple Watch is most accurate of the tested devices (when combining its HR and EE results), but did advise that users don't rely on their fitness trackers' calorie counters too much when working towards a health-improvement program.

Of course, this work was primarily intended to define a set of standards, by which wearables should be tested. Do keep in mind that the devices used are a generation or so old — Sammy has the Gear S3 out, and Microsoft's Band has a second entry as well. Still, the results are an interesting read.