Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round

Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round

Fitness trackers and smartwatches have somewhat merged over the past 18 months. Having discovered that the niche for wearables is just way too tight, manufacturers started bonding features from the two platforms, ultimately ending with devices that function pretty similarly. Sure, dedicated fitness trackers have a rubber band, a more rugged look, a smaller screen, and possibly no touch functionality, while smartwatches will rock leather or nylon bands, have big screens, and look more high-tech. But at the end of the day, feature-wise, both categories will usually have a time display, an HR sensor, step-counter, vibration and alarms for calls and texts, and either have their own GPS or use your phone's GPS.

And, for those of us that wear them, we do rely on our trackers / watches to let us know how much physical activity we've been getting, how well our heart is doing, and how many calories we've burned today. But there's this small caveat slinking through the back of our minds, reminding us that there are not enough scientific tests out there to tell us if any of our wearables' readings are correct or actually mean something. So, that expensive piece of gear may actually be giving us all the wrong data.

The Multidisciplinary Digital Publishing Institute (MDPI) — an academic publisher that produces 178 diverse electronic journals — published a study, which pits 7 popular smartwatches / fitness trackers against each other and measures their error margins. The point of the examination is more to create a framework for wearable device evaluation, but it also shows us some interesting results.

Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round
Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round
Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round
Study tries to measure fitness trackers' actual accuracy, Apple Watch wins first round
The tested devices were the Apple Watch, Samsung Gear S2, Microsoft Band, Basis Peak, Fitbit Surge, Mio Alpha 2, and PulseOn. They were strapped to 60 different volunteers, 29 male and 31 female, with an age range between 27 and 49 years old. In order to provide an accurate reference point in the experiment, the subjects were also continuously monitored with telemetry and indirect calorimetry.

Each subject went through 4 stages — sitting, walking, running, and cycling — and wearable readings were evaluated for each of these actions. Interestingly enough, all devices were most accurate in the cycling stage, while they were most off in the walking stage — the Basis Peak most prominently went way off the mark on its walking reads. Another piece of information, which we found curious — reading errors were higher in males with higher body mass index and / or darker skin tones.

Now, here's the interesting part — error margins and reading accuracy. As far as heart rate readings go, all devices were pretty accurate, with the Apple Watch having the best error margin (a median of 2.0%), and the Samsung Gear S2 being the worst (a median of 6.8%). But when it came to energy expenditure (calories burned), all the wearables were pretty off — none had an error below the 20% mark, and the PulseOn was a staggering 92.6% off with its stats.

The study concluded that the Apple Watch is most accurate of the tested devices (when combining its HR and EE results), but did advise that users don't rely on their fitness trackers' calorie counters too much when working towards a health-improvement program.

Of course, this work was primarily intended to define a set of standards, by which wearables should be tested. Do keep in mind that the devices used are a generation or so old — Sammy has the Gear S3 out, and Microsoft's Band has a second entry as well. Still, the results are an interesting read.

source: MDPI 

FEATURED VIDEO

22 Comments

1. pongkie

Posts: 663; Member since: Aug 20, 2011

my gear s3 has problems detecting floors climbed. also the calories burned on the exercise machine and s3 are far too different that I don't know what to trust.

2. sgodsell

Posts: 6588; Member since: Mar 16, 2013

The only devices tested were "Apple Watch, Samsung Gear S2, Microsoft Band, Basis Peak, Fitbit Surge, Mio Alpha 2, and PulseOn". Right there that is not a large sample at all. My Polar M600 has a very accurate heart rate monitor, and it's GPS lasts a long time as well.

6. AmashAziz

Posts: 2867; Member since: Jun 30, 2014

The tested devices are some of the most popular.

17. ph00ny

Posts: 1948; Member since: May 26, 2011

Isn't MS band on it's third version?

22. Jason78

Posts: 281; Member since: Apr 10, 2013

Per the article - all the devices tested had accurate heart rate monitors. It was estimating calories burned that was way off on all the devices.

11. nikhil23

Posts: 411; Member since: Dec 07, 2016

hmmm wierd. My s3 detects floors perfectly. Although a floor height is kind of fixed so that users don't feel bad climbing a large staircase (equivalent to 2 floors or so) only to find out that it records only 1 floor. So, in a scenario where you climb one floor but the floor is too high, s3 detects that as 1+ floors. I can't speak for calories burned on ex machine as your ex machine knows the pace at which you are working out while any smartwatch doesn't have that info, so smartwatch takes your HR to calculate the calories while ex machine uses your speed and other metrics. if your ex machine supports HR, calculations would be more accurate

3. Phonehex

Posts: 699; Member since: Feb 16, 2016

My garmin is 10 times better than my apple watch.

4. SuperAndroid507

Posts: 361; Member since: Jan 06, 2014

After almost two weeks not visiting the site and the first post I see makes me don't want to come here anymore...

7. AmashAziz

Posts: 2867; Member since: Jun 30, 2014

We know you hate to see Apple winning. Get over it.

9. pongkie

Posts: 663; Member since: Aug 20, 2011

if you are referring to my post, i'm sorry but that is my feedback on my watch. I bought to pair it with my s8+ so i can track my activities better with samsung health. I even wear it during asleep. It's been months and samsung hasn't done anything to patch the problem. but I like the design better, the iwatch looks like too much of a girl's watch, fit bit looks ok but i prefer to have the spotify app. for microsoft looks like they just given up and went all enterprise.

18. ph00ny

Posts: 1948; Member since: May 26, 2011

It's odd. S3 seems to be pretty close to my daily stairway usage. What kind of building? They had posted what they considered a floor since not all the buildings are designed the same

16. therealestmc

Posts: 678; Member since: Jul 23, 2012

Well don't come here then. No one is forcing you to do anything.

5. krystian

Posts: 423; Member since: Mar 16, 2016

Why wouldn't they use Microsoft band 2 which during the time was already out? It is an insane improvement and one of the best for sensors. It was just designed poorly with the band causing issues.

8. AmashAziz

Posts: 2867; Member since: Jun 30, 2014

Also the Watch series 2, gear S3, etc.

10. jeroome86

Posts: 2314; Member since: Apr 12, 2012

Might be the best smart watch but not ready to jump on board yet. I rather have a fitness band. Other wise my Smart enough Casio will do just fine.

12. palmguy

Posts: 949; Member since: Mar 22, 2011

3 - 4 hundred fitness Apple tracker. Magical.

13. palmguy

Posts: 949; Member since: Mar 22, 2011

3 - 4 hundred plus fitness Apple tracker. Magical.

14. Crispin_Gatieza

Posts: 3039; Member since: Jan 23, 2014

I have an Apple Watch, Gear S2 and Band 2. Though it may lack many of the functions the others have, the Band 2 is a lot more accurate and it doesn't silence notifications on my phone like the Apple does.

15. brasstax

Posts: 534; Member since: Apr 16, 2014

Okay, i admit i clicked on this article after seeing the girl's picture :D

19. cnour

Posts: 2305; Member since: Sep 11, 2014

As usual, Apple is always perfect.

21. RebelwithoutaClue

Posts: 5465; Member since: Apr 05, 2013

Hahahahaha Oh you weren't sarcastic?

20. Techist

Posts: 311; Member since: Jan 27, 2015

My Moto 360 (original) is seriously accurate as a step counter, much more so than my Fitbit Charge or Fitbit Charge 2. Whenever I have tested for accuracy by literally counting my steps as I have walked around a 295 yard circuit, the Moto 360 has only ever been off by 1 step, whilst my Fitbits have been off by as many as 19 steps! But my Moto 360 doesn't automatically recognize and log my workout sessions nor do sleep tracking and metrics like my Fitbits do. I wish there was just one device to do it all accurately!

Latest Stories