Why benchmarks are more important than you think

This article may contain personal views and opinion from the author.
Why benchmarks are more important than you think
If you’ve spent even a small amount of time reading about smartphones, you’ve probably seen various benchmark apps being mentioned and scores from different phones being compared. My goal today is not to explain how each benchmark works, but rather why it’s important to have such test results at our disposal.

Benchmark scores often start heated debates since they give a numeric value which objectively (ideally) shows how much better one phone performed than another. And with brands and models usually having their fair share of supporters, you often hear fans defending the “losing” phone by saying things such as “benchmarks don’t translate to real-world use” or something similar. And while this is somewhat true, despite benchmark apps like AnTuTu trying to make their tests as real-world-like as possible, it doesn’t make the results meaningless.

Real-world use is impossible to simulate consistently enough between phones without using preprogrammed software. So unless you want to add the variable that is a human, you’re stuck with a synthetic load. But that’s not a bad thing. Without tests that put chips to the absolute limit, how will we be able to tell which one performs better? If the test is somehow limited to a point where performance gains become unnoticeable for humans during use, then we’ll have 20+ phones with the equal top, what’s the point of that? It’s like capping top speed tests for cars at 100mph because no one should drive faster anyway.

The fight for bragging rights


Performance tests have always been for enthusiasts and finding out where a machine’s limit is. No one goes to a carrier’s store to ask the customer service representative about AnTuTu scores before choosing between a Samsung or an LG smartphone.

But people can make a competition out of anything and smartphones are no exception. Once companies noticed people are paying attention to benchmarks, the race was on. Finally, they could show in plain numbers how much better their device was. AnTuTu and Geekbench scores became the mobile equivalent of a lap time around the Nurburgring.

 
Does a car’s time around the famous race track tell you much about how it will drive during your daily commute? Far from it. But it does tell you it’s freakin’ fast. With phones, it’s not much different.


Of course, as with any competition, some take it too far. A few smartphone makers have been caught cheating on benchmarks to get a higher score. They do this by optimizing the phone to be faster when the specific benchmarking app is running. This is like having extra horsepower only while on a racetrack (which some cars actually do, but it’s clearly stated). This does make some people wary about the trustworthiness of benchmarks as a whole, but so far the app developers have proven to be on top of things, removing scores that were achieved using shenanigans.

What benchmarks really show us?


Continuing with the car analogy, the 0-60 mph seconds for cars isn’t really strongly connected to real-world use either since you rarely ever need to do that, but it’s still an important measurement because it shows the acceleration your car is capable of when you do need it, even if you’re not starting from a standstill. Similarly, when a benchmark app runs an overly complicated 3D scene on your smartphone, it’s meant to show how your phone will handle heavy 3D games.

Since benchmarks test devices under consistent loads, they are also helpful to check manufacturers’ claims for improvements between generations. If they say the new GPU is performing 25% faster then the results should be in that ballpark as well. If everyone is consistently getting results closer to 15% then you know something doesn’t add up.


With smartphones getting to the point that the high-end experience is almost indistinguishable between generations, without benchmarking apps it would be almost impossible to tell if the latest chips are actually as fast as they’re said to be. Now, you might say: “Why do manufacturers even make faster chips if people can’t tell the difference?” Because eventually, software developers start making apps that use the extra performance devices have at their disposal. In a way, software and hardware are pushing each other to achieve more.

Not all benchmarks give us abstract numbers that only mean something in the context of other phone’s results, however. Some, like those that measure battery life, for example, immediately give users a fair idea of what to expect from a device. Sure, brightness settings and all sorts of things will have an effect during day-to-day use, but when phones are tested under the same conditions, the best performers are usually the ones that will last you the longest as well.

If anything, we’d like to see newer, more sophisticated benchmarks being introduced. For example, ones that test more obscure features that manufacturers are marketing, as the various AI-powered functions that have sprung up like mushrooms after rain. That way people might get a better idea if these are actually improving things noticeably or are just another gimmick meant to sell more phones.


All in all, benchmarks are not only here to stay, but they’ll be playing an even bigger part in the future, as performance across the board improves and we start needing tests for the new technologies that will make their way into smartphones in the future. And who knows, maybe one day we’ll get a humanoid robot controlled by AI to benchmark overall user experience for us, complete with all the frustrations we might experience when handling a phone.

FEATURED VIDEO

50 Comments

1. phonefanboy

Posts: 49; Member since: Oct 06, 2011

It’s important because iPhone is losing in real world use and test - iPhoneArena

2. alligator

Posts: 99; Member since: Jan 09, 2016

You read my minds

10. almostdone

Posts: 448; Member since: Sep 25, 2012

Did anyone actually read through the whole piece of trash?

33. sgodsell

Posts: 7368; Member since: Mar 16, 2013

With Toms hardware site, when they benchmark, they use the same resolution displays at least for graphics test. Plus they use the same platform. Benchmarks are only relevant with the same platform, and if testing specifics, then you have to use the same tests. The benchmarking test apps are not on the same platforms, or using the same hardware, with the same resolutions displays or class of CPUs. Why doesn't PhoneArena, or this guy do tests with a PC, and a Note 9, or a PC, and a iPhone XS. Oh, wait the iPhones still to this day do not have any split screen multitasking, and as soon as you switch apps, the one app you left, is halted. That doesn't happen on PCs, or Android. Why not show off some benchmark tests with the Intel i9 - 16 cores, 32 GB of RAM to an iPhone XS? I wonder which one is faster? Oh, wait we could put a lowly graphics card in, or an expensive graphics card with 4096 graphic cores. Hmm, I wonder which is going to be faster. This guy needs to take some computer courses.

48. nodes

Posts: 1159; Member since: Mar 06, 2014

how many points difference between 1080p vs QHD on Samsung phones?

4. shaineql

Posts: 522; Member since: Apr 28, 2014

I know right, this year they even lose to S10 in games LMAO. Wait for cheap Chinese phones to get SD855

9. almostdone

Posts: 448; Member since: Sep 25, 2012

Yep all over the web and YouTube videos the S10 outperforms the latest iPhones in real world usage. iPA you suck more than Apple.

18. c.m.s

Posts: 236; Member since: Dec 10, 2017

Why are you hanging around here? You guys accuses PA to be so Pro-Apple yet there are more Anti-Apple people on this site them any other techsite I visit (including pure Android sites). Seriously, there are so many other techsites to visit, why not leave if you dislike it so much?

19. Cat97

Posts: 1893; Member since: Mar 02, 2017

Before they would talk about benchmarks, PhoneArena should do a proper battery life benchmark when reviewing products. One which takes into account real-life usage patterns, like stand-by power draw. Besides the N hours of screen time, the phone also spends up to 5-6 times more time in stand-by, which no benchmark captures. Because of their poor benchmarking skills phone manufacturers are not enticed to optimize their phones for real-world usage. In their 'benchmarks' iPhones are among the best, while in real life some high-end Androids last up to twice longer compared to iPhones.

43. cheetah2k

Posts: 2256; Member since: Jan 16, 2011

This is exactly why I choose GSM Arena's battery tests over iPA

50. WingMan

Posts: 263; Member since: Mar 28, 2008

"Real world usage" - 8 hours the phone should be on standby while in the office... Amazing. The realistic world usage is more than half of the phone's battery life is while the device is on standby and not in use. Who uses his device from full to dead on a daily basis with the screen on permanently and apps running.

25. blingblingthing

Posts: 963; Member since: Oct 23, 2012

Remember all the talk of iOS being more optimised, benchmarks saying A12 is better and Apple's flash memory being the fastest? Based on what the really word tests are showing, all three of these CAN'T be true. Based on what the S10 is doing, it seems Android is way more optimised to be beating iOS is the real world tests.

27. ph00ny

Posts: 2031; Member since: May 26, 2011

If they're putting this much emphasis on benchmark, they should actually have a consistent procedure created (average from 3 middle scores out of 5 runs?)

3. shaineql

Posts: 522; Member since: Apr 28, 2014

Lol , iPhone XS loosing at speed tests even in games to S10 now , so suddenly BENCHMARKS ARE IMPORTANT HAHAHAHAHAHHAHA

26. blingblingthing

Posts: 963; Member since: Oct 23, 2012

They said the number of cores don't count, real word performance is that does.

39. worldpeace

Posts: 3135; Member since: Apr 15, 2016

Right.. It even beat XS on video encoding test, something that really niche but sheeple love to brag.

5. Boast_Rider

Posts: 535; Member since: Sep 14, 2017

I love extra performance. That way, I can underclock and enjoy epic battery life and enjoy good enough performance. Power scales with V^2*f (v being voltage, f being frequency), so lowering the frequency by about 20% (which will lower voltage by about 15% as well) will result in you getting 80% performance in just 50% power consumption. The screen-on time on my OP6 went from 6.5-7 hours to almost 10 hours with a custom kernel and some manual underclocking (while keeping touchboost) and losing almost nothing in performance.

6. shaineql

Posts: 522; Member since: Apr 28, 2014

Good for you man.

7. LawnBoy

Posts: 189; Member since: Feb 23, 2019

Benchmark......bwahahaha

8. almostdone

Posts: 448; Member since: Sep 25, 2012

iPA will do anything to suck up to rotten fruit company. The good news is the majority of people don't listen to anything iPA has to say.

13. Leo_MC

Posts: 7432; Member since: Dec 02, 2011

Than why are you here?!

28. sissy246

Posts: 7112; Member since: Mar 04, 2015

Do i think they favor apple, they sure do. Why do I stay here, because PA also has a lot of other very good articles and deals they find that i don't see sometimes. PA is not just about phones, but other things also. So maybe that is why others stay also.

35. Leo_MC

Posts: 7432; Member since: Dec 02, 2011

Other than editorials - that are the author opinion on a subject (which you say favors Apple) - every other news is available elsewhere.

11. kracio

Posts: 12; Member since: Jul 09, 2014

Benchmarks are more misleading than useful in real life. Example - iPhones - the numbers shown are "impressive" but when it comes to real life usage .....we see videos on youtube where iPhones are outperformed even from SD 845 and now we have SD855...... ALL i care is which phone will open my apps faster and not which one has a higher score and PhoneArena wants to brag about it.....

14. Leo_MC

Posts: 7432; Member since: Dec 02, 2011

And why do you care?

20. Anonymous.

Posts: 423; Member since: Jun 15, 2016

Hi you still here on PA?

22. Leo_MC

Posts: 7432; Member since: Dec 02, 2011

Rarely, now that the socializing part of pa is gone.

46. RebelwithoutaClue unregistered

I know, I really hate the new look and comment section. Even thinking of deleting my account altogether.

23. c.m.s

Posts: 236; Member since: Dec 10, 2017

That is just BS but anyway benchmarks are not misleading if you know what they test. A CPU benchmark like geekbench does not show which phone opens up an app fastest. Its a CPU test, raw number crunshing. Opening an app (especially light ones) is not a CPU demanding task, it depends on storage speed, animations ans so on. So having the most powerful CPU (the A12) does not mean its nessessary faster to open apps.

12. Rager722 unregistered

This year marks the year iPhone starts dying. It started with the iPhone X and will continue to fall off as the years progress.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.