Why benchmarks are more important than you think
This article may contain personal views and opinion from the author.
If you’ve spent even a small amount of time reading about smartphones, you’ve probably seen various benchmark apps being mentioned and scores from different phones being compared. My goal today is not to explain how each benchmark works, but rather why it’s important to have such test results at our disposal.
Benchmark scores often start heated debates since they give a numeric value which objectively (ideally) shows how much better one phone performed than another. And with brands and models usually having their fair share of supporters, you often hear fans defending the “losing” phone by saying things such as “benchmarks don’t translate to real-world use” or something similar. And while this is somewhat true, despite benchmark apps like AnTuTu trying to make their tests as real-world-like as possible, it doesn’t make the results meaningless.
Performance tests have always been for enthusiasts and finding out where a machine’s limit is. No one goes to a carrier’s store to ask the customer service representative about AnTuTu scores before choosing between a Samsung or an LG smartphone.
But people can make a competition out of anything and smartphones are no exception. Once companies noticed people are paying attention to benchmarks, the race was on. Finally, they could show in plain numbers how much better their device was. AnTuTu and Geekbench scores became the mobile equivalent of a lap time around the Nurburgring.
Does a car’s time around the famous race track tell you much about how it will drive during your daily commute? Far from it. But it does tell you it’s freakin’ fast. With phones, it’s not much different.
Of course, as with any competition, some take it too far. A few smartphone makers have been caught cheating on benchmarks to get a higher score. They do this by optimizing the phone to be faster when the specific benchmarking app is running. This is like having extra horsepower only while on a racetrack (which some cars actually do, but it’s clearly stated). This does make some people wary about the trustworthiness of benchmarks as a whole, but so far the app developers have proven to be on top of things, removing scores that were achieved using shenanigans.
Continuing with the car analogy, the 0-60 mph seconds for cars isn’t really strongly connected to real-world use either since you rarely ever need to do that, but it’s still an important measurement because it shows the acceleration your car is capable of when you do need it, even if you’re not starting from a standstill. Similarly, when a benchmark app runs an overly complicated 3D scene on your smartphone, it’s meant to show how your phone will handle heavy 3D games.
Since benchmarks test devices under consistent loads, they are also helpful to check manufacturers’ claims for improvements between generations. If they say the new GPU is performing 25% faster then the results should be in that ballpark as well. If everyone is consistently getting results closer to 15% then you know something doesn’t add up.
With smartphones getting to the point that the high-end experience is almost indistinguishable between generations, without benchmarking apps it would be almost impossible to tell if the latest chips are actually as fast as they’re said to be. Now, you might say: “Why do manufacturers even make faster chips if people can’t tell the difference?” Because eventually, software developers start making apps that use the extra performance devices have at their disposal. In a way, software and hardware are pushing each other to achieve more.
Not all benchmarks give us abstract numbers that only mean something in the context of other phone’s results, however. Some, like those that measure battery life, for example, immediately give users a fair idea of what to expect from a device. Sure, brightness settings and all sorts of things will have an effect during day-to-day use, but when phones are tested under the same conditions, the best performers are usually the ones that will last you the longest as well.
If anything, we’d like to see newer, more sophisticated benchmarks being introduced. For example, ones that test more obscure features that manufacturers are marketing, as the various AI-powered functions that have sprung up like mushrooms after rain. That way people might get a better idea if these are actually improving things noticeably or are just another gimmick meant to sell more phones.
All in all, benchmarks are not only here to stay, but they’ll be playing an even bigger part in the future, as performance across the board improves and we start needing tests for the new technologies that will make their way into smartphones in the future. And who knows, maybe one day we’ll get a humanoid robot controlled by AI to benchmark overall user experience for us, complete with all the frustrations we might experience when handling a phone.
Real-world use is impossible to simulate consistently enough between phones without using preprogrammed software. So unless you want to add the variable that is a human, you’re stuck with a synthetic load. But that’s not a bad thing. Without tests that put chips to the absolute limit, how will we be able to tell which one performs better? If the test is somehow limited to a point where performance gains become unnoticeable for humans during use, then we’ll have 20+ phones with the equal top, what’s the point of that? It’s like capping top speed tests for cars at 100mph because no one should drive faster anyway.
The fight for bragging rights
Performance tests have always been for enthusiasts and finding out where a machine’s limit is. No one goes to a carrier’s store to ask the customer service representative about AnTuTu scores before choosing between a Samsung or an LG smartphone.
The only difference is, you probably can't afford any of these cars (image from newatlas.com)
Does a car’s time around the famous race track tell you much about how it will drive during your daily commute? Far from it. But it does tell you it’s freakin’ fast. With phones, it’s not much different.
These phones vroom-vroom the best, that's all you need to know!
Of course, as with any competition, some take it too far. A few smartphone makers have been caught cheating on benchmarks to get a higher score. They do this by optimizing the phone to be faster when the specific benchmarking app is running. This is like having extra horsepower only while on a racetrack (which some cars actually do, but it’s clearly stated). This does make some people wary about the trustworthiness of benchmarks as a whole, but so far the app developers have proven to be on top of things, removing scores that were achieved using shenanigans.
What benchmarks really show us?
Continuing with the car analogy, the 0-60 mph seconds for cars isn’t really strongly connected to real-world use either since you rarely ever need to do that, but it’s still an important measurement because it shows the acceleration your car is capable of when you do need it, even if you’re not starting from a standstill. Similarly, when a benchmark app runs an overly complicated 3D scene on your smartphone, it’s meant to show how your phone will handle heavy 3D games.
Benchmarks clearly show the difference between Samsung flagships using Snapdragon chips and those with Exynos inside (source AnTuTu.com)
With smartphones getting to the point that the high-end experience is almost indistinguishable between generations, without benchmarking apps it would be almost impossible to tell if the latest chips are actually as fast as they’re said to be. Now, you might say: “Why do manufacturers even make faster chips if people can’t tell the difference?” Because eventually, software developers start making apps that use the extra performance devices have at their disposal. In a way, software and hardware are pushing each other to achieve more.
If anything, we’d like to see newer, more sophisticated benchmarks being introduced. For example, ones that test more obscure features that manufacturers are marketing, as the various AI-powered functions that have sprung up like mushrooms after rain. That way people might get a better idea if these are actually improving things noticeably or are just another gimmick meant to sell more phones.
Okay, AI, tell me how many mAh this battery is!
All in all, benchmarks are not only here to stay, but they’ll be playing an even bigger part in the future, as performance across the board improves and we start needing tests for the new technologies that will make their way into smartphones in the future. And who knows, maybe one day we’ll get a humanoid robot controlled by AI to benchmark overall user experience for us, complete with all the frustrations we might experience when handling a phone.
Things that are NOT allowed: