300 tests later, these phone makers were found to indeed cheat on benchmarks

300 tests later, these phone makers were found to indeed cheat on benchmarks
Recent #Benchmarkgate reports caught manufacturers revving up their phones' performance when they detected popular system testing apps being run. Huawei and its offshoot Honor come to mind in fresh memory, but plenty of other manufacturers have been shown to do the same, including Samsung at one point. 

Apple's #batterygate drama, on the other hand, showed it did the opposite - its chipsets are so overboard with power consumption on max performance, that they get throttled artificially because the peak demand can overwhelm an aging battery and shut down the iPhone.

When confronted, manufacturers argue that one can't really call this cheating, since everyone wants their product to be at its best when tested, and there is no actual overclocking from the official spec sheet maximums. Huawei even mentioned that its AI engine automatically puts the phone in the high-performance mode when it detects tough loads. A new report, however, tried to delve deeper into the phenomenon, running about both public and private versions of popular benchmarks... about 300 times on various phones.

Public means a version of AnTuTu or Geekbench taken off the Play Store, while private is the same apps provided by their creators, but under different names. Lo and behold, revving up the processor for benchmark purposes is indeed a thing, and some of the phones got so hot while running the public app, they could hardly be held in the palm, as you can see in the charts below. 

The problem seems to be that CPU throttling, which idles the cores to their minimum frequency when there's no workload, immediately goes to the backburner when popular benchmarking app names were recognized. Cheating, massaging the truth, whatever one calls it, the end result is that benchmarking companies and users alike aren't pleased, ruining some reputations in the long run for no good reason, as synthetic benchies aren't really indicative of a real-world performance.



source: FirstPost

FEATURED VIDEO

14 Comments

1. AmashAziz

Posts: 2923; Member since: Jun 30, 2014

Thank you Xiaomi for not cheating.

2. Atechguy0

Posts: 918; Member since: Aug 03, 2018

Omg Daniel look at the test you guys (PhoneArena), and FirstPost are using. They are using 3DMark Slingshot Extreme ES 3.1. If you guys actually know what that test is, then you wouldn't be using that test anymore on Android, period It is using OpenGL ES 3.1, hint, hint, the part ES 3.1. It is 2018 last time I looked. The thing is Vulkan Graphics has been on Android since 2016. All new flagships and mid range devices last year supported Vulkan. This year all new Android smartphones support Vulkan Graphics. So benchmark tests like 3DMark that use OpenGL shouldn't be used any more, period. There is such a big difference in speed between OpenGL and Vulkan. Come on PhoneArena you can do better than this crap reporting. Even the benchmark tests on iOS these days use Apples Metal, which is closer to Vulkan Graphics, verses the older and slower OpenGL.

5. bucky

Posts: 3784; Member since: Sep 30, 2009

from what i understand you have to have Android 7.0 and SD 820+. If i recall, Android 6.0 is still the most popular by a good margin. Thats probably why the test was done the way it is.

11. Cicero

Posts: 1129; Member since: Jan 22, 2014

They test new phones. This new phones are not coming with android 6.

12. Atechguy0

Posts: 918; Member since: Aug 03, 2018

The Android 7 and above is true. But even the SD 425, 427, 430, 450, and many others, including many SD 6XX models all support Vulkan graphics. So its definitely not just SD 820 and above.

13. Atechguy0

Posts: 918; Member since: Aug 03, 2018

BTW Android 6 Marshmallow has 23.7% Android 7.x has 30.8%. Android 8.x has 14.6% So out of the billions of Android smartphones on the current market, there is a potential of 45.4% than are using Android 7 or above right at this moment.

14. Atechguy0

Posts: 918; Member since: Aug 03, 2018

Oh, well...

3. baldilocks

Posts: 1505; Member since: Dec 14, 2008

OnePlus 6 is good. :)

8. yalokiy

Posts: 997; Member since: Aug 01, 2016

Or they have more advance cheating :D That would add one more thing they beat huawei at!

4. strategic_developer

Posts: 1627; Member since: Jul 17, 2018

This is an will always be stupid. Bevan arms are designed to push. A product to its theoretical limits. It doesn't matter if your usage will ever for eca. Pls push a GPU or SoC, to its maximum capability. The idea of a benchmark is to see what a computer or product can do in extreme conditions. Whether you can duplicate them or not. The fact is all benchmarks are BS. Because for one no two phones that are the same perform the same even out of the box with zero changes. But we all install different apps, which changes how background and foreground precede happen. Which means they change how the device will perform when benched. As long as they didn't overcook the chips to perform pass their certified speeds, then it's not cheating. What you all sound like is this: well in the US since the maximum speed on the highway is 70mph in most places, if your care can do 200mph and you push it to the max, you are cheating to make your car seem better. How if it cheating if I am. Just going as fast as I actually can go? . In the beginning this was just hate on Android because we have the ability to actually push our device to max. And because you can't on iOS, the whiners cried fowl. It's simply not cheating because no bench. Ark can duicate real world usage because we all. Use phones differently.

6. JCASS889

Posts: 539; Member since: May 18, 2018

no, alot of companies set cpu and gpu profiles that would not work in the long run when running benchmarks. same idea as what you just said, u think that 4 cylinder with 900 horsepower will outlast a v6 with 400? may be able to reach the top score but its not a stable method of achieving those scores.

7. walauweh

Posts: 9; Member since: Aug 08, 2018

isnt neural network@ ai need to be trained, through coding or names? like when it detect an app that has the word bench, it will turn on performance mode. for games i think they can target the framework like unity or unreal for example. i believe by checking on how the cpu and gpu frequency works across the allerged cheating on benchmark is more useful. and the temperature is by checking via software or is it using temperature gun @ some sort of temperature tools? at 76 degree, u cant hold the phone at all, but if its using software monitoring, and u still can hold the phone, so its not really accurate. if using temperature tool, then its crazy s**t. u might drop the phone when testing it cuz no one can hold a device around 76 degree at a long time.

9. worldpeace

Posts: 3133; Member since: Apr 15, 2016

AI training itself by doing the same thing again and again, with slight difference in each iteration, and find the most effective route (this one is more like machine learning than AI :-/) CPU+GPU is always at 99%-100% during benchmark (unless when throttled due to overheat) That temperature is obviously from the sensor inside the SoC, monitoring it from outside will be useless because smartphones use different kind of internal layout, internal shielding, and backplate material.

10. JCASS889

Posts: 539; Member since: May 18, 2018

its all machine learning, we are a long way off from this so called A.I.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.