Is your device cheating on benchmarks? Here's how to find out

Is your device cheating on benchmarks? Here's how to find out
As time has gone by, so has people's reliance on benchmarks as a dependable and authentic tool for actual performance gauging. So much so, actually, that there are now those who would rather completely dismiss their usefulness (yes, they are useful, read on), and simply refuse to try and understand why it is that benchmarks are still exactly as relevant and exactly as dependable as they ever were. "How is that?", you may wonder, and that's a fair question, seeing as pretty much every major manufacturer has been caught red-handed trying to cheat their way to the top of the benchmark ladders. This has served to obfuscate the relative value of those tools. The truth is, benchmarks still have their role to play -- they're relatively rigid (i.e. standardized) in their approach, and a select few of them are actually pretty good at forecasting real-life results. In fact, if you spend your time benchmarking smartphones every other day, and then using them, it won't be at all that long until you start seeing the patterns that emerge. Are benchmarks foolproof? Heck no. They never were, and never will be. But they still play a role, and once you get used to their particular weaknesses and understand the infinite value of cross-benching, you start enjoying the fruit of your efforts and knowledge. Given enough practice, you'll start understating that these otherwise abstract scores should rarely be seen as absolutes, but rather -- as relative figures.

But how do you fight cheaters? Several apps that can monitor the behavior of your chipset are already available, and there are already available benchmarks that altogether avoid cheaters. We'll be showing you how to easily find out whether your particular device is cheating or not, and you needn't be a techie to make sense of this.

FEATURED VIDEO

39 Comments

34. Xednyl04

Posts: 12; Member since: Apr 26, 2013

I don't get it! they call it cheating when the cpu max out? the CPU was intended to work to its maximal speed. in real life, if we are playing games, the CPUs are stressed to go up the roof and why is that called cheating when the test (e.g. benchmark test) was supposed to see how good the CPU is when being stressed.

35. mattkl

Posts: 255; Member since: Feb 01, 2010

When you artificially stress out the CPU cores you don't get accurate results how the benchmarks were designed to show. You want the test to stress the cores not have the phone stress it to max before the test starts is the issue.

39. Taters

Posts: 6474; Member since: Jan 28, 2013

Naw not cheating is even more stupid than cheating really. If you think of things that way, your smartphone doesn't even open an app the majority of the time and you are mostly interacting with the UI the majority of the time and the only 4 apps you probably use a lot is phone, message, gmail, and what's app. I highly doubt that benchmarks are meant to test those things. Benchmarks are design to test the GPU and CPU and why wouldn't you want to see the theoretical limits of your GPU and CPU? I know that is what I would look for in a benchmark. If I wanted to see the UI performance I would just play with the UI in a store.....

30. alrightihatepickingusernames

Posts: 474; Member since: Dec 29, 2013

If you look at benchmarks as a reflection of actual real-life performance none of the OEMs should be doing this. If you look at them as a way to show the devices full power, this "cheating" should be standard.

33. mattkl

Posts: 255; Member since: Feb 01, 2010

Exactly. The point is that for real world performance the phone needs to run how it normally would when faced with the tests that the benchmark app throws at it. While peaking the CPU cores out works fine for the devices full power as alright said, it is definitely cheating to pass the results off as real world. Our cell phones are dynamic in the way they use their CPU cores. They don't run 100% all the time. They monitor the phones current state and change for any given situation. That means the app must be able to factor in drops in performance, or changes that occur from the dynamic nature of the device. Running flat out doesn't allow for these dynamic changes which are needed for fairness and accuracy.

24. CreeDiddy

Posts: 2292; Member since: Nov 04, 2011

This is aimed at Android, but not iOS....I wonder why?

19. jroc74

Posts: 6023; Member since: Dec 30, 2010

I dont get the point of this either. Are they trying to say the only time the phones will ever ramp up that high is in benchmarks? If so...that could be cheating. But how can we be sure it doesnt ramp up that high in everyday use? Some games might max it out. Using remote desktop apps might max it out. Web browsing might max it out depending on what site you visit. I dont get it.

22. Ninetysix

Posts: 2967; Member since: Oct 08, 2012

Samsung uses a whitelist of benchmark apps/games that will only max out the frequency if it sees it in the list. If it's not in the list, it runs at a lower frequency.

28. tedkord

Posts: 17532; Member since: Jun 17, 2009

Like I said above, I ran this app in overlay while I ran quadrant. Both cores pegged the entire run. There is no Samsung whitelist on my phone because I'm running AOSP. A good benchmark is supposed to stress your system. If it doesn't peg your cores at max, it isn't a good benchmark.

29. Ninetysix

Posts: 2967; Member since: Oct 08, 2012

Not everyone is running AOSP. TW is Samsung's bread and butter and is used by the majority of their phones. It's cheating when you're only allowing certain apps/games to run at full speed. Change the app name and it runs at "normal" levels since it's now outside the whitelist.

11. sprockkets

Posts: 1612; Member since: Jan 16, 2012

Uh, why not just go to anandtech's phone reviews and see the chart for yourself? The only ones that don't cheat 100% are nexus devices and Moto phones, *not surprised*.

15. Extradite

Posts: 316; Member since: Dec 30, 2013

Cause i mentioned above that Nexus device lookoff and blocks at 370Mhz. There are games you can't play on this device cause some games need more Fps and Gou power to run smoothly. Intense games that are over 1Gb to download will require full Gpu power, that's where the Nexus lags and sometimes shutsdown when playing.

17. bigstrudel

Posts: 621; Member since: Aug 20, 2012

Shuts down? That sounds like Rom or Kernel instability. Or your clock speed is too high. That's kinda the point at capping hardware clock speeds afterall. To prevent thermal damage and shutdowns.

18. bigstrudel

Posts: 621; Member since: Aug 20, 2012

People here don't believe Anandtech because their writers are so impressed with Apple SoC's.

27. sprockkets

Posts: 1612; Member since: Jan 16, 2012

That's ridiculous - maybe those people can't even understand Anand's analysis of the A7. To deny that it isn't a good chip is the hallmark of a fanboy.

26. apple4never

Posts: 1064; Member since: May 08, 2013

nexus devices throttle way to much, its not cheating, its under powering the phone

10. tedkord

Posts: 17532; Member since: Jun 17, 2009

Further proof that this benchmark cheating talking point is just crap. I ran this app overlaid over a quadrant run. Both cores pegged at top speed the entire run. I have a cm based aosp ROM installed, no Samsung on my phone at all. A benchmark is supposed to stress your system. Your CPU is supposed to run at full speed when benching.

21. bigstrudel

Posts: 621; Member since: Aug 20, 2012

"Fast" Roms and kernels usually appeal to techies but it gives plenty of incentive to pad benchmark results just like the OEM's do. I've certainly run into plenty with questionable benchmark scores.

9. xfire99

Posts: 1207; Member since: Mar 14, 2012

Run at max frequency call cheating, what about runs at lowest frequency while phone is idling and with screen off? Cheating too, cause it runs at 300Mhz to cheat on battery life? Im sure all peoples will go havok, if CPU/GPU dont max out while running benchmarks and wonder why they got so low score and why CPU/GPU arent run att full speed. -.- In benchmarks, CPU/GPU are supposed to run at full speed too see what it capable of and not at 50/70/90% speed. Why care to run benchmarks, if the device dont run at full speed?

13. Extradite

Posts: 316; Member since: Dec 30, 2013

Well said+1

20. jroc74

Posts: 6023; Member since: Dec 30, 2010

Well said +2 For PC's....there is never this scrutiny. Some sites even do separate benchmarks for out the box and overclocked. Now...if these phones are doing an automatic overclock....thats a different story. But if they are running at the listed speeds...I dont see the problem.

37. blinx

Posts: 23; Member since: Oct 09, 2013

well said

40. Chris.P

Posts: 567; Member since: Jun 27, 2013

Actually, no, poorly said. At least incorrectly. This isn't the actual purpose of benchmarks, and perhaps it's on me for not adding an extra paragraph to clarify this. This was supposed to be a quick guide, not a full-blown essay. Anyway, the cheating is not so much related to, say, Samsung, forcing their chip to go full-out -- this is, after all, within its capabilities. It's cheating, because the OEMs artificially lower thermal throttling limits when a specific app is detected, allowing its chip to run at full speeds for periods that it ordinarily can't, for one reason or another.

41. poikilo

Posts: 49; Member since: Jun 07, 2012

You run a high intensity game for 10 minutes while running at full frequency, then the chip reaches the maximum throttling limit and the freq lowers. Agreed. Now you run a benchmark, the phone reaches maximum frequency and SAY the phone lowers the throttling temperature. Nevertheless you run the benchmark for a few minutes at most which doesn't allow the device to reach the temperature rendering the so called cheating useless. Now I'm not saying that OEM's don't do that, I'm saying that increasing the thermal throttling limit just doesn't do anything useful And in your reply its increasing the throttling limits not " lower thermal throttling limit."

7. Extradite

Posts: 316; Member since: Dec 30, 2013

A great app to find out what your phone has to offer, knowing A-Z and infos that techys and reafers like us who come to websites to show how much intrest they have on there devices. I like Quadrant and Antutu the most... I don't call this cheating at all. I call this smart Gpu processing. When the device picks it needs to exceed maximum power to apply for it to run stable and push those apps where it needs lots of power. Nexus 4 Gpu is locked to 400Mhz, it will not go any further, if Google wanted to they can, but if it did, the device would reach temp at 40+ degrees... I wouldnt mind that cause i know ile get more Fps when playing games than lagging. That's why so many games lag.. cause all Oems clock there Gpus to max 400+Mhz, but Samsung pushes the boundaries and takes the Gpu to near 590mhz.. my S4 S800 Gpu is raised to max 700Mhz, i get 60 Fps when playing games

16. bigstrudel

Posts: 621; Member since: Aug 20, 2012

Antutu and Quadrant are the worst benchmarks around. They aren't even cross platform and are the most often cheated. Serious tech folk avoid these benchmarks at all costs.

25. apple4never

Posts: 1064; Member since: May 08, 2013

dude my un over clocked DUAL CORE sony gets 59.9 fps, whats your point?

6. poikilo

Posts: 49; Member since: Jun 07, 2012

Its not cheating if the cores run at the said frequency, while gaming you use the cores to its full potential, which this replicates. It would be cheating if the phone runs above the marketed frequency.

36. blinx

Posts: 23; Member since: Oct 09, 2013

I fink u r ryt....

5. joey_sfb

Posts: 6794; Member since: Mar 29, 2012

Its false advertising but there nothing the consumer could do about it. Just like 1G = 1000M instead of mathematically correct 1024M.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.
FCC OKs Cingular's purchase of AT&T Wireless