Is your device cheating on benchmarks? Here's how to find out

Is your device cheating on benchmarks? Here's how to find out
As time has gone by, so has people's reliance on benchmarks as a dependable and authentic tool for actual performance gauging. So much so, actually, that there are now those who would rather completely dismiss their usefulness (yes, they are useful, read on), and simply refuse to try and understand why it is that benchmarks are still exactly as relevant and exactly as dependable as they ever were. "How is that?", you may wonder, and that's a fair question, seeing as pretty much every major manufacturer has been caught red-handed trying to cheat their way to the top of the benchmark ladders. This has served to obfuscate the relative value of those tools. The truth is, benchmarks still have their role to play -- they're relatively rigid (i.e. standardized) in their approach, and a select few of them are actually pretty good at forecasting real-life results. In fact, if you spend your time benchmarking smartphones every other day, and then using them, it won't be at all that long until you start seeing the patterns that emerge. Are benchmarks foolproof? Heck no. They never were, and never will be. But they still play a role, and once you get used to their particular weaknesses and understand the infinite value of cross-benching, you start enjoying the fruit of your efforts and knowledge. Given enough practice, you'll start understating that these otherwise abstract scores should rarely be seen as absolutes, but rather -- as relative figures.

But how do you fight cheaters? Several apps that can monitor the behavior of your chipset are already available, and there are already available benchmarks that altogether avoid cheaters. We'll be showing you how to easily find out whether your particular device is cheating or not, and you needn't be a techie to make sense of this.

FEATURED VIDEO

39 Comments

1. av911

Posts: 206; Member since: Nov 19, 2012

Who cares? How much of a difference does it make for everyday use if it scores 1 point higher or lower?

12. Bioload25

Posts: 213; Member since: Nov 12, 2012

You missed the point of this article,bro Chris is telling us how to know if the benchmark result in fact translates into actual performance,because nearly all major OEMs use cheating tools in their benchmark tests (instant overclock for example) (Benchmarks without these methods are useful as you know from the article)

38. apple4never

Posts: 1064; Member since: May 08, 2013

i think its cheating if lets say samsung says their soc runs at 2.2ghz MAX but on a benchmark it runs 2.5 running at max clock just means its working hard. cheating is doing something the oem says it can't in order to do better

2. c.django

Posts: 13; Member since: Jan 09, 2014

According that LG G2 has the latest qualcom's chipset still makes the fastest smartphone right now on the market and beats everyone, so cheating or no cheating he is still the number one, until another phones gets the 805 cpu chipset by qualcom.

14. Bioload25

Posts: 213; Member since: Nov 12, 2012

I dont think so,as nearly all the top-end android smartphones these days are using the same chipset (SD 800) so saying that LG G2 is faster than a Note 3,when they're using exactly the same processors/gpus and nearly at the same clock speed...

23. c.django

Posts: 13; Member since: Jan 09, 2014

okey , my clue was that those smarthphones that have the latest chipsets, are ranket in the top of the list, it means they are based on the cpus clockspeed and gpu, so i don't see cheating here, cheating would be imense if for exemple a 1.7 ghz cpu is ranked better than a 2.3 cpu smartphone.

3. 007akshay

Posts: 37; Member since: Jun 20, 2012

dear PA now use this app n u only give the results of all "hot phones" including note 3 octa core version....

4. c.django

Posts: 13; Member since: Jan 09, 2014

i wana say that posting this article does not make sense at all, or it makes, if they are promoting another foolish app.

5. joey_sfb

Posts: 6794; Member since: Mar 29, 2012

Its false advertising but there nothing the consumer could do about it. Just like 1G = 1000M instead of mathematically correct 1024M.

8. sprockkets

Posts: 1612; Member since: Jan 16, 2012

actually we have a correct way of measuring that stuff. Instead of megabytes, use mebibytes, or gibibytes, if I spelled that correctly. miB, giB

6. poikilo

Posts: 49; Member since: Jun 07, 2012

Its not cheating if the cores run at the said frequency, while gaming you use the cores to its full potential, which this replicates. It would be cheating if the phone runs above the marketed frequency.

36. blinx

Posts: 23; Member since: Oct 09, 2013

I fink u r ryt....

7. Extradite

Posts: 316; Member since: Dec 30, 2013

A great app to find out what your phone has to offer, knowing A-Z and infos that techys and reafers like us who come to websites to show how much intrest they have on there devices. I like Quadrant and Antutu the most... I don't call this cheating at all. I call this smart Gpu processing. When the device picks it needs to exceed maximum power to apply for it to run stable and push those apps where it needs lots of power. Nexus 4 Gpu is locked to 400Mhz, it will not go any further, if Google wanted to they can, but if it did, the device would reach temp at 40+ degrees... I wouldnt mind that cause i know ile get more Fps when playing games than lagging. That's why so many games lag.. cause all Oems clock there Gpus to max 400+Mhz, but Samsung pushes the boundaries and takes the Gpu to near 590mhz.. my S4 S800 Gpu is raised to max 700Mhz, i get 60 Fps when playing games

16. bigstrudel

Posts: 574; Member since: Aug 20, 2012

Antutu and Quadrant are the worst benchmarks around. They aren't even cross platform and are the most often cheated. Serious tech folk avoid these benchmarks at all costs.

25. apple4never

Posts: 1064; Member since: May 08, 2013

dude my un over clocked DUAL CORE sony gets 59.9 fps, whats your point?

9. xfire99

Posts: 1205; Member since: Mar 14, 2012

Run at max frequency call cheating, what about runs at lowest frequency while phone is idling and with screen off? Cheating too, cause it runs at 300Mhz to cheat on battery life? Im sure all peoples will go havok, if CPU/GPU dont max out while running benchmarks and wonder why they got so low score and why CPU/GPU arent run att full speed. -.- In benchmarks, CPU/GPU are supposed to run at full speed too see what it capable of and not at 50/70/90% speed. Why care to run benchmarks, if the device dont run at full speed?

13. Extradite

Posts: 316; Member since: Dec 30, 2013

Well said+1

20. jroc74

Posts: 6023; Member since: Dec 30, 2010

Well said +2 For PC's....there is never this scrutiny. Some sites even do separate benchmarks for out the box and overclocked. Now...if these phones are doing an automatic overclock....thats a different story. But if they are running at the listed speeds...I dont see the problem.

37. blinx

Posts: 23; Member since: Oct 09, 2013

well said

40. Chris.P

Posts: 567; Member since: Jun 27, 2013

Actually, no, poorly said. At least incorrectly. This isn't the actual purpose of benchmarks, and perhaps it's on me for not adding an extra paragraph to clarify this. This was supposed to be a quick guide, not a full-blown essay. Anyway, the cheating is not so much related to, say, Samsung, forcing their chip to go full-out -- this is, after all, within its capabilities. It's cheating, because the OEMs artificially lower thermal throttling limits when a specific app is detected, allowing its chip to run at full speeds for periods that it ordinarily can't, for one reason or another.

41. poikilo

Posts: 49; Member since: Jun 07, 2012

You run a high intensity game for 10 minutes while running at full frequency, then the chip reaches the maximum throttling limit and the freq lowers. Agreed. Now you run a benchmark, the phone reaches maximum frequency and SAY the phone lowers the throttling temperature. Nevertheless you run the benchmark for a few minutes at most which doesn't allow the device to reach the temperature rendering the so called cheating useless. Now I'm not saying that OEM's don't do that, I'm saying that increasing the thermal throttling limit just doesn't do anything useful And in your reply its increasing the throttling limits not " lower thermal throttling limit."

10. tedkord

Posts: 17131; Member since: Jun 17, 2009

Further proof that this benchmark cheating talking point is just crap. I ran this app overlaid over a quadrant run. Both cores pegged at top speed the entire run. I have a cm based aosp ROM installed, no Samsung on my phone at all. A benchmark is supposed to stress your system. Your CPU is supposed to run at full speed when benching.

21. bigstrudel

Posts: 574; Member since: Aug 20, 2012

"Fast" Roms and kernels usually appeal to techies but it gives plenty of incentive to pad benchmark results just like the OEM's do. I've certainly run into plenty with questionable benchmark scores.

11. sprockkets

Posts: 1612; Member since: Jan 16, 2012

Uh, why not just go to anandtech's phone reviews and see the chart for yourself? The only ones that don't cheat 100% are nexus devices and Moto phones, *not surprised*.

15. Extradite

Posts: 316; Member since: Dec 30, 2013

Cause i mentioned above that Nexus device lookoff and blocks at 370Mhz. There are games you can't play on this device cause some games need more Fps and Gou power to run smoothly. Intense games that are over 1Gb to download will require full Gpu power, that's where the Nexus lags and sometimes shutsdown when playing.

17. bigstrudel

Posts: 574; Member since: Aug 20, 2012

Shuts down? That sounds like Rom or Kernel instability. Or your clock speed is too high. That's kinda the point at capping hardware clock speeds afterall. To prevent thermal damage and shutdowns.

18. bigstrudel

Posts: 574; Member since: Aug 20, 2012

People here don't believe Anandtech because their writers are so impressed with Apple SoC's.

27. sprockkets

Posts: 1612; Member since: Jan 16, 2012

That's ridiculous - maybe those people can't even understand Anand's analysis of the A7. To deny that it isn't a good chip is the hallmark of a fanboy.

26. apple4never

Posts: 1064; Member since: May 08, 2013

nexus devices throttle way to much, its not cheating, its under powering the phone

19. jroc74

Posts: 6023; Member since: Dec 30, 2010

I dont get the point of this either. Are they trying to say the only time the phones will ever ramp up that high is in benchmarks? If so...that could be cheating. But how can we be sure it doesnt ramp up that high in everyday use? Some games might max it out. Using remote desktop apps might max it out. Web browsing might max it out depending on what site you visit. I dont get it.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.