x PhoneArena is looking for new authors! To view all available positions, click here.
  • Home
  • News
  • Is your device cheating on benchmarks? Here's how to find out

Is your device cheating on benchmarks? Here's how to find out

Posted: , by Chris P.

Tags:

Is your device cheating on benchmarks? Here's how to find out
As time has gone by, so has people's reliance on benchmarks as a dependable and authentic tool for actual performance gauging. So much so, actually, that there are now those who would rather completely dismiss their usefulness (yes, they are useful, read on), and simply refuse to try and understand why it is that benchmarks are still exactly as relevant and exactly as dependable as they ever were. "How is that?", you may wonder, and that's a fair question, seeing as pretty much every major manufacturer has been caught red-handed trying to cheat their way to the top of the benchmark ladders. This has served to obfuscate the relative value of those tools. The truth is, benchmarks still have their role to play -- they're relatively rigid (i.e. standardized) in their approach, and a select few of them are actually pretty good at forecasting real-life results. In fact, if you spend your time benchmarking smartphones every other day, and then using them, it won't be at all that long until you start seeing the patterns that emerge. Are benchmarks foolproof? Heck no. They never were, and never will be. But they still play a role, and once you get used to their particular weaknesses and understand the infinite value of cross-benching, you start enjoying the fruit of your efforts and knowledge. Given enough practice, you'll start understating that these otherwise abstract scores should rarely be seen as absolutes, but rather -- as relative figures.

But how do you fight cheaters? Several apps that can monitor the behavior of your chipset are already available, and there are already available benchmarks that altogether avoid cheaters. We'll be showing you how to easily find out whether your particular device is cheating or not, and you needn't be a techie to make sense of this.

41 Comments
  • Options
    Close




posted on 10 Jan 2014, 10:08

1. av911 (Posts: 149; Member since: 19 Nov 2012)


Who cares? How much of a difference does it make for everyday use if it scores 1 point higher or lower?

posted on 10 Jan 2014, 11:37

12. Bioload25 (Posts: 213; Member since: 12 Nov 2012)


You missed the point of this article,bro
Chris is telling us how to know if the benchmark result in fact translates into actual performance,because nearly all major OEMs use cheating tools in their benchmark tests (instant overclock for example)
(Benchmarks without these methods are useful as you know from the article)

posted on 11 Jan 2014, 20:03 1

38. apple4never (Posts: 969; Member since: 08 May 2013)


i think its cheating if lets say samsung says their soc runs at 2.2ghz MAX but on a benchmark it runs 2.5 running at max clock just means its working hard. cheating is doing something the oem says it can't in order to do better

posted on 10 Jan 2014, 10:12

2. c.django (Posts: 13; Member since: 09 Jan 2014)


According that LG G2 has the latest qualcom's chipset still makes the fastest smartphone right now on the market and beats everyone, so cheating or no cheating he is still the number one, until another phones gets the 805 cpu chipset by qualcom.

posted on 10 Jan 2014, 11:39 1

14. Bioload25 (Posts: 213; Member since: 12 Nov 2012)


I dont think so,as nearly all the top-end android smartphones these days are using the same chipset (SD 800) so saying that LG G2 is faster than a Note 3,when they're using exactly the same processors/gpus and nearly at the same clock speed...

posted on 10 Jan 2014, 12:25

23. c.django (Posts: 13; Member since: 09 Jan 2014)


okey , my clue was that those smarthphones that have the latest chipsets, are ranket in the top of the list, it means they are based on the cpus clockspeed and gpu, so i don't see cheating here, cheating would be imense if for exemple a 1.7 ghz cpu is ranked better than a 2.3 cpu smartphone.

posted on 10 Jan 2014, 10:15 3

3. 007akshay (Posts: 37; Member since: 20 Jun 2012)


dear PA now use this app n u only give the results of all "hot phones" including note 3 octa core version....

posted on 10 Jan 2014, 10:15

4. c.django (Posts: 13; Member since: 09 Jan 2014)


i wana say that posting this article does not make sense at all, or it makes, if they are promoting another foolish app.

posted on 10 Jan 2014, 10:15

5. joey_sfb (Posts: 3072; Member since: 29 Mar 2012)


Its false advertising but there nothing the consumer could do about it. Just like 1G = 1000M instead of mathematically correct 1024M.

posted on 10 Jan 2014, 11:20

8. sprockkets (Posts: 1296; Member since: 16 Jan 2012)


actually we have a correct way of measuring that stuff. Instead of megabytes, use mebibytes, or gibibytes, if I spelled that correctly.

miB, giB

posted on 10 Jan 2014, 10:39 2

6. poikilo (Posts: 47; Member since: 07 Jun 2012)


Its not cheating if the cores run at the said frequency, while gaming you use the cores to its full potential, which this replicates.
It would be cheating if the phone runs above the marketed frequency.

posted on 11 Jan 2014, 16:49

36. blinx (Posts: 21; Member since: 09 Oct 2013)


I fink u r ryt....

posted on 10 Jan 2014, 10:45 1

7. Extradite (banned) (Posts: 316; Member since: 30 Dec 2013)


A great app to find out what your phone has to offer, knowing A-Z and infos that techys and reafers like us who come to websites to show how much intrest they have on there devices.

I like Quadrant and Antutu the most...

I don't call this cheating at all. I call this smart Gpu processing. When the device picks it needs to exceed maximum power to apply for it to run stable and push those apps where it needs lots of power.

Nexus 4 Gpu is locked to 400Mhz, it will not go any further, if Google wanted to they can, but if it did, the device would reach temp at 40+ degrees... I wouldnt mind that cause i know ile get more Fps when playing games than lagging. That's why so many games lag.. cause all Oems clock there Gpus to max 400+Mhz, but Samsung pushes the boundaries and takes the Gpu to near 590mhz.. my S4 S800 Gpu is raised to max 700Mhz, i get 60 Fps when playing games

posted on 10 Jan 2014, 12:03

16. bigstrudel (Posts: 518; Member since: 20 Aug 2012)


Antutu and Quadrant are the worst benchmarks around. They aren't even cross platform and are the most often cheated. Serious tech folk avoid these benchmarks at all costs.

posted on 10 Jan 2014, 14:18 1

25. apple4never (Posts: 969; Member since: 08 May 2013)


dude my un over clocked DUAL CORE sony gets 59.9 fps, whats your point?

posted on 10 Jan 2014, 11:21 8

9. xfire99 (Posts: 560; Member since: 14 Mar 2012)


Run at max frequency call cheating, what about runs at lowest frequency while phone is idling and with screen off? Cheating too, cause it runs at 300Mhz to cheat on battery life?
Im sure all peoples will go havok, if CPU/GPU dont max out while running benchmarks and wonder why they got so low score and why CPU/GPU arent run att full speed. -.-

In benchmarks, CPU/GPU are supposed to run at full speed too see what it capable of and not at 50/70/90% speed. Why care to run benchmarks, if the device dont run at full speed?

posted on 10 Jan 2014, 11:38 2

13. Extradite (banned) (Posts: 316; Member since: 30 Dec 2013)


Well said+1

posted on 10 Jan 2014, 12:17

20. jroc74 (Posts: 5171; Member since: 30 Dec 2010)


Well said +2

For PC's....there is never this scrutiny. Some sites even do separate benchmarks for out the box and overclocked.

Now...if these phones are doing an automatic overclock....thats a different story. But if they are running at the listed speeds...I dont see the problem.

posted on 11 Jan 2014, 16:51

37. blinx (Posts: 21; Member since: 09 Oct 2013)


well said

posted on 13 Jan 2014, 01:20

40. Chris.P (Posts: 299; Member since: 27 Jun 2013)


Actually, no, poorly said. At least incorrectly. This isn't the actual purpose of benchmarks, and perhaps it's on me for not adding an extra paragraph to clarify this. This was supposed to be a quick guide, not a full-blown essay.

Anyway, the cheating is not so much related to, say, Samsung, forcing their chip to go full-out -- this is, after all, within its capabilities. It's cheating, because the OEMs artificially lower thermal throttling limits when a specific app is detected, allowing its chip to run at full speeds for periods that it ordinarily can't, for one reason or another.

posted on 13 Jan 2014, 03:20

41. poikilo (Posts: 47; Member since: 07 Jun 2012)


You run a high intensity game for 10 minutes while running at full frequency, then the chip reaches the maximum throttling limit and the freq lowers. Agreed.

Now you run a benchmark, the phone reaches maximum frequency and SAY the phone lowers the throttling temperature. Nevertheless you run the benchmark for a few minutes at most which doesn't allow the device to reach the temperature rendering the so called cheating useless.

Now I'm not saying that OEM's don't do that, I'm saying that increasing the thermal throttling limit just doesn't do anything useful
And in your reply its increasing the throttling limits not " lower thermal throttling limit."

posted on 10 Jan 2014, 11:26 2

10. tedkord (Posts: 5079; Member since: 17 Jun 2009)


Further proof that this benchmark cheating talking point is just crap. I ran this app overlaid over a quadrant run. Both cores pegged at top speed the entire run. I have a cm based aosp ROM installed, no Samsung on my phone at all.

A benchmark is supposed to stress your system. Your CPU is supposed to run at full speed when benching.

posted on 10 Jan 2014, 12:19

21. bigstrudel (Posts: 518; Member since: 20 Aug 2012)


"Fast" Roms and kernels usually appeal to techies but it gives plenty of incentive to pad benchmark results just like the OEM's do. I've certainly run into plenty with questionable benchmark scores.

posted on 10 Jan 2014, 11:26 1

11. sprockkets (Posts: 1296; Member since: 16 Jan 2012)


Uh, why not just go to anandtech's phone reviews and see the chart for yourself? The only ones that don't cheat 100% are nexus devices and Moto phones, *not surprised*.

posted on 10 Jan 2014, 11:42 1

15. Extradite (banned) (Posts: 316; Member since: 30 Dec 2013)


Cause i mentioned above that Nexus device lookoff and blocks at 370Mhz. There are games you can't play on this device cause some games need more Fps and Gou power to run smoothly. Intense games that are over 1Gb to download will require full Gpu power, that's where the Nexus lags and sometimes shutsdown when playing.

posted on 10 Jan 2014, 12:08

17. bigstrudel (Posts: 518; Member since: 20 Aug 2012)


Shuts down? That sounds like Rom or Kernel instability. Or your clock speed is too high. That's kinda the point at capping hardware clock speeds afterall. To prevent thermal damage and shutdowns.

posted on 10 Jan 2014, 12:12

18. bigstrudel (Posts: 518; Member since: 20 Aug 2012)


People here don't believe Anandtech because their writers are so impressed with Apple SoC's.

posted on 10 Jan 2014, 15:29

27. sprockkets (Posts: 1296; Member since: 16 Jan 2012)


That's ridiculous - maybe those people can't even understand Anand's analysis of the A7.

To deny that it isn't a good chip is the hallmark of a fanboy.

posted on 10 Jan 2014, 19:07

32. Heatfan316 (Posts: 445; Member since: 21 Aug 2011)


It's a terrible chip and operating system, you have no access to Apple's file system so you can't overclock the processor, which I find boring and useless, many other tech fans feel the same way FYI.

posted on 10 Jan 2014, 14:20

26. apple4never (Posts: 969; Member since: 08 May 2013)


nexus devices throttle way to much, its not cheating, its under powering the phone

posted on 10 Jan 2014, 12:14

19. jroc74 (Posts: 5171; Member since: 30 Dec 2010)


I dont get the point of this either.

Are they trying to say the only time the phones will ever ramp up that high is in benchmarks? If so...that could be cheating.

But how can we be sure it doesnt ramp up that high in everyday use? Some games might max it out. Using remote desktop apps might max it out. Web browsing might max it out depending on what site you visit.

I dont get it.

posted on 10 Jan 2014, 12:22

22. Ninetysix (Posts: 1669; Member since: 08 Oct 2012)


Samsung uses a whitelist of benchmark apps/games that will only max out the frequency if it sees it in the list. If it's not in the list, it runs at a lower frequency.

posted on 10 Jan 2014, 17:00

28. tedkord (Posts: 5079; Member since: 17 Jun 2009)


Like I said above, I ran this app in overlay while I ran quadrant. Both cores pegged the entire run. There is no Samsung whitelist on my phone because I'm running AOSP. A good benchmark is supposed to stress your system. If it doesn't peg your cores at max, it isn't a good benchmark.

posted on 10 Jan 2014, 17:06

29. Ninetysix (Posts: 1669; Member since: 08 Oct 2012)


Not everyone is running AOSP. TW is Samsung's bread and butter and is used by the majority of their phones. It's cheating when you're only allowing certain apps/games to run at full speed. Change the app name and it runs at "normal" levels since it's now outside the whitelist.

posted on 10 Jan 2014, 12:26

24. CreeDiddy (Posts: 324; Member since: 04 Nov 2011)


This is aimed at Android, but not iOS....I wonder why?

posted on 10 Jan 2014, 18:58

31. Heatfan316 (Posts: 445; Member since: 21 Aug 2011)


What's the point in even benchmarking an iPhone anyway? You can't even improve the score because you can't add a custom rom or overclock it. Your overpriced feature phone is worthless and I have no respect for it!

posted on 10 Jan 2014, 18:37 1

30. alrightihatepickingusernames (Posts: 425; Member since: 29 Dec 2013)


If you look at benchmarks as a reflection of actual real-life performance none of the OEMs should be doing this. If you look at them as a way to show the devices full power, this "cheating" should be standard.

posted on 10 Jan 2014, 19:40

33. mattkl (Posts: 176; Member since: 01 Feb 2010)


Exactly. The point is that for real world performance the phone needs to run how it normally would when faced with the tests that the benchmark app throws at it. While peaking the CPU cores out works fine for the devices full power as alright said, it is definitely cheating to pass the results off as real world.

Our cell phones are dynamic in the way they use their CPU cores. They don't run 100% all the time. They monitor the phones current state and change for any given situation. That means the app must be able to factor in drops in performance, or changes that occur from the dynamic nature of the device.

Running flat out doesn't allow for these dynamic changes which are needed for fairness and accuracy.

posted on 10 Jan 2014, 20:59

34. Xednyl04 (Posts: 5; Member since: 26 Apr 2013)


I don't get it! they call it cheating when the cpu max out? the CPU was intended to work to its maximal speed. in real life, if we are playing games, the CPUs are stressed to go up the roof and why is that called cheating when the test (e.g. benchmark test) was supposed to see how good the CPU is when being stressed.

posted on 10 Jan 2014, 21:02

35. mattkl (Posts: 176; Member since: 01 Feb 2010)


When you artificially stress out the CPU cores you don't get accurate results how the benchmarks were designed to show. You want the test to stress the cores not have the phone stress it to max before the test starts is the issue.

posted on 12 Jan 2014, 09:58 1

39. Taters (Posts: 3461; Member since: 28 Jan 2013)


Naw not cheating is even more stupid than cheating really. If you think of things that way, your smartphone doesn't even open an app the majority of the time and you are mostly interacting with the UI the majority of the time and the only 4 apps you probably use a lot is phone, message, gmail, and what's app. I highly doubt that benchmarks are meant to test those things.

Benchmarks are design to test the GPU and CPU and why wouldn't you want to see the theoretical limits of your GPU and CPU? I know that is what I would look for in a benchmark. If I wanted to see the UI performance I would just play with the UI in a store.....

Want to comment? Please login or register.

Latest stories