Is your device cheating on benchmarks? Here's how to find out
Is your device cheating on benchmarks? Here's how to find out
1. Step #1
2. Step #2
3. Step #3
4. Misc: quad- and eight-core smartphones
5. Misc: quad- and eight-core smartphones
6. The easy, but expensive way
1. av911 (Posts: 143; Member since: 19 Nov 2012)
Who cares? How much of a difference does it make for everyday use if it scores 1 point higher or lower?
12. Bioload25 (Posts: 211; Member since: 12 Nov 2012)
You missed the point of this article,bro
Chris is telling us how to know if the benchmark result in fact translates into actual performance,because nearly all major OEMs use cheating tools in their benchmark tests (instant overclock for example)
(Benchmarks without these methods are useful as you know from the article)
38. apple4never (Posts: 966; Member since: 08 May 2013)
i think its cheating if lets say samsung says their soc runs at 2.2ghz MAX but on a benchmark it runs 2.5 running at max clock just means its working hard. cheating is doing something the oem says it can't in order to do better
2. c.django (Posts: 13; Member since: 09 Jan 2014)
According that LG G2 has the latest qualcom's chipset still makes the fastest smartphone right now on the market and beats everyone, so cheating or no cheating he is still the number one, until another phones gets the 805 cpu chipset by qualcom.
14. Bioload25 (Posts: 211; Member since: 12 Nov 2012)
I dont think so,as nearly all the top-end android smartphones these days are using the same chipset (SD 800) so saying that LG G2 is faster than a Note 3,when they're using exactly the same processors/gpus and nearly at the same clock speed...
23. c.django (Posts: 13; Member since: 09 Jan 2014)
okey , my clue was that those smarthphones that have the latest chipsets, are ranket in the top of the list, it means they are based on the cpus clockspeed and gpu, so i don't see cheating here, cheating would be imense if for exemple a 1.7 ghz cpu is ranked better than a 2.3 cpu smartphone.
3. 007akshay (Posts: 37; Member since: 20 Jun 2012)
dear PA now use this app n u only give the results of all "hot phones" including note 3 octa core version....
4. c.django (Posts: 13; Member since: 09 Jan 2014)
i wana say that posting this article does not make sense at all, or it makes, if they are promoting another foolish app.
5. joey_sfb (Posts: 3006; Member since: 29 Mar 2012)
Its false advertising but there nothing the consumer could do about it. Just like 1G = 1000M instead of mathematically correct 1024M.
8. sprockkets (Posts: 1252; Member since: 16 Jan 2012)
actually we have a correct way of measuring that stuff. Instead of megabytes, use mebibytes, or gibibytes, if I spelled that correctly.
6. poikilo (Posts: 47; Member since: 07 Jun 2012)
Its not cheating if the cores run at the said frequency, while gaming you use the cores to its full potential, which this replicates.
It would be cheating if the phone runs above the marketed frequency.
7. Extradite (banned) (Posts: 316; Member since: 30 Dec 2013)
A great app to find out what your phone has to offer, knowing A-Z and infos that techys and reafers like us who come to websites to show how much intrest they have on there devices.
I like Quadrant and Antutu the most...
I don't call this cheating at all. I call this smart Gpu processing. When the device picks it needs to exceed maximum power to apply for it to run stable and push those apps where it needs lots of power.
Nexus 4 Gpu is locked to 400Mhz, it will not go any further, if Google wanted to they can, but if it did, the device would reach temp at 40+ degrees... I wouldnt mind that cause i know ile get more Fps when playing games than lagging. That's why so many games lag.. cause all Oems clock there Gpus to max 400+Mhz, but Samsung pushes the boundaries and takes the Gpu to near 590mhz.. my S4 S800 Gpu is raised to max 700Mhz, i get 60 Fps when playing games
16. bigstrudel (Posts: 518; Member since: 20 Aug 2012)
Antutu and Quadrant are the worst benchmarks around. They aren't even cross platform and are the most often cheated. Serious tech folk avoid these benchmarks at all costs.
25. apple4never (Posts: 966; Member since: 08 May 2013)
dude my un over clocked DUAL CORE sony gets 59.9 fps, whats your point?
9. xfire99 (Posts: 544; Member since: 14 Mar 2012)
Run at max frequency call cheating, what about runs at lowest frequency while phone is idling and with screen off? Cheating too, cause it runs at 300Mhz to cheat on battery life?
Im sure all peoples will go havok, if CPU/GPU dont max out while running benchmarks and wonder why they got so low score and why CPU/GPU arent run att full speed. -.-
In benchmarks, CPU/GPU are supposed to run at full speed too see what it capable of and not at 50/70/90% speed. Why care to run benchmarks, if the device dont run at full speed?
20. jroc74 (Posts: 5022; Member since: 30 Dec 2010)
Well said +2
For PC's....there is never this scrutiny. Some sites even do separate benchmarks for out the box and overclocked.
Now...if these phones are doing an automatic overclock....thats a different story. But if they are running at the listed speeds...I dont see the problem.
40. Chris.P (Posts: 292; Member since: 27 Jun 2013)
Actually, no, poorly said. At least incorrectly. This isn't the actual purpose of benchmarks, and perhaps it's on me for not adding an extra paragraph to clarify this. This was supposed to be a quick guide, not a full-blown essay.
Anyway, the cheating is not so much related to, say, Samsung, forcing their chip to go full-out -- this is, after all, within its capabilities. It's cheating, because the OEMs artificially lower thermal throttling limits when a specific app is detected, allowing its chip to run at full speeds for periods that it ordinarily can't, for one reason or another.
41. poikilo (Posts: 47; Member since: 07 Jun 2012)
You run a high intensity game for 10 minutes while running at full frequency, then the chip reaches the maximum throttling limit and the freq lowers. Agreed.
Now you run a benchmark, the phone reaches maximum frequency and SAY the phone lowers the throttling temperature. Nevertheless you run the benchmark for a few minutes at most which doesn't allow the device to reach the temperature rendering the so called cheating useless.
Now I'm not saying that OEM's don't do that, I'm saying that increasing the thermal throttling limit just doesn't do anything useful
And in your reply its increasing the throttling limits not " lower thermal throttling limit."
10. tedkord (Posts: 4984; Member since: 17 Jun 2009)
Further proof that this benchmark cheating talking point is just crap. I ran this app overlaid over a quadrant run. Both cores pegged at top speed the entire run. I have a cm based aosp ROM installed, no Samsung on my phone at all.
A benchmark is supposed to stress your system. Your CPU is supposed to run at full speed when benching.
21. bigstrudel (Posts: 518; Member since: 20 Aug 2012)
"Fast" Roms and kernels usually appeal to techies but it gives plenty of incentive to pad benchmark results just like the OEM's do. I've certainly run into plenty with questionable benchmark scores.
11. sprockkets (Posts: 1252; Member since: 16 Jan 2012)
Uh, why not just go to anandtech's phone reviews and see the chart for yourself? The only ones that don't cheat 100% are nexus devices and Moto phones, *not surprised*.
15. Extradite (banned) (Posts: 316; Member since: 30 Dec 2013)
Cause i mentioned above that Nexus device lookoff and blocks at 370Mhz. There are games you can't play on this device cause some games need more Fps and Gou power to run smoothly. Intense games that are over 1Gb to download will require full Gpu power, that's where the Nexus lags and sometimes shutsdown when playing.
17. bigstrudel (Posts: 518; Member since: 20 Aug 2012)
Shuts down? That sounds like Rom or Kernel instability. Or your clock speed is too high. That's kinda the point at capping hardware clock speeds afterall. To prevent thermal damage and shutdowns.
18. bigstrudel (Posts: 518; Member since: 20 Aug 2012)
People here don't believe Anandtech because their writers are so impressed with Apple SoC's.
27. sprockkets (Posts: 1252; Member since: 16 Jan 2012)
That's ridiculous - maybe those people can't even understand Anand's analysis of the A7.
To deny that it isn't a good chip is the hallmark of a fanboy.
32. Heatfan316 (Posts: 435; Member since: 21 Aug 2011)
It's a terrible chip and operating system, you have no access to Apple's file system so you can't overclock the processor, which I find boring and useless, many other tech fans feel the same way FYI.
26. apple4never (Posts: 966; Member since: 08 May 2013)
nexus devices throttle way to much, its not cheating, its under powering the phone
19. jroc74 (Posts: 5022; Member since: 30 Dec 2010)
I dont get the point of this either.
Are they trying to say the only time the phones will ever ramp up that high is in benchmarks? If so...that could be cheating.
But how can we be sure it doesnt ramp up that high in everyday use? Some games might max it out. Using remote desktop apps might max it out. Web browsing might max it out depending on what site you visit.
I dont get it.
22. Ninetysix (Posts: 1661; Member since: 08 Oct 2012)
Samsung uses a whitelist of benchmark apps/games that will only max out the frequency if it sees it in the list. If it's not in the list, it runs at a lower frequency.
28. tedkord (Posts: 4984; Member since: 17 Jun 2009)
Like I said above, I ran this app in overlay while I ran quadrant. Both cores pegged the entire run. There is no Samsung whitelist on my phone because I'm running AOSP. A good benchmark is supposed to stress your system. If it doesn't peg your cores at max, it isn't a good benchmark.
29. Ninetysix (Posts: 1661; Member since: 08 Oct 2012)
Not everyone is running AOSP. TW is Samsung's bread and butter and is used by the majority of their phones. It's cheating when you're only allowing certain apps/games to run at full speed. Change the app name and it runs at "normal" levels since it's now outside the whitelist.
24. CreeDiddy (Posts: 311; Member since: 04 Nov 2011)
This is aimed at Android, but not iOS....I wonder why?
31. Heatfan316 (Posts: 435; Member since: 21 Aug 2011)
What's the point in even benchmarking an iPhone anyway? You can't even improve the score because you can't add a custom rom or overclock it. Your overpriced feature phone is worthless and I have no respect for it!
30. alrightihatepickingusernames (Posts: 410; Member since: 29 Dec 2013)
If you look at benchmarks as a reflection of actual real-life performance none of the OEMs should be doing this. If you look at them as a way to show the devices full power, this "cheating" should be standard.
33. mattkl (Posts: 174; Member since: 01 Feb 2010)
Exactly. The point is that for real world performance the phone needs to run how it normally would when faced with the tests that the benchmark app throws at it. While peaking the CPU cores out works fine for the devices full power as alright said, it is definitely cheating to pass the results off as real world.
Our cell phones are dynamic in the way they use their CPU cores. They don't run 100% all the time. They monitor the phones current state and change for any given situation. That means the app must be able to factor in drops in performance, or changes that occur from the dynamic nature of the device.
Running flat out doesn't allow for these dynamic changes which are needed for fairness and accuracy.
34. Xednyl04 (Posts: 5; Member since: 26 Apr 2013)
I don't get it! they call it cheating when the cpu max out? the CPU was intended to work to its maximal speed. in real life, if we are playing games, the CPUs are stressed to go up the roof and why is that called cheating when the test (e.g. benchmark test) was supposed to see how good the CPU is when being stressed.
35. mattkl (Posts: 174; Member since: 01 Feb 2010)
When you artificially stress out the CPU cores you don't get accurate results how the benchmarks were designed to show. You want the test to stress the cores not have the phone stress it to max before the test starts is the issue.
39. Taters (Posts: 3288; Member since: 28 Jan 2013)
Naw not cheating is even more stupid than cheating really. If you think of things that way, your smartphone doesn't even open an app the majority of the time and you are mostly interacting with the UI the majority of the time and the only 4 apps you probably use a lot is phone, message, gmail, and what's app. I highly doubt that benchmarks are meant to test those things.
Benchmarks are design to test the GPU and CPU and why wouldn't you want to see the theoretical limits of your GPU and CPU? I know that is what I would look for in a benchmark. If I wanted to see the UI performance I would just play with the UI in a store.....