The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores
Samsung has made the news a couple times because the company has been caught artificially boosting benchmark scores on various devices, most recently with the Galaxy Note 3 which saw 20-50% increases in benchmark scores depending on the tests run. And now, more research from AnandTech is showing that almost no manufacturers are innocent in this practice, which really shouldn't come as a surprise.

AnandTech has been looking through the data since July, and has concluded that benchmark gaming is a pretty common occurrence. In fact, the only devices that were found to not change CPU behavior when a benchmarking app is detected were Nexus devices and newer Motorola devices (most pre-Google Moto devices weren't tested). The testing showed that at least one device from major manufacturers like HTC, LG, and Asus all push higher CPU frequencies in order to score better on benchmark tests. 

So far, Samsung seems to be the only manufacturer that has also changed GPU frequencies in order to game benchmark tests, and not all tests are being gamed. AnandTech talks about the need for manufacturers to simply stop the entire practice of altering test results, but that idea seems unlikely to us. Manufacturers all want to gain advantages, even if the number of users who would care about said advantages is relatively small. 

In reality, the most likely scenario is that benchmark makers and OEMs will continue to play cat-and-mouse. Benchmark makers will try to stay ahead of the optimizations, and OEMs will keep trying to gain artificial advantages. We can always try to rename every benchmark test, and get around the optimization whitelists. But really, the best option all around is just to stop putting all that much faith in the raw numbers (which were never that useful to begin with), and just look at real world performance. 

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores
source: AnandTech

FEATURED VIDEO

61 Comments

1. syedzain

Posts: 175; Member since: Oct 08, 2012

Shame on u Android devices.. always tend to cheat. We always knew iPhone 5s ( Apple Products ) were fast and smooth. No force close ... iPhone FTW !!!!

16. ardent1

Posts: 2000; Member since: Apr 16, 2011

That was a dumb study because the ceteris paribus condition wasn't met. If you no controlled benchmark, then using absolute changes as opposed to relative changes, you are going to see biases. It's not like people took college-level statistics.

35. Dr.Phil

Posts: 2293; Member since: Feb 14, 2011

Of course it doesn't meet that requirement. We are talking about two different operating systems on several different pieces of hardware. However, the fact that that iOS crashed several more times than any Android device is something worth looking at considering that iOS standardizes their hardware whereas Android doesn't. So, how do we know that those crashes weren't on a device running a 512 mHz processor with 256 MB of RAM? I agree with your notion that we can't accurately compare the two which is why I wouldn't use studies such as that to make a point. However, if you're going to argue with this guy over posting some study then you should also take an equal shot at the guy who said "We always knew iPhone 5s ( Apple Products ) were fast and smooth. No force close ... "

32. davenycept

Posts: 200; Member since: Jul 03, 2012

Nice stat

6. Finalflash

Posts: 4062; Member since: Jul 23, 2013

It is disappointing that this is being done and is reminiscent of the GPU wars on the PC as well as the CPU wars during AMD and Intels run at each other. They got nothing but marginal gains from this and the risk was serious publicity damage. Why would you even want to rig benchmarks when really, we are coming to a point where no one cares about how much horsepower your phone has that it will never use. Besides, they rigged the benchmarks that matter least, the ones that are just purely theoretical. In my experience, I just skim over those and look at the FPS charts from actual games to determine ability. Tragically, this is a pretty one sided article and although I respect Anand and Brian for covering it so well, I would like to see what their methodology and results were when testing the iPhone and iPad for any optomizations/cheating Apple is doing. At the moment all we have is Anand's word that Apple does not do this as stated in the comments first page and in the Article where Apple, Google and Moto are cleared of any wrong doing. But really, the latter two were tested and shown the results off while Apple we are just supposed to take at their word. They really need show their work for those otherwise this just loses some serious credibility and seeming like a biased targeted test against Android OEMs. It would be harder to do I suppose because you cannot just rewrite the code a bit and install apps in iOS as easily, so methodology is very important and the results would then be verified based on how they tackled the problem.

25. Shatter

Posts: 2036; Member since: May 29, 2013

Do not even half believe any benchmarks unless its a game or program like unzipping a 5Gb rar or something. Even games/programs are still faked sometimes so go a bit further and actually look at video comparisons by unbiased youtubers.

19. sprockkets

Posts: 1612; Member since: Jan 16, 2012

Moto didn't cheat with the current crop of devices. Sony wasn't tested.

29. jacko1977

Posts: 428; Member since: Feb 11, 2012

No force close coz it dont tell u it just closes at lease with android u can send report to the dev i bet ijunk as some cheating in software too apple new theme we steal we lie we cheat

43. boosook

Posts: 1442; Member since: Nov 19, 2012

But I can't understand what's the problem in running your cpu at full speed in a benchmark… it seems logical to me. I want to score well, and I'm simply using all the power I have. Is this cheating?

58. theruleslawyer

Posts: 108; Member since: Apr 23, 2012

It's not representing the performance the user will actually see on the device. Such a setting will trash battery life and possibly overheat and damage hardware in the long run.

50. Berzerk000

Posts: 4275; Member since: Jun 26, 2011

Notice how the Motorola and Nexus devices on the chart aren't boosting benchmarks, and those devices still beat the iPhone handily (aside from the 5S, but the Nexus 5 will fix that).

55. kabhijeet.16

Posts: 884; Member since: Dec 05, 2012

Arguments (TROLLS) before reading article... Only Nexus / Moto devices are fair... There is nothing mentioned about iPhone.....

61. MobileKicker

Posts: 212; Member since: Sep 19, 2013

U said right syedzain iPhone FTW And the android worm is proving otherwise

2. spiderpig2894

Posts: 597; Member since: Jan 10, 2012

So what about Sony? Could Sony be the only big phone manufacturer to be honest in benchmarks? Or are they refereing to almost all Android OEM?

5. Jason2k13

Posts: 1447; Member since: Mar 28, 2013

Probably all major OEM's

30. Taters

Posts: 6474; Member since: Jan 28, 2013

Sony is the worst. LOL There is no way in hell the Xperia Z should destroy the Nexus 4 in benchmarks when it the Z has the same internals and pushes way more pixels.

36. dmn666

Posts: 244; Member since: Oct 19, 2011

Aww - little poop - pulling out more poop out of itself. Xperia Z doesn't cheat on benchmarks. N4 scores higher on GPU tests as it has lower res screens. Use google before you poop again http://www.zdnet.com/benchmarks-htc-one-sony-xperia-z-googlelg-nexus-4-7000014067/

47. Taters

Posts: 6474; Member since: Jan 28, 2013

Look at every other benchmark dumb ass. I can link Anandtech for you but i'll let you Google it yourself. The Xperia Z whips the Nexus 4 in most benchmarks even though technically it isn't possible with the same S4 pro chipset and a 1080p screen vs. a 720p screen.

57. Diazene

Posts: 129; Member since: May 01, 2013

well, I recall the nexus 4 had throttling issues

7. ocilfa

Posts: 334; Member since: Aug 03, 2012

If everyone is cheating, then isn't it not cheating?

12. GeekMovement unregistered

No, if everyone is cheating, then it's everyone cheating duh.

53. Berzerk000

Posts: 4275; Member since: Jun 26, 2011

No, because everyone isn't cheating. Motorola and Google devices don't overclock themselves in benchmarks, as you can see in the chart in the article.

8. DukeX

Posts: 327; Member since: Aug 28, 2013

Lmfao like i said the other day that other oems also do this. No one listened. And now this article pops up magically. Lol it's funny how they singled out Samsung. Freakin haters. No i don't have a samsung but fanboys will hate pantech,lg,sony,moto,zte,Huawei,oppo,the list goes on. Only because anyone of those could be the top oem and that's the reason why they hate! People only hate the one's who's winning the most!

33. HouTexan

Posts: 461; Member since: Jan 08, 2012

Facts

9. GoBears

Posts: 456; Member since: Apr 27, 2012

If they were really trying to "cheat" the consumers, I'm sure they would make it much more difficult to detect.

11. Scott93274

Posts: 6025; Member since: Aug 06, 2013

Well, I would hope that Motorolai's honest with its benchmark scores. I'm planning on getting a Droid Maxx around Black Friday if there are any deals to take advantage of. I got to play around with the phone today and I loved it.

22. DukeX

Posts: 327; Member since: Aug 28, 2013

Tell yourself. Will those scores really matter if the phone runs smooth? Caringabout benchmark scores is like jumping for joy and freezing in midair. The joy is there but everyone will see you and be like "why is he so happy?"

13. xperiaDROID

Posts: 5629; Member since: Mar 08, 2013

What about Sony, Huawei, ZTE, Xiaomi, Motorola and Oppo? Are these the only OEMs that are honest in benchmarks? I really hope so.

18. Jason2k13

Posts: 1447; Member since: Mar 28, 2013

no you dont, no one cares about benchmarks in 2013... i dont know about other OEM's, but samsung did not lie about benchmarks since they clearly advertised their phones to run at that full speed. Samsung even mentioned games and most apps don't run at full potential like what the benchmark says, just to save battery which is perfectly fine, its not like we will see a difference in performance, but we will in battery life and overheating.

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.