x PhoneArena is looking for new authors! To view all available positions, click here.
  • Home
  • News
  • The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores

Posted: , by Michael H.

Tags:

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores
Samsung has made the news a couple times because the company has been caught artificially boosting benchmark scores on various devices, most recently with the Galaxy Note 3 which saw 20-50% increases in benchmark scores depending on the tests run. And now, more research from AnandTech is showing that almost no manufacturers are innocent in this practice, which really shouldn't come as a surprise.

AnandTech has been looking through the data since July, and has concluded that benchmark gaming is a pretty common occurrence. In fact, the only devices that were found to not change CPU behavior when a benchmarking app is detected were Nexus devices and newer Motorola devices (most pre-Google Moto devices weren't tested). The testing showed that at least one device from major manufacturers like HTC, LG, and Asus all push higher CPU frequencies in order to score better on benchmark tests. 

So far, Samsung seems to be the only manufacturer that has also changed GPU frequencies in order to game benchmark tests, and not all tests are being gamed. AnandTech talks about the need for manufacturers to simply stop the entire practice of altering test results, but that idea seems unlikely to us. Manufacturers all want to gain advantages, even if the number of users who would care about said advantages is relatively small. 

In reality, the most likely scenario is that benchmark makers and OEMs will continue to play cat-and-mouse. Benchmark makers will try to stay ahead of the optimizations, and OEMs will keep trying to gain artificial advantages. We can always try to rename every benchmark test, and get around the optimization whitelists. But really, the best option all around is just to stop putting all that much faith in the raw numbers (which were never that useful to begin with), and just look at real world performance. 

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores
source: AnandTech

61 Comments
  • Options
    Close




posted on 02 Oct 2013, 20:01 19

1. syedzain (Posts: 174; Member since: 08 Oct 2012)


Shame on u Android devices.. always tend to cheat.

We always knew iPhone 5s ( Apple Products ) were fast and smooth. No force close ...

iPhone FTW !!!!

posted on 02 Oct 2013, 20:08 34

3. willard12 (Posts: 702; Member since: 04 Jul 2012)


Oh really?

http://www.phonearena.com/news/Which-platform-crashes-more-Android-or-iOS_id26542

posted on 02 Oct 2013, 20:51 6

16. ardent1 (Posts: 1991; Member since: 16 Apr 2011)


That was a dumb study because the ceteris paribus condition wasn't met. If you no controlled benchmark, then using absolute changes as opposed to relative changes, you are going to see biases.

It's not like people took college-level statistics.

posted on 02 Oct 2013, 23:15 7

35. Dr.Phil (Posts: 863; Member since: 14 Feb 2011)


Of course it doesn't meet that requirement. We are talking about two different operating systems on several different pieces of hardware. However, the fact that that iOS crashed several more times than any Android device is something worth looking at considering that iOS standardizes their hardware whereas Android doesn't. So, how do we know that those crashes weren't on a device running a 512 mHz processor with 256 MB of RAM?

I agree with your notion that we can't accurately compare the two which is why I wouldn't use studies such as that to make a point. However, if you're going to argue with this guy over posting some study then you should also take an equal shot at the guy who said "We always knew iPhone 5s ( Apple Products ) were fast and smooth. No force close ... "

posted on 02 Oct 2013, 23:07

32. davenycept (Posts: 169; Member since: 03 Jul 2012)


Nice stat

posted on 02 Oct 2013, 20:16 4

6. Finalflash (Posts: 1441; Member since: 23 Jul 2013)


It is disappointing that this is being done and is reminiscent of the GPU wars on the PC as well as the CPU wars during AMD and Intels run at each other. They got nothing but marginal gains from this and the risk was serious publicity damage. Why would you even want to rig benchmarks when really, we are coming to a point where no one cares about how much horsepower your phone has that it will never use. Besides, they rigged the benchmarks that matter least, the ones that are just purely theoretical. In my experience, I just skim over those and look at the FPS charts from actual games to determine ability.

Tragically, this is a pretty one sided article and although I respect Anand and Brian for covering it so well, I would like to see what their methodology and results were when testing the iPhone and iPad for any optomizations/cheating Apple is doing. At the moment all we have is Anand's word that Apple does not do this as stated in the comments first page and in the Article where Apple, Google and Moto are cleared of any wrong doing. But really, the latter two were tested and shown the results off while Apple we are just supposed to take at their word. They really need show their work for those otherwise this just loses some serious credibility and seeming like a biased targeted test against Android OEMs. It would be harder to do I suppose because you cannot just rewrite the code a bit and install apps in iOS as easily, so methodology is very important and the results would then be verified based on how they tackled the problem.

posted on 02 Oct 2013, 21:24 1

25. Shatter (Posts: 1967; Member since: 29 May 2013)


Do not even half believe any benchmarks unless its a game or program like unzipping a 5Gb rar or something. Even games/programs are still faked sometimes so go a bit further and actually look at video comparisons by unbiased youtubers.

posted on 02 Oct 2013, 21:00 3

19. sprockkets (Posts: 1077; Member since: 16 Jan 2012)


Moto didn't cheat with the current crop of devices. Sony wasn't tested.

posted on 02 Oct 2013, 22:04 1

29. jacko1977 (Posts: 382; Member since: 11 Feb 2012)


No force close coz it dont tell u it just closes at lease with android u can send report to the dev
i bet ijunk as some cheating in software too
apple new theme
we steal we lie we cheat

posted on 03 Oct 2013, 01:35 2

43. boosook (Posts: 901; Member since: 19 Nov 2012)


But I can't understand what's the problem in running your cpu at full speed in a benchmark… it seems logical to me. I want to score well, and I'm simply using all the power I have. Is this cheating?

posted on 03 Oct 2013, 13:25

58. theruleslawyer (Posts: 108; Member since: 23 Apr 2012)


It's not representing the performance the user will actually see on the device. Such a setting will trash battery life and possibly overheat and damage hardware in the long run.

posted on 03 Oct 2013, 04:51 1

50. Berzerk000 (Posts: 3877; Member since: 26 Jun 2011)


Notice how the Motorola and Nexus devices on the chart aren't boosting benchmarks, and those devices still beat the iPhone handily (aside from the 5S, but the Nexus 5 will fix that).

posted on 03 Oct 2013, 10:19

55. kabhijeet.16 (Posts: 627; Member since: 05 Dec 2012)


Arguments (TROLLS) before reading article...
Only Nexus / Moto devices are fair...
There is nothing mentioned about iPhone.....

posted on 18 Jan 2014, 05:39

61. MobileKicker (Posts: 212; Member since: 19 Sep 2013)


U said right syedzain iPhone FTW

And the android worm is proving otherwise

posted on 02 Oct 2013, 20:05 4

2. spiderpig2894 (Posts: 413; Member since: 10 Jan 2012)


So what about Sony? Could Sony be the only big phone manufacturer to be honest in benchmarks? Or are they refereing to almost all Android OEM?

posted on 02 Oct 2013, 20:11 3

5. Jason2k13 (Posts: 627; Member since: 28 Mar 2013)


Probably all major OEM's

posted on 02 Oct 2013, 22:42 4

30. Taters (Posts: 2590; Member since: 28 Jan 2013)


Sony is the worst. LOL There is no way in hell the Xperia Z should destroy the Nexus 4 in benchmarks when it the Z has the same internals and pushes way more pixels.

posted on 03 Oct 2013, 00:00 5

36. dmn666 (Posts: 194; Member since: 19 Oct 2011)


Aww - little poop - pulling out more poop out of itself. Xperia Z doesn't cheat on benchmarks. N4 scores higher on GPU tests as it has lower res screens. Use google before you poop again

http://www.zdnet.com/benchmarks-htc-one-sony-xperia-z-googlelg-nexus-4-7000014067/

posted on 03 Oct 2013, 02:56 1

47. Taters (Posts: 2590; Member since: 28 Jan 2013)


Look at every other benchmark dumb ass. I can link Anandtech for you but i'll let you Google it yourself. The Xperia Z whips the Nexus 4 in most benchmarks even though technically it isn't possible with the same S4 pro chipset and a 1080p screen vs. a 720p screen.

posted on 03 Oct 2013, 12:22

57. Diazene (Posts: 123; Member since: 01 May 2013)


well, I recall the nexus 4 had throttling issues

posted on 02 Oct 2013, 20:19 4

7. ocilfa (Posts: 329; Member since: 03 Aug 2012)


If everyone is cheating, then isn't it not cheating?

posted on 02 Oct 2013, 20:45 7

12. GeekMovement (Posts: 1499; Member since: 09 Sep 2011)


No, if everyone is cheating, then it's everyone cheating duh.

posted on 03 Oct 2013, 07:09 1

53. Berzerk000 (Posts: 3877; Member since: 26 Jun 2011)


No, because everyone isn't cheating. Motorola and Google devices don't overclock themselves in benchmarks, as you can see in the chart in the article.

posted on 02 Oct 2013, 20:22 9

8. DukeX (Posts: 327; Member since: 28 Aug 2013)


Lmfao like i said the other day that other oems also do this. No one listened. And now this article pops up magically. Lol it's funny how they singled out Samsung. Freakin haters. No i don't have a samsung but fanboys will hate pantech,lg,sony,moto,zte,Huawei,oppo,the list goes on. Only because anyone of those could be the top oem and that's the reason why they hate! People only hate the one's who's winning the most!

posted on 02 Oct 2013, 23:07

33. HouTexan (Posts: 300; Member since: 08 Jan 2012)


Facts

posted on 02 Oct 2013, 20:30 3

9. GoBears (Posts: 333; Member since: 27 Apr 2012)


If they were really trying to "cheat" the consumers, I'm sure they would make it much more difficult to detect.

posted on 02 Oct 2013, 20:43 1

11. Scott93274 (Posts: 521; Member since: 06 Aug 2013)


Well, I would hope that Motorolai's honest with its benchmark scores. I'm planning on getting a Droid Maxx around Black Friday if there are any deals to take advantage of. I got to play around with the phone today and I loved it.

posted on 02 Oct 2013, 21:08 4

22. DukeX (Posts: 327; Member since: 28 Aug 2013)


Tell yourself. Will those scores really matter if the phone runs smooth? Caringabout benchmark scores is like jumping for joy and freezing in midair. The joy is there but everyone will see you and be like "why is he so happy?"

posted on 02 Oct 2013, 20:45 2

13. xperiaDROID (Posts: 5047; Member since: 08 Mar 2013)


What about Sony, Huawei, ZTE, Xiaomi, Motorola and Oppo? Are these the only OEMs that are honest in benchmarks? I really hope so.

posted on 02 Oct 2013, 20:59 5

18. Jason2k13 (Posts: 627; Member since: 28 Mar 2013)


no you dont, no one cares about benchmarks in 2013... i dont know about other OEM's, but samsung did not lie about benchmarks since they clearly advertised their phones to run at that full speed.

Samsung even mentioned games and most apps don't run at full potential like what the benchmark says, just to save battery which is perfectly fine, its not like we will see a difference in performance, but we will in battery life and overheating.

posted on 02 Oct 2013, 21:44

28. xperiaDROID (Posts: 5047; Member since: 08 Mar 2013)


But reviewers still care about benchmarks, so you can't really say no one cares about benchmarks in 2013. About Samsung, it's really hard to say that Samsung lied or not, because there are news that said Samsung caught for boosting benchmark scores, or maybe the news were lying?

posted on 02 Oct 2013, 20:50

14. KineticArc (Posts: 11; Member since: 29 Jun 2011)


The only way to stop nonsensical, benchmark boosting actions from manufacturers is to stop buying any product that has a high rate of artificial, performance inducing inflation. I don't care who it is; if they cannot be honest about their devices, then I want no part in their business. Unfortunately, it seems all companies have this mantra of skirting ethics for the sake of profits...

posted on 02 Oct 2013, 21:11 2

23. vikingsfootball09 (Posts: 41; Member since: 02 Oct 2013)


did samsung specify any bench mark numbers? if so, can you provide a link please? because the only ones that i know of that does these bench marks are people from anandtech (and the like)

and if samsung, lg, asus and htc were found to be doing it, whos to say that others arent doing it? everyone thought it was only samsung...now we find out that lg, asus and htc are also doing it. alot of android/samsung haters are quick to judge, haha. OMG!!!

posted on 02 Oct 2013, 21:32 3

26. Shatter (Posts: 1967; Member since: 29 May 2013)


Samsung wasn't doing it, what they did was when the phone was not hot it would overclock the chip if the program was CPU hungry. It has been proven the CPU Overclocking/boost function works outside of benchmarks in games and a few other things so it is not artificial. The reason it activates in benchmarks is because they use 100% of the CPU which tells the software to up the chips clocks.

Artificial scores are from them optimizing their software around making benchmarks think it is better than it really is -cough- Apple does it too -cough-

posted on 03 Oct 2013, 00:31

41. taojj (Posts: 33; Member since: 01 Aug 2013)


And you are Android Lawyer. hahaha

posted on 02 Oct 2013, 21:03 3

20. Finest (Posts: 76; Member since: 16 Aug 2013)


So that makes SONY Honest ?

posted on 03 Oct 2013, 02:48

45. Tsepz_GP (Posts: 779; Member since: 12 Apr 2012)


Nobody has actually bothered to check Sony phones, they only checked popular OEMs in the USA.

posted on 02 Oct 2013, 21:05

21. vikingsfootball09 (Posts: 41; Member since: 02 Oct 2013)


anyone with common sense would see that this is not cheating...its not like the codes were used to overclock at a certain ghz, other than the specified, when detecting a benchmarking tool.

posted on 02 Oct 2013, 21:19

24. CX3NT3_713 (Posts: 1650; Member since: 18 Apr 2011)


:-) they should all just have a Lagg benchmark test ..;-) hehe

posted on 02 Oct 2013, 21:34 2

27. DigitalBoy05 (Posts: 200; Member since: 04 Jun 2011)


just a thought here but if all the OEM's are already cheating on the benchmarks then wouldnt the phone that scored the highest still be the most capable of the phones? You can only fudge the numbers as much as the hardware will physically support before overheating and crapping out.

posted on 03 Oct 2013, 00:27

39. taojj (Posts: 33; Member since: 01 Aug 2013)


Go buy your devices on that basis.

posted on 02 Oct 2013, 22:47 6

31. Taters (Posts: 2590; Member since: 28 Jan 2013)


If you want true cheating, look no further than Apple. Hiding behind a below 720p screen and Safari to artificially inflate their scores. Yes, below 720p screen, 1gb of ram, and 1570 mah battery masquerading as a beast and Apple fans falling for it.

Run browsermark and whatever else the iphone lovers champion using chrome on the iphone 5 and the Nexus 4 beats it handily. Similarly, run the same tests but using Chrome instead of Safari and the 5s and it probably gets trounced by mid range Android phones too.

posted on 02 Oct 2013, 23:13 1

34. alaw.14 (Posts: 119; Member since: 10 Sep 2013)


Now im wondering why arent they saying anything about apple exept for positive articles.To me i think this isnt even real and that they just post bad articles on android and make android users switch to apple.And why did they they do this fake benchmark on the note 3 first and now they are just now doing it with other android phones and that proves they hate samsung the most then other android phones.just like they gave bad scores for the note 3 and note 10.1 2014 edition

posted on 03 Oct 2013, 00:01 5

37. ilia1986 (unregistered)


I think most people fail to realize that it doesn't really matter which flagship has more horsepower but rather which unique features in terms of software and hardware design does each one have.

Both the G2, the Z1 and the Note 3 not to mention a few others already have the S800 CPU. The difference? A whole bunch of other stuff.

posted on 03 Oct 2013, 00:22 1

38. taojj (Posts: 33; Member since: 01 Aug 2013)


"Samsung has made the news a couple times because the company has been caught artificially boosting benchmark scores on various devices, most recently with the Galaxy Note 3 which saw 20-50% increases in benchmark scores depending on the tests run."

Still you don't find it important to put Samsung's name in the Title of your story. How mean or paid by Samsung?

posted on 03 Oct 2013, 00:29 3

40. xfire99 (Posts: 412; Member since: 14 Mar 2012)


In which way is these cheating?
Samsung never publicly claimed max GPU frequencies for the Exynos 5 Octa andhttp://en.wikipedia.org/wiki/Exynos_%28system_on_chip%29
"T PowerVR SGX544MP3 (tri-core) @ 480 MHz 51.84 GFLOPS (532 MHz in some full-screen apps)"
So 5410 GPU can run at 532 Mhz, so whats the problem here?
If its cheat, both CPU/GPU will runs at overclock speed, higher then its official CPU speeds 1,6Ghz/GPU 532Mhz.
As long it stay within specs, then I cant see its cheating and if CPU/GPU downclock the frequencies (android governor) when idling to save battery. Is it cheating too? Since it dont runs at max speed as it should?
Cant see why its so big huss in becnhmarking numbers.

posted on 03 Oct 2013, 00:35

42. Whateverman (Posts: 3187; Member since: 17 May 2009)


Wow, this isn't good. But I'm suspicious as to why there is no mention of the iPhone at all. In every study you can think of, every tech site is quick to "Oh" and "Ah" the same three year old iPhone like parents over baby's first steps, first word, first tooth... Was there no test done? Did it pass with flying colors? What? We just want to know.

posted on 03 Oct 2013, 02:31 1

44. Trex95 (Posts: 541; Member since: 03 Mar 2013)


That's why i prefer Nexus.

posted on 03 Oct 2013, 02:51 2

46. twens (Posts: 518; Member since: 25 Feb 2012)


Cheating or no cheating. My international note 3 is the smoothers phone I have ever used in my entire life. I have been pushing this bad boy to the wall and I still can't find a single lag. Even with power saver on. Damn, Sammy really made this bad boy fly.

posted on 03 Oct 2013, 04:32

48. wiiandds (Posts: 60; Member since: 15 Mar 2013)


did anyone notice the nexus 7 has and updated processor. its suppose to have a S4 Pro and this say it has a 600 in it

posted on 03 Oct 2013, 04:48 1

49. Berzerk000 (Posts: 3877; Member since: 26 Jun 2011)


They said it was an S4 Pro at the announcement, but it was revealed later that it used Krait 300 and DDR3 RAM rather than Krait 200 and DDR2 like a traditional S4 Pro, so it is essentially an underclocked S600.

posted on 03 Oct 2013, 06:46

52. wiiandds (Posts: 60; Member since: 15 Mar 2013)


ok that makes sense

posted on 03 Oct 2013, 05:09

51. justsayit (Posts: 159; Member since: 09 May 2013)


i don care about benchmark, i just care about real life perform
as a lg fans,
little disappointing,the lg g2 look more slower than lg optimus g :(
a phone with 4k video playback ability(means greater gpu) and greater cpu look slower than a with only 1080p playback and older gen cpu...

posted on 03 Oct 2013, 10:08

54. jroc74 (Posts: 4720; Member since: 30 Dec 2010)


Not too surprised about HTC. They almost exclusively use Qualcomm chips, there is a benchmark tool thats specifically made by Qualcomm....

Then you have benchmark tools like Quadrant that can easily be tweaked for better results.

I like Anandtech, can be relied on to get the to bottom of most things tech related. I also like that Motorola isnt too involved in this. Even tho they have used Qualcomm chips the last few years. They need to team back up with Texas Instruments...seriously.

posted on 03 Oct 2013, 11:47

56. saiki4116 (Posts: 325; Member since: 31 Mar 2011)


The Playground is leveled.. :P

posted on 03 Oct 2013, 20:34

59. Muayyad (Posts: 96; Member since: 05 Oct 2012)


who really cares! use the phone and you will know from the lag if present which is the faster phone.

posted on 05 Oct 2013, 15:59

60. Ronny666 (Posts: 108; Member since: 08 Jun 2013)


We are unable to detect this in iphones mainly because it is running on closed operating system unlike androids. Most OEMs optimise their devices for benchmarking apps so that it can display the full potential of their hardware. That's not called cheating unless if they overclocked it mainly for benchmarking tools.

In Samsung's case, the GPU does hit at the highest frequency when running certain apps but most of the time it runs on lower frequency so that it doesn't generate too much of heat and suffer shorter life span. The normal GPU speed is adequate for running most applications even graphic extensive games.

* Some comments have been hidden, because they don't meet the discussions rules.

Want to comment? Please login or register.

Latest stories