x PhoneArena is hiring! Reviewer in the USA
  • Hidden picShow menu
  • Home
  • News
  • Hall of Shame: Futuremark's 3DMark de-lists HTC and Samsung for cheating

Hall of Shame: Futuremark's 3DMark de-lists HTC and Samsung for cheating

Hall of Shame: Futuremark's 3DMark de-lists HTC and Samsung for cheating
Whenever any kind of a synthetic benchmark is involved, we find ourselves reiterating over and over that these results should be taken with caution, and certainly in the context of a controlled environment. And while this latter quality allows for standardized testing to take place, it's also inherently predictable, meaning that manufacturers could rig the procedure.

Such cases are common in the PC arena, and they obviously have spread to the mobile space, with numerous manufacturers downright cheating, despite their disappointing denials. For a while now, we simply thought that the marketing geniuses at these companies apparently consider the extra sales from the perceived performance edge over rivals to be sufficient to outweigh the effects of a negative press. If that's indeed the case, those people may want to reconsider, as Futuremark, the company behind the hugely-popular 3DMark benchmark, have now officially de-listed and shamed a number of Samsung and HTC devices as cheats. These include the Galaxy Note 10.1 2014 and Galaxy Note 3, both the Exynos and Snapdragon variants, along with the HTC One and One mini.

Hall of Shame: Futuremark's 3DMark de-lists HTC and Samsung for cheating

Said otherwise, the folks over at Futuremark are siding with the community, and seem to agree that at least Samsung and HTC have been rigging the tool. Unfortunately, the aforementioned duo is far from the only manufacturers to engage in this kind of activity, and we'll be expecting Futuremark to continue taking action.

People rely on Futuremark benchmarks to produce accurate and unbiased results. That’s why we have clear rules for hardware manufacturers and software developers that specify how a platform can interact with our benchmark software,” said Futuremark President Oliver Baltuch. “In simple terms, a device must run our benchmarks without modification as if they were any other application.”

In case you're a little dim on the whole affair, the story first broke out after a forum user found that the Samsung Galaxy Note 3 has a string of code in its software that forces the cores of its chipset to run at full power and not idle at all whenever a known benchmark is detected. This results in a higher score, but does not represent real-world scenarios, and basically forces competitors to also jump on the bandwagon, lest their devices are perceived as less powerful.

51 Comments
  • Options
    Close






posted on 26 Nov 2013, 04:42 23

1. D.Aceveda (Posts: 396; Member since: 30 Jun 2012)


Good. No cheating

posted on 26 Nov 2013, 05:16 2

5. SAYED-EJAZ (Posts: 225; Member since: 10 Oct 2013)


I think almost all brands do cheating.
First we come to know with samsung , then we understood its not samsung alone :P

posted on 26 Nov 2013, 05:21

6. androiphone20 (Posts: 1654; Member since: 10 Jul 2013)


Agreed, I also think some of them just go way too far with it

posted on 26 Nov 2013, 05:25 3

7. SAYED-EJAZ (Posts: 225; Member since: 10 Oct 2013)


but still cheating is cheating,,small or big.
right!

posted on 26 Nov 2013, 05:35 1

8. androiphone20 (Posts: 1654; Member since: 10 Jul 2013)


True, you also know how it feels when you get the word that you're smartphone isn't all that powerful as it was said to be

posted on 26 Nov 2013, 05:45 2

9. marbovo (Posts: 658; Member since: 16 May 2013)


We can see the cheaters comparing nexus 5 result with other phones with snapdragon 800, most of the them have a very higher result comparing to N5, probably because it is the only one not cheating

posted on 26 Nov 2013, 06:06 1

11. SAYED-EJAZ (Posts: 225; Member since: 10 Oct 2013)


That might be right,
but i dont like neXus 5.
Two things are must for me,thats why i always love sammy.
1) Removable battery (MEGA MEGA IMPORTANT,I dont want to hang like a dog near to charger point especially during long train journey,but with nexus i have to do in my home also)
2) MicroSD slot (MUST MUST,companies simply taking our money telling 16gb,32gb when memory card is cheap) .
last but not least Super Amoled, it is like a drug for me !

posted on 26 Nov 2013, 06:17 1

13. Smokin (Posts: 120; Member since: 10 Oct 2013)


i dont think cheating is the problem here. pure android is known for its not very tuned software for processors..

posted on 26 Nov 2013, 07:26

17. FYoung (Posts: 42; Member since: 08 Oct 2013)


So, that's why Futuremark delisted several Samsung and HTC models.

Thank you for clarifying that, or I would have kept on believing Futuremark President Oliver Baltuch, who said: “In simple terms, a device must run our benchmarks without modification as if they were any other application.” I can't believe how naive or outright dishonest he is to say that Samsung and HTC have been cheating.

posted on 26 Nov 2013, 07:23 1

16. JakeLee (banned) (Posts: 1021; Member since: 02 Nov 2013)


Nexus is shipped with pure vanilla AOSP on it while the OEMs with highly optimized AOSP for the particular SoC.

I mean *real* optimizations, cheating benchmarks being optional.

posted on 26 Nov 2013, 07:40

26. marbovo (Posts: 658; Member since: 16 May 2013)


Who says that?

posted on 26 Nov 2013, 10:48

35. JakeLee (banned) (Posts: 1021; Member since: 02 Nov 2013)


EVERYONE working at/with OEMs or SoC vendors knows that.

Satisfied?

Nexus isn't the best choice for performance freaks.

The difference in Webkit, Dalvik JIT, etc is huge.

posted on 26 Nov 2013, 12:57

42. marbovo (Posts: 658; Member since: 16 May 2013)


No man, I´m not satisfied,

First, who is everyone? because I don´t se anyone saying that.

Second, do you really believe in OEM vendor(assuming that anyone of them have said that)?

Third, what is the huge diference between them?

Fourth, Nexus doesn´t come with pure AOSP...

Give a source plz...

posted on 26 Nov 2013, 04:48 3

2. mr.techdude (Posts: 571; Member since: 19 Nov 2012)


That's why I never listen to such numbers, numbers are lies only when completion is involved. So just a quick question how much is apple using on their iPhone cause their benchmark results squashes the completion? Just out of curiosity. PA

posted on 26 Nov 2013, 07:33

22. Shatter (Posts: 2036; Member since: 29 May 2013)


Its impossible to tell but I would put $ on Apple getting caught boosting benchmark scores in the next year to make their iphone look just slightly better than an android phone to trick people. Now that people are realizing the scores are lies the benchmark apps will start preventing cheating.

Motorala, Nexus devices and the Nvidia shield are real scores not cheaters.

LG also cheats in 3dmark why is it now banned

posted on 26 Nov 2013, 07:52

30. wargreymon (Posts: 742; Member since: 05 Nov 2013)


Got any proof of LG cheating in 3DMark? I tried the benchmarks that were in the list before and there was no sign of cheating in 3DMark with G2 or optimus G.

posted on 26 Nov 2013, 11:35 1

36. ardent1 (Posts: 2000; Member since: 16 Apr 2011)


Why would Apple cheat?

They've been building hardware and software as an integrated product for 30+ years. It's in their DNA.

The difference today is that Apple is now designing its own chips. It was first to market with 64-bit processor in the mobile space. Apple gets it -- the world is becoming graphics oriented.

posted on 26 Nov 2013, 05:06 2

3. androiphone20 (Posts: 1654; Member since: 10 Jul 2013)


That's why some OEMs refrain from trying to wow consumers with insanely high specs

posted on 26 Nov 2013, 07:00 1

14. ardent1 (Posts: 2000; Member since: 16 Apr 2011)


Everyone knows Samsung never cheats.

posted on 26 Nov 2013, 09:10

32. androiphone20 (Posts: 1654; Member since: 10 Jul 2013)


lulz :D

posted on 26 Nov 2013, 05:11 3

4. Tomfromsouth (Posts: 76; Member since: 02 Apr 2012)


Ouch!

posted on 26 Nov 2013, 06:03 2

10. _Bone_ (Posts: 2155; Member since: 29 Oct 2012)


You know the SOLE PURPOSE I look at benchmarks is to see the absolute end-performance of the hardware, without software cap or any limit, and I bet you wanna know the Koneigsegg Agera's max speed as well, not where the street limiter caps it.

It's a FACT that a "rigged Note 3" can really pull those high numbers and that's why benchmarks are for, to see the SD800/DRAM/A330 hardware maximum then know where we can get with custom ROMs and tweaks. The actual rigging is putting a frequency cap on every other app, and there's a reason for you calendar doesn't run at full speed: to save you battery.

When I look for everyday performance I do everyday things with the product. Still, when I look up car performance I wanna know the maximums without caps, don't you?

posted on 26 Nov 2013, 07:22 2

15. FYoung (Posts: 42; Member since: 08 Oct 2013)


No way. Normal people want to know how it would perform if they bought a regular unit from a store and used it normally.

It is cheating to add code that would not be enabled in real life so as to make it run unsustainably fast to make it look like it performs better than the competition.

posted on 26 Nov 2013, 07:52 2

29. _Bone_ (Posts: 2155; Member since: 29 Oct 2012)


As already said, the only way you're gonna get a look at real life everyday usage is using it in real life, every day, end of story. :)

Benchmarking was only the fool's tool for that, ppl. who actually know what to do with them want the maximum numbers however, just like ppl. who buy cars to modify it etc. Average Jane shouldn't care much about it let alone express something that's far beyond her understanding, cause, you know she doesn't tune cars.

posted on 26 Nov 2013, 12:02

37. FYoung (Posts: 42; Member since: 08 Oct 2013)


"...the only way you're gonna get a look at real life everyday usage is using it in real life, every day, end of story. "

No, the only way to predict performance in real life for normal people is to use benchmarks that mimic real life (eg rendering web pages, playing a game, enhancing a photo, playing a video) in a standardized way and that cannot be tricked (e.g. by buying the units anonymously and randomly renaming the benchmark's executables and scripts).

Everything else is useless due to subjectivity, bias and anecdota.

posted on 26 Nov 2013, 07:32 1

21. JakeLee (banned) (Posts: 1021; Member since: 02 Nov 2013)


Bull. What are you smoking?

posted on 26 Nov 2013, 07:40 1

27. tasior (Posts: 265; Member since: 04 Nov 2012)


One of previous articles, had source with technical details of cheats. Almost all producers made the optimization, that cuts off power saving system, while running benchmark. Samsung however has gone further. It cut off temperature limits for whole CPU u

posted on 26 Nov 2013, 07:43 1

28. tasior (Posts: 265; Member since: 04 Nov 2012)


Samsung however has gone further. It cut off temperature limits for whole CPU unit. That means, that if the benchmark took longer (like when You play games), Your phone would crash. That means that such a benchmark doesn't test real computing power, because it allows phone to go to unstable processing power. In my mind this is serious cheating, and Samsung deserves to be banned, for such operations.

PS. Sorry for two posts. It wasn't intended.

posted on 26 Nov 2013, 08:18 2

31. _Bone_ (Posts: 2155; Member since: 29 Oct 2012)


Sammy already shared and proved that more than a few apps use the full values just like benchmarks. The limitations are the other way around: you are being "cheated off" max frequencies in most cases so your device lasts longer. It's called 3... 2... 1... optimization for efficiency.

From this bullish attitude floating around, we're one step away from some idiot suing Samsung for not getting full 2.26GHz running the Note 3 calendar app. "IT'S ON THE BOX GODDAMNING, I WANT MY 2.3GHZ 100% OF THE TIME WHEN I RUN THE CALENDAR, OR I WANT MY MONAH BACK!!!"

Sammy should just provide them a ROM and when they start whining that their 6,5 screen on time was cut down to 2, they'll shut their mouth is a little understanding how modern technology works.

posted on 26 Nov 2013, 06:15

12. Smokin (Posts: 120; Member since: 10 Oct 2013)


Oh finally great news. I debated this stupid guy on twitter saying that S200 is MORE powerful than S4 Pro and my wifi is effin slow today.

OT : IMO its not cheating because like a bodybuilder, while he's not flexing his body its already great but if he's flexing the body will become even greater and i think the these phones only flex their muscles here so no cheating

Want to comment? Please login or register.

Latest stories