Hall of Shame: Futuremark's 3DMark de-lists HTC and Samsung for cheating
Such cases are common in the PC arena, and they obviously have spread to the mobile space, with numerous manufacturers downright cheating, despite their disappointing denials. For a while now, we simply thought that the marketing geniuses at these companies apparently consider the extra sales from the perceived performance edge over rivals to be sufficient to outweigh the effects of a negative press. If that's indeed the case, those people may want to reconsider, as Futuremark, the company behind the hugely-popular 3DMark benchmark, have now officially de-listed and shamed a number of Samsung and HTC devices as cheats. These include the Galaxy Note 10.1 2014 and Galaxy Note 3, both the Exynos and Snapdragon variants, along with the HTC One and One mini.
Said otherwise, the folks over at Futuremark are siding with the community, and seem to agree that at least Samsung and HTC have been rigging the tool. Unfortunately, the aforementioned duo is far from the only manufacturers to engage in this kind of activity, and we'll be expecting Futuremark to continue taking action.
In case you're a little dim on the whole affair, the story first broke out after a forum user found that the Samsung Galaxy Note 3 has a string of code in its software that forces the cores of its chipset to run at full power and not idle at all whenever a known benchmark is detected. This results in a higher score, but does not represent real-world scenarios, and basically forces competitors to also jump on the bandwagon, lest their devices are perceived as less powerful.
5. SAYED-EJAZ (Posts: 210; Member since: 10 Oct 2013)
I think almost all brands do cheating.
First we come to know with samsung , then we understood its not samsung alone :P
6. androiphone20 (Posts: 1114; Member since: 10 Jul 2013)
Agreed, I also think some of them just go way too far with it
7. SAYED-EJAZ (Posts: 210; Member since: 10 Oct 2013)
but still cheating is cheating,,small or big.
8. androiphone20 (Posts: 1114; Member since: 10 Jul 2013)
True, you also know how it feels when you get the word that you're smartphone isn't all that powerful as it was said to be
9. marbovo (Posts: 475; Member since: 16 May 2013)
We can see the cheaters comparing nexus 5 result with other phones with snapdragon 800, most of the them have a very higher result comparing to N5, probably because it is the only one not cheating
11. SAYED-EJAZ (Posts: 210; Member since: 10 Oct 2013)
That might be right,
but i dont like neXus 5.
Two things are must for me,thats why i always love sammy.
1) Removable battery (MEGA MEGA IMPORTANT,I dont want to hang like a dog near to charger point especially during long train journey,but with nexus i have to do in my home also)
2) MicroSD slot (MUST MUST,companies simply taking our money telling 16gb,32gb when memory card is cheap) .
last but not least Super Amoled, it is like a drug for me !
13. Smokin (Posts: 97; Member since: 10 Oct 2013)
i dont think cheating is the problem here. pure android is known for its not very tuned software for processors..
17. FYoung (Posts: 27; Member since: 08 Oct 2013)
So, that's why Futuremark delisted several Samsung and HTC models.
Thank you for clarifying that, or I would have kept on believing Futuremark President Oliver Baltuch, who said: “In simple terms, a device must run our benchmarks without modification as if they were any other application.” I can't believe how naive or outright dishonest he is to say that Samsung and HTC have been cheating.
16. JakeLee (Posts: 372; Member since: 02 Nov 2013)
Nexus is shipped with pure vanilla AOSP on it while the OEMs with highly optimized AOSP for the particular SoC.
I mean *real* optimizations, cheating benchmarks being optional.
35. JakeLee (Posts: 372; Member since: 02 Nov 2013)
EVERYONE working at/with OEMs or SoC vendors knows that.
Nexus isn't the best choice for performance freaks.
The difference in Webkit, Dalvik JIT, etc is huge.
42. marbovo (Posts: 475; Member since: 16 May 2013)
No man, I´m not satisfied,
First, who is everyone? because I don´t se anyone saying that.
Second, do you really believe in OEM vendor(assuming that anyone of them have said that)?
Third, what is the huge diference between them?
Fourth, Nexus doesn´t come with pure AOSP...
Give a source plz...
2. mr.techdude (Posts: 491; Member since: 19 Nov 2012)
That's why I never listen to such numbers, numbers are lies only when completion is involved. So just a quick question how much is apple using on their iPhone cause their benchmark results squashes the completion? Just out of curiosity. PA
22. Shatter (Posts: 1763; Member since: 29 May 2013)
Its impossible to tell but I would put $ on Apple getting caught boosting benchmark scores in the next year to make their iphone look just slightly better than an android phone to trick people. Now that people are realizing the scores are lies the benchmark apps will start preventing cheating.
Motorala, Nexus devices and the Nvidia shield are real scores not cheaters.
LG also cheats in 3dmark why is it now banned
30. wargreymon (Posts: 53; Member since: 05 Nov 2013)
Got any proof of LG cheating in 3DMark? I tried the benchmarks that were in the list before and there was no sign of cheating in 3DMark with G2 or optimus G.
36. ardent1 (Posts: 1983; Member since: 16 Apr 2011)
Why would Apple cheat?
They've been building hardware and software as an integrated product for 30+ years. It's in their DNA.
The difference today is that Apple is now designing its own chips. It was first to market with 64-bit processor in the mobile space. Apple gets it -- the world is becoming graphics oriented.
3. androiphone20 (Posts: 1114; Member since: 10 Jul 2013)
That's why some OEMs refrain from trying to wow consumers with insanely high specs
10. _Bone_ (Posts: 2061; Member since: 29 Oct 2012)
You know the SOLE PURPOSE I look at benchmarks is to see the absolute end-performance of the hardware, without software cap or any limit, and I bet you wanna know the Koneigsegg Agera's max speed as well, not where the street limiter caps it.
It's a FACT that a "rigged Note 3" can really pull those high numbers and that's why benchmarks are for, to see the SD800/DRAM/A330 hardware maximum then know where we can get with custom ROMs and tweaks. The actual rigging is putting a frequency cap on every other app, and there's a reason for you calendar doesn't run at full speed: to save you battery.
When I look for everyday performance I do everyday things with the product. Still, when I look up car performance I wanna know the maximums without caps, don't you?
15. FYoung (Posts: 27; Member since: 08 Oct 2013)
No way. Normal people want to know how it would perform if they bought a regular unit from a store and used it normally.
It is cheating to add code that would not be enabled in real life so as to make it run unsustainably fast to make it look like it performs better than the competition.
29. _Bone_ (Posts: 2061; Member since: 29 Oct 2012)
As already said, the only way you're gonna get a look at real life everyday usage is using it in real life, every day, end of story. :)
Benchmarking was only the fool's tool for that, ppl. who actually know what to do with them want the maximum numbers however, just like ppl. who buy cars to modify it etc. Average Jane shouldn't care much about it let alone express something that's far beyond her understanding, cause, you know she doesn't tune cars.
37. FYoung (Posts: 27; Member since: 08 Oct 2013)
"...the only way you're gonna get a look at real life everyday usage is using it in real life, every day, end of story. "
No, the only way to predict performance in real life for normal people is to use benchmarks that mimic real life (eg rendering web pages, playing a game, enhancing a photo, playing a video) in a standardized way and that cannot be tricked (e.g. by buying the units anonymously and randomly renaming the benchmark's executables and scripts).
Everything else is useless due to subjectivity, bias and anecdota.
27. tasior (Posts: 125; Member since: 04 Nov 2012)
One of previous articles, had source with technical details of cheats. Almost all producers made the optimization, that cuts off power saving system, while running benchmark. Samsung however has gone further. It cut off temperature limits for whole CPU u
28. tasior (Posts: 125; Member since: 04 Nov 2012)
Samsung however has gone further. It cut off temperature limits for whole CPU unit. That means, that if the benchmark took longer (like when You play games), Your phone would crash. That means that such a benchmark doesn't test real computing power, because it allows phone to go to unstable processing power. In my mind this is serious cheating, and Samsung deserves to be banned, for such operations.
PS. Sorry for two posts. It wasn't intended.
31. _Bone_ (Posts: 2061; Member since: 29 Oct 2012)
Sammy already shared and proved that more than a few apps use the full values just like benchmarks. The limitations are the other way around: you are being "cheated off" max frequencies in most cases so your device lasts longer. It's called 3... 2... 1... optimization for efficiency.
From this bullish attitude floating around, we're one step away from some idiot suing Samsung for not getting full 2.26GHz running the Note 3 calendar app. "IT'S ON THE BOX GODDAMNING, I WANT MY 2.3GHZ 100% OF THE TIME WHEN I RUN THE CALENDAR, OR I WANT MY MONAH BACK!!!"
Sammy should just provide them a ROM and when they start whining that their 6,5 screen on time was cut down to 2, they'll shut their mouth is a little understanding how modern technology works.
12. Smokin (Posts: 97; Member since: 10 Oct 2013)
Oh finally great news. I debated this stupid guy on twitter saying that S200 is MORE powerful than S4 Pro and my wifi is effin slow today.
OT : IMO its not cheating because like a bodybuilder, while he's not flexing his body its already great but if he's flexing the body will become even greater and i think the these phones only flex their muscles here so no cheating
18. jroc74 (Posts: 3959; Member since: 30 Dec 2010)
Not only do some companies cheat benchmarks....some benchmarks are easily fooled, tweaked.
Its funny because HTC and Samsung have used custom browsers on their Android phones for years. And always scored high in those benchmarks.
23. jroc74 (Posts: 3959; Member since: 30 Dec 2010)
One is Quadrant. I see benchmarks from that listed and just shake my head.
Years ago, with the lil ole Droid 1....rooting the phone, making a small change to the build.prop (something like a Registry) resulted in higher scores for something. I think it was something to do with not reading a type of media or something.
Also...Samsung went with a different file system years ago and it caused benchmarks to not be as good. Just changing the file system resulted in higher benchmarks. They learned and use the file system that most other Android phones use now.
19. JakeLee (Posts: 372; Member since: 02 Nov 2013)
Sammy's N3 is the undisputed king in this, covering six out of seven benchmarks while most others are "optimized" only for two or three benchmarks.
That's quite a performance!
24. Shatter (Posts: 1763; Member since: 29 May 2013)
Has anyone figured out how to disable the benchmark boosting to get real scores?
38. FYoung (Posts: 27; Member since: 08 Oct 2013)
Well, a simple method that worked to disable the cheating was to rename the executable file of the benchmark. That is how cheating was first proven. Maybe this method still works.
20. jroc74 (Posts: 3959; Member since: 30 Dec 2010)
Oh well....I am a Moto fan, looking at LG lately. For shame HTC and Sammy...for shame.....
Never really liked their phones too much, too many reception issues. If they wanna cheat benchmarks, they should get out the phone business and jump in the PC business.
25. AfterShock (Posts: 1250; Member since: 02 Nov 2012)
Once your crooked ways get out, reputation will make sure no one will ever forget.
I don't think it's normal usage if I can't have access to that power at my demand without rooting.
It's cheating plain and simple.
34. Taters (Posts: 2361; Member since: 28 Jan 2013)
Sure you can. All they would have to do is design a fart app that uses max power for fun. Or play the most graphic intensive game that exists.
Benchmark cheating is the same cheating as graphics demos on the ps4 and xbox that doesn't represent most games. Sure it's possible to make games approaching graphic demos but they never reach that level of polish.
Misleading? Yes, a little. Cheating. f**k no! Graphic demos represent best case scenarios or close to it at the time. Benchmarks do the same and should hardly be considered cheating.
33. Taters (Posts: 2361; Member since: 28 Jan 2013)
Um if the devices can run a benchmark at full power, it can run any app at full power if they programmed it to...... How is that cheating?
Like Bones said, if I floored my McLaren F1 to the max mph that I can whenever a racing sign appears, which is what a benchmark basically is(something to show how powerful my device can be), it doesn't represent my daily performance which would be driving 10 km over the speed limit but it DOES represent what the car can do. And that SHOULD be the whole point of benchmarks.
If I wanted to see what my phone could do on idle, I would just use my judgement. It is easy to see how well it performs day to day. The hard part is seeing how well it can perform at it's theoretical peak.
40. Jaeel (Posts: 20; Member since: 18 Apr 2013)
-phone A, benchmark score 1000, cheats
-phone B, benchmark score 1000, no cheats
Phone B will be faster is everyday use.
41. AfterShock (Posts: 1250; Member since: 02 Nov 2012)
If I can't use the full power, its slower then product A, that is deceitful/cheating.
43. Taters (Posts: 2361; Member since: 28 Jan 2013)
No. That would mean they both perform 1000 at the high end, it says nothing about everyday use. For all we know phone A can cheat on high end but perform better at everyday use.
It is more like phone A benchmark 1000 cheat or not will perform better than benchmark 500 at everyday tasks cheating regardless. If it has the power to cheat it is most likely performs better than a product that can't flex it's cheating muscles even if it wanted to.
It is called a higher ceiling. I suggest you haters learn more about it.
44. AfterShock (Posts: 1250; Member since: 02 Nov 2012)
Try an reword it all you want, its still the same outcome, you never get potential to run hardware at max on anything outside of the white listed apps.
Haters lol, yeah I can't stand deceit by anyone.
Nothing to do with hate dude, just facts.
46. Taters (Posts: 2361; Member since: 28 Jan 2013)
You can easily run a game at that level, dummy....
So I guess if a game runs at a high performance and a calendar app doesn't, the game is cheating? That is the retarded hate logic that is currently coming out of you....
47. AfterShock (Posts: 1250; Member since: 02 Nov 2012)
Name calling, Taters I thought you was better then that.
On subject, thermal throttling is a reality that these OEMs are shirking to show a brief run and not sustained capability.
If you buy into that fine, but most do not and see it for what it is.
39. FYoung (Posts: 27; Member since: 08 Oct 2013)
"Um if the devices can run a benchmark at full power, it can run any app at full power if they programmed it to...... How is that cheating?"
It's cheating precisely because it is untrue that a phone you can buy through normal channels could run any app at that power, not unless you're a hacker who can tweak the operating system. And if you did, the phone would be unstable and would fail if run normally at those settings, and this would void the warranty.
"It is easy to see how well it performs day to day."
No, it isn't, not unless you buy the phone first. The point of the benchmarks is to help buyers decide which phones offer the best performance before they buy.
The point of benchmark cheating is to mislead consumers on how that phone compares to its competitors.
45. Taters (Posts: 2361; Member since: 28 Jan 2013)
It's not untrue though, it can run any app at that power. It just chooses to run a benchmark app at that power. You people are so clueless that you cant see the big picture.
Do you honestly believe that a company will allow any app, even a benchmark to run so high that it risks harming the internals? They know the throttle limits. They won't open themselves up to that chance that joe blow might right run antutu and quadrant all day.
If tech sites can run antutu and quadrant all day without destroying the internals, it is a real indicator of performance, period. You idiots make it sound like if you run these benchmarks all day on these phones they will explode or something. The devices can handle that performance easily so it's not cheating.
If you can run a benchmark app all day on high performance then you can run ANY app all day at that performance. Not every developer will choose to and in fact most don't, but you DEFINITELY can. How is that cheating?
That is like I said, less cheating than graphic demos or those nvidia or Qualcomm reference devices. There is nothing wrong with those. They are just showing you the capabilities and the ceiling.
If you want true cheating and deception, look no further than Apple. They do the equivalent of lifting 1 pound(iOS 7and safari)better than Android lifts 20 pounds ( android and chrome , much heavier) and then claim they have power and idiots like you believe them.
48. AfterShock (Posts: 1250; Member since: 02 Nov 2012)
Face it, you bought the farm and all that goes with it, pastures an manure combined.
Sorry but it stinks, can't cover aroma with more bs.
49. Ninetysix (Posts: 1070; Member since: 08 Oct 2012)
--Run GLBenchmark and get a score of 100.
--Rename GLBenchmark to PoTatersBenchmark and get a score of 75.
50. philfoster (Posts: 34; Member since: 01 Mar 2012)
Still many devices that cheating outside, aren't they?
51. Teja171 (unregistered)
Samsung and all other cheaters, " you are worse than chinese copycats "