Android engineer speaks his mind about OnePlus' benchmark shenanigans

Android engineer speaks his mind about OnePlus' benchmark shenanigans
So, the OnePlus 5 is only a couple of days old and controversies are already stirring up. It has been alleged that the phone "cheats" on benchmarks by enabling a specific profile, which forces the processor to work at high clock speeds. The reason this is considered cheating is because a handset would not operate in this way during normal usage, since prolonged times at high load results in overheating issues.

However, co-founder Carl Pei addressed this issue by stating that this profile is indeed used when the user boots up a game and will also kick in at app launches to reduce loading times. "OK", critics said, "but aren't you exposing the phone to overheating risks that way?". No specific answer has been given yet.

Now, Tim Murray — Google Senior Staff Software Engineer and Android performance lead (mainly working on the Pixel) — has spoken his mind on the issue via a series of tweets. And yes, Mr. Murray is highly critical of the practice of "app detection and frequency ramping". In his words, he has actually spoken to a lot of Android manufacturers and specifically insisted against them doing this... twice.

Mr. Murray's main concerns is that this sort of practice risks battery life and threatens to introduce overheating dangers. The full tweetstorm can be seen below.

Related phones

5
  • Display 5.5" 1080 x 1920 pixels
  • Camera 16 MP / 16 MP front
  • Processor Qualcomm Snapdragon 835, Octa-core, 2457 MHz
  • Storage 128 GB
  • Battery 3300 mAh

FEATURED VIDEO

30 Comments

1. splus

Posts: 148; Member since: Nov 26, 2011

I don't get it - what's wrong if the CPU gets a boost for a limited time? If this happens with ANY resource demanding app then how can this be called cheating? It's cheating if the CPU gets boosted ONLY in benchmark apps, but not if this happens in ALL resource demanding apps.

11. Cat97

Posts: 1804; Member since: Mar 02, 2017

The CPU already gets full boost as long as temperatures permit. The boost in question for the One Plus 5 is given only after the safe operating temperatures are already passed. And no, this is not ok !

18. Lumberjack

Posts: 306; Member since: May 04, 2017

But it only happens when benchmarks are used according to the news.

19. may_czos

Posts: 955; Member since: Nov 22, 2014

In the OnePlus 5 CPU is boosted only in the benchmark apps. It doesn't work that way in any other app or game.

20. Macready

Posts: 1813; Member since: Dec 08, 2014

It's not. It's mainly designed to do it in these benchmarks and not within Qualcomm's recommended frequency guidelines.

24. dimas

Posts: 3324; Member since: Jul 22, 2014

"It's cheating if the CPU gets boosted ONLY in benchmark apps, but not if this happens in ALL resource demanding apps." They boost it in benchmarks during promotions to impress people with numbers that will literally cook their phones in real life usage. If you're gullible to overclockings, you're gonna buy it expecting it to run on such speed then just get disappointed because it's a different story when used everyday. If that's not cheating potential customers, then I have no more arguments to present.

31. Ironboned

Posts: 77; Member since: Jun 16, 2016

It's cheating because the phone does pass hit limit and sets to hot ( 82°) it is not safe and will kill the battery and phone in short time . So if it's a profile to push the COUNTRY to 82° it's not safe and will kill the device in a short time and if it is not a passive profile then IT IS CHEATING . easy to understand .

35. yalokiy

Posts: 923; Member since: Aug 01, 2016

But it's not true. Same benchmark app with different process name runs slower.

3. maherk

Posts: 6750; Member since: Feb 10, 2012

It's not like they're manipulating the numbers, they're simply letting the CPU to run at it's max power, so how is that cheating? Isn't that like the new Chiron having top soeed of 300 mph but only limited to 280 mph? I dislike OnePlus and find them highly overrated as phones and as a company, but I can't see how this is called cheating.

17. KingSam

Posts: 1403; Member since: Mar 13, 2016

Do you guys even know how the cheating works. When it detects a benchmark the little cores' minimum frequency is set to 1.9 ghz effectively locking it at 100%. Big cores do the longing in these scenarios so why force little cores to Max?

4. plsnoregforcomments

Posts: 25; Member since: Mar 29, 2017

just like the way every other smartphone does it... Pretty standard these days and the reason why no major review seems to give any thoughts about benchmarks...

5. MattPerkins1

Posts: 94; Member since: Mar 25, 2017

Android companies are notorious for cheating benchmarks. Samsung has been busted at least 5 times with the Galaxy S. This is nothing new. And the reason they cheat benchmarks is because Android is a far slower operating system than iOS is. So to give the impression it's a level playing field they cheat benchmarks. Notice Apple has never been busted cheating benchmarks. Even Android companies know specs only matter so much when there's such a speed difference between iOS and Android so got to also cheat the benchmarks to make it seem more meaningful. The sad thing in all this is even after all the benchmark cheating by Android companies, the newest iPhone almost always ends up on top anyways.

21. Macready

Posts: 1813; Member since: Dec 08, 2014

Show me all 5 Galaxy S phones that cheated. You can't, because there aren't 5. Guess who was early to the cheating party: your beloved Apple with their Mac's. With the G5 arrival for example, they didn't just cheat their Mac to higher scores in a benchmark comparison, they also purposely crippled the competition in a presentation by Jobs. There goes the "Apple never cheats".

26. magnanimus

Posts: 565; Member since: Mar 29, 2013

The Galaxy Note 3 and the 10.1 have been accussed of cheating in benchmarks before http://bgr.com/2013/11/27/samsung-benchmark-cheating-banned-htc/

28. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

Except they weren't. Its obvious the fools didnt understand what benchmark software is for. Any fool can read what benchmark softhearted does. If your CPU is design to run a 2.1GHz...as long as your CPU doesn't exceed it native set clock, then how is it cheating? ITS NOT.. It is cheating if the test is suppose to be a real world test and you punch the CPU to max which would not be real world.

27. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

You aren't cheating the benchmarks stupid. Benchmarks are designed to push hardware to list limits. So if the frequency limit of the CPU is 1.9GHz and the benchmark app pushes it to 1.9Ghz, then how is that cheating. Cheating would be, the hardware overclocks the CPU above its maximum frequency.. You don't even know WTF you are talkign about which is why you should shutup. No one is cheating. The fools is claiming its cheating because, in the real world and when you use your phone, the device is gonna use its design speedstep-like technology to control the CPU speed to move it up and down based on application demand. Because benchmarks scores and ratings woudl be misleding if peopel think their device will run this way all the time. HERE IS WHY YOU ALL ARE WRONG. First off, 90% of the people who buy phone, could care less or have never seen or will see benchmarks. Benchmark was design by tech enthusiasts, who want to see how powerful hardware can be when ran to it maximum potential without overclocking. And then when they do overclock, they watch the marks to understand how cooling effect performance and to see how cool they need to keep a chip in order to achieve a stable max speed with minimal cooling. The average consumer will never look up or care. On nerds and tech dudes love this stuff. So what customers are goign to be misled? NONE of them. People who buy products based on benchmarks are usually tech nerds and people who need hardware in special group, that need to hit a certain performance metric. This excludes the vast majority of consumers. Its not cheating because benchmarks are not about duplicating everyday usage. Its about pushing hardware to its limits. That's why its not cheating and fool who says it is, should even be posting because you are wrong. This is the same flack Samsung got for making the CPU run full on benchmark. Well its suppose to to begin with.

6. vasra

Posts: 122; Member since: Feb 27, 2014

It's app specific. If the OnePlus hasn't programmed the recognition for your app, then you won't get the speed boost, even though you'd like to get it. So, it's a good thing, but maybe not good thing enough and can give a false impression of overall speed, because it's app specific.

7. pixel_ftw

Posts: 83; Member since: Jun 19, 2017

Google should punish these companies by banning their phones from entering the play store. That should tech them a lesson about cheating. The cheating by these companies also give android and maker of android google a bad name. Take a hard stand against cheating. Whatever the people say about how it's not an issues don't know what they are talking about. I'll always take google engineers advice over some random "tech engineers" who always claims that it's not cheating/it's not an issues.

8. Dr.Phil

Posts: 2315; Member since: Feb 14, 2011

While I wouldn't go so far as to call it cheating, I will say that I don't necessarily see the need for it anymore. Android manufacturers in the high end market for the most part use Snapdragon 800 series chipsets. Their performance is going to be the same except for RAM and software UI differences. But, you're going to get the same result in terms of raw performance. The only reason I look at benchmarks today is to see where a certain chipset falls in performance. In other words, to compare the Snapdragon 821 to say a Snapdragon 660. But I don't use it to compare Phone X with a Snapdragon 835 to Phone Y with the same processor. Benchmarks used to be very useful back when we had multiple big name processors were being used almost equally in the same space. We had Texas Instruments, nVidia Tegra, Qualcomm Snapdragon, and Samsung Exynos all competing in the same space. But now it seems that Qualcomm is being used 85% of the time (not a real number) and I just don't see the need for benchmarks as much anymore.

12. adi90

Posts: 554; Member since: Dec 21, 2015

Maybe you do not compare your phones benchmark performance to other. You are correct. But in countries like china and India, main market for these chinese phones, people do compare major benchmarks before buying a device. The trend is declining. But still people do that. People should also understand that op5 is very close to price of lg g6, mi 6, etc. After few months even htc u11 will sell at almost same price as op5 in these countries. So this practice might mislead some customers. Moreover, as you rightly said. What is the need of it? Can we say that op5 is the only honest manufacturer left on the face of the earth, which care so much about the power of sd835 ? People are not interested in the raw power of sd835. People want to see what it can do in their hand.

15. adi90

Posts: 554; Member since: Dec 21, 2015

Google and other android OEM should come together and analyze the pros and cons of such practice and issue uniform guidelines.

30. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

Yes and no. What Google shoudl do is justy mind its own business. Google laoows OEm to use Android. They arent breaking nay rules. Google make sits won devices. Google has the right to setup tehjir device the way they want. If Samsung, LG, OnePlus or whoever wants to use Android another way with their hardware, then they can. These company's have a right to make their hardware do whatever they want. If you buy a car, you can charge whatever you want as long as what you do is street legal. If your car is no longer street legal, then whatever you did, take it to a track where it is legal. But you have the right to modify your vehicle however you want as long as it doesn't break any laws. When my car is on the street, I keep it at legal; limits. When I hit a track, I get under the hood, make some changes and now its more for track vs street. I'm not cheating you unless I do something to the car that exceeds restrictions like in a real sponsored race.

29. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

Just because peopel use benchmarks, doesn't mean this is cheating. First off, if anyone uses benchmarks, then they have an undertsanding as to what tyhey are lookign for. If you asked such persons if this was cheating, they would say know. Maybe such buyers will unlock the device so they can overclock the CPU or run it at full tilt when they want too. But question, pl;ease list the names of people you know who buy devices over benchmark scores....even if you could, don't you think they understand what benches are for? Anyone who uses benchmarks, understand that benchmarks scores are first and foremost, theoretic and are not based on any daily expected performance. They understand that these apps are designed to push hardware to a certain maximum, and that the results are not expect daily numbers. The only fools with an issue are fools who claim anyone would be expecting a device to perform this way all day everyday.

9. bencozzy

Posts: 24; Member since: Oct 29, 2012

Who cares? Honestly if I get the OnePlus 5 it will have TWRP, ROOT, ROM, and KERNEL the first day. And that's all done to give best performance and battery life. So it doesn't matter to me what OnePlus does with Oxygen OS as long as they continue to release source code. Same goes for AOSP pixels are lady without a custom ROM and kernel.

10. sunnyfpy

Posts: 277; Member since: May 12, 2013

Turbo charged !

13. L0n3n1nja

Posts: 1527; Member since: Jul 12, 2016

On a side note, the "Never Settle" slogan makes me want to buy a real flagship instead of settling for this imitation.

23. TeriusRose

Posts: 108; Member since: May 12, 2017

While I can't necessarily disparage one plus over their slogan, at the same time I struggle to think of a real reason to buy one when I could purchase an S8 or the upcoming G6 Plus.

16. trojan_horse

Posts: 5868; Member since: May 06, 2016

I think some rules needs to be established in regards to benchmarks, so that OEMs who are found guilty of cheating (depending on the rules in question) should be sanctioned. Don't know if Google could pull this off.

32. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

What rules? Its a benchmark. There are 2 types of benches...real world and not real. Real world are tests that give you some idea how a device will perform based on everyday usage. Then you have benchmarks tests that are designed to push hardware to its physical limit to understand its potential vs another. So if your CPU clock has a max limit of 2.1Ghz, and you are using this benchmark for full performance testing, then you woudl expect the software will push the CPU to 2.1Ghz. If it pushes it to 2.4Ghz, then that woudl be cheating because it is outside the maximum set limit of the hardware. Again the type of benchmark already set the limit. You don't get to set another and this is what you all aren't getting. A non real test would be allowing the chip set to behave how iot would through natural use. So malign the CPU run full tilt is not real world and that would be cheating.

25. TechieXP1969

Posts: 14967; Member since: Sep 25, 2013

It does not matter what Google thinks will happen. It doesn't matter if they ask manufacturers to not do this as a suggestion. If an OEM can pull it off without the cons of battery-life or overheating, then dont get mad if it works. Is anyone complaining the OnePlus5 is overheating or has excessive battery drain? NO! Just beating a dead horse. They didn't cheat. They simply ran the hardware at full tilt. It doesn't matter that it wont run this fast all the time. Just like exotics can do more than 200 MPH. Just because you wont do 200MPH all the time, doesn't mean you never will and yes even if I dont drive 200mph, I wanna see what its like. I got to drive a Lambo for the first time at Disney and I managed to get to 150mph before I got to scared. Me and the more experience driver switched and he drove up to 190MPH. I was scared, but it was also a thrill at the same time. But if I was to buy a similar car, just because I wont even drive 150 all the time, doesn't mean I never will and its nice to know A, how long it would take to reach that speed, B learn how to control tp and C, enjoy the benefits of it. The phone is still runnign within the limitations of its design. Its not gonna run full tilt all the time. Because when ther chip heats up, the core speed is gonna drop until it cools. DUH, running a chip at full speed drains the battery...SO WHAT! That's why it has a fast charging system! All the haters can drop dead. Chips are design to run at full tilt. Yes they darin more battery, yes without active cooling they will get hot. When they do, they will slow down in order too cool and dash charge will replenish the battery. I I like that Samsung allows me to run my games at a full 60FPS all the time. Yes the phoen gets ot, yes that emans it runs at a very high clock, and yes it drains the battery. But for an 1 hour or 2 of supped up gameplay I love it. Any fool who speaks against it is a fool. Use your phone how you want. Dont ruin it for others who don't want to be conservative. I paid $9800+ for my phone. There are times I want to push it to its limits. Do with your phone whatever you like...you bought it.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.