Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests

Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests

Like fashion, the devices in our pockets have — kind of inexplicably — grown into a sort of public statement. You buy iPhone, I buy Android, therefore you and I have fundamentally different views of the world. Reconciliation of those two... belief systems, if you will, is improbable at best. But for the most part, it just doesn't seem to occur naturally very often. Even among non-techies, preferences are so firmly lodged that people tend to identify themselves with brands instead of the more sensible, reversed scenario. And that's all kinds of problematic.

Problematic because it often means that objective feedback is swept under the carpet in what is increasingly looking like a game of only winners and losers and no middle ground. Frank discussion is, therefore, a pipe dream, as the opposing camps are in on a too personal a level to allow for it. This benefits phone makers, not consumers, as the former are allowed a bit too much slack when it comes to solving the big problems plaguing their product lines. Like fragmentation in Android world.

The "F" word


4 charts that perfectly exemplify the state of fragmentation in Android. (1) Shows version distribution, (2) and (3) give us the various screen configurations, and (4) breaks down the usage of OpenGL ES API.

4 charts that perfectly exemplify the state of fragmentation in Android. (1) Shows version distribution, (2) and (3) give us the various screen configurations, and (4) breaks down the usage of OpenGL ES API.

Put plainly, we say fragmentation when we think of Android's rather unsurprising — but still problematic — diversity in hardware and software. Compared to the Apple camp, which only has to worry about supporting a handful of devices, third-party Android developers, even OEM engineers, have quite the task in front of them when it comes to optimizing for performance. And since it's a task too great for any one company to tackle on its own, the net result is inferior performance. 

In fact, a new, comprehensive study by the fine folk over at GameBench — a company dedicated to benchmarking real-world mobile app and game performance instead of arbitrary synthetic loads — delivers some much needed data corroborating this. I also had the opportunity to talk with the team at the company to get some clarity on the methodology and their insight.

What is to follow is a bit on the technical side — though only a bit — and so here's the obligatory 'executive summary'. GameBench works by counting the frame rate an Android or iOS device is capable of delivering within apps and games. I won't get into too many details regarding frame rates and why it's the preferred (and objective) method of benchmarking for performance, but you can think of it this way: The more frames per second your device is capable of producing, the smoother events such as transitions, enemy fire, pane switching, etc. will appear. GameBench also tracks frame rate stability, which is expressed in percents, and denotes the time spent at median FPS (the most common frame rate), and the lowest observed frame rate. These are not unimportant since a low frame rate stability detracts from the user experience, as your eye will be drawn to the event when the picture goes from smooth to choppy and back again.

Since the idea of the study is to benchmark performance across the two leading operating systems, the methodology behind the report includes a basket of 10 popular game titles, all based on the cross-platform Unity game engine. Each of the games was played for a period of 15 minutes, with the same graphical settings where configurable, and on the same levels wherever possible, all the while frame rates were logged. Unlike similar previous tests, the games tested here all ran at, or at close to, their native display resolution, so 750p for the iPhone 5s, 6, and 6s, ~1080p for the iPhone 6s Plus, HTC One M9, Huawei Mate 7, and Xiaomi Mi Note, and 1440p for the Samsung Galaxy S6, LG G4, and Meizu MX4 Pro.

Key findings


If, like myself, you're an Android user, it's unlikely that you'll like hearing about the key takeaways of the study. In short, the iPhone camp is having a field day with its Android competitors, with even the 2-year old iPhone 5s managing an incredible showing despite its aging hardware. In fact, between the 6 Android devices included in the test, only Samsung's Galaxy S6 was capable of at least 25 frames per second (FPS) median in each of the games tested. To add some perspective, the gaming community is mostly in agreement that 30 FPS is the preferred minimum, dropping under which is considered problematic. As a previously active gamer myself, I definitely agree and would even go as far as to say that gaming in 2015 should be done at no less than 60 FPS wherever reasonable.

Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests

Why 'reasonable'? Well, depending on the genre of the game, anything over 30 FPS can well be considered an overkill. Turn-based games, for example, don't require as much fluidity in order not to impact the gaming experience, and so developers behind a few of the tested titles here have actually limited their games to just 30 FPS in order to save battery. Quite frankly, this is the main reason why the iPhone 6 and iPhone 6 Plus ended up higher on the top-level averages table above — one title (Lara Croft: Go) was capped for an inexplicable reason at just 30 FPS for the 6s and 6s Plus, but not the 6 and 6 Plus. Furthermore, with 4 out of 10 games being capped at 30FPS, there was simply no room for the iPhone 6s and 6s Plus to flex their silicon muscle, as their respective predecessors were already delivering at/near the cap no problem. 

Another incredibly important metric, as mentioned, is FPS stability, where the entire crop of iPhones fared best, followed by the Samsung Galaxy S6 and HTC One M9. You can expect these devices to deliver the most stutter-free gameplay of the tested lineup. A higher FPS stability indicates that the device is having comparatively little issue running the content, while the opposite holds true when the stability is low. While I don't have the required data at hand to make any serious claim about this, it is also possible that devices with higher frame rate stability are potentially drawing less power as the GPU is obviously having an easier time keeping up.

Wrench icon indicates these devices faced compatibility issues and couldn't run all the titles from the test basket

Wrench icon indicates these devices faced compatibility issues and couldn't run all the titles from the test basket


The reasons for these results are several, and some of them pretty darn complex, and I'll go over them in a minute — and yes, including the question I'm sure is on many currently ticked off Android fans: resolution. Before I address these and talk about the insight I gained in the process of working on this piece, however, let's get the rest of the test results out of the way. Here's how each of the devices on the list performed in every game in the test basket:


The resolution fallacy


You don't have to be much of a techie to figure out that the more pixels a processor has to push, the harder it will be for it to deliver high frame rates. In this respect, devices like the 1440p Galaxy S6, LG G4, and Meizu MX4 Pro are at a disadvantage when compared to the 750p iPhone 5s, 6, and 6s, and even the 1080p iPhone 6s and iPhone 6s Plus. For a fair comparison, one would think that only the iPhone 6 Plus and 6s Plus need to be tested against the 1080p HTC One M9, Huawei Mate 7, and Xiaomi Mi Note. In a simpler world, that would be the case, but that's only half the story here.

Obviously, the Galaxy S6's Exynos 7420 processor has more work to do at any one moment than the A9 chip of the iPhone 6s, as it has many times more pixels to take care of. Even disregarding the futility of packing so many pixels into what is essentially a tiny screen, however, there's more to consider. It's what I've taken to referring to as the resolution fallacy.

There are a few components to this fallacy, and all of them make perfect sense if you sit your preconceptions and think about it. First off, the vast majority of gaming titles offer comparatively rudimentary textures and animations that benefit little from extra clarity (resolution). There's simply no more information to be gleaned — it just looks sharper when you zoom in at, say 300% in an image editor. All these extra pixels, then, are actually pretty inconsequential as they don't actually improve the gaming experience. What's even worse, in fact, is that these resolutions essentially impose a tax on Android gamers, as developers are downgrading graphics to Medium by default in many cases — and there isn't always an option to set it to high. I'll come back to this one and try to offer some more depth as to why that is, but the key takeaway here is that, ironically, a higher resolution screen on an Android device could actually downgrade instead of upgrade the quality of the image. From what the people at GameBench tell me, this isn't a rare scenario, and is not how things are on iOS, where developers default their games to 'High' graphics on the tested devices.

Even assuming that you can set graphics to high at the expense of performance in some Android titles — like you do with Unkilled and Hitman: Sniper — the above argument about the simplistic nature of textures still holds with full strength. At that scale, such tiny improvements in clarity are imperceptible in the real world. Here are two examples:

Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests
Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests
Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests
Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests

Are the lines less jagged on the higher resolution devices? Absolutely. Do they reveal more information, however? Not at all — it's the same, plain texture. Just smoother. But most importantly, this is an incredible magnification of the screen image — you don't actually see these imperfections when gaming on a lower-res screen because they're too far out. Our eyes simply aren't that good.

Why? Fragmentation, of course


As a sort of a build-up to the report, GameBench tells me they held an event in London for game developers, game publishers, phone manufacturers, and platform holders. Reps from across the industry all had a similar take on the state of Android game development: there is just way too much variety in software/hardware configurations for a small — even medium — sized dev teams to properly and confidently support their titles and ensure great performance. At the same time, the limited iOS hardware is a boon, as pretty much any developer can afford owning most or all devices that will run his or her software. 

As importantly, Apple's low-level API, Metal, was mentioned as a defining factor in allowing iOS devs to deliver great performance across the board. Metal's Android equivalent is called Vulkan, but it's still not a market reality. Vulkan could introduce significant advantages to Android development, as it will significantly limit abstraction, which is simply the disconnect between developer code and how it's intended, and how the hardware reads it. The more hardware and software combinations available on the market, the bigger the chance of a miscommunication with this or that device. That's among the bigger issues with Android development right now, and a main reason for gaming studios' almost religious entanglement with iOS and their shunning of Android.

All of this paints a rather gloomy picture of Android, but in closing, I'd like to go on record and point out that despite inferior performance, I think fragmentation was worth it from the point of view of consumers. It's what allowed for smartphones to essentially turn into a commodity and it's what makes $200 devices that don't suck a reality. It's also what will continue to drive prices down until total ubiquity. But there's a price to pay, and I for one think the Android community needs to stop ignoring it.

references: GameBench

FEATURED VIDEO

67 Comments

1. zeeBomb

Posts: 2318; Member since: Aug 14, 2014

Tsk tsk....

14. XperiaFanZone

Posts: 2277; Member since: Sep 21, 2012

21. Finalflash

Posts: 4062; Member since: Jul 23, 2013

The title was click baity as hell and the rest of the article was just as sad in terms of critical thinking. This article was a bit dumb seeing as how they cherry picked as much as they could to make the iPhone look better than it was. Compared directly with the S6, the only phone not being crippled by a qualcomm chipset this generation, you can see that while running 80% more pixels than the iPhone 6/s plus and about 250% more pixels than an iPhone 6/s, the S6 averages about 80% frame stability. Where as the iPhone 6/s/pluses only average about 10-20% better frame stability comparatively. The same situation is present with the FPS with the iPhones only doing 10-20% better than the S6 even running almost half the resolution or 1/4th the resolution (6/s plus and 6/s respectively). That trade off is totally worth it especially since you can lower the resolution on Samsung phones and any hardcore gamer will know how to google that at least. This is a fairly laughable article that showed nothing but the fact that iPhoneArena doesn't know how to analyze data objectively and that iPhones are neither future proof nor are they the equivalent of hardware from 6 months ago. Also, this whole article speaks to the greatest part about fragmentation which is that Samsung, followed their own path and were not crippled by qualcomms stupidity. Without it, all flagships would have been crippled this generation. The fragmentation in hardware allows Android to have better cameras, hardware, software in almost every respect. Maybe not all in the same device, but more so than the iOS equivalent, which now holds the lead in almost nothing generation to generation equivalents compared. Also, if metal is that great, t

24. Finalflash

Posts: 4062; Member since: Jul 23, 2013

damn, got cut off. Anyway, all that metal praise is in contrast to the anandtech comparison seeing as how it maximum offered another 2-10% gain at best. Even with that, they couldn't outscore the S6 by more than 10-20% running half or quarter the number of pixels depending on the iDevice. http://www.anandtech.com/show/9223/gfxbench-3-metal-ios

30. Chris.P

Posts: 567; Member since: Jun 27, 2013

I'll offer two comments here. First off, due to the highlighted, comparatively gigantic leaps in GPU performance relative to CPU performance, the need for a low-level API will only grow in the future. So Vulkan — and therefore Metal — is more important than you make it out to be. Even if the gains in performance are limited to 10%, that's still a a lot RIGHT NOW, when you consider that you're running the same hardware. More importantly, however, Metal was only mentioned in order to introduce Vulkan, which is going to be crucial for future Android development. Right now, devs are limited to using a higher-level API (OpenGL), which still requires a lot of extra effort and skill to optimize. Now factor in the smorgasbord of Android devices, and you can see why many games are iOS only, and many more only come out to Android after a ton of wait. So the utility of Metal is not just limited to performance, but also ease of use. With that in mind, I suggest you read through the comments of actual developers, available in the report linked at the bottom.

28. Chris.P

Posts: 567; Member since: Jun 27, 2013

@Finalflash It's a commentary on the challenges that fragmentation poses for Android developers and real world performance. As in, what you actually get from these devices in reality. On a 5.1-inch screen, all the pixels in the world will NOT change your gaming experience, simply because textures are super basic, the screen is tiny, and so jumps in pixel count are of questionable benefit and of certain disadvantage — especially going forward, where developers will be forced to stick with simpler graphics just to accommodate QHD and 4K screens. So they will continue to set graphics to medium by default. It's probably why Samsung allows you to downgrade resolution. Also, do keep in mind that in games like Lara Croft: Go, the 's' series iPhones were capped at 30FPS, dragging down the average. And by the way, you're playing down the importance of FPS stability. It's crucial, since we're looking at median frame rates, not averages. Finally, it's not a chip vs chip comparison. Also, the Galaxy S6 is just -one- device out of hundreds of high-ends in Android land. Even the HTC One M9 is something of an outlier, for HTC devices have always fared great in graphics tests historically, despite sharing hardware with others. The rest? It ain't pretty, even when comparing the 1080p 6s Plus to a 1080p Android.

34. Unordinary unregistered

Why do people make fun of you and your "bias" articals? I think youre a brilliant author and a well educated techie. Cheers, enjoyed reading comment #30 and 28, as well as the entire article and not just the graphs..

41. Finalflash

Posts: 4062; Member since: Jul 23, 2013

For the first point of "what you get?": You get a higher resolution screen, which allows you to watch 2k videos which are becoming very common on good youtube channels. You get better future proofing with the upcoming VR tech. What you do lose is about 10-20% performance in high end games, which only a small subset of mobile gamers actually play and even that for short periods of time. Most casual mobile games don't require anywhere near that level of performance and for the an equivalent cost, one would rather have the option to have a higher resolution and downscale than no option at all. Second point on FPS stability: It is important, but once again, what is the margin of difference and is it worth it. The iPhone 6 plus had the same overall stability as the S6 having half the resolution. If toggling the S6 to 1080p or lower was tested, then it would have easily outpaced its iOS counterparts. THAT option is available and anyone playing these higher end games will know of it. The key point here being that the best of both worlds is available at the same cost (if not lower) as the competition. Also, I do not know how you can say it isn't a chip to chip comparison when pretty much all of the same chipped devices with a similar resolution scored pretty much exactly the same. The snapdragon 810 devices all scored pretty much neck and neck with each other especially in FR stability. All this goes just to show that counter to the implication of the title and theme of this article, fragmentation is the best thing that ever happened for Android. For a minor 10-20% degradation in high end gaming capacity at unequal resolutions (biased towards the iPhones in most cases), you get so much more. Better screen resolutions. better screen technologies, more features, better cameras, better battery life, different form factors, different prices, etc. Anyone with an S6 will enjoy better gaming on it by adjusting the resolution than probably all of the iPhones to date while having the best screen tech and resolution. Anyone with an S6 or Nexus 6P will enjoy better camera performance than all of the iOS devices listed here. Anyone with a need for a small long lasting device could get a Z5 compact or an LG G4 that want a phablet with a replaceable battery. What I want to know is why, in the mobile world, cutting off features that don't cost anything is considered an advantage (usually only when it comes to iPhones)? If I chop my arms off, my body will be more efficient energy wise since it no longer has to feed 2 arms. But then I have no arms. That isn't an advantage, it doesn't make the whole device more efficient and if you have an option between the two (maybe) at a minor cost, how is that better than no option at all.

47. Unordinary unregistered

News flash, look at all the OFF SCREEN tests. Whats your excuse for most Android flagships being rekt on that one?

64. buccob

Posts: 2945; Member since: Jun 19, 2012

Even then, he is still right about everything he stated... only a loss in 10% of performance when playing high end games..... But smartphones are so much more than handheld gaming devices... and the iPhone is not the top dog on everything else... pretty much the opposite...

48. Macready

Posts: 1793; Member since: Dec 08, 2014

Nonsense, you can run the S6 at 1080p using the dedicated Samsung app and get higher framerates similar to those of a 6S plus. Apples vs Apples. The S6 user is given the choice. Choice is an evil word for some though, who rather replace it with fragmentation.

54. BattleBrat

Posts: 1476; Member since: Oct 26, 2011

The high resolution screens are for Virtual Reality, which sucks on an iPhone. I have a 5s and just ordered a 6s plus.

55. cheetah2k

Posts: 2146; Member since: Jan 16, 2011

Its still not apples for apples. iPhones have low res screens, and Android phones high res screens.. Why is the Sony Z5 with its 1080p screen in the mix, as well as the Note 5. Zero credibility PA. zero.

31. Unordinary unregistered

So, now there are excuses for packing in all these "great" features like a QHD screen and all that pizzaz into a phone, and exemptions for them NOT finding a way to keep speed and performance at the top of their priority, or even just valued imporatance? LMFAO you Samsung/Android fanboys are a joke. Lets just keep saying "THEY PUSH 23,498,572,309,473,838,754 KAZILLION MORE PIXELS SO ITS FINE THAT THEY ARE BEHIND!!!!!!!!!" You remind me of the skit on The Amanda Show.. BlockBLISTER! NOT BUSTER!. MOAR PIXELS BETTTURR MUHCHA BETTAAAAAAA!!!! Hypocrisy and idiocy at its finest.

44. NoToFanboys

Posts: 3231; Member since: Oct 03, 2015

Do not talk about the fandroid's hypocrisy when you yourself are frequently spewing apple hypocrisy. You are the same as them, just on a different side.

46. Unordinary unregistered

I dont sit here for 3 hours typing a 16 page essay to defend a simple issue LOL. I comment on what I think is funny, and make short simple comments. You guys just get way to offended, Ordinary especially lol. Ive gotten an FBI/NSA type of investigation going on from him lmfao. Back to the point. Most Android flagships are a joke in terms of screen FPS. If you have to make excuses (see #31) because you know the issue is real, then refer to OFF SCREEN tests where pixels dont matter. iPhone still rocks the socks off most devices. Which is sad, because I want myself an S7, and am hoping they work up to the standard Apple has in terms of performance :)

50. NoToFanboys

Posts: 3231; Member since: Oct 03, 2015

While I would agree that those essays are annoying, only XP does that. I am not offended, you are the one who is offended, you even named your account against Ordinary lol, that's a different level of butthurt. I am not going to defend Android, as I'm not a fandroid as you think. It's just that you haters and fanboys from BOTH sides are annoying.

63. Msean.owensby

Posts: 1; Member since: Dec 16, 2015

Is that your only Apple defense? Frames per second? LOL! Yea, because the difference between 55 FPS and 45 FPS is STAGGERING! Psyche. Any difference it makes is miniscule. I've handled an iPhone 6+ side by side my LG G4 and noticed no difference whatsoever. Cause this is the real world. Not a benchmark test. And in the real world, my QHD display, camera and OS reign supreme. Like, burrito supreme :)

65. downphoenix

Posts: 3165; Member since: Jun 19, 2010

I thnk the main problem is that even though mobile phones are like PCs and there are a wide number of configurations, development setups are more akin to console development. Thus devices that do have advantages over iphones in some regards, like resolution, will have their advantages effectively muted.

2. HugoBarraCyanogenmod

Posts: 1408; Member since: Jul 06, 2014

Very informative article as ever, well done PA

4. Derekjeter

Posts: 1440; Member since: Oct 27, 2011

Didn't you guys put out this article like three weeks ago? I think you did but since you had over 200 idiot fans come in and talk sh!t about Apple/Android I would see why you had to re-post.

5. TyrionLannister unregistered

it's always been like this and will always be. A closed walled garden and vertical integration will lead to better performance. If there is a reason for a game like angry birds 2 to run below 60 fps on flagships, it has all to do with optimization. Then again, they could have made the GS6 run at 720p or 1080p as the Samsung game tuner allows to do that for any game. That would help in further determining if it really helps in gameplay or not. If you are looking for serious gaming on a handheld( which you shouldn't apart from casual games), simply get an iOS device. My s6 handles games exceptionally well. But games look better on my air 2. The iPad also lasts longer while gaming ( duh!) and hence its the ultimate casual gaming handheld for when you are bored while on travel or just in general.

6. Kumar123 unregistered

Google should seriously do something about their update policy otherwise the fragmentation will put serious threat in the android ecosystem. Android smartphone always I mean always ruin their performance by putting higher resolution screen even if the hardware can't keep up with it. If any device put 1080p resolution display and SD820 that would be an amazing performance device. But I don't think any android manufacturer will be dared to put 1080p resolution in 2015 flagship phones because android fans thinks, More resolution = Better phone. They hardly care about real world performance but they are concern with superficial benchmark score. Such a pity.

8. wando77

Posts: 1166; Member since: Aug 23, 2012

More resolution does make for a better phone. All you guys that think hi res screens aren't necessary obviously haven't tried out a vr headset yet. Once you have been demoed these for gaming and watching movies you will see they are the future. I have a gear vr for my s6 edge+ and I had no idea how good they were. Watching movies feels like you are REALLY in a cinema watching a 70ft screen and you can look around at the seats next to you. And gaming has so much potential. You simply won't be able to do this on the iPhone with its 720p screen. Even the s6 has the bare minimum for a vr headset. Bring on the 11k screens that Sammy are developing. Mark my words vr is the way gaming is heading

15. elitewolverine

Posts: 5192; Member since: Oct 28, 2013

Why would I want to completely enclose my head? A few times? Maybe...but AR is vastly superior to an experience imo than any VR can deliver. You cant even move in VR...(well very little)

18. RELAXyougeeks

Posts: 24; Member since: Apr 07, 2015

Your comment is shortsighted. All that oculus would have to do would be to combine movement sensing tech (think a better version of XBOX Kinect) with the VR and then... well picture a fighting game where your punches and kicks register in the game. Or COR with a special controller where it could sense aiming, ducking and strafing. The possibilities are only as limited as your your space being safe to move around in.

52. joey_sfb

Posts: 6794; Member since: Mar 29, 2012

Because VR is cheaply and widely available for those that bother. AR is not accessible to me at this point of time. Would be interested in AR if i can get my hands on one. I also agree that Google should do better with software update, a different software update system must be devise moving forward. This is where both Apple and Microsoft Windows do well.

53. joey_sfb

Posts: 6794; Member since: Mar 29, 2012

One way for Google to do that is to take back software update control from OEM that deploy the Google API version of Android meaning those that have official Playstore service installed. If Microsoft Windows desktop could do this for the past 20 years, I don't why Google can't do it, I think its technically possible to at least push security update as the first milestone.

67. elitewolverine

Posts: 5192; Member since: Oct 28, 2013

Because of their policy and open source. There is a limit to how much they can close it off. It is why they are trying to make everything a service to close off what they can. But in the end, if google shuts them out, the carriers; who give code by the hundreds in some cases. Would loose out. Because they can take the open code and do what they want, ala amazon store stepping in. The Open Handset Alliance would certainly have their say to.

10. VyktorJonas

Posts: 40; Member since: Mar 26, 2014

"They hardly care about real world performance but they are concern with superficial benchmark score" agreed 10000%

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.