Fragmentation is alive and well: Apple's iPhones blaze competing Androids in comprehensive performance tests

Like fashion, the devices in our pockets have — kind of inexplicably — grown into a sort of public statement. You buy iPhone, I buy Android, therefore you and I have fundamentally different views of the world. Reconciliation of those two... belief systems, if you will, is improbable at best. But for the most part, it just doesn't seem to occur naturally very often. Even among non-techies, preferences are so firmly lodged that people tend to identify themselves with brands instead of the more sensible, reversed scenario. And that's all kinds of problematic.

Problematic because it often means that objective feedback is swept under the carpet in what is increasingly looking like a game of only winners and losers and no middle ground. Frank discussion is, therefore, a pipe dream, as the opposing camps are in on a too personal a level to allow for it. This benefits phone makers, not consumers, as the former are allowed a bit too much slack when it comes to solving the big problems plaguing their product lines. Like fragmentation in Android world.

Put plainly, we say fragmentation when we think of Android's rather unsurprising — but still problematic — diversity in hardware and software. Compared to the Apple camp, which only has to worry about supporting a handful of devices, third-party Android developers, even OEM engineers, have quite the task in front of them when it comes to optimizing for performance. And since it's a task too great for any one company to tackle on its own, the net result is inferior performance. 

In fact, a new, comprehensive study by the fine folk over at GameBench — a company dedicated to benchmarking real-world mobile app and game performance instead of arbitrary synthetic loads — delivers some much needed data corroborating this. I also had the opportunity to talk with the team at the company to get some clarity on the methodology and their insight.

What is to follow is a bit on the technical side — though only a bit — and so here's the obligatory 'executive summary'. GameBench works by counting the frame rate an Android or iOS device is capable of delivering within apps and games. I won't get into too many details regarding frame rates and why it's the preferred (and objective) method of benchmarking for performance, but you can think of it this way: The more frames per second your device is capable of producing, the smoother events such as transitions, enemy fire, pane switching, etc. will appear. GameBench also tracks frame rate stability, which is expressed in percents, and denotes the time spent at median FPS (the most common frame rate), and the lowest observed frame rate. These are not unimportant since a low frame rate stability detracts from the user experience, as your eye will be drawn to the event when the picture goes from smooth to choppy and back again.

Since the idea of the study is to benchmark performance across the two leading operating systems, the methodology behind the report includes a basket of 10 popular game titles, all based on the cross-platform Unity game engine. Each of the games was played for a period of 15 minutes, with the same graphical settings where configurable, and on the same levels wherever possible, all the while frame rates were logged. Unlike similar previous tests, the games tested here all ran at, or at close to, their native display resolution, so 750p for the iPhone 5s, 6, and 6s, ~1080p for the iPhone 6s Plus, HTC One M9, Huawei Mate 7, and Xiaomi Mi Note, and 1440p for the Samsung Galaxy S6, LG G4, and Meizu MX4 Pro.

If, like myself, you're an Android user, it's unlikely that you'll like hearing about the key takeaways of the study. In short, the iPhone camp is having a field day with its Android competitors, with even the 2-year old iPhone 5s managing an incredible showing despite its aging hardware. In fact, between the 6 Android devices included in the test, only Samsung's Galaxy S6 was capable of at least 25 frames per second (FPS) median in each of the games tested. To add some perspective, the gaming community is mostly in agreement that 30 FPS is the preferred minimum, dropping under which is considered problematic. As a previously active gamer myself, I definitely agree and would even go as far as to say that gaming in 2015 should be done at no less than 60 FPS wherever reasonable.

Why 'reasonable'? Well, depending on the genre of the game, anything over 30 FPS can well be considered an overkill. Turn-based games, for example, don't require as much fluidity in order not to impact the gaming experience, and so developers behind a few of the tested titles here have actually limited their games to just 30 FPS in order to save battery. Quite frankly, this is the main reason why the iPhone 6 and iPhone 6 Plus ended up higher on the top-level averages table above — one title (Lara Croft: Go) was capped for an inexplicable reason at just 30 FPS for the 6s and 6s Plus, but not the 6 and 6 Plus. Furthermore, with 4 out of 10 games being capped at 30FPS, there was simply no room for the iPhone 6s and 6s Plus to flex their silicon muscle, as their respective predecessors were already delivering at/near the cap no problem. 

Another incredibly important metric, as mentioned, is FPS stability, where the entire crop of iPhones fared best, followed by the Samsung Galaxy S6 and HTC One M9. You can expect these devices to deliver the most stutter-free gameplay of the tested lineup. A higher FPS stability indicates that the device is having comparatively little issue running the content, while the opposite holds true when the stability is low. While I don't have the required data at hand to make any serious claim about this, it is also possible that devices with higher frame rate stability are potentially drawing less power as the GPU is obviously having an easier time keeping up.

The reasons for these results are several, and some of them pretty darn complex, and I'll go over them in a minute — and yes, including the question I'm sure is on many currently ticked off Android fans: resolution. Before I address these and talk about the insight I gained in the process of working on this piece, however, let's get the rest of the test results out of the way. Here's how each of the devices on the list performed in every game in the test basket:

You don't have to be much of a techie to figure out that the more pixels a processor has to push, the harder it will be for it to deliver high frame rates. In this respect, devices like the 1440p Galaxy S6, LG G4, and Meizu MX4 Pro are at a disadvantage when compared to the 750p iPhone 5s, 6, and 6s, and even the 1080p iPhone 6s and iPhone 6s Plus. For a fair comparison, one would think that only the iPhone 6 Plus and 6s Plus need to be tested against the 1080p HTC One M9, Huawei Mate 7, and Xiaomi Mi Note. In a simpler world, that would be the case, but that's only half the story here.

Obviously, the Galaxy S6's Exynos 7420 processor has more work to do at any one moment than the A9 chip of the iPhone 6s, as it has many times more pixels to take care of. Even disregarding the futility of packing so many pixels into what is essentially a tiny screen, however, there's more to consider. It's what I've taken to referring to as the resolution fallacy.

There are a few components to this fallacy, and all of them make perfect sense if you sit your preconceptions and think about it. First off, the vast majority of gaming titles offer comparatively rudimentary textures and animations that benefit little from extra clarity (resolution). There's simply no more information to be gleaned — it just looks sharper when you zoom in at, say 300% in an image editor. All these extra pixels, then, are actually pretty inconsequential as they don't actually improve the gaming experience. What's even worse, in fact, is that these resolutions essentially impose a tax on Android gamers, as developers are downgrading graphics to Medium by default in many cases — and there isn't always an option to set it to high. I'll come back to this one and try to offer some more depth as to why that is, but the key takeaway here is that, ironically, a higher resolution screen on an Android device could actually downgrade instead of upgrade the quality of the image. From what the people at GameBench tell me, this isn't a rare scenario, and is not how things are on iOS, where developers default their games to 'High' graphics on the tested devices.

Even assuming that you can set graphics to high at the expense of performance in some Android titles — like you do with Unkilled and Hitman: Sniper — the above argument about the simplistic nature of textures still holds with full strength. At that scale, such tiny improvements in clarity are imperceptible in the real world. Here are two examples:

Are the lines less jagged on the higher resolution devices? Absolutely. Do they reveal more information, however? Not at all — it's the same, plain texture. Just smoother. But most importantly, this is an incredible magnification of the screen image — you don't actually see these imperfections when gaming on a lower-res screen because they're too far out. Our eyes simply aren't that good.

As a sort of a build-up to the report, GameBench tells me they held an event in London for game developers, game publishers, phone manufacturers, and platform holders. Reps from across the industry all had a similar take on the state of Android game development: there is just way too much variety in software/hardware configurations for a small — even medium — sized dev teams to properly and confidently support their titles and ensure great performance. At the same time, the limited iOS hardware is a boon, as pretty much any developer can afford owning most or all devices that will run his or her software. 

As importantly, Apple's low-level API, Metal, was mentioned as a defining factor in allowing iOS devs to deliver great performance across the board. Metal's Android equivalent is called Vulkan, but it's still not a market reality. Vulkan could introduce significant advantages to Android development, as it will significantly limit abstraction, which is simply the disconnect between developer code and how it's intended, and how the hardware reads it. The more hardware and software combinations available on the market, the bigger the chance of a miscommunication with this or that device. That's among the bigger issues with Android development right now, and a main reason for gaming studios' almost religious entanglement with iOS and their shunning of Android.

All of this paints a rather gloomy picture of Android, but in closing, I'd like to go on record and point out that despite inferior performance, I think fragmentation was worth it from the point of view of consumers. It's what allowed for smartphones to essentially turn into a commodity and it's what makes $200 devices that don't suck a reality. It's also what will continue to drive prices down until total ubiquity. But there's a price to pay, and I for one think the Android community needs to stop ignoring it.

references: GameBench