x PhoneArena is hiring! Reviewer in the USA
  • Hidden picShow menu
  • Home
  • ›
  • News
  • ›
  • Are benchmark tests a good representation of real life performance?

Are benchmark tests a good representation of real life performance?

Posted: , by Ian M.

Share Share

Tags :

Are benchmark tests a good representation of real life performance?
Quadrant

Quadrant

The performance of mobile devices has rapidly increased over the last few years. Major cell phone manufacturers are constantly trying to one-up each other more than ever before. It seems like every other day a new phone comes out with an even higher-resolution screen, a new-age GPU (Graphics Processing Unit), a ridiculously fast processor and an increasing amount of RAM (Random Access Memory). The mobile power race hasn’t gone unnoticed as consumers everywhere are hungry for beefier performance at the palm of their hands. Those curious to see how powerful their device truly is put it through a test called a benchmark.

The most common handsets that are put to the benchmark tests are those running on Google’s highly popular Android platform. There is an abundance of benchmarking applications available in the Android Market, in which the most notable is called Quadrant. What benchmarking does is measure multiple performance aspects of the device and add the results together to get a final score. The higher the score your smartphone receives, the better.

Neocore

Neocore

The main points measured in each test using one of the benchmarking applications are CPU, memory, I/O, and 2D/3D graphics. Powerful devices such as the Motorola XOOM tablet and the HTC Inspire 4G have performed well in these benchmark tests, but, interestingly, devices with similar specifications can often end up with an entirely different score altogether. So the question must be asked; can we trust these results to resemble real world performance? Or are these results skewed by other variables such as different screen resolution and user interfaces (custom software)?

As we saw years ago with the progression of PCs, the speed of the processor isn’t everything. Fortunately, manufacturers seem to be aware of this as evidenced by their advancements. There are many attributes that can affect the performance of a mobile device positively or negatively. A solid example is going back to the HTC EVO 3D benchmark that we did just a few weeks ago. While it may boast a 1.2GHz dual-core Qualcomm Snapdragon MSSM8660 processor to go along with a whopping 1GB of RAM and an Adreno 220 GPU, the EVO 3D was outperformed by other similar devices such as the Samsung Galaxy S II. But why?

One of the things that these tests do not take into consideration is the added user interfaces that manufacturers like to build on top of the Android platform. The HTC EVO 3D features the latest of HTC’s Sense UI which not only likes to eat away at your power but also occupies a good chunk of the device's memory. Given that memory is one of the main things measured by a benchmark test like Quadrant, it should be no surprise that the EVO 3D scores lower than the Galaxy SII which uses a somewhat less invasive user interface.

AnTuTu Benchmark

AnTuTu Benchmark

One of the great things about the Samsung Galaxy S II is the gorgeous Super AMOLED Plus technology of the display. However, this screen only has a resolution of 480x800. The HTC EVO 3D may not have the big, eye-popping colors of Samsung’s latest flagship, but it does however boast a sharp qHD screen with a higher resolution of 540x960. And the higher resolution display will take away from the EVO 3D’s score because it is taxing the processor more during usage.

So what can you take from all of this? Can we trust benchmark results to resemble real world performance? The answer can actually be both yes and no. First of all, we have to be sure that  there aren't any hardware/software incompatibilities between the device and the benchmarking app, which can be difficult to figure out, but suspiciously high or low results should lead to such thoughts. Then, it should be kept in mind that it's best to compare handsets with roughly the same system software installed, since it's obvious that custom solutions by manufacturers/developers may alter the benchmark results. That, however, doesn't mean that real-life performance will truly be different.

What do you all think? Are benchmark tests accurate means of testing the overall power of a device? Is it a good way to determine how well the phone will perform under real life conditions? We’d love to hear what you think on the subject so tell us what you think in the comments below!

33 Comments
  • Options
    Close





posted on 08 Jul 2011, 09:10
Reply

1. Larry_ThaGr81 (Posts: 449; Member since: 26 May 2011)


I think they could be an accurate way of testing the overall performance of a device if the developer(s) can find a way to turn everything off that uses up resources except for what is needed for the phone to function and run test that way.

posted on 11 Jul 2011, 10:17
Reply

32. deezy (Posts: 46; Member since: 10 Feb 2011)


Considering you can overclock and tweek your device to get over a 3000 benchmark, yet that device is unstable and would glitch/perform slower than a 90s computer full of porn, indicates that no it isn't a reliable way to judge performance. It had gingerbread on an X with a higher benchmark than my rooted/overclocked X with darkslide 4.2 even though my phone performed better than the stock gingerbread. After I updated to darkslide X, my benchmark was higher than stock gingerbread, yet I feel it's less snappy and responsive and darkslide 4.2

posted on 08 Jul 2011, 09:22 2
Reply

2. Droid_X_Doug (Posts: 5993; Member since: 22 Dec 2010)


The problem with turning off certain functions is that is in itself, an artificial test - who is going to use the EVO 3D without the Sense UI or the qHD display? I would argue for categories of handsets - Category 1 could be dual core CPU with minimum 1 ghz clock rate, 1 Gb RAM, dedicated GPU, qHD resolution display, HD video at a minimum 30 fps recording. Category 2 would be a lesser specification. To keep the Category relevant from year-to-year, there could be a Category 1-2011 which would have higher specs than, say Category 1-2010.

posted on 08 Jul 2011, 09:24 1
Reply

3. ngo2dd (Posts: 896; Member since: 08 Jul 2011)


The Qualcomm Snapdragon MSSM8660 processor only use one core at a time until a second is need that is why the Quadrant score is so low. People who use the 3D know how smooth it run. So no the benchmark test can not replace real world test.

posted on 08 Jul 2011, 13:13 2
Reply

8. SuperAndroidEvo (Posts: 4888; Member since: 15 Apr 2011)


The HTC Evo 3D uses one core at a time, while the Exynos & Tegra 2 use BOTH cores at the same time. That is why I get great battery life on this phone. Since June 24th I average 18 hours of heavy usage from personal fun to heavy business use during my work hours. So the benchmarks will be different. If there was a way to have Qualcomm's 2 cores be running at the same time during the benchmarks the scores will be closer to Nvidia's & Samsung's. PLUS the HTC Evo 3D's Sense 3.0 UI is a huge memory eater. I can't wait until I can slap a vanilla ROM of Gingerbread on the HTC Evo 3D to really see how the benchmarks are affected by the Sense 3.0 UI. Also the HTC Evo 3D has a 540x960 qHD display which hinders benchmark performance. Those extra pixels make a HUGE difference. So for the HTC Evo 3D to score a 2318 on Quadrant (which I just got on my phone) with all those factors is so impressive. Imagine the score it would receive if it had no qHD display and had a normal 480x800 resolution, no HTC Sense 3.0 UI, & if both cores ran at the same time like the Exynos & Tegra 2 does! So in real world performance my HTC Evo 3D is a beast which can STAND toe to toe with the BEST out there. The HTC Evo 3D IS one of the BEST phones period! It’s so smooth & does everything I want excellently. It’s incredible on the web, on Wi-Fi & 4G. I can have multiple apps running at the same time with no LAG. There is no game I play that it cannot handle. So it’s light years ahead of my old HTC Evo 4G. Just wait until Android releases Ice Cream Sandwich which is optimized for multiple-core phones. Then the HTC Evo 3D will shine even brighter. The HTC Evo 3D is indeed a future proof phone. I can’t be more satisfied with its performance at this moment.

posted on 08 Jul 2011, 14:02
Reply

10. Droid_X_Doug (Posts: 5993; Member since: 22 Dec 2010)


The only problem for Sprint is their spotty WiMax coverage. There are some places on the S.F. peninsula where you have 4G coverage, and if you walk to the other end of the parking lot (300-ish yards), no coverage. Also, it seems that both WiMax and LTE don't play well indoors. :-(

posted on 08 Jul 2011, 14:39 2
Reply

12. SuperAndroidEvo (Posts: 4888; Member since: 15 Apr 2011)


Yes all companies 4G service is in its infancy. I remember 3G had the same problem way back then. In 5 years or so both Sprint's & Verizon's LTE will be more the norm than anything else, but for now that is to be expected! The tech is moving so fast that we need time to catch up to it!

posted on 11 Jul 2011, 10:27
Reply

33. deezy (Posts: 46; Member since: 10 Feb 2011)


Have you noticed the LTE radios seem more responsive connected to 3G than say...stock 3G CDMA radios? I have an X and my wife has the Charge and the speedtests show .5-.8 download for me and .9-1.6 download for her yet my upload stays around .2 where her upload is around 1.2 on 3G. Our city/town is surrounded by mountains but coverage has been great with both. The 700 mHz frequency of the LTE radios is supposed to do a better job for inner city and mountainous areas. I do notice the preliminary problems with the LTE radios but it shows to be a promising network.

posted on 09 Jul 2011, 04:45
Reply

19. ph00ny (Posts: 1509; Member since: 26 May 2011)


Similar to how both cores aren't use to their fullest unless needed?

CPUs do not run as an on and off switch. Even if both cores aren't used, if it's not enough to tax qualcomm's dual core solution than i highly doubt there is a 100% utilization on both cores for tegra2 and exynos. Face the music, A9 is superior to A8 in cpu performance clock for clock

posted on 08 Jul 2011, 10:36
Reply

4. ddedsg (unregistered)


The Exynos can't use both cores at a time as well, yet its score is much higher. The Snapdragon is just two 8255s (Desire HD) slapped onto a single die while the Exynos and Tegra 2 use the newer ARM A9 architecture, which provides at least 25% improvement per core.

posted on 08 Jul 2011, 11:11 2
Reply

5. Krishaun27 (unregistered)


Wow i never took into consideration the fact that high resolution qhd displays use considerably more cpu than amoled displays. This could explain why htc sensation benchmarks are lower than SGS2, and of course the taxing new Sense UI interface.

posted on 08 Jul 2011, 14:37
Reply

11. exo2u (unregistered)


Definitely decreases performance. One reason I'm picking SGSII screen over other higher res screens. Personally waiting for a retina display equivalent to be excited about resolution.

posted on 08 Jul 2011, 15:27
Reply

13. not true (unregistered)


http://androidandme.com/2011/03/news/preview-how-will-upcoming-dual-core-cpus-like-exynos-omap4-and-snapdragon-match-up-with-tegra-2/

it gives benchmarks showing that what really matters is not really the amount of pixels but rather the processors really. A tegra 2 device with a qHD display scores significantly higher than another device with qHD and a snapdragon.

posted on 08 Jul 2011, 17:01
Reply

14. exo2u (unregistered)


You misunderstood me. Everyone knows that without the processor there wouldn't be any performance lol. just saying that higher resolution will make any device work harder

posted on 09 Jul 2011, 04:50
Reply

20. ph00ny (Posts: 1509; Member since: 26 May 2011)


It probably has more to do with the fact that Qualcomm's solution is built on modified A8 architecture whereas both Tegra2 and Exynos are built on A9 architecture. Resolution doesn't impact much on CPU usage during day to day usage. Webpages are rendered as whole and high def videos are decoded and downscaled accordingly

posted on 09 Jul 2011, 10:52 1
Reply

21. exo2u (unregistered)


Geez Phoony AND Not true....I must be hard to understand. The facts i was stating are pretty indisputable and if you think otherwise then you clearly just didn't get what i meant.
Ph00ny-your comment is so far off im not even sure you were trying to reply to me lol

posted on 10 Jul 2011, 01:27
Reply

25. ph00ny (Posts: 1509; Member since: 26 May 2011)


i'm pretty sure my post is listed below the op's post

also how is my post far from the truth? GPU performance is impacted by the resolution difference more than anything

Outside of games, what else is taking the performance hits on day to day operation?

posted on 08 Jul 2011, 11:31 3
Reply

6. corps1089 (Posts: 492; Member since: 20 Jan 2010)


Like any statistic, the meaning of the results can be skewed if the variables are ignored. There is no simple answer as to which performs better unless you compare apples to apples and the nature of these tests are to compare apples to oranges.
As we all know, items mentioned such as screen resolution, memory speed, GPU speed, etc. will all have an effect on performance even if the core chipsets have the same specifications. You have to take all into account and look at the data as a whole versus focusing on just the benchmark score.

Statistically speaking, taking a qualititive poll on how well a handset performs would yield better results IF AND ONLY IF everyone who was polled could remove thier carrier/manufacturer/OS biases. Yeah good luck getting that to work...


@Steve Jobs, I meant apples in the context of the edible fruit. You haven't trademarked or received a patent for that yet... have you?

posted on 08 Jul 2011, 11:56 1
Reply

7. josh (unregistered)


Benchmarks don't matter. I have the galaxy s and it never slows down or lags and it only gets like a 900.

posted on 08 Jul 2011, 13:44 2
Reply

9. KingB (unregistered)


Yes, but you have know how to interpret the numbers.

Obviously very few people do that.

That is why we have so many GS2 fan boys.

posted on 08 Jul 2011, 17:10
Reply

15. twill (unregistered)


one of the best ways to test this will be when the US version of the GS2 is released with tegra 2 onboard. Barring any other hardware changes you will be able to compare the exact same hardware and software with the processor being the only difference.

posted on 08 Jul 2011, 17:12
Reply

16. barfbagger (unregistered)


Its not about how fast it runs, its how efficiently it uses the resources and how good the programmers are at maximizing its efficiency.

Android isnt even meant for dual core yet so its inefficiently burning through resources to get from point A to point B vs utilizing the cores for maximum performance across the board.

Its like comparing a Honda with Ferrari. Sure they both can reach 60mph, but there is no real world reason why you would need to get to 60mph faster than the other on 1/4 mile track.

posted on 08 Jul 2011, 21:15 2
Reply

17. RORYREVOLUTION (Posts: 3117; Member since: 12 Jan 2010)


I don't care what my phone scores at on a benchmark. I care about the real performance of the device itself. If it can't handle simple things like sliding through your homescreens and it lags like crazy. That annoys the crap out of me. I don't care if it gets a score of 1960(which my Droid X got) after a while it became very laggy when you put widgets and stuff on your homescreens.

posted on 09 Jul 2011, 23:19
Reply

24. biophone (Posts: 1994; Member since: 15 Jun 2011)


I couldnt agree more and that's why people love the iPhone. May not score well but it never seems to lag or crash. That's real world speed that people care about and buy! Androids are frustrating to some because of there constant lagging and freezing. People don't like how open android is its too much work for the average consumer. People just want there phone to work.

posted on 10 Jul 2011, 16:24
Reply

29. taz89 (Posts: 2014; Member since: 03 May 2011)


i think that that mainly any android before gingerbread lags and crashes...my galaxy s2 is lag free so far and has yet to slow down even with widgets and lots of apps running...you say iphone does not lag well thats not true at all...my ip4 has lagged on me a few times for example sometimes the animations when an app is closing will skip...also if you are downloading apps and doing other things you will see lag...although it does not happen to often and that is why you dont hear many complains....you also have to remember that a lot less is happening on a iphone compared to android..background syncing live widgets,live wallpapers etc....also i dont really get how android is to hard for average consumers, it really isnt that hard at all and the s2 is my 1st android and i found it just as easy as using my iphone....if you know how to read and have common sense then using any phone shouldnt b an issue unless you are going to root and mod your phone which is then understandable but i doubt the average user will be buying android to mod it.

posted on 10 Jul 2011, 21:54 1
Reply

31. biophone (Posts: 1994; Member since: 15 Jun 2011)


Even with gb i now will all that android does in the background it still lags. iPhones are simple and therefore have little lagging problems and battery problems. With android non tech savy consumers have to deal with things like background data and battery saving tips. I had a thunderbolt for two weeks and was driving myself crazy toggling on/off 4g turning my brightness higher and lower playing with the settings to save battery and reading forums to learn how to get a day of battery life. People don't want to do that and that's the lure of an iPhone. It has real world speed it's simple and beautifully made. Plus things like apple makes everything to work eachother not like the skins androids have.

posted on 09 Jul 2011, 04:29
Reply

18. ph00ny (Posts: 1509; Member since: 26 May 2011)


It does reflect real world performance up to a certain point. There are some terrible benchmarks that really needs to be updated to keep up with the time *cough* quadrant *cough* and there are those which doesn't take some of the variables such as screen resolution into consideration during the benchmark (CF-bench, linpack, etc). Even going back to Quadrant which is flawed to begin with, people tend to cry about how one device has more pixels to push or there is a frame rate cap but the graphics portion only accounts for a very small portion of the benchmark's total score.

posted on 09 Jul 2011, 17:10 1
Reply

22. IEatApples (Posts: 66; Member since: 06 Jul 2011)


Tegra 2 is far superior to any dual core on the market. Samsung knows benchmark sells phones and tweaks their exynos to perform better on benchmark in addition to useing lower resolution to make the processor seem better. Benchmarks however don't matter to 99% of ppl. But in comparing processors in our world nvidia is tops. Qualcomm should be ashamed of puking out 2 8255 a8 scorpion cores and overclocking them to look better to most uninformed ppl. Most ppl just see 1.2ghz and assume 1.2 is better then 1. Great marketing skeem but once again just like sammy its all smoke and mirrors. Nvidia will dominate 2012 with kal. El quad core as well. All you nerds with the evo 3d and sensation, stop bragging about your crappy processor and blaming crappy benchmarks on sense.

posted on 09 Jul 2011, 23:11
Reply

23. biophone (Posts: 1994; Member since: 15 Jun 2011)


There are other things besides speed that are important. Single core is better it will save you battery. Although it is outdated until larger battery capacieties are made smaller dual core isn't worth it. Or if ur dual core phone only uses 2 when it needs it. Evo 3 d is best phone out there love the dual core 4g ram sense it just needs to be in vzw and smaller!

posted on 10 Jul 2011, 01:31
Reply

26. ph00ny (Posts: 1509; Member since: 26 May 2011)


Let's talk about tegra2 for a sec.

Lowest fpu performance out of the three and terrible native video codec support. Show me an example where Nvidia tops others in CPU performance not GPU performance. We all know that nvidia tegra2 devices aren't capped to 60fps like other devices hence the score it is able achieve in game benches.

Want to comment? Please login or register.

Latest stories