Here are the new benchmark tests we will be using in our reviews

Here are the new benchmark tests we will be using in our reviews
Synthetic benchmarks should be approached with care, because putting too much trust in those numbers may result in unnecessarily high (or low) expectations of a device's performance. Indeed, it is real-world usage that counts, but still, benchmarking shouldn't be ignored altogether. After all, there are some tests that can actually give us a pretty good picture about certain aspects of a phone or a tablet.

We think you know where this is going. Ladies and gents, allow us to introduce our new benchmarks! Let us elaborate: we've come up with a number of somewhat more scientific tests that we'll use to study the devices' displays, system performance, camera speed, audio output and... that's right - battery life! The results will be easily visible in the respective device's review. Here's a glimpse of how the new benchmarks are going to look!

  • When it comes to displays, we'll be providing you with quite a lot of information. For example, you'll be able to see a display's maximum and minimum brightness, contrast, color temperature, Delta E, color gamut and more. We've tried to make it all easy to use and understand.


  • Up next, we'll continue to check performance of Android phones by running some of the most popular performance benchmarks out there, but we'll be displaying the results in a new, more appealing type of visualization.
Quadrant Higher is better
Sony Xperia Z Ultra 18158
HTC One 12481
Samsung Galaxy S4 12078
AnTuTu Higher is better
Sony Xperia Z Ultra 29732
HTC One 23308
Samsung Galaxy S4 24701
GFXBench Egypt HD 2.5 onscreen (fps) Higher is better
Sony Xperia Z Ultra 52
HTC One 31
Samsung Galaxy S4 39
Vellamo Metal Higher is better
Sony Xperia Z Ultra 1153
HTC One 781
Samsung Galaxy S4 704
Vellamo HTML 5 Higher is better
Sony Xperia Z Ultra 2937
HTC One 2395
Samsung Galaxy S4 1702


  • We'll be testing camera speeds in a couple of ways. The first one is our custom "Taking a pic" test, which will measure the time needed for a camera to launch, focus, take a picture, and get ready to take the next one. This test will also have a separate result for HDR photo-taking. We'll also post the scores we've got using the cross-platform Camspeed benchmarking app.
Taking a pic Lower is better Taking an HDR pic (sec) Lower is better CamSpeed score Higher is better CamSpeed score with flash Higher is better
Apple iPhone 5 2.7
3.7
544
286
Sony Xperia Z 3
4
987
948
Google Nexus 4 2.8
4.3
434
395


  • Moving on to the audio output, there are two things we're going to measure in this regard. The first is the output power of the 3.5 mm headphone jack (in volts), and the second one is the loudspeaker's loudness (in decibels).
Headphones output power (Volts) Higher is better
Apple iPhone 5 0.54
Sony Xperia Z 0.37
Google Nexus 4 0.28
Loudspeaker loudness (dB) Higher is better
Apple iPhone 5 71
Sony Xperia Z 67
Google Nexus 4 80


  • Finally, we've created a custom battery life benchmark to test phones with. It's a special script, designed to replicate real-life usage. The results for each handset or tablet will be provided in hours and minutes.

We measure battery life by running a custom web-script, designed to replicate the power consumption of typical real-life usage. All devices that go through the test have their displays set at 200-nit brightness.

hours Higher is better
Sony Xperia Z
4h 48 min (Poor)
Apple iPhone 5
4h 22 min (Poor)
Google Nexus 4
4h 47 min (Poor)

Of course, keep in mind that all of these benchmarks are quite new, so we might continue polishing and improving them during the days ahead.

FEATURED VIDEO

45 Comments

1. Ninetysix

Posts: 2962; Member since: Oct 08, 2012

Looks good.

17. buccob

Posts: 2963; Member since: Jun 19, 2012

Looks very good... but will this test be updated to show information of older phones or reviews? or will this only work from now on?

18. Ray.S

Posts: 456; Member since: Jul 19, 2011

We'll make sure to test a good number of recent smartphones so that there's a useful database with results.

19. adrialk

Posts: 20; Member since: Sep 04, 2013

Please make sure to include the Droid MAXX!

23. buccob

Posts: 2963; Member since: Jun 19, 2012

Thanks, good to hear

28. rd_nest

Posts: 1656; Member since: Jun 06, 2010

Why still using Quadrant and AnTuTu? Nobody so far defined how exactly Quadrant and AnTuTu gives the score. IMO, much better using something like 3DMark (cross-platform). Browsermark, Geekbench etc.,

29. neutralguy

Posts: 1152; Member since: Apr 30, 2012

This is actually a very good upgrade to the site. Especially with regards to the battery, camera, and display tests.

2. rawbow

Posts: 428; Member since: Mar 30, 2012

Nice work, and you have made it interactive. Pretty Neat!

3. PhoneArenaUser

Posts: 5498; Member since: Aug 05, 2011

Nice, PhoneArena! Now let's hope that results will be fair and not biased. :)

44. rihel_95

Posts: 305; Member since: Mar 21, 2012

hey PA, check fps of Nexus 4 in GFXBench Egypt HD 2.5 onscreen using new benchmark chart...it says 4917! WTF...others are around 50...

4. PapaSmurf

Posts: 10457; Member since: May 14, 2012

This should put an end to many fanboys saying the review was biased.

8. rawbow

Posts: 428; Member since: Mar 30, 2012

The reviews are always biased in the eye of the beholder. What people fail to understand is that reviews too are done by humans and thus are still perspectives.

15. buccob

Posts: 2963; Member since: Jun 19, 2012

You are correct, but Reviewers should understand that you can ONLY rate quantitative characteristics such as display measurements, sound loudness... etc... like what they just update here And things like "how pretty" or "we like this skin better" is complete subjective opinion, which is finally up to each user or buyer, and should NOT be rated... or at least it should follow certain parameters So yeah this should help lessen both the bias in reviews and the hating in the comments

9. Immolate

Posts: 310; Member since: Jun 17, 2011

But some aspects of a phone's appeal are subjective and can't be captured by a benchmark. Aesthetic is one. The "feel" of a handset is another. The quality of the outbound and inbound voice signal isn't really entirely subjective, but there are so many variables involved that it might as well be. The verdict on such subjective criteria tends to evolve from the aggregate. But still, your point is taken. Having hard data on the real capabilities removes much of the subjectivity.

11. PhoneArenaUser

Posts: 5498; Member since: Aug 05, 2011

"But some aspects of a phone's appeal are subjective and can't be captured by a benchmark." True, but since these benchmarks are not about these aspects...

12. LordDavon

Posts: 153; Member since: Sep 19, 2011

I wish, but fanboys are fanboys. This is only one aspect of a review, and while the benchmarks are going to be cleaner and interactive, there is still so much more making up a review. But, we can hope it helps. ;-)

5. bubbadoes

Posts: 1225; Member since: May 03, 2012

we'll

6. tech2

Posts: 3487; Member since: Oct 26, 2012

Finally phonearena..............Finally !

7. Immolate

Posts: 310; Member since: Jun 17, 2011

Very nice. But just as a phones real test is how it works in real life, the real test of these imaginative benchmarks will be how well they parallel our actual experiences. My take on benchmarks and companies designing their device to "cheat" them is that if your benchmarks measure actual benefits to the end-user, then cheating them simply makes your device better for the end user. Cheat all you want.

10. LordDavon

Posts: 153; Member since: Sep 19, 2011

I like that you can add a device, but is there a way to remove one if accidentally added?

16. theruleslawyer

Posts: 108; Member since: Apr 23, 2012

Just refresh the page. It'll reset it.

34. LordDavon

Posts: 153; Member since: Sep 19, 2011

Yes, but when there are several benchmarks on a single page, it will reset all of them. Would rather not start from the beginning due to one mistake.

13. Zakriya

Posts: 11; Member since: Aug 05, 2013

Sony Xperia Z has everything on par in displays with others...Why the contrast is too low? i think you should re do the display measurements of Z as it has some improvments in my opinion

35. sulre

Posts: 18; Member since: Sep 08, 2013

well, because it's TFT TN. Check out the result of cantrast ratio on Xperia Z on GSMArena, it tells u the same.

14. buccob

Posts: 2963; Member since: Jun 19, 2012

Great work... Hopefully this lessen the bias reviewer issue and also the fan rage in the comments

20. McLTE

Posts: 922; Member since: Oct 18, 2011

Wait.. PA can hardly write an article without obviously fanboyism, massive punctuation and grammatical errors... now they have developed their own benchmarks? I don't know whether I should be encouraged since maybe their poor writing is due to their talents residing with things more technical.. or should I be scared? I'll go for scared.

21. cezarepc

Posts: 718; Member since: Nov 23, 2012

There's something like this on another popular review site, but I like this one better because you can add phones (so you can compare easier). Pretty neat. One suggestion I can make is to use a theoretical limit for a pixel that is off (Display measurements and quality). You can use 0.000001 nits for all AMOLED screens so that we can have a figure to compare with. It's not the real result yes, but will give us an idea of how AMOLED screen from one phone can compare to an AMOLED screen of another.

22. aries.phills

Posts: 161; Member since: May 17, 2011

This is pretty cool and intuitive. Great idea by a great tech website. Now I feel like I should be part of phone arena team. Kudos guys.

39. pwnarena

Posts: 1129; Member since: Feb 15, 2013

the only problem is that they don't seem credible. look at the battery numbers. sony phones get good scores in comparison to others that are known for having better battery longevity. i don't think that's accurate.

24. Tuxedo

Posts: 356; Member since: Mar 19, 2013

Two comments: 1. When measuring output voltage of the headphone jack, you need to use a standard load (headset), otherwise, results will be meaningless. 2. I'd like to see a "sleep mode" battery life test... say how much battery is used in a given period of time (say 24hrs) in which the device is in sleep mode, radios on, sync on etc.

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.