x PhoneArena is hiring! Reviewer in the USA
  • Hidden picShow menu
  • Home
  • News
  • Here are the new benchmark tests we will be using in our reviews

Here are the new benchmark tests we will be using in our reviews

Posted: , by Ray S.

Here are the new benchmark tests we will be using in our reviews
Synthetic benchmarks should be approached with care, because putting too much trust in those numbers may result in unnecessarily high (or low) expectations of a device's performance. Indeed, it is real-world usage that counts, but still, benchmarking shouldn't be ignored altogether. After all, there are some tests that can actually give us a pretty good picture about certain aspects of a phone or a tablet.

We think you know where this is going. Ladies and gents, allow us to introduce our new benchmarks! Let us elaborate: we've come up with a number of somewhat more scientific tests that we'll use to study the devices' displays, system performance, camera speed, audio output and... that's right - battery life! The results will be easily visible in the respective device's review. Here's a glimpse of how the new benchmarks are going to look!

  • When it comes to displays, we'll be providing you with quite a lot of information. For example, you'll be able to see a display's maximum and minimum brightness, contrast, color temperature, Delta E, color gamut and more. We've tried to make it all easy to use and understand.

Display measurements and quality

Maximum brightness (nits)Higher is better Minimum brightness (nits)Lower is better Contrast Higher is better Color temperature (Kelvins) Gamma Delta E rgbcmy Lower is better Delta E grayscale Lower is better
LG G2 438
(Good)
8
(Good)
1:1338
(Excellent)
8109
(Poor)
2.25
4.27
(Average)
6.22
(Average)
Samsung Galaxy S4 289
(Poor)
9
(Average)
unmeasurable
(Excellent)
7316
(Good)
2.16
5.52
(Average)
7.31
(Average)
HTC One 460
(Good)
14
(Poor)
1:1758
(Excellent)
8008
(Poor)
2.21
5.89
(Average)
5.33
(Average)
View all


  • Up next, we'll continue to check performance of Android phones by running some of the most popular performance benchmarks out there, but we'll be displaying the results in a new, more appealing type of visualization.

Performance benchmarks

Quadrant
Higher is better
Sony Xperia Z Ultra 18158
HTC One 12481
Samsung Galaxy S4 12078
AnTuTu
Higher is better
Sony Xperia Z Ultra 29732
HTC One 23308
Samsung Galaxy S4 24701
GFXBench Egypt HD 2.5 onscreen (fps)
Higher is better
Sony Xperia Z Ultra 52
HTC One 31
Samsung Galaxy S4 39
Vellamo Metal
Higher is better
Sony Xperia Z Ultra 1153
HTC One 781
Samsung Galaxy S4 704
Vellamo HTML 5
Higher is better
Sony Xperia Z Ultra 2937
HTC One 2395
Samsung Galaxy S4 1702
View all


  • We'll be testing camera speeds in a couple of ways. The first one is our custom "Taking a pic" test, which will measure the time needed for a camera to launch, focus, take a picture, and get ready to take the next one. This test will also have a separate result for HDR photo-taking. We'll also post the scores we've got using the cross-platform Camspeed benchmarking app.

Camera speed

Taking a pic (sec)Lower is better Taking an HDR pic (sec)Lower is better CamSpeed score Higher is better CamSpeed score with flash Higher is better
Apple iPhone 5 2.7
3.7
544
286
Sony Xperia Z 3
4
987
948
Google Nexus 4 2.8
4.3
434
395
View all


  • Moving on to the audio output, there are two things we're going to measure in this regard. The first is the output power of the 3.5 mm headphone jack (in volts), and the second one is the loudspeaker's loudness (in decibels).

Audio output

Headphones output power (Volts)
Higher is better
Apple iPhone 5 0.54
Sony Xperia Z 0.37
Google Nexus 4 0.28
Loudspeaker loudness (dB)
Higher is better
Apple iPhone 5 71
Sony Xperia Z 67
Google Nexus 4 80
View all


  • Finally, we've created a custom battery life benchmark to test phones with. It's a special script, designed to replicate real-life usage. The results for each handset or tablet will be provided in hours and minutes.

Battery life

We measure battery life by running a custom web-script, designed to replicate the power consumption of typical real-life usage. All devices that go through the test have their displays set at 200-nit brightness.

hours
Higher is better
Sony Xperia Z
4h 48 min (Poor)
Apple iPhone 5
4h 22 min (Poor)
Google Nexus 4
4h 47 min (Poor)
View all

Of course, keep in mind that all of these benchmarks are quite new, so we might continue polishing and improving them during the days ahead.

45 Comments
  • Options
    Close






posted on 17 Sep 2013, 10:14 12

1. Ninetysix (Posts: 2461; Member since: 08 Oct 2012)


Looks good.

posted on 17 Sep 2013, 10:53 1

17. buccob (Posts: 2578; Member since: 19 Jun 2012)


Looks very good... but will this test be updated to show information of older phones or reviews? or will this only work from now on?

posted on 17 Sep 2013, 11:13 17

18. Ray.S (Posts: 339; Member since: 19 Jul 2011)


We'll make sure to test a good number of recent smartphones so that there's a useful database with results.

posted on 17 Sep 2013, 11:16 2

19. adrialk (Posts: 20; Member since: 04 Sep 2013)


Please make sure to include the Droid MAXX!

posted on 17 Sep 2013, 12:02

23. buccob (Posts: 2578; Member since: 19 Jun 2012)


Thanks, good to hear

posted on 17 Sep 2013, 12:37

28. rd_nest (Posts: 1607; Member since: 06 Jun 2010)


Why still using Quadrant and AnTuTu?

Nobody so far defined how exactly Quadrant and AnTuTu gives the score.

IMO, much better using something like 3DMark (cross-platform). Browsermark, Geekbench etc.,

posted on 17 Sep 2013, 12:45 1

29. neutralguy (Posts: 1152; Member since: 30 Apr 2012)


This is actually a very good upgrade to the site. Especially with regards to the battery, camera, and display tests.

posted on 17 Sep 2013, 10:14 4

2. rawbow (Posts: 428; Member since: 30 Mar 2012)


Nice work, and you have made it interactive. Pretty Neat!

posted on 17 Sep 2013, 10:16 7

3. PhoneArenaUser (Posts: 5498; Member since: 05 Aug 2011)


Nice, PhoneArena! Now let's hope that results will be fair and not biased. :)

posted on 18 Sep 2013, 04:43

44. rihel_95 (Posts: 301; Member since: 21 Mar 2012)


hey PA, check fps of Nexus 4 in GFXBench Egypt HD 2.5 onscreen using new benchmark chart...it says 4917! WTF...others are around 50...

posted on 17 Sep 2013, 10:17 16

4. PapaSmurf (Posts: 10457; Member since: 14 May 2012)


This should put an end to many fanboys saying the review was biased.

posted on 17 Sep 2013, 10:21 5

8. rawbow (Posts: 428; Member since: 30 Mar 2012)


The reviews are always biased in the eye of the beholder. What people fail to understand is that reviews too are done by humans and thus are still perspectives.

posted on 17 Sep 2013, 10:50 6

15. buccob (Posts: 2578; Member since: 19 Jun 2012)


You are correct, but Reviewers should understand that you can ONLY rate quantitative characteristics such as display measurements, sound loudness... etc... like what they just update here

And things like "how pretty" or "we like this skin better" is complete subjective opinion, which is finally up to each user or buyer, and should NOT be rated... or at least it should follow certain parameters

So yeah this should help lessen both the bias in reviews and the hating in the comments

posted on 17 Sep 2013, 10:25 3

9. Immolate (Posts: 310; Member since: 17 Jun 2011)


But some aspects of a phone's appeal are subjective and can't be captured by a benchmark. Aesthetic is one. The "feel" of a handset is another. The quality of the outbound and inbound voice signal isn't really entirely subjective, but there are so many variables involved that it might as well be. The verdict on such subjective criteria tends to evolve from the aggregate. But still, your point is taken. Having hard data on the real capabilities removes much of the subjectivity.

posted on 17 Sep 2013, 10:31

11. PhoneArenaUser (Posts: 5498; Member since: 05 Aug 2011)


"But some aspects of a phone's appeal are subjective and can't be captured by a benchmark."

True, but since these benchmarks are not about these aspects...

posted on 17 Sep 2013, 10:33 2

12. LordDavon (Posts: 135; Member since: 19 Sep 2011)


I wish, but fanboys are fanboys. This is only one aspect of a review, and while the benchmarks are going to be cleaner and interactive, there is still so much more making up a review. But, we can hope it helps. ;-)

posted on 17 Sep 2013, 10:18 1

5. bubbadoes (Posts: 1172; Member since: 03 May 2012)


we'll

posted on 17 Sep 2013, 10:20

6. tech2 (Posts: 3477; Member since: 26 Oct 2012)


Finally phonearena..............Finally !

posted on 17 Sep 2013, 10:20 1

7. Immolate (Posts: 310; Member since: 17 Jun 2011)


Very nice. But just as a phones real test is how it works in real life, the real test of these imaginative benchmarks will be how well they parallel our actual experiences. My take on benchmarks and companies designing their device to "cheat" them is that if your benchmarks measure actual benefits to the end-user, then cheating them simply makes your device better for the end user. Cheat all you want.

posted on 17 Sep 2013, 10:28

10. LordDavon (Posts: 135; Member since: 19 Sep 2011)


I like that you can add a device, but is there a way to remove one if accidentally added?

posted on 17 Sep 2013, 10:52

16. theruleslawyer (Posts: 108; Member since: 23 Apr 2012)


Just refresh the page. It'll reset it.

posted on 17 Sep 2013, 13:46

34. LordDavon (Posts: 135; Member since: 19 Sep 2011)


Yes, but when there are several benchmarks on a single page, it will reset all of them. Would rather not start from the beginning due to one mistake.

posted on 17 Sep 2013, 10:45

13. Zakriya (Posts: 11; Member since: 05 Aug 2013)


Sony Xperia Z has everything on par in displays with others...Why the contrast is too low?
i think you should re do the display measurements of Z as it has some improvments in my opinion

posted on 17 Sep 2013, 15:22

35. sulre (Posts: 18; Member since: 08 Sep 2013)


well, because it's TFT TN. Check out the result of cantrast ratio on Xperia Z on GSMArena, it tells u the same.

posted on 17 Sep 2013, 10:47 2

14. buccob (Posts: 2578; Member since: 19 Jun 2012)


Great work... Hopefully this lessen the bias reviewer issue and also the fan rage in the comments

posted on 17 Sep 2013, 11:26

20. McLTE (Posts: 877; Member since: 18 Oct 2011)


Wait.. PA can hardly write an article without obviously fanboyism, massive punctuation and grammatical errors... now they have developed their own benchmarks?

I don't know whether I should be encouraged since maybe their poor writing is due to their talents residing with things more technical.. or should I be scared?

I'll go for scared.

posted on 17 Sep 2013, 11:30

21. cezarepc (Posts: 716; Member since: 23 Nov 2012)


There's something like this on another popular review site, but I like this one better because you can add phones (so you can compare easier). Pretty neat.

One suggestion I can make is to use a theoretical limit for a pixel that is off (Display measurements and quality). You can use 0.000001 nits for all AMOLED screens so that we can have a figure to compare with. It's not the real result yes, but will give us an idea of how AMOLED screen from one phone can compare to an AMOLED screen of another.

posted on 17 Sep 2013, 11:48

22. aries.phills (Posts: 140; Member since: 17 May 2011)


This is pretty cool and intuitive. Great idea by a great tech website. Now I feel like I should be part of phone arena team.
Kudos guys.

posted on 17 Sep 2013, 22:17

39. pwnarena (Posts: 1129; Member since: 15 Feb 2013)


the only problem is that they don't seem credible. look at the battery numbers. sony phones get good scores in comparison to others that are known for having better battery longevity. i don't think that's accurate.

posted on 17 Sep 2013, 12:06 1

24. Tuxedo (Posts: 351; Member since: 19 Mar 2013)


Two comments:

1. When measuring output voltage of the headphone jack, you need to use a standard load (headset), otherwise, results will be meaningless.

2. I'd like to see a "sleep mode" battery life test... say how much battery is used in a given period of time (say 24hrs) in which the device is in sleep mode, radios on, sync on etc.

* Some comments have been hidden, because they don't meet the discussions rules.

Want to comment? Please login or register.

Latest stories