x PhoneArena is looking for new authors! To view all available positions, click here.
  • Home
  • News
  • Here are the new benchmark tests we will be using in our reviews

Here are the new benchmark tests we will be using in our reviews

Posted: , by Ray S.

Here are the new benchmark tests we will be using in our reviews
Synthetic benchmarks should be approached with care, because putting too much trust in those numbers may result in unnecessarily high (or low) expectations of a device's performance. Indeed, it is real-world usage that counts, but still, benchmarking shouldn't be ignored altogether. After all, there are some tests that can actually give us a pretty good picture about certain aspects of a phone or a tablet.

We think you know where this is going. Ladies and gents, allow us to introduce our new benchmarks! Let us elaborate: we've come up with a number of somewhat more scientific tests that we'll use to study the devices' displays, system performance, camera speed, audio output and... that's right - battery life! The results will be easily visible in the respective device's review. Here's a glimpse of how the new benchmarks are going to look!

  • When it comes to displays, we'll be providing you with quite a lot of information. For example, you'll be able to see a display's maximum and minimum brightness, contrast, color temperature, Delta E, color gamut and more. We've tried to make it all easy to use and understand.

Display measurements and quality

Maximum brightness (nits)Higher is better Minimum brightness (nits)Lower is better Contrast Higher is better Color temperature (Kelvins) Gamma Delta E rgbcmy Lower is better Delta E grayscale Lower is better
HTC One 460
(Good)
14
(Poor)
1:1758
(Excellent)
8008
(Poor)
2.21
5.89
(Average)
5.33
(Average)
LG G2 438
(Good)
8
(Good)
1:1338
(Excellent)
8109
(Poor)
2.25
4.27
(Average)
6.22
(Average)
Samsung Galaxy S4 289
(Poor)
9
(Average)
unmeasurable
(Excellent)
7316
(Good)
2.16
5.52
(Average)
7.31
(Average)
View all


  • Up next, we'll continue to check performance of Android phones by running some of the most popular performance benchmarks out there, but we'll be displaying the results in a new, more appealing type of visualization.

Performance benchmarks

Quadrant
Higher is better
Sony Xperia Z Ultra 18158
HTC One 12481
Samsung Galaxy S4 12078
AnTuTu
Higher is better
Sony Xperia Z Ultra 29732
HTC One 23308
Samsung Galaxy S4 24701
GFXBench Egypt HD 2.5 onscreen (fps)
Higher is better
Sony Xperia Z Ultra 52
HTC One 31
Samsung Galaxy S4 39
Vellamo Metal
Higher is better
Sony Xperia Z Ultra 1153
HTC One 781
Samsung Galaxy S4 704
Vellamo HTML 5
Higher is better
Sony Xperia Z Ultra 2937
HTC One 2395
Samsung Galaxy S4 1702
View all


  • We'll be testing camera speeds in a couple of ways. The first one is our custom "Taking a pic" test, which will measure the time needed for a camera to launch, focus, take a picture, and get ready to take the next one. This test will also have a separate result for HDR photo-taking. We'll also post the scores we've got using the cross-platform Camspeed benchmarking app.

Camera speed

Taking a pic (sec)Lower is better Taking an HDR pic (sec)Lower is better CamSpeed score Higher is better CamSpeed score with flash Higher is better
Apple iPhone 5 2.7
3.7
544
286
Google Nexus 4 2.8
4.3
434
395
Sony Xperia Z 3
4
987
948
View all


  • Moving on to the audio output, there are two things we're going to measure in this regard. The first is the output power of the 3.5 mm headphone jack (in volts), and the second one is the loudspeaker's loudness (in decibels).

Audio output

Headphones output power (Volts)
Higher is better
Apple iPhone 5 0.54
Sony Xperia Z 0.37
Google Nexus 4 0.28
Loudspeaker loudness (dB)
Higher is better
Apple iPhone 5 71
Sony Xperia Z 67
Google Nexus 4 80
View all


  • Finally, we've created a custom battery life benchmark to test phones with. It's a special script, designed to replicate real-life usage. The results for each handset or tablet will be provided in hours and minutes.

Battery life

We measure battery life by running a custom web-script, designed to replicate the power consumption of typical real-life usage.

Time
Higher is better
Sony Xperia Z
4h 48 min (Average)
Google Nexus 4
4h 47 min (Average)
Apple iPhone 5
4h 22 min (Poor)
View all

Of course, keep in mind that all of these benchmarks are quite new, so we might continue polishing and improving them during the days ahead.

45 Comments
  • Options
    Close




posted on 17 Sep 2013, 10:14 12

1. Ninetysix (Posts: 1351; Member since: 08 Oct 2012)


Looks good.

posted on 17 Sep 2013, 10:53 1

17. buccob (Posts: 1305; Member since: 19 Jun 2012)


Looks very good... but will this test be updated to show information of older phones or reviews? or will this only work from now on?

posted on 17 Sep 2013, 11:13 17

18. Ray.S (Posts: 250; Member since: 19 Jul 2011)


We'll make sure to test a good number of recent smartphones so that there's a useful database with results.

posted on 17 Sep 2013, 11:16 2

19. adrialk (Posts: 20; Member since: 04 Sep 2013)


Please make sure to include the Droid MAXX!

posted on 17 Sep 2013, 12:02

23. buccob (Posts: 1305; Member since: 19 Jun 2012)


Thanks, good to hear

posted on 17 Sep 2013, 12:37

28. rd_nest (Posts: 683; Member since: 06 Jun 2010)


Why still using Quadrant and AnTuTu?

Nobody so far defined how exactly Quadrant and AnTuTu gives the score.

IMO, much better using something like 3DMark (cross-platform). Browsermark, Geekbench etc.,

posted on 17 Sep 2013, 12:45 1

29. neutralguy (Posts: 1152; Member since: 30 Apr 2012)


This is actually a very good upgrade to the site. Especially with regards to the battery, camera, and display tests.

posted on 17 Sep 2013, 10:14 4

2. rawbow (Posts: 406; Member since: 30 Mar 2012)


Nice work, and you have made it interactive. Pretty Neat!

posted on 17 Sep 2013, 10:16 7

3. PhoneArenaUser (Posts: 5433; Member since: 05 Aug 2011)


Nice, PhoneArena! Now let's hope that results will be fair and not biased. :)

posted on 18 Sep 2013, 04:43

44. rihel_95 (Posts: 197; Member since: 21 Mar 2012)


hey PA, check fps of Nexus 4 in GFXBench Egypt HD 2.5 onscreen using new benchmark chart...it says 4917! WTF...others are around 50...

posted on 17 Sep 2013, 10:17 16

4. PapaSmurf (Posts: 7322; Member since: 14 May 2012)


This should put an end to many fanboys saying the review was biased.

posted on 17 Sep 2013, 10:21 5

8. rawbow (Posts: 406; Member since: 30 Mar 2012)


The reviews are always biased in the eye of the beholder. What people fail to understand is that reviews too are done by humans and thus are still perspectives.

posted on 17 Sep 2013, 10:50 6

15. buccob (Posts: 1305; Member since: 19 Jun 2012)


You are correct, but Reviewers should understand that you can ONLY rate quantitative characteristics such as display measurements, sound loudness... etc... like what they just update here

And things like "how pretty" or "we like this skin better" is complete subjective opinion, which is finally up to each user or buyer, and should NOT be rated... or at least it should follow certain parameters

So yeah this should help lessen both the bias in reviews and the hating in the comments

posted on 17 Sep 2013, 10:25 3

9. Immolate (Posts: 270; Member since: 17 Jun 2011)


But some aspects of a phone's appeal are subjective and can't be captured by a benchmark. Aesthetic is one. The "feel" of a handset is another. The quality of the outbound and inbound voice signal isn't really entirely subjective, but there are so many variables involved that it might as well be. The verdict on such subjective criteria tends to evolve from the aggregate. But still, your point is taken. Having hard data on the real capabilities removes much of the subjectivity.

posted on 17 Sep 2013, 10:31

11. PhoneArenaUser (Posts: 5433; Member since: 05 Aug 2011)


"But some aspects of a phone's appeal are subjective and can't be captured by a benchmark."

True, but since these benchmarks are not about these aspects...

posted on 17 Sep 2013, 10:33 2

12. LordDavon (Posts: 107; Member since: 19 Sep 2011)


I wish, but fanboys are fanboys. This is only one aspect of a review, and while the benchmarks are going to be cleaner and interactive, there is still so much more making up a review. But, we can hope it helps. ;-)

posted on 17 Sep 2013, 10:18 1

5. bubbadoes (Posts: 319; Member since: 03 May 2012)


we'll

posted on 17 Sep 2013, 10:20

6. tech2 (Posts: 1757; Member since: 26 Oct 2012)


Finally phonearena..............Finally !

posted on 17 Sep 2013, 10:20 1

7. Immolate (Posts: 270; Member since: 17 Jun 2011)


Very nice. But just as a phones real test is how it works in real life, the real test of these imaginative benchmarks will be how well they parallel our actual experiences. My take on benchmarks and companies designing their device to "cheat" them is that if your benchmarks measure actual benefits to the end-user, then cheating them simply makes your device better for the end user. Cheat all you want.

posted on 17 Sep 2013, 10:28

10. LordDavon (Posts: 107; Member since: 19 Sep 2011)


I like that you can add a device, but is there a way to remove one if accidentally added?

posted on 17 Sep 2013, 10:52

16. theruleslawyer (Posts: 108; Member since: 23 Apr 2012)


Just refresh the page. It'll reset it.

posted on 17 Sep 2013, 13:46

34. LordDavon (Posts: 107; Member since: 19 Sep 2011)


Yes, but when there are several benchmarks on a single page, it will reset all of them. Would rather not start from the beginning due to one mistake.

posted on 17 Sep 2013, 10:45

13. Zakriya (Posts: 11; Member since: 05 Aug 2013)


Sony Xperia Z has everything on par in displays with others...Why the contrast is too low?
i think you should re do the display measurements of Z as it has some improvments in my opinion

posted on 17 Sep 2013, 15:22

35. sulre (Posts: 10; Member since: 08 Sep 2013)


well, because it's TFT TN. Check out the result of cantrast ratio on Xperia Z on GSMArena, it tells u the same.

posted on 17 Sep 2013, 10:47 2

14. buccob (Posts: 1305; Member since: 19 Jun 2012)


Great work... Hopefully this lessen the bias reviewer issue and also the fan rage in the comments

posted on 17 Sep 2013, 11:26

20. McLTE (Posts: 637; Member since: 18 Oct 2011)


Wait.. PA can hardly write an article without obviously fanboyism, massive punctuation and grammatical errors... now they have developed their own benchmarks?

I don't know whether I should be encouraged since maybe their poor writing is due to their talents residing with things more technical.. or should I be scared?

I'll go for scared.

posted on 17 Sep 2013, 11:30

21. cezarepc (Posts: 482; Member since: 23 Nov 2012)


There's something like this on another popular review site, but I like this one better because you can add phones (so you can compare easier). Pretty neat.

One suggestion I can make is to use a theoretical limit for a pixel that is off (Display measurements and quality). You can use 0.000001 nits for all AMOLED screens so that we can have a figure to compare with. It's not the real result yes, but will give us an idea of how AMOLED screen from one phone can compare to an AMOLED screen of another.

posted on 17 Sep 2013, 11:48

22. aries.phills (Posts: 132; Member since: 17 May 2011)


This is pretty cool and intuitive. Great idea by a great tech website. Now I feel like I should be part of phone arena team.
Kudos guys.

posted on 17 Sep 2013, 22:17

39. pwnarena (Posts: 820; Member since: 15 Feb 2013)


the only problem is that they don't seem credible. look at the battery numbers. sony phones get good scores in comparison to others that are known for having better battery longevity. i don't think that's accurate.

posted on 17 Sep 2013, 12:06 1

24. Tuxedo (Posts: 110; Member since: 19 Mar 2013)


Two comments:

1. When measuring output voltage of the headphone jack, you need to use a standard load (headset), otherwise, results will be meaningless.

2. I'd like to see a "sleep mode" battery life test... say how much battery is used in a given period of time (say 24hrs) in which the device is in sleep mode, radios on, sync on etc.

posted on 17 Sep 2013, 12:16

25. artealis (Posts: 98; Member since: 15 Dec 2010)


wow, finally camera speed test
please bring wifi strength test too PA, its important for day to day uses more than antutu benchmark number.
thx u.

posted on 17 Sep 2013, 12:18 1

26. jellmoo (Posts: 618; Member since: 31 Oct 2011)


Very nice. It's good to see that some standards are being applied to reviews, rather than a vague description and "feel".

Two things I'd still like to see:

1) A better breakdown on battery life (which you may do, it just isn't shown here).

2) A better wrap up at the end of the review, where each category is properly weighted so that we can better understand why one phone gets a 9, and another a 9.4 for example.

posted on 17 Sep 2013, 12:35

27. Mahdi007 (Posts: 5; Member since: 12 Sep 2013)


It's fantastic but please include all the phones(at least important phones) for example:Google nexus 4 is missed from battery life test

posted on 17 Sep 2013, 12:52

30. ocilfa (Posts: 329; Member since: 03 Aug 2012)


Please include the Nook HD+. Great tablet at a great price.

posted on 17 Sep 2013, 13:29

31. _Bone_ (Posts: 2104; Member since: 29 Oct 2012)


These screen measurements are pretty well off, Ray. DisplayMate use 100% accurate tools for proper numbers and the difference in the S4's numbers stand out at most areas. It has 475nit peak brightness (100% white, auto mode), 0 nits black, 6,5k Kelvin (almost perfect) color temperature in Movie Mode and 2.42 gamma cause it's an AMOLED screen. Your numbers are well off.

Now as far as the benchmarks go, I recommend adding the new AnTuTu 3DRating for OpenGL ES 3.0, Basemark X (on and off screen), some memory meausrement and most of all, battery testing and it'll be cool.

posted on 17 Sep 2013, 13:42

32. Kevinphantom (Posts: 82; Member since: 17 Jun 2013)


nice but also I recommend adding heat (temperature of the device ) while using aggressive cpu power and aggressive gpu power

posted on 17 Sep 2013, 13:44

33. mrturner (Posts: 16; Member since: 10 Sep 2013)


Hmm...useful. Would love to how the iphone 5same performs against the best androids.

posted on 17 Sep 2013, 21:23 1

37. 14545 (Posts: 1057; Member since: 22 Nov 2011)


Why is it that NOT ONE REVIEW SITE actually measures a phones radio performance. You can look at the actual dbm to compare radio performance. As well as synthetic tests at specific places with all phones. This is just the tip of what a good review should actually consist of. Good start, but still no where near enough.

posted on 17 Sep 2013, 22:14

38. pwnarena (Posts: 820; Member since: 15 Feb 2013)


your battery script is probably erroneous or inaccurate. it's impossible for the galaxy s3 to have lower longevity than the xperia z. i'm not really sure about these tools you're using. they don't seem reliable. they seem to be designed to favor sony devices.

posted on 18 Sep 2013, 00:49

40. XiphiasGladius (Posts: 801; Member since: 21 Aug 2011)


A welcome addition, keep it up P.A.

posted on 18 Sep 2013, 01:33

42. zachattack (Posts: 532; Member since: 31 Jul 2013)


What PA should do is update the G2's scores LOL. You honestly think the HTC One scores better than the G2 you must be on drugs

posted on 18 Sep 2013, 04:31

43. jsdechavez (Posts: 699; Member since: 20 Jul 2012)


Include Windows Phones. Lumia!

posted on 18 Sep 2013, 06:26

45. Retro-touch (Posts: 248; Member since: 24 Oct 2011)


Phone Arena why are you running GFX onscreen and not offscreen? To rule out pixel differences between phones I think it would be better to run it offscreen

Also Wi-Fi signal strength would be good to add if possible

posted on 19 Sep 2013, 02:21

46. hulkkii (Posts: 5; Member since: 19 Sep 2013)


Nice to have camera speed tests!

Here some notes about CamSpeed:
CamSpeed score is calculated from following measurements: camera start-up time, focus time and image capture time, so it presents the total time to take first picture. (Image saving to sdcard/disk is not part of measurements/score).

Info about latest app updates:
Galaxy S4 (exynos) should be measured again with CamSpeed (app version 2.1.2 or newer).
Samsung had little different implementation of Camera API's startpreview-call in Galaxy S4 than others causing little too fast camera start-up time results for it (and so a higher score than it should have), but the measurements for S4 were fixed in latest CamSpeed version.

* Some comments have been hidden, because they don't meet the discussions rules.

Want to comment? Please login or register.

Latest stories