Here are the new benchmark tests we will be using in our reviews
0. phoneArena 17 Sep 2013, 10:10 posted on
Synthetic benchmarks should be approached with care, because putting too much trust in those numbers may result in unnecessarily high (or low) expectations of a device's performance. Indeed, it is real-world usage that counts, but still, benchmarking shouldn't be ignored altogether. After all, there are some tests that can actually give us a pretty good picture about certain aspects of a phone or a tablet...
This is a discussion for a news. To read the whole news, click here
17. buccob (Posts: 1656; Member since: 19 Jun 2012)
Looks very good... but will this test be updated to show information of older phones or reviews? or will this only work from now on?
18. Ray.S (Posts: 267; Member since: 19 Jul 2011)
We'll make sure to test a good number of recent smartphones so that there's a useful database with results.
19. adrialk (Posts: 20; Member since: 04 Sep 2013)
Please make sure to include the Droid MAXX!
28. rd_nest (Posts: 876; Member since: 06 Jun 2010)
Why still using Quadrant and AnTuTu?
Nobody so far defined how exactly Quadrant and AnTuTu gives the score.
IMO, much better using something like 3DMark (cross-platform). Browsermark, Geekbench etc.,
29. neutralguy (Posts: 1152; Member since: 30 Apr 2012)
This is actually a very good upgrade to the site. Especially with regards to the battery, camera, and display tests.
2. rawbow (Posts: 428; Member since: 30 Mar 2012)
Nice work, and you have made it interactive. Pretty Neat!
3. PhoneArenaUser (Posts: 5498; Member since: 05 Aug 2011)
Nice, PhoneArena! Now let's hope that results will be fair and not biased. :)
44. rihel_95 (Posts: 245; Member since: 21 Mar 2012)
hey PA, check fps of Nexus 4 in GFXBench Egypt HD 2.5 onscreen using new benchmark chart...it says 4917! WTF...others are around 50...
4. PapaSmurf (Posts: 9071; Member since: 14 May 2012)
This should put an end to many fanboys saying the review was biased.
8. rawbow (Posts: 428; Member since: 30 Mar 2012)
The reviews are always biased in the eye of the beholder. What people fail to understand is that reviews too are done by humans and thus are still perspectives.
15. buccob (Posts: 1656; Member since: 19 Jun 2012)
You are correct, but Reviewers should understand that you can ONLY rate quantitative characteristics such as display measurements, sound loudness... etc... like what they just update here
And things like "how pretty" or "we like this skin better" is complete subjective opinion, which is finally up to each user or buyer, and should NOT be rated... or at least it should follow certain parameters
So yeah this should help lessen both the bias in reviews and the hating in the comments
9. Immolate (Posts: 292; Member since: 17 Jun 2011)
But some aspects of a phone's appeal are subjective and can't be captured by a benchmark. Aesthetic is one. The "feel" of a handset is another. The quality of the outbound and inbound voice signal isn't really entirely subjective, but there are so many variables involved that it might as well be. The verdict on such subjective criteria tends to evolve from the aggregate. But still, your point is taken. Having hard data on the real capabilities removes much of the subjectivity.
11. PhoneArenaUser (Posts: 5498; Member since: 05 Aug 2011)
"But some aspects of a phone's appeal are subjective and can't be captured by a benchmark."
True, but since these benchmarks are not about these aspects...
12. LordDavon (Posts: 108; Member since: 19 Sep 2011)
I wish, but fanboys are fanboys. This is only one aspect of a review, and while the benchmarks are going to be cleaner and interactive, there is still so much more making up a review. But, we can hope it helps. ;-)
7. Immolate (Posts: 292; Member since: 17 Jun 2011)
Very nice. But just as a phones real test is how it works in real life, the real test of these imaginative benchmarks will be how well they parallel our actual experiences. My take on benchmarks and companies designing their device to "cheat" them is that if your benchmarks measure actual benefits to the end-user, then cheating them simply makes your device better for the end user. Cheat all you want.
10. LordDavon (Posts: 108; Member since: 19 Sep 2011)
I like that you can add a device, but is there a way to remove one if accidentally added?
16. theruleslawyer (Posts: 108; Member since: 23 Apr 2012)
Just refresh the page. It'll reset it.
34. LordDavon (Posts: 108; Member since: 19 Sep 2011)
Yes, but when there are several benchmarks on a single page, it will reset all of them. Would rather not start from the beginning due to one mistake.
13. Zakriya (Posts: 11; Member since: 05 Aug 2013)
Sony Xperia Z has everything on par in displays with others...Why the contrast is too low?
i think you should re do the display measurements of Z as it has some improvments in my opinion
35. sulre (Posts: 15; Member since: 08 Sep 2013)
well, because it's TFT TN. Check out the result of cantrast ratio on Xperia Z on GSMArena, it tells u the same.
14. buccob (Posts: 1656; Member since: 19 Jun 2012)
Great work... Hopefully this lessen the bias reviewer issue and also the fan rage in the comments
20. McLTE (Posts: 714; Member since: 18 Oct 2011)
Wait.. PA can hardly write an article without obviously fanboyism, massive punctuation and grammatical errors... now they have developed their own benchmarks?
I don't know whether I should be encouraged since maybe their poor writing is due to their talents residing with things more technical.. or should I be scared?
I'll go for scared.
21. cezarepc (Posts: 626; Member since: 23 Nov 2012)
There's something like this on another popular review site, but I like this one better because you can add phones (so you can compare easier). Pretty neat.
One suggestion I can make is to use a theoretical limit for a pixel that is off (Display measurements and quality). You can use 0.000001 nits for all AMOLED screens so that we can have a figure to compare with. It's not the real result yes, but will give us an idea of how AMOLED screen from one phone can compare to an AMOLED screen of another.
22. aries.phills (Posts: 134; Member since: 17 May 2011)
This is pretty cool and intuitive. Great idea by a great tech website. Now I feel like I should be part of phone arena team.
39. pwnarena (Posts: 1001; Member since: 15 Feb 2013)
the only problem is that they don't seem credible. look at the battery numbers. sony phones get good scores in comparison to others that are known for having better battery longevity. i don't think that's accurate.
24. Tuxedo (Posts: 179; Member since: 19 Mar 2013)
1. When measuring output voltage of the headphone jack, you need to use a standard load (headset), otherwise, results will be meaningless.
2. I'd like to see a "sleep mode" battery life test... say how much battery is used in a given period of time (say 24hrs) in which the device is in sleep mode, radios on, sync on etc.
25. artealis (Posts: 98; Member since: 15 Dec 2010)
wow, finally camera speed test
please bring wifi strength test too PA, its important for day to day uses more than antutu benchmark number.
26. jellmoo (Posts: 725; Member since: 31 Oct 2011)
Very nice. It's good to see that some standards are being applied to reviews, rather than a vague description and "feel".
Two things I'd still like to see:
1) A better breakdown on battery life (which you may do, it just isn't shown here).
2) A better wrap up at the end of the review, where each category is properly weighted so that we can better understand why one phone gets a 9, and another a 9.4 for example.
27. Mahdi007 (Posts: 5; Member since: 12 Sep 2013)
It's fantastic but please include all the phones(at least important phones) for example:Google nexus 4 is missed from battery life test
30. ocilfa (Posts: 334; Member since: 03 Aug 2012)
Please include the Nook HD+. Great tablet at a great price.
31. _Bone_ (Posts: 2142; Member since: 29 Oct 2012)
These screen measurements are pretty well off, Ray. DisplayMate use 100% accurate tools for proper numbers and the difference in the S4's numbers stand out at most areas. It has 475nit peak brightness (100% white, auto mode), 0 nits black, 6,5k Kelvin (almost perfect) color temperature in Movie Mode and 2.42 gamma cause it's an AMOLED screen. Your numbers are well off.
Now as far as the benchmarks go, I recommend adding the new AnTuTu 3DRating for OpenGL ES 3.0, Basemark X (on and off screen), some memory meausrement and most of all, battery testing and it'll be cool.
32. Kevinphantom (Posts: 82; Member since: 17 Jun 2013)
nice but also I recommend adding heat (temperature of the device ) while using aggressive cpu power and aggressive gpu power
33. mrturner (Posts: 16; Member since: 10 Sep 2013)
Hmm...useful. Would love to how the iphone 5same performs against the best androids.
37. 14545 (Posts: 1158; Member since: 22 Nov 2011)
Why is it that NOT ONE REVIEW SITE actually measures a phones radio performance. You can look at the actual dbm to compare radio performance. As well as synthetic tests at specific places with all phones. This is just the tip of what a good review should actually consist of. Good start, but still no where near enough.
38. pwnarena (Posts: 1001; Member since: 15 Feb 2013)
your battery script is probably erroneous or inaccurate. it's impossible for the galaxy s3 to have lower longevity than the xperia z. i'm not really sure about these tools you're using. they don't seem reliable. they seem to be designed to favor sony devices.
40. XiphiasGladius (Posts: 805; Member since: 21 Aug 2011)
A welcome addition, keep it up P.A.
42. zachattack (Posts: 598; Member since: 31 Jul 2013)
What PA should do is update the G2's scores LOL. You honestly think the HTC One scores better than the G2 you must be on drugs
45. Retro-touch (Posts: 265; Member since: 24 Oct 2011)
Phone Arena why are you running GFX onscreen and not offscreen? To rule out pixel differences between phones I think it would be better to run it offscreen
Also Wi-Fi signal strength would be good to add if possible
46. hulkkii (Posts: 5; Member since: 19 Sep 2013)
Nice to have camera speed tests!
Here some notes about CamSpeed:
CamSpeed score is calculated from following measurements: camera start-up time, focus time and image capture time, so it presents the total time to take first picture. (Image saving to sdcard/disk is not part of measurements/score).
Info about latest app updates:
Galaxy S4 (exynos) should be measured again with CamSpeed (app version 2.1.2 or newer).
Samsung had little different implementation of Camera API's startpreview-call in Galaxy S4 than others causing little too fast camera start-up time results for it (and so a higher score than it should have), but the measurements for S4 were fixed in latest CamSpeed version.