Only one smartphone assistant can be considered the best after crushing its rivals in a new test

Only one smartphone assistant can be considered the best after crushing its rivals in a new test
Venture Capital firm Loup Ventures recently tested the three major virtual digital assistants by running a test. The Google Assistant, Amazon's Alexa and Apple's Siri were each given the same 800 questions and tasks and scored on their ability to both understand the query and answer it or respond correctly. The results? Google Assistant understood all 800 questions asked and answered 93% of them correctly. Siri understood 99.8% of the questions and correctly responded 83.1% of the time. Alexa nearly matched Google Assistant's perfect comprehension with a score of 99.9% but did the worst in terms of accuracy with a score of 79.8%.

All three digital assistants improved over the scores they had last year when Google Assistant answered 86% of the questions right, Siri was correct 79% of the time and Alexa gave the right response 61% of the time. The report, written by analysts Gene Munster and Will Thompson, notes an important factor; this test, like the one last year, measures the response of these digital assistants on a smartphone instead of a smart speaker. This is important because the lack of a screen could change the response on a speaker as opposed to a phone. Since this test was made using phones, the questions were shorter and the use of a screen allowed the digital assistants to answer some questions without having to verbally announce the response.

Test shows that Google Assistant is the best digital helper found on a smartphone




Each of the three digital assistants was asked questions broken down into five different categories: Local, Commerce, Navigation, Information and Command. Google Assistant had the best score for each one except for Command. The latter group of questions dealt with phone-related tasks such as email, texting, calendar, and music. That category was topped by Siri, which beat out Google Assistant by a score of 93% to 86%. Siri finished second in the Local category ("Where is the nearest book store?") and Navigation ("Which subway do I take to get downtown?") while finishing last in Commerce ("Order me a 24-pack of Coca-Cola and three boxes of M&M cones"). In that department, Alexa finished second while it also placed second in Information ("What time do the Yankees play tonight?"). Alexa, with its Amazon parentage, was the favorite in the Commerce department but was unable to top Google Assistant. Alexa also had the disadvantage of being an app rather than a native OS feature. This prevented it from doing well in the Command category.


The test was conducted over an iPhone running iOS 12.4, a Pixel XL with Android 9 Pie installed, and Alexa was tested using the iOS app. Interestingly, while Alexa finished last in accuracy, Amazon's digital assistant showed an 18 percentage point improvement over the last 13 months, the best showing among the trio. Google Assistant had a 7 percentage point increase in its score over the same time period while Siri did 5 percentage points better.


Since the first test was conducted in April 2017, Google Assistant and Siri have improved the most in the Commerce category. Alexa, which wasn't part of the first test, has seen its greatest improvement over the last 13 months in the Commerce category also.

FEATURED VIDEO

52 Comments

55. caelironstaff

Posts: 21; Member since: Nov 02, 2018

I use Microsoft's Cortana on my Sony Xperia phone (both the app and the baked-in version in the Microsoft Launcher), in my Harmon Kardon Invoke speakers, on my personal PC's, and even at work on company PC's. People are always amazed at how well Cortana works on all of those platforms. Sad to see her not rated here.

50. Demo-jay

Posts: 106; Member since: Feb 13, 2018

Apple fan or not..we both know siri sucks..standing up for it is just to look for unnecessary arguments..i use both android and apple..i prefer apple but honestly siri sucks compared to google assistance..

30. jeffpom

Posts: 69; Member since: Dec 11, 2016

This is not a definitive test - only from a standpoint of answering questions. What it does not take into effect is capabilities and features of those assistants. I love my iPhone and will defend iPhone overall - but Siri is still lacking in capabilities to Google or even Alexa. (Of course, even this could be subjective, based on what a person may want their assistant to be able to do for them.) Apple's hope seems to be that Siri shortcuts will help - that developers will help make Siri better, and that's entirely possible, but is not being brought to full potential at this time. Google's integration with their own products (email, calendar, to-do, etc...) is better, and the overall features are by far better than Siri. Most Apple lovers would even agree with this.

20. whatev

Posts: 2459; Member since: Oct 28, 2015

The best part is that iOS devices can have all those assistants, while the others...

24. sgodsell

Posts: 7673; Member since: Mar 16, 2013

Siri is still a POS. Plus iOS doesn't allow you to change system defaults. So you can't just change to another assistant. At least with Android your default is the best Assistant on the market. Plus if you like Alexa, Cortana, or anything else, then you can change the system default to use another Assistant. Also while the others don't have access to Siri, then they are definitely not missing out on using one of the worst assistants.

28. apple-rulz

Posts: 2198; Member since: Dec 27, 2016

Bixby

31. sgodsell

Posts: 7673; Member since: Mar 16, 2013

All Android smartphones with the Play store come with Google Assistant. Samsung defaults to Bixby. But at least if you press and hold the home button, then Google Assistant comes up. Others have put on Alexa. But Google Assistant is still there to bring up and use. If you actually used Android, then you would know this. The thing is Android gives you options and allows others to change the system defaults. Something that Apple doesn't allow. But your only Apple-foolz so I guess you don't know this kind of stuff.

32. apple-rulz

Posts: 2198; Member since: Dec 27, 2016

Samsung has Bixby.

37. sgodsell

Posts: 7673; Member since: Mar 16, 2013

You numb nut, Samsung smartphones also come with Google Assistant. I would tell you to actually look, but since you only own iPhones, then once again you like to look like a fool. Or basically an apple foolz.

41. apple-rulz

Posts: 2198; Member since: Dec 27, 2016

How do you feel about Samsung’s effort know as Bixby?

42. tedkord

Posts: 17529; Member since: Jun 17, 2009

So you're hyped that you can use the one that got 76% of info queries correct instead of the one that got 93%? And, you can't use them natively. I can use GA natively.

46. whatev

Posts: 2459; Member since: Oct 28, 2015

I can use all of them, as an iOS user I always get the best treatment from developers, I’m a first class citizen, you’re second class. Next time try to cry not so loud, the same goes to sgodsell, your whining is so loud I can’t even hear my thoughts

49. Blink85333

Posts: 2; Member since: Aug 17, 2019

Second class, lol. Apple doesn't enjoy an open platform, but Android does. Besides assistant, which Google wins, Apple can't beat an Android camera.

51. whatev

Posts: 2459; Member since: Oct 28, 2015

Son, take some English lessons and try not to use google translator so often please

16. dmdzordilla

Posts: 60; Member since: Jul 31, 2018

If Cortana was invested by Microsoft with R&D she would have been best

11. Charlie2k

Posts: 178; Member since: Jan 11, 2016

Cortana?

13. lyndon420

Posts: 6938; Member since: Jul 11, 2012

I remember her... the hype. And she was was killing it when it came to sports predictions. We rarely hear her name mentioned anymore....

22. Crispin_Gatieza

Posts: 3200; Member since: Jan 23, 2014

Cortana is being integrated with Alexa. It's been happening for some time now and it shows why Alexa has performed so much better since last year.

9. apple-rulz

Posts: 2198; Member since: Dec 27, 2016

Bixby never seems to be included much in these kinds of tests.

10. Alcyone

Posts: 650; Member since: May 10, 2018

How did one guess you'd mention anything Samsung? Doesn't matter both siri and bixby suck. Assistant will always be the best, and it'll be hard for Apple to ever catch up. Especially, after the latest fiascos, considering Apples lack of control on user statistics. I'd be more concerned that what you say and your conversations as a whole aren't as private as Apple states. Siri sucked on my iphone and I'm glad i ditched ios as daily use. Can't do my daily job with a walled garden and an inept helper. Btw, you just can't get Samsung off your mind? Jealous little boy.

23. sgodsell

Posts: 7673; Member since: Mar 16, 2013

Apple-foolz at least Samsung smartphones come with two voice assistants. Bixby and Google Assistant. You don't like Bixby, then you can easily use Google Assistant instead. Problem solved. At the end of the day Siri is a POS, and unreliable.

25. apple-rulz

Posts: 2198; Member since: Dec 27, 2016

At the end of the day Apple owns your mind, and it's hysterical.

33. sgodsell

Posts: 7673; Member since: Mar 16, 2013

You have to be a child, or you like to deflect just like Tim cook. Especially when something is not going Apple's way. This is a perfect example, Siri is a POS, and if customers think that all voice assistants are roughly the same, or they offer and do the same kinds of things. Then Siri is giving the other voice assistants a bad name, or better yet Siri is not helping in the cause to help promote voice assistants, and that's because Siri fails to much and cannot get the users results that they actually requested. Google Assistant on the other hand gets the user the results that they requested, and even presents it in a nice layout with display cards. Google Assistant even supports interactive canvas with support for HTML5, JavaScript, CSS all of which is missing from Siri, but then again Apples Siri doesn't even support smart displays. But you knew all this right it's me. I and many others know that Siri is a POS, and have come to terms with that fact. Now apple-foolz have you come to terms with that fact as well?

26. Alcyone

Posts: 650; Member since: May 10, 2018

No need for me to dry my tears lol. My voice assistant works on a more reliable basis than siri. If anything your trolling needs work, its kinda weak. Ask siri how to be a better troll. Siri won't even know what a troll is, much less even understand what you ask her. I'm sorry, your pos siri is a joke. Then, again I may laugh hard enough I might tear up. I miss screwing around with siri, all the nonsense responses were funny. Now, please, get off your moms pc, gotta be sure your ready for school.

18. sissy246

Posts: 7129; Member since: Mar 04, 2015

"How did one guess you'd mention anything Samsung" Samsung is in their brains Google for the win as always.

35. Vancetastic

Posts: 1965; Member since: May 17, 2017

See my query to your buddy Whatev.

39. tedkord

Posts: 17529; Member since: Jun 17, 2009

That's because Samsung users use the class leading Google Assistant on their class leading phones.

52. whatev

Posts: 2459; Member since: Oct 28, 2015

Class leading and Samsung in the same sentence, I have to laugh, thanks :)))

5. sgodsell

Posts: 7673; Member since: Mar 16, 2013

There is five categories for their tests. However how many questions were used for each category. If you have a lot more questions for one category, then that can skew the actual percentages. For instance the information category would have the most queries over every other category. Google Assistant got 96% of those queries correct. Alexa got 92% of those queries correct. Siri only got 76% of the queries correct. So you can't rely on Siri to get you the correct information. Also when you recalculate the 5 categories. You can see that the calculations were wrong. You get 93% for Google Assistant, which they listed 92.9% 93+92+98+96+86 = 465 465 / 5 = 93% Siri you get 82.4%, and it was listed at 83.1% 89+68+86+76+93 = 412 412 / 5 = 82.4% Alexa you get 78%, and it was listed at 79.8% 85+71+72+93+69 = 390 390 / 5 = 78%

3. lyndon420

Posts: 6938; Member since: Jul 11, 2012

Google was the most accurate ok, but I wouldn't say they 'crushed' the competition....these are much better scores compared to a year ago.

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.
FCC OKs Cingular's purchase of AT&T Wireless