Only one smartphone assistant can be considered the best after crushing its rivals in a new test

Only one smartphone assistant can be considered the best after crushing its rivals in a new test
Venture Capital firm Loup Ventures recently tested the three major virtual digital assistants by running a test. The Google Assistant, Amazon's Alexa and Apple's Siri were each given the same 800 questions and tasks and scored on their ability to both understand the query and answer it or respond correctly. The results? Google Assistant understood all 800 questions asked and answered 93% of them correctly. Siri understood 99.8% of the questions and correctly responded 83.1% of the time. Alexa nearly matched Google Assistant's perfect comprehension with a score of 99.9% but did the worst in terms of accuracy with a score of 79.8%.

All three digital assistants improved over the scores they had last year when Google Assistant answered 86% of the questions right, Siri was correct 79% of the time and Alexa gave the right response 61% of the time. The report, written by analysts Gene Munster and Will Thompson, notes an important factor; this test, like the one last year, measures the response of these digital assistants on a smartphone instead of a smart speaker. This is important because the lack of a screen could change the response on a speaker as opposed to a phone. Since this test was made using phones, the questions were shorter and the use of a screen allowed the digital assistants to answer some questions without having to verbally announce the response.

Test shows that Google Assistant is the best digital helper found on a smartphone




Each of the three digital assistants was asked questions broken down into five different categories: Local, Commerce, Navigation, Information and Command. Google Assistant had the best score for each one except for Command. The latter group of questions dealt with phone-related tasks such as email, texting, calendar, and music. That category was topped by Siri, which beat out Google Assistant by a score of 93% to 86%. Siri finished second in the Local category ("Where is the nearest book store?") and Navigation ("Which subway do I take to get downtown?") while finishing last in Commerce ("Order me a 24-pack of Coca-Cola and three boxes of M&M cones"). In that department, Alexa finished second while it also placed second in Information ("What time do the Yankees play tonight?"). Alexa, with its Amazon parentage, was the favorite in the Commerce department but was unable to top Google Assistant. Alexa also had the disadvantage of being an app rather than a native OS feature. This prevented it from doing well in the Command category.


The test was conducted over an iPhone running iOS 12.4, a Pixel XL with Android 9 Pie installed, and Alexa was tested using the iOS app. Interestingly, while Alexa finished last in accuracy, Amazon's digital assistant showed an 18 percentage point improvement over the last 13 months, the best showing among the trio. Google Assistant had a 7 percentage point increase in its score over the same time period while Siri did 5 percentage points better.


Since the first test was conducted in April 2017, Google Assistant and Siri have improved the most in the Commerce category. Alexa, which wasn't part of the first test, has seen its greatest improvement over the last 13 months in the Commerce category also.

FEATURED VIDEO

52 Comments

1. blingblingthing

Posts: 967; Member since: Oct 23, 2012

Apple fan logic - Apple's ecosystem is bigger, so Siri is the better choice.

6. sgodsell

Posts: 7377; Member since: Mar 16, 2013

The one category labeled information, or basically queries would most likely have the most queries. Google Assistant gets 96%, Alexa 92%, and Siri get 76%. So Siri sucks with information and query lookups.

12. Mike88

Posts: 415; Member since: Mar 05, 2019

I’m yet to see someone using virtual assistants in public places, it’s weird and doesn’t help much. I use Siri sometimes for showing my movie tickets and other passes via Siri shortcuts but that’s it. The use of these assistants is very limited and all of them are equally good in doing those limited tasks. They also pose a security risk so can’t be trusted for sensitive tasks.

21. sgodsell

Posts: 7377; Member since: Mar 16, 2013

Mike88 if all you use is crappy Siri, then you clearly expect to be disappointed, and are use to getting crappy results, a lot of the time. Now with Google Assistant it goes way beyond anything that Siri can do, and continues to get better all the time. Google Assistant doesn't need to wait for any OS updates, because GA has been decoupled from Android and platforms for years now. Plus developers can extend GA, and those apps don't have to be tied to any OS, platform, or hardware. Siri apps are locked down to iOS, and even then Apples Siri is too restrictive and limiting. Siri developers still to this day cannot make any standalone Siri voice apps for Apples one and only HomePod. GA developers plus are not limited to any voice category or domain like Siri is. GA developers can make make any voice app they want, including games, food and nutrition apps, doctor and checkup apps, hospital guidance and help apps, buy and sell apps, and a lot more. All of which Siri cannot do because Apple has voice restrictive domain limitations in place. With iOS 13 Siri will now support 12 domains or voice categories that developers can make their Siri voice app to fit under. GA doesn't restrict it's developer's, they can make any 3rd party voice app they want. I love how you bring up Siri Shortcuts which is a kludge for Siri and SiriKits deficiencies in the first place. Then you go on to say that the use of these assistants is very limited, and all of them are equally good in doing those limited tasks. Siri is most definitely not equally as good. Let's get that one out of the way. Third party Siri and SiriKit apps are very limiting, because they require that the user has installed their app. With GA users don't have to rely on any actual app to be installed on their smartphone, tablet, smart speaker, or smart display. Most voice apps for Google Assistant these days reside on the back end or server side. Something that Siri app developers cannot do. Then you go on to say they also pose a security risk so can’t be trusted for sensitive tasks. So Apple has decided for its developers what is safe and secure, and just limits their Siri voice assistant altogether. Okay we get it that Siri cannot do much compared to the other voice assistants. Got it! Oh, btw just in case you didn't realize users can quickly type in a query to GA, and get a response back. Also Mike88 anyone comparing Siri to the other voice assistants clearly doesn't know their head from their @$$, because Siri is a POS in comparison to the others.

29. Mike88

Posts: 415; Member since: Mar 05, 2019

Well that make’s sense. But tell me a few commands you use with your google assistant. What are those amazing things it can do and what commands do you use personally? Plus about your last line you see I didn’t compare Siri with any assistant, I called them all as equally good at doing tasks, you are the only one who compared Siri with google assistant and other voice assistants.

36. sgodsell

Posts: 7377; Member since: Mar 16, 2013

Mike88 there is restaurant voice apps. Like MCDONALD'S, where you just talk to Google Assistant to place your order, using nothing but your voice. It can even remember previous orders to make future orders quicker. The best part is if I change smartphones, then I don't have to worry about installing any app. It just works. "Talk to McDonalds". Starbucks is coming online soon, and many others. There is banking apps and more. I even check on my bank balances. Oh, and since your worried about security, then it has secure id with a fob that changes its 6 numbers every minute. So I could use your iPhone to check my bank balance, because the fob stays with me, and if you tried then it would fail. I can check on food and nutrition and even track some meals using nothing but my voice. I can even play some card games using once again nothing but my voice, and still no apps to install. The approach that Apple went with Siri, means that Siri is never going to catch up to Google Assistant. Siri from here on out should never be compared to Google Assistant, because Siri is a POS, and ultimately gives the other voice assistants a bad name, especially if they think voice assistants are all similar.

45. Mike88

Posts: 415; Member since: Mar 05, 2019

You can do most of those things with Siri too but I would never like google assistant or Siri to access my bank information,my balance and statements are sensitive info. You know what google is trying to do? It know how rich or how poor you are, it knows what and where you spends, it monitors everything you do in your life, no one should have that much information about someone.. And it’s not a big thing when you say I can do this thing using nothing but my voice, Someone without an assistant can do all that too, using nothing but his hand touches lol voice will not replace touch UI as you don’t need to shout everything you want to do with it

53. sgodsell

Posts: 7377; Member since: Mar 16, 2013

"You can do most of those things with Siri too." No you can't. First of all the way that Apple has implemented it's Siri voice assistant, you are limited to where she can run, and what she can do. Siri requires iOS apps, so that limits where these apps can run. The HomePods can't run any standalone Siri voice apps today. The HomePods are over a year an a half old. They can't even recognize a users voice. A Google Home mini can do that, but not Apple's overpriced HomePod. I know iOS 13 is going to add multiple accounts. But the Siri restrictions are still there. With Siri there is some banking apps, which are full blown iOS apps (Android has those as well). But these iOS banking apps might have some limited Siri integration to pay a bill or some other limited interaction, or you are scripted with a series of voice prompts to complete a transaction. Because Siri and SiriKit are not that robust and have to many restrictions when it comes to voice interactions. Siri even has to many restrictions on its intents. So not really a true voice banking app at all. With Siri it's just as easy to open up the banking app, and tap on the screen, then to try and use Siri. Like you said. Now I heard you that you don't want GA or Siri to ever have access to your banking information. I totally get it that it's not for you, and you have trust issues. I don't totally trust banking and banking apps on my smartphones either. It's why I have multiple banks and multiple accounts. Some are exposed to some online banking, and most are not. But at the end of the day Google Assistant gives me the choice to try online banking using nothing but my voice. Plus you can do this from a smartphone, tablet, smart speaker, smart display, smart watch, or car. Seeing as how it works, that is why I said I would use your iPhone or any smartphone to do my banking as long as GA is there, and I have my fob with me. However with Google Assistant I can talk like I normally talk to any person, and I don't have to wait for that person to feed me with specific voice prompts that lead me down a certain path in order to complete a transaction. That's where Siri is failing, because now you have to learn how to talk to Siri and that banking app. That's not how things should work with your voice. This is where Google Assistant and the banks voice app have nailed it. A user can say in one sentence that has the complete transaction, or if that users sentence is missing anything to complete that transaction, then like a teller, the user will be prompted for any missing or needed information to complete the transaction. It's like talking to a real teller or person at a bank. This holds true for GA in general. Siri needs to open up a lot more to developers, and Siri needs less restrictions. Otherwise Siri is always going to be last place, especially compared to Google Assistant.

43. Matte_Black13

Posts: 26; Member since: Oct 12, 2018

Lol. Ask a question and it actually give you an answer, such as “when does [insert store] open?” And it gives you a time instead of STILL giving you a link. Siri is dumb. Sincerely, and iPhone user who is probably going back to Android.

2. cmdacos

Posts: 4216; Member since: Nov 01, 2016

For Siri to actually do well these were clearly softball questions.

4. lyndon420

Posts: 6793; Member since: Jul 11, 2012

They dumbed down everything so Siri and Alexa could appear smarter? I suppose that's possible...lol But seriously though...I'm sure they've been working hard to improve their AI, and maybe this is a genuine result of their efforts.

7. sgodsell

Posts: 7377; Member since: Mar 16, 2013

It is, especially since we don't know how many of those 800 quests were split between these 5 categories. That alone can skew the actual percentages.

17. whatev

Posts: 2276; Member since: Oct 28, 2015

Sadly they didn’t dump down everything enough to include Bixby :( It’s so far behind its counterparts or maybe it’s so stupid that it can’t be considered an assistant :((((

34. Vancetastic

Posts: 1365; Member since: May 17, 2017

Samsung still keeping you awake at night?

47. whatev

Posts: 2276; Member since: Oct 28, 2015

Not, it’s actually your mother who calls me like at 3 am to come to my bed

54. Vancetastic

Posts: 1365; Member since: May 17, 2017

Phone calls from the grave. Sounds like a Twilight Zone episode.

38. tedkord

Posts: 17358; Member since: Jun 17, 2009

Luckily, anyone who had Bixby can simply use Google Assistant, and they aren't stuck with an also ran like Siri. Another reason Samsung phones have been the best in the market the past five plus years.

48. whatev

Posts: 2276; Member since: Oct 28, 2015

I can have GA, Alexa, and others, Samsung’s wet dreams are to put Bixby inside an iPhone but that’s never gonna happen, and it’s time for you to wake up and stop sucking all that bs those so called youtubers and tech reviewers throw at you, they all use iPhones as their main device, they only praise Samsung because they are paid, Samsung can’t even make a celebrity comply with a contract, WAKE UP!!!!

3. lyndon420

Posts: 6793; Member since: Jul 11, 2012

Google was the most accurate ok, but I wouldn't say they 'crushed' the competition....these are much better scores compared to a year ago.

8. sgodsell

Posts: 7377; Member since: Mar 16, 2013

The one main category for information or query lookup for information. Siri only got 76% of the correct results. That makes Siri unreliable. Google got 96% of the correct results. Alexa got 92% correct.

44. Matte_Black13

Posts: 26; Member since: Oct 12, 2018

Not sure how Siri got the scores it did, and I am an iPhone user. Generally when are you Siri I am the cringing and pining for GA. The GA that can be used with the iPhone is very dumbed down because of the nature of iOS

5. sgodsell

Posts: 7377; Member since: Mar 16, 2013

There is five categories for their tests. However how many questions were used for each category. If you have a lot more questions for one category, then that can skew the actual percentages. For instance the information category would have the most queries over every other category. Google Assistant got 96% of those queries correct. Alexa got 92% of those queries correct. Siri only got 76% of the queries correct. So you can't rely on Siri to get you the correct information. Also when you recalculate the 5 categories. You can see that the calculations were wrong. You get 93% for Google Assistant, which they listed 92.9% 93+92+98+96+86 = 465 465 / 5 = 93% Siri you get 82.4%, and it was listed at 83.1% 89+68+86+76+93 = 412 412 / 5 = 82.4% Alexa you get 78%, and it was listed at 79.8% 85+71+72+93+69 = 390 390 / 5 = 78%

9. apple-rulz

Posts: 2120; Member since: Dec 27, 2016

Bixby never seems to be included much in these kinds of tests.

10. Alcyone

Posts: 406; Member since: May 10, 2018

How did one guess you'd mention anything Samsung? Doesn't matter both siri and bixby suck. Assistant will always be the best, and it'll be hard for Apple to ever catch up. Especially, after the latest fiascos, considering Apples lack of control on user statistics. I'd be more concerned that what you say and your conversations as a whole aren't as private as Apple states. Siri sucked on my iphone and I'm glad i ditched ios as daily use. Can't do my daily job with a walled garden and an inept helper. Btw, you just can't get Samsung off your mind? Jealous little boy.

23. sgodsell

Posts: 7377; Member since: Mar 16, 2013

Apple-foolz at least Samsung smartphones come with two voice assistants. Bixby and Google Assistant. You don't like Bixby, then you can easily use Google Assistant instead. Problem solved. At the end of the day Siri is a POS, and unreliable.

25. apple-rulz

Posts: 2120; Member since: Dec 27, 2016

At the end of the day Apple owns your mind, and it's hysterical.

33. sgodsell

Posts: 7377; Member since: Mar 16, 2013

You have to be a child, or you like to deflect just like Tim cook. Especially when something is not going Apple's way. This is a perfect example, Siri is a POS, and if customers think that all voice assistants are roughly the same, or they offer and do the same kinds of things. Then Siri is giving the other voice assistants a bad name, or better yet Siri is not helping in the cause to help promote voice assistants, and that's because Siri fails to much and cannot get the users results that they actually requested. Google Assistant on the other hand gets the user the results that they requested, and even presents it in a nice layout with display cards. Google Assistant even supports interactive canvas with support for HTML5, JavaScript, CSS all of which is missing from Siri, but then again Apples Siri doesn't even support smart displays. But you knew all this right it's me. I and many others know that Siri is a POS, and have come to terms with that fact. Now apple-foolz have you come to terms with that fact as well?

26. Alcyone

Posts: 406; Member since: May 10, 2018

No need for me to dry my tears lol. My voice assistant works on a more reliable basis than siri. If anything your trolling needs work, its kinda weak. Ask siri how to be a better troll. Siri won't even know what a troll is, much less even understand what you ask her. I'm sorry, your pos siri is a joke. Then, again I may laugh hard enough I might tear up. I miss screwing around with siri, all the nonsense responses were funny. Now, please, get off your moms pc, gotta be sure your ready for school.

18. sissy246

Posts: 7120; Member since: Mar 04, 2015

"How did one guess you'd mention anything Samsung" Samsung is in their brains Google for the win as always.

35. Vancetastic

Posts: 1365; Member since: May 17, 2017

See my query to your buddy Whatev.

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.