Latest test reveals the top virtual digital assistant; do you have it on your device?

Latest test reveals the top virtual digital assistant; do you have it on your device?
Every so often, someone feeds a number of virtual digital assistants a number of questions to determine which one is the best. Among the four major AI-based helpers, Google Assistant usually comes out on top with Apple's Siri, Amazon's Alexa and Microsoft's Cortana ranking behind the leader. And this seems to be the case regardless of whether the testing is done on a smart speaker or a smartphone.

Last week, Perficient digital conducted the most recent battle of the virtual assistants and asked 4,999 questions to Alexa, an Echo Show, Cortana, Google Assistant on a Google Home, Google Assistant on a Google Home Hub, Google Assistant on a Smartphone and Siri. The scoring was based on a number of factors including whether the response was made verbally, if the answer came from a database or from a third-party source, if the assistant did not understand the question, and if the answer was just flat-out wrong.

You're almost better off flipping a coin to get a correct response instead of relying on Cortana


The results? Google Assistant on a smartphone answered nearly 90% of the questions thrown at it "fully and correctly." Perficient explains that if it asked one of the virtual helpers "How old is Abraham Lincoln?" and was shown only his birthdate, that would not be a fully correct answer to the question. Google Assistant on the Google Home smart speaker was next with an 85% accuracy score and Assistant on a Home Hub finished third with a score of 82%. Siri followed with a 70% score with Alexa (67%) right behind. Cortana's score, slightly better than 50%, means that you're almost better off flipping a coin to get an answer.


Interestingly, Cortana did attempt to answer the most questions which could partially explain the low accuracy score. Perficient explains this as the percentage of questions where "the personal assistant thinks it understands the question and makes an overt effort to provide a response. This does not include results where the response was 'I'm still learning' or 'Sorry, I don't know that' or responses where an answer was attempted, but the query was heard incorrectly."


This is the third year that Perficient has run a similar test and looking at the results of the last three years one might get the impression that digital assistants have been getting dumber. Outside of Cortana, which fully and correctly answered over 90% of the queries thrown at it last year, the digital helpers taking part in all three contests have seen their score decline each year. What hasn't changed over the three years are the results at the top. The smartphone version of Google Assistant consistently beats out the other contenders.


During this year's testing, Siri had the most incorrect answers amounting to nearly three times the number of incorrect responses from the smartphone version of Google Assistant. Accord to Perficient, about a third of the wrong answers from Siri and Alexa, came from questions that it considered to be "poorly structured." An example would be "What movies do The Rushmore, New York appear in?" Additionally, most incorrect responses were so obviously wrong that a user would know not to believe the answer.

If you're curious about which of the personal assistants has the best sense of humor, the test shows that Siri and Alexa are neck and neck at the top. As an example, when Siri was asked "Where is Waldo?," Apple's digital helper responded, "He's well hidden. I can't see him." The most serious of the assistants was Cortana.

Lastly, Siri and Alexa used "snippets" the least. These are responses sourced from a third-party and are generally attributed to them. The overall trend among virtual assistants is lower but Google Assistant (on every device), by far, delivers such answers the most.

FEATURED VIDEO

4 Comments

3. monoke

Posts: 1193; Member since: Mar 14, 2015

Is it just me or do u ask it questions u already know the answers to cuz u can't think of any other questions?

5. MsPooks

Posts: 214; Member since: Jul 08, 2019

I think the idea of virtual assistants turned out to be better than the execution. Not many of us need assistants to begin with, and the people who do aren't likely to trust the likes of an ad company (Google) with their sensitive business info. So most of us end up using them as audible search tools.

7. Phullofphil

Posts: 1829; Member since: Feb 10, 2009

I use mine when I don’t want to use my hands or my hands are doing something else. They are good for a lot of stuff around the house to. They are nothing more. You want to know something important get on with it your self and read watch it.

9. leonho708

Posts: 3; Member since: Oct 27, 2019

Alan Friedman, maybe you should start learning how to read a graph before BS and try to make Siri look better. "Siri followed with a 70% score with Alexa (67%) right behind." Alexa & Echo Show both have nearly 80% score, which makes Siri down to 6th, just better than Cortana. And Siri attempted the least questions, just over 40%, way behind every other assistant.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.