Siri finishes last in test against other smart speaker virtual assistants

Siri finishes last in test against other smart speaker virtual assistants
A smart speaker is made up of a speaker and a virtual assistant. Apple finally got around to launching its HomePod smart speaker yesterday, and while it delivers what users are calling a premium audio experience, the device is apparently falling short in the virtual assistant department according to a test run by a venture capital firm named Loup Ventures. The test featured four virtual assistants including Apple's Siri, Amazon's Alexa, Google Assistant and Microsoft's Cortana.

In a report released on Saturday written by former Piper Jaffray analyst Gene Munster, it was disclosed that the aforementioned quartet of virtual assistants were asked 782 queries. HomePod's Siri answered 52.3% of the questions correctly. That placed it dead last among the four smart speaker assistants. Slightly above Siri, answering 57% of the questions correctly, was the Microsoft Invoke's Cortana. Alexa, found on the Amazon Echo line of smart speakers, had the right answer 64% of the time. And Google Home's Google Assistant was on the top of the list as the virtual assistant answered 81% of the queries correctly.

Munster pointed out that part of the problem with Siri is that on the HomePod, Apple's virtual assistant cannot handle tasks or answer questions related to navigation, calendar, email, and calling. Siri would answer questions related to these "domains" with a stock response that said, "I can’t ___ on HomePod." Removing queries from these domains took Siri's accuracy from 52.3% to 67%. That score would have placed it above Alexa and Cortana.

The analyst said that Siri understood 99.4% of the questions asked, and despite answering the fewest questions correctly among the four assistants tested, he wrote "Overall, Siri performed above our expectations given the limited scope of HomePod’s music focus."

The report adds that HomePod has better listening abilities compared to other smart speakers, and that Siri sounds more human on the device than on the iPhone. Communications with Siri are more human-like according to Munster, since she doesn't repeat every query like Alexa and Google Assistant both do. Apple was also praised for the packaging of the device.


A survey conducted by Loup Ventures suggests that demand for the product will be similar to demand for the Apple Watch. Munster expects that HomePod will grab 12% of the market for smart speakers in 2018, compared to a leading 52% for Amazon's Echo, and 32% for the Google Home. Other units will nab 4% of the market.

source: LoupVentures via AppleInsider

FEATURED VIDEO

25 Comments

24. izim1

Posts: 1599; Member since: Feb 04, 2013

I see youre still repeating this nonsense... the echo and GA work exactly the same in that they pick up key words in a command, no matter the sequence. You can speak naturally to both.

33. RebelwithoutaClue unregistered

Both Amazon and Google have a similar thing; Amazon calls them skills, Google calls them actions. They are pretty much similar, but the number of currently available skills are a lot greater than available actions. Also, Alexa is a little bit better integrated in the variety of home appliances.

4. Zylam

Posts: 1816; Member since: Oct 20, 2010

Oh the hard on all the fandroids probably got just by reading the title.... Have to admit, turned me on too. Siri sucks.

7. GreenMan

Posts: 2697; Member since: Nov 09, 2015

A few months ago; I saw a comparison review between Google's Assistant and Siri conducted by popular YouTuber Marques Brownlee. https://www.youtube.com/watch?v=JFiu5rfnhzo Siri used to have a sheer edge over Cortana and Google Now but... Alexa and Assistant has really upped the game as seen in this chart! In any case; I don't think I'm going to own any of these smart speakers any time soon! My S6 is enough technology for me! G'Day!

8. Sparkxster

Posts: 1228; Member since: Mar 31, 2017

I agree don't have any use for smart speakers yet.

9. BullDozer

Posts: 154; Member since: Dec 18, 2015

Siri is just like all software from Apple. Very basic and limited.

10. S4NDY

Posts: 240; Member since: Mar 14, 2016

and full of strange bugs just like its hardware.

11. torr310

Posts: 1659; Member since: Oct 27, 2011

I don't feel like any voice-activated devices... I like it quiet at my space except listening to music or watching movies. They are noise pollution IMO.

18. lyndon420

Posts: 6788; Member since: Jul 11, 2012

Agreed. The VA on my phone is enough for my needs. There are no decent smart watches on the market either to justify spending money on.

12. cmdacos

Posts: 4200; Member since: Nov 01, 2016

Did we really need an article for this? The world is aware of Siri being two years behind...

22. ViewRoyal

Posts: 10; Member since: Feb 11, 2018

" The world is aware of Siri being two years behind" Actually... No. See the test results of Siri (on iPhone and iPad) by Loop Ventures shown in the comment below. Siri on the HomePod has been limited in its initial version, but will be just as proficient as Siri on iPhone and iPad, when it is updated in the near future. The skewed opinion that Siri in general is "behind" other AI assistants, is a fallacy that has been proven by methodical tests to be wrong.

14. jacksmith21006

Posts: 68; Member since: Nov 03, 2015

This test is consistent with our experience. We started with the Echo but since replaced with Google Homes. We do have Siri on our phones but it is so horrible rarely use it. But the big difference in these is not discussed nearly enough. Siri and the Google Assistant try to use natural language instead of commands like Alexa. So if willing to memorize the commands for the Echo it works extremely well and better than Siri. It is taking your words and looking it up in a data base or doing very good voice recognition. But Google and Apple are doing the next step and natural language and trying to understand what you are asking. Google is really good at it and Apple is not. What will be interesting to see is how long it takes Amazon to catch up and get the Echo working with natural language. I am really surprised it still has not happened and suspect a big part is the web data that Google has makes it possible. Years ago Larry Page, founder of Google, was asked about using AI to make search better and he replied they were going to use search to make AI better.

25. izim1

Posts: 1599; Member since: Feb 04, 2013

You. Dont. Need. To. Memorize. Specific. Commands. On. The. Echo. For the billionth and last time, man. Why do you keep repeating this nonsense? Seriously? Do you sell GA speakers? You own google stock? I own both and rely mostly on GA because of its integration with my google play accounts, but Alexa turn my light on, alexa turn on my light, alexa my light on, all work exactly the same. There is no specific command.

17. darkkjedii

Posts: 31034; Member since: Feb 05, 2011

We have an Amazon Alexa, and won't be changing it anytime soon.

19. Crispin_Gatieza

Posts: 3137; Member since: Jan 23, 2014

Don't fall asleep on Cortana. The Invoke is a first release and very crude. Cortana on phones is phenomenal.

20. RoryBreaker

Posts: 218; Member since: Oct 11, 2015

Google really got it right with the Home. Bought one for my mom & she is using it more than I ever thought she would.

21. ViewRoyal

Posts: 10; Member since: Feb 11, 2018

We have known since the release of the HomePod that many of Siri's capabilities and commands that ARE available on iPhone and iPad, are NOT YET available on the HomePod. So it is NOT unusual that Siri on the HomePod (only) scores low. But the missing Siri features will be coming in upcoming software updates. In fact, the same organization that did this test, Loop Ventures, also did testing of Siri versus the various other AI assistants in December 2017, using 800 questions divided into five categories. Here are the test results: Understood Query: Apple Siri: 99.0% Amazon Echo: 97.9% Google Home: 99.9% MS Cortana: 98.9% Answered Correctly: Apple Siri: 75.4% Amazon Echo: 63.8% Google Home: 81.1% MS Cortana: 56.4% Siri (on iPhone and iPad) is currently only slightly behind Google Home, but well ahead of both Amazon Echo and MS Cortana. And Siri's ratings have been consistently improving. We can expect the same high level of performance from Siri on HomePod in the very near future.

27. worldpeace

Posts: 3133; Member since: Apr 15, 2016

Slightly behind? lol... Based on your number, Siri fail to understand query 10 times more than Google Home. And Google Home give around 45 more correct answers based on that test. (not exact number, too lazy to use calculator). Google Home also have more natural voice, which is far better than Siri. Siri is improving (like you said), but don't forget that Google Assistant is also improving.

31. cheetah2k

Posts: 2256; Member since: Jan 16, 2011

I just don't get Apple some times. They're last to the party implementing 12-18 month old technology, and then they consistently don't do it very well, or haven't learnt from the lessons of their competitors.. Everything Apple sells lately wreaks of skank.. like a used jock strap that's missed the wash after a football match and worn again 6 months later in an emergency when the other clean ones are no where to be seen...

32. midan

Posts: 2871; Member since: Oct 09, 2017

"Google Home also have more natural voice, which is far better than Siri. Siri is improving (like you said), but don't forget that Google Assistant is also improving." Actually siri is now most natural sounding AI.

34. Back_from_beyond

Posts: 1420; Member since: Sep 04, 2015

No, it really isn't. It's improved a hell of a lot though. But still a bit behind.

30. cheetah2k

Posts: 2256; Member since: Jan 16, 2011

So what you're saying is that Apple released another half-a55ed product? I didn't see that one coming haha

28. Joosty

Posts: 473; Member since: Mar 14, 2013

Love my iPhone X but Siri effin blows.

29. cheetah2k

Posts: 2256; Member since: Jan 16, 2011

Geesus someone's got the iBlinkers on.. been butt raped already by an apple and loving it even more. Apple - its totally irrelevant

* Some comments have been hidden, because they don't meet the discussions rules.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.