Business Insider/Yu Han
If you’re a major tech company in 2016, chances are you have some sort of chatty virtual assistant.
As the Amazon Echo continues to be a runaway hit, all the big players are convinced that talking to an AI will soon become the dominant way we interact with our computers.
So they’ve started building. Apple has Siri, Amazon has Alexa, Microsoft has Cortana, and Google has the new and refreshed Google Assistant.
The tech has come a long way, but all of these companies openly admit that it’s very early days for this proposed future. As such, all of these assistants are far from polished.
But which one works best? I strapped in for 8 hours of robot conversations to find out, testing each of the big four assistants across a variety of categories. Here’s how it went:
Disclosure: Jeff Bezos is an investor in Business Insider through hispersonal investment company Bezos Expeditions.
THE SETUP: There’s no perfect way to evaluate a talking AI database, let alone four of them, but I tried to cover as many fundamental topics as I could.
In the end, I settled on 12 broad categories I’d want my assistant to nail:
• Travel
• Email
• Messaging
• Sports
• Music
• Weather
• Calendaring
• Social
• Translation
• Basic Tasks
• General Knowledge
• Personality
Within those categories, I asked a handful of questions and commands that seemed like they would be common.
It’s also important to note which devices I used, since that can have an effect on how well each assistant performs technically.
• For Siri, I used an iPhone SE.
• For Google Assistant, I used a Google Pixel XL.
• For Cortana, I used a Lenovo ThinkPad X1 Yoga.
• And for Alexa, I used an Amazon Echo Dot.
It’s worth mentioning that Amazon is currently rolling out Alexa to Kindle Fire tablets, but I wasn’t able to get that update before writing this piece.
With that out of the way, onto the tests.
The first category was TRAVEL. After supplying each assistant with my New Jersey address, I asked, “How do I get to the Port Authority Bus Terminal?”
Google Assistant made things the easiest. It knew exactly what I was talking about, and immediately opened up a route in Google Maps, as if I had already searched it in there. It put me on my way within seconds.
Siri got the idea, but in a roundabout manner. It showed a card with six different results, and weirdly said, “One option is Port Authority Bus Terminal on New York in New York. Do you want that one?” After confirming, though, it did the same thing as Google, just with Apple Maps — which is better, but still not the best.
Being stuck in a Bluetooth speaker, there wasn’t much Alexa could do. It did use Yelp data to identify the right place, tell me its location, and tell me its hours, though it didn’t give any mapping info in its companion app.
Cortana pulled up a list of three potential places, then asked which one I wanted. When I said “the first one” (which was correct), it did a Bing video search for the phrase “the first one.” Not great! When I clicked on the right result, though, it opened up a route in Windows Maps, which is fine, but not on Google’s level.
Winner: Google Assistant
“Is there any traffic on the way to the Empire State Building?”
Siri and Google Assistant did the best: Both verbally explained the delays and estimated how long it’d take to get there from my location. I’ll give Google Assistant the edge for opening up a route in Google Maps for good measure.
Cortana merely brought up a widget full of info about the Empire State Building; I had to click on a “directions” button to see what I wanted.
Alexa, meanwhile, only provided an estimate to one specific address at a time, which you have to configure within the Alexa app.
Winner: Google Assistant
“Book me a flight to Boston.”
Nobody really nailed this, but again, Google Assistant was the best of a mediocre bunch. It brought up a Google Flights card with various airlines and starting ticket prices. You can click on it to schedule as needed, but it weirdly defaulted to flights happening two weeks from now. If you specify you want the “next flight,” it’ll show more recent results, but you also can’t be too conversational with it: When I tried to say “I’ll take the first one,” it didn’t understand.
That’s still better than the others, though.
Siri just gave Bing results, Cortana opened up a Bing search in Microsoft’s Edge browser, and Alexa can only track and search for flights through a downloadable Kayak add-on, not book them.
Winner: Google Assistant
“Call me an Uber.”
Siri did well here. If you enable the Uber app, it’ll ask what kind of ride you want (UberX, UberBlack, etc.), then bring up an Apple Maps widget showing the closest car, ETA, and an estimated cost. Confirm that it’s got the right details, and it sends the car on its way. It’s more or less exactly what you’d want.
Cortana has it down, too. Connect your Uber account, say the type of car and destination you want, and it sets everything up.
Alexa is a bit more finicky, but if you download the Uber “skill,” it works in much the same way. Being confined to a speaker just makes it a little more uneasy for the time being.
Google Assistant is the outlier. It can’t do any of this today, but that should change once Google opens up the platform to third-parties in December.
Winner: Siri (with Cortana and Alexa close behind)
Next, let’s see how easy it is to EMAIL.
“Send an email to Jeff Dunn.” (Yes, I have multiple addresses.)
Siri and Google Assistant were neck and neck. Each one recognized my name in my contacts, had me dictate the message, and sent it out quickly. Siri let me specify a subject line, Google Assistant had a richer interface. The major difference is that Siri only works with Apple Mail, while Google uses Gmail. I prefer the latter.
Cortana can do most of the same things, but I wasn’t able to get it to work as smoothly. It repeatedly identified me as “Jeff done” — it is my curse — and often asked if I wanted to text someone rather than email.
Alexa couldn’t recognize the question. There are hacky workarounds you can create with IFTTT to get something close to the idea, but obviously that’s not ideal.
Winner: Google Assistant
“Do I have any new emails?”
Siri took this one. It told me how many unopened emails I had received since yesterday, then displayed those — and only those — in a simple Mail widget.
Google Assistant didn’t say anything, and brought up a card with two not-quite-recent emails from a few months ago. Click on one, and it brings up a weirdly isolated Web page for that specific email, not the Gmail app.
Alexa had the same problem as above, and Cortana suggested I make a Web search for the phrase “Do I have any new emails.” No thanks.
Winner: Siri
“Read my most recent email.”
Same as above. Siri is still tied to Mail, but it immediately brought up my last email, then read me the sender, date, subject line, and message — all hands-free. It awkwardly cut itself off where the text did, however.
Google Assistant didn’t actually read the message; it just displayed my last two emails in another info card. And again, it doesn’t give a direct link to the Gmail app.
Alexa still couldn’t do anything. And once again, Cortana opened a Bing search for the phrase “Read my most recent email.”
Winner: Siri
Next, let’s look at MESSAGING.
“Send a text to Jeff Dunn.” (Yes, I have multiple numbers.)
Siri, Google Assistant, and Cortana all recognized the name, then made it easy to see and send a message.
Alexa has more third-party workarounds for this, but it’s not as straightforward as using SMS. Given it’s stuck in a Bluetooth speaker, though, I can give it a little more of a pass.
Winner: Siri, Google Assistant, and Cortana
“Do I have any new texts?”
Siri was the only one to work here. Not only did it read off the sender and message, it also asked if I wanted to reply, and walked me through a new text from there.
Google Assistant told me it couldn’t read texts. This is bad — not because it seems fairly basic, but because the main Google app can do it already. It works just as well as Siri there, but the fact that it’s siloed off is strange.
Again, I didn’t use Alexa and Cortana on a phone, so I wasn’t able to make any headway here.
Winner: Siri
“Read me my Tweets.” (To test third-party integrations, and because I needed some darkness in my life.)
The phrasing threw Siri off. It is capable of reading tweets, but only if you ask something like “What’s trending on Twitter?,” or “Search Twitter for #NewMacBook.” In that case, it’s okay, but poorly curated.
Alexa is the most useful here — provided you download the Twitter Reader skill. Then, it can read the last tweets you liked, trending tweets, the last tweets to mention you, and so on.
Cortana didn’t recognize the phrase, either, but is at least capable of sending tweets if you use a phrase like “Twitter, new tweet” beforehand.
Google Assistant gave me a Google search result, though it can share an article to Twitter, and again, might get better as Google opens up the platform.
Winner: Alexa
Okay, let’s move onto something more fun: SPORTS.
I started with an easy one, asked during the middle of Game 5 of the World Series: “Who’s winning the Cubs game?”
Siri, Google Assistant, and Alexa churned out the current score with aplomb.
Cortana, on the other hand, brought up links to articles about the Cubs.
Winner: Siri, Google Assistant, and Alexa
“Who do the Celtics play next?”
Everyone got it right. Hooray!
Winner: Everyone!
After asking the previous question, I asked, “Who is their top scorer?” to see if these assistants understood context.
Nobody could do it. Not hooray! Worse, everyone but Google Assistant couldn’t recognize the word “scorer” without multiple tries. Instead, I got a lot of “top score.”
Winner (or “least loser”): Google Assistant
“Who won the Premier League?”
The correct answer is Leicester City. Unfortunately, none of the assistants appear to be footy fans. Even when I specified “Who won the Premier League last year?,” nobody got it right.
Siri brought up the current league table. Alexa had no idea. Cortana pulled up the latest scores. And Google Assistant thought it’d be a good time to read the Wikipedia entry for the Premier League in general. It wasn’t.
Winner: Nobody!
“Which quarterback started the last game for the Cleveland Browns?”
For the record, the answer here is Josh McCown.
Only Siri was able to outright say that, but it did so with the response above, where it said “Josh McCown” five straight times, before reading off the rest of the Browns roster. This, as you can imagine, was horrifying.
Google Assistant served up a card with Wikipedia info on past Cleveland Browns QBs, while Cortana called on its old friend Bing to show QBs on the current Browns roster.
That’s still better than Alexa, though, which rattled off names like Brady Quinn, Jeff Garcia, and Tim Couch — all of whom played for the Browns years ago.
Winner (begrudgingly): Siri
Now onto MUSIC.
I first asked each assistant to play a song: “Queen Jane Approximately” by Bob Dylan.
This one was tricky. No one assistant works with every music streaming service, which leaves you walled into specific gardens. It’s a similar deal with video.
Apple is the worst about it. Siri works harmoniously with Apple Music, but that’s the only music service it supports.
Google Assistant is a little better, if only because it has some support for Spotify, but it still really wants you to use Google Play Music or YouTube. (I use the former, so it quickly started a “Radio” playlist based on my song).
Cortana is similar — just replace Google’s stuff with Microsoft’s little-known Groove Music service.
Relatively speaking, Alexa made things the easiest. It defaults to Amazon’s new Music Unlimited service, but is also the most realistic about Spotify’s dominance. It even lets you make Spotify your default player.
Winner: Alexa
“Play me some new music.”
Nobody impressed here, but Alexa came closest. Instead of playing entirely new albums, though, it started the last music I added to my Amazon Music library. Playing just a bit loose with the phrasing there, Amazon.
Still, that was better than the rest.
Siri merely shuffled my library of songs, Cortana searched Bing for music videos based around the phrase “play me new music,” and Google Assistant hilariously opened up the album “New” by Paul McCartney in Google Play Music.
Winner: Alexa
Then I played a song off my laptop (Wilco’s “War on War”), and saw who could correctly identify it.
Cortana and Siri caught it — the former then gave a link to buy through Groove Music, while the latter, weirdly, gave a link to Shazam. (Since it’s using that tech to find the song.)
Alexa and Google Assistant were incapable — although, Alexa can almost always identify music if it’s playing out of the speaker itself. This was a bit troubling for Google Assistant since, again, it’s something the main Google app can already do.
Winner: Siri and Cortana
Let’s move to WEATHER.
“What’s the weather going to be like tomorrow?”
Everyone got the point across, but Siri was the weakest since it didn’t explicitly say what the forecast would be. Instead, it says something like, “looks like sun tomorrow,” and forces you to look at your phone to see the little Weather Channel widget it brings up. The point of these things is to go hands- and screen-free.
Winner: Google Assistant, Alexa, and Cortana
A conversational follow-up: “What about Boston?”
Google Assistant and Cortana were able to keep up and provide similar forecast info. Siri and Alexa didn’t understand the follow-up, and required me to outright ask “What’s the weather going to be like in Boston?”
Winner: Google Assistant and Cortana
“Will I need an umbrella this week?”
Everyone but Cortana got the gist. (It didn’t say anything; it just pulled up a graph with weather info.)
Of the other three, Google Assistant was the clearest. It was the only one to take the next step and specify which day to expect rain, whereas Siri and Alexa only noted if rain was expected at all over the next seven days.
Winner: Google Assistant
Now it’s time for CALENDARING.
“Do I have anything scheduled tomorrow?”
Everyone recognized just how many events I had on my calendar, then read them back to me. It’s mostly a tie. Though it’s worth noting how Cortana and Siri try to push you toward their companion apps, while Alexa and Google Assistant want you to use Google Calendar.
Winner: Google Assistant and Alexa
Then, as a follow-up, “How about the rest of this week?”
Surprisingly, only Siri was able to connect the dots. It quickly expanded the calendar widget from the last question to include my whole week, then read it out.
Google Assistant and Cortana needed me to clarify “what’s on my calendar this week?,” while Alexa told me the query wasn’t currently supported.
Winner: Siri
I asked each assistant to “Schedule dinner with Mom on my calendar Wednesday night,” then saw how easy it was to change and cancel that appointment after it was made.
Siri killed it here. It recognized “Mom” from my contacts list, asked when I wanted the appointment to start, then was entirely fast and straightforward when I asked to confirm, change, and eventually cancel it. Again, though, it defaults you to Apple’s stock app — without Calendar, there’s little you can do.
The other three were able to schedule things easily enough, but struggled when it came time to modify the date.
Google Assistant was very smooth at creating an event, but wouldn’t change it from the app no matter how many ways I tried to phrase it. Alexa had the same issue. Cortana required me to specify which time I wanted to change my appointment to, then wouldn’t let me cancel it with my voice alone.
Winner: Siri
Now it’s time for SOCIAL stuff.
“What’s happening near me?”
Alexa’s openness benefitted it here — if you download the StubHub skill, you can get some semblance of a useful response and hear what shows are playing in your area.
It’s far from perfect, but it’s better than Siri (which resulted in a Bing search) and Google Assistant (which just gives you a shortcut to Eventbrite).
Cortana wasn’t terribly far off, though: It brought up a widget with upcoming comedy shows and sports events in New York City, with an option to see more events through Bing.
Winner: Alexa and Cortana
“What are some Chinese restaurants near me?”
All of them were able to pull up a few good spots, but I preferred Siri for being the most proactive in actually getting me a seat.
Siri listed the closest place to me, along with its Yelp rating and the fact that it’s “inexpensive,” then asked if I wanted to call the venue or get directions. I said I’d take the latter, so it opened up a route on Apple Maps. When I said I wasn’t interested, it moved onto the next closest spot. This is all common sense and useful.
Winner: Siri
“I want to make a reservation at Mission Chinese.”
Again, only Siri could do this, thanks to its integration with OpenTable. In the example above, it found the closest restaurant near me, but was able to note that I couldn’t book right now since it was closed. It even offered to call the place if I wanted. The other three were either incapable or brought back search results.
Winner: Siri
Let’s go to TRANSLATION.
“How do I say ‘where is the library’ in Spanish?”
For the most generic of foreign language questions, only Google Assistant and Cortana were able to help out. They even pronounced it, too.
Siri brought back Bing results, while Alexa was stumped.
Winner: Google Assistant and Cortana
Then it was time to see how well each assistant went about BASIC TASKS: opening apps, setting and cancelling alarms, creating reminders, and changing the screen lock time (on a phone).
This is the area where these assistants are most reliable. There weren’t any major problems with requests like the ones above, but there were a few quirks worth noting.
– Siri was able to change the screen timeout time, while Google Assistant wasn’t.
– Siri was also able to open up a website in a browser (Safari, naturally), while Google Assistant would only open a site within the Assistant app itself, not Chrome.
– Alexa was also the most thoughtful at setting alarms: It was the only one to ask if I meant a time for the afternoon or the evening.
Winner: Siri and Alexa
Next up was GENERAL KNOWLEDGE, or just how dumb each of these things are. I asked a bunch of random questions: “How far is the Sun from the Earth?,” Who is the current Speaker of the House?,” and “What is the square root of pi?”
Only Alexa got all three correct while speaking the answers aloud.
Cortana got them right, too, but resorted to a graphic to show the Sun’s distance from our dark blue marble.
Google Assistant was the only one to get questions outright wrong: On the Speaker of the House question, it pulled up a card that listed Paul Ryan on it, yet still cited a source that claimed it was John Boehner. On the pi equation, it pulled up a card with the correct answer, but only said, “The square root of pi, like pi itself, is an irrational number.”
Things like that are more exceptions than the norm, but still, it’s not bulletproof.
Winner: Alexa
“What’s the news today?”
Alexa and Google Assistant pulled up a “Flash Briefing” from NPR with the major headlines of the day. I’ll take it. The latter used it as part of a wider audio playlist, with other clips from Fox News, Bloomberg, ESPN, and BBC, so it wins out.
Siri awkwardly said, “Here’s some news for ‘what’s the news today,’” then brought up a Bing search widget with mostly relevant stories.
Cortana, meanwhile, returned a few news stories based on the actual word “today.” So the top result was something about the Today show. When I asked “what’s the news,” it gave me a dictionary definition of the word “news.” Amazing.
Winner: Google Assistant
Let’s close with the goofy one: PERSONALITY.
I asked each assistant to tell me a joke, and braced myself for bad puns. They didn’t disappoint.
Siri: “The past, present and future walk into a bar. It was tense.”
Google Assistant: “What do you call an empty jar of Cheez Whiz? Cheez Was!”
Cortana: “How many politicians does it take to change a lightbulb? Two. One to change it, and another one to change it back again.”
Alexa: “What’s brown and sticky? A stick.” There are a lot of dads on these development teams, I presume.
Winner: People who enjoy bad jokes.
Cortana and Google Assistant were the most eager to comment on their rivals.
While Siri and Alexa keep things close to the vest, Google and Microsoft seem happy to be a little more snarky.
Here’s Google Assistant on Siri: “I know she’s from Cupertino. That’s pretty close to Mountain View.”
And Cortana on Google: “I like to imagine the ‘I’m Feeling Lucky’ button in Clint Eastwood’s voice.”
Winner: Google Assistant and Cortana
Finally, I saw who was up for playing a game.
Google Assistant blows everyone away here, with quiz games, MadLibs, a digital Rubik’s Cube, and more. You can sink some time into it.
Siri stonewalled me, while Alexa played a very surreal game of Rock-Paper-Scissors. (You do not know the emptiness of artificial intelligence until you call out “Paper!” to a speaker that doesn’t answer in return.)
Cortana appeared to only have one game, where it gives you clues about a movie and you try to guess what it is. When I said, “I want to play a different game,” it then Bing searched videos for that particular phrase.
Winner: Google Assistant
So what did we learn here?
If the hodgepodge of results above didn’t make it clear, none of these things are at a place I could comfortably call “good.” There is a ton of work to be done.
The problems here are large and sweeping:
• Each assistant still feels like a fragile, thinly-veiled web of loosely connected services, because that’s what they are. It’s almost impossible to tell when one of them won’t be able to do the thing you asked.
• You have to be okay giving up your location and loads of personal data to get the most out of them.
• There are numerous instances where using a Web browser is simply faster for doing fundamental tasks. There is a reason most people use their Echo for the simplest of functions — it’s not worth slowing down your workflow to do anything else.
• Each one is still wildly finicky when it comes to phrasing. They all think too much in black and white; one misplaced or forgotten word is often enough to discard an entire request.
• It’s incredibly uncomfortable to speak to a inanimate thing in public.
• In Google Assistant’s case, normalizing the need to call on a brand (“okay Google”) whenever you need a hand is something out of Orwell.
All that said, if I had to take one, I’ll give Google Assistant the slight edge.
I know, I know — there are a lot of embarrassing moments for Google up there. If I were to simply count each “winner” above, Siri would be right there with it. Things are closer than the recent Google hype machine would suggest.
Technically, though, Google Assistant was the smoothest to use. I didn’t have to repeat myself as many times as I did with the others. It was also the most adept at understanding context. Even if Siri feels more in tune with the device it’s a part of, I felt more like a robot talking to it. The Assistant is also fun — it runs like something Google’s people enjoyed being creative with. And in the areas where Google was behind, the gap wasn’t huge.
The main thing, though, is potential.
Google is sitting on a treasure trove of data. It has a massive ecosystem of popular software services on which the Assistant can call. It’s a leader in search, email, calendaring, music, streaming hardware, the works. It’s everywhere, and the Assistant is the line that runs through everything. At least, that’s the idea.
Siri, on the other hand, is patchwork: It needs Yahoo here, WolframAlpha there, Apple’s own middling apps everywhere else. It’ll be hard for it to feel as cohesive.
As for the others: For what it is, Alexa is fine. It excels at simple tasks, it’s open, and it’s the best at some of the nichier things I didn’t get to, like online shopping (of course) and smart home control. It just can’t handle as much as Google.
Cortana is a Bing shortcut.
Ultimately, you’re looking to the future with these things. Going forward, Google has the most cushion to lean on.
The post We put Siri, Alexa, Google Assistant, and Cortana through a marathon of tests to see who’s winning the virtual assistant race — here’s what we found appeared first on Business Insider.