With the release of Windows 10, Microsoft has brought its digital assistant Cortana, previously only available to those few who had taken the plunge with a Windows phone, to millions of people’s laptops and desktops. After spending a few days playing around with Cortana’s features, and as a long-time user of Google Now, I thought it would be fun to put both, along with Apple’s Siri, through the paces and see which voice-activated digital assistant comes out on top. To do this, I came up with a series of questions and asked each one. I then judged their responses based on the accuracy and relevancy of the response, whether or not the answer was given audibly or just visually, and the speed with which the answer was delivered.
Names:
I didn’t use this is a factor is critiquing the three systems, but I thought I’d add a quick note on their names for those who might be curious.
Cortana from the Halo video game franchise.
Cortana is named for the artificial intelligence system in Microsoft’s incredibly successful Halo series of games on the Xbox. I’ve been a fan of Halo since the first game came out in 2001, and absolutely love the fact that Microsoft named their assistant after the character.
According to Adam Cheyer, co-founder of Siri, the startup that Apple would eventually buy, there’s some debate about where their system’s name actually came from, but all of the stories are nice. CEO Dag Kittlaus knew that Siri was Norse for “beautiful woman who leads you to victory” and had considered using the name for his child. Cheyer had discovered that the word was Swahili for “secret” and was the reverse spelling of the name of an earlier project that Siri had developed out of. The word also means “beauty” in Sinhalese.
Google Now’s name is clearly the result of a committee meeting.
Set up:
I tested Google Now via my Samsung Galaxy S4, running Android 5.0. For Siri, I used my son’s iPad Mini running iOS 8. Cortana I got through my laptop, running Windows 10. While the disparity of the devices potentially impacted the results when it came to responsiveness, I accounted for that in my analysis.
All three systems have a setting available to allow them to respond anytime to a voice command: “OK, Google,” “Hey, Cortana,” or “Hey, Siri.” In all three cases, this poses a potential battery drain, but Google and Microsoft at least leave it up to the user to decide. Apple doesn’t give you that choice: you can only set Siri to respond to the voice command when the device is plugged in. That’s unfortunate, as it dramatically limits the overall availability of Siri.
In my testing, Google Now never failed to respond. Once I plugged the iPad in, Siri activated correctly every time to the voice command. Cortana correctly responded more often than not, but failed quite a bit as well. However, I suspect that could have had as much to do with my laptop’s microphone as the system, so I wouldn’t be ready to hold that against Cortana without further testing on other devices.
The Questions
As I mentioned above, the core of the test was asking each system a set of questions to seeing how the responded.
Did the Giants win last night?
I’m a San Francisco Giants fan, and, while I watch as many games as I can, I still miss a lot. So I began by simply asking the question, “Did the Giants win last night?” Two things about this question: I was intentionally being vague in asking about the “Giants” and not the “San Francisco Giants” because it’s reasonable to think that that is how a normal human would ask the question, but I wanted to know if any of the systems would be confused and give me information about the New York (football) Giants. Also, the Giants didn’t actually play last night at all–their last game was Sunday night.
The Giants score via Google Now.
Google Now answered with “No, the Giants lost to the Pirates five to two. They are playing the Cubs today at 7:15 pm.” It also displayed the box score on the screen. One might argue that that answer is technically wrong, as they didn’t lose last night at all, but you could argue that instead it gave me the answer I was actually looking for, which is their most recent score. Added points for stating the score out loud along with displaying it, and for letting me know when the next game is.
Cortana’s answer to the Giants question.
Cortana delivered a slightly more correct answer: “On Sunday, the Giants lost to the Pirates two to five.” It also displayed the full box score, which was nice (or would have been nice if the game had gone better). It didn’t provide any information on the next game.
Siri’s response to the Giants question
The combination of the lack of a game for San Francisco and the vagueness of using “the Giants” stumped Siri. It’s response: “the Giants didn’t play yesterday. They play the Jets in a preseason game on Saturday at 4pm.”
Even when I was more specific and asked Siri specifically about the San Francisco Giants, it gave the more technically correct answer that they didn’t play, and let me know that their next game was tonight against the Cubs. What it didn’t do was give me the score to the last game.
To be fair, both Google Now and Cortana allow you to go into settings and specify favorite teams, virtually eliminating the possibility of confusion over which Giants team (unless, I guess, you set both as favorites). I’m still willing to call this against Siri, though, since it doesn’t provide any such setting.
So this one I have to call as a tie for Google Now and Cortana, and a solid loss for Siri.
Who is the president?
Next, I thought I’d go into some general information stuff. I started with what I thought was an easy one: “Who is the president?” And right away, I hit an AI limitation.
Siri’s answer
Google brought up the Wikipedia entry for “President of the United States” and read aloud the first paragraph. It did, however, display a picture of Obama, but only because that picture happens to be on the Wikipedia entry. Siri, after pausing with a “let me check that” message, brought up a list of search results. Cortana did as well, but it did it by opening a new tab in my browser with the Bing search results. So I did get the answer I wanted–in all three cases, a quick glance at the screen revealed the answer–but none gave me the answer I wanted out loud.
Google Now’s somewhat oddly different answer to “who’s the president?”
But then I accidentally stumbled on an oddity. In the first test, I clearly enunciated “who is the president.” But then I did a second test, asking “who’s the president.” And, surprisingly, I got a different answer as Google stated out loud, “President Barack Obama. According to the White House, Barack Obama is the 44th President of the United States.” Both Cortana and Siri gave the same answer as before.
So again, Google and Siri tie on these questions, while Cortana tied with the other two on “who is” but lost giving the better answer to “who’s.”
What time is it in Tokyo?
Tokyo time, Cortana-style
Next, I asked all three “what time is it in Tokyo?” Google answered the fastest. Cortana was a bit slower but still got it right. Both read the answer and displayed it on the screen.
Siri really struggled with this question. I’ve discovered in experimenting with it that it is the most likely of the three to be somewhat slow to actually start listening, so the first time it picked up “I’m is it in Tokyo”, which, of course, it didn’t understand. On the next try, it only heard “is it in Tokyo,” to which it brought up a list of search results about the city. Deciding that I was asking too quickly, I said “Hey, Siri”, then paused… too long, as it responded with “Hello, Rob.” Grrr. Finally, my fourth or fifth attempt I managed to time things exactly right and get the answer.
So all three eventually gave me what I wanted, but Siri definitely had issues.
How long will it take me to get to work?
I decided to try them out as actual digital assistants rather than information systems, so my next question was “how long will it take me to get to work?” I should mention that all three give the ability to set home and work locations, and I had gone in and preset both.
Here again Google was fastest, with an almost-immediate answer of “your work is 30 minutes from your location by car in light traffic.” It also offered directions. It’s worth noting that my screen showed the actual travel time as 29 minutes, so the AI was doing what we humans do and rounding.
Cortana’s answer was, again, slightly slower, but still very helpful: “It’ll take about 30 minutes to drive to work. It’s about 26 miles away.” Again, on the screen I could see that Cortana thought the actual travel time was 34 minutes.
Siri was the clear loser here. It merely responded with “getting directions to work,” and then popped me over to the map app to show the route. The travel time was displayed on that screen, but the answer clearly wasn’t as helpful as the other two.
What is the square root of 1764?
Next: math. I asked all three for the square root of 1764. Easy, right? Well, Google and Siri thought so, and both stated the answer right away. (42, of course.) Cortana was noticeably slower, and once again resorted to opening a browser window and showing me Bing’s calculator. Another tie for Google and Siri, and a loss for Cortana.
How many days until Christmas?
My next question was one that would be important to my kids, and, interestingly, was one of the questions with the greatest variance: “how many days until Christmas?” Google Now brought up an answer from website emailsanta.com, reading “There are 122 more sleeps to the big day.” Siri went vague: “It’s three months until then.” It’s OK, I guess, but not the answer to the question I actually asked.
Siri’s answer to the Christmas question.
Cortana’s audible answer wasn’t close to what I asked: “Christmas is on Friday, December 25, 2015.” It’s display showed the right answer, 122.
So on this question, Google was the only one that answered the question I asked out loud. Cortana gave me the answer, but I had to look at the screen. Siri gave an OK answer, but it wasn’t the question I asked (days aren’t months, after all), and I think pretty much every parent knows that “three months from now” isn’t close to an acceptable answer to this question if you’re under 10.
How long until Star Wars: Episode VII opens?
One last serious, and seriously important question: “how long until Star Wars Episode VII opens?” Google Now and Cortana simply brought up search results, but Siri got it–“the answer is 3 months, 29 days.”
But are they funny?
When Siri was first released, it was famous for its humorous answers. Some of them have changed, so, for example, it no longer tells you where to bury a body, replying instead with “I used to know the answer to this.” Cortana and Google Now both brought up a web search on burying bodies. (The NSA is going to love looking at my search history this month.) Siri still plays around with the answer to “when is the world going to end,” with a set of funny answers. Google Now brought up a page from IFL Science debunking some Facebook meme about the imminent end of the world, while Cortana added yet another tab to my browser with a web search. Siri will read a haiku if asked, while the others won’t.
“Hey, Cortana. Are you her?”
But Siri doesn’t quite have a monopoly on humor. If asked, “are you her?”–a reference to the Spike Jonze movie Her, in which a man falls in love with his phone’s operating system, Siri responds with “No. She is a fictional construct, whereas I am a virtual entity. But we can still be friends.” Cortana, I think, had the better answer: “Dude! I’m Cortana.” Google Now had no clue what I was asking.
Cortana telling a joke.
Cortana also tells jokes much better than the other two. Siri will tell jokes if asked, but the engineers at Microsoft managed to get Cortana to deliver punchlines with some awesome inflection. In the joke shown above, Cortana reads the punchline as “Pick a cahd. Any cahhhd.” Google Now again falls flat, merely bringing up a search screen.
Recently, there’s been some press about Siri’s answer to “what is zero divided by zero.” It’s cute, with a whole story about cookies and whatnot. Cortana is also good here, responding that it’s not possible, and something that nature abhors, like vacuuming. Google Now, though, has the most unintentionally hilarious result: it once again falls back on a search, but thanks to the aforementioned press, the results are all about its competitor Siri.
Pretty much the only silly question I could think to ask that Google Now did better with was “what is the answer to the ultimate question of life, the universe, and everything?” Google correctly answered with “42.” Cortana brought up a search result page that had the right answer as the first hit. And Siri flubbed this one: “Life: the condition that distinguishes animals and plants” blah blah blah.
So if you want pure silliness, Siri and Cortana are clearly better, while Google Now apparently has no sense of humor at all.
End Results
In most cases, I found that Google Now was quicker and more likely to give me the answer to the question I asked, and much more likely to read it out loud. Siri comes in second: it usually gives the correct answer, and is very likely to both read the answer and display it. However, I found it generally slower and I very often had to repeat my question and over-enunciate to get Siri to listen to the whole thing. It seems to still really struggle with listening.
Siri also has another incredibly frustrating feature: it’s the only one of the three that doesn’t stick around. When my phone goes to sleep and I wake it back up, my previous Google Now screen is still there, with my previous question. Cortana remains open and on top until you close it. But if your iPad or iPhone goes to sleep from Siri and you wake it up, you’ll find yourself back on the home screen with no choice but to ask Siri the question again to get back to the answer.
I’d probably have to rate Cortana last in my tests, but it was close. It resorted to simply opening a browser search far more often than the other two, and, while it didn’t have troubles understanding as often as Siri, I still had to repeat my questions more often than I’d like. The one area where Siri and Cortana are clear winners is in giving silly answers, but, while that’s fun for a short time, it’s not really what most people probably want from these systems.
To be fair, Cortana is the newest of all three systems, and the Windows 10 version at least is officially still in beta. So it’s possible that the results will improve over time, and in fact it might be interesting to revisit this test in a year or so and see where things stand. But to be equally fair, Siri is the oldest of the systems, and it has some big issues.
In the end, if you want a voice-activated digital assistant, your best bet these days is Google Now. It may not be able to tell you a joke, but it’ll get you where you need to go and reliably answer your day-to-day questions.
If you value content from GeekDad, please support us via Patreon.