Adam D'Angelo, CEO of the question and answer website, Quora, talks with EconTalk host Russ Roberts about the history, evolution, and challenges of Quora. Along the way they discuss the aggregation of knowledge and the power of experiments for improving the day-to-day performance of the site.
Play
Time: 1:06:14
How do I listen to a podcast?
Download
Size:30.4 MB
Right-click or Option-click, and select "Save Link/Target As MP3.
Readings and Links related to this podcast episode
Related Readings
HIDE READINGS
This week's guest:
Adam D'Angelo on Twitter.
This week's focus:
Quora: The best answer to any question.
Additional ideas and people mentioned in this podcast episode:
Pedro Domingos on Machine Learning and the Master Algorithm. EconTalk. May 2016.
A few more readings and background resources:
Monopoly, by George Stigler. Concise Encyclopedia of Economics.
A few more EconTalk podcast episodes:Ryan Holiday on Ego is the Enemy. EconTalk. July 2016.
Abby Smith Rumsey on Remembering, Forgetting, and When We Are No More. EconTalk. June 2016.
Hanson on Signaling. EconTalk. May 2008.
Marc Andreessen on Venture Capital and the Digital Future. EconTalk. May 2014.
Highlights
Time
Podcast Episode Highlights
HIDE HIGHLIGHTS
0:33
Intro. [Recording date: July 19, 2016.] Russ: I'm recording this episode at Quora's headquarters in California, high above Castro St., but not that high; so we may hear the occasional honk or truck sound in the background. Adam welcome to EconTalk. Guest: Thanks for having me. Russ: So, what is Quora, for our listeners who haven't been there? Guest: So, Quora is a knowledge-sharing platform. And we basically want to connect people who have knowledge with other people who need it. And the product takes the format of questions and answers. So, anyone can come and ask a question, and then we try to show those questions to people who are going to be especially qualified to answer them; and then those people can write answers. And over time we try to build up this big data base of high-quality answers to questions that can be useful to everyone. Russ: How did you get here? How did your career path end up in this? I think it's about 6 years old now. Guest: Yes. Let's see. So, I started out--I studied computer science in college, but I was also interested in social science and economics. And after college I went to Facebook-- Russ: That's a website, I think, that also--yeah, go ahead. Guest: I've heard of them. Yup. So, I think Facebook is a very interesting application of both computer science, but also social science and some of the theory about signaling and how users behave in these large-scale social products. Russ: Because there's a weird online community. We talked beforehand about signaling; and I don't usually think of social media as being a signaling phenomenon. For me, it's mainly a place where I can get information, or try to spread information about, say, EconTalk. But I probably did some signaling with realizing it. Why did you mention that? Guest: I think a lot of what motivates people to participate in these social networks, especially things like Twitter, Facebook, Instagram, Snapchat is a desire to signal things about themselves--information that's going to be useful to other people. So, it's one of the first chances that we've had to really apply signaling theory to product design. And so I studied--one summer in college I studied, there was a group at MIT (Massachusetts Institute of Technology) called the Sociable Media Group, doing some research on that, that really opened my eyes to doing some of that. So, Facebook was a great chance to apply it. And then after leaving Facebook I looked around, sort of decided what I wanted to do; and I'd always been interested in knowledge sharing and questions and answers and this idea that there has been--there is a huge amount of knowledge that is out in people's heads, that's not on the Internet; it's not very easy to get access to. There's a huge amount of knowledge that I think is locked in research papers where you need to know a certain amount of very domain-specific language to even navigate, to figure out what you want to get access to. And so I thought that there was just an opportunity to really build a platform that would get all this knowledge out to the world and make something that could last for a really long time. Russ: So, some people would say that's what the Internet is. It's just a knowledge-sharing thing. So, right now, if I wanted to solve a problem I had or figure out something I didn't know the answer to, I would just enter it into Google; and there are other sites--Yahoo Answers--that lets people answer stuff. How does Quora differ from those, and what's the importance of that difference? Guest: So, compared to the Internet in general--one of the problems on the Internet--so, if you want to get knowledge that's already on the Internet, there's a lot of good ways to do that. Google does a very good job of indexing all that information and making it available to everyone. The challenge we're interested in is: How do you get more information onto the Internet in the first place? And so there have been some other attempts to do this, other question and answer products-- Russ: Jeeves. AskJeeves. Guest: Yup.Russ: Is that still around? Guest: It's still around. It's actually more of a search engine than a question and answer platform. So, there's been all these other attempts to make this work. But it turns out to be really challenging to run one of these systems at scale and not have the quality just degrade to this kind of, you know, mess where there's nothing really good going on and there's no experts participating. Russ: Right: I might get a hundred answers to my question and I'm going to have to read all of them to figure out anything. Guest: Yep. Russ: Which would be a disaster. So, how do you try to avoid that? Guest: So, we're able to have things like--we've had Hillary Clinton answering some questions; we've had Obama answering questions. We were able to set up an environment where experts want to participate. And there's a lot of different pieces that go into the puzzle. One of the first things that we did--one of the ways that we differentiated early on was that we required everyone to use their real names. And real names are really important because they mean--a real name means that an expert or a someone with sort of real-world-- Russ: Some reputation-- Guest: real-world reputation-- Russ: Even if they don't always tell the truth. So, I'm not sure politicians are your best example of experts. But go ahead. Guest: Yep. So, it lets someone with a real-world reputation bring that reputation into the platform and start out with this credibility. And so, these other platforms at the time that we started didn't really use real names. And so that just put them at a disadvantage. And it's very discouraging to an expert to have to start from zero in an online system. So, that was one piece of the puzzle. But since then, one of the most important things for us has been personalization. So, we've invested very heavily in getting the right answers to the right people, and getting the questions to the people who are going to be able to write really good answers. And that's something that we've gotten better and better at over time and we're still continuing to invest and get better at it. So, ideally you have this environment where people have seen the content--the questions and the answers that are the right questions and answers for them--and that's usually a totally different experience than what other users are having.
7:18
Russ: So, sometimes you go to the Internet--you might to Quora with a particular question because you really want an answer: you've got a pressing question or something you're just really curious about. But sometimes you just go because, 'I'm bored. I'm looking for some interesting content; this might be fun.' Do you have any perception of the mix of users in those two camps? Guest: It's mixed. I'd say maybe half and half. There are a lot of people who have important questions every day that are trying to get an answer to that. And then there's a lot of people who just--they're bored, maybe they are really are digesting [?]; they get provoked by a particular question that they are curious about and then they'll go and read that. And I almost think of the reading behavior as your mind is rewarding you for getting some information that might be helpful to you later. So you get this sort of short-term payoff that it feels good to learn new things and to express your curiosity. Russ: Speaking of signaling: so, before this interview I thought maybe I should--not 'maybe I should'--I went on the site, right, which I've been on before; but I thought to refresh my memory, make sure--I'll explore some things; maybe I'll get some ideas for questions. And then I thought--gee, should I ask a question? And then I realized: 'Do I really want people to know I'm asking that question?' Right? Because that's one thing--one of the questions I ended up asking--I didn't ask this question; I searched on it--was: What should I ask Adam D'Angelo? Which I thought was really unclever--not a bad idea--but obviously the personalization, the non-anonymity changes people's willingness to participate in the site in certain ways, right? Guest: Yeah. So, we actually have a feature where you can ask an anonymous question, if that's important. But the vast majority of users don't ask questions. It turns out that almost all questions that people have are questions that someone else has had in the past. Russ: Nothing new under the sun. Guest: Yeah. And that's actually one of our strategies, that we don't want to have 20 versions of each question. We want to have one version; and that allows us to concentrate all the answers' energy into that one place. And then that helps to generate the best possible answers. Because people know that there's this one place where they are going to answer the question that's going to be kind of definitive for a long time. Russ: So, that's a big challenge. For example, I really like photography, and I'll be on a photography website like DPReview (Digital Photography Review), which is my favorite. And in the forum someone will ask a question that's been asked 273 times. But this [?] person who has got in, just joined the forum, was lazy, didn't search, or they asked the question a slightly different way and they didn't find it. And everyone jumps on them--and sometimes they are kind and they'll put a link that's been asked 40 times: 'Here's one of the better ones.' But obviously one of the challenges is: how do you figure out what question is like another question. So, how do you solve that? Give me, if you can, if you're comfortable, how many questions get asked a day, roughly? I mean, what kind of volume are we talking about here? Guest: So, we don't share numbers on the volume of questions. Russ: But the answer is: A lot. Guest: Yes. It's a lot. Russ: Even though most people don't ask, doesn't matter; I'm sure there's a lot who ask a lot of questions. Guest: Yeah. It's a lot, and it's growing very quickly, at the rate that the rest of the product is growing. So, if you want a sense of scale, we have about 100 million monthly unique visitors--that was the last number we announced. Russ: That's a big number. Guest: Yeah. It's a lot of people. But most of them are looking at questions that someone else asked. Back to your question about how do you [?]-- Russ: How do you merge, how do you decide to merge a question? Guest: So, there's a few things. One is we have a machine learning algorithm that can look at two questions and determine whether, make a guess at whether those are likely the same question or not. And so, if you are going to ask a question that we think is a duplicate then we'll go and show you that question and say, 'Hey, you probably wanted to look at this question.' Some questions get through that filter, though; and so we have another set of systems that try to merge questions later on. There's an offline machine learning process that takes a little bit longer. Then, there's--users of the product can actually go in and merge two questions. We have a process for how those get reviewed and how we make sure that we control the quality on that. Russ: Cool. Guest: So it's not perfect, but it ends up being a much better experience than, say, on the photography feed forum where there were 20 versions of the same thing.
11:59
Russ: So, if I could go back and visit the Adam D'Angelo of 2010--you had some kind of vision. How has it turned out? Something like you imagined? Nothing like it? Guest: Pretty much-- Russ: Or just bigger? Guest: Yeah. I mean, it's definitely bigger than I expected. I think everyone in our industry has been surprised over the last few years at just how big all these markets have gotten. One--you know, you have all these people coming onto the Internet for the first time, all these people have phones so they are using the product more. But, there's also another thing I think is a huge factor here is that personalization is a really powerful technology as far as its effect on markets. So, in the offline world, let's say for magazines, everyone reading a magazine has to see the same magazine and the same order of the articles. Russ: A little bit--they might put a different cover on the West Coast versus the East Coast. Guest: Yeah-- Russ: That was a big breakthrough. Guest: Right. There might be 5 variants or something like that. There might be 10, 20. But still, they are the same stories in the magazine; and you have this limited amount of physical space to hold them in. But with the Internet, you can have, sort of an infinite amount of content; and you can have every user getting a totally different experience. And so that's meant that markets that used to be fragmented, like, say, the magazine industry, now in the Internet you can just have a very small number of products that are able to reach a much bigger audience. Because they can show the content to the right people. So I think, you can see this with Google. There's just one search engine: everyone uses the one search engine. There's NetFlix, NetFlix instead of all these different TV channels and different shows on different channels and different cable providers: You just basically have NetFlix. And I think personalization is this--it basically has this effect on markets where it makes the market a lot bigger, because a single company can address this very diverse, needs [?] Russ: It's one, but it's providing a million channels instead of--issues of monopoly where they may not have the incentive to customize it. Because that's not really the case. Because they are constantly trying to customize it to maximize their reach and then their revenue, right? Guest: Right. And though they may not have head-to-head competition every day every way you might have had in some previous industries. But instead they have this very strong incentive to just get people to use, get the existing users to use the product more. Which, you know, effectively Netflix is in competition with Facebook even though you think about them as totally different markets. People have this discretionary time that they are going to spend online; and the better Netflix gets, the more time is going to Netflix; and the better Facebook gets, the more time is going to Facebook. Russ: And the same is true for Quora, obviously. Hulu is a competitor even though it is not a question-and-answer site, because it's in the same category of interesting things I can do on my computer or phone. Guest: Yep. Russ: Absolutely. When I have some spare time.
15:12
Russ: You don't appear to have any ads on the site. And yet, according to my research outside of Quora you are worth something over probably a billion--your valuation of the company is over a billion dollars. Do you have plans for monetization that you can talk about? Guest: So, we actually do have ads. We are running ads on a small percentage of our pages. It's just a small test right now. So far the results have been very promising. And we're looking at--we're going to scale that up over the years ahead. Our valuation--you know, I'm not going to take a particular stance on what our valuation is. The valuation that investors give us is based on projections of the future and based on the amount of usage that a product has. You can assume that we are going to monetize in a similar way to other similar products per unit of usage. Russ: But do people worry that there's a big difference between an ad-free site and a site that's full of ads, and how that might affect the culture and the ease and the pleasure that people get from visiting the site? Is there a tradeoff there? Guest: Yeah. I mean, we've been very up front with our users that we will run ads in the future. We don't intend to have a product that is full of ads. We think we can make a lot of money, enough to sustain the business, without having to make the user experience a lot worse. I think a good example of this is Google. You do a search; most of the time you don't see any ads. If you happen to search for something that's commercial, then you'll see more ads. Russ: All the time. And I want to say, 'Who do I call to tell them I already bought that? They can stop.' Guest: Yeah. Russ: It's interesting. They'll get better at that, I assume. Guest: Yeah.
17:04
Russ: So, the company is 6 years old. You've had a lot of growth, obviously, because you didn't have any users when you started and now you've got 100 million. But, as an outsider there's a temptation to assume that you're bored. You had this great idea; it came to fruition: Big enough, we're done, the site's up and running, it's growing steadily. Is the thrill gone, or is it still fun to come to work every morning? Guest: Yeah. Well, see, it's still really fun, and I think one of the things that really exciting to me is just the scale that we're getting to. We're starting to reach like a big percentage of the United States and the rest of the world every month. That's inspiring sort of on its own, but it also creates a lot of challenges internally. So, the growth basically follows this exponential growth pattern, so that there's a lot more people using the product every week than the week before. And that creates a lot of challenges internally. So we have to make sure that our infrastructure and our technical systems are able to be ready for the load. We have to make sure that our personalization technology can do better and better as we get bigger and bigger. The bigger we get, we also get more data. And so that lets us do a better job in machine learning, in personalizing. It lets us run more experiments. That's something else that I think is a pretty interesting part of what we do. Russ: And we'll talk about that--but I have to warn you now, because you can have this big spike when this EconTalk episode is released. So you're going to want to prepare for that, probably. But, buy an extra server. Guest: Yeah. Well, we're on Amazon Web Services, so luckily we can just turn on some more servers and just run them by the hour. Russ: Oh, whew. Guest: So I get a warning when it is about to be close. Russ: You have to borrow some money. Um, so, in terms of the product itself, you know, as a user, I see questions; I see the answers. Do you do other things besides questions and answers? Is there product variation or innovation beside just the Q&A part of the site? Guest: We've experimented with some formats outside of questions and answers, but over and over we've just seen that the question-and-answer product is just doing really well and getting bigger and bigger. And it takes--it's almost taken our full energy as a company to just keep up with the question-and-answer product and make sure that the quality stays really high. And just bring it--we've brought it to mobile phones; as new technology has come out we have to keep pace with that. And we have to keep our costs under control. So there's a lot to do just for questions and answers. So that's kept us focused. But I imagine in the long term we will branch out into other formats of knowledge besides questions and answers. Russ: So, besides asking questions or reading questions other people have asked and answered, you can sign up for Topics on the site and see Q&A related to a particular topic. Do you, Adam D'Angelo, do you have topics that you subscribe to yourself, personally? Guest: Yeah. Yeah. I follow a lot of topics. Some of the ones I'm particularly in are related to machine learning. So that's just a personal interest of mine. But also it's super-applicable to what we are doing every day. Russ: It reminds me--I've told this story before, but one of my favorite management stories, Sam Walton supposedly used to fly on a private plane--he was the founder of Wal-Mart. And he'd see a WalMart truck on the road, land in a cornfield--I don't know if it was literally a corn field, but land somewhere ahead of the truck--and then hitchhike--stick his thumb out, get a ride with the trucker into town and chat with the trucker and then go into the actual store. You know, it's called manage by walking around, among other things. But I always like this image of this trucker driving along, trying to stay awake, and seeing the Founder and CEO (Chief Executive Officer) of his company there on the side of the road with his thumb out. And trying not to have a heart attack. But that's the standard way, in brick-and-mortar businesses, are renowned for getting information. They get their hands dirty; they get in the trenches. The kind of [?] of the CEO is up in the executive suite, never really tries the product and just relies on, say, the marketing team, etc. So, you--you've got a ton of information about how the site's doing. But you also have the one data point of you as user. Does that affect you at all? Does that ever lead into the decisions you make? Do you ever, like, slam your fist down on the desk saying, 'Why am I getting these bad answers?' or 'bad questions?' or whatever? Or is it all just driven by the data? Guest: I'd say most of it goes on the data. But I definitely use my own experience. And one thing I try to do that you might find interesting: I actually try to behave more like what I think a normal user behaves-- Russ: Sure-- Guest: so that my own experience is not too distorted from that. So you could say that, as the CEO maybe I would just artificially kind of like force myself to write tons of answers and follow lots of topics and give the system lots of information about me so that it could do a really good job. But then I would have this experience that was nothing like a normal user. Russ: Correct. Guest: So I actually go out of my way with using Quora--and other Internet products--I just try to like act like a normal user so that the information I get isn't so biased. But yeah, you know, in terms of it affecting my--it's good for generating theories and ideas. Usually I'd like to test those theories and ideas on some, ideally on a controlled experiment. But also we look at surveys of our users; we look at things users are reporting. We look at--we have all kinds of sources of information. So I try to make sure that the ideas I generate from my own usage are consistent with the data. Because if you don't do that, you can really just go off in a direction that's crazy. [More to come, 23:11]