Interview: Inside Google Brain Founder Andrew Ng's Plans To Transform Baidu

This article is more than 2 years old.





Q: How did you get interested in artificial intelligence?

A: I just thought making machines intelligent was the coolest thing you could do. I had a summer internship in AI in high school, writing neural networks at National University of Singapore--early versions of deep learning algorithms. I thought it was amazing you could write software that would learn by itself and make predictions.

If we can make computers more intelligent--and I want to be careful of AI hype--and understand the world and the environment better, it can make life so much better for many of us. Just as the Industrial Revolution freed up a lot of humanity from physical drudgery, I think AI has the potential to free up humanity from a lot of the mental drudgery.

Q: For a period, you were turned off by AI given the lack of progress, right?

A: The big AI dreams of making machines that could someday evolve to do intelligent things like humans could, I was turned off by that. I didn’t really think that was feasible, when I first joined Stanford. It was seeing the evidence that a lot of human intelligence might be due to one learning algorithm that I thought maybe we could mimic the human brain and build intelligence that’s a bit more like the human brain and make rapid progress. That particular set of ideas has been around for a long time, but [AI expert and Numenta cofounder] Jeff Hawkins helped popularize it.

Q: How close are you to that?

A: Oh, very far away. There are a lot of problems. One is just scale. We’re nowhere near processing enough images today. Two is I’m very sure we don’t have the right algorithm yet.

Q: So why has AI seen a resurgence in interest and results in recent years?

A: For a long time, AI was off doing spam filters, autofocus functions for cameras, whatever. That was very valuable. But it was only about seven years ago, seeing the latest evidence from neuroscience, that many of us felt more hope for the bigger AI visions to build machines that don’t just do a few tasks really well but that capture intelligence more broadly.

About four years ago, late 2010, we had figured out a number of these algorithms and realized that the biggest bottleneck to taking the technology to the next level was scale. It turns out we could take software that we had written in the 1980s, maybe in my high school internship hypothetically, and just run it on a modern-day computer, it would do much better.

So for myself, in late 2010, I wound up looking around Silicon Valley to figure out what would be a good way to scale up these algorithms. Google, no surprise, had a lot of computers, so I wanted to start a project at Google to take these algorithms and build a larger version than anyone else had built. In hindsight, what made this project successful was we gave the team a very simple instruction: Build the biggest neural network possible.

Q: This was the Google Brain project in 2012 that recognized images of cats.

A: The remarkable thing was that it had discovered the concept of a cat itself. No one had ever told it what a cat is. That was a milestone in machine learning. That was the milestone that helped inspire many other companies, including Facebook and Baidu and a few others.

Q: What was so striking about that experiment to these companies’ businesses?

A: Most of the economically important applications so far have been just learning from tagged or labeled data. Take speech recognition. The way that Baidu and the Google project have improved speech recognition performance is that deep learning algorithms can just soak up massive amounts of transcripted speech data. Today we have tens of thousands of hours of audio together with the transcript of what was actually said, so that’s labeled data, so we can train these networks to make predictions.

In the longer term, there’s a different type of deep learning that I’m excited about. The second set of ideas is called unsupervised learning, or learning from unlabeled data. The Google cat thing was actually an example of learning from unlabeled data. We had the neural network watch YouTube for a week, we didn’t tell it anything, we just had it watch videos for a week, and then at the end of the week we probed it to see what it had learned, and it learned to detect human faces, cat faces, a few other things. It turns out that that was kind of a milestone in AI. That was unsupervised learning, not what was driving most of the economic value of deep learning today.

Q: Why are you excited about unsupervised learning?

A: One is that it’s closest to how animals and infants learn. If you think about learning to recognize cars, the way we would do it today is to find 50,000 pictures of cars and label them as cars and feed them as labeled data to a supervised learning algorithm. But if you think about how children learn to recognize cars, no parent, no matter how patient or loving, will point out 50,000 cars. Most neuroscientists believe that most animal and children learning is from just going out into the world and experiencing it for ourselves. If we make progress in this, I think we can make systems that are much better able to understand images.

Two is that for some applications, we’re starting to run out of data. So let’s say you want to recognize cars, it turns out today we can get 50,000 pictures of cars, maybe we could get 500,000 if we do a lot of work, but at some point, there just aren’t that many cars in this country, and no matter how much you drive around the country, it’s very hard to get 5 million pictures of [labeled] cars. Or with medical imaging, not that many X-ray scans are done in the entire country, so there’s just a cap on the amount of data you could possibly get.

Q: Is unsupervised learning the main thing you’ll be focusing on here at the lab?

A: It will be one of the major things--the initial focus. Unsupervised is harder, less proven, we don’t know what the right algorithms are. We have some ideas, but it will be much harder.

Q: Is the trend toward computing and communications on mobile devices and the explosion of sensory data driving this interest?

A: It’s more than that. Big Data comes from two things. It comes from the increased digitization of society, creating digital data that computers can examine. Second, storage and computation costs continue to decrease, so that gives us finally the ability to store and process all this data. So if you believe that digitization of society will continue and storage and computing costs will continue to decrease, I think that predicts the extent to which Big Data will continue as a trend.

Q: I gather you’re thinking as far out as 10 years at this lab, but what specifically do you plan to do near-term and longer-term?

A: Baidu Research has three labs--two in Beijing that are already largely built up, and the Silicon Valley one is being built from scratch. We’re hiring pretty rapidly, about one person a week, but we are about a month in, so honestly we haven’t done that much work yet.

Q: Do you have a model for the lab, such as Bell Labs, PARC, Google, Microsoft, or whatever?

A: One of the things I was doing prior to starting with Baidu over the last two months was going to speak to directors of the great labs. I met with Bill Mark (VP of information and computing sciences) at SRI this morning and spoke with a bunch of people, like the former director of Xerox PARC. I was at Google X. A long time ago, I was a summer intern at AT&T Bell Labs, throughout my undergrad and early part of Ph.D., every single summer I was there. So I’m trying to be a bit humble, learning from others. Some models of research work better than others.

Q: What do you consider the most important factor for a successful lab?

A: I think a lot about team culture. When I was younger and trying to build organizations and people would try to talk to me about culture, I used to dismiss that as that fluffy stuff I didn’t what to make of. Now that I’m a bit older, I really appreciate culture and the importance of being thoughtful about it.

Q: It matters right from the start, doesn’t it? If you don’t get it right from the start, it’s a problem because it’s very hard to change.

A: Even changing a little bit is very hard. There’s an opportunity to create a culture that’s great for research and great for changing the world.

Ultimately, it’s all about the mission. All of the organizations I’ve been at tend to be mission-driven. The mission of Coursera was giving everyone a great education. We decided the best way to do that was to start a company. For the same reason, the mission here is changing the world through AI technology. I felt strongly that the most efficient way to do that would be to join Baidu.

Q: Why Baidu?

A: It turns out Baidu has had very advanced deep learning technology. One of my good friends, Kai Yu [director of Baidu’s deep-learning lab in Beijing], is a leader in deep learning. Deep learning has been very useful for Web search, ads, speech recognition, OCR, a whole bunch of Baidu products including ones at the core of the company. Robin Li, the CEO, has a deep personal passion for AI. One thing I really appreciated is that if you read Baidu’s official annual report, in the letter to shareholders, Robin talked about deep learning. So this is leadership that gets it.

There are three reasons I joined. One is that AI today is a capital-intensive. You need the data and the computational resources in order to make progress. It turns out the data is harder to get than the computational resources, but you need both.

Two is nimbleness. One thing I’ve learned about Baidu is the incredible degree of nimbleness the organization has. To give you an example, Kai Yu, when he decided to build a graphics processor cluster, he just made a decision and then it happened.

Third is just the people. The engineers at Baidu work incredibly hard. Engineers in China work much harder than the average Silicon Valley engineer.

Q: Really? Engineers at startups here seem to work pretty hard.


Q: Is it a little strange being at a company that is often called China’s Google after having worked at Google itself?

A: I really don’t mind it. I used to work at the USA’s Baidu. Baidu is the tech powerhouse of China. That’s not really in question. There are other organizations that do product very well--the user interface, the shiny apps and so on. That’s less Baidu’s strength, but Baidu’s really good at building the really hard technologies.

Q: Is that by virtue of doing search, which is inherently a technically hard problem, or because of the choice of the leadership?

A: Company leadership. Search is a hard technology. There’s a willingness to build hard technologies. It creates a more solid base to build on. I feel like the shiny apps, who knows who’s going to be the next--boy, I’m even losing track, Snapchat? I think what Snapchat did was not easy. But maybe there’ll be a different Snapchat++ next year. When you build some of the hard technologies that companies like Baidu try to, it gives you a more lasting base to build on top of.

Q: How do you intend to apply the work to be done in this lab? Could it go beyond Internet applications to robots, autonomous cars, and the like?

A: Initially we just plan to focus on the technology. Usually I tend to start with product and then worry about the technology. Looking around Silicon Valley, most of our failures are not because we fail to solve a hard technical problem. Most of our failures, including my own failures, are spending two years on solving a hard technical problem only to realize that nobody cares. The piece you solved wasn’t critical.

With deep learning, because it’s already used in a lot of Baidu products and because AI is just useful to so many things, I’m actually less worried about whether they’ll be useful to current or future products.

Q: Will you leverage Baidu’s technical infrastructure or do you need to build some of that new?

A: What Adam and I want to do is build tooling--the infrastructure and developer tools--to enable the deep learning team working with us to be able to very efficiently generate and test ideas and to learn from them. To take a illustration: A lot of the speech recognition experiments take about a week to run. When it takes you a week to get your results back, it’s very hard to learn that quickly. So if you can halve that time, it makes the team twice as efficient. We want to enable the future heroes of deep learning.

Q: What kind of new technologies might result?

A: I hope that this will allow us to make incremental improvements to some of the current deep learning applications within Baidu, to support those teams--search, advertising, language translations, optical character recognition, speech recognition.

Improvements in technology enable two types of progress. One is incremental, which is not to be disparaging. If we can improve the performance of Web search by 5%, that will benefit a lot of users.

Q: And the other type?

A: Improvements in technology sometimes create breakthrough moments which enable whole new classes of applications that you just wouldn’t have used before. To take an example, if we can make progress in speech recognition, to recognize speech at human levels, imagine if when you’re driving, you could send text messages to your friends by just talking to your cell phone even in a noisy car. That’s a whole new mode of interaction with cell phones that just isn’t possible today.

If we can actually solve speech, I would love to redesign the cell phone around the speech interface. Maybe your email app on your mobile device should have only two buttons: reply and delete. These are only ideas. They may be bad ideas. But improvements in core technologies such as speech recognition could enable much larger changes like these.

Looking even further out, improvements like these could build us personal assistants that all of us will actually use. Can we build software that could answer some fraction of our emails for us? That would make life better.

Q: What other ideas are you looking at?

A: Maybe some robotics things, we’re brainstorming a bunch of all that too. Or retail stores: Big Data is really influencing online retailing because websites like Amazon, Netflix, get a ton of data about what decisions. Physical, big-box retailers, most of them don’t really know what’s going on in the stores. Is it possible to get better data so that they know, did the customer pick up this good [product], did they make a face before putting it down and walking away? Is it possible to get retailers better data about what’s happening in their stores so they can run A/B tests the way that online retailers do every day and help them?

One question I often ask people in terms of deciding which project to focus on is: If what you’re working on succeeds beyond your wildest dreams, would anyone care? The answer had better be a very strong yes, because otherwise I think you should find something else to work on. I love to focus on projects that if they succeed will allow us to develop a service or product that will significantly influence or impact at least 100 million people.

Q: How are you attracting talent given competition in deep learning from Google, Facebook and others?

A: We’re able to attract very, very strong talent so far. We’ve been hiring about one person a week. Six as of today. Every offer we have extended has been accepted so far. One is pending. We’ve been keeping the bar high. Hires so far have been all from Silicon Valley. We’re talking to a small number of people from farther away. Those will take longer.

We want to be part of the ecosystem here, to be part of the conversation with researchers and developers. We’re committed to building an organization that’s a little bit porous.

Q: What do you mean?

A: As we develop ideas, we’ll try to share them with the developer community. We hope to be more porous than many. There are some research organizations whose main aspiration is to write papers. That’s not ours. The mission is to develop hard AI technology that will benefit hundreds of millions of people. There is an inclination of business-to-consumer impact rather than business-to-business impact.

Q: The lab is called the Silicon Valley AI lab, so are you looking beyond deep learning in particular?

A: Deep learning is a very promising approach. There could be other pieces as well. Like decision-making. Deep learning algorithms are very good at perceiving the world and understanding the world. In Adam’s and my former life, we worked on decision-making controls, technology called reinforcement learning, where you make decisions on how do you steer this helicopter or how do you steer this car, how do you control this robot? That could be another piece of the puzzle. But that’s not what we’re focusing on right now, because the perception, deep-learning piece seems like the harder one.

Q: It seems that one of your jobs is to reduce the seemingly unlimited number of things the lab could work on.

A: One sign of creative engineers and researchers is that you always have more ideas than you can possibly work on.

We’re going to do some cool things here.

Follow me on Twitter or LinkedIn. Check out my website.
