Deep Learning is a new area of Machine Learning research that aims to move Machine Learning closer to one of its original goals: Artificial Intelligence (AI). It allows computational models that are made up of numerous processing layers to learn representations of data with numerous levels of abstraction. The field has helped radically improve developments in visual object recognition, speech recognition, object detection and various other areas, including genomics and drug discovery.
Some branches of Deep Learning draw inspiration from how the human brain works. Instead of using a digital model when all computations manipulate zeros and ones, a neural network functions instead by creating connections between processing elements, sometimes compared to the electrical relationships found between neurons in the brain.
Deep Learning is particularly effective when it can draw on large amounts of data. The term Deep Learning came into being because its systems typically have five to twenty hidden layers compared to one or two for shallow learning systems. Deep Learning has recently found itself at the vanguard of the tech industry and specialists in the field are highly sought after. However, it was not always this way.
“The AI Winter”
Up until recent years, the Deep Learning field was in the bitter throes of an “AI Winter” as the potential of its theory outweighed its practical potential. In the late 90s, financial backing for this branch of AI began to shrink and many researchers left the field. The scientists still working within Deep Learning had advanced theories and ideas for how neural networks operated, but without the backing, computing power or vast troves of data, they weren’t able to fully explore them. Deep Learning was seen as something of a renegade subject, unlikely to have much future impact upon daily life.
The Google Brain Breakthrough
In 2011, Stanford University professor Andrew Ng, one of the most prominent researchers who had begun his work during the AI Winter, set up a Deep Learning initiative at Google. It was one of the company’s Google X projects – a semi-secret division dedicated to making major technological advances. Ng and Google Fellow Jeff Dean set out to develop the “Google Brain”, aiming to create the largest artificial neural network then in existence. They did so by connecting 16,000 computer processors, which they then released onto the Internet to learn on its own. The neural network taught itself to recognize cats, human faces and other parts of the human body with an approximately 75% accuracy rate, far higher than the team expected. Its impact was game changing because it was one of the most significant examples yet of “unsupervised learning”.
The breakthrough in Deep Learning was made possible by recent advances in technology, those being primarily: (i) the lower costs of computing, which allowed the researchers to annex so many computers at once, and (ii) the vast amount of data that has become available because of the ever-growing scale of the Internet, which gave the computers access to the vast datasets necessary to practice finding and classifying patterns. Exposure to the world’s largest video library with YouTube’s 10 million digital images allowed the Google Brain to begin to teach itself through repetition. “Deep Learning has been around for decades,” said Andrew Ng, “but the main reason it has been taking off in the last few years is scale.”
The Present Day Boom in Deep Learning
As a result of the AI Winter, there are only a fairly limited number of scientists who are Deep Learning specialists. Consequently, tech companies are snatching up even current students before they have yet completed their studies. Naveen Rao, the CEO of Deep Learning startup Nervana Systems, told Re/Code recently that, “There’s been a huge brain drain from academia”. An engineer who is knowledgeable about Deep Learning can make over $250,000 a year at a company like Facebook or Google. Yoshua Bengio, one of the key researchers in the field who runs a machine-learning lab at the University of Montreal, has seen his student numbers swell from 15 to 60 in recent years because of renewed interest in the field. However, Bengio has concurrently seen the impact of students leaving academia early for tech companies and worries about its long-term significance, saying, “it’s hard to find the more senior experts in Deep Learning”.
“The Canadian Mafia”
A number of computer scientists continued to work within Deep Learning across the AI Winter, providing vital sustenance for the field while others had moved away. Perhaps the best-known group was the then Canadian based trio: Geoffrey Hinton, Yann LeCun and Yoshua Bengio. Yann LeCun met Hinton while studying at Hinton’s seminal neural networks lab at the University of Toronto. LeCun then met Yoshua Bengio at AT&T’s Bell Labs. All three also began to work collaboratively at the Canadian Institute for Advanced Research (CIFAR).
Tongue in cheek, they have referred to themselves as the “Deep Learning conspiracy”. Others have dubbed them the “Canadian Mafia” because they are so close-knit. They still work together, recently co-publishing an article in Nature magazine on Deep Learning.
Geoffrey Hinton
Geoffrey Hinton is a British cognitive psychologist and computer scientist. He has been a professor in the computer science department at the University of Toronto since 1983 (barring a three year break at UCL in London from 1998-2001) where he directs the program on “Neural Computation and Adaptive Perception” (NCAP) for CIFAR. During the 1990s and early 2000s, Hinton’s work on mimicking the human brain as a route to artificial intelligence was seen as a project on the fringes of academia. The AI community instead largely worked to find shortcuts to brain-like behavior. Hinton’s lab, however, continued to focus on building artificial neural nets to gather information and react to it.
In 2004, Hinton decided to found NCAP based at CIFAR, in order to spur innovation in Deep Learning. The group was invite-only and consisted of celebrated computer scientists, neuroscientists, psychologists, biologists, engineers and physicists. Hinton wanted to build a team of world-class thinkers dedicated to generating computer systems that mimic organic intelligence (or what we know of it). The members of NCAP saw their research accelerating as a result of the regular workshops and increased collaboration, and they began to build more powerful Deep Learning algorithms that could operate on much larger datasets.
Many see Hinton, now 67, as an elder statesman within the field. Michael Mozer who studied under Hinton and now works at the University of Colorado described him as having ideas “at a rate that it’s a little hard to fathom”, always related to Hinton’s belief that machine learning mimics human development. Mozer said that a Hinton refrain was: “I know this is crazy, but just suspend your disbelief and suppose this is what’s going on in the brain.” While studying psychology at Cambridge, Hinton says he was struck by how little scientists understood the brain. He was driven to join the search for how interactions among billions of neurons might give rise to intelligence. Deep Learning is now a mainstream pursuit. Hinton phrases it humorously, “we ceased to be the lunatic fringe. We’re now the lunatic core.”
In 2013, Google recruited Hinton into its Deep Learning division, although he continues to act as a part-time professor in Toronto and remains at the helm of the CIFAR group. In addition to running the yearly NCAP conference and talking at worldwide events, Hinton also runs a summer school for students to help foster the next generation of AI researchers.
Hinton at Google
In 2012 around the time Ng started Google Brain, the company made some internal changes that freed up the AI researchers working there, allowing them to attract some of the biggest names in the field. Rather than having AI researchers working on specific product teams, Google unhooked them from that responsibility. Now, they act more as independent researchers and consultants. The AI team develops machine-learning advances, shares them company-wide and consults on specific products when other teams request their expertise. This has allowed them to work quickly and yet consult widely. Up to 100 teams inside Google currently use neural network tools.
Andrew Ng now works for Chinese search giant Baidu with a similar kind of team to the one he launched at Google. Before leaving, however, he recruited Geoffrey Hinton. Jeff Dean, Google’s senior fellow and co-founder of Google Brain, explained that Hinton largely works to encourage the AI experts to explore more speculative ideas. “The main thing he brings is lots of interesting ideas and how to take what we’re doing and look out five years.” Currently, Hinton is using Deep Learning techniques to improve voice recognition, image tagging, and many other online tools.
Yann LeCun
LeCun was born near Paris, France in 1960. After completing his postdoctoral studies in France, LeCun moved to Canada to be a postdoctoral research associate in Hinton’s lab at the University of Toronto. In 1988, he moved to New Jersey to join the Adaptive Systems Research Department at AT&T’s Bell Labs where he met Bengio. While a researcher at BellLabs, LeCun developed the convolutional neural networks technique, including building a biologically inspired model of image recognition focused on recognizing visual patterns in pixels. This led to the widely deployed bank check recognition system, allowing computers to scan and register handwritten bank checks for the first time.
LeCun persevered with neural networks after they had fallen out of favor, becoming a professor at New York University in 2003 where he has helmed many more advances in Deep Learning. In an interview with Spectrum, LeCun credits his perseverance in the field while it had fallen out of favor to having been long enamored “of the idea of being able to train an entire system from end to end. You hit the system with essentially raw input, and because the system has multiple layers, each layer will eventually figure out how to transform the representations produced by the previous layer so that the last layer produces the answer. This idea – that you should integrate learning from end to end so that the machine learns good representations of the data – is what I have been obsessed with for over 30 years.” Lecun is stresses that his models do not try to replicate the brain, and that while he draws inspiration from neuroscience; many components of his work derive from theory, intuition or empirical exploration.
In 2013, LeCun was cherry-picked to become the inaugural director of Facebook ‘s Artificial Intelligence Research Lab centered in New York City.
LeCun at Facebook
When LeCun became the Head of Facebook’s AI division in 2013, he intentionally kept his AI team separate from the product teams, learning from Google’s early mistakes. His unit at Facebook now employs 50 researchers, including fellow NYU professor Chris Bregler. He has recently opened a new office in Paris, and has plans to continue expanding.
In speaking to Re/code journalists Mark Berger and Kurt Wagner recently, LeCun made clear that while his team at Facebook of course develops products for the company, it also has the freedom and large resources to simultaneously pursue long-term projects. “There may or may not be products that come out of this for the next two or three or four or five years”, LeCun explained. “It’s not clear. They may come faster, they may not.”
In terms of immediate application at Facebook, one of the focuses of LeCun’s team is on getting AI to better understand content so that the most relevant content can be selected and shown to each user. Much of LeCun’s work at Facebook centers on creating new theories, methods and systems to get machines to understand images, language, speech and video – and then to reason about them. Additionally, LeCun’s team is working on building a neural net on one side and a separate module on the other that is used as a memory. The aim is to help improve question-answering and language translation through bolstering the machine’s ability to have associative memory, an area that hasn’t been much focused on previously in AI.
Currently, the power of Deep Learning can be seen in apps like Moments, which has facial recognition technology embedded into it. It’s also rumored that Facebook is developing a personal assistant product within Messenger, nicknamed “Moneypenny” that would draw heavily on Deep Learning’s computing powers.
Yoshua Bengio
Like Lecun, Bengio is also French born. He and LeCun became friends whilst working together at AT&T’s Bell Labs where Bengio was a postdoc student; and the pair pushed at the edge of the theories LeCun had studied with Hinton in Toronto. In a recent interview with Re/Code, Bengio recalled, “We innovated in many exciting ways.”
Bengio didn’t stay in the corporate world and has been a Professor in the Department of Computer Science and Operations Research at the University of Montreal since 1994. He has authored two books and over 200 publications in the areas of Deep Learning, neural networks, probabilistic learning algorithms, language modeling and pattern recognition.
In 2013, Bengio started to develop a set of equations for functional, intelligent algorithms aimed at taking advantage of unlabeled data. ‘Labeled data’ is information that’s been categorized by people, which machine learning has typically depended on in the past. However, unlabeled information far outweighs the amount which people have been able to label, meaning that for machine learning to take major steps forwards, it must tackle areas in which labeled data is less common, such as image recognition and language translation.
He says that one of the reasons AI failed in the 70s or 80s is that the programming of the machines was too narrowly focused on building every step through reasoning. “It’s much easier”, he told Wired in 2013, “to train machines to develop intuitions to make the right decisions.” Bengio’s new models build on the neural nets generated at Google for its image search and photo-tagging systems, however, Bengio described his models as more intuitive, thus better at exploring data that is thrown at them.
Bengio and LeCun co-founded the International Conference on Learning Representations in 2013.
Criticism and Rebuttal of the Trio’s Impact on AI
In response to the trio’s recent Nature article, Juergen Schmidhuber, a Swiss AI researcher, posted an online critique stating that LBH (LeCun, Bengio & Hinton) take too much credit themselves while failing to “credit the pioneers of the field, which originated half a century ago”. Schmidhuber, while acknowledging LBH’s work, wants to see better credit assignment within the field of machine learning. He hints that LBH are more popularizers than inventors.
The “Canadian mafia”, however, have overall high standing within the Deep Learning community and are seen as deserving credit for being part of a select number of scientists keeping AI on “life support” across its critical years, meriting the headline success they are now garnering. A former Facebook researcher, Rob Fergus, put it this way: “In the lean times when no one believed in neural nets, these are the people who really kept the torch burning and really inspired a lot of people.” They continue to do so today through their varied contributions to the field, and through a spirit of close collaboration. When LeCun recently spoke about his leadership style of his lab, he discussed the need to let young people exercise their creativity. “The creativity of old people is based on stuff they know”, LeCun said, “whereas the creativity of young people is based on what they don’t know. Which allows for a little wider exploration.”
The post Deep Learning and the “Canadian Mafia” appeared first on Tech News.