This discussion topic – about the structural design decisions we build into our neural architectures, and how those correspond to certain assumptions and inductive biases – is an important one in AI right now.
On one side, Manning is a prominent advocate for incorporating more linguistic structures into deep learning systems. On the other, LeCun is a leading proponent for the ability of simple but powerful neural architectures to perform sophisticated tasks without extensive task-specific feature engineering. For this reason, anticipation for disagreement between the two was high, with one Twitter commentator describing the event as “the AI equivalent of Batman vs Superman”.
However, LeCun and Manning agreed on more than you may expect. LeCun’s most famous contribution (the Convolutional Neural Network) is all about an innate prior – the assumption that an image processing system should be translationally invariant – which is enforced through an architectural design choice (weight sharing). For his part, Manning has spoken publicly to say that the Deep Learning renaissance is A Good Thing for NLP.
While the two professors agreed on many other things during the discussion, certain key differences emerged – you can watch the full video above. The rest of this post is a summary of the main themes that emerged throughout the discussion, plus some links to relevant further materials.
ABOUT THE EVENT:
Deep Learning has achieved huge success largely on the basis of end-to-end learning: the idea that a single neural network with many layers can learn to perform a complex task, from the raw input to the final output, with minimal human guidance on the intermediate stages. This approach has many advantages: it reduces the need for human-designed features and allows us to design systems that are less pipelined and less task-specific. However, it may sometimes be advantageous to include certain innate priors – such as linguistic information for text-based tasks, for example – in our deep learning systems.
At this event, Professors LeCun and Manning discuss what role these innate priors should play in modern Deep Learning – an active debate in the AI community.
ABOUT THE SPEAKERS:
Yann LeCun is the Chief AI Scientist at Facebook AI Research, a Silver Professor at New York University, and one of the leading voices in AI. He pioneered the early use of convolutional neural networks, which have been central to the modern success of Deep Learning. LeCun has been a leading proponent for the ability of simple but powerful neural architectures to perform sophisticated tasks without extensive task-specific feature engineering.
Christopher Manning is the Thomas M. Siebel Professor in Machine Learning at Stanford University, and the world’s leading NLP researcher. His work on applying deep learning to NLP includes influential research on Tree Recursive Neural Networks, GLoVe word vectors, and neural dependency parsing. Manning is a proponent for the importance of incorporating linguistic structure into deep learning systems.
