Recent advances in Natural Language Processing and Deep Learning

Thomas Delteil

2019-09-12 17:30 #machinelearning

In recent years, Machine Learning (ML) approaches to Natural Language Processing (NLP) drastically changed. From a feature-based modelling, where NLP tasks are tackled by using a set of manually engineered features, we moved to the so-called representation learning: text is naturally represented as a sequence of symbols (e.g., words, characters, etc.) and the role of learning how these symbols should be represented is entirely left to the ML method. Deep Learning dominates such paradigmatic shift.nIn this presentation we will review the Deep Learning approach to NLP and discuss different alternatives for representing text, including word embeddings[1] and character embeddings[2].nFinally, we will focus on the latest advances, including: ELMo[3], a model for generating deep contextualized word embeddings, and BERT[4], a very recent Neural Network Model that allows to achieve state-of-the-art results on several NLP tasks with very little task-specific fine-tuning.nThis presentation will mix scientific theory and actual code examples from Apache MXNet (incubating) demonstrating how to leverage these advanced techniques to solve NLP tasks. [1] Word2Vec - Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013n[2] CNN character embedding layer - Character-Aware Neural Language Models, Kim et al. 2015n[3] ELMo - Deep contextualized word representations, Peters et al. 2018n[4] BERT - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al. 2018