Deep Learning for NLP - Part 6

This course is a part of "Deep Learning for NLP" Series. In this course, I will talk about various popular Transformer models beyond the ones I have already covered in the previous sessions in this series. Such Transformer models including encoder as well as decoder based models and differ in terms of various aspects like form of input, pretraining objectives, pretraining data, architecture variations, etc.

These Transformer models have been all proposed after 2019 and some of them are also from early 2025. Thus, as of Aug 2025, these models are very recent and state of the art across multiple NLP tasks. The course consists of three main sections as follows.

In the first section, I will talk about a few Transformer encoder and decoder models which extend the original Transformer framework. Specifically I will cover SpanBERT, Electra, DeBERTa and DialoGPT. SpanBERT, Electra and DeBERTa are Transformer encoders while DialoGPT is a Transformer decoder model.

For each model, we will also talk about their architecture or pretraining differs from standard Transformer. We will also talk important results on various NLP tasks. In the second section, I will talk about multi-modal Transformer models.