Transformer-based language models
Abstract
Transformer-based models have recently gained much attention, especially with the release of ChatGPT. Since 2017, deep learning models based on the Transformer architecture have become an important research tool. Their development and application in various fields, including the social sciences, continue to expand. In this talk, we will examine the components that make up these language models and explore how to train state-of-the-art models with HuggingFace for your research. We will also discuss these models’ limitations and open challenges, including open-source availability, the growing need for resources, responsibility, and more.