Talk: Lessons learned from training multilingual language models for Scandinavian languages

Title

Lessons learned from training multilingual language models for Scandinavian languages

Abstract

Training high-quality language models for languages other than English can be challenging both because of the lack of resources, but also because of the often unclear transfer effects between languages. In this presentation, I am going to give an overview of the GPT-SW3 model series, which were the first Generative Language Models covering the Scandinavian Languages. In addition, I am going to discuss our recent paper on studying the cross-lingual forward and backward effects on the continual pre-training setup.