Image
The University of Tartu is the leading university in the Baltics in the field of information technology and one of the 250 best universities in the world according to the ranking of Times Higher Education Universities in the field of computer science. The University's Chair of Natural Language Processing focuses on machine learning and data mining on text data and machine translation from one natural language into another.
Technical/scientific Challenge:
Neural networks have caused rapid growth in output quality for many natural language processing tasks, including neural machine translation (NMT). However, the output quality crucially depends on the availability of large amounts of parallel and monolingual data for the covered languages. Due to lack of training material, several lower-resource Finno-Ugric languages are not included in the existing massively multilingual models. In terms of the number of speakers, they range from 20 near-native speakers of Livonian to several hundred thousand speakers of Mordvinic languages.
Solution:
Large-scale LLM training requires the coordinated use of multiple accelerator nodes, high-bandwidth and low-latency interconnects, distributed data pipelines, resource scheduling, parallel training strategies, checkpointing, and efficient utilisation of heterogeneous CPU/GPU resources. These are fundamental HPC system and performance-engineering challenges. The scalability and efficiency of the training process depend not only on the model architecture and training algorithm, but also on the underlying HPC system, including accelerator performance, interconnect bandwidth and latency, storage throughput, and job scheduling.
An NMT system was developed between 20 low-resource Finno-Ugric languages and 7 high-resource languages. Monolingual corpora was collected mainly by crawling texts off the web and combining with pre-existing corpora. Three main categories of texts can be distinguished: news, Wikipedia, and biblical. All the NMT systems were trained on the LUMI supercomputer. All models were fine-tuned with the Fairseq framework implementation of M2M-100 for 350k updates with a batch size of 3840 tokens (the number was chosen to match earlier versions of models trained with the Huggingface implementation of M2M-100). The models were fine-tuned on 4 AMD Mi250X GPU-s.
Scientific impact:
23 Finnish-Ugric languages can now be translated using the machine translation engine of the University of Tartu. Most of these languages were added to a public translation engine for the first time. The translation engine allows researchers to translate materials that would otherwise be incomprehensible to them. It provides an opportunity to better study the history of languages and regions without knowing the respective language. Moreover, since most of Finno-Ugric languages are not widely spoken today, a translation engine is necessary to preserve these languages.
Benefits:
Collection of parallel and monolingual corpora that can be used for training NMT systems for 20 low-resource Finno-Ugric languages.
Expansion of the 200-language translation benchmark FLORES-200 with manual translations into nine new languages (Komi, Udmurt, Hill and Meadow Mari, Erzya, Livonian, Mansi, Moksha and Livvi Karelian).
The collected data can be used to create NMT systems for the included languages and investigate the impact of back-translation data on the NMT performance for low-resource languages.
This success story was produced within the framework of the EuroCC 2 project.