Loading...
DataDecon: Data Cleansing Tools for Large Language Model with Efficient Decontamination Techniques
Author(s)
Yuenyong, Sumeth
Buppodom, Norapat
Sangkaew, Koravich
Boonmeeprakob, Konthee
Boonkwan, Prachya
Jaroenkantasima, Jillaphat
Khlaisamniang, Pitikorn
Lertpiya, Anuruth
Piyatumrong, Apivadee
Rojratchadakorn, Peerawat
Rugsujarit, Thaweewat
Saengsukhiran, Teerapol
Saetan, Kriangkrai
Sukprapa, Isada
Thavornmongkol, Thanachot
Thongthungwong, Nucharee
Triamamornwooth, Patteera
Utupon, Chanon
Viriyayudhakorn, Kobkrit
Witchutanon, Phoochit
Wongprayon, Sadit
Supnithi, Thepchai
Date Issued
January 1, 2024
Type
Conference Paper
Abstract
Large language models (LLMs) play an important role in modern NLP technology as they are versatile for a wide array of NLP tasks. However, constructing an LLM is challenging due to concealed construction pipelines, the lack of cleansed datasets, and hyperparameter settings, making it almost irreproducible. This paper presents an efficient pipeline for constructing an LLM tailored to a low-to-medium-sourced language with a high level of data contamination and tools to cleanse the dataset. Following our pipeline, we constructed OpenThaiGPT, an LLM for Thai, with only open-sourced datasets such as CC100, OSCAR, and mC4, and achieved the state-of-the-art accuracies on our downstream tasks. Here, we disclosed the data statistics and all hyperparameter settings for reproducibility.
Citation
19th International Joint Symposium on Artificial Intelligence and Natural Language Processing Isai Nlp 2024, 2024