WangchanLion and WangchanX MRC Eval
Wannaphong Phatthiyaphaibun Surapon Nonesung Patomporn Payoungkhamdee Peerat Limkonchotiwat Can Udomcharoenchaikit Ekapol Chuangsuwanich Sarana Nutanong
This technical report describes the development of WangchanLion, an
instruction fine-tuned model focusing on Machine Reading Comprehension (MRC) in
the Thai language. Our model is based on SEA-LION and a collection of
instruction following datasets. To promote open research and reproducibility,
we publically release all training data, code, and the final model weights
under the Apache-2 license. To assess the contextual understanding capability,
we conducted extensive experimental studies using two Thai MRC datasets, XQuAD
and Iapp_wiki_qa_squad. Experimental results demonstrate the model's ability to
comprehend the context and produce an answer faithful to the reference one in
0-shot and 1-shot settings. In addition, our evaluation goes beyond the
traditional MRC. We propose a new evaluation scheme assessing the answer's
correctness, helpfulness, conciseness, and contextuality. Evaluation results
provide insight into how we can improve our model in the future. Our code is
public at https://github.com/vistec-AI/WangchanLion.