Juru: Legal Brazilian Large Language Model from Reputable Sources
Roseval Malaquias Junior Ramon Pires Roseli Romero Rodrigo Nogueira
The high computational cost associated with pretraining large language models
limits their research. Two strategies have emerged to address this issue:
domain specialization and pretraining with high-quality data. To explore these
strategies, we specialized the Sabi\'a-2 Small model with 1.9 billion unique
tokens from reputable Brazilian legal sources and conducted few-shot
evaluations on legal and general knowledge exams. Our model, Juru, demonstrates
the benefits of domain specialization with a reduced amount of pretraining
data. However, this specialization comes at the expense of degrading
performance in other knowledge areas within the same language. This study
contributes to the growing body of scientific evidence showing that pretraining
data selection may enhance the performance of large language models, enabling
the exploration of these models at a lower cost.