A Bit of a Problem: Measurement Disparities in Dataset Sizes Across
  Languages
Catherine Arnett Tyler A. Chang Benjamin K. Bergen
How should text dataset sizes be compared across languages? Even for
content-matched (parallel) corpora, UTF-8 encoded text can require a
dramatically different number of bytes for different languages. In our work, we
define the byte premium between two languages as the ratio of bytes used to
encode content-matched text in those languages. We compute byte premiums for
1155 languages, and we use linear regressions to estimate byte premiums for
other languages. We release a tool to obtain byte premiums for any two
languages, enabling comparisons of dataset sizes across languages for more
equitable multilingual model development and data practices.