bilingual evaluation evaluating quality text machine translated natural language quality considered correspondence machine output human closer machine translation professional human translation better central idea achieve high human quality remains popular automated inexpensive scores calculated individual translated segments generally sentences comparing set good quality reference translations scores averaged corpus reach estimate translation overall quality taken account designed approximate human judgement corpus level performs badly evaluate quality individual sentences output number value indicates similar candidate reference texts values closer representing similar texts human translations attain score candidate texts identical reference translation reason necessary attain score opportunities match adding additional reference translations increase score uses modified form precision compare candidate translation multiple reference translations metric simple precision machine translation systems known generate words reference text illustrated following example seven words candidate translation appear reference translations candidate text given precision formula formula number words candidate found reference formula total number words candidate perfect score despite fact candidate translation retains little content references modification makes fairly straightforward word candidate translation takes maximum total count formula reference translations example word appears twice reference reference formula candidate translation count formula word maximum formula word case formula formula formula formula summed words candidate sum divided total number words candidate translation example modified precision score formula practice individual words unit comparison optimal instead modified precision metric grams length highest human found scores found account translation information retained longer formula scores account translation extent reads like good english problem scores tend favor short translations produce high precision scores modified precision example candidate translation references cat example modified precision formula word word cat appear candidate total number words modified precision formula cat appears candidate pointed precision usually recall overcome problem recall example formula formula problem multiple reference translations bad translation easily inflated recall translation consisted words references produce score corpus modified precision scores segments combined geometric mean multiplied penalty prevent short candidates receiving high score let formula total length reference corpus formula total length translation corpus formula penalty applies defined formula case multiple reference sentences formula taken sum lengths sentences lengths closest lengths candidate sentences version metric prior shortest reference sentence instead interactive version allows user visually examine scores obtained candidate translations allows comparing different systems visual interactive manner useful system development performance frequently reported human judgement remains assessment new evaluation metric number criticisms voiced noted principle capable evaluating translations language present form deal languages lacking word boundaries argued significant advantages guarantee increase score indicator improved translation quality inherent problem metric based comparing reference translations real life sentences translated different ways overlap approach comparing given translation result computer differs human translations flawed automated metric compares translations reference grammar defined human human effort involved correctly defining ways render meaning translation practice means
BLEU