Predicting the Quality of Machine Translation

Lucia Specia

The assessment of Machine Translation (MT) quality has been primarily done using metrics that compare the MT output to one or more human translations. These metrics are useful for MT system development and comparison but cannot be applied in practice, when MT systems are being used in production (e.g. online systems such as Google Translate), since human translations will not be available. This lecture will introduce an alternative way of assessing MT quality which has two main advantages: (i) it can be used for any unseen translations and (ii) it allows for flexible modeling of specific dimensions of quality. Such an approach, so called Quality Estimation, is based on machine learning models built and used to predict translation quality. The lecture will cover prediction at different levels (word, sentence, etc.) and different prediction approaches (including neural models). It will also highlight some of the challenges faced when predicting the quality of neural MT systems.

Made on