An expert public-health translator did not close the MT-plus-postediting quality gap with human translation

Description #

In a follow-up test of whether posteditor expertise explained the quality gap, a highly experienced public-health translator (P6) postedited four documents under the original ("edit only as needed") instructions. When five native-Chinese raters blindly compared these against the human translations, three of the five still preferred the human translation over the MT+PE for all four documents, and the other two rated only one document pair as equivalent. Greater posteditor expertise thus did not close the quality gap with human translation.

"Three of the 5 raters selected the human translation over the MT+PE for all four documents; 2 raters rated one of the HT and MT+PE documents as equivalent." (Turner, 2015, p. 7)

Methods Context #

What? #ⓘ

The observable: raters' blinded preference (human translation vs MT+PE) for documents postedited by an expert translator, collected by repeating the quality-rating procedure.

"We then repeated the quality rating procedure with those documents, asking five native Chinese speakers to review them." (Turner, 2015, p. 5)

How? #ⓘ

An expert posteditor corrected four documents under the original minimal-edit instructions; the resulting MT+PE documents were then blindly rated against their original HTs by five native-Chinese raters.

"She was given the original set of instructions to correct only as much as needed and to not rewrite the text extensively." (Turner, 2015, p. 7)

Who? #ⓘ

One highly trained/experienced public-health translator (P6), who performed translation for a large metropolitan health department, served as the posteditor.

"we engaged the services of a public health professional who performed translation for a large metropolitan health department in Washington State (P6)." (Turner, 2015, p. 7)

Other Notes #

This follow-up isolates posteditor expertise as a candidate explanation for the headline preference finding, and finds expertise alone was insufficient (under minimal-edit instructions). It supports the claim that HT is preferred over MT+PE. Contrast with the companion EVD showing that changing the instructions (edit as needed) did move MT+PE toward equivalence.

Caveats #

• MT-plus-postediting received only one round of postediting whereas the human translations underwent multiple editing cycles The blinded quality comparison pitted MT+PE documents that had undergone only a single round of postediting against professional human translations that were collected already-finished from health-department websites and had likely gone through several rounds of editing and quality control. Because the two arms did not receive equal editing effort — and the human translations' process was opaque — the raters' unanimous preference for the human translation may reflect the amount of editing applied rather than an intrinsic ceiling on MT+PE quality. The authors themselves note that additional postediting rounds would likely have further improved the MT+PE product.