Instructions to make all necessary corrections moved MT-plus-postediting toward quality equivalence with human translation

Description #

When posteditor instructions were changed from "edit only where necessary" to "use as much time and effort as necessary to ensure a high-quality translation," three returning posteditors reworked 10 documents. Postediting slowed sharply (e.g., P2 fell from 35.38 to 23.43 CPM, P4 from 54.33 to 17.46 CPM, P5 from 53.96 to 19.69 CPM), and on blinded rating the quality rater chose the human translation for only 6 of 10 documents while rating four as equivalent — a notable improvement over the original instructions, under which raters had preferred the HT for every document.

"The rater chose the manual human translations for 6/10 documents, while rating four as equivalent—a notable improvement over the original instructions." (Turner, 2015, p. 7)

"As anticipated, posteditors took longer to produce the MT+PE translations with the updated instructions: P2's average speed dropped from 35.38 CPM to 23.43 CPM, P4's fell from 54.33 to 17.46 CPM, and P5's decreased from 53.96 to 19.69 CPM." (Turner, 2015, p. 7)

Methods Context #

What? #ⓘ

The observable: raters' blinded preference (human translation vs MT+PE) and postediting speed (CPM) under revised, quality-emphasizing instructions.

"we modified the instructions to emphasize quality and recruited 3 posteditors to come back for another postediting session with the new instructions." (Turner, 2015, p. 7)

How? #ⓘ

The original minimal-edit instructions were replaced with instructions to spend whatever time and effort quality required; three prior posteditors reworked 10 documents, which were then blindly rated by one quality rater with language and public-health expertise.

"The updated instructions directed posteditors to use as much time and effort as necessary to ensure a high-quality translation. The 3 returning posteditors corrected a total of 10 documents, which were then blindly rated by a quality rater with language and public health expertise." (Turner, 2015, p. 7)

Who? #ⓘ

Three returning native-Chinese posteditors (P2, P4, P5) and one quality rater with language and public-health expertise.

"To test posteditor engagement and whether the instructions to edit only as necessary were problematic, we asked 3 posteditors (P2, P4, and P5) to return to edit a total of 10 more documents, this time with instructions to make as many corrections as needed to ensure the quality of the translation." (Turner, 2015, p. 5)

Other Notes #

This finding qualifies the headline result: the HT-over-MT+PE preference was not fixed — instructing posteditors to edit thoroughly (at a large time cost) moved MT+PE toward equivalence. It is wired as contradicting the strong form of "HT is preferred over MT+PE" and as supporting the best-practice claim that thorough-edit instructions improve MT+PE quality. Sample is small (10 documents, one rater).