Blinded quality raters preferred human translation over MT-plus-postediting for all 20 Chinese public-health documents

Description #

In a blinded head-to-head comparison, two public-health professional quality raters selected the professional human translation (HT) over the machine-translation-plus-postediting (MT+PE) version as the preferred document for all 20 document sets rated. Reasons cited for preferring the HT included better word order, a more professional reading level, smoother flow, more accurate word choice, preserved meaning, and cultural appropriateness. This unanimous preference contrasts with the authors' prior English-to-Spanish work, where HT and MT+PE were rated as roughly equivalent.

"Unlike our previous experience with English to Spanish translations, in a blind comparison of HT and MT+PE, the quality raters selected the HT document as the preferred version for all 20 documents." (Turner, 2015, p. 6)

"Reasons given for the preference were better word order, a more professional reading level, smoother flow, more accurate translated word use, preserved meaning, and cultural appropriateness of the original English document." (Turner, 2015, p. 6)

Methods Context #

What? #ⓘ

The observable: which translated version — the professional human translation or the MT+PE version — a rater preferred for each document set.

"Using a questionnaire, we asked the quality raters to read each set carefully, indicate which of the translated versions they preferred, and describe why they chose that version, based on five dimensions: grammar, adequacy, word choice, cultural appropriateness, and reading level." (Turner, 2015, p. 4)

How? #ⓘ

Two raters, blinded to translation method, each received 20 sets (of the 24 available) containing the original English, an HT version, and an MT+PE version; documents were unlabeled and set order was randomized.

"Two public health professionals, blinded to the method of translation, compared the quality of the postedited documents to the quality of the HT documents from the health department websites." (Turner, 2015, p. 4)

"The documents were not labeled as human- or machine-translated, and the order in which they were presented in each set was randomized." (Turner, 2015, p. 4)

Who? #ⓘ

Two public-health professional raters: one a professional public-health translator and DSHS-certified medical interpreter, the other a health researcher (Table 1).

"One rater was a professional public health translator and a Department of Social and Health Services Certified Medical Interpreter at a local clinic; the other was a health researcher (Table 1)." (Turner, 2015, p. 4)

Other Notes #

This is the study's headline quality finding and is wired as supporting the claim that human translation is preferred over MT+PE, and as contradicting the (prior Spanish-pair) expectation of HT/MT+PE equivalence. It is qualified by caveats that MT+PE received only one editing round (versus multiple, opaque rounds for the HT) and that a single MT engine was used.

Caveats #

• MT-plus-postediting received only one round of postediting whereas the human translations underwent multiple editing cycles The blinded quality comparison pitted MT+PE documents that had undergone only a single round of postediting against professional human translations that were collected already-finished from health-department websites and had likely gone through several rounds of editing and quality control. Because the two arms did not receive equal editing effort — and the human translations' process was opaque — the raters' unanimous preference for the human translation may reflect the amount of editing applied rather than an intrinsic ceiling on MT+PE quality. The authors themselves note that additional postediting rounds would likely have further improved the MT+PE product.