Machine translation of a SACT booklet introduced 11 critical errors vs 1 and failed CIoL assessment (51 vs 73 of 100)

Description #

On blinded independent assessment against the Chartered Institute of Linguists (CIoL) Diploma-in-Translation criteria, the unsupervised machine (Google Translate) Bengali translation of the SACT booklet scored 51/100 (a CIoL "fail") versus 73/100 (a "pass") for the professional translation. The machine translation introduced 11 critical (meaning-changing) errors and 19 accuracy issues, compared with 1 critical error and 4 accuracy issues in the professional translation. One machine mistranslation rendered patient-directed instruction as "Kill as many myeloma cells as possible."

"Independent translation assessment scored the machine-translated booklet 51/100 ("fail" by CIoL criteria) and the professional translation 73/100 ("pass" by CIoL criteria)." (Hibbs, 2026, p. 4)

"The professional translation introduced 1 critical error and 4 accuracy issues, compared to 11 critical errors and 19 accuracy issues introduced by the machine translation." (Hibbs, 2026, p. 4)

"Mistranslations in the machine translation included: "The medicine will help you to: Kill as many myeloma cells as possible", as instructing the patient to kill as many myeloma cells as possible." (Hibbs, 2026, p. 4)

Methods Context #

What? #ⓘ

The observable: translation quality, scored as an overall CIoL mark out of 100 and as counts of critical (meaning-changing) errors and accuracy issues.

"an additional secondary outcome was blinded independent assessment of translation quality using the Chartered Institute of Linguists (CIoL) criteria for the Diploma in Translation." (Hibbs, 2026, p. 3)

""Very serious errors" are defined by CIoL as errors that completely change meaning or make translated texts unusable unless retranslated or heavily revised (also defined as "critical errors")" (Hibbs, 2026, p. 3)

How? #ⓘ

Two independent assessors, blinded to translation type, graded both Bengali translations of the same source booklet against CIoL Comprehension and Language-Quality criteria; full assessments are in Supplementary Results.

"Independent assessors of translated information booklets were blinded to translation type." (Hibbs, 2026, p. 3)

Who? #ⓘ

The object assessed was a single English Easy-Read myeloma booklet ("Easy Read: How is myeloma treated?", Myeloma UK) translated into Bengali two ways — professionally and by unsupervised Google Translate NMT.

"An unsupervised neural machine translation (NMT) was generated using Google Translate" (Hibbs, 2026, p. 3)

Other Notes #

The machine translation was generated with Google Translate specifically because many UK healthcare providers and cancer charities deliver non-English patient information through the Recite Me accessibility tool, which uses Google Translate. This is a translation-quality (accuracy) measurement, not a patient-comprehension outcome; the parallel comprehension EVD found no downstream difference in understanding despite this large quality gap.

Caveats #

• Bengali is low-resource, so machine-translation results may not generalize to higher-resource languages Machine-translation quality is language-specific. Bengali is a low-resource language with a limited linguistic corpus for training machine-translation tools, so it is likely to be translated worse than higher-resource languages (e.g. French, Spanish, German) for which such tools have far more training data. The poor machine-translation performance and high critical-error count observed here may therefore not generalize to other language groups, and conclusions about machine translation should not be extrapolated across languages.