Readers raised significantly more comprehension issues about the literal than the functionalist survey translation

Description #

Across the semistructured interviews, participants raised significantly more issues (unclear items, unusual expressions, uncertainty) about the literal translation A (M = 6.375) than about the functionalist translation B (M = 2.125), by a two-sample Welch t-test (t[8] = 2.121, p = .033). The authors read the higher issue count as an indication that translation A was harder to understand. A per-item analysis reinforced this: five questions (Q1, Q2, Q4, Q6, Q8) in survey A were flagged by more than five participants.

"The two-sample Welch t-test confirmed that the participants raised significantly more issues about the survey questions or items in A (M = 6.375) than in B (M = 2.125) (t [8] = 2.121, p = .033), which further confirmed expectations; the higher number of issues raised suggests that translation A was more difficult to understand than translation B." (Colina, 2022, p. 33)

"The per-item analysis of the data shows that five questions (Q1, Q2, Q4, Q6, and Q8) in survey A were mentioned by more than five participants, another indication of the preference for translation B." (Colina, 2022, p. 33)

Methods Context #

What? #ⓘ

The observable: the number of issues (points of unclarity, unusual expressions, or uncertainty) each participant raised per translation during the interview.

"The quantitative analysis focused on identifying the number of issues raised in each survey and per item as well as the participants' translation preferences." (Colina, 2022, p. 33)

How? #ⓘ

During semistructured interviews, participants were directed to identify unclear, unusual, or uncertain items in each translation; issue counts per translation were then compared with a two-sample Welch t-test (used to accommodate unequal variance in the small sample), run in R.

"During these interviews, the participants were instructed to identify questions that were unclear, that contained unusual expressions, or that they were unsure about." (Colina, 2022, p. 33)

"Additionally, a two-sample Welch t-test was conducted to compare the mean of issues raised between both translations. The Welch t-test was used to account for unequal variance due to the small sample." (Colina, 2022, p. 33)

Who? #ⓘ

The 20 English-Spanish bilingual adults reviewing the two Mexican-Spanish translations of the Perceived Stress Scale.

"Being bilingual in English and Spanish was a criterion for inclusion in the study. Participants were between the ages of 19 and 39, with educational backgrounds that ranged from high school to advanced university degrees from institutions in the U.S." (Colina, 2022, p. 32)

Other Notes #

The issue count is an indirect index of comprehension difficulty (perceived, interview-elicited), not a direct reading-comprehension test; the authors note their study measured only perceived difficulty.

Caveats #

• Study measured only perceived difficulty not actual reading comprehension of the translations The study captured only participants' perceptions of difficulty and their stated preferences; it included no reading-comprehension test or readability instrument, so the findings speak to perceived difficulty and preference, not to objectively measured comprehension. The authors caution that translation B might be only perceived as more formal or difficult and note that future studies should test actual difficulty.
• Spelling errors in the literal translation could have influenced participants perceptions Translation A (the literal version) was a real published translation that contained spelling errors, which the authors retained because their interest was the translation approach rather than spelling. However, those spelling errors are a confound: they could independently have depressed participants' evaluations of translation A, so part of the observed preference and issue-count gap may reflect spelling rather than the literal-versus-functionalist approach. The authors recommend using an error-free text in future studies.
• Sample size of 20 is too small to statistically confirm the main effects The study enrolled only 20 participants. The authors acknowledge that a larger sample is needed to statistically confirm the main effects from the t-tests; the small n is also why the logistic-regression language-dominance trend reached only p = 0.058 rather than conventional significance. The authors frame the current findings as sufficient illustration rather than definitive confirmation.
• Sample was bilingual not monolingual LEP limiting generalizability to true limited-English readers All 20 participants were bilingual to some degree rather than monolingual or truly limited-English-proficient, because recruiting participants with no English skills in Tucson, Arizona was difficult. Their knowledge of English could make them more accepting of English-influenced, literal renderings than a genuinely LEP reader would be, which limits how far the preference and comprehension findings generalize to the target population — the authors call for a future study recruiting monolingual speakers.