Cloud-based VERAA mapped Spanish voice survey responses at 90% median accuracy, below English

Description #

Using the cloud-based pipeline, VERAA mapped scripted Spanish voice responses (translated to English by Whisper's large model, then mapped by Llama-2 70B) to the correct structured SDoH answer choices with a median per-question accuracy of 90% (interquartile range 80%–100%) across the 28 questions (Fig. 3; Table 3). Spanish accuracy was lower than the 100% observed for the same cloud pipeline on English responses, which the authors attribute to Spanish-to-English transcription/translation errors.

"The median and interquartile accuracy for the MCoPet Hybrid English approach was 85% [80%-95%]. Similarly, the accuracy for the Cloud English and Cloud Spanish approaches was 100% [90%-100%] and 90% [80%-100%], respectively." (Krishnamoorthy, 2024, p. 261)

"These factors contributed to reducing the accuracy of the Cloud Spanish approach as opposed to the Cloud English approach." (Krishnamoorthy, 2024, p. 263)

Methods Context #

What? #ⓘ

The observable: per-question mapping accuracy (ACC) for Spanish responses — the proportion of spoken Spanish answers correctly matched to the discrete survey answer choices, scored against the human consensus ground truth.

"The accuracy (ACC) for a question represents the accuracy of matching 20 English responses and 10 Spanish responses to the set of response choices for each question." (Krishnamoorthy, 2024, p. 261)

How? #ⓘ

Scripted Spanish audio responses were transcribed and translated to English by Whisper's large model on AWS, then mapped to structured answers by Llama-2 70B; accuracy was scored against the reviewer consensus ground truth. See A-0019ArtifactA-0019Initial AI draftVERAA (Voice-Enabled Response Analysis Agent)Why might someone use this thing? The Voice-Enabled Response Analysis Agent (VERAA) is a multilingual, voice-based conversational agent, embedded in the bedside "My Companion Pet" (MCoPet) edge-computing device, designed….

"In the second and third experiments, the English and Spanish audio files were passed to the AWS instance where Whisper's large model (1.550 Billion parameters) was used to produce audio transcription that Llama-2 70B used for inference." (Krishnamoorthy, 2024, p. 261)

Who? #ⓘ

The Spanish dataset comprised 10 scripted Spanish audio responses per question across the same 28 SDoH questions (from four All of Us surveys), read by one Spanish speaker; no real patients were involved.

"A panel of two English speakers and one Spanish speaker then read the generated scripts for a total of 20 English audio responses and 10 Spanish audio responses per question." (Krishnamoorthy, 2024, p. 259)

Other Notes #

This is the only non-English configuration evaluated. The Spanish accuracy loss relative to English is qualified by a caveat on translation errors (e.g., "Rarely"/"Fair" mistranslated), and the evaluation used scripted rather than spontaneous patient speech.

Caveats #

• VERAA was evaluated on scripted pre-formed responses, not spontaneous patient speech The reported mapping accuracies were obtained on a fixed set of pre-formed, scripted responses read aloud by a small panel of speakers, not on spontaneous, "wild type" answers from real patients. Scripted responses were used specifically so that accuracy could be compared across languages and speakers given the same response, but they under-represent the variability, disfluency, dialect, accent, and open-endedness of genuine patient speech. The full range of real-world responses therefore remains unevaluated, so the high accuracies may not transfer to deployment with actual patients.
• Spanish-to-English transcription and translation errors lowered VERAA's Cloud Spanish accuracy The lower accuracy of the Cloud Spanish configuration relative to Cloud English was driven not by the LLM mapping step but by errors in the upstream Spanish-to-English speech transcription/translation performed by Whisper's large model. The authors observed that Spanish answer words were sometimes mistranslated — for example, "Rarely" and "Fair" (for questions O2 and O3) were rendered as "Almost never" and "Just" — which then propagated into incorrect LLM mappings. This means the machine-translation stage is a distinct source of error for non-English responses, and the reported Spanish accuracy reflects combined ASR+translation+mapping error rather than mapping alone.
• VERAA was tested in only two languages (English and Spanish) The multilingual capability was evaluated in only two languages — English and Spanish — which does not represent the diverse linguistic backgrounds encountered in US healthcare. Because Spanish is a comparatively high-resource language for both speech recognition and machine translation, the demonstrated cross-language accuracy may not generalize to the lower-resource languages where LEP patients most often lack concordant services. The authors note the method is extendable to 99 languages, but performance across those languages was not measured.