S-MINDS translated the full English-Spanish-English medication-counseling string with ~98% adjusted accuracy

Description #

Across 379 diabetes medication action plan (MAP) recommendations tested over 608 attempts by 21 Spanish-speaking LEP patients, the S-MINDS device translated the full English–Spanish–English communication string with an overall adjusted accuracy of ~98% (Table 6). The clinician-side steps (pharmacist speaking English to the device, then the device's audio Spanish translation) were 100% correct. The patient-side steps had lower raw accuracy — 91% for the patient restating the Spanish recommendation and 90% for the device's audio English translation of the patient — rising to 99% and 98% respectively on adjusted scores that exclude "patient forgot" responses. Absolutely incorrect full communication strings occurred in roughly 2% of tries.

"Translation accuracy was 100% for the pharmacist speaking English to the device followed by playing an audio Spanish translation of the pharmacist's MAP recommendation (Table 6). Raw accuracy scores were 91% and 90%, respectively, for (a) the patient speaking, in Spanish, the Spanish MAP recommendation that he/she heard from the device in Spanish and (b) the device's audio English translation of what the patient said in Spanish." (Soller, 2012, p. 931)

"The total number of absolutely incorrect full communication strings was approximately 2%—or an overall 98% accuracy score." (Soller, 2012, p. 932)

"S-MINDS demonstrated 98% accuracy based on adjusted scores to correct for patients' forgetfulness." (Soller, 2012, p. 934)

Methods Context #

What? #ⓘ

The observable: the accuracy of each verbal and written step of the English–Spanish–English communication string, coded as correct, conceptually correct, partial, or incorrect.

"Verbal and written statements at each communication step were categorized as (a) correct, if the statement was an exact representation of the stated or written statement; (b) conceptually correct, if the statement had the same meaning as determined by the researcher (e.g., I will test [versus check] my blood sugars twice a day); (c) partial, if only a portion of the statement was represented correctly; or (d) incorrect, if the entire statement was a misrepresentation." (Soller, 2012, p. 929)

How? #ⓘ

Prospective assessment of each communication step of the A-0020ArtifactA-0020Initial AI draftS-MINDS concept-based speech translation systemWhy might someone use this thing? S-MINDS (Speaking Multilingual Interactive Natural Dialog System), developed by Fluential Inc. and evaluated with the UCSF School of Pharmacy, is a bidirectional English–Spanish speech t… prototype, with two bilingual researchers plus the practitioner coding each step, using a three-attempt teach-back procedure for the patient's Spanish restatement.

"This was a prospective assessment of each communication step of the prototype system with documentation of accuracy, patient satisfaction, and laboratory comparisons with other translation systems." (Soller, 2012, p. 928)

"Accuracy was documented by two researchers who were fluent in medical English and Spanish and the practitioner." (Soller, 2012, p. 929)

Who? #ⓘ

21 Spanish-preferring LEP patients (mainly women, 60%) with type 2 diabetes at the UCSF–St. Anthony Diabetes Telepharmacy Clinic, contributing 379 recommendations over 608 attempts.

"Patients were included if they were current patients in the UCSF–St. Anthony Diabetes Telepharmacy Clinic, were 21–85 years of age, had a physician diagnosis of type 2 diabetes, were prescribed one or more oral or injectable diabetes medicines, had electronic medical record (EMR) documentation of LEP status and patient preference of Spanish for medical visits, and had completed a consent form." (Soller, 2012, p. 929)

"A total of 379 recommendations were tested over 608 attempts by 21 patients." (Soller, 2012, p. 931)

Other Notes #

The headline 98% is an adjusted figure. Raw exact-plus-conceptual accuracy on the patient side was 91%/90% and depended on allowing up to three teach-back attempts; the adjustment removes "patient forgot" responses (see the qualifying caveats).

Caveats #

• The 98% S-MINDS accuracy is an adjusted figure excluding patient-forgot responses and allowing three teach-back attempts The headline ~98% accuracy is an adjusted score: it excludes "patient forgot" responses (patients who failed to repeat the device's Spanish audio due to cognitive difficulty) and it credits recommendations that were only conveyed correctly after a second or third teach-back attempt. On the raw single-pass patient side, only 66% were exactly correct and a further 25% were partial; 28% of tries needed a second attempt and 11% a third. The device-to-patient-to-device leg therefore depended heavily on patient memory and repeated attempts, so the 98% figure overstates first-pass performance in routine use.
• S-MINDS accuracy and satisfaction come from a small single-clinic underserved sample limiting generalizability The accuracy and satisfaction findings were generated in a small feasibility study of 21 underserved Spanish-speaking LEP diabetes patients at a single clinic, with a small number of practitioners. The authors themselves flag limited generalizability. There was also no comparator arm (e.g., a human interpreter or a general-purpose app used with the same patients), so the in-patient results speak to feasibility rather than to relative effectiveness.