SLT 2026
Multimodal speech processing and controllable generation for healthcare and accessibility
Ph.D. Candidate, Language Technologies Institute
Karen Rosero
(she/her)
Ph.D. Candidate, Language Technologies Institute
About Me
I am a Ph.D. candidate at the Language Technologies Institute, Carnegie Mellon University, where I am a member of the Multimodal Speech Processing (MSP) Lab led by Prof. Carlos Busso. My research sits at the intersection of multimodal speech processing and controllable generation: I model atypical speech and speech-grounded facial dynamics, and use generative models to make speech and facial technologies work for the people they most often fail, including children with speech disorders, individuals with craniofacial anomalies, and second-language learners.
My work spans generative reconstruction of disordered speech, synthetic data for low-resource clinical ASR, anatomically consistent facial analysis and synthesis, and interpretable multimodal biomarkers of speech disorders. In summer 2026 I interned with the Apple Accessibility team in Seattle, working on atypical speech generation.
My Ph.D. research is developed in collaboration with Children’s Health Dallas and UT Southwestern Medical Center, with Dr. Rami Hallac, Director of Research at the Analytical Imaging and Modeling (AIM) Center at Children’s Medical Center and Associate Professor at UT Southwestern. I communicate my work to both the AI and the medical communities: beyond AI venues such as ICMI and Interspeech, I have presented at clinical meetings, including the American Cleft Palate–Craniofacial Association annual meeting (ACPA 2024) and the International Congress on Cleft Lip/Palate (CLEFT 2025), and published in The Cleft Palate Craniofacial Journal.
Education
Ph.D. in Language and Information Technology
2023 – Expected Summer 2027
Carnegie Mellon University
Master of Language Technologies (M.L.T.)
2025 – Expected Summer 2027
Carnegie Mellon University
M.S. in Electrical Engineering
2020 – 2022
University of Campinas (UNICAMP), Brazil
B.S. in Electronics and Telecommunications Engineering
2014 – 2019
Universidad de las Fuerzas Armadas ESPE, Ecuador
Education
Ph.D. in Language and Information Technology
2023 – Expected Summer 2027
Carnegie Mellon University
Transferred from The University of Texas at Dallas
Advisor: Prof. Carlos Busso
Master of Language Technologies (M.L.T.)
2025 – Expected Summer 2027
Carnegie Mellon University
Earned en route to the Ph.D.
Language Technologies Institute
M.S. in Electrical Engineering
2020 – 2022
University of Campinas (UNICAMP), Brazil
FAPESP Research Excellence Fellowship
Advisor: Prof. Bruno S. Masiero
B.S. in Electronics and Telecommunications Engineering
2014 – 2019
Universidad de las Fuerzas Armadas ESPE, Ecuador
Best Graduate Award
Advisor: Prof. Felipe Grijalva
Interests
AI for communication disordersMultimodal assessment of speech productionArticulatory modeling of speechPersonalized speech generationGenerative AI for limited clinical dataSpeech and vision large language modelsAccessible voice technologyInterpretable and responsible AI in healthcare
Note
I am looking for full-time research positions starting in Fall 2027. If my profile looks like a good fit, please reach out at kroseroj at cmu dot edu or karenrosero14 at gmail dot com.
Research
Clinicians assess speech by listening, watching the face, and reasoning about how sounds are produced, yet most AI tools analyze audio alone and are trained mostly on adults with typical speech. I develop AI tools that analyze speech production the way clinicians observe it, for cleft lip and palate, speech sound disorders in children, and dysarthria.
Multimodal assessment of speech production
Combining speech, facial video, and articulatory information, with cues tailored to each condition: anatomy-aware landmarks and lip assessment after cleft repair, automatic consonant accuracy for children’s speech that correlates with expert annotation (ρ = 0.63), and dysarthria severity from speech, the intended transcript, and facial asymmetry.
ICMI 2026
SLT 2026
FG 2024
+1 more
Generative models for underrepresented speakers
Clinical data are scarce and identify the patient. I adapt models pretrained on typical data to new populations using little real data: synthetic disordered speech that reduces the data needed to train speech recognition for children, text-guided synthesis of cleft lip images, and anonymization that keeps the clinically relevant anatomy (98.9% of anonymized faces identified as different people).
Interspeech 2025
ICMI 2025
CLEFT 2025
Interpretable tools for clinicians, patients, and families
Assessments that report what differs and where, by comparing an atypical production with a typical reference from the same person, such as the child’s intended utterance in their own voice or a normalized version of the same face. Next: feedback for home practice, and voice technology that works for speakers with speech disorders.
SLT 2026
ICMI 2024
Code & Demos
Open-source releases accompanying my papers
ICMI 2026
CLEFT 2025
Anonymizing Faces, Preserving Anomalies
Privacy protection for clinical photos of patients with cleft lip. A diffusion model synthesizes a new face around the patient’s preserved orofacial region.
ICMI 2024
Lip Abnormality Detection for CLP
Lip normalization and Siamese abnormality detection for patients with repaired cleft lip and palate, trained without patient data.
IEEE/ACM TASLP 2023
Selected Publications
The full list is available on Google Scholar and on the publications page.
(2026).
Generative Reconstruction of Pediatric Disordered Speech for Automated Clinical Evaluation.
IEEE Spoken Language Technology Workshop (SLT), Palermo, Italy.
(2026).
Multimodal Biomarkers of Dysarthria: Severity-Conditioned Contrastive Alignment of Speech, Text, and Facial Asymmetry.
Proc. 28th ACM International Conference on Multimodal Interaction (ICMI), Naples, Italy.
(2025).
DiffusionCleft: Facial Anomaly Synthesis Guided by Text.
Proc. 27th ACM International Conference on Multimodal Interaction (ICMI), pp. 506–515.
(2025).
Anonymizing Faces, Preserving Anomalies: Deep-Learning Based Privacy Protection for Data of Patients with Cleft Lip.
International Congress on Cleft Lip/Palate and Related Craniofacial Anomalies (CLEFT), Kyoto, Japan (abstract).
(2025).
Advancing Pediatric ASR: The Role of Voice Generation in Disordered Speech.
Proc. Interspeech 2025, Annual Conference of the International Speech Communication Association (ISCA), Rotterdam, The Netherlands, pp. 2890–2894.
(2025).
Deep Learning-Based Assessment of Lip Symmetry for Patients with Repaired Cleft Lip.
The Cleft Palate Craniofacial Journal, 62(2), 289–299.
(2024).
Lip Abnormality Detection for Patients with Repaired Cleft Lip and Palate: A Lip Normalization Approach.
Proc. 26th ACM International Conference on Multimodal Interaction (ICMI), pp. 479–487.
(2024).
Enhanced Facial Landmarks Detection for Patients with Repaired Cleft Lip and Palate.
Proc. IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG).
(2023).
Sound Events Localization and Detection Using Bio-Inspired Gammatone Filters and Temporal Convolutional Neural Networks.
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2314–2324.
News
- Oct 2026Multimodal Biomarkers of Dysarthria oral presentation at ICMI 2026 (Naples, Italy).
- Sep 2026Released code and checkpoints for MMDyS (ICMI 2026) on GitHub and Hugging Face.
- Sep 2026ChiReSSD (SLT 2026): code, model and an audio demo are out.
- Aug 2026Completed a 3-month summer internship with the Apple Accessibility team in Seattle, mentored by Dianna Yee, Colin Lea and Leah Findlater.
- Jul 2026PRiSM, a benchmark for phone realization in speech models, in collaboration with the ChangeLing Lab, appears at ACL 2026.
- Oct 2025Presented DiffusionCleft at ICMI 2025.
- Aug 2025Two papers at Interspeech 2025 (Rotterdam): pediatric disordered-speech ASR as first author, and CS-FLEURS in collaboration with WAVLab.
- May 2025Presented my Ph.D. thesis proposal, which was approved. Committee: Berrak Sisman (Johns Hopkins University), Jeffrey Cohn (University of Pittsburgh), Louis-Philippe Morency (CMU), Visar Berisha (Arizona State University), David Mortensen (CMU), and my advisor Carlos Busso (CMU).
- Dec 2024Transferred from The University of Texas at Dallas to Carnegie Mellon University to continue working with Prof. Carlos Busso.
- 2023Finished my Master's degree at UNICAMP (Brazil) and moved to Texas, USA, to start my Ph.D. at The University of Texas at Dallas!