Bilingual Conversational Agents for Parent-Child Shared Reading

In collaboration with the John Ganz Coony Center at Sesame Workshop, we developed a bilingual conversational agent embodied in the Sesame Street character, Rosita, along with a story reading interface, Rosita Reads with My Family. The goal of this project is to support culturally-engaging shared reading in Hispanic families. Unlike most existing conversational agents that only supports one-to-one dialogue with individual children, our conversational agent, Rosita, not only engages children in back-and-forth conversations about the story plots but also facilitates parental involvement by providing discussion prompts that connect the story to children’s life experience. This project contributes 1) a child-friendly bilingual conversational agent, 2) family-oriented discussion prompts, and 3) a culturally-engaging learning experience.

System Development and Initial User Study

In June 2022, we developed the Rosita Reads with My Family system and conducted a user study with 18 parent-child pairs from local Hispanic communities. We found that the bilingual agent effectively engages children verbally and encourages parental involvement in reading processes. The study also provides design insights for creating conversational agents for bilingual children.

Highlights

We identified two distinct interaction styles during child and family questions. This suggests parents naturally took different roles when interacting with the two types of questions despite that they received minimal instructions, indicating that our design met its goals.

1. The interactions during child questions were mostly Rosita-driven, meaning that Rosita initiated the question and interacted with the child independently.
2. The interactions during family questions were mostly co-driven by parents and Rosita. Parents typically extended the conversations with their children based on Rosita’s prompts.

Interviews with parents revealed that parents valued the cultural significance of Rosita Reads with My Family and the bilingual learning opportunities offered by Rosita.

System Improvement and Home Deployment Study

In September 2023, we carried out a one-week home deployment study of our system with 15 Hispanic families. We improved our system by creating an additonal book, Pinata Adventure, and different sets of family questions for daily rotation to make the use of the system more sustaining and engaging. In addition to interviewing the parents, this time, we also interviewed children about their perceptions of Rosita and their understanding of the two stories and the key vocabulary in the stories. Detailed analyses will be available soon.

Highlights

1. Children demonstrated significant gains in Spanish vocabulary, which were linked to their verbal engagement with the bilingual conversational agent, particularly in terms of response rate and accuracy.
2. The more elaborative the parents were during family discussions, the more language the children produced. These discussions included metalinguistic conversations that facilitated connections between the two languages.
3. Overall, the combination of child-directed questions and family discussion prompts provided by the bilingual conversational agent positively impacts children’s bilingual language development and enriches the home literacy environment.

Publications

1. He, K., Levine, J., Cervera, K., Ojeda-Ramirez, S., Xu, Y., & Warschauer, M. (2024, May). A Home Study of Parent-Child Co-Reading with a Bilingual Conversational Agent. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI’24), Association for Computing Machinery, Honolulu, HI, USA.
2. Xu, Y., He, K., Vigil, V., Ojeda-Ramirez, S., Liu, X., Levine, J., Cervera, K., & Warschauer, M. (2023, June) . “Rosita Reads With My Family”: Developing A Bilingual Conversational Agent to Support Parent-Child Shared Reading. In Proceedings of the 22nd ACM International Conference on Interaction Design and Children (IDC ’23). Association for Computing Machinery, Chicago, IL, USA
3. He, K., Xu, Y., Vigil, V., Ojeda-Ramirez, S., Cervera, K., & Warschauer, M. (2023, April). Developing a Bilingual Dialogic Ebook with a Conversational Agent for Hispanic Families [Paper session]. American Educational Research Association Annual Meeting, Chicago, IL.