A Family-oriented Bilingual Conversational Agent for Co-reading

In collaboration with the John Ganz Coony Center at Sesame Workshop, we developed a bilingual conversational agent embodied in the Sesame Street character, Rosita, along with a story reading interface, Rosita Reads with My Family. The goal of this project is to support culturally-engaging shared reading in Hispanic families. Unlike most existing conversational agents that only supports one-to-one dialogue with individual children, our conversational agent, Rosita, not only engages children in back-and-forth conversations about the story plots but also facilitates parental involvement by providing discussion prompts that connect the story to children’s life experience. This project contributes 1) a child-friendly bilingual conversational agent, 2) a family-oriented conversational interface, and 3) a culturally-engaging learning experience.

System Development and Initial User Study

In June 2022, we developed the Rosita Reads with My Family system and conducted a user study with 18 parent-child pairs from local Hispanic communities. We found that the bilingual agent effectively engages children verbally and encourages parental involvement in reading processes. The study also provides design insights for creating conversational agents for bilingual children.

Highlights

We identified two distinct interaction styles during child and family questions. This suggests parents naturally took different roles when interacting with the two types of questions despite that they received minimal instructions, indicating that our design met its goals.

1. The interactions during child questions were mostly Rosita-driven, meaning that Rosita initiated the question and interacted with the child independently.
2. The interactions during family questions were mostly co-driven by parents and Rosita. Parents typically extended the conversations with their children based on Rosita’s prompts.

Interviews with parents revealed that parents valued the cultural significance of Rosita Reads with My Family and the bilingual learning opportunities offered by Rosita.

System Improvement and Home Deployment Study

In September 2023, we carried out a one-week home deployment study of our system with 15 Hispanic families. We improved our system by creating an additonal book, Pinata Adventure, and different sets of family questions for daily rotation to make the use of the system more sustaining and engaging. In addition to interviewing the parents, this time, we also interviewed children about their perceptions of Rosita and their understanding of the two stories and the key vocabulary in the stories. Detailed analyses will be available soon.

What’s Next

We are considering two potential future directions:

1. Conduct a larger-scale home study to systematically evaluate the learning effect from co-reading with the bilingual conversational agent, in comparison to reading regular e-books.
2. Given that our team leveraged generative AI to create the second book in this study, I want to explore the potential of generative AI as an authoring tool. Specifically, I am interested in how teachers can collaborate with generative AI to create culturally engaging science stories for young learners to support science education.

Publications

1. Xu, Y., He, K., Vigil, V., Ojeda-Ramirez, S., Liu, X., Levine, J., Cervera, K., & Warschauer, M. (2023, June) . “Rosita Reads With My Family”: Developing A Bilingual Conversational Agent to Support Parent-Child Shared Reading. In Proceedings of the 22nd ACM International Conference on Interaction Design and Children (IDC ’23). Association for Computing Machinery, Chicago, IL, USA
2. He, K., Xu, Y., Vigil, V., Ojeda-Ramirez, S., Cervera, K., & Warschauer, M. (2023, April). Developing a Bilingual Dialogic Ebook with a Conversational Agent for Hispanic Families [Paper session]. American Educational Research Association Annual Meeting, Chicago, IL.