Scaffolding Language Learning via Multi-modal Tutoring Systems with
  Pedagogical Instructions
Zhengyuan Liu Stella Xin Yin Carolyn Lee Nancy F. Chen
Intelligent tutoring systems (ITSs) that imitate human tutors and aim to
provide immediate and customized instructions or feedback to learners have
shown their effectiveness in education. With the emergence of generative
artificial intelligence, large language models (LLMs) further entitle the
systems to complex and coherent conversational interactions. These systems
would be of great help in language education as it involves developing skills
in communication, which, however, drew relatively less attention. Additionally,
due to the complicated cognitive development at younger ages, more endeavors
are needed for practical uses. Scaffolding refers to a teaching technique where
teachers provide support and guidance to students for learning and developing
new concepts or skills. It is an effective way to support diverse learning
needs, goals, processes, and outcomes. In this work, we investigate how
pedagogical instructions facilitate the scaffolding in ITSs, by conducting a
case study on guiding children to describe images for language learning. We
construct different types of scaffolding tutoring systems grounded in four
fundamental learning theories: knowledge construction, inquiry-based learning,
dialogic teaching, and zone of proximal development. For qualitative and
quantitative analyses, we build and refine a seven-dimension rubric to evaluate
the scaffolding process. In our experiment on GPT-4V, we observe that LLMs
demonstrate strong potential to follow pedagogical instructions and achieve
self-paced learning in different student groups. Moreover, we extend our
evaluation framework from a manual to an automated approach, paving the way to
benchmark various conversational tutoring systems.