When and What to Ask Through World States and Text Instructions: IGLU
  NLP Challenge Solution
Zhengxiang Shi Jerome Ramos To Eun Kim Xi Wang Hossein A. Rahmani Aldo Lipani
In collaborative tasks, effective communication is crucial for achieving
joint goals. One such task is collaborative building where builders must
communicate with each other to construct desired structures in a simulated
environment such as Minecraft. We aim to develop an intelligent builder agent
to build structures based on user input through dialogue. However, in
collaborative building, builders may encounter situations that are difficult to
interpret based on the available information and instructions, leading to
ambiguity. In the NeurIPS 2022 Competition NLP Task, we address two key
research questions, with the goal of filling this gap: when should the agent
ask for clarification, and what clarification questions should it ask? We move
towards this target with two sub-tasks, a classification task and a ranking
task. For the classification task, the goal is to determine whether the agent
should ask for clarification based on the current world state and dialogue
history. For the ranking task, the goal is to rank the relevant clarification
questions from a pool of candidates. In this report, we briefly introduce our
methods for the classification and ranking task. For the classification task,
our model achieves an F1 score of 0.757, which placed the 3rd on the
leaderboard. For the ranking task, our model achieves about 0.38 for Mean
Reciprocal Rank by extending the traditional ranking model. Lastly, we discuss
various neural approaches for the ranking task and future direction.