ChatShop: Interactive Information Seeking with Language Agents
Sanxing Chen Sam Wiseman Bhuwan Dhingra
The desire and ability to seek new information strategically are fundamental
to human learning but often overlooked in current language agent development.
Using a web shopping task as an example, we show that it can be reformulated
and solved as a retrieval task without a requirement of interactive information
seeking. We then redesign the task to introduce a new role of shopper, serving
as a realistically constrained communication channel. The agents in our
proposed ChatShop task explore user preferences in open-ended conversation to
make informed decisions. Our experiments demonstrate that the proposed task can
effectively evaluate the agent's ability to explore and gradually accumulate
information through multi-turn interaction. We also show that LLM-simulated
shoppers serve as a good proxy to real human shoppers and discover similar
error patterns of agents.