Search Beyond Queries: Training Smaller Language Models for Web
  Interactions via Reinforcement Learning
Moghis Fereidouni A. B. Siddique
Traditional search systems focus on query formulation for effective results
but face challenges in scenarios such as product searches where crucial product
details (e.g., size, color) remain concealed until users visit specific product
pages. This highlights the need for intelligent web navigation agents capable
of formulating queries and navigating web pages according to users' high-level
intents. In response to this need, this work introduces a Grounded Language
Agent for Intelligent Web Interactions, called GLAINTEL. Drawing upon
advancements in language modeling and reinforcement learning, GLAINTEL
investigates the efficacy of transformer-based models in enhancing the search
capabilities of interactive web environments. Given the dynamic action space
for each state in web navigation, GLAINTEL employs the Flan-T5 architecture and
incorporates language modeling and value estimation heads. This work focuses on
training smaller language models as agents across various scenarios,
systematically evaluating the impact of human demonstrations on the training
process. Specifically, we investigate scenarios where no human demonstrations
are available and subsequently assess the effective utilization of such
demonstrations. We also explore unsupervised domain adaptation for situations
where demonstrations are confined to a specific domain. Experimental
evaluations across diverse setups demonstrate the effectiveness of training
agents in unsupervised settings, outperforming in-context learning-based
approaches that employ larger models with up to 540 billion parameters.
Surprisingly, behavioral cloning-based methods that straightforwardly use human
demonstrations do not outperform unsupervised learning-based methods.
Additionally, combining human demonstrations with Reinforcement Learning-based
training yields results comparable to models utilizing GPT-4.