Enhancing Virtual Assistant Intelligence: Precise Area Targeting for
  Instance-level User Intents beyond Metadata
Mengyu Chen Zhenchang Xing Jieshan Chen Chunyang Chen Qinghua Lu
Virtual assistants have been widely used by mobile phone users in recent
years. Although their capabilities of processing user intents have been
developed rapidly, virtual assistants in most platforms are only capable of
handling pre-defined high-level tasks supported by extra manual efforts of
developers. However, instance-level user intents containing more detailed
objectives with complex practical situations, are yet rarely studied so far. In
this paper, we explore virtual assistants capable of processing instance-level
user intents based on pixels of application screens, without the requirements
of extra extensions on the application side. We propose a novel cross-modal
deep learning pipeline, which understands the input vocal or textual
instance-level user intents, predicts the targeting operational area, and
detects the absolute button area on screens without any metadata of
applications. We conducted a user study with 10 participants to collect a
testing dataset with instance-level user intents. The testing dataset is then
utilized to evaluate the performance of our model, which demonstrates that our
model is promising with the achievement of 64.43% accuracy on our testing
dataset.