Multi-modal Representation Learning for Social Post Location Inference
Ruiting Dai Jiayi Luo Xucheng Luo Lisi Mo Wanlun Ma Fan Zhou
Inferring geographic locations via social posts is essential for many
practical location-based applications such as product marketing,
point-of-interest recommendation, and infector tracking for COVID-19. Unlike
image-based location retrieval or social-post text embedding-based location
inference, the combined effect of multi-modal information (i.e., post images,
text, and hashtags) for social post positioning receives less attention. In
this work, we collect real datasets of social posts with images, texts, and
hashtags from Instagram and propose a novel Multi-modal Representation Learning
Framework (MRLF) capable of fusing different modalities of social posts for
location inference. MRLF integrates a multi-head attention mechanism to enhance
location-salient information extraction while significantly improving location
inference compared with single domain-based methods. To overcome the noisy
user-generated textual content, we introduce a novel attention-based
character-aware module that considers the relative dependencies between
characters of social post texts and hashtags for flexible multi-model
information fusion. The experimental results show that MRLF can make accurate
location predictions and open a new door to understanding the multi-modal data
of social posts for online inference tasks.