HyperFormer: Learning Expressive Sparse Feature Representations via
  Hypergraph Transformer
Kaize Ding Albert Jiongqian Liang Bryan Perrozi Ting Chen Ruoxi Wang Lichan Hong Ed H. Chi Huan Liu Derek Zhiyuan Cheng
Learning expressive representations for high-dimensional yet sparse features
has been a longstanding problem in information retrieval. Though recent deep
learning methods can partially solve the problem, they often fail to handle the
numerous sparse features, particularly those tail feature values with
infrequent occurrences in the training data. Worse still, existing methods
cannot explicitly leverage the correlations among different instances to help
further improve the representation learning on sparse features since such
relational prior knowledge is not provided. To address these challenges, in
this paper, we tackle the problem of representation learning on feature-sparse
data from a graph learning perspective. Specifically, we propose to model the
sparse features of different instances using hypergraphs where each node
represents a data instance and each hyperedge denotes a distinct feature value.
By passing messages on the constructed hypergraphs based on our Hypergraph
Transformer (HyperFormer), the learned feature representations capture not only
the correlations among different instances but also the correlations among
features. Our experiments demonstrate that the proposed approach can
effectively improve feature representation learning on sparse features.