Rewrite the Stars
Xu Ma Xiyang Dai Yue Bai Yizhou Wang Yun Fu
Recent studies have drawn attention to the untapped potential of the "star
operation" (element-wise multiplication) in network design. While intuitive
explanations abound, the foundational rationale behind its application remains
largely unexplored. Our study attempts to reveal the star operation's ability
to map inputs into high-dimensional, non-linear feature spaces -- akin to
kernel tricks -- without widening the network. We further introduce StarNet, a
simple yet powerful prototype, demonstrating impressive performance and low
latency under compact network structure and efficient budget. Like stars in the
sky, the star operation appears unremarkable but holds a vast universe of
potential. Our work encourages further exploration across tasks, with codes
available at https://github.com/ma-xu/Rewrite-the-Stars.