T-SaS: Toward Shift-aware Dynamic Adaptation for Streaming Data
Weijieying Ren Tianxiang Zhao Wei Qin Kunpeng Liu
In many real-world scenarios, distribution shifts exist in the streaming data
across time steps. Many complex sequential data can be effectively divided into
distinct regimes that exhibit persistent dynamics. Discovering the shifted
behaviors and the evolving patterns underlying the streaming data are important
to understand the dynamic system. Existing methods typically train one robust
model to work for the evolving data of distinct distributions or sequentially
adapt the model utilizing explicitly given regime boundaries. However, there
are two challenges: (1) shifts in data streams could happen drastically and
abruptly without precursors. Boundaries of distribution shifts are usually
unavailable, and (2) training a shared model for all domains could fail to
capture varying patterns. This paper aims to solve the problem of sequential
data modeling in the presence of sudden distribution shifts that occur without
any precursors. Specifically, we design a Bayesian framework, dubbed as T-SaS,
with a discrete distribution-modeling variable to capture abrupt shifts of
data. Then, we design a model that enable adaptation with dynamic network
selection conditioned on that discrete variable. The proposed method learns
specific model parameters for each distribution by learning which neurons
should be activated in the full network. A dynamic masking strategy is adopted
here to support inter-distribution transfer through the overlapping of a set of
sparse networks. Extensive experiments show that our proposed method is
superior in both accurately detecting shift boundaries to get segments of
varying distributions and effectively adapting to downstream forecast or
classification tasks.