Improve Long-term Memory Learning Through Rescaling the Error Temporally
Shida Wang Zhanglu Yan
This paper studies the error metric selection for long-term memory learning
in sequence modelling. We examine the bias towards short-term memory in
commonly used errors, including mean absolute/squared error. Our findings show
that all temporally positive-weighted errors are biased towards short-term
memory in learning linear functionals. To reduce this bias and improve
long-term memory learning, we propose the use of a temporally rescaled error.
In addition to reducing the bias towards short-term memory, this approach can
also alleviate the vanishing gradient issue. We conduct numerical experiments
on different long-memory tasks and sequence models to validate our claims.
Numerical results confirm the importance of appropriate temporally rescaled
error for effective long-term memory learning. To the best of our knowledge,
this is the first work that quantitatively analyzes different errors' memory
bias towards short-term memory in sequence modelling.