Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in
  Energy and Contingency Reserve Markets
Jinhao Li Changlong Wang Yanru Zhang Hao Wang
The battery energy storage system (BESS) has immense potential for enhancing
grid reliability and security through its participation in the electricity
market. BESS often seeks various revenue streams by taking part in multiple
markets to unlock its full potential, but effective algorithms for joint-market
participation under price uncertainties are insufficiently explored in the
existing research. To bridge this gap, we develop a novel BESS joint bidding
strategy that utilizes deep reinforcement learning (DRL) to bid in the spot and
contingency frequency control ancillary services (FCAS) markets. Our approach
leverages a transformer-based temporal feature extractor to effectively respond
to price fluctuations in seven markets simultaneously and helps DRL learn the
best BESS bidding strategy in joint-market participation. Additionally, unlike
conventional "black-box" DRL model, our approach is more interpretable and
provides valuable insights into the temporal bidding behavior of BESS in the
dynamic electricity market. We validate our method using realistic market
prices from the Australian National Electricity Market. The results show that
our strategy outperforms benchmarks, including both optimization-based and
other DRL-based strategies, by substantial margins. Our findings further
suggest that effective temporal-aware bidding can significantly increase
profits in the spot and contingency FCAS markets compared to individual market
participation.