Sibyl: Forecasting Time-Evolving Query Workloads
Hanxian Huang Tarique Siddiqui Rana Alotaibi Carlo Curino Jyoti Leeka Alekh Jindal Jishen Zhao Jesus Camacho-Rodriguez Yuanyuan Tian
Database systems often rely on historical query traces to perform
workload-based performance tuning. However, real production workloads are
time-evolving, making historical queries ineffective for optimizing future
workloads. To address this challenge, we propose SIBYL, an end-to-end machine
learning-based framework that accurately forecasts a sequence of future
queries, with the entire query statements, in various prediction windows.
Drawing insights from real-workloads, we propose template-based featurization
techniques and develop a stacked-LSTM with an encoder-decoder architecture for
accurate forecasting of query workloads. We also develop techniques to improve
forecasting accuracy over large prediction windows and achieve high scalability
over large workloads with high variability in arrival rates of queries.
Finally, we propose techniques to handle workload drifts. Our evaluation on
four real workloads demonstrates that SIBYL can forecast workloads with an
$87.3\%$ median F1 score, and can result in $1.7\times$ and $1.3\times$
performance improvement when applied to materialized view selection and index
selection applications, respectively.