LearnedWMP: Workload Memory Prediction Using Distribution of Query
  Templates
Shaikh Quader Andres Jaramillo Sumona Mukhopadhyay Ghadeer Abuoda Calisto Zuzarte David Kalmuk Marin Litoiu Manos Papagelis
In a modern DBMS, working memory is frequently the limiting factor when
processing in-memory analytic query operations such as joins, sorting, and
aggregation. Existing resource estimation approaches for a DBMS estimate the
resource consumption of a query by computing an estimate of each individual
database operator in the query execution plan. Such an approach is slow and
error-prone as it relies upon simplifying assumptions, such as uniformity and
independence of the underlying data. Additionally, the existing approach
focuses on individual queries separately and does not factor in other queries
in the workload that may be executed concurrently. In this research, we are
interested in query performance optimization under concurrent execution of a
batch of queries (a workload). Specifically, we focus on predicting the memory
demand for a workload rather than providing separate estimates for each query
within it. We introduce the problem of workload memory prediction and formalize
it as a distribution regression problem. We propose Learned Workload Memory
Prediction (LearnedWMP) to improve and simplify estimating the working memory
demands of workloads. Through a comprehensive experimental evaluation, we show
that LearnedWMP reduces the memory estimation error of the
state-of-the-practice method by up to 47.6%. Compared to an alternative
single-query model, during training and inferencing, the LearnedWMP model and
its variants were 3x to 10x faster. Moreover, LearnedWMP-based models were at
least 50% smaller in most cases. Overall, the results demonstrate the
advantages of the LearnedWMP approach and its potential for a broader impact on
query performance optimization.