In-Database Data Imputation
Massimo Perini Milos Nikolic
Missing data is a widespread problem in many domains, creating challenges in
data analysis and decision making. Traditional techniques for dealing with
missing data, such as excluding incomplete records or imputing simple estimates
(e.g., mean), are computationally efficient but may introduce bias and disrupt
variable relationships, leading to inaccurate analyses. Model-based imputation
techniques offer a more robust solution that preserves the variability and
relationships in the data, but they demand significantly more computation time,
limiting their applicability to small datasets.
  This work enables efficient, high-quality, and scalable data imputation
within a database system using the widely used MICE method. We adapt this
method to exploit computation sharing and a ring abstraction for faster model
training. To impute both continuous and categorical values, we develop
techniques for in-database learning of stochastic linear regression and
Gaussian discriminant analysis models. Our MICE implementations in PostgreSQL
and DuckDB outperform alternative MICE implementations and model-based
imputation techniques by up to two orders of magnitude in terms of computation
time, while maintaining high imputation quality.