Auto-Validate by-History: Auto-Program Data Quality Constraints to
  Validate Recurring Data Pipelines
Dezhan Tu Yeye He Weiwei Cui Song Ge Haidong Zhang Han Shi Dongmei Zhang Surajit Chaudhuri
Data pipelines are widely employed in modern enterprises to power a variety
of Machine-Learning (ML) and Business-Intelligence (BI) applications.
Crucially, these pipelines are \emph{recurring} (e.g., daily or hourly) in
production settings to keep data updated so that ML models can be re-trained
regularly, and BI dashboards refreshed frequently. However, data quality (DQ)
issues can often creep into recurring pipelines because of upstream schema and
data drift over time. As modern enterprises operate thousands of recurring
pipelines, today data engineers have to spend substantial efforts to
\emph{manually} monitor and resolve DQ issues, as part of their DataOps and
MLOps practices.
  Given the high human cost of managing large-scale pipeline operations, it is
imperative that we can \emph{automate} as much as possible. In this work, we
propose Auto-Validate-by-History (AVH) that can automatically detect DQ issues
in recurring pipelines, leveraging rich statistics from historical executions.
We formalize this as an optimization problem, and develop constant-factor
approximation algorithms with provable precision guarantees. Extensive
evaluations using 2000 production data pipelines at Microsoft demonstrate the
effectiveness and efficiency of AVH.