Can I trust my fake data -- A comprehensive quality assessment framework
  for synthetic tabular data in healthcare
Vibeke Binz Vallevik Aleksandar Babic Serena Elizabeth Marshall Severin Elvatun Helga Brøgger Sharmini Alagaratnam Bjørn Edwin Narasimha Raghavan Veeraragavan Anne Kjersti Befring Jan Franz Nygård
Ensuring safe adoption of AI tools in healthcare hinges on access to
sufficient data for training, testing and validation. In response to privacy
concerns and regulatory requirements, using synthetic data has been suggested.
Synthetic data is created by training a generator on real data to produce a
dataset with similar statistical properties. Competing metrics with differing
taxonomies for quality evaluation have been suggested, resulting in a complex
landscape. Optimising quality entails balancing considerations that make the
data fit for use, yet relevant dimensions are left out of existing frameworks.
We performed a comprehensive literature review on the use of quality evaluation
metrics on SD within the scope of tabular healthcare data and SD made using
deep generative methods. Based on this and the collective team experiences, we
developed a conceptual framework for quality assurance. The applicability was
benchmarked against a practical case from the Dutch National Cancer Registry.
We present a conceptual framework for quality assurance of SD for AI
applications in healthcare that aligns diverging taxonomies, expands on common
quality dimensions to include the dimensions of Fairness and Carbon footprint,
and proposes stages necessary to support real-life applications. Building trust
in synthetic data by increasing transparency and reducing the safety risk will
accelerate the development and uptake of trustworthy AI tools for the benefit
of patients. Despite the growing emphasis on algorithmic fairness and carbon
footprint, these metrics were scarce in the literature review. The overwhelming
focus was on statistical similarity using distance metrics while sequential
logic detection was scarce. A consensus-backed framework that includes all
relevant quality dimensions can provide assurance for safe and responsible
real-life applications of SD.