The METRIC-framework for assessing data quality for trustworthy AI in
  medicine: a systematic review
Daniel Schwabe Katinka Becker Martin Seyferth Andreas Klaß Tobias Schäffter
The adoption of machine learning (ML) and, more specifically, deep learning
(DL) applications into all major areas of our lives is underway. The
development of trustworthy AI is especially important in medicine due to the
large implications for patients' lives. While trustworthiness concerns various
aspects including ethical, technical and privacy requirements, we focus on the
importance of data quality (training/test) in DL. Since data quality dictates
the behaviour of ML products, evaluating data quality will play a key part in
the regulatory approval of medical AI products. We perform a systematic review
following PRISMA guidelines using the databases PubMed and ACM Digital Library.
We identify 2362 studies, out of which 62 records fulfil our eligibility
criteria. From this literature, we synthesise the existing knowledge on data
quality frameworks and combine it with the perspective of ML applications in
medicine. As a result, we propose the METRIC-framework, a specialised data
quality framework for medical training data comprising 15 awareness dimensions,
along which developers of medical ML applications should investigate a dataset.
This knowledge helps to reduce biases as a major source of unfairness, increase
robustness, facilitate interpretability and thus lays the foundation for
trustworthy AI in medicine. Incorporating such systematic assessment of medical
datasets into regulatory approval processes has the potential to accelerate the
approval of ML products and builds the basis for new standards.