BESTMVQA: A Benchmark Evaluation System for Medical Visual Question
  Answering
Xiaojie Hong Zixin Song Liangzhi Li Xiaoli Wang Feiyan Liu
Medical Visual Question Answering (Med-VQA) is a very important task in
healthcare industry, which answers a natural language question with a medical
image. Existing VQA techniques in information systems can be directly applied
to solving the task. However, they often suffer from (i) the data insufficient
problem, which makes it difficult to train the state of the arts (SOTAs) for
the domain-specific task, and (ii) the reproducibility problem, that many
existing models have not been thoroughly evaluated in a unified experimental
setup. To address these issues, this paper develops a Benchmark Evaluation
SysTem for Medical Visual Question Answering, denoted by BESTMVQA. Given
self-collected clinical data, our system provides a useful tool for users to
automatically build Med-VQA datasets, which helps overcoming the data
insufficient problem. Users also can conveniently select a wide spectrum of
SOTA models from our model library to perform a comprehensive empirical study.
With simple configurations, our system automatically trains and evaluates the
selected models over a benchmark dataset, and reports the comprehensive results
for users to develop new techniques or perform medical practice. Limitations of
existing work are overcome (i) by the data generation tool, which automatically
constructs new datasets from unstructured clinical data, and (ii) by evaluating
SOTAs on benchmark datasets in a unified experimental setup. The demonstration
video of our system can be found at https://youtu.be/QkEeFlu1x4A. Our code and
data will be available soon.