A Scenario-Based Functional Testing Approach to Improving DNN
  Performance
Hong Zhu Thi Minh Tam Tran Aduen Benjumea Andrew Bradley
This paper proposes a scenario-based functional testing approach for
enhancing the performance of machine learning (ML) applications. The proposed
method is an iterative process that starts with testing the ML model on various
scenarios to identify areas of weakness. It follows by a further testing on the
suspected weak scenarios and statistically evaluate the model's performance on
the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is
confirmed by test results, the treatment of the model is performed by
retraining the model using a transfer learning technique with the original
model as the base and applying a set of training data specifically targeting
the treated scenarios plus a subset of training data selected at random from
the original train dataset to prevent the so-call catastrophic forgetting
effect. Finally, after the treatment, the model is assessed and evaluated again
by testing on the treated scenarios as well as other scenarios to check if the
treatment is effective and no side effect caused. The paper reports a case
study with a real ML deep neural network (DNN) model, which is the perception
system of an autonomous racing car. It is demonstrated that the method is
effective in the sense that DNN model's performance can be improved. It
provides an efficient method of enhancing ML model's performance with much less
human and compute resource than retrain from scratch.