Hacking Predictors Means Hacking Cars: Using Sensitivity Analysis to
  Identify Trajectory Prediction Vulnerabilities for Autonomous Driving
  Security
Marsalis Gibson David Babazadeh Claire Tomlin Shankar Sastry
Adversarial attacks on learning-based trajectory predictors have already been
demonstrated. However, there are still open questions about the effects of
perturbations on trajectory predictor inputs other than state histories, and
how these attacks impact downstream planning and control. In this paper, we
conduct a sensitivity analysis on two trajectory prediction models,
Trajectron++ and AgentFormer. We observe that between all inputs, almost all of
the perturbation sensitivities for Trajectron++ lie only within the most recent
state history time point, while perturbation sensitivities for AgentFormer are
spread across state histories over time. We additionally demonstrate that,
despite dominant sensitivity on state history perturbations, an undetectable
image map perturbation made with the Fast Gradient Sign Method can induce large
prediction error increases in both models. Even though image maps may
contribute slightly to the prediction output of both models, this result
reveals that rather than being robust to adversarial image perturbations,
trajectory predictors are susceptible to image attacks. Using an
optimization-based planner and example perturbations crafted from sensitivity
results, we show how this vulnerability can cause a vehicle to come to a sudden
stop from moderate driving speeds.