ADL4D: Towards A Contextually Rich Dataset for 4D Activities of Daily
  Living
Marsil Zakour Partha Pratim Nath Ludwig Lohmer Emre Faik Gökçe Martin Piccolrovazzi Constantin Patsch Yuankai Wu Rahul Chaudhari Eckehard Steinbach
Hand-Object Interactions (HOIs) are conditioned on spatial and temporal
contexts like surrounding objects, previous actions, and future intents (for
example, grasping and handover actions vary greatly based on objects proximity
and trajectory obstruction). However, existing datasets for 4D HOI (3D HOI over
time) are limited to one subject interacting with one object only. This
restricts the generalization of learning-based HOI methods trained on those
datasets. We introduce ADL4D, a dataset of up to two subjects interacting with
different sets of objects performing Activities of Daily Living (ADL) like
breakfast or lunch preparation activities. The transition between multiple
objects to complete a certain task over time introduces a unique context
lacking in existing datasets. Our dataset consists of 75 sequences with a total
of 1.1M RGB-D frames, hand and object poses, and per-hand fine-grained action
annotations. We develop an automatic system for multi-view multi-hand 3D pose
annotation capable of tracking hand poses over time. We integrate and test it
against publicly available datasets. Finally, we evaluate our dataset on the
tasks of Hand Mesh Recovery (HMR) and Hand Action Segmentation (HAS).