Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was proposed, which has demonstrated improved performances over the conventional convolu- tional neural networks (CNN). In this paper, we address several limitations of the baseline negated architecture by proposing two further optimized architectures: a coarser-grained gated architecture employing (feature) group-level fusion weights and a two-stage gated architectures leveraging both the group-level and feature- level fusion weights. Using driving mode prediction and human activity recogni- tion datasets, we demonstrate the significant performance improvements brought by the proposed gated architectures and also their robustness in the presence of sensor noise and failures.
 Sensor fusion is an essential technology to autonomous systems such as self-driving cars and mobile robots. In advanced driver-assistance systems (ADAS), many sensors such as cameras, ultrasonic sensors, and LiDARs are utilized for enhanced safety and driving experience. Sensor fusion based vehicle and robot control technologies have been explored BID9 BID7 BID4 BID2 BID8 . In addition, devices like smartphones and smartwatches typically integrate a number of sensors, making these devices a suitable platform for running sensor fusion applications such as activity recognition. Several sensor fusion techniques for activity recognition have been proposed BID10 BID3 BID11 BID5 .More specifically, in BID2 , a deep reinforcement learning based sensor fusion algorithm is discussed for robot control. A Kuka YouBot is used with multiple LiDAR sensors for simulations. In terms of sensor fusion, early fusion which concatenates all inputs as feature planes is compared with three other fusion techniques: concatenating convolution layer outputs, reducing the concatenated features with a 1x1 convolution, and accumulating convolution outputs. However, sensor noise and failures are not considered in this work. In addition , due to the fact that only the same type of sensory inputs is used in the experiments, the performance of sensor fusion based on different kinds of sensory inputs is unclear.Among sensor fusion techniques employed in automobiles, BID9 exploit neural networks with a Kalman filter for vehicle roll angle estimation, and show the advantage of using an inertial measurement unit (IMU) without additional suspension deflection sensors. BID7 consider a sensor-rich platform with a learning algorithm for maneuver prediction. Long short-term memory (LSTM) networks, which are a type of recurrent neural networks (RNN) are used with sensory inputs from cameras, GPS, and speedometers. BID4 propose joint probabilistic data fusion for road environments. However, neither a multiplicity of sensory inputs nor sensory noise and failures are considered. The adopted architecture is simplistic where input data are only fused in one layer.In the field of wearable devices, BID10 utilize early fusion, which concatenates sensory inputs. With this simple fusion approach, classical supervised learning methods such as Bayesian classifiers, k-nearest-neighbors, support vector machines, and artificial neural networks are compared. BID3 use deep convolutional neural networks with IMU data. BID11 use a CNN with angle embedded gate dynamic images, which are pre-processed inputs for gait recognition. BID5 summarize three fusion methods, namely, data, feature, and decision level fusion. However, effective sensor fusion network architectures for coping with sensor failures are not deeply investigated.In terms of sensor fusion architectures, BID8 propose a so-called netgated architecture in which the information flow in a given convolutional neural network (CNN) is gated by fusion weights extracted from camera and LiDAR inputs. These fusion weights are used for computing a weighted sum of the sensory inputs. The weighted sum passes through fully connected layers to create a steering command. The gated networks (netgated) is shown to be robust to sensor failures, comparing to basic CNNs. However, a deep understanding of the relationships between sensory inputs, fusion weights, network architecture, and the resulting performances are not examined.The main objective of this paper is to propose optimized gated architectures that can address three limitations of the baseline netgated architecture of BID8 and investigate how different fusion architectures operate under clean sensory data and in the presence of snesor noise and failures. Our main contributions are:• Propose a new coarser-grained gated architecture which learns robustly a set of fusion weights at the (feature) group level;• Further propose a two-stage gating architecture which exploits both the feature-level and group-level fusion weights, leading to further performance improvements.• Analyze the characteristics of the proposed architectures and show how they may address the limitations of the negated architecture in terms of inconsistency of fusion weights, overfitting, and lack of diverse fusion mechanisms.By utilizing driving mode prediction and human activity recognition datasets, we demonstrate the significant performance improvements brought by the proposed architectures over the conventional CNN and netgated architectures under various settings including cases where random sensor noise and failures are presented. Empirical evidence is also analyzed to help shed light on the underlying causes that may be responsible for the observed improvements of the two proposed architectures.
Optimized gated deep learning architectures for sensor fusion is proposed. The authors improve upon several limitations of the baseline negated architecture by proposing a coarser-grained gated fusion architecture and a two-stage gated fusion architecture Proposes two gated deep learning architectures for sensor fusion and by having the grouped features, demonstrates improved performance, especially in the presence of random sensor noise and failures.