---
abstract: 'The ability to simultaneously leverage multiple modes of sensor information is critical for perception of an automated vehicle’s physical surroundings. Spatio-temporal alignment of registration of the incoming information is often a prerequisite to analyzing the fused data. The persistence and reliability of multi-modal registration is therefore the key to the stability of decision support systems ingesting the fused information. LiDAR-video systems like on those many driverless cars are a common example of where keeping the LiDAR and video channels registered to common physical features is important. We develop a deep learning method that takes multiple channels of heterogeneous data, to detect the misalignment of the LiDAR-video inputs. A number of variations were tested on the Ford LiDAR-video driving test data set and will be discussed. To the best of our knowledge the use of multi-modal deep convolutional neural networks for dynamic real-time LiDAR-video registration has not been presented.'
author:
- |
    Michael Giering, Vivek Venugopalan and Kishore Reddy\
    United Technologies Research Center\
    E. Hartford, CT 06018, USA\
    Email: {gierinmj, venugov, reddykk}@utrc.utc.com
bibliography:
- 'references.bib'
title: 'Multi-modal Sensor Registration for Vehicle Perception via Deep Neural Networks'
---

Motivation {#sec:motivation}
==========

Navigation and situational awareness of optionally manned vehicles requires the integration of multiple sensing modalities such as Light Detection and Ranging (LiDAR) and video, but could just as easily be extended to other modalities including Radio Detection And Ranging (RADAR), Short-Wavelength Infrared (SWIR) and Global Positioning System (GPS). Spatio-temporal registration of information from multi-modal sensors is technically challenging in its own right. For many tasks such as pedestrian and object detection tasks that make use of multiple sensors, decision support methods rest on the assumption of proper registration. Most approaches [@Bodensteiner2012Real-time-] in LiDAR-video for instance, build separate vision and LiDAR feature extraction methods and identify common anchor points in both. Alternatively, by generating a single feature set on LiDAR, Video and optical flow, it enables the system to to capture mutual information among modalities more efficiently. The ability to dynamically register information from the available data channels for perception related tasks can alleviate the need for anchor points *between* sensor modalities. We see auto-registration as a prerequisite need for operating on multi-modal information with confidence.

Deep neural networks (DNN) lend themselves in a seamless manner for data fusion on time series data. For some challenges in which the modalities share significant mutual information, the features generated on the fused information can provide insight that neither input alone can [@Ngiam2011Multimodal]. In effect the ML version of, “the whole is greater than the sum of it’s parts”.

Autonomous navigation places significant constraints on the speed of perception algorithms and their ability to drive decision making in real-time. Though computationally intensive to train, our implemented DCNN run easily within our real-time frame rates of 8 fps and could accommodate more standard rates of 30 fps. With most research in deep neural networks focused on algorithmic improvements and novel applications, a significant benefit to applied researchers is sometimes under appreciated. The automated feature generation of DNNs enables us to create mutli-modal systems with far less overhead. The need for domain experts and hand-crafted feature design are lessened, allowing more rapid prototyping and testing. The generalization of auto-registration across multiple assets is clearly a path to be explored.

In this paper, the main contributions are: (i) formulation of an image registration problem as a fusion of modalities from different sensors, namely LIDAR (L), video (Grayscale or R,G,B) and optical flow (U,V); (ii) performance evaluation of deep convolutional neural networks (DCNN) with various input parameters, such as kernel filter size and different combinations of input channels (R,G,B,Gr,L,U,V); (iii) fusion of patch-level and image-level predictions to generate alignment at the frame-level. The experiments were conducted using a publicly available dataset from FORD and the University of Michigan [@Pandey2011Ford-Campu]. The DCNN implementation was executed on an NVIDIA Tesla K40 GPU with 2880 cores and compute power of 5 TFLOPS (single precision). The paper is organized into the following sections: Section \[sec:motivation\] describes the introduction and motivation for this work; Section \[sec:previous\_work\] provides a survey of the related work; the problem formulation along with the dataset description and the preprocessing is explained in Section \[sec:problem\_statement\]; Section \[sec:model\_description\] gives the details of the DCNN setup for the different experiments; Section \[sec:experiments\] describes the experiments and the post-processing steps for visualizing the qualitative results; finally Section \[sec:conclusions\_and\_future\_work\] summarizes the paper and concludes with future research thrusts.

Previous Work {#sec:previous_work}
=============

A great amount has been published on various multi-modal fusion methods [@Ross2003Informatio], [@Gregor2011Learning-R], [@Wu2004Optimal-Mu], [@Snoek2006The-Challe]. The most common approaches taken generate features of interest in each modality separately and create a decision support mechanism that aggregates features across modalities. If spatial alignment is required across modalities, as it is for LiDAR-video such filter methods [@Thrun2011Googles-dr] are required to ensure proper inter-modal registration. These filter methods for leveraging 3D LiDAR and 2D images are often geometric in nature and make use of projections between the different data spaces.

Automatic registration of 2D video and 3D LiDAR has been a widely researched topic for over a decade [@Wang2009A-Robust-A],  [@Kim2014Automatic-],  [@Mastin2009Automatic-],  [@Bodensteiner2012Real-time-]. Its application in real-time autonomous navigation makes it a challenging problem. Majority of the 2D-3D registration algorithms are based on feature matching. Geometric features like corners and edges are extracted from detected vanishing points  [@Liu2007-Vanishing-points], [@Ding2008-Vanishing-point], line segments  [@Frueh2004-Linesegment],  [@Stamos2008-Linesegment], and shadows  [@Troccoli2004-ashadow]. Feature based approaches generally rely on dense 3D point cloud and additional knowledge of relative sun position and GPS/inertial navigation system (INS). Another approach used for video and LiDAR auto-registration is to reconstruct 3D point cloud from video sequences using structure from motion (SFM) and performing 3D-3D registration  [@Zhao2004-alignment-3Dcloud],  [@Liu2006-alignment-sfm]. 3D-3D registration is more difficult and computationally expensive compared to 2D-3D registration. The use of deep neural networks to analyze multi-modal sensor inputs has increased sharply in just the last few years, including audio-video [@Ngiam2011Multimodal], [@Kim2013Deep-Learn], image/text [@Srivastava2012Multimodal], image/depth [@Lenz2013Deep-Learn] and LiDAR-video To the best of our knowledge the use of multi-modal deep neural networks for dynamic LiDAR-video registration has not been presented.

A common challenge for data fusion methods is deciding at what level features from the differing sensor streams should be brought together. The deep neural network (DNN) approach most similar to the more traditional data fusion methods is to train DNNs independently on sensor modalities and then use the high-level outputs of those networks as inputs to a subsequent aggregator, which could also be a DNN. This is analogous to the earlier example of learning 3D/2D features and the process of identifying common geometric features.

It is possible however to apply DNNs with a more agnostic view enabling a unified set of features to be learned across multi-modal data. In these cases the input channels aren’t differentiated. Unsupervised methods including deep Boltzmann machines and deep auto-encoders for learning such joint representations have been successful.

Deep convolutional neural networks (DCNN) enable a similar agnostic approach to input channels. A significant difference is that target data is required to train them as classifiers. This is the approach chosen by us for automating the registration of LiDAR-video and optical-flow, in which we are combining 1D/3D/2D data representations respectively to learn a unified model across as many as 6D.

Problem Statement {#sec:problem_statement}
=================

Being able to detect and correct the misalignment (registration, calibration) among sensors of the same or different kinds, is critical for decision support systems operating on their fused information streams. For our work DCNNs were implemented for the detection of small spatial misalignments in LiDAR and Video frames. The methodology is directly applicable to temporal registration as well. LiDAR-video data collected from a driverless car was chosen for the multi-modal fusion test case. LiDAR-video is a common combination for providing perception capabilities to many types of ground and airborne platforms including driverless cars [@Thrun2011Googles-dr].

Ford LiDAR-video Dataset and Experimental Setup {#sub:ford_lidar_video_dataset_and_experimental_setup}
-----------------------------------------------

The FORD LiDAR-video dataset [@Pandey2011Ford-Campu] is collected by an autonomous Ford F-250 vehicle integrated with the following perception and navigation sensors as follows:

-   Velodyne HDL-64E LiDAR with two blocks of lasers spinning at 10 Hz and a maximum range of 120m.

-   Point Grey Ladybug3 omni-directional camera system with six 2-Mega-pixel cameras collecting video data at 8fps with $1600\times1600$ resolution.

-   Two Riegl LMS-Q120 LIDAR sensors installed in the front of the vehicle generating range and intensity data when the laser sweeps its $80\degree$ field of view (FOV).

-   Applanix POS-LV420 INS with Trimble GPS system providing the 6 degrees of freedom (DOF) estimates at 100 Hz.

-   Xsens MTi-G sensor consisting of accelerometer, gyroscope, magnetometer, integrated GPS receiver, static pressure sensor and temperature sensor. It measures the GPS co-ordinates of the vehicle and also provides the 3D velocity and 3D rate of turn.

![Training (A to B) and testing (C to D) tracks in the downtown Dearborn Michigan.[]{data-label="fig:ford_train_test_track"}](ford_train_test_track.jpg)

This dataset is generated by the vehicle while driving in and around the Ford research campus and downtown Michigan. The data includes feature rich downtown areas as well as featureless empty parking lots. As shown in Figure \[fig:ford\_train\_test\_track\], we divided the data set into training and testing sections A to B and C to D respectively. They were chosen in a manner that minimizes the likelihood of contamination between training and testing. Because of this, the direction of the light source is never the same in the testing and training sets.

Optical Flow {#sub:optical_flow}
------------

![Optical flow: Hue indicates orientation and saturation indicates magnitude[]{data-label="fig:Figures_OptFlow placeholder"}](OpticalFlow_example_final.png)

In the area of navigation of mobile robots, optical flow has been widely used to estimate egomotion [@Prazdny1980-egomotion-OF], depth maps [@Shahraray1988-depthestimation-OF], reconstruct dynamic 3D scene depth [@Yang2012-reconstruction-OF], and segment moving objects [@Shao2002-seg-OF]. Optical flow provides information of the scene dynamics and is expressed as an estimate of velocity at each pixel from two consecutive frames, denoted by $\vec{u}$ and $\vec{v}$. The motion field from these two frames is measured by the motion of the pixel brightness pattern, where the changes in image brightness is due to the camera or object motion. [@Liu2009Beyond-Pix] describes an algorithm for computing optical flow from images, which is used during the preprocessing step. Figure \[fig:Figures\_OptFlow placeholder\] shows an example of the optical flow computed using two consecutive frames from the Ford LiDAR-video dataset. By including optical flow as input channels, we imbue the DCNN with information on the dynamics observed across time steps.

Preprocessing {#sub:preprocessing}
-------------

At each video frame timestep, the inputs to our model consist of *C* channels of data with *C* ranging from 3-6 channels. Channels consist of grayscale *Gr* or *(R,G,B)* information from the video, horizontal and vertical components of optical flow *(U,V)* and depth information *L* from LiDAR The data from each modality is reshaped to a fixed size of $800\times256$ values, which are partitioned into $p\times p$ patches at a prescribed stride. Each patch $p\times p$ is stacked across *C* channels, effectively generating a vector of *C* dimensions. The different preprocessing parameters are denoted by patch size *p*, stride *s* and the number of input channels *C*.

Preprocessing is repeated *N* times, where *N* is the number of offset classes. For each offset class, the video (R,G,B) and optical flow (U,V) channels are kept static and the depth (L) channel from the LiDAR is moved by the offset simulating a misalignment between the video and the LiDAR sensors. In order to accurately detect the misalignment in the LiDAR and Video sensor data, a threshold is set to limit the information available in each channel. The LiDAR data has regions of sparsity and hence the LiDAR patches with a variance (${\sigma}^2 < 15\%$) are dropped from the final dataset. This leads to the elimination of the majority of foreground patches in the data set, reducing the size of the training and testing set by approximately $80\%$. Figure \[fig:Figures\_Ellipse\] shows a $N = 9$ class elliptically distributed set of offsets and Figure \[fig:ImageChStride\] shows a $p\times p$ patch stacked across all the different *C* channels.

Model Description {#sec:model_description}
=================

![image](lidar_dcnn_setup3.pdf)

Our models for auto-registration are DCNNs trained to classify the current misalignment of the LiDAR-video data streams into one of a predefined set of offsets. DCNNs are probably the most successful deep learning model to date on fielded applications. The fact that the algorithm shares weights in the training phase, results in fewer model parameters and more efficient training. DCNNs are particularly useful for problems in which local structure is important, such as object recognition in images and temporal information for voice recognition. The alternating steps of convolution and pooling generates features at multiple scales which in turn imbues DCNN’s with scale invariant characteristics.

The model shown in Figure \[fig:Figures\_lidar\_dcnn\_setup1\] consists of 3 pairs of convolution-pooling layers, that estimates the offset between the LiDAR-video inputs at each time step. For each patch within a timestep, there are $N$ variants with the LiDAR-video-optical flow inputs offset by the predetermined amounts. The CNN outputs to a softmax layer, thereby providing an offset classification value for each patch of the frame. As described in Section \[sub:preprocessing\], $32\times32$ patches were stacked across the different channels and provided as the input to the DCNN. All the $6$ channels *RGBLUV* were used for the majority of the experiments, whereas only $4$ channels were required for the *RGBL* and the *GrLUV* experiments. The first convolutional layer uses $32$ filters (or kernels) of size $5 \times 5 \times \mathit{C} $ with a stride of $1$ pixel and padding of $2$ pixels on the edges. The following pooling layer generates the input data (of size $16 \times 16 \times 32$) for the second convolutional layer. This layer uses $32$ filters of size $5 \times 5 \times 32$ with a stride of $1$ pixel and padding of $2$ pixels on the edges. A second pooling layer, similar to the first one is used to generate input with size $8 \times 8 \times 32$ for the third convolutional layer that uses $64$ filters of size $5 \times 5 \times 32$ with the stride and padding same as previous convolutional layer. The third pooling layer with similar configuration as the two previous pooling layers connects to an output softmax layer with labels corresponding to the $N=9$ classes. The DCNN described above was trained using stochastic gradient descent with a mini-batch size of $100$ epochs. The DCNN is configured with Rectified Linear Units (ReLUs), as they train several times faster than their equivalents with $\tanh$ connections [@Nair2010Rectified-]

The NVIDIA Kepler series K40 GPUs [@NVIDIA-Inc.2012NVIDIAs-Ne] are very FLOPS/Watt efficient and are being used to drive real-time image processing capabilities [@Venugopal2013Accelerati]. These GPUs consist of 2880 cores with 12 GB of on-board device memory (RAM). Deep Learning applications have been targeted on GPUs previously in [@Krizhevsky2012Imagenet-C] and these implementations are both compute and memory bound. Stacking of channels results in a vector of $32 \times 32 \times \mathit{C}$, which is suitable for the Single Instruction Multiple Datapath (SIMD) architecture of the GPUs. At the same time, the training batch size caches in the GPU memory, so the utilization of the K40 GPU’s memory is very high. This also results in our experiments to run successfully on a single GPU instead of partitioning the different layers over multiple GPUs.

Experiments {#sec:experiments}
===========

Dataset using elliptically distributed offsets {#sub:dataset_using_elliptically_distributed_offsets}
----------------------------------------------

![image](Voting_final.png)

In our experiments, elliptically distributed set of $N = 9$ offsets of the LiDAR-video data were considered. The LiDAR data is displaced along an ellipse with a major axis of $32$ pixels and a minor axis of $16$ pixels rotated clockwise from x-axis by $45\degree$ as shown in Figure \[fig:Figures\_Ellipse\]. Separate training and testing sets were generated from two different tracks as shown in Figure \[fig:ford\_train\_test\_track\] for all the $N = 9$ offsets of LiDAR data. Training and testing tracks have never seen regions and also have different lighting conditions. Our preprocessing step described in Section \[sub:preprocessing\] results in $223,371$ and $126,513$ patches for testing and training extracted from $469$ and $224$ images respectively.

In the testing phase, for each frame a simple voting scheme is used to aggregate the patch level offset predictions to a single frame level prediction. A sample histogram of the patch level predictions is show in Figure \[fig:Figures\_Voting\]. We color each patch of the frame with a color corresponding to the predicted class as shown in Figure \[fig:Figures\_Voting\].

Experimental results {#sub:experimental_results}
--------------------

Table \[table:cnn\_param\] lists the inputs and CNN parameters explored ranked in the order of increasing accuracy. We averaged the values across the diagonal of the confusion matrix to determine the image level and patch level accuracy. Patch level accuracy is the individual performance of all the $32\times32$ patches from the testing images. Classification of patches belonging to a single time-step are voted to predict the shift for image level accuracy. In Table \[table:cnn\_param\], the first 3 columns show the results for different number of filter combinations in the convolutional layers with fixed number of filters and input channels *RGBLUV*. We observed that the image and patch level accuracy decreased with the increase in the number of filters. For experiments shown in columns 4 and 5, the filter size was increased, with the number of filters constant at $(32,32,64)$. We observed that for the 6 channels *RGBLUV*, filter size of 9 gave the best image level accuracy of $63.03\%$. Column 6 shows the results of our experiment after dropping the optical flow *UV* channels. The image and patch level accuracy decreased for this case, indicating that optical flow contributed significantly towards image registration. The remaining experiments utilized the Grayscale information *Gr* instead of *RGB* and produced the best results with $76.69\%$ and $41.05\%$ image and patch level accuracy respectively. Table \[table:temporal\_performance\] shows that by using information from consecutive frames the performance increases significantly.

  **Channels**                                   RGBL                                                  
  ----------------------------- ------------ ------------ ------------ ------- ------- ------- ------- -----------
  **Filter size**                                 7            9                  9                    
  **\# of filters**              (32,32,32)   (32,32,64)   (64,64,64)                                  
  **Image Level accuracy(%)**      61.75        61.06        60.09      61.79   63.03   60.66   68.03   **76.69**
  **Patch Level accuracy(%)**      38.74        38.57        38.49      38.03   39.00   39.28   40.96   **41.05**

\[table:cnn\_param\]

  **Number of consecutive time-steps used**   **1**   **2**   **3**   **4**   **5**   **6**   **7**   **8**
  ------------------------------------------- ------- ------- ------- ------- ------- ------- ------- -------
  **Accuracy(%)**                             76.33   85.42   88.88   90.30   92.52   93.85   94.29   95.12

\[table:temporal\_performance\]

Conclusions and Future Work {#sec:conclusions_and_future_work}
===========================

In this paper, we proposed a deep learning method to do LiDAR-Video registration. We demonstrated the effect of filter size, number of filters and different channels. We also showed the advantage of using temporal information, optical flow and grayscale. The next step in taking this work forward is to complete our development of a deep auto-registration method for ground and aerial platforms requiring no a priori calibration ground truth. Our aerospace applications in particular present noisier data with an increased number of degrees of freedom. The extension of these methods to simultaneously register information across multiple platforms and larger numbers of modalities will provide interesting challenges that we look forward to working on.
