Deep neural networks (DNN) have shown promising performance in computer vision. In medical imaging, encouraging results have been achieved with deep learning for applications such as segmentation, lesion detection and classification. Nearly all of the deep learning based image analysis methods work on reconstructed images, which are obtained from original acquisitions via solving inverse problems (reconstruction). The reconstruction algorithms are designed for human observers, but not necessarily optimized for DNNs which can often observe features that are incomprehensible for human eyes. Hence, it is desirable to train the DNNs directly from the original data which lie in a different domain with the images. In this paper, we proposed an end-to-end DNN for abnormality detection in medical imaging. To align the acquisition with the annotations made by radiologists in the image domain, a DNN was built as the unrolled version of iterative reconstruction algorithms to map the acquisitions to images, and followed by a 3D convolutional neural network (CNN) to detect the abnormality in the reconstructed images. The two networks were trained jointly in order to optimize the entire DNN for the detection task from the original acquisitions. The DNN was implemented for lung nodule detection in low-dose chest computed tomography (CT), where a numerical simulation was done to generate acquisitions from 1,018 chest CT images with radiologists' annotations. The proposed end-to-end DNN demonstrated better sensitivity and accuracy for the task compared to a two-step approach, in which the reconstruction and detection DNNs were trained separately. A significant reduction of false positive rate on suspicious lesions were observed, which is crucial for the known over-diagnosis in low-dose lung CT imaging. The images reconstructed by the proposed end-to-end network also presented enhanced details in the region of interest. Deep neural networks (DNN) have shown promising performance in computer vision for various applications such as segmentation, detection and recognition. In medical imaging, DNNbased computer vision is also desirable because that radiologists' routine work requires handling of large amount of data, and the possibility exists that the intensive labor may lead to misdiagnosis BID14 BID38 . Furthermore, in some radiation related applications such as computed tomography (CT), low-dose scans are always preferred to decrease the potential harm that ionized radiation may do to human body. The increased noise level in lowdose data made it even more challenging for the radiologists to make correct decisions BID19 .Almost all the DNNs in medical image analysis are constructed in the image domain, which is the same domain where radiologists do the observations. However , in most medical imaging modalities, the acquired data are in a different domain from the images, and inverse problems have to be solved to reconstruct the images. For example , magnetic resonance imaging (MRI) acquires data in the Fourier domain, and CT acquires data in the Radon transform domain BID10 . During the reconstruction, there is great possibility of information lost due to the presence of noise in the measurements, especially for low-dose scans BID27 . To compensate for the noise, iterative methods that exploit prior knowledge about human body have been proposed BID11 BID30 . To achieve better representation of the med-ical images, DNN based reconstruction methods were also proposed recently BID36 b; BID12 . However, there is still gap between the objective image quality improvement and its utility in diagnosis, which means that both radiologists and computer aided diagnosis (CAD) systems are working with sub-optimal images.There is an emerging trend on task-based (end-to-end) signal processing with DNNs in recent years, where the decisions were directly made by the DNNs without explicit intermediate representations. BID13 used DNN for speech recognition directly from audio data without and intermediate phonetic representations. BID4 trained a DNN for self-driving cars which learned commands directly from images without recognition of land markers. BID18 used a classification criteria for the colorization of grey-scale images. BID35 detected words directly from scenes without doing a two-step text detection and optical character recognition (OCR). It was demonstrated that end-to-end DNNs had improved performance compared to multiple-step learning in these applications.In this paper, we proposed an end-to-end DNN which predicts the location of abnormalities in the images from the acquisitions. A reconstruction DNN was built first to map the acquired data to the images in order to align the data with annotations made by the radiologists. The DNN approximated a 10-iteration unrolled sequential quadratic surrogates (SQS) algorithm BID11 . A 3D convolutional neural network ( CNN) was used to detect abnormalities from the reconstructed images. The entire DNN was optimized jointly with regard to the total detection cross entropy loss. The method was implemented on The Lung Image Database Consortium image collection (LIDC-IDRI) from The Cancer Image Archive (TCIA), where we simulated ultra low-dose CT scans from the original patients' data BID2 BID8 BID3 . The task of the DNN was lung nodule detection , which is essential for early stage cancer screening (Team et al., 2011) . The performance of the end-to-end method was evaluated with entropy loss and receiver operating characteristic (ROC), and compared to a two-step approach, where the reconstruction DNN was trained first and the detection DNN was trained on the reconstructed images. Furthermore, the intermediate reconstructed images and features from the end-to-end network were studied for more comprehensive understanding of the DNN. In this paper a novel end-to-end DNN was proposed for abnormality detection in medical imaging. A reconstruction network and detection network were trained jointly to maximize the abnormality detection accuracy. We implemented the method on simulated chest CT data and achieved higher non-small lung nodule detection accuracy compared to two-step training scheme. There was significant false positive rate reduction on suspicious lesions (annotated non-nodules), and fair improvement on the overall detection sensitivity and accuracy. The images reconstructed by the end-to-end method resembled ordinary CT images, with more details and increased noise level compared to that from a two-step approach.Among the 102 validation cases, the mean entropy loss of nodule detection of the end-to-end method was smaller or similar than the two-step method for most of the cases, which indicated a statistical improvement on nodule detection from the proposed method. However, there was one case where the end-to-end entropy loss was significantly higher than the two-step loss. We studied the case further and confirmed that it was due to a strong misclassification of the positive samples, which was shown in figure 9(d).Although there was no significant improvement on the total AUC as shown in table 2, the ROC study in FIG4 indicated a significantly improved true positive rate at small false positive rate. The U.S. carried a national lung cancer screening with low-dose CT, which was considered to cause overdiagnosis due to the high false positive rate (Team et al., 2011; BID26 . The sensitivity improvement on the low false positive rate end indicated that the end-to-end DNN had great potential value to cancer screening tasks.There was great difference in the appearance of the reconstructed images from the two methods. The two-step training gave images with smaller overall noise level, but some of the details in lung were smoothed out, which caused misclassification for the detection network, as shown in FIG5 (a) and (b). The end-to-end training revealed more details in lung with higher spatial resolution, and the images were more suitable for the automatic nodule detection task. Though there were some misclassification due to the increased noise level, the overall performance of the nodule detection was improved with the end-to-end training scheme.The analysis on the intermediate results from the reconstruction network further revealed the difference between the two approaches. Whereas both methods kept similar structural component, the end-to-end method had more focus on the edges and tissues inside lung compared to the two-step method. As observed in FIG0 , the structures of the lung tissue were much more clearer in the end-to-end networks. This observation indicated that sharper edge and structures were of more importance for the detection network than the noise level in the reconstructed images, which is in accordance with human perceptions when radiologists perform the same task.Selecting appropriate representations of the data for further tasks such as detection is crucial, and the philosophy in end-to-end training is to leave the representation selection problem to machine rather than hand-crafting it. In this work, we demonstrated the feasibility of the end-to-end DNN for abnormality detection in medical imaging for a specific lung nodule detection problem in chest CT, and concluded that better results can be achieved for CAD systems by doing so. Nowadays most CAD systems are trained on reconstructed images, which are designed and tuned for radiologists rather than machines. By integrating the reconstruction process into the detection pipeline, better detection accuracy could be achieved for the CAD systems, and will increase its value to radiologists.
Detection of lung nodule starting from projection data rather than images. DNNs are used for patch-based lung nodule detection in CT projection data. Jointly modeling computed tomography reconstruction and lesion detection in the lung by training the mapping from raw sinogram to detection outputs in an end-to-end manner Presents an end to end training of a CNN architecture that combines CT image signal processing and image analysis.