Neural network training relies on our ability to find ````````"good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood.

 In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple ``"filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers. Training neural networks requires minimizing a high-dimensional non-convex loss function -a task that is hard in theory, but sometimes easy in practice. Despite the NP-hardness of training general neural loss functions BID0 , simple gradient methods often find global minimizers (parameter configurations with zero or near-zero training loss), even when data and labels are randomized before training BID39 . However, this good behavior is not universal; the trainability of neural nets is highly dependent on network architecture design choices, the choice of optimizer, variable initialization, and a variety of other considerations. Unfortunately, the effect of each of these choices on the structure of the underlying loss surface is unclear. Because of the prohibitive cost of loss function evaluations (which requires looping over all the data points in the training set), studies in this field have remained predominantly theoretical.Our goal is to use high-resolution visualizations to provide an empirical characterization of neural loss functions, and to explore how different network architecture choices affect the loss landscape. Furthermore, we explore how the non-convex structure of neural loss functions relates to their trainability, and how the geometry of neural minimizers (i.e., their sharpness/flatness, and their surrounding landscape), affects their generalization properties.To do this in a meaningful way, we propose a simple "filter normalization" scheme that enables us to do side-by-side comparisons of different minima found by different methods. We then use visualizations to explore sharpness/flatness of minimizers found by different methods, as well as the effect of network architecture choices (use of skip connections, number of filters, network depth) on the loss landscape. Out goal is to understand how differences in loss function geometry effect the generalization of neural nets. In this paper, we presented a new, more accurate visualization technique that provided insights into the consequences of a variety of choices facing the neural network practitioner, including network architecture, optimizer selection, and batch size.Neural networks have advanced dramatically in recent years, largely on the back of anecdotal knowledge and theoretical results with complex assumptions. For progress to continue to be made, a more general understanding of the structure of neural networks is needed. Our hope is that effective visualization, when coupled with continued advances in theory, can result in faster training, simpler models, and better generalization. Figure 4 , the first row uses zero weight decay and the second row uses 5e-4 weight decay.Generalization error for each plot is shown in TAB2 10 TEST AND TRAINING DATA FOR VARIOUS NETWORKS FIG0 , the first row uses zero weight decay and the second row sets weight decay to 5e-4.
We explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. This paper proposes a method to visualize the loss function of a NN and provides insights on the trainability and generalization of NNs. Investigates the non-convexity of the loss surface and optimization paths.