---
author:
- 'Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu'
title: Index Networks
---

[Lu : Index Networks]{}

is an essential stage for dense prediction tasks using deep convolutional neural networks (CNNs). The frequently used upsampling operators include transposed convolution [@zeiler2014visualizing; @long2015fully], unpooling [@badrinarayanan2017segnet], *periodic shuffling* [@shi2016real] (a.k.a.  depth-to-space), and naive interpolation [@lin2017refine; @chen18v3] followed by convolution. These operators, however, are not general-purpose designs and often exhibit different behaviors in different tasks.

The widely-adopted upsampling operator in semantic segmentation and depth estimation is bilinear interpolation, while unpooling is less popular. A reason might be that the feature map generated by max unpooling is sparse, while the bilinearly interpolated feature map has dense and consistent representations for local regions (compared to the feature map before interpolation). This is particularly true for semantic segmentation and depth estimation where pixels in a region often share the same class label or have similar depth. However, we observe that bilinear interpolation can perform significantly worse than unpooling in boundary-sensitive tasks such as image matting. A fact is that the leading deep image matting model [@xu2017deep] largely borrows the design from the SegNet method [@badrinarayanan2017segnet], where unpooling was first introduced. When adapting other state-of-the-art segmentation models, such as DeepLabv3+ [@chen18v3] and RefineNet [@lin2017refine], to this task, we observe that they tend to fail to recover boundary details (Fig. \[fig:alpha\_mattes\]). A plausible explanation is that, compared to the bilinearly upsampled feature map, unpooling uses max-pooling indices to guide upsampling. Since boundaries in the shallow layers usually have the maximum responses, indices extracted from these responses record the boundary locations. The feature map projected by the indices thus shows improved boundary delineation.

![Alpha mattes of different models for the task of image matting. From left to right, Deeplabv3+ [@chen18v3], RefineNet [@lin2017refine], Deep Matting [@xu2017deep] and IndexNet (Ours). Bilinear upsampling tends to fail to recover subtle details, while unpooling and our learned upsampling operator can produce much clear mattes with good local contrast. []{data-label="fig:alpha_mattes"}](mattes_intro.pdf){width="\linewidth"}

We thus believe that different upsampling operators may exhibit different characteristics, and we expect a specific behavior of the upsampling operator when dealing with specific image content for a particular vision task. A question of interest is: *Can we design a generic operator to upsample feature maps that better predict boundaries and regions simultaneously?* A key observation of this work is that unpooling, bilinear interpolation or other *upsampling operators are some forms of index functions*. For example, the nearest neighbor interpolation of a point is equivalent to allocating indices of one to its neighbor and then map the value of the point. In this sense, indices are models [@kraska2018case], therefore indices can be modeled and learned.

In this work, *we model indices as a function of the local feature map and learn index functions to implement upsampling within deep CNNs.* In particular, we present a novel index-guided encoder-decoder framework, which naturally generalizes models like SegNet. Instead of using max-pooling and unpooling, we introduce indexed pooling and indexed upsampling where downsampling and upsampling are guided by learned indices. The indices are generated *dynamically* conditioned on the feature map and are learned using a fully convolutional network, termed IndexNet, without extra supervision needed. IndexNet is a highly flexible module. It can be applied to almost all convolutional networks that have coupled downsampling and upsampling stages. Compared to the fixed $\max$ function or bilinear interpolation, learned index functions show potentials for simultaneous boundary and region delineation.

IndexNet is a high-level concept and represents a broad family of networks modeling the so-called index function. In this work, we instantiate and investigate five families of IndexNet. Different designs correspond to different assumptions. We compare the behavior of IndexNet with existing upsampling operators and demonstrate its superiority in delivering spatial information. We show that IndexNet can be incorporated into many CNNs to benefit a number of visual tasks, for instance: i) *image matting*: our MobileNetv2-based [@sandler2018mobilenetv2] model with IndexNet exhibits at least $16.1\%$ improvement against the VGG-16-based DeepMatting baseline [@xu2017deep] on the Composition-1k matting dataset; by visualizing learned indices, the indices automatically learn to capture the boundaries and textural patterns; ii) *image denoising*: a modified DnCNN model with IndexNet can achieve performance comparable to the baseline DnCNN [@zhang2017beyond] that has no downsampling stage on the BSD68 and Set12 datasets [@roth2009fields], thus reducing the computational cost and memory consumption significantly; iii) *semantic segmentation*: consistently improved performance is observed when SegNet [@badrinarayanan2017segnet] is equipped with IndexNet on the SUN RGB-D dataset [@song2015sun]; and iv) *monocular depth estimation*: IndexNet also improves the performance of a recent light-weight FastDepth model on the NYUDv2 dataset [@silberman2012indoor], with negligible extra computation cost.

We make the following main contributions:

-   We present a unified perspective of existing upsampling operators with the notion of the index function;

-   We introduce Index Networks—a novel family of networks that can be included into standard CNNs to provide dynamic, adaptive downsampling and upsampling capabilities; to the best of our knowledge, IndexNet is one of the first attempts towards the design of generic upsampling operators;

-   We instantiate, and investigate five designs of IndexNet and demonstrate their effectiveness on four vision tasks.

A preliminary conference version of this work appeared in [@hao2019indexnet]. We extend [@hao2019indexnet] by i) further investigating two light-weight IndexNets; ii) comparing properties and computational complexity of IndexNets; iii) presenting a taxonomy of upsampling operators with the notion of guided upsampling and blind upsampling; and iv) designing image reconstruction experiments on synthetic data to compare two upsampling paradigms in recovering spatial information. Besides image matting in [@hao2019indexnet], we now v) apply IndexNets to a few more vision tasks including image denoising, semantic segmentation and monocular depth estimation and report extensive experimental results, not only between index networks but also across different vision tasks.

Literature Review
=================

We review upsampling operators and a closely-related group of networks: dynamic networks.

[**Upsampling in Deep Networks.**]{} Compared with other components in the design of deep networks, downsampling and upsampling of feature maps are relatively less studied. Since learning a CNN without sacrificing the spatial resolution is computationally expensive and memory intensive, and suffers from limited receptive fields, downsampling operators are common choices, such as strided convolution and max/average pooling. To recover the resolution, upsampling is thus an essential stage for almost all dense prediction tasks. This poses a fundamental question: *What is the principal approach to recover the resolution of a downsampled feature map (decoding)*. A few upsampling operators are proposed. The *deconvolution* operator, a.k.a. transposed convolution, was initially used in [@zeiler2014visualizing] to visualize convolutional activations and introduced to semantic segmentation [@long2015fully], but this operator sometimes can be harmful due to its behavior in producing checkerboard artifacts [@odena2016deconvolution]. To avoid this, a suggestion is the “resize+convolution” paradigm, which has currently become the standard configuration in state-of-the-art semantic segmentation models [@chen18v3; @lin2017refine]. Apart from these, *perforate* [@osendorfer2014image] and *unpooling* [@badrinarayanan2017segnet] generate sparse indices to guide upsampling. The indices are able to capture and keep boundary information, but one issue is that the two operators can induce much sparsity after upsampling. Convolutional layers with large filter sizes must follow for densification. In addition, *periodic shuffling* ($\mathcal{PS}$) was introduced in [@shi2016real] as a fast and memory-efficient upsampling operator for image super-resolution. $\mathcal{PS}$ recovers resolution by rearranging the feature map of size $H\times W\times Cr^2$ to $rH\times rW\times C$. It is also used in some segmentation models [@yang2019deeperlab].

Our work is primarily inspired by the unpooling operator [@badrinarayanan2017segnet]. We remark that, it is important to extract spatial information before its loss during downsampling, and more importantly, to use stored information during upsampling. Unpooling shows a simple and effective use case, while we believe that there is much room to improve. Here we show that unpooling is a special form of index function, and we can learn an index function beyond unpooling.

We notice that concurrent work of [@jiaqi2019carafe] also pursues the idea of data-dependent upsampling and proposes an universal upsampling operator termed CARAFE. Although the idea is similar, IndexNet is different from CARAFE in several aspects. First, CARAFE does not associate upsampling with the notion of the index function. Second, the kernels used in CARAFE are generated conditioned on decoder features, while IndexNet builds upon encoder features, so the generated indices can also be used to guide downsampling. Third, CARAFE can be viewed as one of our investigated index networks—holistic index networks, but with different upsampling kernels and normalization strategies. We further compare IndexNet with CARAFE in Section \[ssec:comparison\] and in experiments.

**Dynamic Networks.** If considering the dynamic property of IndexNet, [IndexNet]{} shares similarity with an interesting group of networks—dynamic networks. Dynamic networks are often implemented with adaptive modules to extend the modeling capabilities of CNNs. These networks share the following characteristics. The output is *dynamic*, conditioned on the input feature map. Since dynamic networks are learnable modules, they are *generic* in the sense that they can be used as building blocks in many network architectures. They are also *flexible* to allow modifications according to target tasks. *Spatial Transformer Networks (STNs)* [@jaderberg2015spatial]. STN allows explicit manipulation of spatial transformation within the network. It achieves this by regressing transformation parameters $\theta$ with a side-branch network. A spatially-transformed output is then produced by a sampler parameterized by $\theta$. This results in a holistic transformation of the feature map. The dynamic nature of STN is reflected by the fact that, given different inputs, the inferred $\theta$ is different, allowing to learn some forms of invariance to translation, scale, rotation, etc. *Dynamic Filter Networks (DFNs)* [@jia2016dynamic]. DFN implements a filter generating network to dynamically generate kernel filter parameters Compared to conventional filter parameters that stay fixed during inference, filter parameters in DFN are dynamic and sample-specific. *Deformable Convolutional Networks (DCNs)* [@dai2017deformable]. DCNs introduce deformable transformation into convolution. The key idea is to predict offsets for convolutional kernels. With offsets, convolution can be executed on irregular sampling grids, enabling adaptive manipulation of the receptive field.

*Attention Networks* [@mnih2014recurrent]. Attention networks are a broad family of networks that use attention mechanisms. The mechanisms introduce multiplicative interactions between the inferred attention map and the feature map. In computer vision, attention mechanisms are usually referred to spatial attention [@wang2017residual], channel attention [@hu2018squeeze] or both [@woo2018cbam]. These network modules are widely applied in CNNs to force the network focusing on specific regions and therefore to refine feature maps. Essentially, attention is about feature selection. No attentional module has been designed to deal with the downsampling/upsampling stage.

In contrast to above dynamic networks, IndexNet focuses on upsampling, rather than manipulating filters or refining features. Akin to dynamic networks above, the dynamics in IndexNet also has a physical definition—indices. Such a definition also closely relates to attention networks. Later we show that the downsampling and upsampling operators used with IndexNet can, to some extent, be viewed as attentional operators. Indeed, max-pooling indices are a form of hard attention. It is worth noting that, despite that IndexNet in its current implementation may closely relate to attention, it focuses on upsampling rather than refining feature maps. IndexNet also shares the other characteristics mentioned above. It is implemented in a convolutional side-branch network, is trained without extra supervision and is generic and flexible. We demonstrate its effectiveness on four dense prediction tasks and five variants of IndexNet.

![image](indexnet_framework.pdf){width="5in"}

An Indexing Perspective of Upsampling
=====================================

With the argument that upsampling operators are index functions, here we offer a unified indexing perspective of upsampling operators. The unpooling operator is straightforward. We can define its index function in a $k\times k$ local region as an indicator function $$I_{max}(x)=\mathbbm{1}(x=\max ({\mbox{\boldsymbol{$X$}}}))\,,x\in{\mbox{\boldsymbol{$X$}}} \,,$$ where ${\mbox{\boldsymbol{$X$}}}\in\mathbb{R}^{k\times k}$. $ \mathbbm{1}( \cdot ) $ is the indicator function with output being a binary matrix. Similarly, if one extracts indices from average pooling, its index function takes the form $$I_{avg}(x)=\mathbbm{1}(x\in{\mbox{\boldsymbol{$X$}}})\,.$$ If further using $I_{avg}(x)$ during upsampling, it is equivalent to the nearest neighbor interpolation. As for the bilinear interpolation and deconvolution operators, their index functions have an identical form $$\label{eq:dconv}
I_{bilinear/dconv}(x)={\mbox{\boldsymbol{$W$}}}\otimes\mathbbm{1}(x\in{\mbox{\boldsymbol{$X$}}})\,,$$ where ${\mbox{\boldsymbol{$W$}}}$ is the weight/filter of the same size as ${\mbox{\boldsymbol{$X$}}}$, and $\otimes$ denotes the element-wise multiplication. The difference is that, ${\mbox{\boldsymbol{$W$}}}$ is learned in deconvolution but predefined in bilinear interpolation. Indeed bilinear interpolation has been shown to be a special case of deconvolution [@long2015fully]. Note that, in this case, the index function generates soft indices. The sense of index for the $\mathcal{PS}$ operator [@shi2016real] is also clear, because the rearrangement of the feature map is an indexing process. Considering $\mathcal{PS}$ a tensor $\mathcal{Z}$ of size $1\times1\times r^2$ to a matrix ${\mbox{\boldsymbol{$Z$}}}$ of size $r\times r$, the index function can be expressed by the one-hot encoding $$I^l_{ps}(x)=\mathbbm{1}(x= \mathcal{Z}_l)\,,l=1,...,r^2\,,$$ such that ${\mbox{\boldsymbol{$Z$}}}_{m,n}=\mathcal{Z}[I^l_{ps}(x)]$, where $m=1,...,r$, $n=1,...,r$, and $l=(r-1) \cdot m+n$. $\mathcal{Z}_l$ denotes the $l$-th element of $\mathcal{Z}$. Similar notation applies to ${\mbox{\boldsymbol{$Z$}}}_{m,n}$.

Since upsampling operators can be unified by the notion of the index function, it is plausible to ask whether one can learn an index function to dynamically capture local spatial patterns.

Learning to Index, to Pool, and to Upsample
===========================================

Before introducing the designs of IndexNet, we first present the general idea about how learned indices may be used in downsampling and upsampling with a new index-guided encoder-decoder framework. Our framework is a generalization of SegNet, as illustrated in Fig. \[fig:framework\]. For ease of exposition, let us assume the downsampling and upsampling rates to be $2$, and the pooling operator to use a kernel size of $2\times2$. The IndexNet module dynamically generates indices given the feature map. The proposed indexed pooling and indexed upsampling operators further receive generated indices to guide downsampling and upsampling, respectively. In practice, multiple such modules can be combined and used analogous to the max pooling layers for every downsampling and upsampling stage. *IndexNet* models the index as a function of the feature map $\mathcal{X}\in\mathbb{R}^{H\times W\times C}$. Given $\mathcal{X}$, it generates two index maps for downsampling and upsampling, respectively. An important concept for the index is that an index can either be represented in a natural order, e.g., 1, 2, 3, ..., or be represented in a logical form, i.e., 0, 1, 0, ..., meaning that an index map can be used as a mask. This is exactly how we use the index map in downsampling/upsampling. The predicted index shares the same definition of the index in computer science, except that we generate *soft* indices for smooth optimization, i.e., for any index $i$, $i\in[0,1]$.

IndexNet consists of a predefined index block and two index normalization layers. An index block can simply be a heuristically defined function, e.g., a $\max$ function, or more generally, a parameterized function such as neural network. In this work, we use a fully convolutional network to be the index block. More details are presented in Sections \[subsec:holistic\_networks\] and \[subsec:depthwise\_networks\]. Note that the index maps sent to the encoder and decoder are normalized differently. The decoder index map only goes through a *sigmoid* function such that for any predicted index $i\in(0,1)$. As for the encoder index map, indices of each local region $L$ are further normalized by a *softmax* function such that $\sum_{i\in{L}}i=1$. The second normalization guarantees the magnitude consistency of the feature map after downsampling.

*Indexed Pooling* ($\mathcal{IP}$) performs downsampling using generated indices. Given a local region $E\in\mathbb{R}^{k\times k}$, $\mathcal{IP}$ calculates a weighted sum of activations and corresponding indices over $E$ as $\mathcal{IP}(E)=\sum_{x\in E}I(x)x$, where $I(x)$ is the index of $x$. It is easy to see that max pooling and average pooling are special cases of $\mathcal{IP}$. In practice, this operator can be easily implemented with an element-wise multiplication between the feature map and the index map, an average pooling layer, and a multiplication of a constant used to compensate the effect of averaging, as instantiated in Fig. \[fig:framework\]. The current implementation is equivalent to $2\times2$ stride-2 convolution with dynamic kernels, but is more efficient than explicit on-the-fly kernel generation.

*Indexed Upsampling* ($\mathcal{IU}$) is the inverse operator of $\mathcal{IP}$. $\mathcal{IU}$ upsamples $d\in\mathbb{R}^{1\times 1}$ that spatially corresponds to $E$ taking the same indices into account. Let $I\in\mathbb{R}^{k\times k}$ be the local index map formed by $I(x)$s, $\mathcal{IU}$ upsamples $d$ as $\mathcal{IU}(d)=I\otimes D$, where $\otimes$ denotes the element-wise multiplication, and $D$ is of the same size as $I$ and is upsampled from $d$ with the nearest neighbor interpolation. $\mathcal{IU}$ also relates to deconvolution, but an important difference between $\mathcal{IU}$ and deconvolution is that, deconvolution applies a fixed kernel to all local regions (even if the kernel is learned), while $\mathcal{IU}$ upsamples different regions with different kernels (indices).

![Conceptual differences between holistic and depthwise index.[]{data-label="fig:holistic_depthwise"}](holistic_depthwise.pdf){width="3in"}

Index Networks
--------------

![A taxonomy of proposed index networks.[]{data-label="fig:indexnet_taxonomy"}](indexnet_taxonomy.pdf){width="2.43in"}

Here we present a taxonomy of proposed index networks. According to the shape of the output index map, index networks can be first categorized into two branches: *holistic index networks* (HINs) and *depthwise (separable) index networks* (DINs). Their conceptual differences are shown in Fig. \[fig:holistic\_depthwise\]. HINs learn an index function $I(\mathcal{X}):\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W\times1}$. In this case, all channels of the feature map share a holistic index map. By contrast, DINs learn an index function $I(\mathcal{X}):\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W \times C}$, where the index map is of the same size as the feature map.

Since the index map generated by DINs can correspond to individual slices of the feature map, we can incorporate further assumptions into DINs to simplify the designs. If assuming that each slice of the index map only relates to its corresponding slice of the feature map, this is the One-to-One (O2O) assumption and *O2O DINs*. If each slice of the index map relates to all channels of the feature map, this leads to Many-to-One (M2O) assumption and *M2O DINs*. In O2O DINs, one can further consider sharing IndexNet. In the most simplified case, the same IndexNet can be applied to every slice of the feature map and can be shared across different downsampling/upsampling stages, like the $\max$ function. We name this IndexNet *Modelwise O2O DINs*. If IndexNet is stage-specific, i.e., only sharing indices in individual stages, we call this IndexNet *Shared Stagewise O2O DINs*. Finally, without sharing any parameter (each feature slice has its specific index function), we obtain the standard design, termed *Unshared Stagewise O2O DINs*. Fig. \[fig:indexnet\_taxonomy\] shows the tree diagram of these index networks. The difference between modelwise IndexNet and stagewise IndexNet is also shown in Fig. \[fig:modelwise\_stagewise\]. Notice that, HINs and M2O DINs are both stagewise.

With the taxonomy, we investigate five families of IndexNet. Each family can be designed to have either linear mappings or nonlinear mappings, as we discuss next.

![Modelwise IndexNet vs.  stagewise IndexNet.[]{data-label="fig:modelwise_stagewise"}](modelwise_stagewise.pdf){width="3.4in"}

Holistic Index Networks {#subsec:holistic_networks}
-----------------------

Recall that HINs learn an index function $I(\mathcal{X}):\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W\times1}$. A naive design choice is to assume a linear mapping between the feature map and the index map.

*Linear HINs.* An example is shown in Fig. \[fig:holistic\_networks\](a). The network is implemented in a fully convolutional network. It first applies stride-$2$ $2\times2$ convolution (assuming that the downsampling rate is $2$) to the feature map of size $H\times W\times C$, generating a concatenated index map of size $H/2\times W/2\times 4$. Each slice of the index map ($H/2\times W/2\times 1$) is designed to correspond to the indices of a certain position of all local regions, e.g., the top-left corner of all $2\times 2$ regions. The network finally applies a $\mathcal{PS}$-like shuffling operator to rearrange the index map to the size of $H\times W\times 1$.

In many situations, a linear relationship is not sufficient. For example, a linear function even cannot approximate the max function. Thus, the second design choice is to introduce nonlinearity into the network.

*Nonlinear HINs.* Fig. \[fig:holistic\_networks\](b) illustrates a nonlinear HIN where the feature map is first projected to a map of size $H/2\times W/2\times 2C$, followed by a batch normalization layer and a ReLU function for nonlinear mappings. We then use point-wise convolution to reduce the channel dimension to an indices-compatible size. The remaining transformations follow its linear counterpart.

![Holistic index networks. (a) a linear index network; (b) a nonlinear index network.[]{data-label="fig:holistic_networks"}](holistic_index_networks.pdf){width="2.4in"}

Depthwise Index Networks {#subsec:depthwise_networks}
------------------------

In DINs, we seek $I(\mathcal{X}):\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W \times C}$, i.e., each spatial index corresponds to each spatial activation. As aforementioned, this type of networks further has two different high-level design strategies that correspond to two different assumptions.

### One-to-One Depthwise Index Networks

O2O assumption assumes that each slice of the index map only relates to its corresponding slice of the feature map. It can be denoted by a local index function $l(\mathcal{X}): \mathbb{R}^{k\times k\times1}\rightarrow\mathbb{R}^{k\times k\times1}$, where $k$ denotes the size of the local region. Since the local index function operates on individual feature slices, we can design whether different feature slices share the same local index function. Such a weight sharing strategy can be applied at a modelwise level or at a stagewise level, which leads to the following designs of O2O DINs:

1.  *Modelwise O2O DINs*: the model only has a unique index function that is shared by all feature slices, even in different downsampling and upsampling stages. This is the most light-weight design;

2.  *Shared Stagewise O2O DINs*: the index function is also shared by feature slices, but every stage has stage-specific IndexNet. This design is also light-weight;

3.  *Unshared Stagewise O2O DINs*: even in the same stage, different feature slices have distinct index functions.

Similar to HINs, DINs can also be designed to have linear/nonlinear modeling ability. Fig. \[fig:depthwise\_networks\] shows an example when $k=2$. Note that, in contrast to HINs, DINs follow a multi-column architecture. Each column is responsible for predicting indices specific to a certain spatial location of all local regions. We implement DINs with group convolutions.

*Linear O2O DINs.* According to Fig. \[fig:depthwise\_networks\], the feature map first goes through four parallel convolutional layers with the same kernel size. Modelwise O2O DINs and Shared Stagewise O2O DINs only use a kernel size of $2\times2\times 1$, a stride of $2$, and $1$ group, while Unshared Stagewise O2O DINs has a kernel size of $2\times2\times C$, a stride of $2$, and $C$ groups. One can simply reshape the feature map, i.e., reshaping $H\times W \times C$ to be $C \times H\times W \times 1$, to enable a $2\times2\times 1$ kernel operating on each $H\times W \times 1$ feature slice, respectively. All O2O DINs lead to four downsampled feature maps of size $H/2\times W/2 \times C$. The final index map of size $H\times W \times C$ is composed from the four feature maps by shuffling and rearrangement. Note that the parameters of four columns are not shared.

*Nonlinear O2O DINs.* Nonlinear DINs can be easily modified from linear DINs by inserting four extra convolutional layers. Each of them is followed by a batch normalization (BN) layer and a ReLU unit, as shown in Fig. \[fig:depthwise\_networks\]. The rest remains the same as the linear DINs.

![Depthwise index networks. $M=1,N=1$ for Modelwise O2O DINs and Shared Stagewise O2O DINs; $M=C,N=C$ for Unshared Stagewise O2O DINs; and $M=C,N=1$ for the M2O DINs. The masked modules are invisible to linear networks.[]{data-label="fig:depthwise_networks"}](depthwise_index_networks.pdf){width="3.2in"}

### Many-to-One Depthwise Index Networks

M2O assumption assumes that all feature slices have contributions to each index slice. The local index function is defined by $l(\mathcal{X}): \mathbb{R}^{k\times k\times C}\rightarrow\mathbb{R}^{k\times k\times1}$. Compared to O2O DINs, the only difference in implementation is the use of standard convolution instead of group convolution, i.e., $M=C,N=1$ in Fig. \[fig:depthwise\_networks\].

Property and Model Complexity
-----------------------------

Both HINs and DINs have merits and drawbacks. Here we discuss some important properties of IndexNet. We also present an analysis of computational complexity.

Index maps generated by HINs and used by the $\mathcal{IP}$ and $\mathcal{IU}$ operators are related to spatial attention.

The holistic index map is shared by all feature slices, which means that the index map is required to be expanded to the size of $H\times W\times C$ when feeding into $\mathcal{IP}$ and $\mathcal{IU}$. This index map can be thought as a collection of local attention maps [@mnih2014recurrent] applied to individual local spatial regions. In this case, the $\mathcal{IP}$ and $\mathcal{IU}$ operators can also be referred to as “attentional pooling” and “attentional upsampling”. However, it should be noted that spatial attention has no pooling or upsampling operators like $\mathcal{IP}$ and $\mathcal{IU}$.

HINs are more flexible than DINs and more friendly for decoder design.

Since the holistic index map is expandable, the decoder feature map does not need to forcibly increase/reduce its dimensionality to fit the shape of the index map during upsampling. This gives much flexibility for decoder design, while it is not the case for DINs.

The number of parameters in Modelwise O2O DINs and Shared Stagewise O2O DINs is independent of the dimensionality of feature maps.

No matter how large the model capacity is or how wide the feature channels are, the number of parameters in Modelwise O2O DINs remains at a constant level, and that in Shared Stagewise O2O DINs is only proportional to the number of downsampling/upsampling stages. This is desirable as the number of parameters introduced by IndexNet is not significant. However, these two types of IndexNet may be limited to capture sophisticated local patterns.

M2O DINs have the most powerful modeling capability among IndexNet variants, but also introduce many extra parameters.

M2O DINs have higher capacity than HINs and O2O DINs due to the use of standard convolution. Another desirable property of IndexNet is that they may be able to predict the indices from a large local feature map, e.g., $l(\mathcal{X}): \mathbb{R}^{2k\times 2k\times C}\rightarrow\mathbb{R}^{k\times k\times1}$. An intuition behind this idea is that, if one identifies a local maximum point from a $k\times k$ region, its surrounding $2k\times2k$ region can further support whether this point is a part of a boundary or only an isolated noise point. This idea can be easily implemented by enlarging the convolutional kernel size and with appropriate padding.

In Table \[tab:complexity\_comparison\], we summarize the model complexity of different index networks used at a single downsampling and upsampling stage. We assume the convolution kernel has a size of $K\times K$ applied on a $C$-channel feature map. The number of parameters in BN layers is excluded. When considering weak context, we assume the kernel size is $2K\times 2K$. Since $C\gg K$, generally we have the model complexity *M2O DINs*$>$*HINs*$>$*Unshared Stagewise O2O DINs*$>$*Shared Stagewise O2O DINs*$>$*Modelwise O2O DINs*.

  IndexNet    Type                       \# Param.
  ---------- ------ ----------------------------------------------------
               L                 $K\times K\times C\times4$
               NL          $K\times K\times C\times2C+2C\times4$
              NL+C        $2K\times 2K\times C\times2C+2C\times4$
               L                    $(K\times K)\times4$
               NL              $(K\times K\times2+2)\times4$
              NL+C            $(2K\times 2K\times2+2)\times4$
               L                    $(K\times K)\times4$
               NL              $(K\times K\times2+2)\times4$
              NL+C            $(2K\times 2K\times2+2)\times4$
               L                $(K\times K\times C)\times4$
               NL         $(K\times K\times 2C+2C\times C)\times4$
              NL+C       $(2K\times 2K\times 2C+2C\times C)\times4$
               L            $(K\times K\times C\times C)\times4$
               NL     $(K\times K\times C\times 2C+2C\times C)\times4$
              NL+C   $(2K\times 2K\times C\times 2C+2C\times C)\times4$

  : A Comparison of Model Complexity of Different Index Networks[]{data-label="tab:complexity_comparison"}

L: Linear; NL: Nonlinear; C: Context.

Guided Upsampling or Blind Upsampling: A Reconstruction-Based Justification
===========================================================================

Here we introduce the concept of *guided upsampling* and *blind upsampling* to summarize existing upsampling operators. Particularly, here we present a comparison between two data-dependent upsampling operators—IndexNet and CARAFE [@jiaqi2019carafe]. In addition, we show results of an image reconstruction task on synthetic data, highlighting the difference between guided upsampling and blind upsampling.

Guided Upsampling vs.  Blind Upsampling
---------------------------------------

By blind upsampling, we mean that an upsampling operator that is pre-defined with fixed parameters. Guided upsampling, instead, is guided with the involvement of auxiliary information. Thus, most widely-used upsampling operators peform blind upsampling. These operators include nearest-neighbor (NN) interpolation, bilinear interpolation, space-to-depth and deconvolution. It is worth noting that the recent data-dependent upsampling operator CARAFE is also a blind upsampling operator. By contrast, guided upsampling operators are rare in literature. Max unpooling, albeit being simple, is a guided upsampling operator. The auxiliary information used in upsampling comes from the max-pooling indices. Thefore, our proposed IndexNet clearly implements guided upsampling, with inferred dynamic indices as the auxiliary information.

A taxonomy of commonly-used upsampling operators and their corresponding downsampling operators is summarized in Table \[tab:upsampling\]. An upsampling operator should generally have an corresponding downsampling operator, and vice versa.

            Upsampling          Downsampling
  -- ------------------------ -----------------
         NN Interpolation      Average Pooling
      Bilinear Interpolation     Convolution
          Deconvolution          Convolution
          Space-to-Depth       Depth-to-Space
              CARAFE             Convolution
          Max Unpooling          Max Pooling
        Indexed Upsampling     Indexed Pooling

  : Blind Upsampling and Guided Upsampling Operators[]{data-label="tab:upsampling"}

The main difference here is that guided upsampling is made possible to exploit extra information to better recover the spatial information during upsampling. Thus, it is important that the spatial information is properly encoded during downsampling and is transferred to unsampling.

IndexNet vs.  CARAFE {#ssec:comparison}
--------------------

Both IndexNet and CARAFE are one of the few attempts pursuing the idea of data-dependent upsampling. The similarities include:

1.  They both are related to dynamic networks.

2.  Both are parametric upsampling operators;

3.  CARAFE and HINs both perform holistic upsampling.

4.  CARAFE also learns an index function. The index function has an identical form to Eq. , but with dynamic and normalized ${\mbox{\boldsymbol{$W$}}}$. In this sense, CARAFE may be considered as a single-input version of $\mathcal{IU}$, where the index map is generated internally.

The differences are:

1.  CARAFE is a blind upsampling operator, while IndexNet implements guided upsampling;

2.  The reassembly kernels in CARAFE are generated conditioned on the low-resolution decoder feature map. The index maps predicted by IndexNet, however, build upon the high-resolution encoder feature map, before spatial information is lost;

3.  In IndexNet, each upsampled feature point only associates with a single point in the low-resolution feature map. From low resolution to high resolution, it is a one-to-many mapping. In CARAFE, each upsampled point is a weighted sum of a local region from the low-resolution feature map. This is a many-to-one mapping;

4.  Compared to CARAFE which is presented as a single upsampling operator, IndexNet is a more general framework.

In particular, the key difference lies in the intermediate path that allows spatial information to be visible to upsampling. To further demonstrate the benefit of this intermediate path, we present an image reconstruction experiment on synthetic data, namely, the Fashion-MNIST dataset [@xiao2017fashion].

Fashion-MNIST Image Reconstruction
----------------------------------

The idea is that, if an upsampling operator can recover spatial information well from downsampled feature maps, the reconstructed output should be visually closer to the input image. The quality of reconstruction results can be a good indicator how well spatial information is recovered by an upsampling operator.

*Network Architecture and Baselines.* We use a standard encoder-decoder architecture. Let **C**($k$) denote a 2D convolutional layer with $k$-channel $3\times3$ filters, followed by BN and ReLU. **$D$** represents a downsampling operator with a downsampling ratio of $2$, and **$U$** an upsampling operator with an upsampling ratio of $2$. The reconstruction network can therefore be defined by **C**($32$)-**$D$**-**C**($64$)-**$D$**-**C**($128$)-**$D$**-**C**($256$)-**C**($128$)-**$U$**-**C**($64$)-**$U$**-**C**($32$)-**$U$**-**C**($1$). Note that BN and ReLU are not included in the last **C**($1$). We build the following baselines:

1.  Average Pooling–NN interpolation (AvgPool–NN);

2.  stride-2 Convolution–Bilinear interpolation (Conv~/2~–Bilinear);

3.  Space-to-Depth–Depth-to-Space (S2D–D2S);

4.  stride-2 Convolution–2-stride Deconvolution (Conv~/2~–Deconv~/2~);

5.  stride-2 Convolution–CARAFE (Conv~/2~–CARAFE);

6.  Max Pooling–Max Unpooling (MaxPool–MaxUnpool);

7.  Indexed Pooling–Indexed Upsampling ($\mathcal{IP}$–$\mathcal{IU}$).

                                                 PSNR         SSIM         MAE          MSE
  ------------------------------------------- ----------- ------------ ------------ ------------
  AvgPool–NN                                     25.88       0.9811       0.0259       0.0509
  Conv~/2~–Bilinear                              24.45       0.9726       0.0320       0.0600
  S2D–D2S                                        28.93       0.9901       0.0204       0.0358
  Conv~/2~–Deconv~/2~                            28.75       0.9903       0.0187       0.0366
  Conv~/2~–CARAFE                                25.55       0.9798       0.0277       0.0529
  MaxPool–MaxUnpool                              29.33       0.9920       0.0202       0.0342
  $\mathcal{IP}$–$\mathcal{IU}$^\*^              37.83       0.9989       0.0089       0.0128
  $\mathcal{IP}$–$\mathcal{IU}$^$\dagger$^       45.93       0.9998       0.0032       0.0051
  $\mathcal{IP}$–$\mathcal{IU}$^$\ddagger$^    **48.37**   **0.9999**   **0.0026**   **0.0038**

  : Performance of Image Reconstruction on the Fashion-MNIST Dataset[]{data-label="tab:reconstruction"}

$^*$ denotes Modelwise O2O DIN; $^\dagger$ indicates HIN; $^\ddagger$ refers to M2O DIN. All IndexNets are with nonlinearity and weak context. The best performance is boldfaced.

*Training Details.* The Fashion-MNIST dataset [@xiao2017fashion] includes $60,000$ training images and $10,000$ testing images. The input images are resized to $32\times32$. $\ell_1$ loss is used in training. The initial learning rate is set to $0.01$. We train the network for $100$ epochs with a batch size of $100$. The learning rate is decreased by $\times10$ at the $50$-th, $70$-th and $85$-th epoch, respectively. We report Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity index (SSIM), Mean Absolute Error (MAE) and root Mean Square Error (MSE).

*Discussions.* Quantitative and qualitative results are shown in Table \[tab:reconstruction\] and Fig. \[fig:recover\], respectively. We observe in Fig. \[fig:recover\] that, all baselines that exploit blind upsampling fail to reconstruct the input images. While in some cases they may produce reasonable reconstructions, images of trousers and high-heeled shoes for instance, in most cases these baselines generate blurred results when complex textural patterns appear, tops and wallets for example. By contrast, other baselines that leverage guided upsampling produce visually pleasing reconstructions, in all circumstances. In particular, by comparing MaxPool–MaxUnpool with $\mathcal{IP}$–$\mathcal{IU}$, the former tends to yield jittering artifacts. This suggests index maps extract and encode richer spatial information than max-pooling indices. In addition, by disabling the intermediate path, $\mathcal{IP}$–Bilinear leads to significantly poor reconstructions, which means that the intermediate path matters. The differences between two upsampling paradigms are also supported by the numerical results in Table \[tab:reconstruction\] where guided upsampling exhibits significantly better PSNR and lower errors than blind upsampling. Indeed, the intermediate path distinguishes guided upsampling from blind upsampling, and also IndexNet from CARAFE.

![image](Figures/recover.pdf){width=".6999\linewidth"}

Applications
============

In this section, we show several applications of IndexNet on the tasks of image matting, image denoising, semantic segmentation, and monocular depth estimation.

Image Matting
-------------

We first evaluate IndexNet on the task of image matting. Image matting is defined as a problem of estimating soft foreground from images. This problem is ill-posed due to the fact that solving a linear system of $7$ unknown variables with only $3$ known inputs: given the RGB color at pixel $i$, $I_i$, one needs to estimate the corresponding foreground color $F_i$, background color $B_i$, and matte $\alpha_i$, such that $I_i=\alpha_iF_i+(1-\alpha_i)B_i$, for any $\alpha_i\in[0,1]$. Previous methods have extensively studied this problem from a low-level view [@chen2013knn; @chuang2001bayesian; @he2011global; @levin2008closed]; and particularly, they have been designed to solve the above matting equation. Despite being theoretically elegant, these methods heavily rely on the color cues, rendering failures of matting in general natural scenes where colors are not reliable. With the tremendous success of deep CNNs in high-level vision tasks [@girshick2014rich; @krizhevsky2012imagenet; @long2015fully], deep matting methods are emerging. Recently deep image matting was proposed [@xu2017deep]. In [@xu2017deep] the authors presented the first deep image matting approach (DeepMatting) based on SegNet [@badrinarayanan2017segnet] and significantly outperformed other competitors. In this application, we use DeepMatting as our baseline.

![Customized MobileNetv2-based encoder-decoder network architecture. Our modifications are boldfaced.[]{data-label="fig:network_architecture"}](network_architecture.pdf){width="2.8in"}

Image matting is particularly suitable for evaluating the effectiveness of IndexNet, because the quality of learned indices can be visually observed from inferred alpha mattes. We conduct experiments on the Adobe Image Matting dataset [@xu2017deep]. This is so far the largest publicly available matting dataset. The training set has $431$ foreground objects and ground-truth alpha mattes. Each foreground is composited with $100$ background images randomly chosen from MS COCO [@lin2014microsoft]. The validation set termed Composition-1k includes $100$ unique objects. Each of them is composited with 10 background images chosen from Pascal VOC [@everingham2010pascal]. Overall, we have $43,100$ training images and $1,000$ testing images. We evaluate the results using widely-used Sum of Absolute Differences (SAD), root Mean Square Error (MSE), and perceptually-motivated Gradient (Grad) and Connectivity (Conn) errors [@rhemann2009perceptually]. The evaluation code implemented by [@xu2017deep] is used. In what follows, we first describe our modified MobileNetv2-based architecture and training details. We then perform extensive ablation studies to justify choices of model design, make comparisons of different index networks, and visualize learned indices.

### Network Architecture and Implementation Details

Here we describe the network architecture and training details.

  No.   Architecture                   Backbone     Fusion   Indices   Context   OS     SAD         MSE        Grad       Conn    
  ----- ---------------------------- ------------- -------- --------- --------- ---- ---------- ----------- ---------- ---------- --
  B1    DeepLabv3+ [@chen18v3]        MobileNetv2   Concat     No       ASPP     16     60.0       0.020       39.9       61.3    
  B2    RefineNet [@lin2017refine]    MobileNetv2    Skip      No        CRP     32     60.2       0.020       41.6       61.4    
  B3    SegNet [@xu2017deep]             VGG16        No       Yes       No      32   **54.6**   **0.017**     36.7       55.3    
  B4    SegNet                           VGG16        No       No        No      32    122.4       0.100      161.2      130.1    
  B5    SegNet                        MobileNetv2     No       Yes       No      32     60.7       0.021       40.0       61.9    
  B6    SegNet                        MobileNetv2     No       No        No      32     78.6       0.031      101.6       82.5    
  B7    SegNet                        MobileNetv2     No       Yes      ASPP     32     58.0       0.021       39.0       59.5    
  B8    SegNet                        MobileNetv2    Skip      Yes       No      32     57.1       0.019       36.7       57.0    
  B9    SegNet                        MobileNetv2    Skip      Yes      ASPP     32     56.0     **0.017**     38.9       55.9    
  B10   UNet                          MobileNetv2   Concat     Yes       No      32     54.7     **0.017**     34.3     **54.7**  
  B11   UNet                          MobileNetv2   Concat     Yes      ASPP     32     54.9     **0.017**   **33.8**     55.2    

Fusion: fuse encoder features; Indices: max-pooling indices (where Indices is ‘No’, bilinear interpolation is used for upsampling); CRP: chained residual pooling [@lin2017refine]; ASPP: atrous spatial pyramid pooling [@chen18v3]; OS: output stride. The lowest errors are boldfaced.

*Network Architecture*. We build our model based on MobileNetv2 [@sandler2018mobilenetv2] with only slight modifications to the backbone. We choose MobileNetv2 for its lightweight model and fast inference. The basic network configuration is shown in Fig. \[fig:network\_architecture\]. It also follows the encoder-decoder paradigm same as SegNet. We simply change all 2-stride convolution to be 1-stride and attach 2-stride $2\times2$ max pooling after each encoding stage for downsampling, which allows us to extract indices. If applying the IndexNet idea, max pooling and unpooling layers can be replaced with $\mathcal{IP}$ and $\mathcal{IU}$, respectively. We also investigate alternative ways for low-level feature fusion and whether encoding context (Section \[subsubsec:ablation\_study\]). Note that, the matting refinement stage [@xu2017deep] is not applied here.

*Training Details*. To enable a direct comparison with deep matting [@xu2017deep], we follow the same training configurations used in [@xu2017deep]. The 4-channel input concatenates the RGB image and its trimap. We follow exactly the same data augmentation strategies, including $320\times320$ random cropping, random flipping, random scaling, and random trimap dilation. We use a combination of the alpha prediction loss and the composition loss during training as in [@xu2017deep]. Only losses from the unknown region of the trimap are calculated. Encoder parameters are pretrained on ImageNet [@deng2009imagenet]. The parameters of the $4$-th input channel are initialized with zeros. The Adam optimizer [@kingma2015adam] is used. We update parameters with $30$ epochs (around $90,000$ iterations). The learning rate is initially set to $0.01$ and reduced by $10\times$ at the $20$-th and $26$-th epoch respectively. We use a batch size of $16$ and fix the BN layers of the backbone.

[&gt;p[.5cm]{}|&gt;p[.5cm]{}|c|c|c|c|c|c]{} & \#Param. & GFLOPs & SAD & MSE & Grad & Conn\
& 130.55M & 32.34 & 54.6 & 0.017 & 36.7 & 55.3\
& 3.75M & 4.08 & 54.9 & 0.017 & 33.8 & 55.2\
& 8.86M & 7.61 & 55.6 & 0.016 & 36.4 & 55.7\
& 4.06M & 5.01 & 50.2 & 0.015 & 27.9 & 50.0\
& 3.75M & 4.08 & 56.5 & 0.021 & 33.0 & 56.4\
NL & C & $\Delta$\
& &\
& & +4.99K & 4.09 & 55.1 & 0.018 & 32.1 & 55.2\
& & +0.26M & 4.22 & 50.6 & 0.015 & 27.9 & 49.4\
& & +1.04M & 4.61 & 49.5 & 0.015 & **25.6** & 49.2\
& &\
& & +16 & 4.08 & 57.3 & 0.017 & 37.3 & 57.4\
& & +56 & 4.08 & 52.4 & 0.016 & 30.1 & 52.2\
& & +152 & 4.08 & 59.1 & 0.018 & 39.0 & 59.7\
& &\
& & +80 & 4.08 & 48.9 & 0.014 & 26.2 & 48.0\
& & +280 & 4.08 & 51.1 & 0.016 & 30.2 & 50.7\
& & +760 & 4.08 & 56.0 & 0.016 & 37.5 & 55.9\
& &\
& & +4.99K & 4.09 & 50.3 & 0.015 & 33.7 & 50.0\
& & +17.47K & 4.10 & 50.6 & 0.016 & 26.5 & 50.3\
& & +47.42K & 4.15 & 50.2 & 0.016 & 26.8 & 49.3\
& &\
& & +0.52M & 4.34 & 51.0 & 0.015 & 33.7 & 50.5\
& & +1.30M & 4.73 & 48.9 & 0.015 & 32.1 & 47.9\
& & +4.40M & 6.30 & **45.8** & **0.013** & 25.9 & **43.7**\
& 50.4 & 0.014 & 31.0 & 50.8\

NL: Non-Linearity; C: Context. $\Delta$ indicates increased parameters compared to B11. GFLOPs are measured on a $224\times224\times4$ input. The lowest errors are boldfaced.

![image](composed_1k.pdf){width="\linewidth"}

### Results on the Adobe Image Matting Dataset {#subsubsec:ablation_study}

*Ablation Study on Model Design*. To establish a better baseline comparable to DeepMatting, here we first investigate strategies for fusing low-level features (no fusion, skip fusion as in ResNet [@he2016deep] or concatenation as in UNet [@ronneberger2015u]) and whether encoding context for image matting. $11$ baselines are consequently built to justify model design. Results on the Composition-1k testing set are reported in Table \[tab:architecture\]. B3 is cited from [@xu2017deep]. We can make the following observations:

1.  Indices are of great importance. Matting can significantly benefit from only indices (B3 vs. B4, B5 vs. B6);

2.  State-of-the-art semantic segmentation models cannot be directly applied to image matting (B1/B2 vs. B3);

3.  Fusing low-level features help, and concatenation is better than skip connection but at a cost of increased computation (B6 vs. B8 vs. B10 or B7 vs. B9 vs. B11);

4.  Modules such as ASPP may improve the results (e.g., B6 vs. B7 or B8).

5.  A MobileNetv2-based matting model can work as well as a VGG-16-based one (B3 vs. B11).

For the following experiments, we now mainly use B11.

*Ablation Study on Index Networks*. Here we compare different index networks and justify their effectiveness. The configurations of index networks used in the experiments follow Figs. \[fig:holistic\_networks\] and \[fig:depthwise\_networks\]. We primarily investigate the $2\times2$ kernel with a stride of $2$. Whenever the weak context is considered, we use a $4\times4$ kernel in the first convolutional layer of index networks. To highlight the effectiveness of HINs, we further build a baseline called *holistic max index* (HMI) where max-pooling indices are extracted from a squeezed feature map $\mathcal{X}'\in\mathbb{R}^{H\times W\times1}$. $\mathcal{X}'$ is generated by applying the max function along the channel dimension of $\mathcal{X}\in\mathbb{R}^{H\times W\times C}$. Furthermore, since IndexNet increases extra parameters, we introduce another baseline *B11-1.4* where the width multiplier of MobilieNetV2 is adjusted to be $1.4$ to increase the model capacity. In addition, to compare IndexNet against CARAFE in this task, we build an additional baseline *B11-carafe* where the unpooling operator in B11 is replaced with CARAFE. Results on the testing dataset are listed in Table \[tab:index\_function\]. We observe that, most index networks reduce the errors notably, except for some low-capacity IndexNet modules (due to limited modeling capabilities). In particular, nonlinearity and the context generally have a positive effect on deep image matting, but they do not work effectively in O2O DINs. A possible reason may be that the limited dimensionality of the intermediate feature map is not sufficient to model complex patterns in matting. Compared to holistic max index, the direct baseline of HINs, the best HIN (“Nonlinearity+Context”) has at least $12.3\%$ relative improvement. Compared to B11, the baseline of DINs, M2O DIN with “Nonlinearity+Context” exhibits at least $16.5\%$ relative improvement. Notice that, our best model outperforms the DeepMatting approach [@xu2017deep] that even has the refinement stage. In addition, according to the results of B11-1.4, the performance improvement does not come from increased parameters. Moreover, CARAFE also enhances matting performance, but it falls behind M2O DIN. Some qualitative results are shown in Fig. \[fig:composition-1k-results\]. Our predicted mattes show improved delineation for edges and textures like hair and water drops.

       Encoder       Decoder     SAD         MSE        Grad       Conn
  ----------------- --------- ---------- ----------- ---------- ----------
       sigmoid       sigmoid     52.7       0.016       29.3       52.4
       softmax       softmax     51.6       0.015       29.2       51.6
   softmax+sigmoid   softmax     57.3       0.016       43.5       57.3
   sigmoid+softmax   sigmoid   **45.8**   **0.013**   **25.9**   **43.7**

  : Ablation Study of Different Normalization Choices on Index Maps[]{data-label="tab:normalize"}

The lowest errors are boldfaced.

*Ablation Study on Index Normalization*. Index normalization is important for the final performance. Here we justify this by evaluating different normalization choices. Apart from the sigmoid function used for the decoder and the sigmoid+softmax function for the encoder, we compare other three different combinations of normalization strategies listed in Table \[tab:normalize\]. The experiment is conducted based on M2O DIN with “Nonlinearity+Context”. It is clear that keeping the magnitude consistency during downsampling matters. In fact, both max pooling and average pooling satisfy this property naturally, and our normalization design is inspired from this fact.

![Visualization of the randomly initialized index map (left) and the learned index map (right) of HINs (top) and DINs (bottom). Best viewed on screen.[]{data-label="fig:indices_visualization"}](indices_visualization.pdf){width=".46\textwidth"}

*Index Map Visualization*. It is interesting to see what indices are learned by IndexNet. For the holistic index, the index map itself is a 2D matrix and can be easily visualized. Regarding the depthwise index, we squeeze the index map along the channel dimension and calculate the average responses. Two examples of learned index maps are visualized in Fig. \[fig:indices\_visualization\]. We observe that, initial random indices have poor delineation for edges, while learned indices automatically capture the complex structural and textual patterns, e.g., the fur of the dog, and even air bubbles in the water.

Image Denoising
---------------

The goal of image denoising is to recover a clean image ${\mbox{\boldsymbol{$x$}}}$ from a corrupted observation ${\mbox{\boldsymbol{$y$}}}$ following an image degradation model ${\mbox{\boldsymbol{$y$}}}={\mbox{\boldsymbol{$x$}}} + {\mbox{\boldsymbol{$v$}}}$, where ${\mbox{\boldsymbol{$v$}}}$ is commonly assumed to be additive white Gaussian noise (AWGN) parameterized by $\sigma$. While such an assumption has been challenged in recent real-image denoising [@chen2018learning; @jaroensri2019generating], we still follow the AWGN paradigm in evaluation because our focus is not to improve image denoising. When deep CNNs are widely accepted, the data-driven paradigm now becomes the first-class choice for image denoising [@mao2016image; @zhang2017beyond]. Most deep denoising models are designed with the same high-level idea—processing the feature map without decreasing its spatial resolution. Indeed, it has been observed that, when the feature map is downsampled, the performance drops remarkably [@mao2016image]. For such networks, although the model parameters largely reduce, computational complexity of training and inference becomes much heavier.

We show that, by inserting IndexNet into a denoising model, it can effectively compensate the loss of spatial information, achieving performance comparable to or even better than the network without downsampling. Thus, despite the number of parameters increases, computation us much reduced. We choose DnCNN [@zhang2017beyond] as our baseline to demonstrate this on standard benchmarks. We follow the experimental setting of [@chen2016trainable] that uses a 400-image training set. The performance is reported on a 68-image Berkeley segmentation dataset (BSD68) and the other 12-image test set (Set12). The networks are trained for Gaussian denoising, with three noise levels, i.e., $\sigma=15,25$ and $50$. PSNR and SSIM are used as evaluation metrics.

![The DnCNN architecture and our modified SegNet-like DnCNN.[]{data-label="fig:dncnn"}](dncnn.pdf){width="\linewidth"}

[&gt;p[.85cm]{}|&gt;p[1.25cm]{}|c|c|ccc|ccc]{} & \#Param. & GFLOPs & &\
& & & 15 & 25 & 50 & 15 & 25 & 50\
& 0.56M & 25.89 & 31.74/0.9410 & 29.22/0.9015 & 26.23/0.8269 & 32.87/0.9544 & 30.42/0.9296 & 27.17/0.8775\
& 7.09M & 18.14 & 30.74/0.9278 & 28.27/0.8752 & 24.88/0.7437 & 31.91/0.9395 & 28.98/0.8881 & 24.99/0.7485\
& 7.29M & 19.11 & 25.70/0.7578 & 21.41/0.5670 & 15.40/0.2988 & 24.64/0.6997 & 20.19/0.4965 & 14.22/0.2489\
& 7.29M & 19.11 & 25.70/0.7578 & 21.43/0.5673 & 15.56/0.2968 & 24.64/0.6997 & 20.20/0.4968 & 14.26/0.2460\
& 7.29M & 15.23 & 25.70/0.7578 & 21.43/0.5672 & 15.50/0.2994 & 24.64/0.6997 & 20.20/0.4967 & 14.22/0.2496\
NL & C & $\Delta$\
& & & &\
& & +7.17K & 18.16 & 31.13/0.9357 & 29.02/0.8997 & 26.29/0.8281 & 32.71/0.9536 & 30.28/0.9285 & 27.20/0.8789\
& & +0.69M & 19.30 & 31.15/0.9356 & 29.01/0.8999 & 26.29/0.8301 & 32.77/0.9537 & 30.36/0.9295 & 27.18/0.8799\
& & +2.76M & 22.75 & 31.20/0.9365 & 29.05/0.9004 & 26.30/0.8305 & 32.79/0.9541 & 30.37/0.9300 & 27.22/0.8804\
& & & &\
& & +16 & 18.14 & 31.22/0.9366 & 29.06/0.9002 & 25.84/0.8294 & 32.83/0.9545 & 30.42/0.9302 & 26.21/0.8782\
& & +56 & 18.14 & 30.64/0.9255 & 27.39/0.8391 & 24.15/0.6776 & 31.92/0.9386 & 27.97/0.8330 & 24.14/0.6747\
& & +152 & 18.14 & 30.87/0.9296 & 27.70/0.8617 & 24.09/0.6939 & 32.23/0.9432 & 28.31/0.8677 & 23.85/0.6634\
& & & &\
& & +48 & 18.14 & 31.14/0.9364 & 29.05/0.9002 & 26.32/0.8310 & 32.80/0.9542 & 30.41/0.9302 & 27.24/0.8807\
& & +168 & 18.14 & 31.20/0.9365 & 28.97/0.9000 & 26.18/0.8272 & 32.83/0.9545 & 30.43/0.9302 & 27.24/0.8801\
& & +456 & 18.14 & 31.22/0.9366 & 29.07/0.9004 & 26.31/0.8311 & 32.82/0.9543 & 30.41/0.9300 & 27.27/0.8814\
& & & &\
& & +7.17K & 18.16 & 31.17/0.9366 & 28.25/0.8944 & 25.02/0.8235 & 32.80/0.9544 & 30.23/0.9286 & 26.41/0.8675\
& & +25.1K & 18.19 & 31.25/0.9368 & 29.06/0.9002 & 26.33/0.8306 & 32.77/0.9541 & 30.43/0.9303 & 27.29/0.8814\
& & +68.1K & 18.32 & 31.21/0.9364 & 27.68/0.8740 & 26.33/0.8312 & 32.83/0.9544 & 30.32/0.9288 & 27.24/0.8807\
& & & &\
& & +1.38M & 20.44 & 31.22/0.9365 & 29.03/0.9005 & 26.33/0.8316 & 32.82/0.9544 & 30.42/0.9302 & 27.28/0.8812\
& & +3.45M & 23.88 & 31.23/0.9368 & 29.07/0.9002 & 26.26/0.8278 & 32.84/0.9546 & 30.44/0.9304 & 27.28/0.8808\
& & +11.7M & 37.67 & 31.23/0.9365 & 29.06/0.8996 & 26.34/0.8315 & 32.82/0.9545 & 30.43/0.9301 & 27.29/0.8803\

NL: Non-Linearity; C: Context. $\Delta$ indicates increased parameters compared to the SegNet-DnCNN baseline. GFLOPs are measured on a $224\times224\times1$ input.

### Network Architecture and Implementation Details

*Network Architecture*. We use the 17-layer DnCNN model [@zhang2017beyond], implemented by PyTorch. To enable the use of IndexNet, we modify DnCNN to a SegNet-like architecture with $3$ downsampling and upsampling stages (the input image size is $40\times40$). The number of layers remains the same to ensure a relatively fair comparison. Fig. \[fig:dncnn\] illustrates the original DnCNN and our modified architecture. The first $9$ layers follow VGG-16 except that the first layer is a single-channel input, and the rest are $7$ decoding layers formed by unpooling and convolution and the final prediction layer. All convolutional operations use $3\times 3$ kernels. To incorporate IndexNet, it is straightforward to replace max pooling and unpooling with $\mathcal{IP}$ and $\mathcal{IU}$.

*Training Details*. We follow the same experimental configurations used in [@zhang2017beyond]. At each epoch, $40\times40$ image patches are cropped from multiple scales ($0.7, 0.8, 0.9, 1$) with a stride of $10$ and are added with Gaussian noise of a certain noise level ($\sigma=15,25,$ or $50$); image patches are further augmented with random flipping and random rotation. This results in around $240,000$ training samples. $\ell_2$ loss is used. All networks are trained from scratch with a batch size of $128$. Model parameters are initialized with the improved Xavier [@he2015delving]. The Adam optimizer is also used. Parameters are updated with $60$ epochs. The learning rate is initially set to $0.001$ and reduced by $10\times$ at the $45$-th and $55$-th epoch, respectively.

### Results on the BSD68 and Set12 Datasets

Apart from the DnCNN baseline, we also report the performance of our modified DnCNN-SegNet with max pooling and unpooling. Furthermore, to compare IndexNet against CARAFE, we build three additional baselines where CARAFE is combined with different downsampling strategies, including max pooling, average pooling, and stride-2 convolutions, denoted by DnCNN-max-carafe, DnCNN-avg-carafe, and DnCNN-conv-carafe, respectively. Results are shown in Table \[tab:indexnet\_denoising\]. It can be observed that, simply downsampling with max pooling and upsampling by unpooling as in DnCNN-SegNet lead to significant drops in both PSNR (generally $>1 dB$) and SSIM ($>0.1$). This suggests that spatial information plays an important role in image denoising. Denoising is content-irrelevant (the model is unaware of regions coming from the foreground or the background). Downsampling without recording sufficient spatial information (only the boundary information is not sufficient) impedes the model from recovering the appearance and the structure in the original image. This is particularly true for baselines adopting CARAFE. Since CARAFE applies blind upsampling, no spatial information is transferred during upsampling, which may lead to inferior results. Interestingly, after IndexNet is inserted into downsampled DnCNN, the loss of PSNR and SSIM is effectively compensated. The compensation behaviors can be observed from almost all types of IndexNet, except the two cases in Modelwise O2O DINs with nonlinearity. The poor performance of Modelwise O2O DINs may attribute to the insufficient modeling ability, particularly when $\sigma=50$. Nonlinearity and weak context generally have a positive effect on image denoising, and the effectiveness of different IndexNets is similar. Hence, Shared Stagewise O2O DINs appear to be a preferred choice due to slightly increased parameters and negligible extra computation costs.

Semantic Segmentation
---------------------

Here we further evaluate IndexNet on semantic segmentation. Semantic segmentation aims to predict a dense labeling map for each image where each pixel is labeled into one category. Since the FCNs were introduced [@long2015fully], FCN-based encoder-decoder architectures have been studied extensively [@chen2017deeplab; @lin2017refine; @badrinarayanan2017segnet; @chen18v3]. Efforts have been spent on how to encode contextual information, We use SegNet [@badrinarayanan2017segnet] as our baseline because IndexNet is primarily inspired by the unpooling operator in SegNet. We follow the experimental setting in [@badrinarayanan2017segnet] and report performance on the SUN RGB-D [@song2015sun] dataset. We use RGB as the input (depth is not used). The standard mean Intersection-over-Union (mIoU) is used as the evaluation metric. In addition, we also compare against the recent UperNet [@xiao2018unified]. We evaluate UperNet on the ADE20K dataset [@zhou2017scene].

![IndexNet-guided feature pyramid network and multi-level feature fusion.[]{data-label="fig:indexnet-fpn-mff"}](indexnet-fpn-mff.pdf){width="\linewidth"}

### Network Architecture and Implementation Details

*Network Architecture.* The architecture of SegNet employs the first $13$ layers of the VGG-16 model pretrained on ImageNet as the encoder. The decoder uses unpooling for upsampling. Each unpooling layer is followed by the same number of convolutional layers as in the corresponding encoder stage. Overall, SegNet has $5$ downsampling and $5$ upsampling stages. Convolutional layers in the decoding stage mainly play a role to smooth the feature maps generated by unpooling. To insert IndexNet, the only modification is to replace max pooling and unpooling layers with $\mathcal{IP}$ and $\mathcal{IU}$, respectively, which is straightforward.

UperNet builds upon the idea of Pyramid Pooling Module (PPM) [@zhao2017pyramid] and Feature Pyramid Network (FPN) [@lin2017feature]. UperNet also implements a Multi-level Feature Fusion (MFF) module that fuses multi-resolution feature maps by concatenation. In FPN, downsampling is implemented by 2-stride convolution, and upsampling uses bilinear interpolation. It produces four feature levels $\{D_2, D_3, D_4, D_5\}$ with output strides of $\{4,8,16,32\}$, conditioned on the encoder features $\{E_2, E_3, E_4, E_5\}$. MFF further fuses four levels of features and generates the output $M_2$ with an output stride of $4$. To insert IndexNet, three IndexNet blocks can be inserted into the encoder to generate index maps to guide upsampling. The same index maps can also be used in MFF in a sequential upsampling manner to fuse features, as shown in Fig. \[fig:indexnet-fpn-mff\]. Note that, in theory IndexNet can also be applied to PPM, because PPM itself has internal downsampling and upsampling stages. However, we discourage the use of IndexNet in PPM, because it will significantly increase parameters (due to mixed downsampling/upsampling rates). In this case blind upsampling such as NN/bilinear interpolation may be a better choice.

*Training Details.* On the SUN RGB-D dataset, the VGG-16 model pretrained on ImageNet with BN layers is used. We employ the standard data augmentation strategies: random scaling, random cropping $320\times320$ sub-images, and random horizontal flipping. We learn the model with the standard softmax loss. Encoder parameters are pretrained on ImageNet. All other parameters are initialized with the improved Xavier [@he2015delving]. The SGD optimizer [@kingma2015adam] is used with a momentum of $0.9$ and a weight decay of $0.0001$. We train the model with a batch size of $16$ for $300$ epochs (around $90,000$ iterations). The learning rate is initially set to $0.01$ and reduced by $10\times$ at the $250$-th and $280$-th epoch, respectively. The BN layers of the encoder are fixed.

On the ADE20K benchmark, we use the MobileNetV2 pretrained on ImageNet as the encoder and UperNet the decoder. Due to limited computational resources, only this setting enables us to train a model on $4$ GPUs with a batch size of $16$ following the official implementation of UperNet and provided experimental settings.[^1]

![image](segmentation_results.pdf){width=".8\textwidth"}

### Results on the SUN RGB-D Dataset

We report the results in Table \[tab:indexnet\_segmentation\]. All index networks show improvements over the baseline, among which Modelwise and Shared Stagewise O2O DINs improve the baseline with few extra parameters and GFLOPs. Compared with other types of IndexNet, M2O DINs and HINs (particularly under the setting of “Nonlinearity+Context”) increase many parameters and GFLOPs but do not exhibit clear advantages. From the qualitative results shown in Fig. \[fig:segmentation\_results\], we can see that the improvement comes from the ability of IndexNet suppressing fractured predictions that frequently appears in the baseline SegNet. IndexNet seemingly does better in producing predictions at boundaries.

Notice that, in contrast to the behaviour in matting and denoising, CARAFE significantly enhances the performance in segmentation ($32.47\rightarrow36.30$), outperforming IndexNet. We observe that CARAFE tends to produce consistent region-wise predictions. A plausible explanation is that, CARAFE is designed in a way to tackle region-sensitive tasks such as semantic segmentation where region-wise matching between predictions and ground truths matters, while IndexNet prefers detail-sensitive tasks like image matting where errors come from detail-rich regions.

        \#Param.    GFLOPs      mIoU     
  ---- ---------- ---------- ----------- -------
         24.96M     24.76       32.47    
         25.35M     25.75     **36.30**  
   NL      C       $\Delta$              
                                         
                   +23.55K      24.79     33.25
                    +4.90M      26.40     33.11
                   +19.55M      31.28     33.31
                                         
                     +16        24.76     33.18
                     +56        24.76     33.70
                     +152       24.77     33.26
                                         
                     +80        24.76     33.26
                     +280       24.76     33.97
                     +760       24.77     33.41
                                         
                    +0.02M      24.79     33.27
                    +0.08M      24.82     33.59
                    +0.22M      24.96     33.50
                                         
                    +9.76M      28.02     33.28
                   +24.44M      32.90     33.51
                   +83.02M      52.42     33.48

  : Performance on the SUN RGB-D Dataset.[]{data-label="tab:indexnet_segmentation"}

NL: Non-Linearity; C: Context. $\Delta$ indicates increased parameters compared to the SegNet baseline. GFLOPs are measured on a $224\times224\times3$ input.

     FPN        MFF        mIoU      Pixel Accuracy ($\%$)
  ---------- ---------- ----------- -----------------------
   Bilinear   Bilinear     37.08             78.29
   IndexNet   Bilinear     36.25             78.23
   Bilinear   IndexNet     36.90             78.27
   IndexNet   IndexNet     37.62             78.29
    CARAFE    Bilinear     37.76             78.81
   Bilinear    CARAFE      38.03             78.51
    CARAFE     CARAFE    **38.31**         **78.90**

  : Performance on the ADE-20K Dataset[]{data-label="tab:indexnet_segmentation_2"}

MFF: Multi-level Feature Fusion. Only HINs (‘Linear’) are evaluated due to varied decoder feature dimensionality. The best performance is boldfaced.

### Results on the ADE20K Dataset

Here we conduct ablative studies to highlight the role of upsampling in UperNet. Both IndexNet and CARAFE are considered. In addition to the full replacement of upsampling operators following Fig. \[fig:indexnet-fpn-mff\], we also replace bilinear upsampling either in FPN or in MFF with IndexNet/CARAFE. Results are shown in Table \[tab:indexnet\_segmentation\_2\]. It can be observed that, IndexNet improves UperNet ($37.08\rightarrow37.62$) only when bilinear upsampling in FPN and MFF is simultaneously replaced. However, when only one component is modified, IndexNet even leads to negative results. This suggests that the guided information should be used consistently in the decoder. In addition, CARAFE also works better than IndexNet, showing that spatial information may not play a critical role in semantic segmentation.

Monocular Depth Estimation
--------------------------

Estimating per-pixel depth from a single image is challenging because one needs to recover 3D information from a 2D plane. With deep learning, significant progress has been witnessed [@liu2015learning; @xian2018monocular; @wofk2019fastdepth]. We use the recent FastDepth [@wofk2019fastdepth] as our baseline. We compare the performance with/without IndexNet on the NYUv2 dataset [@silberman2012indoor] with the official train/test split. To be in consistent with [@wofk2019fastdepth], the following metrics are used to quantify the performance:

-   root mean square error (rms): $\sqrt { \frac { 1 }{ T } \sum _{ i=1 }^{ T }{ { \left( { d }_{ i }-{ g }_{ i } \right)  }^{ 2 } }  } $;

-   accuracy with threshold $th$: percentage (%) of $d_1$, $\rm s.t.$ $\max { \left( \frac { { d }_{ 1 } }{ { g }_{ 1 } } ,\frac { { g }_{ 1 } }{ { d }_{  } }  \right)  } =\delta_{1} <$ $th$.

### Network Architecture and Implementation Details

*Network Architecture.* FastDepth is an encoder-decoder architecture, with MobileNet as its backbone. Here we choose the best upsampling option suggested by the authors [@wofk2019fastdepth] where upsampling is implemented by $\times2$ NN interpolation and $5\times5$ convolution. Hence, our baseline is FastDepth-NNConv5: downsampling with 2-stride convolution and upsampling via NN interpolation. We also modify this baseline by changing the stride-2 convolution to be stride-1 followed by max-pooling, named as FastDepth-P-NNConv5. Fig. \[fig:fastdepth\_architecture\] shows how we insert IndexNet into FastDepth. Similar to the modifications applied to the matting network, stride-2 convolution layers in the encoder are changed to be stride-1, followed by $\mathcal{IP}$, and the NN interpolation in the decoder is replaced with $\mathcal{IU}$. To compare IndexNet with CARAFE, we build two additional baselines: FastDepth-carafe and FastDepth-P-carafe, where NNConv5 is modified to CARAFE in FastDepth-NNConv5 and FastDepth-P-NNConv5.

![Our modified FastDepth [@wofk2019fastdepth] architecture.[]{data-label="fig:fastdepth_architecture"}](fastdepth_architecture.pdf){width="3.5in"}

*Training Details.* We follow similar training settings used by FastDepth [@wofk2019fastdepth]. $\ell_1$ loss is used. Random rotation, random scaling and random horizontal flipping are used for data augmentation. The initial learning rate is set to $0.01$ and reduced by $\times10$ every $5$ epochs. The SGD optimizer is used with a momentum of $0.9$ and a weight decay of $0.0001$. Encoder weights are pretrained on ImageNet [@deng2009imagenet]. A batch size of $16$ is used to train the network for $30$ epochs in total.

[&gt;p[1.1cm]{}|&gt;p[1.45cm]{}|c|c|c|c]{} & \#Param. & GFLOPs & rms & $\delta_{1}<1.25$\

& 3.96M & 0.69 & 0.567 & 0.781\
& 3.96M & 1.01 & 0.577 & 0.778\
& 4.31M & 1.63 & 0.558 & **0.790**\
& 4.31M & 1.96 & 0.571 & 0.782\
NL & C & $\Delta$\
& &\
& & +31.23K & 1.03 & 0.566 & 0.784\
& & +11.17M & 2.65 & 0.565 & 0.786\
& & +44.62M & 7.53 & 0.559 & 0.787\
& &\
& & +16 & 1.02 & 0.569 & 0.778\
& & +56 & 1.02 & 0.568 & 0.785\
& & +152 & 1.02 & 0.564 & 0.786\
& &\
& & +80 & 1.02 & 0.562 & 0.783\
& & +280 & 1.02 & 0.565 & 0.786\
& & +760 & 1.02 & 0.567 & 0.783\
& &\
& & +31.23K & 1.03 & **0.556** & 0.789\
& & +0.11M & 1.06 & 0.564 & 0.786\
& & +0.30M & 1.16 & 0.562 & 0.788\
& &\
& & +22.30M & 4.27 & 0.563 & 0.783\
& & +55.78M & 9.15 & 0.562 & 0.786\
& & +189.57M & 28.67 & 0.565 & 0.787\

NL: Non-Linearity; C: Context. $\Delta$ indicates increased parameters compared to the standard FastDepth baseline. GFLOPs are measured on a $224\times224\times3$ input. The best performance is boldfaced.

                                rms         rel        log10     $\delta<1.25$   $\delta<1.25^2$   $\delta<1.25^3$
  ------------------------- ----------- ----------- ----------- --------------- ----------------- -----------------
   Hu  [@hu2019revisiting]     0.558       0.129       0.055         0.837          **0.968**         **0.992**
        Hu + IndexNet        **0.554**   **0.128**   **0.054**     **0.843**        **0.968**         **0.992**

Only HIN (‘Nonlinear+Context’) is evaluated due to varied feature dimensionality of decoder and multi-level feature fusion. *rel* and *log10* denote the average relative error and average $\log_{10}$ error [@xian2018monocular], respectively. The best performance is boldfaced.

### Results on the NYUDv2 Dataset

We report the results in Table \[tab:indexnet\_fastdepth\]. We observe that almost all types of IndexNet improve the performance compared to the baselines except for the most light-weight design—linear Modelwise O2O DIN. It may be because only $16$ parameters are not sufficient to model local variations of high-dimensional feature maps. Note that, Unshared Stagewise O2O DINs (with only linear mappings) shows clear improvements with only slightly increased parameters. HINs and M2O DINs increase a large amount of parameters and floating-point calculations because of the high dimensionality of feature maps, while the improved performance is not proportional to such a high cost. Some qualitative results are further illustrated in Fig. \[fig:fastdepth\]. We observe that IndexNet exhibits better boundary delineation than the baseline, e.g., the edge of the desk, and the contour of the woman. Moreover, CARAFE achieves comparable performance against IndexNet in this task. In addition, we report the results of applying IndexNet to a state-of-the-art model [@hu2019revisiting] in Table \[tab:indexnet\_depth\_2\]. We modify the usage of IndexNet here by taking the same feature fusion strategies shown in Fig. \[fig:indexnet-fpn-mff\]. Other implementation details and evaluations are kept consistent with [@hao2019indexnet]. Compared with the baseline, IndexNet shows improvement in the first four metrics.

![image](fastdepth.pdf){width="65.00000%"}

Insights Towards Good Practices {#ssec:practices}
-------------------------------

As a summary of our evaluations, here we provide some guidelines for using guided/blind upsampling:

1.  In detail-sensitive tasks, such as image matting, image restoration, and edge detection, the spatial information is important. Thus guided upsampling may be preferred.

2.  Blind upsampling may be used in the situation when computational budget is limited because most blind upsampling operators are non-parametric and computationally efficient.

3.  In image matting, the best IndexNet configuration is “M2O DINs+Nonlinearity+Context”. This configuration is also true for the image reconstruction experiment and image denoising, where M2O DINs exhibit the best performance and the most stable behavior, respectively. Hence, the capacity of IndexNet is closely related to the complexity of local patterns. M2O DINs is preferred in a detail- or boundary-sensitive task, but one should also be aware of the increased model parameters and computation costs, especially when the feature maps are high-dimensional.

4.  If one prefers a flexible decoder design, e.g., squeezing/enlarging the dimensionality of the decoder feature map, HINs are good choices, because DINs only generate index maps whose dimensionality is identical to the input feature map.

5.  For real-time applications, Shared Stagewise O2O DINs are the first choices. Model parameters increased by Shared Stagewise O2O DINs are comparable to Modelwise O2O DINs, and the extra GFLOPs are also negelectable. Shared Stagewise O2O DINs, however, always work better than Modelwise O2O DINs for applications considered in this work. It implies that each upsampling stage should learn a stage-specific index function;

6.  It is worth noting that, the current implementation of IndexNet has some limitations. Currently IndexNet only implements single-point upsampling—each upsampled feature point is only associated with a single point. In this sense, we may not simulate the behavior of bilinear interpolation where each upsampled point is affected by multiple points of a local region.

Conclusion
==========

Inspired by an observation in image matting, we examine the role of indices and present a unified view of upsampling operators using the notion of index functions. We show that an index function can be learned within a proposed index-guided encoder-decoder framework. In this framework, indices are learned with a flexible network module termed IndexNet, and are used to guide downsampling and upsampling using $\mathcal{IP}$ and $\mathcal{IU}$. IndexNet itself is also a sub-framework that can be designed depending on the task at hand. We investigate five index networks, and demonstrate their effectiveness on four dense prediction tasks. We believe that IndexNet is an important step towards generic upsampling operators for deep networks.

[10]{} \[1\][\#1]{} url@samestyle \[2\][\#2]{} \[2\][[l@\#1=l@\#1\#2]{}]{}

M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in *Proc. European Conference on Computer Vision (ECCV)*, 2014, pp. 818–833.

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015, pp. 3431–3440.

V. Badrinarayanan, A. Kendall, and R. Cipolla, “[SegNet]{}: [A]{} deep convolutional encoder-decoder architecture for image segmentation,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 39, no. 12, pp. 2481–2495, 2017.

W. Shi, J. Caballero, F. Husz[á]{}r, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016, pp. 1874–1883.

G. Lin, A. Milan, C. Shen, and I. Reid, “[RefineNet]{}: [Multi-path]{} refinement networks for high-resolution semantic segmentation,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 1925–1934.

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in *Proc. European Conference on Computer Vision (ECCV)*, 2018.

N. Xu, B. Price, S. Cohen, and T. Huang, “Deep image matting,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 2970–2979.

T. Kraska, A. Beutel, E. H. Chi, J. Dean, and N. Polyzotis, “The case for learned index structures,” in *Proc. International Conference on Management of Data*, 2018, pp. 489–504.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018, pp. 4510–4520.

K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a [G]{}aussian denoiser: residual learning of deep [CNN]{} for image denoising,” *IEEE Transactions on Image Processing*, vol. 26, no. 7, pp. 3142–3155, 2017.

S. Roth and M. J. Black, “Fields of experts,” *International Journal of Computer Vision*, vol. 82, no. 2, p. 205, 2009.

S. Song, S. P. Lichtenberg, and J. Xiao, “[SUN RGB-D]{}: [A RGB-D]{} scene understanding benchmark suite,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015, pp. 567–576.

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in *Proc. European Conference on Computer Vision (ECCV)*, 2012, pp. 746–760.

H. Lu, Y. Dai, C. Shen, and S. Xu, “Indices matter: Learning to index for deep image matting,” in *Proc. IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019, pp. 3266–3275.

A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” *Distill*, vol. 1, no. 10, p. e3, 2016.

C. Osendorfer, H. Soyer, and P. Van Der Smagt, “Image super-resolution with fast approximate convolutional sparse coding,” in *Proc. International Conference on Neural Information Processing (ICONIP)*, 2014, pp. 250–257.

T.-J. Yang, M. D. Collins, Y. Zhu, J.-J. Hwang, T. Liu, X. Zhang, V. Sze, G. Papandreou, and L.-C. Chen, “[DeeperLab]{}: Single-shot image parser,” *arXiv*, 2019.

J. Wang, K. Chen, R. Xu, Z. Liu, C. C. Loy, and D. Lin, “[CARAFE]{}: Context-aware reassembly of features,” in *Proc. IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019.

M. Jaderberg, K. Simonyan, A. Zisserman *et al.*, “Spatial transformer networks,” in *Advances in Neural Information Processing Systems (NIPS)*, 2015, pp. 2017–2025.

X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool, “Dynamic filter networks,” in *Advances in Neural Information Processing Systems (NIPS)*, 2016, pp. 667–675.

J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in *Proc. IEEE International Conference on Computer Vision (ICCV)*, 2017, pp. 764–773.

V. Mnih, N. Heess, A. Graves *et al.*, “Recurrent models of visual attention,” in *Advances in Neural Information Processing Systems (NIPS)*, 2014, pp. 2204–2212.

F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 3156–3164.

J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition*, 2018, pp. 7132–7141.

S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “[CBAM]{}: Convolutional block attention module,” in *Proc. European Conference on Computer Vision (ECCV)*, 2018, pp. 3–19.

H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” *arXiv*, 2017.

Q. Chen, D. Li, and C.-K. Tang, “[KNN]{} matting,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 35, no. 9, pp. 2175–2188, 2013.

Y.-Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A bayesian approach to digital matting,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2001, pp. 264–271.

K. He, C. Rhemann, C. Rother, X. Tang, and J. Sun, “A global sampling method for alpha matting,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.1em plus 0.5em minus 0.4emIEEE, 2011, pp. 2049–2056.

A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 30, no. 2, pp. 228–242, 2008.

R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2014, pp. 580–587.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “[ImageNet]{} classification with deep convolutional neural networks,” in *Advances in Neural Information Processing Systems (NIPS)*, 2012, pp. 1097–1105.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll[á]{}r, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in *Proc. European Conference on Computer Vision (ECCV)*, 2014, pp. 740–755.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” *International Journal of Computer Vision*, vol. 88, no. 2, pp. 303–338, 2010.

C. Rhemann, C. Rother, J. Wang, M. Gelautz, P. Kohli, and P. Rott, “A perceptually motivated online benchmark for image matting,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2009, pp. 1826–1833.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “[ImageNet]{}: A large-scale hierarchical image database,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.1em plus 0.5em minus 0.4emIeee, 2009, pp. 248–255.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in *Proc. International Conference on Learning Representations (ICLR)*, 2015.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016, pp. 770–778.

O. Ronneberger, P. Fischer, and T. Brox, “[U-Net]{}: [C]{}onvolutional networks for biomedical image segmentation,” in *Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)*, 2015, pp. 234–241.

C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018, pp. 3291–3300.

R. Jaroensri, C. Biscarrat, M. Aittala, and F. Durand, “Generating training data for denoising real rgb images via camera pipeline simulation,” *arXiv*, 2019.

X. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in *Advances in Neural Information Processing Systems (NIPS)*, 2016, pp. 2802–2810.

Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 39, no. 6, pp. 1256–1272, 2016.

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in *Proc. IEEE International Conference on Computer Vision (ICCV)*, 2015, pp. 1026–1034.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 40, no. 4, pp. 834–848, 2017.

T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” in *Proc. European Conference on Computer Vision (ECCV)*, 2018, pp. 418–434.

B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017.

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 2881–2890.

T.-Y. Lin, P. Doll[á]{}r, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 2117–2125.

F. Liu, C. Shen, G. Lin, and I. Reid, “Learning depth from single monocular images using deep convolutional neural fields,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 38, no. 10, pp. 2024–2039, 2015.

K. Xian, C. Shen, Z. Cao, H. Lu, Y. Xiao, R. Li, and Z. Luo, “Monocular relative depth perception with web stereo data supervision,” in *Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018, pp. 311–320.

D. Wofk, F. Ma, T.-J. Yang, S. Karaman, and V. Sze, “Fastdepth: Fast monocular depth estimation on embedded systems,” in *Proc. IEEE International Conference on Robotics and Automation (ICRA)*, 2019.

J. Hu, M. Ozay, Y. Zhang, and T. Okatani, “Revisiting single image depth estimation: toward higher resolution maps with accurate object boundaries,” in *Proc. IEEE Winter Conference on Applications of Computer Vision (WACV)*, 2019, pp. 1043–1051.

[Hao Lu]{} received the Ph.D. degree from Huazhong University of Science and Technology, Wuhan, China, in 2018. He is currently a Postdoctoral Fellow at School of Computer Science, The University of Adelaide, Australia. His research interests include computer vision and machine learning. He is currently working on dense prediction problems.

plus -1fil

[Yutong Dai]{} received the M.Sc. degree from Southeast University, Nanjing, in 2018. She is currently pursuing the Ph.D.  degree at The University of Adelaide, Australia. Her research interests include computer vision and deep learning. She is currently working on image matting.

plus -1fil

[Chunhua Shen]{} is a Professor of Computer Science at The University of Adelaide, Australia.

plus -1fil

[Songcen Xu]{} received the Ph.D. degree in electronic engineering from the University of York, U.K. in 2015. He is with Noah’s Ark Lab, Huawei Technologies.

plus -1fil

[^1]: <https://github.com/CSAILVision/semantic-segmentation-pytorch>
