---
abstract: 'In this paper, we address the *boundary detection* task motivated by the ambiguities in current definition of edge detection. To this end, we generate a large database consisting of more than 10k images (which is 20$\times$ bigger than existing edge detection databases) along with ground truth boundaries between 459 semantic classes including both foreground objects and different types of background, and call it the *PASCAL Boundaries* dataset, which will be released to the community. In addition, we propose a novel deep network-based multi-scale semantic boundary detector and name it Multi-scale Deep Semantic Boundary Detector (M-DSBD). We provide baselines using models that were trained on edge detection and show that they transfer reasonably to the task of boundary detection. Finally, we point to various important research problems that this dataset can be used for.'
author:
- |
    Vittal Premachandran Boyan Bonev Alan L. Yuille\
    University of California, Los Angeles,\
    [{vittalp@, bonev@, yuille@stat.}ucla.edu]{}
bibliography:
- 'egbib.bib'
title: 'PASCAL Boundaries: A Class-Agnostic Semantic Boundary Dataset'
---

Introduction
============

Edge detection has been a fundamental problem in computer vision since the 1970’s [@fram1975quantitative]. Detecting edges is beneficial for many vision tasks, for example, object detection [@zitnick2014edge], image segmentation [@arbelaez2011contour], neural circuit reconstruction from brain images [@ciresan2012deep], and autonomous navigation, among others. This problem is under active research and potential solutions include local filtering-based approaches like the Canny edge detector [@canny1986computational] and the zero-crossing algorithm [@torre1986edge], to pixel-level classification methods that use features obtained by careful manual design like *gPb* [@arbelaez2011contour], to patch-based clustering algorithms such as Structured Edges (SE) [@dollar2014fast], to the more-recent deep learning based approaches such as the N4-network [@ciresan2012deep] or HED [@xie2015holistically].

![This figure shows the differences between edge annotations from the BSDS500 dataset (top row) and our class-agnostic object-level boundary annotations (bottom row). Our annotations are restricted to object outlines and background classes and generate boundaries around 459 semantic classes.[]{data-label="fig:teaser"}](./img/taj.jpg "fig:"){width="0.49\linewidth"} ![This figure shows the differences between edge annotations from the BSDS500 dataset (top row) and our class-agnostic object-level boundary annotations (bottom row). Our annotations are restricted to object outlines and background classes and generate boundaries around 459 semantic classes.[]{data-label="fig:teaser"}](./img/taj_bsds.pdf "fig:"){width="0.49\linewidth"}\
![This figure shows the differences between edge annotations from the BSDS500 dataset (top row) and our class-agnostic object-level boundary annotations (bottom row). Our annotations are restricted to object outlines and background classes and generate boundaries around 459 semantic classes.[]{data-label="fig:teaser"}](./img/8.jpg "fig:"){width="0.49\linewidth"} ![This figure shows the differences between edge annotations from the BSDS500 dataset (top row) and our class-agnostic object-level boundary annotations (bottom row). Our annotations are restricted to object outlines and background classes and generate boundaries around 459 semantic classes.[]{data-label="fig:teaser"}](./img/bdr_gt_8.png "fig:"){width="0.49\linewidth"}

However, edge detection is an ambiguous task making it difficult to evaluate. There is no clear answer to the question, ‘What is an edge?’ An accepted definition of an edge is those sets of pixels with strong gradients. Existing edge detection databases such as the BSDS300 [@martin2004learning] and BSDS500 [@arbelaez2011contour] were generated by asking the annotators to divide the image into multiple segments resulting in different annotators dividing the images into different segments. This lack of consistency arises because of the fact that edges can occur at different levels of granularity; i) just the exterior boundaries of objects, which divide the image into different object instances (car, road, tree, etc.), ii) interior boundaries dividing an object into its constituent parts (head, neck, torso, etc.), or iii) non-semantic contours emanating from texture (stripes on a tiger) or artificial design (writings on clothes). Hou et al. [@hou2013boundary] discuss the ambiguities in these datasets in more detail. In addition to the ambiguity, the BSDS500 dataset has only 500 images and cannot be considered as a large database. This motivates us to construct a new, large, class-agnostic semantic boundary dataset, that is not only large in comparison to BSDS500, but is also without the ambiguity of edge detection.

In this work, we wish to eliminate this ambiguity by restricting ourselves to the coarsest level of granularity, i.e., semantic instance-level object boundaries. Further extensions of our work is possible by introducing edges of other levels of granularity. Figure \[fig:teaser\] shows the differences between the BSDS500 annotations (top row) and our annotations (bottom row).

This paper makes the following contributions: i) We define a precise task, namely, *boundary detection*. To enable progress on this problem, we construct a large dataset with $\sim$10k images taken from the PASCAL VOC2010 challenge and provide boundary annotations between 459 semantic classes including both foreground objects and different types of background. These boundary annotations will be released publicly. The dataset generation process is described in more detail in Section \[sec:dataset\]. ii) We propose a novel multi-scale deep network-based class-agnostic semantic boundary detector (M-DSBD) to solve the boundary detection task. This is described in detail in Section \[sec:multiscale-deepNetwork\].

The rest of the paper is organized as follows. In Section \[sec:exp\], we provide baselines on this new dataset using two well-performing edge detectors, i.e., Structured Edges (SE) [@dollar2014fast] and Holistically-Nested Edge detector (HED) [@xie2015holistically]. We also provide results obtained using M-DSBD, and its single-scale counterpart DSBD, using existing evaluation methodologies like the F-measure to enable fair comparisons with current and future approaches. Finally, we conclude the paper in Section \[sec:conclusion\] by pointing to various future directions that this dataset paves way to, which would enable progress in many computer vision problems.

Related Work
============

One of the first databases for edge detection was the Sowerby database [@bowyer1998empirical], which consisted of $100$ color images from the streets of Sowerby. At about the same time, there was the South Florida dataset [@bowyer1998empirical], which was simpler than the Sowerby dataset, and consisted of $50$ gray scale images. These datasets enabled the identification of the fact that low-level filtering techniques such as the Canny edge detector [@canny1986computational] were limited in many ways. Moreover, the first real statistical techniques for edge detection [@konishi1999fundamental; @konishi2003statistical] were developed and tested on these datasets.

The Sowerby dataset being too small motivated Martin et al. [@martin2001database] to start creating a public dataset of image segmentations. A set of $300$ images from this dataset was then used in [@martin2004learning], who cast the problem of edge detection as a per-pixel classification problem and evaluated their results using a precision-recall curve, which can be summarized by the now-standard F-Measure. This set of $300$ images later came to be known as the BSDS300 dataset. The BSDS300 dataset enabled the development of some notable edge detectors such as BEL [@dollar2006supervised].

Recently, the BSDS300 dataset was extended to incorporate $200$ additional images [@arbelaez2011contour] and the new superset dataset was named as the BSDS500 dataset. The BSDS500 dataset has since been heavily worked upon producing significant efforts on edge detection algorithms such as the *gPb*-edge detector [@arbelaez2011contour], Sketch Tokens [@lim2013sketch], SE [@dollar2014fast] and, the now state-of-the-art method, HED [@xie2015holistically]. Over the last couple of years, many deep network-based edge detection methods, such as the N4-network [@ciresan2012deep], Deep Edge [@bertasius2014deepedge], Deep Contour [@shendeepcontour] and HED [@xie2015holistically], have shown significant improvements in terms of the F-Measure on the BSDS500 dataset. While these algorithmic improvements are welcome, we feel that further jumps in performance will be limited by the size of the BSDS500 dataset. The issue with regards to the scale of the BSDS500 dataset and its ambiguity was addressed to a certain extent (though not consciously) by Hariharan et al. [@hariharan2011semantic]. They built an instance-level segmentation dataset for the $20$ PASCAL object categories in about $10,000$ images corresponding to the trainval set of the PASCAL VOC2010 challenge. However, their aim in building this dataset was to tackle the problem of obtaining class-specific object boundaries, thus requiring them to train $\mathcal{O}(N)$ boundary detectors corresponding to $N$ object classes. Uijlings and Ferrari [@uijlings2015situational] go even more extreme by subdividing each object class into $K$ subclasses, which they call “situations", thus requiring them to train $\mathcal{O}(NK)$ situational object boundary detectors. Clearly, neither of these approaches are scalable for large $N$ and $K$. Therefore, we adopt a class-agnostic boundary detection strategy by going back to building a strong monolithic boundary detector, thus requiring us to train a $\mathcal{O}(1)$ boundary detector. Such an approach allows for the sharing of the computation involved in performing the mid-level task across various high-level vision tasks. Sharing of mid-level computations enables seamless scalability across multiple high-level visual tasks.

A similar dataset is the MS-COCO dataset [@mscoco], which contains instance-level masks for 80 object categories and is much larger than the PASCAL dataset. However, this dataset also contains masks only for foreground objects. In comparison, we consider all objects in the image and an initial taxonomy established 459 semantic classes that includes both foreground objects and different types of backgrounds (e.g. sky, water, grass, etc.). Finally, Zhu et al. [@zhu2015semantic] recently proposed an amodal segmentation dataset, where they label the complete extents of an object (even if they are occluded) on the 500 images of the BSDS500 dataset. We restrict ourselves to the unoccluded parts of the object since labeling the occluded regions of objects would again lead to ambiguity in the task.

Multi-scale Deep Semantic Boundary Detector (M-DSBD) {#sec:multiscale-deepNetwork}
====================================================

To complement the PASCAL Boundaries database, we propose a novel multi-scale deep network-based semantic boundary detector (M-DSBD). As an overview, our network takes an RGB image as input and outputs a prediction map that provides the confidence for the presence of a class-agnostic object boundary at each pixel location. To this end, we build upon the fully convolutional network (FCN) architecture [@long2014fully]. Our network architecture is shown in Figure \[fig:arch\]. M-DSBD works on multiple scales of input images, which is a common practice in many computer vision algorithms. Since the objects in our images occur at different scales, we try to provide invariance to it by *explicitly* detecting object boundaries at various scales during both the training *and* testing phases. Note that this is different from HED [@xie2015holistically], where the authors use multi-scale only while training the network. Combining the predictions from multiple scales of the same image allows the deep network model to be scale-invariant, thus leading to a more robust boundary detector (also corroborated by the results from our experiments).

More formally, for a given image, $\mathbf{x}$, the network rescales it to multiple scales, $\mathcal{S} \in \{1,2,...,|\mathcal{S}|\}$, to produce an image pyramid, $\{\mathbf{x}^s\}_{s=1}^{|\mathcal{S}|}$. Our network acts on each rescaled image in this image pyramid, $\mathbf{x}^s$, and outputs a class-agnostic boundary map for each scale, $\hat{\mathbf{y}}^s(=\sigma(\hat{\mathbf{y}}^s_a))$. The final boundary prediction, $\hat{\mathbf{y}}$, involves taking a linear combination of the scale-specific boundary prediction activations, $\hat{\mathbf{y}}^s_a$, $$\hat{\mathbf{y}}(i) = \sigma(\sum_{s=1}^{|\mathcal{S}|} w^s_{scale} \hat{\mathbf{y}}^{s}_a(i)).
\vspace{-1ex}$$ Here, $i$ is used to index the pixel locations in the image and $w^s_{scale}$ is the linear combination weight associated with scale $s$, which can be vectorized and written as $\mathbf{w}_{scale}$, and $\sigma(.)$ is used to denote the sigmoid function that maps the boundary prediction activations into the range $[0,1]$. Each scale-specific boundary prediction, $\hat{\mathbf{y}}^s$, is obtained by passing the rescaled image, $\mathbf{x}^{s}$, though a series of convolutional layers, rectified linear unit (ReLU) layers, and max-pooling layers. We use $CNN(\mathbf{x}^{s}; \mathbf{W}_{base}, \mathbf{w}^{s}_{side})$ to denote the processing done on the rescaled image, $\mathbf{x}^{s}$, by a convolutional neural network parameterized by two sets of weights, $\mathbf{W}_{base}$ and $\mathbf{w}^{s}_{side}$, to produce the scale-specific boundary map, $$\hat{\mathbf{y}}^s = CNN(\mathbf{x}^{s}; \mathbf{W}_{base}, \mathbf{w}^{s}_{side}).$$ Note that $\mathbf{W}_{base}$ is independent of the scale of the image and is shared across all image scales, and $\mathbf{w}^{s}_{side}$ denotes the scale-specific weights. We will explain both these weights in more detail, shortly.

Recently, various works have shown that a boost in performance is achievable by using features from the intermediate layers of the deep network [@long2014fully; @hariharan2014hypercolumns; @xie2015holistically]. M-DSBD also uses features from the intermediate layers of the base network, which we combine using a linear combination to produce a scale-specific boundary prediction map. Let, $\mathbf{f}^{(s,k)}(i) \in \mathbb{R}^{d_k}$ ($d_k$ is the number of convolutional filters in layer $k$) denote the feature vector at a spatial location, $i$, obtained from an intermediate layer, $k$, and, let the subset of the weights of the base network ($\mathbf{W}_{base}$) that are used to produce the features, $\mathbf{f}^{(.,k)}$, be denoted as $\mathbf{W}^{1:k}_{base}$. We fuse these features into a 1-channel feature map, $\mathbf{f}^{(s,k)}_{side}$, which can be extracted at the side of each intermediate layer, $k$, using a $1\times1$ convolution kernel, i.e., $$\mathbf{f}^{(s,k)}_{side}(i) = {\mathbf{w}^{(s,k)}_{feat}}^{\top} \mathbf{f}^{(s,k)}(i)
\vspace{-1ex}$$ where, $\mathbf{f}^{(s,k)}_{side}(i) \in \mathbb{R}$ is the 1-channel feature at the spatial location, $i$, and $\mathbf{w}^{(s,k)}_{feat}$ are the linear weights used to combine the intermediate layer features.

Due to the max-pooling at end of the intermediate layers, the spatial resolution of the side-output features, $\mathbf{f}^{(s,k)}_{side}$, will not be the same as the spatial resolution of the image, $\mathbf{x}^{s}$. So, we upsample the side-output features, using a deconvolution layer with an appropriately sized kernel, $\mathbf{w}^{(s,k)}_{up}$, before taking a linear combination of these side output features to produce the scale-specific boundary prediction activation, $$\hat{\mathbf{y}}^s_a(i) = \sum_{k=1}^{K}{w}^{(s,k)}_{fuse} \mathbf{f}^{(s,k)}_{(side, up)}(i).
\vspace{-1ex}$$ Here, $\mathbf{f}^{(s,k)}_{(side, up)} = UP(\mathbf{f}^{(s,k)}_{side}; \mathbf{w}^{(s,k)}_{up})$ is the upsampled feature map, $\mathbf{w}^{(s,k)}_{up}$ are the weights corresponding to the interpolation kernel, and ${w}^{(s,k)}_{fuse} \in \mathbb{R}$ is the weight associated with the $k$-th layer side output for performing the linear fusion. We combine all linear fusion weights into a vector notation, $\mathbf{w}^{s}_{fuse} \in \mathbb{R}^K$, where K is the total number of layers in the deep network. We group all the side-output weights and denote the set as $\mathbf{w}^{s}_{side} = \{\mathbf{w}^{(s,k)}_{feat}\}_{k=1}^{K} \bigcup \{\mathbf{w}^{(s,k)}_{up}\}_{k=1}^{K} \bigcup \{\mathbf{w}^{s}_{fuse}\}$.

We initialize the base network weights, $\mathbf{W}_{base}$, from the five convolutional layers of the VGG16 network [@simonyan2014very], which was pretrained on the ImageNet database. We encourage the reader to refer to [@simonyan2014very] for the architecture of the base network. From our experiments, we found that augmenting the VGG16 convolutional weights, with an additional convolutional layer (`conv5_4`), improved the performance of the boundary detection task. Therefore, our base network architecture consists of the original convolutional layers from the VGG16 architecture and an additional convolutional layer, `conv5_4`, which consists of 512 filters of size $3\times3$. The weights for this new `conv5_4` layer were initialized randomly by drawing from a Gaussian distribution.

Training Procedure
------------------

We now describe the training procedure that was employed to train the weights in our deep network. As mentioned above, we build on the Fully Convolutional Network architecture, which allows us to backpropagate the gradients computed at each pixel location.

Our training set consists of the image-boundary label pairs, $\mathcal{D} = \{(\mathbf{x}_1,\mathbf{y}_1), (\mathbf{x}_2,\mathbf{y}_2),...,(\mathbf{x}_{|\mathcal{D}|},\mathbf{y}_{|\mathcal{D}|})\}$, where $\mathbf{x}_i$’s are the images and $\mathbf{y}_i$’s are the boundary labels. We employ batch-stochastic gradient descent to update the initialized weights. We make use of a layer-by-layer deep supervision [@lee2014deeply] to warm-start the training procedure. We greedily update the weights [@bengio2007greedy] in each layer by backpropagating the gradients from a side-output loss, $\Delta_k(\mathbf{y}, \hat{\mathbf{y}}^k)$, which is computed between the side output, $\hat{\mathbf{y}}^k(=\sigma(\mathbf{f}^{(s,k)}_{(side, up)}))$, obtained from the intermediate features out of layer $k$, and the ground truth boundary map, $\mathbf{y}$. The side-output loss is the sum of the weighted cross-entropy loss at each pixel location, i.e., $$\begin{aligned}
\Delta_k(\mathbf{y}, \hat{\mathbf{y}}^k) = - \beta \sum_{j \in \{i | \mathbf{y}(i) = 1\}} \log P(\hat{\mathbf{y}}^{k}(j)=1|\mathbf{x}; \mathbf{W}_{(\Delta,k)}) \\
- (1-\beta) \sum_{j \in \{i | \mathbf{y}(i) = 0\}} \log P(\hat{\mathbf{y}}^{k}(j)=0|\mathbf{x}; \mathbf{W}_{(\Delta,k)}),
\end{aligned}
\label{eq:side-loss}
\vspace{-1ex}$$ where $\mathbf{W}_{(\Delta,k)} = \{\mathbf{W}^{1:k}_{base}\} \bigcup \{\mathbf{w}^{(s,k)}_{fuse}\} \bigcup \{\mathbf{w}^{(s,k)}_{up}\}$, and $\beta$ is the class-balancing weight. Class-balancing is needed because of the severe imbalance in between the number of boundary pixels and non-boundary pixels. We fixed $\beta = 0.9$, which we found to work well in our experiments.

The layer-by-layer deep supervision procedure uses a side-output loss, $\Delta_k(\mathbf{y}, \hat{\mathbf{y}}^k)$, to update only the weights corresponding to that layer. The weights of all other layers are not changed. For example, while backpropagating from $\Delta_k(\mathbf{y}, \hat{\mathbf{y}}^k)$, only the weights, $\{\mathbf{W}^{k}_{base}\} \bigcup \{\mathbf{w}^{(s,k)}_{fuse}\} \bigcup \{\mathbf{w}^{(s,k)}_{up}\}$ are updated; $\mathbf{W}^{k}_{base}$ corresponds to the weights in the $k$-th layer of the base network. The rest of the weights are untouched. We sequentially update the weights in each layer starting from layer $1$ and ending at layer $K$.

Once the weights have been fine-tuned using our greedy layer-by-layer update procedure, we switch off the side-output losses and finetune the network using a scale-specific boundary detection loss, $$\begin{aligned}
\Delta_s(\mathbf{y}, \hat{\mathbf{y}}^s) = - \beta \sum_{j \in \{i | \mathbf{y}(i) = 1\}} \log P(\hat{\mathbf{y}}^{s}(j)=1|\mathbf{x}; \mathbf{W}_{(\Delta,s)})\\
- (1-\beta) \sum_{j \in \{i | \mathbf{y}(i) = 0\}} \log P(\hat{\mathbf{y}}^{s}(j)=0|\mathbf{x}; \mathbf{W}_{(\Delta,s)}),
\end{aligned}
\label{eq:scale-scecific-loss}
\vspace{-1ex}$$ where $\mathbf{W}_{(\Delta,s)} = \{\mathbf{W}_{base}\} \bigcup \{\mathbf{w}^{s}_{side}\}$. This is different from the training procedure in [@xie2015holistically], where the authors employ deep supervision and force each side-output prediction to be a boundary map. We, on the other hand, only use deep supervision to warm-start the training procedure and switch off the gradients from the side-output loss while updating the fusion weights. In other words, we *do not* enforce each side output to correspond to a boundary prediction map, but use these side outputs as features for the scale-specific boundary map. Enforcing each side output to be a boundary predictor of its own right prevents the fusion layer from providing the best performance. Allowing the side outputs to only act as features for the fusion layer, by switching off the gradients from the side-output loss, enables a layer’s features to be complementary to other layers’ features, thus permitting the fusion weights to extract the best possible performance.

All that is left is to learn the optimal weights to fuse the various scale-specific predictions. To this end, we define the final boundary detection loss, $\Delta_b(\mathbf{y}, \hat{\mathbf{y}})$ as, $$\begin{aligned}
\Delta_b(\mathbf{y}, \hat{\mathbf{y}}) = - \beta \sum_{j \in \{i | \mathbf{y}(i) = 1\}} \log P(\hat{\mathbf{y}}(j)=1|\mathbf{x}; \mathbf{W}_{(\Delta,b)}) \\
- (1-\beta) \sum_{j \in \{i | \mathbf{y}(i) = 0\}} \log P(\hat{\mathbf{y}}(j)=0|\mathbf{x}; \mathbf{W}_{(\Delta,b)}),
\end{aligned}
\label{eq:boundary-loss}
\vspace{-1ex}$$ where $\mathbf{W}_{(\Delta,b)} = \{\mathbf{W}_{base}\} \bigcup \{\mathbf{w}^{s}_{side}\}_{s=1}^{|\mathcal{S}|} \bigcup \{\mathbf{w}_{scale}\}$. In this final stage of learning, we switched off the gradients from the side-output losses and the scale-specific losses, and backpropagated the gradients only from the boundary detection loss. Moreover, the base network weights $\mathbf{W}_{base}$ were not updated during this final stage, and only the side-output weights, $\{\mathbf{w}^{s}_{side}\}_{s=1}^{|\mathcal{S}|}$, and the scale-fusion weights, $\mathbf{w}_{scale}$, were updated.

Experiments {#sec:exp}
===========

We predominantly experimented on the newly collected PASCAL Boundaries dataset that was described in Section \[sec:dataset\]. This database consists of images from the `trainval` set of the PASCAL VOC2010 challenge. There are a total of 10,103 images that have been labeled. We train our deep network on the `train` set of the dataset and test on the `test` set. Note that since we label only the images from the `trainval` set of the PASCAL VOC2010 challenge, the `test` set of the PASCAL Boundaries dataset corresponds to the `val` set of the PASCAL VOC2010 challenge.

**Implementation details:** We used the publicly available FCN code [@long2014fully], which is built on top of the Caffe framework to train our deep network. We modified the `sigmoid_cross_entropy_loss_layer` to compute the weighted cross entropy loss. In addition, we provide functionalities within the Caffe framework that resizes (downsample and upsample) data blobs to arbitrary resize factors[^1]. Weight updates were performed using batch-SGD with a batch size of $5$ images. To enable batch training on a GPU, we resized all images from the `train` set to a standard resolution of $400\times400$. The learning rate was fixed to 1e-7, and weight decay was set to 0.0002. We did not augment our training data since the PASCAL Boundaries dataset has $\sim5000$ training images.

**Evaluation Criterion:** The standard evaluation criterion for evaluating edge detection algorithms is the F-score. We also use the same evaluation criterion for evaluating our boundary detector. In addition, we provide baselines on the new dataset using two other well-known edge detection algorithms; SE [@dollar2014fast] and HED [@xie2015holistically]. We use the helper evaluation functions provided in the SE Detection Toolbox [@dollar_edges_code] to obtain all the numbers we report in this paper.

Transfer from Edge Detection
----------------------------

We tested the baseline edge detection methods, SE [@dollar2014fast] and HED [@xie2015holistically], on the 5105 images present in the `test` set of the PASCAL Boundaries dataset, and Figure \[fig:pasal-plot\] shows the precision/recall curves. A more detailed, and exhaustive comparison is provided in Table \[tab:pascal\]. SE and HED models were trained on the BSDS dataset and were released by the respective authors. To make this explicit, we call them SE-BSDS and HED-BSDS, respectively.

We see that both SE-BSDS and HED-BSDS transfer reasonably on to the PASCAL Boundaries dataset; SE-BSDS achieves an F-score of 0.541, while HED-BSDS achieves an F-score of 0.553. The ranking order of SE’s and HED’s performance when tested on the BSDS500 dataset also transfers over when tested on the PASCAL Boundaries dataset. This shows that BSDS500 edges are not entirely different from our definition of segment boundaries. The BSDS500 boundaries constitute object-level boundaries, object part-level boundaries, and boundaries emanating texture. Our database, in comparison, deals only with object-level boundaries.

**Retraining HED on PASCAL Boundaries:** To provide a fair comparison, we tried training HED using their publicly-released training code. We retained all the parameters that were set by the authors. We only replaced the training set from the BSDS500’s augmented training set (which HED uses) to PASCAL Boundaries’ `train` set. To account for an increase in the complexity of the PASCAL Boundaries dataset (in comparison to the BSDS500 dataset), we trained HED for a total of 100k iterations (as opposed to the 10k terations that the authors report in [@xie2015holistically]). We snapshotted the model every 1000 iterations and used a validation set of $25$ images (randomly chosen from the `train` set) to select the best model. Surprisingly, we found the performance of the best model when tested on the PASCAL Boundaries’ `test` set to be worse by several $\%$ points compared to the performance obtained using their released model (HED-BSDS)[^2]. We believe that the optimal parameters set by the authors of HED to train on the BSDS500 dataset might not be the optimal parameters to train on the PASCAL Boundaries dataset. We did not experiment with different parameter settings.

![image](img/results/12.jpg){width="0.19\linewidth"} ![image](img/results/10.jpg){width="0.19\linewidth"} ![image](img/results/13.jpg){width="0.095\linewidth"} ![image](img/results/4.jpg){width="0.215\linewidth"} ![image](img/results/15.jpg){width="0.19\linewidth"} ![image](img/results/7.jpg){width="0.088\linewidth"}\
![image](img/results/context_gt_12.pdf){width="0.19\linewidth"} ![image](img/results/context_gt_10.pdf){width="0.19\linewidth"} ![image](img/results/context_gt_13.pdf){width="0.095\linewidth"} ![image](img/results/context_gt_4.pdf){width="0.215\linewidth"} ![image](img/results/context_gt_15.pdf){width="0.19\linewidth"} ![image](img/results/context_gt_7.pdf){width="0.088\linewidth"}\
![image](img/results/bdr_gt_12.png){width="0.19\linewidth"} ![image](img/results/bdr_gt_10.png){width="0.19\linewidth"} ![image](img/results/bdr_gt_13.png){width="0.095\linewidth"} ![image](img/results/bdr_gt_4.png){width="0.215\linewidth"} ![image](img/results/bdr_gt_15.png){width="0.19\linewidth"} ![image](img/results/bdr_gt_7.png){width="0.088\linewidth"}\
![image](img/results/dollar_12.png){width="0.19\linewidth"} ![image](img/results/dollar_10.png){width="0.19\linewidth"} ![image](img/results/dollar_13.png){width="0.095\linewidth"} ![image](img/results/dollar_4.png){width="0.215\linewidth"} ![image](img/results/dollar_15.png){width="0.19\linewidth"} ![image](img/results/dollar_7.png){width="0.088\linewidth"}\
![image](img/results/hed_12.png){width="0.19\linewidth"} ![image](img/results/hed_10.png){width="0.19\linewidth"} ![image](img/results/hed_13.png){width="0.095\linewidth"} ![image](img/results/hed_4.png){width="0.215\linewidth"} ![image](img/results/hed_15.png){width="0.19\linewidth"} ![image](img/results/hed_7.png){width="0.088\linewidth"}\
![image](img/results/ours_12.png){width="0.19\linewidth"} ![image](img/results/ours_10.png){width="0.19\linewidth"} ![image](img/results/ours_13.png){width="0.095\linewidth"} ![image](img/results/ours_4.png){width="0.215\linewidth"} ![image](img/results/ours_15.png){width="0.19\linewidth"} ![image](img/results/ours_7.png){width="0.088\linewidth"}

Accretion Study and (M-)DSBD Results
------------------------------------

We present our results in a step-by-step modular fashion to show the improvements that the respective components in our model provide.

**Training Strategy:** To test our training strategy, we replaced HED’s training method with greedy layer-by-layer training strategy to warm-start the training process. We then used these updated weights as an initialization to train the HED architecture by backpropagating the gradients from the losses computed at each of the five side-output predictions and the fusion layer prediction, simultaneously, as was done in [@xie2015holistically]. This approach of training the HED architecture provided an improvement of $3\%$ over the results that were obtained while testing with the publicly-released pre-trained model; we were able to obtain an F-score of 0.59. Since this method uses the HED architecture but a different training strategy (i.e., the greedy layer-wise training), we use the term ‘HED-arch-greedy’ to indicate this model.

**More Convolutional Layers:** Since the PASCAL Boundaries dataset is more complex than the BSDS dataset, we experimented with adding more layers to the models so that it could capture the dataset’s complexity. We began by adding an additional convolution layer, `conv5_4`. We built the layer `conv5_4` with $512$ filters, each with a kernel size of $3\times3$. We also added a ReLU layer to rectify the output of `conv5_4`. This enhanced architecture was able to further improve the results by $3\%$ over the previous model by producing an F-score of 0.62 on the `test` set. We experimented with adding more layers to the network, but found that they did not improve the performance of the model. We use the term ‘HED-arch+`conv5_4`-greedy’ for this model.

**Switching deep supervision off:** An interesting outcome was observed when we used deep supervision just to warm-start the training process. Upon completion of the greedy layer-by-layer training process, we switched off the backpropagation of the gradient from the side-output losses (Eq. \[eq:side-loss\]) and backpropagated only from the scale-specific boundary detection loss[^3] (Eq. \[eq:scale-scecific-loss\]). Doing so, improved the performance of the model by another $2\%$. We call this version as the single scale Deep Semantic Boundary Detector (DSBD). We believe that the improvement in performance was achievable because we no longer force the side-output predictions to be boundary detectors of their own right, but use them as features for the fusion layer. That said, we do acknowledge the importance of deep supervision for warm-starting the training process.

**Multi-scale Boundary Detection:** Finally, we experimented with the M-DSBD architecture that was described in Section \[sec:multiscale-deepNetwork\]. We used three scales, $\mathcal{S} = \{1, 0.8, 0.5\}$, for training and testing. The base network weights were not updated at this stage. Only the scale-specific side output weights, and the multi-scale fusion weights were updated during this final training procedure. The gradients were backpropagated from the boundary detection loss (Eq. \[eq:boundary-loss\]). Our experiments supported our hypothesis that multi-scale processing would improve the task of boundary detection by providing a further improvement of $1\%$ on the `test` set of the PASCAL Boundaries dataset. Our model and training procedure produced a final F-score of 0.652, which is significantly more than the other baselines.

We tabulate all the numbers described above in Table \[tab:pascal\]. ‘BSDS’ is used to indicate that the model was trained on the BSDS500 dataset. We also show some qualitative results in Fig. \[fig:qualitative\]. Notice that our boundary detector is capable of identifying the semantic boundaries confidently and detects far less number of internal edges. On the other hand, the edge detectors identify edges across various levels of granularity (which they were trained to detect).

               Method                   ODS         OIS         AP
  --------------------------------- ----------- ----------- -----------
      SE-BSDS [@dollar2014fast]        0.541       0.570       0.486
   HED-BSDS [@xie2015holistically]     0.553       0.585       0.518
           HED-arch-greedy             0.59          -           -
      HED-arch+`conv5_4`-greedy        0.62          -           -
                DSBD                   0.643       0.663       0.650
               M-DSBD                **0.652**   **0.678**   **0.674**

  : Results on the PASCAL Boundaries dataset. SE’s and HED’s results are from the models that were trained on the BSDS500. The results from M-DSBD shows that multi-scale does improve performance over single scale.[]{data-label="tab:pascal"}

**BSDS500:** For completeness, we report the performance of M-DSBD on the BSDS500 dataset. Table \[tab:bsds\] tabulates the results. Note that M-DSBD was trained on the PASCAL Boundaries’ `train` set, but tested on the BSDS500’s `test` set. The numbers show that our model transfers to a different dataset while producing competitive results. Fig. \[fig:bsds\] shows an example image from the BSDS500 dataset along with the edge and boundary detections. We can see from the figure M-DSBD transfers on to the BSDS500 dataset and is successful in providing high confidence for object boundaries and low confidence for internal edges.

             Method              ODS     OIS     AP
  ---------------------------- ------- ------- -------
      SE [@dollar2014fast]      0.746   0.767   0.803
   HED [@xie2015holistically]   0.782   0.804   0.833
        M-DSBD-PASCAL-B         0.751   0.773   0.789

  : Results on the BSDS500 dataset. The M-DSBD model was trained on the PASCAL Boundaries dataset. The results show that methods trained on a boundary detection task perform fairly well on the edge detection task.[]{data-label="tab:bsds"}

\

Conclusion and Future Work {#sec:conclusion}
==========================

In this paper, we pointed to the ambiguity in the definition of edge detection, and, defined a precise task, namely class-agnostic boundary detection. To facilitate progress in solving this problem, we release a large dataset of $\sim10$k images with labeled boundaries, which is 20 times bigger than the widely-used BSDS500 dataset, and without any ambiguity in the annotations. In addition, we proposed a novel multi-scale deep semantic boundary detector and showed that it performs well on the boundary detection task.

We now conclude the paper by pointing to various new research directions that can emerge out of this dataset. Firstly, since boundaries are complementary to pixel-level semantic labeling, it would be interesting to develop joint techniques that can exploit the advantages of each of these respective tasks. Secondly, state-of-the-art object proposal generators are based on edge grouping. It will be interesting to study the effect that instance-level semantic boundary predictions have on object proposals. And, finally, this dataset allows easy access to regions of occlusions because of the presence of occlusion cues (triple points). This dataset provides a good starting point to work on the hard task of occlusion-handling.

[^1]: Source code will be released for public use.

[^2]: We obtained an F-score of 0.36 when we trained HED using the authors’ training code on the PASCAL Boundaries datset. Moreover, we also noticed a drop in performance when we tried replicating HED’s results on the BSDS500 dataset.

[^3]: Please note that the above experiment was done on a single scale. When we use the term “scale-specific loss", the gradients were backpropagated from the loss computed using the original-sized images.
