benchmark and it is efﬁcient to run.
Additionally, we show that due to the semantic nature of
our boundaries we can use them to aid a number of highlevel vision tasks. We demonstrate that by using our boundaries we improve the performance of state-of-the-art methods on the problems of semantic boundary labeling, semantic segmentation and object proposal generation. We can
view this process as a Low-for-High scheme, where lowlevel boundaries aid high-level vision tasks.
Thus, our contributions include a boundary detection
system that is accurate, efﬁcient, generalizes well to multiple datasets, and is also shown to improve existing stateof-the-art high-level vision methods on three distinct tasks.
1. Introduction
In the vision community, boundary detection has always
been considered a low-level problem. However, psychological studies suggest that when a human observer perceives
boundaries, object level reasoning is used [13, 25, 17]. Despite these ﬁndings, most of the boundary detection methods rely exclusively on low-level color and gradient features. In this work, we present a method that uses objectlevel features to detect boundaries. We argue that using
object-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks
BD SBL SS OP
ODS MF AP PI-IOU MR


object-level information to predict boundaries is more sim-Low-Level Task High-Level Tasks
BD SBL SS OP
ODS MF AP PI-IOU MR
SotA 0.76 [27] 28.0 [11] 19.9 [11] 45.8 [5] 0.88 [31]
HFL 0.77 62.5 54.6 48.8 0.90
Table 1: Summary of results achieved by our proposed
method (H FL) and state-of-the-art methods (SotA). We provide results on four vision tasks: Boundary Detection (BD),
Semantic Boundary Labeling (SBL), Semantic Segmentation (SS), and Object Proposal (OP). The evaluation metrics
include ODS F-score for BD task, max F-score (MF) and
average precision (AP) for SBL task, per image intersection over union (PI-IOU) for SS task, and max recall (MR)
for OP task. Our method produces better results in each of
these tasks according to these metrics.
ilar to how humans reason. Our boundary detection scheme
can be viewed as a High-for-Low approach where we use
high-level object features as cues for a low-level boundary
detection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries

High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision

Summary:
Most of the current boundary detection systems rely exclusively on low-level
features, such as color and texture. However, perception studies suggest that
humans employ object-level reasoning when judging if a particular pixel is a
boundary. Inspired by this observation, in this work we show how to predict
boundaries by exploiting object-level features from a pretrained
object-classification network. Our method can be viewed as a "High-for-Low"
approach where high-level object features inform the low-level boundary
detection process. Our model achieves state-of-the-art performance on an
established boundary detection benchmark and it is efficient to run.
  Additionally, we show that due to the semantic nature of our boundaries we
can use them to aid a number of high-level vision tasks. We demonstrate that
using our boundaries we improve the performance of state-of-the-art methods on
the problems of semantic boundary labeling, semantic segmentation and object
proposal generation. We can view this process as a "Low-for-High" scheme, where
low-level boundaries aid high-level vision tasks.
  Thus, our contributions include a boundary detection system that is accurate,
efficient, generalizes well to multiple datasets, and is also shown to improve
existing state-of-the-art high-level vision methods on three distinct tasks.

Gedas Bertasius
Jianbo Shi
Lorenzo Torresani


high-level object features as cues for a low-level boundary
detection process. Throughout the rest of the paper, we refer to our proposed boundaries as High-for-Low boundaries
(HFL).
We present an efﬁcient deep network that uses objectlevel information to predict the boundaries. Our proposed
architecture reuses features from the sixteen convolutional
layers of the network of Simonyan et al. [29], which we
refer to as VGG net. The VGG net has been trained for
object classiﬁcation, and therefore, reusing its features allows our method to utilize high-level object information to
predict H FL boundaries. In the experimental section, we
demonstrate that using object-level features produces semantically meaningful boundaries and also achieves above
state-of-the-art boundary detection accuracy.
Additionally, we demonstrate that we can successfully
apply our H FL boundaries to a number of high-level vision
tasks. We show that by using H FL boundaries we improve
the results of three existing state-of-the-art methods on the
tasks of semantic boundary labeling, semantic segmenta1arXiv:1504.06201v3  [cs.CV]  21 Sep 2015
tion and object proposal generation. Therefore, using H FL
boundaries to boost the results in high level vision tasks


tion and object proposal generation. Therefore, using H FL
boundaries to boost the results in high level vision tasks
can be viewed as a Low-for-High scheme, where boundaries
serve as low-level cues to aid high-level vision tasks.
We present the summarized results for the boundary detection and the three mentioned high-level vision tasks in
Table 1. Speciﬁcally, we compare our proposed method and
an appropriate state-of-the-art method for that task. As the
results indicate, we achieve better results in each of the tasks
for each presented evaluation metric. We present more detailed results for each of these tasks in the later sections.
In summary, our contributions are as follows. First, we
show that using object-level features for boundary detection produces perceptually informative boundaries that outperform prior state-of-the-art boundary detection methods.
Second, we demonstrate that we can use H FL boundaries
to enhance the performance on the high-level vision tasks
of semantic boundary labeling, semantic segmentation and
object proposal. Finally, our method can detect boundaries
in near-real time. Thus, we present a boundary detection
system that is accurate, efﬁcient, and is also applicable to
high level vision tasks.
2. Related Work
Most of the contour detection methods predict boundaries based purely on color, text, or other low-level features.


high level vision tasks.
2. Related Work
Most of the contour detection methods predict boundaries based purely on color, text, or other low-level features.
We can divide these methods into three broad categories:
spectral methods, supervised discriminative methods and
deep learning based methods.
Spectral methods formulate contour detection problem
as an eigenvalue problem. The solution to this problem is
then used to reason about the boundaries. The most successful approaches in this genre are the MCG detector [2], gPb
detector [1], PMI detector [14], and Normalized Cuts [28].
Some of the notable discriminative boundary detection
methods include sketch tokens (ST) [18], structured edges
(SE) [6] and sparse code gradients (SCG) [23]. While SCG
use supervised SVM learning [4], the latter two methods
rely on a random forest classiﬁer and models boundary detection as a classiﬁcation task.
Recently there have been attempts to apply deep learning
to the task of boundary detection. SCT [20] is a sparse coding approach that reconstructs an image using a learned dictionary and then detect boundaries. Both N4ﬁelds [10] and


DeepNet [16] approaches use Convolutional Neural Networks (CNNs) to predict edges. N4ﬁelds rely on dictionary learning and the use of the Nearest Neighbor algorithm
within a CNN framework while DeepNet uses a traditional
CNN architecture to predict contours.
The most similar to our approach is DeepEdge [3],
which uses a multi-scale bifurcated network to perform contour detection using object-level features. However, we
show that our method achieves better results even without
1100x1100
512
500x375...
1024
Input Image
5504 Convolutional     Feature Maps 500x375Upsampled     Image
5504-Dimensional    Feature Vectors  Predicted  Boundaries
|{z}
Feature Interpolation & Concatenation
1100x1100x64- Candidate Contour Points
35x35x512
Shared Weights for all     Candidate Points
|{z}
Figure 1: An illustration of our architecture (best viewed in
color). First we extract a set of candidate contour points.
Then we upsample the image and feed it through 16convolutional layers pretrained for object classiﬁcation. For each
candidate point, we ﬁnd its correspondence in each of the
feature maps and perform feature interpolation. This yields


candidate point, we ﬁnd its correspondence in each of the
feature maps and perform feature interpolation. This yields
a5504 -dimensional feature vector for each candidate point.
We feed each of these vectors to two fully connected layers
and store the predictions to produce a ﬁnal boundary map.
the complicated multi-scale and bifurcated architecture of
DeepEdge. Additionally, unlike DeepEdge, our system can
run in near-real time.
In comparison to prior approaches, we offer several contributions. First, we propose to use object-level information to predict boundaries. We argue that such an approach
leads to semantic boundaries, which are more consistent
with humans reasoning. Second, we avoid feature engineering by learning boundaries from human-annotated data.
Finally, we demonstrate excellent results for both low-level
and high-level vision tasks. For the boundary detection task,
our proposed H FL boundaries outperform all of the prior
methods according to both F-score metrics. Additionally,
we show that because H FL boundaries are based on objectlevel features, they can be used to improve performance in
the high-level vision tasks of semantic boundary labeling,
semantic segmentation, and object proposal generation.
3. Boundary Detection
In this section, we describe our proposed architecture
and the speciﬁc details on how we predict H FL boundaries


3. Boundary Detection
In this section, we describe our proposed architecture
and the speciﬁc details on how we predict H FL boundaries
using our method. The detailed illustration of our architecture is presented in Figure 1.
Figure 2: A visualization of selected convolutional feature maps from VGG network (resized to the input image dimension).
Because VGG was optimized for an object classiﬁcation task, it produces high activation values on objects and their parts.
3.1. Selection of Candidate Contour Points
We ﬁrst extract a set of candidate contour points with
a high recall. Due to its efﬁciency and high recall performance, we use the SE edge detector [6]. In practice, we
could eliminate this step and simply try to predict boundaries at every pixel. However, selecting a set of initial candidate contour points, greatly reduces the computational cost.
Since our goal is to build a boundary detector that is both accurate and efﬁcient, we use these candidate points to speed
up the computation of our method.
3.2. Object-Level Features
After selecting candidate contour points, we up-sample
the original input image to a larger dimension (for example
11001100 ). The up-sampling is done to minimize the loss
of information due to the input shrinkage caused by pooling


the original input image to a larger dimension (for example
11001100 ). The up-sampling is done to minimize the loss
of information due to the input shrinkage caused by pooling
at the different layers. Afterwards, we feed the up-sampled
image through 16convolutional layers of the VGG net.
We use the VGG net as our model because it has been
trained to recognize a large number of object classes (the
1000 categories of the ImageNet dataset [24]) and thus encodes object-level features that apply to many classes. To
preserve speciﬁc location information we utilize only the
16convolutional layers of the VGG net. We don’t use fully
connected layers because they don’t preserve spatial information, which is crucial for accurate boundary detection.
We visualize some of the selected convolutional maps in
Figure 2. Note the high activation values around the various objects in the images, which conﬁrms our hypothesis
that the VGG net encodes object speciﬁc information in its
convolutional feature maps.
3.3. Feature Interpolation
Similarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes


3.3. Feature Interpolation
Similarly to [26, 12, 19], we perform feature interpolation in deep layers. After the up-sampled image passes
through all 16convolutional layers, for each selected candidate contour point we ﬁnd its corresponding point in the
feature maps. Due to the dimension differences in convo-lutional maps these correspondences are not exact. Thus
we perform feature interpolation by ﬁnding the four nearest
points and averaging their activation values. This is done
in each of the 5504 feature maps. Thus, this results in a
5504 -dimensional vector for each candidate point.
We note that the interpolation of convolutional feature
maps is the crucial component that enables our system to
predict the boundaries efﬁciently. Without feature interpolation, our method would need to independently process
the candidate edge points by analyzing a small image patch
around each point, as for example done in DeepEdge [3]
which feeds one patch at a time through a deep network.
However, when the number of candidate points is large
(e.g., DeepEdge considers about 15K points at each of 4 different scales), their patches overlap signiﬁcantly and thus a
large amount of computation is wasted by recalculating ﬁlter response values over the same pixels. Instead, we can


large amount of computation is wasted by recalculating ﬁlter response values over the same pixels. Instead, we can
compute the features for all candidate points with a single
pass through the network by performing deep convolution
over the entire image (i.e., feeding the entire image rather
than one patch at a time) and then by interpolating the convolutional feature maps at the location of each candidate
edge point so as to produce its feature descriptor. Thanks to
this speedup, our method has a runtime of 1:2seconds (using a K40 GPU), which is better than the runtimes of prior
deep-learning based edge detection methods [27, 10, 16, 3].
3.4. Learning to Predict Boundaries
After performing feature interpolation, we feed the
5504 -dimensional feature vectors corresponding to each of
the candidate contour points to two fully connected layers
that are optimized to the human agreement criterion. To be
more precise, we deﬁne our prediction objective as a fraction of human annotators agreeing on the presence of the
boundary at a particular pixel. Therefore, a learning objective aims at mimicking the judgement of the human labelers.
Finally, to detect H FL boundaries, we accumulate the
predictions from the fully connected layers for each of the
candidate points and produce a boundary probability map
as illustrated in Figure 1.


Finally, to detect H FL boundaries, we accumulate the
predictions from the fully connected layers for each of the
candidate points and produce a boundary probability map
as illustrated in Figure 1.
3.5. Implementation Details
In this section, we describe the details behind the training
procedure of our model. We use the Caffe library [15] to
implement our network architecture.
In the training stage, we freeze the weights in all of the
convolutional layers. To learn the weights in the two fully
connected layers we train our model to optimize the least
squares error of the regression criterion that we described
in the previous subsection. To enforce regularization we set
a dropout rate of 0:5in the fully connected layers.
Our training dataset includes 80Kpoints from the
BSDS500 dataset [22]. As described in the previous subsection, the labels represent the fraction of human annotators agreeing on the boundary presence. We divide the label space into four quartiles, and select an equal number of
samples for each quartile to balance the training dataset. In
addition to the training dataset, we also sample a hold-out
dataset of size 40;000. We use this for the hard-positive
mining [21] in order to reduce the number of false-negative
predictions.


dataset of size 40;000. We use this for the hard-positive
mining [21] in order to reduce the number of false-negative
predictions.
For the ﬁrst 25epochs we train the network on the original80;000training samples. After the ﬁrst 25epochs, we
test the network on the hold-out dataset and detect false negative predictions made by our network. We then augment
the original 80;000 training samples with the false negatives and the same number of randomly selected true negatives. For the remaining 25epochs, we train the network on
this augmented dataset.
3.6. Boundary Detection Results
In this section, we present our results on the BSDS500
dataset [22], which is the most established benchmark for
boundary detection. The quality of the predicted boundaries is evaluated using three standard measures: ﬁxed contour threshold (ODS), per-image best threshold (OIS), and
average precision (AP).
We compare our approach to the state-of-the-art based
on two different sets of BSDS500 ground truth boundaries.
First, we evaluate the accuracy by matching each of the
predicted boundary pixels with the ground truth boundaries
that were annotated by anyof the human annotators. This


First, we evaluate the accuracy by matching each of the
predicted boundary pixels with the ground truth boundaries
that were annotated by anyof the human annotators. This
set of ground truth boundaries is referred to as “any”. We
present the results for “any” ground truth boundaries in the
lower half of Table 2. As indicated by the results, H FL
boundaries outperform all the prior methods according to
both F-score measures.
Recently, there has been some criticism raised about
the procedure for boundary detection evaluation on the
BSDS500 dataset. One issue with the BSDS500 dataset
involves the so called “orphan” boundaries: the bound-Consensus GT ODS OIS AP FPS
SCG [23] 0.6 0.64 0.56 1/280
DeepNet [16] 0.61 0.64 0.61 1=5z
PMI [14] 0.61 0.68 0.56 1/900
DeepEdge [3] 0.62 0.64 0.64 1=1000z
N4-ﬁelds [10] 0.64 0.67 0.64 1=6z
HFL 0.65 0.68 0.67 5=6z
Any GT ODS OIS AP FPS


HFL 0.65 0.68 0.67 5=6z
Any GT ODS OIS AP FPS
SE [6] 0.75 0.77 0.80 2.5
MCG [2] 0.75 0.78 0.76 1/24
N4-ﬁelds [10] 0.75 0.77 0.78 1=6z
DeepEdge [3] 0.75 0.77 0.81 1=1000z
MSC [30] 0.76 0.78 0.79 DeepContour [27] 0.76 0.77 0.8 1=30z
HFL 0.77 0.79 0.8 5=6z
Table 2: Boundary detection results on BSDS500 benchmark. Upper half of the table illustrates the results for “consensus” ground-truth criterion while the lower half of the
table depicts the results for “any” ground-truth criterion. In
both cases, our method outperforms all prior methods according to both ODS (optimal dataset scale) and OIS (optimal image scale) metrics. We also report the run-time of
our method (zGPU time) in the FPS column (frames per
second), which shows that our algorithm is faster than prior
approaches based on deep learning [27, 10, 16, 3].


second), which shows that our algorithm is faster than prior
approaches based on deep learning [27, 10, 16, 3].
Figure 5: Qualitative results on the BSDS benchmark. The
ﬁrst column of images represent input images. The second
column illustrates SE [6], while the third column depicts
HFL boundaries. Notice that SE boundaries are predicted
with low conﬁdence if there is no signiﬁcant change in color
between the object and the background. Instead, because
our model is deﬁned in terms of object-level features, it can
predict object boundaries with high conﬁdence even if there
is no signiﬁcant color variation in the scene.
aries that are marked by only one or two human annotators. These “orphan” boundaries comprise around 30% of
BSDS500 dataset but most of them are considered uninformative. However, the standard evaluation benchmark rewards the methods that predict these boundaries. To resolve
500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500024x 10−3Weight Magnitude
Feature Map Number (Sorted in the Order from Early to Deep Layers)Figure 6: We train a linear regression model and visualize
its weight magnitudes in order to understand which features
are used most heavily in the boundary prediction (this linear


its weight magnitudes in order to understand which features
are used most heavily in the boundary prediction (this linear
regression is used only for the visualization purposes and
not for the accuracy analysis). Note how heavily weighted
features lie in the deepest layers of the network, i.e., the layers that are most closely associated with object information.
this issue we also evaluate our H FL boundaries on the so
called “consensus” set of ground truth boundaries. These
“consensus” boundaries involve only boundaries that are
marked by allof the human annotators and hence, are considered perceptually meaningful. In the upper half of Table 2, we present the results achieved by our method on the
“consensus” set of the ground truth boundaries. Our H FL
boundaries outperform or tie all the prior methods in each
of the three evaluation metrics, thus suggesting that H FL
boundaries are similar to the boundaries that humans annotated. We also report the runtimes in Table 2 and note that
our method runs faster than previous deep-learning based
edge detection systems [27, 10, 16, 3].
Our proposed model computes a highly nonlinear function of the 5504-dimensional feature vector of each candidate point. Thus, it is difﬁcult to assess which features are
used most heavily by our edge predictor. However, we can


used most heavily by our edge predictor. However, we can
gain a better insight by replacing the nonlinear function with
a simple linear model. In Fig. 6 we show the weight magnitudes of a simple linear regression model (we stress that
this linear model is used only for feature visualization purposes). From this Figure, we observe that many important
features are located in the deepest layers of the VGG network. As shown in [7], these layers encode high-level object information, which conﬁrms our hypothesis that highlevel information is useful for boundary detection.
Finally, we present some qualitative results achieved by
our method in Figure 5. These examples illustrate the effective advantage that H FL boundaries provide over another
state-of-the-art edge detection system, the SE system [6].
Speciﬁcally, observe the parts of the image where there is a
boundary that separates an object from the background but
where the color change is pretty small. Notice that because
the SE boundary detection is based on low-level color and
texture features, it captures these boundaries with very low
conﬁdence. In comparison, because H FL boundaries rely
on object-level features, it detects these boundaries with
high conﬁdence.4. High-Level Vision Applications
In this section, we describe our proposed Low-for-High


high conﬁdence.4. High-Level Vision Applications
In this section, we describe our proposed Low-for-High
pipeline: using low-level boundaries to aid a number of
high-level vision tasks. We focus on the tasks of semantic
boundary labeling, semantic segmentation and object proposal generation. We show that using H FL boundaries improves the performance of state-of-the-art methods in each
of these high-level vision tasks.
4.1. Semantic Boundary Labeling
The task of semantic boundary labeling requires not only
to predict the boundaries but also to associate a speciﬁc
object class to each of the boundaries. This implies that
given our predicted boundaries we also need to label them
with object class information. We approach this problem
by adopting the ideas from the recent work on Fully Convolutional Networks (FCN) [19]. Given an input image,
we concurrently feed it to our boundary-predicting network
(described in Section 3), and also through the FCN that was
pretrained for 20Pascal VOC classes and the background
class. While our proposed network produces H FL boundaries, the FCN model predicts class probabilities for each
of the pixels. We can then merge the two output maps as
follows. For a given boundary point we consider a 99


of the pixels. We can then merge the two output maps as
follows. For a given boundary point we consider a 99
grid around that point from each of the 21FCN object-class
probability maps. We calculate the maximum value inside
each grid, and then label the boundary at a given pixel with
the object-class that corresponds to the maximum probability across these 21maps. We apply this procedure for each
of the boundary points, in order to associate object-class
labels to the boundaries. Note that we consider the grids
around the boundary pixel because the output of the FCN
has a poor localization, and considering the grids rather than
individual pixels leads to higher accuracy.
We can also merge H FL boundaries with the state-ofthe-art DeepLab-CRF segmentation [5] to obtain higher accuracy. We do this in a similar fashion as just described.
First, around a given boundary point we extract a 99
grid from the DeepLab-CRF segmentation. We then compute the mode value in the grid (excluding the background
class), and use the object-class corresponding to the mode
value as a label for the given boundary point. We do this for
each of the boundary points. By merging H FL boundaries
and the output of FCN or DeepLab-CRF, we get semantic


each of the boundary points. By merging H FL boundaries
and the output of FCN or DeepLab-CRF, we get semantic
boundaries that are highly localized and also contain objectspeciﬁc information.
4.1.1 Semantic Boundary Labeling Results
In this section, we present semantic boundary labeling results on the SBD dataset [11], which includes ground truth
boundaries that are also labeled with one of 20Pascal VOC
classes. The boundary detection accuracy for each class is
Method (Metric) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
InvDet (MF) 42.6 49.5 15.7 16.8 36.7 43.0 40.8 22.6 18.1 26.6 10.2 18.0 35.2 29.4 48.2 14.3 26.8 11.2 22.2 32.0 28.0
HFL-FC8 (MF) 71.6 59.6 68.0 54.1 57.2 68.0 58.8 69.3 43.3 65.8 33.3 67.9 67.5 62.2 69.0 43.8 68.5 33.9 57.7 54.8 58.7


Table 3: Results of semantic boundary labeling on the SBD benchmark using the Max F-score (MF) and Average Precision
(AP) metrics. Our method (H FL) outperforms Inverse Detectors [11] for all 20categories according to both metrics. Note
that using the CRF output to label the boundaries produces better results than using the outputs from the FC8 layer of FCN.
evaluated using the maximum F-score (MF), and average
precision (AP) measures.
Labeling boundaries with the semantic object information is a novel and still relatively unexplored problem.
Therefore, we found only one other approach (Inverse Detectors) that tried to tackle this problem [11]. The basic idea behind Inverse Detectors consists of several steps.
First, generic boundaries in the image are detected. Then,
a number of object proposal boxes are generated. These
two sources of information are then used to construct the
features. Finally, a separate classiﬁer is used to label the
boundaries with the object-speciﬁc information.
Table 3 shows that our approach signiﬁcantly outperforms Inverse Detectors according to both the maximum Fscore and the average precision metrics for all twenty categories. As described in Section 4.1 we evaluate the two


variants of our method. Denoted by H FL-FC8 is the variant for which we label H FL boundaries with the outputs
from the last layer (FC8) of the pretrained FCN. We denote
with H FL-CRF the result of labeling our boundaries with
the output from the DeepLab-CRF [5]. Among these two
variants, we show that the latter one produces better results.
This is expected since the CRF framework enforces spatial
coherence in the semantic segments.
In Figure 7, we present some of the qualitative results
produced by our method. We note that even with multiple
objects in the image, our method successfully recognizes
and localizes boundaries of each of the classes.
4.2. Semantic Segmentation
For the semantic segmentation task, we propose to
enhance the DeepLab-CRF [5] with our predicted H FL
boundaries. DeepLab-CRF is a system comprised of a Fully
Convolutional Network (described in Section 4.1) and a
dense CRF applied on top of FCN predictions.
Speciﬁcally, in the CRF, the authors propose to use
a Gaussian kernel and a bilateral term including position
and color terms as the CRF features (see [5]). While in
most cases the proposed scheme works well, DeepLab-CRF


a Gaussian kernel and a bilateral term including position
and color terms as the CRF features (see [5]). While in
most cases the proposed scheme works well, DeepLab-CRF
sometimes produces segmentations that are not spatially coherent, particularly for images containing small object reFigure 7: A visualization of the predicted semantic boundary labels. Images in the ﬁrst column are input examples.
Columns two and three show semantic H FL boundaries of
different object classes. Note that even with multiple objects appearing simultaneously, our method outputs precise
semantic boundaries.
gions.
We propose to address this issue by adding features
based on our predicted H FL boundaries in the CRF framework. Note that we use predicted boundaries from Section 3
and not the boundaries labeled with the object information
that we obtained in Section 4.1. We use the Normalized
Cut [28] framework to generate our features.
First, we construct a pixel-wise afﬁnity matrix Wusing
our H FL boundaries. We measure the similarity between
two pixels as:
Wij= exp (max
p2ijfM(p)2
2g)
whereWijrepresents the similarity between pixels iand
j,pdenotes the boundary point along the line segment ij
connecting pixels iandj,Mdepicts the magnitude of the


j,pdenotes the boundary point along the line segment ij
connecting pixels iandj,Mdepicts the magnitude of the
boundary at pixel p, anddenotes the smoothness parameter, which is usually set to 14% of the maximum boundary
value in the image.
The intuitive idea is that two pixels are similar (i.e.
Wij= 1) if there is no boundary crossing the line connecting these two pixels (i.e. M(p) = 08p2ij) or if the
boundary strength is low. We note that it is not necessary
to build a full afﬁnity matrix W. We build a sparse afﬁnMetric Method (Dataset) aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean
PP-IOUDL-CRF (VOC) 78.6 41.1 83.5 75.3 72.9 83.1 76.6 80.8 37.8 72.1 66.5 64.7 65.8 75.7 80.5 34.4 75.9 47.4 86.6 77.9 68.9


(per pixel) and PI-IOU (per image) evaluation metrics. We denote the original DeepLab-CRF system and our proposed
modiﬁcation as DL-CRF and DL-CRF+H FL, respectively. According to the PP-IOU metric, our proposed features (DLCRF+H FL) yield almost equivalent results as the original DeepLab-CRF system. However, based on PI-IOU metric, our
proposed features improve the mean accuracy by 3%and1:9%on SBD and VOC 2007 datasets respectively.
ity matrix connecting every pair of pixels iandjthat have
distance 5or less from each other.
After building a boundary-based afﬁnity matrix Wwe
setDii=P
i6=jWijand compute eigenvectors vof the
generalized eigenvalue system:
(DW)v=Dv
We then resize the eigenvectors vto the original image
dimensions, and use them as additional features to the CRF
part of DeepLab-CRF system. In our experiments, we use
the16eigenvectors corresponding to the smallest eigenvalues, which results in 16extra feature channels.
Note that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level


Note that the eigenvectors contain soft segmentation information. Because H FL boundaries predict object-level
contours with high conﬁdence, the eigenvectors often capture regions corresponding to objects. We visualize a few
selected eigenvectors in Figure 8. In the experimental section, we demonstrate that our proposed features make the
output produced by DeepLab-CRF more spatially coherent
and improve the segmentation accuracy according to one of
the metrics.
We also note that our proposed features are applicable to any generic method that incorporates CRF. For instance, even if DeepLab-CRF used an improved DeepLab
network architecture, our features would still be beneﬁcial
because they contribute directly to the CRF part and not the
DeepLab network part of the system.
4.2.1 Semantic Segmentation Results
In this section, we present semantic segmentation results
on the SBD [11] and also Pascal VOC 2007 [8] datasets,
which both provide ground truth segmentations for 20Pascal VOC classes. We evaluate the results in terms of two
metrics. The ﬁrst metric measures the accuracy in terms of
pixel intersection-over-union averaged per pixel (PP-IOU)
across the 20 classes. According to this metric, the accuracy
is computed on a per pixel basis. As a result, the images that


across the 20 classes. According to this metric, the accuracy
is computed on a per pixel basis. As a result, the images that
contain large object regions are given more importance.
Figure 8: In this ﬁgure, the ﬁrst column depicts an input
image while the second and third columns illustrate two selected eigenvectors for that image. The eigenvectors contain
soft segmentation information. Because H FL boundaries
capture object-level boundaries, the resulting eigenvectors
primarily segment regions corresponding to the objects.
We observe that while DeepLab-CRF works well on the
images containing large object regions, it produces spatially
disjoint outputs for the images with smaller and object regions (see Figure 9). This issue is often being overlooked,
because according to the PP-IOU metric, the images with
large object regions are given more importance and thus
contribute more to the accuracy. However, certain applications may require accurate segmentation of small objects.
Therefore, in addition to PP-IOU, we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each
of the images.
For both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a


of the images.
For both of the metrics we compare the semantic segmentation results of a pure DeepLab-CRF [5] and also a
modiﬁcation of DeepLab-CRF with our proposed features
added to the CRF framework. We present the results for
both of the metrics in Table 4.
Based on these results, we observe that according to
the ﬁrst metric (PP-IOU), our proposed features yield almost equivalent results as the original DeepLab-CRF system. However, according to the second metric (PI-IOU) our
features yield an average improvement of 3%and1:9%in
Figure 9: An illustration of the more challenging semantic
segmentation examples. The ﬁrst column depicts the predictions achieved by DeepLab-CRF, while the second column illustrates the results after adding our proposed features to the CRF framework. The last column represents
ground truth segmentations. Notice how our proposed features render the predicted semantic segments more spatially
coherent and overall more accurate.
SBD and VOC 2007 datasets respectively.
We also visualize the qualitative results produced by both
approaches in Figure 9. Notice how our proposed features
make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.
Once again, we want to stress that our H FL features


make the segmentations look smoother relative to the segmentations produced by the original DeepLab-CRF system.
Once again, we want to stress that our H FL features
are applicable to any method that uses the CRF. Therefore,
based on the results presented in this section, we believe that
our proposed features could be beneﬁcial in a wide array of
problems that involve the use of the CRF framework.
4.3. Object Proposals
Finally, we show that our method produces object-level
boundaries that can be successfully exploited in an object
proposal scheme. Speciﬁcally we adopt the EdgeBoxes approach [31], which can be applied to any generic boundaries to produce a list of object proposal boxes. The original EdgeBoxes method uses SE boundaries to generate the
boxes. However, SE boundaries are predicted using lowlevel color and texture features, rather than object-level
features. Thus, here we validate the hypothesis that the
EdgeBoxes proposals can be improved by replacing the SE
boundaries with our H FL boundaries.
4.3.1 Object Proposal Results
In this section, we present object proposal results on the
Pascal VOC 2012 dataset [9]. We evaluate the quality of
bounding-box proposals according to three metrics: area
under the curve (AUC), the number of proposals needed to


Pascal VOC 2012 dataset [9]. We evaluate the quality of
bounding-box proposals according to three metrics: area
under the curve (AUC), the number of proposals needed to
reach recall of 75%, and the maximum recall over 5000 object bounding-boxes. Additionally, we compute the accuracy for each of the metrics for three different intersectionMethodIoU 0.65 IoU 0.7 IoU 0.75
AUC N@75% Recall AUC N@75% Recall AUC N@75% Recall
SE 0:52 413 0.93 0:47 658 0.88 0.41 inf 0:75
HFL 0.53 365 0.95 0.48 583 0.9 0.41 2685 0.77
Table 5: Comparison of object proposal results. We
compare the quality of object proposals using Structured
Edges [6] and H FL boundaries. We evaluate the performance for three different IOU values and demonstrate that
using H FL boundaries produces better results for each evaluation metric and for each IOU value.
over union (IOU) values: 0:65;0:7, and 0:75. We present
these results in Table 5. As described in Section 4.3, we use
EdgeBoxes [31], a package that uses generic boundaries, to
generate object proposals. We compare the quality of the


EdgeBoxes [31], a package that uses generic boundaries, to
generate object proposals. We compare the quality of the
generated object proposals when using SE boundaries and
HFL boundaries. We demonstrate that for each IOU value
and for each of the three evaluation metrics, H FL boundaries produce better or equivalent results. This conﬁrms our
hypothesis that H FL boundaries can be used effectively for
high-level vision tasks such as generating object proposals.
5. Conclusions
In this work, we presented an efﬁcient architecture that
uses object-level information to predict semantically meaningful boundaries. Most prior edge detection methods rely
exclusively on low-level features, such as color or texture, to
detect the boundaries. However, perception studies suggest
that humans employ object-level reasoning when deciding
whether a given pixel is a boundary [13, 25, 17]. Thus,
we propose a system that focuses on the semantic objectlevel cues rather than low level image information to detect
the boundaries. For this reason we refer to our boundary
detection scheme as a High-for-Low approach, where highlevel object features inform the low-level boundary detection process. In this paper we demonstrated that our proposed method produces boundaries that accurately separate
objects and the background in the image and also achieve
higher F-score compared to any prior work.


objects and the background in the image and also achieve
higher F-score compared to any prior work.
Additionally, we showed that, because H FL boundaries
are based on object-level features, they can be employed to
aid a number of high level vision tasks in a Low-for-High
fashion. We use our boundaries to boost the accuracy of
state-of-the-art methods on the high-level vision tasks of semantic boundary labeling, semantic segmentation, and object proposals generation. We show that using H FL boundaries leads to better results in each of these tasks.
To conclude, our boundary detection method is accurate,
efﬁcient, applicable to a variety of datasets, and also useful
for multiple high-level vision tasks. We plan to release the
source code for H FL upon the publication of the paper .
6. Acknowledgements
We thank Mohammad Haris Baig for the suggestions and
help with the software. This research was funded in part by
NSF award CNS-1205521.
References
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2


detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2
[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqués, and J. Malik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition (CVPR) , 2014. 2, 4
[3] G. Bertasius, J. Shi, and L. Torresani. Deepedge: A multiscale bifurcated deep network for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4, 5
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2
[5] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L.
Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR , abs/1412.7062,
2014. 1, 5, 6, 7


2014. 1, 5, 6, 7
[6] P. Dollár and C. L. Zitnick. Fast edge detection using structured forests. PAMI , 2015. 2, 3, 4, 5, 8
[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. CoRR ,
abs/1310.1531, 2013. 5
[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2007 (VOC2007) Results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html.
7
[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
8


Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
8
[10] Y . Ganin and V . S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 3, 4, 5
[11] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In International
Conference on Computer Vision (ICCV) , 2011. 1, 5, 6, 7
[12] B. Hariharan, P. A. Arbeláez, R. B. Girshick, and J. Malik.
Hypercolumns for object segmentation and ﬁne-grained localization. CoRR , abs/1411.5752, 2014. 3
[13] P.-J. Hsieh, E. Vul, and N. Kanwisher. Recognition alters the
spatial pattern of fmri activation in early retinotopic cortex.
Journal of Neurophysiology , 103(3):1501–1507, 2010. 1, 8


spatial pattern of fmri activation in early retinotopic cortex.
Journal of Neurophysiology , 103(3):1501–1507, 2010. 1, 8
[14] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 4[15] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 4
[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction
network and quality dissection. AISTATS , 1(2):9, 2014. 2, 3,
4, 5
[17] Z. Kourtzi and N. Kanwisher. Representation of perceived
object shape by the human lateral occipital complex. Science, 293:1506–1509, 2001. 1, 8


object shape by the human lateral occipital complex. Science, 293:1506–1509, 2001. 1, 8
[18] J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned
mid-level representation for contour and object detection. In
CVPR , 2013. 2
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CVPR (to appear) ,
Nov. 2015. 3, 5
[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse
code transfer for contour detection and semantic labeling. In
Asian Conference on Computer Vision (ACCV) , 2014. 2
[21] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of
exemplar-svms for object detection and beyond. In ICCV ,
2011. 4
[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4


evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4
[23] X. Ren and L. Bo. Discriminatively Trained Sparse Code
Gradients for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 2, 4
[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge, 2014. 3
[25] J. L. Sanguinetti, J. J. Allen, and M. A. Peterson. The ground
side of an object perceived as shapeless yet processed for
semantics. Psychological science , page 0956797613502814,
2013. 1, 8
[26] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 3


and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 3
[27] W. Shen, X. Wang, Y . Wang, X. Bai, and Z. Zhang. Deepcontour: A deep convolutional feature learned by positivesharing loss for contour detection. June 2015. 1, 3, 4, 5
[28] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 22:888–905, 1997. 2, 6
[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 1
[30] A. Sironi, V . Lepetit, and P. Fua. Multiscale centerline detection by learning a scale-space distance transform. June 2014.
4
[31] C. L. Zitnick and P. Dollár. Edge boxes: Locating object
proposals from edges. In ECCV , 2014. 1, 8


Semantic Segmentation with Boundary Neural Fields
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
The state-of-the-art in semantic segmentation is currently represented by fully convolutional networks (FCNs).
However, FCNs use large receptive ﬁelds and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations that are poorly localized around object
boundaries. Prior work has attempted to address this issue
in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional parameters and low-level features that are difﬁcult to tune and integrate into the original
network architecture. Additionally, most CRFs use colorbased pixel afﬁnities, which are not well suited for semantic
segmentation and lead to spatially disjoint predictions.
To overcome these problems, we introduce a Boundary

Summary:
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.

3. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection

Gedas Bertasius, Jianbo Shi, Lorenzo Torresani

4. High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision


segmentation and lead to spatially disjoint predictions.
To overcome these problems, we introduce a Boundary
Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues. The boundary information is used to enhance semantic segment coherence and to improve object localization. Speciﬁcally, we
ﬁrst show that the convolutional ﬁlters of semantic FCNs
provide good features for boundary detection. We then employ the predicted boundaries to deﬁne pairwise potentials
in our energy. Finally, we show that our energy decomposes semantic segmentation into multiple binary problems,
which can be relaxed for efﬁcient global optimization. We
report extensive experiments demonstrating that minimization of our global boundary-based energy yields results superior to prior globalization methods, both quantitatively
as well as qualitatively.
1. Introduction
The recent introduction of fully convolutional networks
(FCNs) [22] has led to signiﬁcant quantitative improvements on the task of semantic segmentation. However, despite their empirical success, FCNs suffer from some limitations. Large receptive ﬁelds in the convolutional layers and
the presence of pooling layers lead to blurring and segmentation predictions at a signiﬁcantly lower resolution than the


the presence of pooling layers lead to blurring and segmentation predictions at a signiﬁcantly lower resolution than the
Figure 1: Examples illustrating shortcomings of prior semantic segmentation methods: the second column shows
results obtained with a FCN [22], while the third column
shows the output of a Dense-CRF applied to FCN predictions [19, 7]. Segments produced by FCN are blob-like and
are poorly localized around object boundaries. Dense-CRF
produces spatially disjoint object segments due to the use of
a color-based pixel afﬁnity function that is unable to measure semantic similarity between pixels.
original image. As a result, their predicted segments tend to
be blobby and lack ﬁne object boundary details. We report
in Fig. 1 some examples illustrating typical poor localization of objects in the outputs of FCNs.
Recently, Chen at al. [7] addressed this issue by applying a Dense-CRF post-processing step [19] on top of coarse
FCN segmentations. However, such an approach introduces
several problems of its own. First, the Dense-CRF adds new
parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods


parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods
based on CRFs or MRFs use low-level pixel afﬁnity functions, such as those based on color. These low-level afﬁnities often fail to capture semantic relationships between objects and lead to poor segmentation results (see last column
in Fig. 1).
We propose to address these shortcomings by means of
a Boundary Neural Field (BNF), an architecture that employs a single semantic segmentation FCN to predict semantic boundaries and then use them to produce semantic
segmentation maps via a global optimization. We demonstrate that even though the semantic segmentation FCN has
not been optimized to detect boundaries, it provides good
1arXiv:1511.02674v2  [cs.CV]  24 May 2016
...
Convolutional Feature Maps RGB Input Image
Segmentation Mask in       Softmax Layer
   Minimize Global Energy
Boundaries
|{z}...
wn()++()w1
Xi✓i(zi)+Xij✓ij(zi,zj)
Linear Combination of Feature MapsFigure 2: The architecture of our system (best viewed in
color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries


color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries
for our global energy; 2) to compute object boundaries.
Speciﬁcally, we deﬁne semantic boundaries as a linear combination of these feature maps (with a sigmoid function applied on top of the sum) and learn individual weights corresponding to each convolutional feature map. We integrate
this boundary information in the form of pairwise potentials
(pixel afﬁnities) for our energy model.
features for boundary detection. Speciﬁcally, the contributions of our work are as follows:
We show that semantic boundaries can be expressed
as a linear combination of interpolated convolutional
feature maps inside an FCN. We introduce a boundary
detection method that exploits this intuition to predict
object boundaries with accuracy superior to the statethe-of-art.
We demonstrate that boundary-based pixel afﬁnities
are better suited for semantic segmentation than the
commonly used color afﬁnity functions.
Finally, we introduce a new global energy that decomposes semantic segmentation into multiple binary
problems and relaxes the integrality constraint. We
show that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs.


problems and relaxes the integrality constraint. We
show that minimizing our proposed energy yields better qualitative and quantitative results relative to traditional globalization models such as MRFs or CRFs.
2. Related Work
Boundary Detection. Spectral methods comprise one
of the most prominent categories for boundary detection.
In a typical spectral framework, one formulates a generalized eigenvalue system to solve a low-level pixel groupingproblem. The resulting eigenvectors are then used to predict the boundaries. Some of the most notable approaches
in this genre are MCG [2], gPb [1], PMI [17], and Normalized Cuts [29]. A weakness of spectral approaches is that
they tend to be slow as they perform a global inference over
the entire image.
To address this issue, recent approaches cast boundary
detection as a classiﬁcation problem and predict the boundaries in a local manner with high efﬁciency. The most notable examples in this genre include sketch tokens (ST) [20]
and structured edges (SE) [9], which employ fast random
forests. However, many of these methods are based on
hand-constructed features, which are difﬁcult to tune.


forests. However, many of these methods are based on
hand-constructed features, which are difﬁcult to tune.
The issue of hand-constructed features have been recently addressed by several approaches based on deep learning, such as N4ﬁelds [11], DeepNet [18], DeepContour [27], DeepEdge [3], H FL [4] and HED [33]. All of
these methods use CNNs in some way to predict the boundaries. Whereas DeepNet and DeepContour optimize ordinary CNNs to a boundary based optimization criterion from
scratch, DeepEdge and H FL employ pretrained models to
compute boundaries. The most recent of these methods is
HED [33], which shows the beneﬁt of deeply supervised
learning for boundary detection.
In comparison to prior deep learning approaches, our
method offers several contributions. First, we exploit the inherent relationship between boundary detection and semantic segmentation to predict semantic boundaries. Specifically, we show that even though the semantic FCN has
not been explicitly trained to predict boundaries, the convolutional ﬁlters inside the FCN provide good features for
boundary detection. Additionally, unlike DeepEdge [3] and
HFL [4], our method does not require a pre-processing step
to select candidate contour points, as we predict boundaries


HFL [4], our method does not require a pre-processing step
to select candidate contour points, as we predict boundaries
on all pixels in the image. We demonstrate that our approach allows us to achieve state-of-the-art boundary detection results according to both F-score and Average Precision metrics. Additionally, due to the semantic nature of
our boundaries, we can successfully use them as pairwise
potentials for semantic segmentation in order to improve
object localization and recover ﬁne structural details, typically lost by pure FCN-based approaches.
Semantic Segmentation. We can group most semantic segmentation methods into three broad categories. The
ﬁrst category can be described as “two-stage” approaches,
where an image is ﬁrst segmented and then each segment
is classiﬁed as belonging to a certain object class. Some
of the most notable methods that belong to this genre include [24, 6, 12, 14].
The primary weakness of the above methods is that they
are unable to recover from errors made by the segmentation
algorithm. Several recent papers [15, 10] address this issue
by proposing to use deep per-pixel CNN features and then
classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmentation problem, they perform predictions independently on


classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmentation problem, they perform predictions independently on
each pixel. This leads to extremely local predictions, where
the relationships between pixels are not exploited in any
way, and thus the resulting segmentations may be spatially
disjoint.
The third and ﬁnal group of semantic segmentation
methods can be viewed as front-to-end schemes where segmentation maps are predicted directly from raw pixels without any intermediate steps. One of the earliest examples of
such methods is the FCN introduced in [22]. This approach
gave rise to a number of subsequent related approaches
which have improved various aspects of the original semantic segmentation [7, 34, 8, 16, 21]. There have also been attempts at integrating the CRF mechanism into the network
architecture [7, 34]. Finally, it has been shown that semantic
segmentation can also be improved using additional training
data in the form of bounding boxes [8].
Our BNF offers several contributions over prior work. To
the best of our knowledge, we are the ﬁrst to present a model
that exploits the relationship between boundary detection
and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from semantic boundaries inside an FCN, and use these boundaries to


and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from semantic boundaries inside an FCN, and use these boundaries to
predict the segmentations in a global fashion. Unlike [21],
which requires a large number of additional parameters to
learn for the pairwise potentials, our global model only
needs5Kextra parameters, which is about 3orders of
magnitudes less than the number of parameters in a typical deep convolutional network (e.g. VGG [30]). We empirically show that our proposed boundary-based afﬁnities
are better suited for semantic segmentation than color-based
afﬁnities. Additionally, unlike in [7, 34, 21], the solution to
our proposed global energy can be obtained in closed-form,
which makes global inference easier. Finally we demonstrate that our method produces better results than traditional globalization models such as CRFs or MRFs.
3. Boundary Neural Fields
In this section, we describe Boundary Neural Fields.
Similarly to traditional globalization methods, Boundary
Neural Fields are deﬁned by an energy including unary
and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image


and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image
and then combine them in a global manner. More precisely,
the coarse segmentations predicted by a semantic FCN are
used to deﬁne the unary potentials of our BNF. Next, we
show that the convolutional feature maps of the FCN can
be used to accurately predict semantic boundaries. These
boundaries are then used to build pairwise pixel afﬁnities,
which are used as pairwise potentials by the BNF. Finally,we introduce a global energy function, which minimizes the
energy corresponding to the unary and pairwise terms and
improves the initial FCN segmentation. The detailed illustration of our architecture is presented in Figure 2. We now
explain each of these steps in more detail.
3.1. FCN Unary Potentials
To predict semantic unary potentials we employ the
DeepLab model [7], which is a fully convolutional adaptation of the VGG network [30]. The FCN consists of 16convolutional layers and 3fully convolutional layers. There are
more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].


more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].
Although these more advanced architectures could be integrated into our framework to improve our unary potentials, in this work we focus on two aspects orthogonal to
this prior work: 1) demonstrating that our boundary-based
afﬁnity function is better suited for semantic segmentation
than the common color-based afﬁnities and 2) showing that
our proposed global energy achieves better qualitative and
quantitative semantic segmentation results in comparison to
prior globalization models.
3.2. Boundary Pairwise Potentials
In this section, we describe our approach for building
pairwise pixel afﬁnities using semantic boundaries. The basic idea behind our boundary detection approach is to express semantic boundaries as a function of convolutional
feature maps inside the FCN. Due to the close relationship
between the tasks of semantic segmentation and boundary
detection, we hypothesize that convolutional feature maps
from the semantic segmentation FCN can be employed as
features for boundary detection.
3.2.1 Learning to Predict Semantic Boundaries.
We propose to express semantic boundaries as a linear combination of interpolated FCN feature maps with a non-linear
function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior


function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior
work (see e.g. [15]) in order to obtain dense pixel-level features from the low-resolution outputs of deep convolutional
layers. Here we adopt interpolation to produce pixel-level
boundary predictions. There are several advantages to our
proposed formulation. First, because we express boundaries
as a linear combination of feature maps, we only need to
learn a small number of parameters, corresponding to the
individual weight values of each feature map in the FCN.
This amounts to5Klearning parameters, which is much
smaller than the number of parameters in the entire network
(15M). In comparison, DeepEdge [3] and HFL [4] need
17M and 6M additional parameters to predict boundaries.
Furthermore, expressing semantic boundaries as a linear
combination of FCN feature maps allows us to efﬁciently
predict boundary probabilities for all pixels in the image
(we resize the FCN feature maps to the original image dimensions). This eliminates the need to select candidate
boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as


boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as
follows. First we use use SBD segmentations [13] to optimize our FCN for semantic segmentation task. We then
treat FCN convolutional maps as features for the boundary detection task and use the boundary annotations from
BSDS 500 dataset [23] to learn the weights for each feature
map. BSDS 500 dataset contains 200training, 100validation, 200testing images, and ground truth annotations by 5
human labelers for each of these images.
To learn the weights corresponding to each convolutional
feature map we ﬁrst sample 80Kpoints from the dataset.
We deﬁne the target labels for each point as the fraction of
human annotators agreeing on that point being a boundary.
To ﬁx the issue of label imbalance (there are many more
non-boundaries than boundaries), we divide the label space
into four quartiles, and select an equal number of samples
for each quartile to balance the training dataset. Given these
sampled points, we then deﬁne our features as the values in
the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up


the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up
and apply a sigmoid function on top of it. We obtain the
weights corresponding to each convolutional feature map by
minimizing the cross-entropy loss using a stochastic batch
gradient descent for 50epochs. To obtain crisper boundaries at test-time we post-process the boundary probabilities
using non-maximum suppression.
To give some intuition on how FCN feature maps contribute to boundary detection, in Fig. 3 we visualize the feature maps corresponding to the highest weight magnitudes.
It is clear that many of these maps contain highly localized
boundary information.
Boundary Detection Results Before discussing how
boundary information is integrated in our energy for semantic segmentation, here we present experimental results assessing the accuracy of our boundary detection scheme. We
tested our boundary detector on the BSDS500 dataset [23],
which is the standard benchmark for boundary detection.
The quality of the predicted boundaries is evaluated using
three standard measures: ﬁxed contour threshold (ODS),
per-image best threshold (OIS), and average precision (AP).
In Table 1 we show that our algorithm outperforms all
prior methods according to both F-score measures and the


per-image best threshold (OIS), and average precision (AP).
In Table 1 we show that our algorithm outperforms all
prior methods according to both F-score measures and the
Average Precision metric. In Fig. 4, we also visualize our
predicted boundaries. The second column shows the pixellevel softmax output computed from the linear combination of feature maps, while the third column depicts our ﬁFigure 3: An input image and convolutional feature maps
corresponding to the largest weight magnitude values. Intuitively these are the feature maps that contribute most heavily to the task of boundary detection.
Method ODS OIS AP
SCG [25] 0.739 0.758 0.773
SE [9] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [11] 0.753 0.769 0.784
DeepEdge [3] 0.753 0.772 0.807
DeepContour [27] 0.756 0.773 0.797
HFL [4] 0.767 0.788 0.795
HED [33] 0.782 0.804 0.833
BNF 0.788 0.807 0.851


HED [33] 0.782 0.804 0.833
BNF 0.788 0.807 0.851
Table 1: Boundary detection results on BSDS500 benchmark. Our proposed method outperforms all prior algorithms according to all three evaluation metrics.
nal boundaries after applying a non-maximum suppression
post-processing step.
We note that our predicted boundaries achieve highconﬁdence predictions around objects. This is important as
we employ these boundaries to improve semantic segmentation results, as discussed in the next subsection.
3.2.2 Constructing Pairwise Pixel Afﬁnities.
We can use the predicted boundaries to build pairwise pixel
afﬁnities. Intuitively, we declare two pixels as similar (i.e.,
likely to belong to the same segment) if there is no boundary crossing the straight path between these two pixels.
Conversely, two pixels are dissimilar if there is a boundary crossing their connecting path. The larger the boundary
magnitude of the crossed path, the more dissimilar the two
pixels should be, since a strong boundary is likely to mark
the separation of two distinct segments. Similarly to [1], we
encode this intuition with a following formulation:
wsb
ij= exp (Mij
sb) (1)


encode this intuition with a following formulation:
wsb
ij= exp (Mij
sb) (1)
Figure 4: A ﬁgure illustrating our boundary detection results. In the second column, we visualize the raw probability output of our boundary detector. In the third column,
we present the ﬁnal boundary maps after non-maximum
suppression. While most prior methods predict the boundaries where the sharpest change in color occurs, our method
captures semantic object-level boundaries, which we subsequently use to aid semantic segmentation.
whereMijdenotes the maximum boundary value that
crosses the straight line path between pixels iandj,sbdepicts the smoothing parameter and wsb
ijdenotes the semantic
boundary-based afﬁnity between pixels iandj.
Similarly, we want to exploit high-level object information in the network to deﬁne another type of pixel similarity.
Speciﬁcally, we use object class probabilities from the softmax (SM) layer to achieve this goal. Intuitively, if pixels i
andjhave different hard segmentation labels from the softmax layer, we set their similarity ( wsm
ij) to0. Otherwise,
we compute their similarity using the following equation:
wsm
ij= exp (Dij
sm) (2)


ij) to0. Otherwise,
we compute their similarity using the following equation:
wsm
ij= exp (Dij
sm) (2)
whereDijdenotes the difference in softmax output values corresponding to the most likely object class for pixels
iandj, andsmis a smoothing parameter. Then we can
write the ﬁnal afﬁnity measure as:
wij= exp (wsm
ij)wsb
ij (3)
We exponentiate the term corresponding to the objectlevel afﬁnity because our boundary-based afﬁnity may be
too aggressive in declaring two pixels as dissimilar. To address this issue, we increase the importance of the objectlevel afﬁnity in (3) using the exponential function. However, in the experimental results section, we demonstrate
that most of the beneﬁt from modeling pairwise potentials
comes from wsb
ijrather thanwsm
ij.We then use this pairwise pixel afﬁnity measure to build
a global afﬁnity matrix Wthat encodes relationships between pixels in the entire image. For a given pixel, we
sample10% of points in the neighborhood of radius 20
around that pixel, and store the resulting afﬁnities into W.


sample10% of points in the neighborhood of radius 20
around that pixel, and store the resulting afﬁnities into W.
3.3. Global Inference
The last step in our proposed method is to combine semantic boundary information with the coarse segmentation
from the FCN softmax layer to produce an improved segmentation. We do this by introducing a global energy function that utilizes the afﬁnity matrix constructed in the previous section along with the segmentation from the FCN
softmax layer. Using this energy, we perform a global inference to get segmentations that are well localized around the
object boundaries and that are also spatially smooth.
Typical globalization models such as MRFs [31],
CRFs [19] or Graph Cuts [5] produce a discrete label assignment for the segmentation problem by jointly modeling a multi-label distribution and solving a non-convex optimization. The common problem in doing so is that the
optimization procedure may get stuck in local optima.
We introduce a new global energy function, which overcomes this issue and achieves better segmentation in comparison to prior globalization models. Similarly to prior
globalization approaches, our goal is to minimize the energy
corresponding to the sum of unary and pairwise potentials.
However, the key difference in our approach comes from the


globalization approaches, our goal is to minimize the energy
corresponding to the sum of unary and pairwise potentials.
However, the key difference in our approach comes from the
relaxation of some of the constraints. Speciﬁcally, instead
of modeling our problem as a joint multi-label distribution,
we propose to decompose it into multiple binary problems,
which can be solved concurrently. This decomposition can
be viewed as assigning pixels to foreground and background
labels for each of the different object classes. Additionally,
we relax the integrality constraint. Both of these relaxations
make our problem more manageable and allow us to formulate a global energy function that is differentiable, and has
a closed form solution.
In [35], the authors introduce the idea of learning
with global and local consistency in the context of semisupervised problems. Inspired by this work, we incorporate
some of these ideas in the context of semantic segmentation. Before deﬁning our proposed global energy function,
we introduce some relevant notation.
For the purpose of illustration, suppose that we only have
two classes: foreground and background. Then we can denote an optimal continuous solution to such a segmentation
problem with variable z. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a


problem with variable z. To denote similarity between pixelsiandjwe usewij. Then,diindicates the degree of a
pixeli. In graph theory, the degree of a node denotes the
number of edges incident to that node. Thus, we set the degree of a pixel to di=Pn
j=1wijfor alljexcepti6=j.
Finally, with fiwe denote an initial segmentation probaInput
 Softmax
 Dense-CRF
 BNF Boundaries
 BNF Segmentation
Figure 5: A ﬁgure illustrating semantic segmentation results. Images in columns two and three represent FCN softmax and
Dense-CRF predictions, respectively. Note that all methods use the same FCN unary potentials. Additionally, observe that
unlike FCN and Dense-CRF, our methods predicts segmentation that are both well localized around object boundaries and
that are also spatially smooth.
bility, which in our case is obtained from the FCN softmax
layer.
Using this notation, we can then formulate our global
inference objective as:
z= argmin
z
2X
idi(zifi
di)2+1
2X
ijwij(zizj)2(4)
This energy consists of two different terms. Similar to


idi(zifi
di)2+1
2X
ijwij(zizj)2(4)
This energy consists of two different terms. Similar to
the general globalization framework, our ﬁrst term encodes
the unary energy while the second term includes the pairwise energy. We now explain the intuition behind each of
these terms. The unary term attempts to ﬁnd a segmentation
assignment ( zi) that deviates little from the initial candidate
segmentation computed from the softmax layer (denoted by
fi). Theziin the unary term is weighted by the degree di
of the pixel in order to produce larger unary costs for pixels
that have many similar pixels within the neighborhood. Instead, the pairwise term ensures that pixels that are similar
should be assigned similar zvalues. To balance the energies of the two terms we introduce a parameter and set it
to0:025throughout all our experiments.
We can also express the same global energy function in
matrix notation:
z= argmin
z
2D(zD1f)T(zD1f)+1
2zT(DW)z
(5)
where zis an1vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree


2zT(DW)z
(5)
where zis an1vector containing an optimal continuous assignment for all npixels, Dis a diagonal degree
matrix, and Wis thennpixel afﬁnity matrix. Finally, f
denotes an1vector containing the probabilities from the
softmax layer corresponding to a particular object class.
An advantage of our energy is that it is differentiable. Ifwe denote the above energy as E(z)then the derivative of
this energy can be written as follows:
@E(z)
@z=D(zD1f) + (DW)z=0 (6)
With simple algebraic manipulations we can then obtain
a closed form solution to this optimization:
z= (DW)1f (7)
where=1
1+and=
1+. In the general case where
we havekobject classes we can write the solution as:
Z= (DW)1F (8)
where Znow depicts a nkmatrix containing assignments for all kobject classes, while Fdenotesnkmatrix
with object class probabilities from softmax layer. Due to


where Znow depicts a nkmatrix containing assignments for all kobject classes, while Fdenotesnkmatrix
with object class probabilities from softmax layer. Due to
the large size of DWit is impractical to invert it. However, if we consider an image as a graph where each pixel
denotes a vertex in the graph, we can observe that the term
DWin our optimization is equivalent to a Laplacian matrix of such graph. Since we know that a Laplacian matrix is
positive semi-deﬁnite, we can use the preconditioned conjugate gradient method [28] to solve the system in Eq. (9).
Alternatively, because our deﬁned global energy in Eq. (5)
is differentiable, we can efﬁciently solve this optimization
problem using stochastic gradient descent. We choose the
former option and solve the following system:
(DW)z=f (9)
To obtain the ﬁnal discrete segmentation, for each pixel
we assign the object class that corresponds to the largest
column value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z


we assign the object class that corresponds to the largest
column value in the row of Z(note that each row in Zrepresents a single pixel in the image, and each column in Z
Metric Inference Method RGB Afﬁnity BNF Afﬁnity
PP-IOUBelief Propagation [31] 75.4 75.6
ICM 74.2 75.8
TRWS [32] 75.9 76.7
QPBO [26] 76.9 77.2
BNF 74.6 77.6
PI-IOUBelief Propagation [31] 45.9 46.2
ICM 45.7 48.8
TRWS [32] 51.5 52.0
QPBO [26] 55.3 57.2
BNF 53.0 58.5
Table 2: We compare semantic segmentation results
when using a color-based pixel afﬁnity and our proposed
boundary-based afﬁnity. We note that our proposed afﬁnity
improves the performance of all globalization techniques.
Note that all of the inference methods use the same FCN
unary potentials . This suggests that for every method our


improves the performance of all globalization techniques.
Note that all of the inference methods use the same FCN
unary potentials . This suggests that for every method our
boundary-based afﬁnity is more beneﬁcial for semantic segmentation than the color-based afﬁnity.
represents one of the object classes). In the experimental
section, we show that this solution produces better quantitative and qualitative results in comparison to commonly used
globalization techniques.
4. Experimental Results
In this section we present quantitative and qualitative results for semantic segmentation on the SBD [13] dataset,
which contains objects and their per-pixel annotations for
20Pascal VOC classes. We evaluate semantic segmentation results using two evaluation metrics. The ﬁrst metric
measures accuracy based on pixel intersection-over-union
averaged per pixels (PP-IOU) across the 20 classes. According to this metric, the accuracy is computed on a perpixel basis. As a result, the images that contain large object regions are given more importance. However, for certain applications we may need to accurately segment small
objects. Therefore, similar to [4] we also consider the PIIOU metric (pixel intersection-over-union averaged per image across the 20 classes), which gives equal weight to each
of the images.


of the images.
We compare Boundary Neural Fields with other commonly used global inference methods. These methods include Belief Propagation [31], Iterated Conditional Mode
(ICM), Graph Cuts [5], and Dense-CRF [19]. Note that in
all of our evaluations we use the same FCN unary potentials
for every model.
Our evaluations provide evidence for three conclusions:
In Subsection 4.1, we show that our boundary-based
pixel afﬁnities are better suited for semantic segmentation than the traditional color-based afﬁnities.
In Subsection 4.2, we demonstrate that our global minimization leads to better results than those achieved byother inference schemes.
In Fig. 5, we qualitatively compare the outputs of FCN
and Dense-CRF to our predicted segmentations. This
comparison shows that the BNF segments are better
localized around the object boundaries and that they
are also spatially smooth.
4.1. Comparing Afﬁnity Functions for Semantic
Segmentation
In Table 2, we consider two global models. Both models use the same unary potentials obtained from the FCN
softmax layer. However, the ﬁrst model uses the popular
color-based pairwise afﬁnities, while the second employs


softmax layer. However, the ﬁrst model uses the popular
color-based pairwise afﬁnities, while the second employs
our boundary-based afﬁnities. Each of these two models
is optimized using several inference strategies. The table
shows that using our boundary based-afﬁnity function improves the results of all global inference methods according to both evaluation metrics. Note that we cannot include Dense-CRF [19] in this comparison because it employs an efﬁcient message-passing technique and integrating our afﬁnities into this technique is a non-trivial task.
However, we compare our method with Dense-CRF in Subsection 4.2.
The results in Table 2 suggest that our semantic boundary based pixel afﬁnity function yields better semantic segmentation results compared to the commonly-used color
based afﬁnities. We note that we also compared the results
of our inference technique using other edge detectors, notably UCM [1] and H FL [4]. In comparison to UCM edges,
we observed that our boundaries provide 1:0%and6:0%
according to both evaluation metrics respectively. When
comparing our boundaries with H FL method, we observed
similar segmentation performance, which suggests that our
method works best with the high quality semantic boundaries.


comparing our boundaries with H FL method, we observed
similar segmentation performance, which suggests that our
method works best with the high quality semantic boundaries.
4.2. Comparing Inference Methods for Semantic
Segmentation
Additionally, we also present semantic segmentation results for both of the metrics (PP-IOU and PI-IOU) in Table 3. In this comparison, all the techniques use the same
FCN unary potentials. Additionally, all inference methods
except Dense-CRF use our afﬁnity measure (since the previous analysis suggested that our afﬁnities yield better performance). We use BNF-SB to denote the variant of our
method that uses only semantic boundary based afﬁnities.
Additionally, we use BNF-SB-SM to indicate the version
of our method that uses both boundary and softmax -based
afﬁnities (see Eq. (3)).
Based on these results, we observe that our proposed
technique outperforms all the other globalization methods
according to both metrics, by 0:3%and1:3%respectively.
Metric Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean


Additionally, we use BNF-SB-SM to indicate our method that uses boundary and softmax based afﬁnities (See Eq. (3)). We
observe that our proposed globalization method outperforms other globalization techniques according to both metrics by
at least 0:3%and1:3%respectively. Note that in this experiment, all of the inference methods use the same FCN unary
potentials . Additionally, for each method except Dense-CRF (it is challenging to incorporate boundary based afﬁnities into
the Dense-CRF framework) we use our boundary based afﬁnities, since those lead to better results.
Additionally, these results indicate that most beneﬁt comes
from the semantic boundary afﬁnity term rather than the
softmax afﬁnity term.
In Fig. 5, we also present qualitative semantic segmentation results. Note that, compared to the segmentation output from the softmax layer, our segmentation is much better localized around the object boundaries. Additionally,
in comparison to Dense-CRF predictions, our method produces segmentations that are much spatially smoother.
4.3. Semantic Boundary Classiﬁcation
We can also label our boundaries with a speciﬁc object


4.3. Semantic Boundary Classiﬁcation
We can also label our boundaries with a speciﬁc object
class, using the same classiﬁcation strategy as in the H FL
system [4]. Since the SBD dataset provides annotations for
semantic boundary classiﬁcation, we can test our results
against the state-of-the-art H FL [4] method for this task.
Due to the space limitation, we do not include full results for
each category. However, we observe that our produced results achieve mean Max F-Score of 54:5%(averaged across
all20classes) whereas H FL method obtains 51:7%.
5. Conclusions
In this work we introduced a Boundary Neural Field
(BNF), an architecture that employs a semantic segmentation FCN to predict semantic boundaries and then uses the
predicted boundaries and the FCN output to produce an improved semantic segmentation maps a global optimization.
We showed that our predicted boundaries are better suited
for semantic segmentation than the commonly used low-level color based afﬁnities. Additionally, we introduced a
global energy function that decomposes semantic segmentation into multiple binary problems and relaxes an integrality constraint. We demonstrated that the minimization
of this global energy allows us to predict segmentations
that are better localized around the object boundaries and


of this global energy allows us to predict segmentations
that are better localized around the object boundaries and
that are spatially smoother compared to the segmentations
achieved by prior methods. We made the code of our globalization technique available at http://www.seas.upenn.
edu/~gberta/publications.html .
The main goal of this work was to show the effectiveness of boundary-based afﬁnities for semantic segmentation. However, due to differentiability of our global energy,
it may be possible to add more parameters inside the BNFs
and learn them in a front-to-end fashion. We believe that
optimizing the entire architecture jointly could capture the
inherent relationship between semantic segmentation and
boundary detection even better and further improve the performance of BNFs. We will investigate this possibility in
our future work.
6. Acknowledgements
This research was funded in part by NSF award CNS1205521.
References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE
Trans. Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2, 4, 7


Trans. Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 2, 4, 7
[2] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqués, and Jitendra
Malik. Multiscale combinatorial grouping. In Computer Vision and
Pattern Recognition (CVPR) , 2014. 2, 4
[3] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A
multi-scale bifurcated deep network for top-down contour detection.
InThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 2, 3, 4
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for-low
and low-for-high: Efﬁcient boundary detection from deep object features and its applications to high-level vision. In The IEEE International Conference on Computer Vision (ICCV) , December 2015. 2,
3, 4, 7, 8
[5] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach.
Intell. , 23(11):1222–1239, November 2001. 5, 7, 8


Intell. , 23(11):1222–1239, November 2001. 5, 7, 8
[6] João Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu.
Semantic segmentation with second-order pooling. In Proceedings
of the 12th European Conference on Computer Vision - Volume Part
VII, ECCV’12, pages 430–443, Berlin, Heidelberg, 2012. SpringerVerlag. 2
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin
Murphy, and Alan L. Yuille. Semantic image segmentation with deep
convolutional nets and fully. In ICLR , 2015. 1, 2, 3
[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference on Computer Vision
(ICCV) , December 2015. 3
[9] Piotr Dollár and C. Lawrence Zitnick. Fast edge detection using
structured forests. PAMI , 2015. 2, 4


[9] Piotr Dollár and C. Lawrence Zitnick. Fast edge detection using
structured forests. PAMI , 2015. 2, 4
[10] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence , August 2013.
2
[11] Yaroslav Ganin and Victor S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014. 2, 4
[12] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik.
Learning rich features from RGB-D images for object detection and
segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2014. 2
[13] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu
Maji, and Jitendra Malik. Semantic contours from inverse detectors.
InInternational Conference on Computer Vision (ICCV) , 2011. 4, 7


Maji, and Jitendra Malik. Semantic contours from inverse detectors.
InInternational Conference on Computer Vision (ICCV) , 2011. 4, 7
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV) , 2014. 2
[15] Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and
Jitendra Malik. Hypercolumns for object segmentation and ﬁnegrained localization. In The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , June 2015. 2, 3
[16] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled
deep neural network for semi-supervised semantic segmentation. In
NIPS) , December 2015. 3
[17] Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson.
Crisp boundary detection using pointwise mutual information. In
ECCV , 2014. 2
[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual
boundary prediction: A deep neural prediction network and quality


[18] Jyri J Kivinen, Christopher KI Williams, and Nicolas Heess. Visual
boundary prediction: A deep neural prediction network and quality
dissection. AISTATS , 1(2):9, 2014. 2
[19] Philipp Krähenbühl and Vladlen Koltun. Efﬁcient inference in fully
connected crfs with gaussian edge potentials. In J. Shawe-Taylor,
R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors,
Advances in Neural Information Processing Systems 24 , pages 109–
117. Curran Associates, Inc., 2011. 1, 5, 7, 8[20] Joseph Lim, C. Lawrence Zitnick, and Piotr Dollár. Sketch tokens:
A learned mid-level representation for contour and object detection.
InCVPR , 2013. 2
[21] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton van den Hengel. Efﬁcient piecewise training of deep structured models for semantic segmentation. CoRR , abs/1504.01013, 2015. 3
[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference


[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1,
3
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human
segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th
Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 4
[24] Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory
Shakhnarovich. Feedforward semantic segmentation with zoom-out
features. CoRR , abs/1412.0774, 2014. 2
[25] X. Ren and L. Bo. Discriminatively Trained Sparse Code Gradients
for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 4
[26] Carsten Rother, Vladimir Kolmogorov, Victor Lempitsky, and Martin
Szummer. Optimizing binary mrfs via extended roof duality. In Proc
Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8
[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang


Comp. Vision Pattern Recogn. (CVPR) , June 2007. 7, 8
[27] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang
Zhang. Deepcontour: A deep convolutional feature learned by
positive-sharing loss for contour detection. June 2015. 2, 4
[28] Jonathan R Shewchuk. An introduction to the conjugate gradient
method without the agonizing pain. Technical report, Pittsburgh, PA,
USA, 1994. 6
[29] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22:888–905, 1997. 2
[30] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 3
[31] Marshall F. Tappen and William T. Freeman. Comparison of graph
cuts with belief propagation for stereo, using identical mrf parameters. In Proceedings of the Ninth IEEE International Conference on
Computer Vision - Volume 2 , ICCV ’03, pages 900–, Washington,
DC, USA, 2003. IEEE Computer Society. 5, 7, 8


Computer Vision - Volume 2 , ICCV ’03, pages 900–, Washington,
DC, USA, 2003. IEEE Computer Society. 5, 7, 8
[32] Martin Wainwright, Tommi Jaakkola, and Alan Willsky. Map estimation via agreement on (hyper)trees: Message-passing and linear
programming approaches. IEEE Transactions on Information Theory, 51:3697–3717, 2002. 7, 8
[33] Saining Xie and Zhuowen Tu. Holistically-nested edge detection.
InThe IEEE International Conference on Computer Vision (ICCV) ,
December 2015. 2, 4
[34] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr.
Conditional random ﬁelds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 3
[35] Dengyong Zhou, Olivier Bousquet, Thomas N. Lal, Jason Weston,
and Bernhard Schölkopf. Learning with local and global consistency.
In S. Thrun, L.K. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16 , pages 321–328. MIT Press,


Unitary Evolution Recurrent Neural Networks
Martin ArjovskyMARJOVSKY @DC.UBA.AR
Amar ShahAS793@ CAM .AC.UK
Yoshua Bengio
Universidad de Buenos Aires, University of Cambridge,
Universit ´e de Montr ´eal. Yoshua Bengio is a CIFAR Senior Fellow.
Indicates ﬁrst authors. Ordering determined by coin ﬂip.
Abstract
Recurrent neural networks (RNNs) are notoriously difﬁcult to train. When the eigenvalues
of the hidden to hidden weight matrix deviate
from absolute value 1, optimization becomes difﬁcult due to the well studied issue of vanishingandexploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture
that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an
expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimization with this parameterization becomes feasible

Summary:
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.


expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimization with this parameterization becomes feasible
only when considering hidden states in the complex domain. We demonstrate the potential of
this architecture by achieving state of the art results in several hard tasks involving very longterm dependencies.
1. Introduction
Deep Neural Networks have shown remarkably good performance on a wide range of complex data problems including speech recognition (Hinton et al., 2012), image
recognition (Krizhevsky et al., 2012) and natural language
processing (Collobert et al., 2011). However, training very
deep models remains a difﬁcult task. The main issue surrounding the training of deep networks is the vanishing
andexploding gradients problems introduced by HochreProceedings of the 33rdInternational Conference on Machine
Learning , New York, NY , USA, 2016. JMLR: W&CP volume
48. Copyright 2016 by the author(s).iter (1991) and shown by Bengio et al. (1994) to be necessarily arising when trying to learn to reliably store bits
of information in any parametrized dynamical system. If
gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no