---
abstract: 'Flow-based generative models, conceptually attractive due to tractability of both the exact log-likelihood computation and latent-variable inference, and efficiency of both training and sampling, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. Despite their computational efficiency, the density estimation performance of flow-based generative models significantly falls behind those of state-of-the-art autoregressive models. In this work, we introduce *masked convolutional generative flow* (**<span style="font-variant:small-caps;">MaCow</span>**), a simple yet effective architecture of generative flow using masked convolution. By restricting the local connectivity in a small kernel, <span style="font-variant:small-caps;">MaCow</span> enjoys the properties of fast and stable training, and efficient sampling, while achieving significant improvements over Glow for density estimation on standard image benchmarks, considerably narrowing the gap to autoregressive models.'
author:
- |
    Xuezhe Ma\
    Language Technologies Institute\
    Carnegie Mellon University\
    Pittsburgh, PA, USA\
    `xuezhem@cs.cmu.edu` Eduard Hovy\
    Language Technologies Institute\
    Carnegie Mellon University\
    Pittsburgh, PA, USA\
    `hovy@cmu.edu`
bibliography:
- 'macow.bib'
title: 'MaCow: Masked Convolutional Generative Flow'
---

Introduction
============

Unsupervised learning of probabilistic models is a central yet challenging problem. Deep generative models have shown promising results in modeling complex distributions such as natural images [@radford2015unsupervised], audio [@van2016wavenet] and text [@bowman2015generating]. A number of approaches have emerged over the years, including Variational Autoencoders (VAEs) [@kingma2014auto], Generative Adversarial Networks (GANs) [@goodfellow2014generative], autoregressive neural networks [@larochelle2011neural; @oord2016pixel], and flow-based generative models [@dinh2014nice; @dinh2016density; @kingma2018glow]. Among these models, flow-based generative models gains popularity for this capability of estimating densities of complex distributions, efficiently generating high-fidelity syntheses and automatically learning useful latent spaces.

Flow-based generative models are typically warping a simple distribution into a complex one by mapping points from the simple distribution to the complex data distribution through a chain of invertible transformations whose Jacobian determinants are efficient to compute. This design guarantees that the density of the transformed distribution can be analytically estimated, making maximum likelihood learning feasible. Flow-based generative model has spawned significant interests in improving and analyzing it from both theoretical and practical perspectives, and applying it to a wide range of tasks and domains.

In their pioneering work, @dinh2014nice proposed *Non-linear Independent Component Estimation* (NICE), where they first applied flow-based models for modeling complex dimensional densities. RealNVP [@dinh2016density] extended NICE with more flexible invertible transformation and experimented on natural images. However, these flow-based generative models have much worse density estimation performance compared to state-of-the-art autoregressive models, and are incapable of realistic-looking synthesis of large images compared to GANs [@karras2017progressive; @brock2018large]. Recently, @kingma2018glow proposed Glow: generative flow with invertible 1x1 convolutions, significantly improving the density estimation performance on natural images. Importantly, they demonstrated that flow-based generative models optimized towards the plain likelihood-based objective are capable of generating realistic-looking high-resolution natural images efficiently. @prenger2018waveglow investigated applying flow-based generative models to speech synthesis by combining Glow with WaveNet [@van2016wavenet]. Unfortunately, the density estimation performance of Glow on natural images still falls behind autoregressive models, such as PixelRNN/CNN [@oord2016pixel; @salimans2017pixelcnn++], Image Transformer [@parmar2018image], PixelSNAIL [@chen2017pixelsnail] and SPN [@menick2018generating]. We noted in passing that there are also some work [@rezende2015variational; @kingma2016improved; @zheng2017] trying to apply flow to variational inference.

In this paper, we propose a novel architecture of generative flow, *masked convolutional generative flow* (**<span style="font-variant:small-caps;">MaCow</span>**), using masked convolutional neural networks [@oord2016pixel]. The bijective mapping between input and output variables can easily be established; meanwhile, computation of the determinant of the Jacobian is efficient. Compared to inverse autoregressive flow (IAF) [@kingma2016improved], <span style="font-variant:small-caps;">MaCow</span> has the merits of stable training and efficient inference and synthesis by restricting the local connectivity in a small “masked” kernel, and large receptive fields by stacking multiple layers of convolutional flows and using reversed ordering masks (§\[subsec:macow\]). We also propose a fine-grained version of the multi-scale architecture adopted in previous flow-based generative models to further improve the performance (§\[subsec:multi-scale\]). Experimentally, on three benchmark datasets for images — CIFAR-10, ImageNet and CelebA-HQ, we demonstrate the effectiveness of <span style="font-variant:small-caps;">MaCow</span> as a density estimator by consistently achieving significant improvements over Glow on all the three datasets. When equipped with the variational dequantization mechanism [@ho2018flow++], <span style="font-variant:small-caps;">MaCow</span> considerably narrows the gap on density estimation to autoregressive models (§\[sec:experiment\]).

Flow-based Generative Models {#sec:background}
============================

In this section, we first setup notations, describe flow-based generative models, and review Glow [@kingma2018glow], which is <span style="font-variant:small-caps;">MaCow</span> built on.

Notations
---------

Throughout we use uppercase letters for random variables, and lowercase letters for realizations of the corresponding random variables. Let $X \in \mathcal{X}$ be the randoms variables of the observed data, e.g., $X$ is an image or a sentence for image and text generation, respectively.

Let $P$ denote the true distribution of the data, i.e., $X \sim P$, and $D = \{x_1, \ldots, x_N\}$ be our training sample, where $x_i, i=1,\ldots, N,$ are usually i.i.d. samples of $X$. Let $\mathcal{P} = \{P_\theta : \theta \in \Theta\}$ denote a parametric statistical model indexed by parameter $\theta \in \Theta$, where $\Theta$ is the parameter space. $p$ is used to denote the density of corresponding distribution $P$. In the literature of deep generative models, deep neural networks are the most widely used parametric models. The goal of generative models is to learn the parameter $\theta$ such that $P_{\theta}$ can best approximate the true distribution $P$. In the context of maximal likelihood estimation, we wish to minimize the negative log-likelihood of the parameters: $$\label{eq:mle}
\min\limits_{\theta \in \Theta} \frac{1}{N} \sum\limits_{i=1}^{N} -\log p_{\theta}(x_i) = \min\limits_{\theta \in \Theta} \mathrm{E}_{\widetilde{P}(X)} [-\log p_{\theta}(X)],$$ where $\tilde{P}(X)$ is the empirical distribution derived from training data $D$.

Flow-based Models
-----------------

In the framework of flow-based generative models, a set of latent variables $Z \in \mathcal{Z}$ are introduced with a prior distribution $p_{Z}(z)$, typically a simple distribution like multivariate Gaussian. For a bijection function $f: \mathcal{X} \rightarrow \mathcal{Z}$ (with $g = f^{-1}$), the change of variable formula defines the model distribution on $X$ by : $$p_{\theta}(x) = p_{Z}(f_{\theta}(x))\left| \det(\frac{\partial f_{\theta}(x)}{\partial x})\right|,$$ where $\frac{\partial f_{\theta}(x)}{\partial x}$ is the Jacobian of $f_{\theta}$ at $x$.

The generative process is defined straight-forwardly as: $$\begin{array}{rcl}
z & \approx & p_{Z}(z) \\
x & = & g_{\theta}(z).
\end{array}$$ Flow-based generative models focus on certain types of transformations $f_{\theta}$ that both the inverse functions $g_{\theta}$ and the Jacobian determinants are tractable to compute. By stacking multiple such invertible transformations in a sequence, which is also called a (normalizing) *flow* [@rezende2015variational], a flow is capable of warping a simple distribution ($p_{Z}(z)$) in to a complex one ($p(x)$): $$X \underset{g_1}{\overset{f_1}{\longleftrightarrow}} H_1 \underset{g_2}{\overset{f_2}{\longleftrightarrow}} H2 \underset{g_3}{\overset{f_3}{\longleftrightarrow}} \cdots \underset{g_K}{\overset{f_K}{\longleftrightarrow}} Z,$$ where $f = f_1 \circ f_2 \circ \cdots \circ f_K$ is a flow of $K$ transformations. For brevity, we omit the parameter $\theta$ from $f_{\theta}$ and $g_{\theta}$.

Glow
----

In recent years, A number types of invertible transformations have emerged to enhance the expressiveness of flows, among which Glow [@kingma2018glow] has stood out for its simplicity and effectiveness on both density estimation and high-fidelity synthesis. We briefly describe the three types of transformations that Glow consists of.

#### Actnorm.

@kingma2018glow proposed an activation normalization layer (Actnorm) as an alternative for batch normalization [@ioffe2015batch] to alleviate the problems in model training. Similar to batch normalization, Actnorm performs an affine transformation of the activations using a scale and bias parameter per channel for 2D images: $$y_{i,j} = s \odot x_{i,j} + b,$$ where both $x$ and $y$ are tensors of shape $[h\times w \times c]$ with spatial dimensions $(h, w)$ and channel dimension $c$.

#### Invertible 1 x 1 convolution.

To incorporate a permutation along the channel dimension, Glow includes a trainable invertible 1 x 1 convolution layer to generalize the permutation operation: $$y_{i, j} = W x_{i,j},$$ where $W$ is the weight matrix with shape $c \times c$.

#### Affine Coupling Layers.

Following @dinh2016density, Glow has affine coupling layers in its architecture: $$\begin{array}{rcl}
x_a, x_b & = & \mathrm{split}(x) \\
y_a & = & x_a \\
y_b & = & \mathrm{s}(x_a) \odot x_b + \mathrm{b}(x_a) \\
y & = & \mathrm{concat}(y_a, y_b),
\end{array}$$ where $\mathrm{s}(x_a)$ and $\mathrm{b}(x_a)$ are outputs of two neural networks with $x_a$ as input. The $\mathrm{split}()$ and $\mathrm{concat}()$ functions perform operations along the channel dimension.

From the designed architecture of Glow, we see that interactions between spatial dimensions are incorporated only in the coupling layers. The coupling layer, however, is typically costly for memory, making it infeasible to stack a large number of coupling layers into a single model, particularly for high-resolution images. The main goal of this work is to design a new type of transformation that is able to simultaneously model dependencies in both the spatial and channel dimensions, meanwhile maintains relatively low-memory footprint, in the hope to improve the capacity of the generative flow.

Masked Convolutional Generative Flows {#sec:macow}
=====================================

In this section, we describe the architectural components that compose the *masked convolutional generative flow* (<span style="font-variant:small-caps;">MaCow</span>). We first introduce the proposed flow transformation using masked convolutions in §\[subsec:macow\]. Then, we present an fine-grained version of the multi-scale architecture adopted in previous generative flows [@dinh2016density; @kingma2018glow] in §\[subsec:multi-scale\]. In §\[subsec:dequant\], we briefly revisit the dequantization problem involved in flow-based generative models.

![Visualization of the receptive field of two masked convolutions with reversed ordering.[]{data-label="fig:mcf"}](figs/conv1.pdf){width="60.00000%"}

Flow with Masked Convolutions {#subsec:macow}
-----------------------------

Applying autoregressive models to normalizing flows has been explored by previous studies [@kingma2016improved; @papamakarios2017masked]. The idea behind autoregressive flows is sequentially modeling the input random variables in an autoregressive order to make sure the model cannot read input variables behind the current one: $$\label{eq:auto-flow}
y_t = \mathrm{s}(x_{<t}) \odot x_t + \mathrm{b}(x_{<t}),$$ where $x_{<t}$ denotes the input variables in $x$ that are in front of $x_t$ in the autoregressive order. $\mathrm{s}()$ and $\mathrm{b}()$ are two autoregressive neural networks typically implemented using spatial masks [@germain2015made; @oord2016pixel].

Despite of the effectiveness in high-dimensional space, autoregressive flows suffer from two crucial problems: (1) The training procedure is unstable when modeling a long range of contexts and stacking multiple layers. (2) Inference and synthesis are inefficient, due to the non-parallelizable inverse function.

We propose to use masked convolutions to restricting the local connectivity in a small “masked” kernel to address these two problems. The two autoregressive neural networks, $\mathrm{s}()$ and $\mathrm{b}()$, are implemented with one-layer masked convolutional networks with small kernels (e.g. $3\times3$) to make sure that they are only able to read contexts in a small neighborhood: $$\mathrm{s}(x_{<t}) = \mathrm{s}(x_{t\star}), \quad \mathrm{b}(x_{<t}) = \mathrm{b}(x_{t\star}),$$ where $x_{t\star}$ denotes the input variables, restricted in a small kernel, that $x_t$ depends on. By using masks in reversed ordering and stacking multiple layers of flows, the model can capture a large receptive field (see Figure \[fig:mcf\]), and is able to model dependencies in both the spatial and channel dimensions.

#### Efficient Synthesis.

As discussed above, synthesis from autoregressive flows is inefficient since the inverse has to be computed by sequentially traversing through the autoregressive order. In the context of 2D images with shape $[h\times w \times c]$, the time complexity of synthesis is $O(h \times w \times \mathrm{NN}(h, w, c))$, where $\mathrm{NN}(h, w, c)$ is the time of computing the outputs from the neural network $\mathrm{s}()$ and $\mathrm{b}()$ with input shape $[h\times w \times c]$. In our proposed flow with masked convolutions, computation of $x_{i,j}$ can begin as soon as all $x_{<i,j}$ are available, contrary to the autoregressive requirement that all $x_{<i,j}$ must be computed. Moreover, at each step we only need to feed $x_{(i,j)\star}$ (with shape $[kh\times kw\times c]$) into $\mathrm{s}()$ and $\mathrm{b}()$. Here $[\mathit{kh}\times \mathit{kw}\times c]$ is the shape of a kernel in the convolution. Thus, the time complexity reduces significantly to $O((h + w) \times \mathrm{NN}(\mathit{kh}, \mathit{kw}, c))$.

\[fig:architecture:step\]

Fine-grained Multi-Scale Architecture {#subsec:multi-scale}
-------------------------------------

@dinh2016density proposed a multi-scale architecture using a squeezing operation, which has been demonstrated to be helpful for training very deep flows. In the original multi-scale architecture, the model factors out half of the dimensions at each scale to reduce computational and memory cost. In this paper, inspired by the size upscaling in subscale ordering [@menick2018generating] which generates an image as a sequence of sub-images of equal size, we propose a fine-grained multi-scale architecture to further improve model performance. In this fine-grained multi-scale architecture, each scale consists of $M/2$ blocks. After each block, the model splits out $1/M$ dimensions of the input[^1]. Figure \[fig:architecture\] illustrates the graphical specification of the two versions of architectures. Experimental improvements demonstrate the effectiveness of the fine-grained multi-scale architecture (§\[sec:experiment\]).

Dequantization {#subsec:dequant}
--------------

From the description in §\[sec:background\], generative flows are defined on continuous random variables. Many real word datasets, however, are recordings of discrete representations of signals, and fitting a continuous density model to discrete data will produce a degenerate solution that places all probability mass on discrete datapoints [@uria2013rnade; @ho2018flow++]. A common solution to this problem is “dequantization” which converts the discrete data distribution into a continuous one.

Specifically, in the context of natural images, each dimension (pixel) of the discrete data $x$ takes on values in $\{0, 1, \ldots, 255\}$. The dequatization process adds a continuous random noise $u$ to $x$, resulting a continuous data point: $$\label{eq:dequant}
y = x + u,$$ where $u \in [0, 1)^{d}$ is continuous random noise taking values from interval $[0, 1)$. By modeling the density of $Y \in \mathcal{Y}$ with $p_{\theta}(y)$, the distribution of $X$ is defined as: $$P_\theta(x) = \int_{\mathcal{Y}} p_\theta(y) \operatorname{d\!}y = \int_{[0, 1)^d} p_\theta(x + u) \operatorname{d\!}u.$$ By restricting the range of $u$ in $[0, 1)$, the mapping between $y$ and a pair of $x$ and $u$ is bijective. Thus, we have $p_\theta(y) = p_\theta(x + u) = p_\theta(x, u)$.

By introducing a *dequantization noise distribution* $q(u|x)$, the training objective in can be re-written as: $$\begin{aligned}
\mathrm{E}_{P(X)} \Big[-\log P_{\theta}(X)\Big] & = \mathrm{E}_{P(X)} \left[-\log \int_{[0, 1)^d} p_\theta(X, u) \operatorname{d\!}u \right] \nonumber \\
 & = \mathrm{E}_{P(X)} \Bigg[\mathrm{E}_{q(u|X)} \left[-\log \frac{p_{\theta}(X, u)}{q(u|X)}\right] - \mathrm{KL}\big(q(u|X) || p_{\theta}(u|X)\big) \Bigg] \nonumber \\
 & \leq \mathrm{E}_{P(X)} \Bigg[\mathrm{E}_{q(u|X)} \Big[-\log p_{\theta}(X, u) \Big] + \mathrm{E}_{q(u|X)} \Big[\log q(u|X) \Big]\Bigg] \nonumber \\
 & = \mathrm{E}_{p(Y)} \Big[-\log p_{\theta}(Y) \Big] + \mathrm{E}_{P(X)} \mathrm{E}_{q(u|X)} \Big[\log q(u|X)\Big], \label{eq:elbo}\end{aligned}$$ where $p(y) = P(x) q(u|x)$ is the distribution of the dequantized variable $Y$ under the dequantization noise distribution $q(u|X)$.

#### Uniform Dequantization.

The most common dequantization method used in prior work is uniform dequantization, where the noise $u$ is sampled from the uniform form distribution $\mathrm{Unif(0, 1)}$: $$q(u|x) \sim \mathrm{Unif(0, 1)}, \forall x \in \mathcal{X}.$$ From , we have $$\mathrm{E}_{P(X)} \left[-\log P_{\theta}(X)\right] \leq \mathrm{E}_{p(Y)} \left[-\log p_{\theta}(Y) \right],$$ as $\log q(u|x) = 0, \forall x \in \mathcal{X}$.

#### Variational Dequantization.

As discussed in @ho2018flow++, uniform dequantization asks $p_\theta(y)$ to assign uniform density to unit hypercubes $[0, 1)^d$, which is difficult for smooth distribution approximators. They proposed to use a parametric dequantization noise distribution $q_\phi(u|x)$, and the training objective is to optimize the *evidence lower bound* (ELBO) provided in : $$\min\limits_{\theta, \phi} \mathrm{E}_{p_\phi(Y)} \left[-\log p_{\theta}(Y) \right] + \mathrm{E}_{P(X)} \mathrm{E}_{q_\phi(u|X)} \left[\log q_\phi(u|X) \right],$$ where $p_\phi(y) = P(x) q_\phi(u|x)$. In this paper, we implemented both the two dequantization methods for our <span style="font-variant:small-caps;">MaCow</span> (detailed in §\[sec:experiment\]).

Experiments {#sec:experiment}
===========

We evaluate our <span style="font-variant:small-caps;">MaCow</span> model on both low-resolution and high-resolution datasets. For a step of <span style="font-variant:small-caps;">MaCow</span>, we use $T=4$ masked convolution units, and the Glow step is the same of that described in @kingma2018glow: an *ActNorm* followed by an *Invertible $1\times1$ convolution*, followed by a *coupling layer*. Each the coupling layer has three convolution layers, where the the first and last convolutions are $3\times3$, while the center convolution is $1\times1$. For low resolution images we use affine coupling layers with 512 hidden channels, while for high resolution images we use additive layers with 256 hidden channels to reduce memory cost. ELU [@clevert2015elu] is used as the activation function throughout the flow architecture. For variational dequantization, the dequantization noise distribution $q_\phi(u|x)$ is modeled with a conditional <span style="font-variant:small-caps;">MaCow</span> with shallow architecture. More details on architectures, as well as results, and analysis of the conducted experiments, will be given in a source code release.

Low-Resolution Images
---------------------

We begin our experiments by evaluating the density estimation performance of <span style="font-variant:small-caps;">MaCow</span> on two commonly used datasets of low-resolution images for evaluating deep generative models: CIFAR-10 with images of size $32\times32$ [@krizhevsky2009learning] and $64\times64$ downsampled version of ImageNet [@oord2016pixel].

[ll|c:c]{} & **Model** & **CIFAR-10** & **ImageNet-64**\
& IAF VAE [@kingma2016improved] & 3.11 & –\
& Parallel Multiscale [@reed2017parallel] & – & 3.70\
& PixelRNN [@oord2016pixel] & 3.00 & 3.63\
& Gated PixelCNN [@van2016conditional] & 3.03 & 3.57\
& MAE [@ma2019mae] & 2.95 & –\
& PixelCNN++ [@salimans2017pixelcnn++] & 2.92 & –\
& PixelSNAIL [@chen2017pixelsnail] & **2.85** & **3.52**\
& SPN [@menick2018generating] & – & **3.52**\
& Real NVP [@dinh2016density] & 3.49 & 3.98\
& Glow [@kingma2018glow] & 3.35 & 3.81\
& Flow++:  [@ho2018flow++] & 3.29 & –\
& Flow++:  [@ho2018flow++] & **3.09** & 3.69\
& <span style="font-variant:small-caps;">MaCow</span>: & 3.31 & 3.78\
& <span style="font-variant:small-caps;">MaCow</span>: & 3.28 & 3.75\
& <span style="font-variant:small-caps;">MaCow</span>: & 3.16 & **3.64**\

We run experiments to dissect the effectiveness of each components of our <span style="font-variant:small-caps;">MaCow</span> model by ablation studies. The model utilizes the original multi-scale architecture, while the model augments the original one with the fine-grained multi-scale architecture proposed in §\[subsec:multi-scale\]. The model further implements the variational dequantization (§\[subsec:dequant\]) on the top of to replace the uniform dequantization. For each ablation, we slightly adjust the number of steps in each level so that all the models have similar numbers of parameters for fair comparison.

Table \[tab:density:low\] provides the density estimation performance of different variations of our <span style="font-variant:small-caps;">MaCow</span> model, together with the top-performing autoregressive models (first section) and flow-based generative models (second section). First, on both the two datasets, models outperform ones, demonstrating the effectiveness of the fine-grained multi-scale architecture. Second, with the uniform dequantization, <span style="font-variant:small-caps;">MaCow</span> combined with fine-grained multi-scale architecture significantly improves the performance over Glow on both the two datasets, and obtains slightly better results than Flow++ on CIFAR-10. In addition, with variational dequantization, <span style="font-variant:small-caps;">MaCow</span> achieves 0.05 improvement of bits/dim over Flow++ on ImageNet $64\times64$. On CIFAR-10, however, the performance of is around 0.07 bits/dim behind Flow++.

Compared with PixelSNAIL [@chen2017pixelsnail] and SPN [@menick2018generating], the state-of-the-art autoregressive generative models, the performance of <span style="font-variant:small-caps;">MaCow</span> is around 0.31 bits/dim worse on CIFAR-10 and 0.12 worse on ImageNet $64\times64$. Further improving the density estimation performance of <span style="font-variant:small-caps;">MaCow</span> on natural images has been left to future work.

  **Model**                                               **CelebA-HQ $256\times256$**
  ------------------------------------------------------ ------------------------------
  Glow [@kingma2018glow]                                              1.03
  SPN [@menick2018generating]                                       **0.61**
  <span style="font-variant:small-caps;">MaCow</span>:                0.95
  <span style="font-variant:small-caps;">MaCow</span>:                0.74

  : Negative Log-likelihood scores for 5-bit datasets in bits/dim.[]{data-label="tab:density:high"}

![5-bit $256\times256$ CelebA-HQ samples, with temperature 0.7.[]{data-label="fig:celeba:sample"}](figs/celeba_sample.png){width="99.00000%"}

High-Resolution Images
----------------------

We now demonstrate experimentally that our <span style="font-variant:small-caps;">MaCow</span> model is capable of high fidelity samples at high resolution. Following @kingma2018glow, we choose the CelebA-HQ dataset [@karras2017progressive], which consists of 30,000 high resolution images from the CelebA dataset [@liu2015faceattributes]. We train our models on 5-bit images, with the fine-grained multi-scale architecture and both the uniform and variational dequantization.

### Density Estimation

Table \[tab:density:high\] illustrates the negative log-likelihood scores in bits/dim of two versions of <span style="font-variant:small-caps;">MaCow</span> on the 5-bit $256\times256$ CelebA-HQ dataset. With uniform dequantization, <span style="font-variant:small-caps;">MaCow</span> improves the log-likelihood over Glow from 1.03 bits/dim to 0.95 bits/dim. Equipped with variational dequantization, <span style="font-variant:small-caps;">MaCow</span> obtains 0.74 bits/dim, 0.13 bits/dim behind the state-of-the-art autoregressive generative model SPN [@menick2018generating], significantly narrowing the gap.

### Image Generation

Consistent with previous work on likelihood-based generative models [@parmar2018image; @kingma2018glow], we found that sampling from a reduced-temperature model often results in higher-quality samples. Figure \[fig:celeba:sample\] showcases some random samples obtained from our model for 5-bit CelebA-HQ $256\times256$ with temperature 0.7. The images are extremely high quality for non-autoregressive likelihood models.

Conclusion
==========

In this paper, we proposed a new type of generative flow, coined <span style="font-variant:small-caps;">MaCow</span>, which exploits masked convolutional neural networks. By restricting the local dependencies in a small masked kernel, <span style="font-variant:small-caps;">MaCow</span> enjoys the properties of fast and stable training and efficient sampling. Experiments on both low- and high-resolution benchmark datasets of images show the capability of <span style="font-variant:small-caps;">MaCow</span> on both density estimation and high-fidelity generation, with state-of-the-art or comparable likelihood, and superior quality of samples against previous top-performing models.

One potential direction for future work is to extend <span style="font-variant:small-caps;">MaCow</span> to other forms of data, in particular text on which no attempt (to our best knowledge) has been made to apply flow-based generative models. Another exciting direction is to combine <span style="font-variant:small-caps;">MaCow</span>, or even general flow-based generative models, with variational inference to automatically learn meaningful (low-dimensional) representations from raw data.

[^1]: In our experiments, we set $M=4$. Note that the original multi-scale architecture is a special case of the fine-grained version with $M=2$.
