The problem of visual metamerism is defined as finding a family of perceptually
 indistinguishable, yet physically different images. In this paper, we propose our
 NeuroFovea metamer model, a foveated generative model that is based on a mixture
 of peripheral representations and style transfer forward-pass algorithms. Our
 gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder
 which allows us to encode images in high dimensional space and interpolate
 between the content and texture information with adaptive instance normalization
 anywhere in the visual field. Our contributions include: 1) A framework for
computing metamers that resembles a noisy communication system via a foveated
feed-forward encoder-decoder network – We observe that metamerism arises as a
byproduct of noisy perturbations that partially lie in the perceptual null space; 2)
A perceptual optimization scheme as a solution to the hyperparametric nature of
our metamer model that requires tuning of the image-texture tradeoff coefficients
everywhere in the visual field which are a consequence of internal noise; 3) An
 ABX psychophysical evaluation of our metamers where we also find that the rate
 of growth of the receptive fields in our model match V1 for reference metamers
 and V2 between synthesized samples. Our model also renders metamers at roughly
 a second, presenting a ×1000 speed-up compared to the previous work, which now
 allows for tractable data-driven metamer experiments. The history of metamers originally started through color matching theory, where two light sources were used to match a test light's wavelength, until both light sources are indistinguishable from each other producing what is called a color metamer. This leads to the definition of visual metamerism: when two physically different stimuli produce the same perceptual response (See Figure 1 for an example). Motivated by BID1 's work of local texture matching in the periphery as a mechanism that explains visual crowding, BID7 were the first to create such point-of-fixation driven metamers through such local texture matching models that tile the entire visual field given log-polar pooling regions that simulate the V1 and V2 receptive field sizes, as well as having global image statistics that match the metamer with the original image. The essence of their algorithm is to use gradient descent to match the local texture BID22 ) and image statistics of the original image throughout the visual field given a point of fixation until convergence thus producing two images that are perceptually indistinguishable to each other.However, metamerism research currently faces 2 main limitations: The first is that metamer rendering faces no unique solution. Consider the potentially trivial examples of having an image I and its metamer M where all pixel values are identical except for one which is set to zero (making this difference unnoticeable), or the case where the metameric response arises from an imperceptible equal perturbation across all pixels as suggested in BID16 ; BID7 . This is a concept similar to Just Noticeable Differences BID21 BID5 ). However, like the work of BID7 ; BID17 ; BID23 ; BID1 , we are interested in creating point-of-fixation driven metamers, which create images that preserve information in the fovea, yet lose spatial information in the periphery such that this loss is unnoticeable contingent of a point of fixation (Figure 1 ). The second issue is that the current state of the art for a full field of view rendering of a 512px × 512px metamer takes 6 hours for a grayscale image and roughly a day for a color image. This computational constraint makes data- There has been a recent surge in interest with regards to developing and testing new metamer models: The SideEye model developed by BID8 , uses a fully convolutional network (FCN) as in BID20 and learns to map an input image into a Texture Tiling Model (TTM) mongrel BID23 ). Their end-to-end model is also feedforward like ours, but no use of noise is incorporated in the generation pipeline making their model fully deterministic. At first glance this seems to be an advantage rather a limitation, however it limits the biological plausilibility of metameric response as the same input image should be able to create more than one metamer. Another model which has recently been proposed is the CNN synthesis model developed by BID29 . The CNN synthesis model is gradient-descent based and is closest in flavor to the FS model, with the difference that their texture statistics are provided by a gramian matrix of filter activations of multiple layers of a VGGNet, rather than those used in BID22 .The question of whether the scaling parameter is the only parameter to be optimized for metamerism still seems to be open. This has been questioned early in BID23 , and recently proposed and studied by BID29 , who suggest that metamers are driven by image content, rather than bouma's law (scaling factor). FIG3 suggests that on average, it does seem that α must increase in proportion to retinal eccentricity, but this is conditioned by the image content of each receptive field. We believe that the hyperparametric nature of our model sheds some light into reconciling these two theories. Recall that in FIG0 , we found that certain images can be pushed stronger in the direction of it's texturized version versus others given their location in the encoded space, the local geometry of the surface, and their projection in the perceptual space. This suggests that the average maximal distortion one can do is fixed contingent on the size of the receptive field, but we are allowed to push further (increase α) for some images more than others, because the direction of the distortion lies closer to the perceptual null space (making this difference perceptually un-noticeable to the human observer). This is usually the case for regions of images that are periodic like skies, or grass. Along the same lines, we elaborate in the Supplementary Material on how our model may potentially explain why creating synthesized samples are metameric to each other at the scales of (V1;V2), but only generated samples at the scale of V1 (s = 0.25) are metameric to the reference image.Our model is also different to others (FS and recently Wallis et al. FORMULA0 ) given the role of noise in the computational pipeline. The previously mentioned models used noise as an initial seed for the texture matching pipeline via gradient-descent, while we use noise as a proxy for texture distortion that is directly associated with crowding in the visual field. One could argue that the same response is achieved via both approaches, but our approach seems to be more biologically plausible at the algorithmic level. In our model an image is fed through a non-linear hierarchical system (simulated through a deep-net), and is corrupted by noise that matches the texture properties of the input image (via AdaIN). This perceptual representation is perturbed along the direction of the texture-matched patch for each receptive field, and inverting such perturbed representation results in a metamer. FIG7 illustrates such perturbations which produce metamers when projected to a 2D subspace via the locally linear embedding (LLE) algorithm (Roweis & Saul FORMULA1 ). Indeed, the 10 encoded images do not fully overlap to each other and they are quite distant as seen in the 2D projection. However, foveated representations when perturbed with texture-like noise seem to finely tile the perceptual space, and might act as a type of biological regularizer for human observers who are consistently making eye-movements when processing visual information. This suggests that robust representations might be achieved in the human visual system given its foveated nature as non-uniform high-resolution imagery does not map to the same point in perceptual space. If this holds, perceptually invariant data-augmentation schemes driven by metamerism may be a useful enhancement for artificial systems that react oddly to adversarial perturbations that exploit coarse perceptual mappings (Goodfellow et al. FORMULA0 ; BID26 ; Berardino et al. FORMULA0 ).Understanding the underlying representations of metamerism in the human visual system still remains a challenge. In this paper we propose a model that emulates metameric responses via a foveated feed-forward style transfer network. We find that correctly calibrating such perturbations (a consequence of internal noise that match texture representation) in the perceptual space and inverting such encoded representation results in a metamer. Though our model is hyper-parametric in nature we propose a way to reduce the parametrization via a perceptual optimization scheme. Via a psychophysical experiment we empirically find that the critical scaling factor also matches the rate of growth of the receptive fields in V2 (s = 0.5) as in BID7 when performing visual discrimination between synthesized metamers, and match V1 (0.25) for reference metamers similar to BID29 . Finally, while our choice of texture statistics and transfer is relu4_1 of a VGG19 and AdaIN respectively, our ×1000-fold accelerated feed-forward metamer generation pipeline should be extendible to other models that correctly compute texture/style statistics and transfer. This opens the door to rapidly generating multiple flavors of visual metamers with applications in neuroscience and computer vision.6 Supplementary Material FIG0 : Reference Metamers at the scale of s = 0.25, at which they are indiscriminable to the human observer. The color coding scheme matches the data points of the optimization in Experiment 1 and the psychophysics of Experiment 2. All images used in the experiments were generated originally at 512 × 512 px subtending 26 × 26 d.v.a (degrees of visual angle). for each α ∈ [0 : DISPLAYFORM0 Compute metamer M NF (I) 9:end for 10:Find the α for each receptive field that minimizes: E(∆-SSIM) 2 . 11:Fit the γ s (•) function to collection of α values. 12:endfor 13: end for 14: Perform Permutation test on γ s for all s. 15: if γ s is independent of s then 16: γ s = γ 17: else 18:Perform regression of parameters of γ s as a function f of s. 19: DISPLAYFORM1 end if 21: end procedure
We introduce a novel feed-forward framework to generate visual metamers Proposes a NeuroFovea model for generation of point-of-fixation metamers by using a style transfer approach via and Encoder-Decoder style architecture An analysis of metamerism and a model capable of rapidly producing metamers of value for experimental psychophysics and other domains. The paper proposes a fast method for generating visual metamers – physically different images that cannot be told apart from an original – via foveated, fast, arbitrary style transfer