Recent Colorings And Labelings In Topological Coding. Topological Coding consists of two different kinds of mathematics topological structure and mathematical relation. The colorings and labelings of graph theory are main techniques in topological coding applied in asymmetric encryption system. Topsnut gpws (also, colored graphs) have the following advantages (1) Run fast in communication networks because they are saved in computer by popular matrices rather than pictures. (2) Produce easily text based (number based) strings for encrypt files. (3) Diversity of asymmetric ciphers, one public key corresponds to more private keys, or more public keys correspond more private keys. (4) Irreversibility, Topsnut gpws can generate quickly text based (number based) strings with bytes as long as desired, but these strings can not reconstruct the original Topsnut gpws. (5) Computational security, since there are many non polynomial (NP complete, NP hard) algorithms in creating Topsnut gpws. (6) Provable security, since there are many mathematical conjectures (open problems) in graph labelings and graph colorings. We are committed to create more kinds of new Topsnut gpws to approximate practical applications and antiquantum computation, and try to use algebraic method and Topsnut gpws to establish graphic group, graphic lattice, graph homomorphism etc.


FedDQ Communication Efficient Federated Learning with Descending Quantization. Federated learning (FL) is an emerging privacy preserving distributed learning scheme. The large model size and frequent model aggregation cause serious communication issue for FL. Many techniques, such as model compression and quantization, have been proposed to reduce the communication volume. Existing adaptive quantization schemes use ascending trend quantization where the quantization level increases with the training stages. In this paper, we investigate the adaptive quantization scheme by maximizing the convergence rate. It is shown that the optimal quantization level is directly related to the range of the model updates. Given the model is supposed to converge with training, the range of model updates will get smaller, indicating that the quantization level should decrease with the training stages. Based on the theoretical analysis, a descending quantization scheme is thus proposed. Experimental results show that the proposed descending quantization scheme not only reduces more communication volume but also helps FL converge faster, when compared with current ascending quantization scheme.


Structure of the Nuttall partition for some class of four sheeted Riemann surfaces. The structure of a Nuttall partition into sheets of some class of four sheeted Riemann surfaces is studied. The corresponding class of multivalued analytic functions is a special class of algebraic functions of fourth order generated by the function inverse to the Zhukovskii function. We show that in this class of four sheeted Riemann surfaces, the boundary between the second and third sheets of the Nuttall partition of the Riemann surface, is completely characterized in terms of an extremal problem posed on the two sheeted Riemann surface of the function w defined by the equation w 2 z 2 1 . In particular, we show that in this class of functions the boundary between the second and third sheets does not intersect both the boundary between the first and second sheets and the boundary between the third and fourth sheets.


Theory of and Experiments on Minimally Invasive Stability Preservation in Changing Two Sided Matching Markets. Following up on purely theoretical work of Bredereck et al. AAAI 2020 , we contribute further theoretical insights into adapting stable two sided matchings to change. Moreover, we perform extensive empirical studies hinting at numerous practically useful properties. Our theoretical extensions include the study of new problems (that is, incremental variants of Almost Stable Marriage and Hospital Residents), focusing on their (parameterized) computational complexity and the equivalence of various change types (thus simplifying algorithmic and complexity theoretic studies for various natural change scenarios). Our experimental findings reveal, for instance, that allowing the new matching to be blocked by a few pairs significantly decreases the necessary differences between the old and the new stable matching.


A Companion Curve Tracing Method for Rank deficient Polynomial Systems. We propose a method for tracing implicit real algebraic curves defined by polynomials with rank deficient Jacobians. For a given curve f 1 (0) , it first utilizes a regularization technique to compute at least one witness point per connected component of the curve. We improve this step by establishing a sufficient condition for testing the emptiness of f 1 (0) . We also analyze the convergence rate and carry out an error analysis for refining the witness points. The witness points are obtained by computing the minimum distance of a random point to a smooth manifold embedding the curve while at the same time penalizing the residual of f at the local minima. To trace the curve starting from these witness points, we prove that if one drags the random point along a trajectory inside a tubular neighborhood of the embedded manifold of the curve, the projection of the trajectory on the manifold is unique and can be computed by numerical continuation. We then show how to choose such a trajectory to approximate the curve by computing eigenvectors of certain matrices. Effectiveness of the method is illustrated by examples.


Solvent Effects on the Menshutkin Reaction. The Menshutkin reaction is a methyl transfer reaction relevant in fields ranging from biochemistry to chemical synthesis. In the present work, energetics and solvent distributions for NH 3 MeCl and Pyr MeBr reactions were investigated in the gas phase, in water, methanol, acetonitrile, benzene, and in cyclohexane by means of reactive molecular dynamics simulations. For polar solvents (water, methanol, and acetonitrile) and benzene, strong to moderate catalytic effect for both reactions is found whereas apolar and bulky cyclohexane interacts weakly with the solute and does not show pronounced barrier reduction. Calculated barrier heights for the Pyr MeBr reaction in acetonitrile and cyclohexane are 23.2 and 28.1 kcal mol compared with experimentally measured barriers of 22.5 and 27.6 kcal mol, respectively. The 2 dimensional solvent distributions change considerably between reactant and TS but comparatively little between TS and product conformations of the solute. The simulations also suggest that as the system approaches the transition state, correlated solvent motions that destabilize the solvent solvent interactions are required to surmount the barrier. Finally, the average solvent solvent interaction energies in the reactant, TS, and product state geometries are correlated with changes in the solvent structure around the solute.


Graph Compression with Application to Model Selection. Many multivariate data such as social and biological data exhibit complex dependencies that are best characterized by graphs. Unlike sequential data, graphs are, in general, unordered structures. This means we can no longer use classic, sequential based compression methods on these graph based data. Therefore, it is necessary to develop new methods for graph compression. In this paper, we present universal source coding methods for the lossless compression of unweighted, undirected, unlabelled graphs. We encode in two steps 1) transforming graph into a rooted binary tree, 2) the encoding rooted binary tree using graph statistics. Our coders showed better compression performance than other source coding methods on both synthetic and real world graphs. We then applied our graph coding methods for model selection of Gaussian graphical models using minimum description length (MDL) principle finding the description length of the conditional independence graph. Experiments on synthetic data show that our approach gives better performance compared to common model selection methods. We also applied our approach to electrocardiogram (ECG) data in order to explore the differences between graph models of two groups of subjects.


A Large scale Study on Unsupervised Outlier Model Selection Do Internal Strategies Suffice?. Given an unsupervised outlier detection task, how should one select a detection algorithm as well as its hyperparameters (jointly called a model)? Unsupervised model selection is notoriously difficult, in the absence of hold out validation data with ground truth labels. Therefore, the problem is vastly understudied. In this work, we study the feasibility of employing internal model evaluation strategies for selecting a model for outlier detection. These so called internal strategies solely rely on the input data (without labels) and the output (outlier scores) of the candidate models. We setup (and open source) a large testbed with 39 detection tasks and 297 candidate models comprised of 8 detectors and various hyperparameter configurations. We evaluate 7 different strategies on their ability to discriminate between models w.r.t. detection performance, without using any labels. Our study reveals room for progress we find that none would be practically useful, as they select models only comparable to a state of the art detector (with random configuration).


A consistent BGK model with velocity dependent collision frequency for gas mixtures. We derive a multi species BGK model with velocity dependent collision frequency for a non reactive, multi component gas mixture. The model is derived by minimizing a weighted entropy under the constraint that the number of particles of each species, total momentum, and total energy are conserved. We prove that this minimization problem admits a unique solution for very general collision frequencies. Moreover, we prove that the model satisfies an H Theorem and characterize the form of equilibrium.


Some Combinatorial Problems in Power law Graphs. The power law behavior is ubiquitous in a majority of real world networks, and it was shown to have a strong effect on various combinatorial, structural, and dynamical properties of graphs. For example, it has been shown that in real life power law networks, both the matching number and the domination number are relatively smaller, compared with homogeneous graphs. In this paper, we study analytically several combinatorial problems for two power law graphs with the same number of vertices, edges, and the same power exponent. For both graphs, we determine exactly or recursively their matching number, independence number, domination number, the number of maximum matchings, the number of maximum independent sets, and the number of minimum dominating sets. We show that power law behavior itself cannot characterize the combinatorial properties of a heterogenous graph. Since the combinatorial properties studied here have found wide applications in different fields, such as structural controllability of complex networks, our work offers insight in the applications of these combinatorial problems in power law graphs.


BERT DRE BERT with Deep Recursive Encoder for Natural Language Sentence Matching. This paper presents a deep neural architecture, for Natural Language Sentence Matching (NLSM) by adding a deep recursive encoder to BERT so called BERT with Deep Recursive Encoder (BERT DRE). Our analysis of model behavior shows that BERT still does not capture the full complexity of text, so a deep recursive encoder is applied on top of BERT. Three Bi LSTM layers with residual connection are used to design a recursive encoder and an attention module is used on top of this encoder. To obtain the final vector, a pooling layer consisting of average and maximum pooling is used. We experiment our model on four benchmarks, SNLI, FarsTail, MultiNLI, SciTail, and a novel Persian religious questions dataset. This paper focuses on improving the BERT results in the NLSM task. In this regard, comparisons between BERT DRE and BERT are conducted, and it is shown that in all cases, BERT DRE outperforms BERT. The BERT algorithm on the religious dataset achieved an accuracy of 89.70 , and BERT DRE architectures improved to 90.29 using the same dataset.


Boosting Graph Search with Attention Network for Solving the General Orienteering Problem. Recently, several studies have explored the use of neural network to solve different routing problems, which is an auspicious direction. These studies usually design an encoder decoder based framework that uses encoder embeddings of nodes and the problem specific context to produce node sequence(path), and further optimize the produced result on top by beam search. However, existing models can only support node coordinates as input, ignore the self referential property of the studied routing problems, and lack the consideration about the low reliability in the initial stage of node selection, thus are hard to be applied in real world. In this paper, we take the orienteering problem as an example to tackle these limitations. We propose a novel combination of a variant beam search algorithm and a learned heuristic for solving the general orienteering problem. We acquire the heuristic with an attention network that takes the distances among nodes as input, and learn it via a reinforcement learning framework. The empirical studies show that our method can surpass a wide range of baselines and achieve results close to the optimal or highly specialized approach. Also, our proposed framework can be easily applied to other routing problems. Our code is publicly available.


Exploratory Data Analysis of Urdu Poetry. The study presented here provides numerical insight into ghazal the most appreciated genre in Urdu poetry. Using 48,761 poetic works from 4,754 poets produced over a period of 800 years, this study explores the main features of Urdu ghazal that make it popular and admired more than other forms. A detailed explanation is provided as to the types of words used for expressing love, nature, birds, and flowers etc. Also considered is the way in which the poets addressed their loved ones in their poetry. The style of poetry is numerically analyzed using Multi Dimensional Scaling to reveal the lexical diversity and similarities differences between the different poetic works that have drawn the attention of critics, such as Iqbal and Ghalib, Mir Taqi Mir and Mir Dard. The analysis produced here is particularly helpful for research in computational stylistics, neurocognitive poetics, and sentiment analysis.


Using Shape Constraints for Improving Symbolic Regression Models. We describe and analyze algorithms for shape constrained symbolic regression, which allows the inclusion of prior knowledge about the shape of the regression function. This is relevant in many areas of engineering in particular whenever a data driven model obtained from measurements must have certain properties (e.g. positivity, monotonicity or convexity concavity). We implement shape constraints using a soft penalty approach which uses multi objective algorithms to minimize constraint violations and training error. We use the non dominated sorting genetic algorithm (NSGA II) as well as the multi objective evolutionary algorithm based on decomposition (MOEA D). We use a set of models from physics textbooks to test the algorithms and compare against earlier results with single objective algorithms. The results show that all algorithms are able to find models which conform to all shape constraints. Using shape constraints helps to improve extrapolation behavior of the models.


Two Face Adversarial Audit of Commercial Face Recognition Systems. Computer vision applications like automated face detection are used for a variety of purposes ranging from unlocking smart devices to tracking potential persons of interest for surveillance. Audits of these applications have revealed that they tend to be biased against minority groups which result in unfair and concerning societal and political outcomes. Despite multiple studies over time, these biases have not been mitigated completely and have in fact increased for certain tasks like age prediction. While such systems are audited over benchmark datasets, it becomes necessary to evaluate their robustness for adversarial inputs. In this work, we perform an extensive adversarial audit on multiple systems and datasets, making a number of concerning observations there has been a drop in accuracy for some tasks on CELEBSET dataset since a previous audit. While there still exists a bias in accuracy against individuals from minority groups for multiple datasets, a more worrying observation is that these biases tend to get exorbitantly pronounced with adversarial inputs toward the minority group. We conclude with a discussion on the broader societal impacts in light of these observations and a few suggestions on how to collectively deal with this issue.


Spectrum of the Lam ' e operator along mathrm Re tau 1 2 The genus 3 case. In this paper, we study the spectrum sigma(L) of the Lam ' e operator begin equation L frac d 2 dx 2 12 wp(x z 0 tau) quad text in L 2( mathbb R , mathbb C ), end equation where wp(z tau) is the Weierstrass elliptic function with periods 1 and tau , and z 0 in mathbb C is chosen such that L has no singularities on mathbb R . We prove that a point lambda in sigma(L) is an intersection point of different spectral arcs but not a zero of the spectral polynomial if and only if lambda is a zero of the following cubic polynomial begin equation frac 4 15 lambda 3 frac 8 5 eta 1 lambda 2 3g 2 lambda 9g 3 6 eta 1 g 2 0. end equation We also study the deformation of the spectrum as tau frac 1 2 ib with b 0 varying. We discover 7 different types of graphs for the spectrum as b varies around the double zeros of the spectral polynomial.


Weakly Supervised Named Entity Tagging with Learnable Logical Rules. We study the problem of building entity tagging systems by using a few rules as weak supervision. Previous methods mostly focus on disambiguation entity types based on contexts and expert provided rules, while assuming entity spans are given. In this work, we propose a novel method TALLOR that bootstraps high quality logical rules to train a neural tagger in a fully automated manner. Specifically, we introduce compound rules that are composed from simple rules to increase the precision of boundary detection and generate more diverse pseudo labels. We further design a dynamic label selection strategy to ensure pseudo label quality and therefore avoid overfitting the neural tagger. Experiments on three datasets demonstrate that our method outperforms other weakly supervised methods and even rivals a state of the art distantly supervised tagger with a lexicon of over 2,000 terms when starting from only 20 simple rules. Our method can serve as a tool for rapidly building taggers in emerging domains and tasks. Case studies show that learned rules can potentially explain the predicted entities.


Lecture Notes on Support Preconditioning. These are lectures notes on support preconditioning, originally written in 2007. The notes cover large sparse linear systems of equations, krylov subspace solvers for symmetric diagonally dominant linear systems, sparse Cholesky, support theory, Laplacians, and augmented maximum spanning tree preconditioning.


How Morphological Computation shapes Integrated Information in Embodied Agents. The Integrated Information Theory provides a quantitative approach to consciousness and can be applied to neural networks. An embodied agent controlled by such a network influences and is being influenced by its environment. This involves, on the one hand, morphological computation within goal directed action and, on the other hand, integrated information within the controller, the agent's brain. In this article, we combine different methods in order to examine the information flows among and within the body, the brain and the environment of an agent. This allows us to relate various information flows to each other. We test this framework in a simple experimental setup. There, we calculate the optimal policy for goal directed behavior based on the "planning as inference" method, in which the information geometric em algorithm is used to optimize the likelihood of the goal. Morphological computation and integrated information are then calculated with respect to the optimal policies. Comparing the dynamics of these measures under changing morphological circumstances highlights the antagonistic relationship between these two concepts. The more morphological computation is involved, the less information integration within the brain is required. In order to determine the influence of the brain on the behavior of the agent it is necessary to additionally measure the information flow to and from the brain.


Multiway Storage Modification Machines. We present a parallel version of Sch "onhage's Storage Modification Machine, the Multiway Storage Modification Machine (MWSMM). Like the alternative Association Storage Modification Machine of Tromp and van Emde Boas, MWSMMs recognize in polynomial time what Turing Machines recognize in polynomial space. Falling thus into the Second Machine Class, the MWSMM is a parallel machine model conforming to the Parallel Computation Thesis. We illustrate MWSMMs by a simple implementation of Wolfram's String Substitution System.


ModelingToolkit A Composable Graph Transformation System For Equation Based Modeling. Getting good performance out of numerical equation solvers requires that the user has provided stable and efficient functions representing their model. However, users should not be trusted to write good code. In this manuscript we describe ModelingToolkit (MTK), a symbolic equation based modeling system which allows for composable transformations to generate stable, efficient, and parallelized model implementations. MTK blurs the lines of traditional symbolic computing by acting directly on a user's numerical code. We show the ability to apply graph algorithms for automatically parallelizing and performing index reduction on code written for differential algebraic equation (DAE) solvers, "fixing" the performance and stability of the model without requiring any changes to on the user's part. We demonstrate how composable model transformations can be combined with automated data driven surrogate generation techniques, allowing machine learning methods to generate accelerated approximate models within an acausal modeling framework. These reduced models are shown to outperform the Dymola Modelica compiler on an HVAC model by 590x at 3 error. Together, this demonstrates MTK as a system for bringing the latest research in graph transformations directly to modeling applications.


Provable Limitations of Acquiring Meaning from Ungrounded Form What Will Future Language Models Understand?. Language models trained on billions of tokens have recently led to unprecedented results on many NLP tasks. This success raises the question of whether, in principle, a system can ever understand'' raw text without access to some form of grounding. We formally investigate the abilities of ungrounded systems to acquire meaning. Our analysis focuses on the role of assertions'' textual contexts that provide indirect clues about the underlying semantics. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation of languages that satisfy a strong notion of semantic transparency. However, for classes of languages where the same expression can take different values in different contexts, we show that emulation can become uncomputable. Finally, we discuss differences between our formal model and natural language, exploring how our results generalize to a modal setting and other semantic relations. Together, our results suggest that assertions in code or language do not provide sufficient signal to fully emulate semantic representations. We formalize ways in which ungrounded language models appear to be fundamentally limited in their ability to understand''.


Analogues of Morozov Theorem in characteristic p 0. Let G be a reductive group over an algebraically closed field of positive characteristic. In this article we show an analogue for Morozov theorem for characteristics that are separably good for G (and under additional hypotheses on the group). This theorem characterises, when p 0, subalgebras of parabolic subgroups of G with respect to their nilradical. If now k is an arbitrary field, let C be a smooth projective and geometrically connected curve. Let G be a reductive C group scheme, which is the twisted form of a constant C group. The aforementioned analogue is in particular useful to extend the construction of the canonical parabolic subgroup of G proposed by M. Atiyah and R. Bott when k is of characteristic 0, to the characteristic p 0 framework.


Spinning Pupil Aberration Measurement for anisoplanatic deconvolution. The aberrations in an optical microscope are commonly measured and corrected at one location in the field of view, within the so called isoplanatic patch. Full field correction is desirable for high resolution imaging of large specimens. Here we present a novel wavefront detector, based on pupil sampling with sub apertures, which measures the aberrated wavefront phase at each position of the specimen. Based on this measurement, we propose a region wise deconvolution that provides an anisoplanatic reconstruction of the sample image. Our results indicate that the measurement and correction of the aberrations can be performed in a wide field fluorescence microscope over its entire field of view.


The structure of conservative gradient fields. The classical Clarke subdifferential alone is inadequate for understanding automatic differentiation in nonsmooth contexts. Instead, we can sometimes rely on enlarged generalized gradients called "conservative fields", defined through the natural path wise chain rule one application is the convergence analysis of gradient based deep learning algorithms. In the semi algebraic case, we show that all conservative fields are in fact just Clarke subdifferentials plus normals of manifolds in underlying Whitney stratifications.


A Variational Inequality Model for the Construction of Signals from Inconsistent Nonlinear Equations. Building up on classical linear formulations, we posit that a broad class of problems in signal synthesis and in signal recovery are reducible to the basic task of finding a point in a closed convex subset of a Hilbert space that satisfies a number of nonlinear equations involving firmly nonexpansive operators. We investigate this formalism in the case when, due to inaccurate modeling or perturbations, the nonlinear equations are inconsistent. A relaxed formulation of the original problem is proposed in the form of a variational inequality. The properties of the relaxed problem are investigated and a provenly convergent block iterative algorithm, whereby only blocks of the underlying firmly nonexpansive operators are activated at a given iteration, is devised to solve it. Numerical experiments illustrate robust recoveries in several signal and image processing applications.


Literature review on vulnerability detection using NLP technology. Vulnerability detection has always been the most important task in the field of software security. With the development of technology, in the face of massive source code, automated analysis and detection of vulnerabilities has become a current research hotspot. For special text files such as source code, using some of the hottest NLP technologies to build models and realize the automatic analysis and detection of source code has become one of the most anticipated studies in the field of vulnerability detection. This article does a brief survey of some recent new documents and technologies, such as CodeBERT, and summarizes the previous technologies.


Fundamental Challenges in Deep Learning for Stiff Contact Dynamics. Frictional contact has been extensively studied as the core underlying behavior of legged locomotion and manipulation, and its nearly discontinuous nature makes planning and control difficult even when an accurate model of the robot is available. Here, we present empirical evidence that learning an accurate model in the first place can be confounded by contact, as modern deep learning approaches are not designed to capture this non smoothness. We isolate the effects of contact's non smoothness by varying the mechanical stiffness of a compliant contact simulator. Even for a simple system, we find that stiffness alone dramatically degrades training processes, generalization, and data efficiency. Our results raise serious questions about simulated testing environments which do not accurately reflect the stiffness of rigid robotic hardware. Significant additional investigation will be necessary to fully understand and mitigate these effects, and we suggest several avenues for future study.


Note on Relation between Enhanced Ind Sheaves and Enhanced Subanalytic Sheaves. In this paper, we will explain a relation between Thm. 9.5.3, Andrea D'Agnolo and Masaki Kashiwara, Riemann Hilbert correspondence for holonomic D modules, 2016 and Thm. 6.3, Masaki Kashiwara, Riemann Hilbert correspondence for irregular holonomic D modules . For this purpose, we will also explain a relation between enhanced ind sheaves and enhanced subanalytic sheaves.


Radiation of Charge Moving along Face of Inverted Prism. Cherenkov radiation generated by a charge moving along one of the faces of a dielectric prism is analyzed. Unlike our previous works, here we suppose that the charge moves from the top of the prism to its base. We use the technique which was called by us "the aperture method". However here we develop the new version of this technique which is suitable for objects with plane faces we use inside the object only the expansion over plane waves. This approach is convenient for objects having plane borders especially in the case of two or more borders on which the waves are reflected and or refracted. Using this technique we obtain the field on the aperture and then apply Stretton Chu formulas (aperture integrals). Further, the main attention is paid to the calculation of the radiation field in the Fraunhofer (far field) area. We obtain the expressions for the Fourier transforms of the field components in form of the single integrals. Using them, the series of typical angular diagrams are computed and physical conclusions are made.


Bohr operator on opertor valued polyanalytic functions on simply connected domains. In this article, we study the Bohr operator for the operator valued subordination class S(f) consisting of holomorphic functions subordinate to f in the unit disk mathbb D z in mathbb C z 1 , where f mathbb D rightarrow mathcal B ( mathcal H ) is holomorphic and mathcal B ( mathcal H ) is the algebra of bounded linear operators on a complex Hilbert space mathcal H . We establish several subordination results, which can be viewed as the analogues of a couple of interesting subordination results from scalar valued settings. We also obtain a von Neumann type inequality for the class of self analytic mappings of the unit disk mathbb D which fix the origin. Furthermore, we extensively study Bohr inequalities for operator valued polyanalytic functions in certain proper simply connected domains in mathbb C . We obtain Bohr radius for the operator valued polyanalytic functions of the form F(z) sum l 0 p 1 overline z l , f l (z) , where f 0 is subordinate to an operator valued convex biholomorphic function, and operator valued starlike biholomorphic function in the unit disk mathbb D .


Topological and algebraic genericity and spaceability for an extended chain of sequence spaces. We examine topological and algebraic genericity and spaceability for any pair (X,Y) , X subset Y , X neq Y belonging to an extended chain of sequence spaces which contains the ell p spaces, 0 p leq infty .


On 1 absorbing prime hyperideal and some of its generalizations. Let R be a multiplicative hyperring. In this paper, we define the concept of 1 absorbing prime hyperideals which is a generalization of the prime hyperideals. Several properties of the hyperideals are provided. Moreover, we introduce the notions of 1 absorbing primary hyperideals, strongly 1 absorbing primary hyperideals and weakly 1 absorbing primary hyperideals which are generalizations of the 1 absorbing prime hyperideals and then we will show some properties of them.


Projective well orders and coanalytic witnesses. We further develop a forcing notion known as Coding with Perfect Trees and show that this poset preserves, in a strong sense, definable P points, definable tight MAD families and definable selective independent families. As a result, we obtain a model in which mathfrak a mathfrak u mathfrak i aleph 1 2 aleph 0 aleph 2 , each of mathfrak a , mathfrak u , mathfrak i has a Pi 1 1 witness and there is a Delta 1 3 well order of the reals. Note that both the complexity of the witnesses of the above combinatorial cardinal characteristics, as well as the complexity of the well order are optimal. In addition, we show that the existence of a Delta 1 3 well order of the reals is consistent with mathfrak c aleph 2 and each of the following mathfrak a mathfrak u mathfrak i , mathfrak a mathfrak i mathfrak u , mathfrak a mathfrak u mathfrak i , where the smaller cardinal characteristics have co analytic witnesses. Our methods allow the preservation of only sufficiently definable witnesses, which significantly differs from other preservation results of this type.


Duality for double iterated outer L p spaces. We study the double iterated outer L p spaces, namely the outer L p spaces associated with three exponents and defined on sets endowed with a measure and two outer measures. We prove that in the case of finite sets, under certain conditions between the outer measures, the double iterated outer L p spaces are isomorphic to Banach spaces uniformly in the cardinality of the set. We achieve this by showing the expected duality properties between them. We also provide counterexamples demonstrating that the uniformity does not hold in any arbitrary setting on finite sets, at least in a certain range of exponents. We prove the isomorphism to Banach spaces and the duality properties between the double iterated outer L p spaces also in the upper half 3 space infinite setting described by Uraltsev, going beyond the case of finite sets.


To Talk or to Work Delay Efficient Federated Learning over Mobile Edge Devices. Federated learning (FL), an emerging distributed machine learning paradigm, in conflux with edge computing is a promising area with novel applications over mobile edge devices. In FL, since mobile devices collaborate to train a model based on their own data under the coordination of a central server by sharing just the model updates, training data is maintained private. However, without the central availability of data, computing nodes need to communicate the model updates often to attain convergence. Hence, the local computation time to create local model updates along with the time taken for transmitting them to and from the server result in a delay in the overall time. Furthermore, unreliable network connections may obstruct an efficient communication of these updates. To address these, in this paper, we propose a delay efficient FL mechanism that reduces the overall time (consisting of both the computation and communication latencies) and communication rounds required for the model to converge. Exploring the impact of various parameters contributing to delay, we seek to balance the trade off between wireless communication (to talk) and local computation (to work). We formulate a relation with overall time as an optimization problem and demonstrate the efficacy of our approach through extensive simulations.


A dynamic intermediate state limits the folding rate of a discontinuous two domain protein. Protein folding is an indispensable process for the majority of proteins after their synthesis from ribosomes in the cell. Most in vitro protein folding studies have focused on single domain proteins. Hence, it is important to understand the folding process of multi domain proteins, especially when domains are discontinuous. We choose the Maltose binding protein (MBP) as a model system. In particular, we studied a mutant of MBP that folds slowly. Here, using two and three color single molecule Foerster resonance energy transfer (smFRET) experiments, we study the refolding of both the domains and the interaction between the domains of DM MBP. Initial two color smFRET measurements of the N terminal domain (NTD) reveal the presence of a folding intermediate. The same folding intermediate is observed in measurements monitoring the C terminal domain (CTD) and the NTD CTD (N C) interface. The refolding intermediate is dynamic on the sub millisecond timescale. Quantitative analysis on underlying dynamic interconversions revealed a delay in NTD folding imposed by the entropic barrier being the primary cause for slow DM MBP folding. Moreover, CTD folds after NTD completes the folding. Using three color smFRET, we could show the NTD folds first and CTD later in a same protein. Molecular dynamic simulations for temperature induced unfolding on WT and DM MBP identify a folding nucleus in the NTD, which is rich in hydrophobic residues, and explains why the two mutations slow down the folding kinetics. In the presence of the bacterial Hsp60 chaperonin system GroEL ES, we observe that DM MBP is dynamic within the chaperonin cavity but the chaperonin limits the conformational space of substrate. Hence, confinement aids DM MBP in overcoming the entropic barrier. The study reports on the subtle tuning and co dependency for protein folding between two domains with a discontinuous arrangement.


Multi hierarchical Convolutional Network for Efficient Remote Photoplethysmograph Signal and Heart Rate Estimation from Face Video Clips. Heart beat rhythm and heart rate (HR) are important physiological parameters of the human body. This study presents an efficient multi hierarchical spatio temporal convolutional network that can quickly estimate remote physiological (rPPG) signal and HR from face video clips. First, the facial color distribution characteristics are extracted using a low level face feature Generation (LFFG) module. Then, the three dimensional (3D) spatio temporal stack convolution module (STSC) and multi hierarchical feature fusion module (MHFF) are used to strengthen the spatio temporal correlation of multi channel features. In the MHFF, sparse optical flow is used to capture the tiny motion information of faces between frames and generate a self adaptive region of interest (ROI) skin mask. Finally, the signal prediction module (SP) is used to extract the estimated rPPG signal. The experimental results on the three datasets show that the proposed network outperforms the state of the art methods.


Self similar growth fragmentation processes with types. In this paper, we are interested in self similar growth fragmentation processes with types. It roughly consists of two parts. In the first part, we investigate multitype versions of the self similar growth fragmentation processes introduced by Bertoin, therefore extending the signed case of arXiv 2101.02582 to finitely many types. Our main result in this direction describes the law of the spine in the multitype setting. We stress that our arguments only rely on the structure of the underlying Markov additive processes, and hence is more general than arXiv 2101.02582. In the second part, we study mathbb R d valued self similar growth fragmentation processes driven by an isotropic process. These can be seen as multitype growth fragmentation processes, where the set of types is the sphere mathbb S d 1 . We give the spinal description in this setting. Finally, we prove that such a family of processes shows up when slicing half space excursions with hyperplanes.


Large Deviations of Irreversible Processes. Time irreversible stochastic processes are frequently used in natural sciences to explain non equilibrium phenomena and to design efficient stochastic algorithms. Our main goal in this thesis is to analyse their dynamics by means of large deviation theory. We focus on processes that become deterministic in a certain limit, and characterize their fluctuations around that deterministic limit by Lagrangian rate functions. Our main techniques for establishing these characterizations rely on the connection between large deviations and Hamilton Jacobi equations. We sketch this connection with examples in the introductory parts of this thesis. The second part of the thesis is devoted to irreversible processes that are motivated from molecular motors, Markov chain Monte Carlo (MCMC) methods and stochastic slow fast systems. We characterize the asymptotic dynamics of molecular motors by Hamiltonians defined in terms of principal eigenvalue problems. From our results about the zig zag sampler used in MCMCs, we learn that maximal irreversibility corresponds to an optimal rate of convergence. In stochastic slow fast systems, our main theoretical contributions are techniques to work with the variational formulas of Hamiltonians that one encounters in mean field systems coupled to fast diffusions. In the final part of the thesis, we study a family of Fokker Planck equations whose solutions become singular in a certain limit. The associated gradient flow structures do not converge since the relative entropies diverge in the limit. To remedy this, we propose to work with a different variational formulation that takes fluxes into account, which is motivated by density flux large deviations.


On the Dimension of Kakeya Sets in the First Heisenberg Group. We define Kakeya sets in the Heisenberg group and show that the Heisenberg Hausdorff dimension of Kakeya sets in the first Heisenberg group is at least 3. This lower bound is sharp since, under our definition, the xoy plane is a Kakeya set with Heisenberg Hausdorff dimension 3.


On Single and Multiple Representations in Dense Passage Retrieval. The advent of contextualised language models has brought gains in search effectiveness, not just when applied for re ranking the output of classical weighting models such as BM25, but also when used directly for passage indexing and retrieval, a technique which is called dense retrieval. In the existing literature in neural ranking, two dense retrieval families have become apparent single representation, where entire passages are represented by a single embedding (usually BERT's CLS token, as exemplified by the recent ANCE approach), or multiple representations, where each token in a passage is represented by its own embedding (as exemplified by the recent ColBERT approach). These two families have not been directly compared. However, because of the likely importance of dense retrieval moving forward, a clear understanding of their advantages and disadvantages is paramount. To this end, this paper contributes a direct study on their comparative effectiveness, noting situations where each method under over performs w.r.t. each other, and w.r.t. a BM25 baseline. We observe that, while ANCE is more efficient than ColBERT in terms of response time and memory usage, multiple representations are statistically more effective than the single representations for MAP and MRR 10. We also show that multiple representations obtain better improvements than single representations for queries that are the hardest for BM25, as well as for definitional queries, and those with complex information needs.


The q ary antiprimitive BCH codes. It is well known that cyclic codes have efficient encoding and decoding algorithms. In recent years, antiprimitive BCH codes have attracted a lot of attention. The objective of this paper is to study BCH codes of this type over finite fields and analyse their parameters. Some lower bounds on the minimum distance of antiprimitive BCH codes are given. The BCH codes presented in this paper have good parameters in general, containing many optimal linear codes. In particular, two open problems about the minimum distance of BCH codes of this type are partially solved in this paper.


Dimension bounds of classes of interval orders. In this paper, we discuss the dimension of interval orders having a representation using n different interval lengths, and the dimension of interval orders which has a representation using length in 1, r .


Morse subgroups and boundaries of random right angled Coxeter groups. We study Morse subgroups and Morse boundaries of random right angled Coxeter groups in the Erd H o s R 'enyi model. We show that at densities below left( sqrt frac 1 2 epsilon right) sqrt frac log n n random right angled Coxeter groups almost surely have Morse hyperbolic surface subgroups. This implies their Morse boundaries contain embedded circles and they cannot be quasi isometric to a right angled Artin group. Further, at densities above left( sqrt frac 1 2 epsilon right) sqrt frac log n n we show that, almost surely, the hyperbolic Morse special subgroups of a random right angled Coxeter group are virtually free. We also apply these methods to show that for a random graph Gamma at densities below (1 epsilon) sqrt frac log n n , square( Gamma) almost surely contains an isolated vertex. As a consequence, this provides infinitely many examples of right angled Coxeter groups with no one ended hyperbolic Morse special subgroups that are not quasi isometric to a right angled Artin group.


A strong version of Cobham's theorem. Let k, ell geq 2 be two multiplicatively independent integers. Cobham's famous theorem states that a set X subseteq mathbb N is both k recognizable and ell recognizable if and only if it is definable in Presburger arithmetic. Here we show the following strengthening let X subseteq mathbb N m be k recognizable, let Y subseteq mathbb N m be ell recognizable such that both X and Y are not definable in Presburger arithmetic. Then the first order logical theory of ( mathbb N , ,X,Y) is undecidable. This is in contrast to a well known theorem of B "uchi that the first order logical theory of ( mathbb N , ,X) is decidable.


MTH IDS A Multi Tiered Hybrid Intrusion Detection System for Internet of Vehicles. Modern vehicles, including connected vehicles and autonomous vehicles, nowadays involve many electronic control units connected through intra vehicle networks to implement various functionalities and perform actions. Modern vehicles are also connected to external networks through vehicle to everything technologies, enabling their communications with other vehicles, infrastructures, and smart devices. However, the improving functionality and connectivity of modern vehicles also increase their vulnerabilities to cyber attacks targeting both intra vehicle and external networks due to the large attack surfaces. To secure vehicular networks, many researchers have focused on developing intrusion detection systems (IDSs) that capitalize on machine learning methods to detect malicious cyber attacks. In this paper, the vulnerabilities of intra vehicle and external networks are discussed, and a multi tiered hybrid IDS that incorporates a signature based IDS and an anomaly based IDS is proposed to detect both known and unknown attacks on vehicular networks. Experimental results illustrate that the proposed system can detect various types of known attacks with 99.99 accuracy on the CAN intrusion dataset representing the intra vehicle network data and 99.88 accuracy on the CICIDS2017 dataset illustrating the external vehicular network data. For the zero day attack detection, the proposed system achieves high F1 scores of 0.963 and 0.800 on the above two datasets, respectively. The average processing time of each data packet on a vehicle level machine is less than 0.6 ms, which shows the feasibility of implementing the proposed system in real time vehicle systems. This emphasizes the effectiveness and efficiency of the proposed IDS.


Local Clustering in Contextual Multi Armed Bandits. We study identifying user clusters in contextual multi armed bandits (MAB). Contextual MAB is an effective tool for many real applications, such as content recommendation and online advertisement. In practice, user dependency plays an essential role in the user's actions, and thus the rewards. Clustering similar users can improve the quality of reward estimation, which in turn leads to more effective content recommendation and targeted advertising. Different from traditional clustering settings, we cluster users based on the unknown bandit parameters, which will be estimated incrementally. In particular, we define the problem of cluster detection in contextual MAB, and propose a bandit algorithm, LOCB, embedded with local clustering procedure. And, we provide theoretical analysis about LOCB in terms of the correctness and efficiency of clustering and its regret bound. Finally, we evaluate the proposed algorithm from various aspects, which outperforms state of the art baselines.


Committee selection in DAG distributed ledgers and applications. In this paper, we propose several solutions to the committee selection problem among participants of a DAG distributed ledger. Our methods are based on a ledger intrinsic reputation model that serves as a selection criterion. The main difficulty arises from the fact that the DAG ledger is a priori not totally ordered and that the participants need to reach a consensus on participants' reputation. Furthermore, we outline applications of the proposed protocols, including (i) self contained decentralized random number beacon (ii) selection of oracles in smart contracts (iii) applications in consensus protocols and sharding solutions. We conclude with a discussion on the security and liveness of the proposed protocols by modeling reputation with a Zipf law.


Restricted Positional Games. A positional game is a game where two players sequentially label vertices of a hypergraph, consisting of a board and a collection of winning sets, with colors assigned to each player until all vertices of the board are claimed. The first player to claim all elements of a winning set wins. If no player claims all the elements of a winning set, then the game results in a draw. One such example of a positional game is Tic Tac Toe, where the board is the 3 by 3 grid. The popular game of Connect 4 is an example of what we define to be a "restricted positional game". Here, we introduce another example of a restricted positional game, Connect Tac Toe, which additionally has the influence of gravity like restrictions to affect what plays are possible. It is a generalization of the Connect 4 game for the hypercube. We define a variant of the Hales Jewett number for this game as a way to classify its restrictiveness and provide a logarithmic lower bound for this number.


What Matters for Adversarial Imitation Learning?. Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies. It is therefore difficult to discuss and understand what choices, among the high level algorithmic options as well as low level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning framework and investigate their impacts in a large scale study ( 500k trained agents) with both synthetic and human generated demonstrations. While many of our findings confirm common practices, some of them are surprising or even contradict prior work. In particular, our results suggest that artificial demonstrations are not a good proxy for human data and that the very common practice of evaluating imitation algorithms only with synthetic demonstrations may lead to algorithms which perform poorly in the more realistic scenarios with human demonstrations.


Bolytrope orders. Bolytropes are bounded subsets of an affine building that consist of all points that have distance at most r from some polytrope. We prove that the points of a bolytrope describe the set of all invariant lattices of a bolytrope order, generalizing the correspondence between polytropes and graduated orders.


A NIR to VIS face recognition via part adaptive and relation attention module. In the face recognition application scenario, we need to process facial images captured in various conditions, such as at night by near infrared (NIR) surveillance cameras. The illumination difference between NIR and visible light (VIS) causes a domain gap between facial images, and the variations in pose and emotion also make facial matching more difficult. Heterogeneous face recognition (HFR) has difficulties in domain discrepancy, and many studies have focused on extracting domain invariant features, such as facial part relational information. However, when pose variation occurs, the facial component position changes, and a different part relation is extracted. In this paper, we propose a part relation attention module that crops facial parts obtained through a semantic mask and performs relational modeling using each of these representative features. Furthermore, we suggest component adaptive triplet loss function using adaptive weights for each part to reduce the intra class identity regardless of the domain as well as pose. Finally, our method exhibits a performance improvement in the CASIA NIR VIS 2.0 and achieves superior result in the BUAA VisNir with large pose and emotion variations.


Date Estimation in the Wild of Scanned Historical Photos An Image Retrieval Approach. This paper presents a novel method for date estimation of historical photographs from archival sources. The main contribution is to formulate the date estimation as a retrieval task, where given a query, the retrieved images are ranked in terms of the estimated date similarity. The closer are their embedded representations the closer are their dates. Contrary to the traditional models that design a neural network that learns a classifier or a regressor, we propose a learning objective based on the nDCG ranking metric. We have experimentally evaluated the performance of the method in two different tasks date estimation and date sensitive image retrieval, using the DEW public database, overcoming the baseline methods.


Arbitrary order pressure robust DDR and VEM methods for the Stokes problem on polyhedral meshes. This paper contains two major contributions. First we derive, following the discrete de Rham (DDR) and Virtual Element (VEM) paradigms, pressure robust methods for the Stokes equations that support arbitrary orders and polyhedral meshes. Unlike other methods presented in the literature, pressure robustness is achieved here without resorting to an boldsymbol H ( rm div ) conforming construction on a submesh, but rather projecting the volumetric force onto the discrete boldsymbol H ( bf curl ) space. The cancellation of the pressure error contribution stems from key commutation properties of the underlying DDR and VEM complexes. The pressure robust error estimates in h k 1 (with h denoting the meshsize and k ge 0 the polynomial degree of the DDR or VEM complex) are proven theoretically and supported by a panel of three dimensional numerical tests. The second major contribution of the paper is an in depth study of the relations between the DDR and VEM approaches. We show, in particular, that a complex developed following one paradigm admits a reformulation in the other, and that couples of related DDR and VEM complexes satisfy commuting diagram properties with the degrees of freedom maps.


Towards affinoid Duflo Theorem I Twisted differential operators. For a commutative ring R , we define the notions of deformed Picard algebroids and deformed twisted differential operators on a smooth, separated, locally of finite type R scheme and prove these are in a natural bijection. We then define the pullback of a sheaf of twisted differential operators that reduces to the classical definition when R mathbb C . Finally, for modules over twisted differential operators, we prove a theorem for the descent under a locally trivial torsor.


An Upgrade Path for the Fermilab Accelerator Complex. The completion of the PIP II project and its superconducting linear accelerator will provide up to 1.2 MW of beam power to the LBNF DUNE facility for neutrino physics. It will also be able to produce high power beams directly from the linac that can be used for lower energy particle physics experiments as well, such as directing beam toward the Muon Campus at Fermilab for example. Any further significant upgrade of the beam power to DUNE, however, will be impeded by the limitations of the present Booster synchrotron at the facility. To increase the power to DUNE by a factor of two would require a new accelerator arrangement to feed the Main Injector that does not include the Booster. In what follows, a path toward upgrading the Fermilab accelerator complex to bring the beam power for DUNE to 2.4 MW is presented, using a new rapid cycling synchrotron plus an energy upgrade to the PIP II linac. The path includes the ability to instigate a new lower energy, very high power beam delivery system for experiments that can address much of the science program presented by the Booster Replacement Science Working Group. It also allows for the future possibility to go beyond 2.4 MW up to roughly 4 MW from the Main Injector.


SoK Cryptographic Confidentiality of Data on Mobile Devices. Mobile devices have become an indispensable component of modern life. Their high storage capacity gives these devices the capability to store vast amounts of sensitive personal data, which makes them a high value target these devices are routinely stolen by criminals for data theft, and are increasingly viewed by law enforcement agencies as a valuable source of forensic data. Over the past several years, providers have deployed a number of advanced cryptographic features intended to protect data on mobile devices, even in the strong setting where an attacker has physical access to a device. Many of these techniques draw from the research literature, but have been adapted to this entirely new problem setting. This involves a number of novel challenges, which are incompletely addressed in the literature. In this work, we outline those challenges, and systematize the known approaches to securing user data against extraction attacks. Our work proposes a methodology that researchers can use to analyze cryptographic data confidentiality for mobile devices. We evaluate the existing literature for securing devices against data extraction adversaries with powerful capabilities including access to devices and to the cloud services they rely on. We then analyze existing mobile device confidentiality measures to identify research areas that have not received proper attention from the community and represent opportunities for future research.


Mapping class groups of simply connected high dimensional manifolds need not be arithmetic. It is well known that Sullivan showed that the mapping class group of a simply connected high dimensional manifold is commensurable with an arithmetic group, but the meaning of "commensurable" in this statement seems to be less well known. We explain why this result fails with the now standard definition of commensurability by exhibiting a manifold whose mapping class group is not residually finite. We do not suggest any problem with Sullivan's result rather we provide a gloss for it.


A lifting of the Goulden Jackson cluster method to the Malvenuto Reutenauer algebra. The Goulden Jackson cluster method is a powerful tool for counting words by occurrences of prescribed subwords, and was adapted by Elizalde and Noy for counting permutations by occurrences of prescribed consecutive patterns. In this paper, we lift the cluster method for permutations to the Malvenuto Reutenauer algebra. Upon applying standard homomorphisms, our result specializes to both the cluster method for permutations as well as a q analogue which keeps track of the inversion number statistic. We construct additional homomorphisms using the theory of shuffle compatibility, leading to further specializations which keep track of various "inverse statistics", including the inverse descent number, inverse peak number, and inverse left peak number. This approach is then used to derive formulas for counting permutations by occurrences of two families of consecutive patterns monotone patterns and transpositional patterns refined by these statistics.


SAIS Supervising and Augmenting Intermediate Steps for Document Level Relation Extraction. Stepping from sentence level to document level, the research on relation extraction (RE) confronts increasing text length and more complicated entity interactions. Consequently, it is more challenging to encode the key information sources relevant contexts and entity types. However, existing methods only implicitly learn to model these critical information sources while being trained for RE. As a result, they suffer the problems of ineffective supervision and uninterpretable model predictions. In contrast, we propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for RE. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately so as to enhance interpretability. By assessing model uncertainty, SAIS further boosts the performance via evidence based data augmentation and ensemble inference while reducing the computational cost. Eventually, SAIS delivers state of the art RE results on three benchmarks (DocRED, CDR, and GDA) and outperforms the runner up by 5.04 relatively in F1 score in evidence retrieval on DocRED.


Document AI Benchmarks, Models and Applications. Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques for automatically reading, understanding, and analyzing business documents. It is an important research direction for natural language processing and computer vision. In recent years, the popularity of deep learning technology has greatly advanced the development of Document AI, such as document layout analysis, visual information extraction, document visual question answering, document image classification, etc. This paper briefly reviews some of the representative models, tasks, and benchmark datasets. Furthermore, we also introduce early stage heuristic rule based document analysis, statistical machine learning algorithms, and deep learning approaches especially pre training methods. Finally, we look into future directions for Document AI research.


Maximal independent sets in clique free graphs. Nielsen proved that the maximum number of maximal independent sets (MIS's) of size k in an n vertex graph is asymptotic to (n k) k , with the extremal construction a disjoint union of k cliques with sizes as close to n k as possible. In this paper we study how many MIS's of size k an n vertex graph G can have if G does not contain a clique K t . We prove for all fixed k and t that there exist such graphs with n lfloor frac (t 2)k t 1 rfloor o(1) MIS's of size k by utilizing recent work of Gowers and B. Janzer on a generalization of the Ruzsa Szemer 'edi problem. We prove that this bound is essentially best possible for triangle free graphs when k le 4 .


Sramm short read alignment mapping metrics. Short Read Alignment Mapping Metrics (SRAMM) is an efficient and versatile command line tool providing additional short read mapping metrics, filtering, and graphs. Short read aligners report MAPing Quality (MAPQ), but these methods generally are neither standardized nor well described in literature or software manuals. Additionally, third party mapping quality programs are typically computationally intensive or designed for specific applications. SRAMM efficiently generates multiple different concept based mapping scores to provide for an informative post alignment examination and filtering process of aligned short reads for various downstream applications. SRAMM is compatible with Python 2.6 and Python 3.6 on all operating systems. It works with any short read aligner that generates SAM BAM CRAM file outputs and reports 'AS' tags. It is freely available under the MIT license at http github.com achon sramm.


Formalizing Distribution Inference Risks. Property inference attacks reveal statistical properties about a training set but are difficult to distinguish from the primary purposes of statistical machine learning, which is to produce models that capture statistical properties about a distribution. Motivated by Yeom et al.'s membership inference framework, we propose a formal and generic definition of property inference attacks. The proposed notion describes attacks that can distinguish between possible training distributions, extending beyond previous property inference attacks that infer the ratio of a particular type of data in the training data set. In this paper, we show how our definition captures previous property inference attacks as well as a new attack that reveals the average degree of nodes of a training graph and report on experiments giving insight into the potential risks of property inference attacks.


Introducing and assessing the explainable AI (XAI)method SIDU. Explainable Artificial Intelligence (XAI) has in recent years become a well suited framework to generate human understandable explanations of black box models. In this paper, we present a novel XAI visual explanation algorithm denoted SIDU that can effectively localize entire object regions responsible for prediction in a full extend. We analyze its robustness and effectiveness through various computational and human subject experiments. In particular, we assess the SIDU algorithm using three different types of evaluations (Application, Human and Functionally Grounded) to demonstrate its superior performance. The robustness of SIDU is further studied in presence of adversarial attack on black box models to better understand its performance.


A description of values of Seifert form for punctured n manifolds in (2n 1) space. We study Seifert linking form which is an invariant of embeddings of punctured n manifolds in mathbb R 2n 1 . For punctured n manifold N 0 the values of this invariant are integer valued bilinear symmetric forms on H n 1 (N 0 mathbb Z) . We prove that value modulo two of this invariant at x, y in H n 1 (N 0 mathbb Z) equals mathrm PD bar w n 2 (N 0) cap rho 2x cap rho 2y , where mathrm PD bar w n 2 (N 0) is Poincare dual to Steifel Whitney class. We also prove that any such form can be realized by some embedding N 0 to mathbb R 2n 1 . Also, we survey known results on classification of embeddings of connected manifolds with non empty boundary.


Git Clustering Based on Graph of Intensity Topology. textbf A ccuracy, textbf R obustness to noises and scales, textbf I nterpretability, textbf S peed, and textbf E asy to use (ARISE) are crucial requirements of a good clustering algorithm. However, achieving these goals simultaneously is challenging, and most advanced approaches only focus on parts of them. Towards an overall consideration of these aspects, we propose a novel clustering algorithm, namely GIT (Clustering Based on textbf G raph of textbf I ntensity textbf T opology). GIT considers both local and global data structures firstly forming local clusters based on intensity peaks of samples, and then estimating the global topological graph (topo graph) between these local clusters. We use the Wasserstein Distance between the predicted and prior class proportions to automatically cut noisy edges in the topo graph and merge connected local clusters as final clusters. Then, we compare GIT with seven competing algorithms on five synthetic datasets and nine real world datasets. With fast local cluster detection, robust topo graph construction and accurate edge cutting, GIT shows attractive ARISE performance and significantly exceeds other non convex clustering methods. For example, GIT outperforms its counterparts about 10 (F1 score) on MNIST and FashionMNIST. Code is available at color red https github.com gaozhangyang GIT .


Channel Estimation for Intelligent Reflecting Surface Assisted Wireless Communications. In this paper, the minimum mean square error (MMSE) channel estimation for intelligent reflecting surface (IRS) assisted wireless communication systems is investigated. In the considered setting, each row vector of the equivalent channel matrix from the base station (BS) to the users is shown to be Bessel K distributed, and all these row vectors are independent of each other. By introducing a Gaussian scale mixture model, we obtain a closed form expression for the MMSE estimate of the equivalent channel, and determine analytical upper and lower bounds on the mean square error. Using the central limit theorem, we conduct an asymptotic analysis of the MMSE estimate, and show that the upper bound on the mean square error of the MMSE estimate is equal to the asymptotic mean square error of the MMSE estimation when the number of reflecting elements at the IRS tends to infinity. Numerical simulations show that the gap between the upper and lower bounds are very small, and they almost overlap with each other at medium signal to noise ratio (SNR) levels and moderate number of elements at the IRS.


ALFRED Virtual Memory for Intermittent Computing. We present ALFRED a virtual memory abstraction that resolves the dichotomy between volatile and non volatile memory in intermittent computing. Mixed volatile microcontrollers allow programmers to allocate part of the application state onto non volatile main memory. Programmers are therefore to explore manually the trade off between simpler management of persistent state against the energy overhead for non volatile memory operations and intermittence anomalies due to re execution of non idempotent code. This approach is laborious and yields sub optimal performance. We take a different stand with ALFRED we provide programmers with a virtual memory abstraction detached from the specific volatile nature of memory and automatically determine an efficient mapping from virtual to volatile or non volatile memory. Unlike existing works, ALFRED does not require programmers to learn a new programming model or language syntax, while the mapping is entirely resolved at compile time, reducing the run time energy overhead. We implement ALFRED through a series of program machine level code transformations. Compared to existing systems, we demonstrate that ALFRED reduces energy consumption by up to two orders of magnitude given a fixed workload. This enables the workloads to finish sooner, as the use of available energy shifts from ensuring forward progress to useful application processing.


Passive Flow Control for Series Inflatable Actuators Application on a Wearable Soft Robot for Posture Assistance. This paper presents a passive control method for multiple degrees of freedom in a soft pneumatic robot through the combination of flow resistor tubes with series inflatable actuators. We designed and developed these 3D printed resistors based on the pressure drop principle of multiple capillary orifices, which allows a passive control of its sequential activation from a single source of pressure. Our design fits in standard tube connectors, making it easy to adopt it on any other type of actuator with pneumatic inlets. We present its characterization of pressure drop and evaluation of the activation sequence for series and parallel circuits of actuators. Moreover, we present an application for the assistance of postural transition from lying to sitting. We embedded it in a wearable garment robot suit designed for infants with cerebral palsy. Then, we performed the test with a dummy baby for emulating the upper body motion control. The results show a sequential motion control of the sitting and lying transitions validating the proposed system for flow control and its application on the robot suit.


Solving Traffic4Cast Competition with U Net and Temporal Domain Adaptation. In this technical report, we present our solution to the Traffic4Cast 2021 Core Challenge, in which participants were asked to develop algorithms for predicting a traffic state 60 minutes ahead, based on the information from the previous hour, in 4 different cities. In contrast to the previously held competitions, this year's challenge focuses on the temporal domain shift in traffic due to the COVID 19 pandemic. Following the past success of U Net, we utilize it for predicting future traffic maps. Additionally, we explore the usage of pre trained encoders such as DenseNet and EfficientNet and employ multiple domain adaptation techniques to fight the domain shift. Our solution has ranked third in the final competition. The code is available at https github.com jbr ai labs traffic4cast 2021.


Functional inequalities and applications to doubly nonlinear diffusion equations. We study weighted inequalities of Hardy and Hardy Poincar 'e type and find necessary and sufficient conditions on the weights so that the considered inequalities hold. Examples with the optimal constants are shown. Such inequalities are then used to quantify the convergence rate of solutions to doubly nonlinear fast diffusion equation towards the Barenblatt profile.


High Performance HZO InAlN GaN MIS HEMT with fT fmax of 155 250 GHz. Scaling of GaN high electron mobility transistors (HEMTs) usually increases gate leakage current and deteriorates breakdown characteristic, limiting the maximum drain current and output power density. These bottlenecks can be circumvented by inserting a dielectric material under the gate of HEMTs. Doped HfO2 is an excellent dielectric material but unexplored so far as the gate material of HEMTs for high speed device application. Here we demonstrate that Zr doped HfO2 (HZO) gated InAlN GaN metal insulator semiconductor (MIS) HEMTs exhibit remarkable properties. The device with a gate length (Lg) of 50 nm exhibits maximum drain current (Id,max) of 2.15 A mm, a transconductance (gm) peak of 476 mS mm, an on off current ratio (Ion Ioff) of 9.3 107, a low drain induced barrier lowing (DIBL) of 45 mV V. RF characterizations reveal a current gain cutoff frequency (fT) of 155 GHz and a maximum oscillation frequency (fmax) of 250 GHz, resulting in a (fT fmax)1 2 of 197 GHz.These properties, particularly the high (fT fmax)1 2 and JFOM are highly desirable for the millimeter wave power applications, demonstrating the great technological potential of HZO InAlN GaN MIS HEMTs.


On Guaranteed Optimal Robust Explanations for NLP Models. We build on abduction based explanations for ma chine learning and develop a method for computing local explanations for neural network models in natural language processing (NLP). Our explanations comprise a subset of the words of the in put text that satisfies two key features optimality w.r.t. a user defined cost function, such as the length of explanation, and robustness, in that they ensure prediction invariance for any bounded perturbation in the embedding space of the left out words. We present two solution algorithms, respectively based on implicit hitting sets and maximum universal subsets, introducing a number of algorithmic improvements to speed up convergence of hard instances. We show how our method can be con figured with different perturbation sets in the em bedded space and used to detect bias in predictions by enforcing include exclude constraints on biased terms, as well as to enhance existing heuristic based NLP explanation frameworks such as Anchors. We evaluate our framework on three widely used sentiment analysis tasks and texts of up to100words from SST, Twitter and IMDB datasets,demonstrating the effectiveness of the derived explanations.


Trans Encoder Unsupervised sentence pair modelling through self and mutual distillations. In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence pair tasks bi encoders and cross encoders. Bi encoders produce fixed dimensional sentence representations and are computationally efficient, however, they usually underperform cross encoders. Cross encoders can leverage their attention heads to exploit inter sentence interactions for better performance but they require task fine tuning and are computationally more expensive. In this paper, we present a completely unsupervised sentence representation model termed as Trans Encoder that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi and cross encoders. Specifically, on top of a pre trained Language Model (PLM), we start with converting it to an unsupervised bi encoder, and then alternate between the bi and cross encoder task formulations. In each alternation, one task formulation will produce pseudo labels which are used as learning signals for the other task formulation. We then propose an extension to conduct such self distillation approach on multiple PLMs in parallel and use the average of their pseudo labels for mutual distillation. Trans Encoder creates, to the best of our knowledge, the first completely unsupervised cross encoder and also a state of the art unsupervised bi encoder for sentence similarity. Both the bi encoder and cross encoder formulations of Trans Encoder outperform recently proposed state of the art unsupervised sentence encoders such as Mirror BERT and SimCSE by up to 5 on the sentence similarity benchmarks.


InfoPrint Embedding Information into 3D Printed Objects. We present a technique to embed information invisible to the eye inside 3D printed objects. The information is integrated in the object model, and then fabricated using off the shelf dual head FDM (Fused Deposition Modeling) 3D printers. Our process does not require human intervention during or after printing with the integrated model. The information can be arbitrary symbols, such as icons, text,binary, or handwriting. To retrieve the information, we evaluate two different infrared based imaging devices that are readily available thermal cameras and near infrared scanners. Based on our results, we propose design guidelines for a range of use cases to embed and extract hidden information. We demonstrate how our method can be used for different applications, such as interactive thermal displays, hidden board game tokens, tagging functional printed objects, and autographing non fungible fabrication work.


Adversarial Perturbations Are Not So Weird Entanglement of Robust and Non Robust Features in Neural Network Classifiers. Neural networks trained on visual data are well known to be vulnerable to often imperceptible adversarial perturbations. The reasons for this vulnerability are still being debated in the literature. Recently Ilyas et al. (2019) showed that this vulnerability arises, in part, because neural network classifiers rely on highly predictive but brittle "non robust" features. In this paper we extend the work of Ilyas et al. by investigating the nature of the input patterns that give rise to these features. In particular, we hypothesize that in a neural network trained in a standard way, non robust features respond to small, "non semantic" patterns that are typically entangled with larger, robust patterns, known to be more human interpretable, as opposed to solely responding to statistical artifacts in a dataset. Thus, adversarial examples can be formed via minimal perturbations to these small, entangled patterns. In addition, we demonstrate a corollary of our hypothesis robust classifiers are more effective than standard (non robust) ones as a source for generating transferable adversarial examples in both the untargeted and targeted settings. The results we present in this paper provide new insight into the nature of the non robust features responsible for adversarial vulnerability of neural network classifiers.


Jamming Resilient Path Planning for Multiple UAVs via Deep Reinforcement Learning. Unmanned aerial vehicles (UAVs) are expected to be an integral part of wireless networks. In this paper, we aim to find collision free paths for multiple cellular connected UAVs, while satisfying requirements of connectivity with ground base stations (GBSs) in the presence of a dynamic jammer. We first formulate the problem as a sequential decision making problem in discrete domain, with connectivity, collision avoidance, and kinematic constraints. We, then, propose an offline temporal difference (TD) learning algorithm with online signal to interference plus noise ratio (SINR) mapping to solve the problem. More specifically, a value network is constructed and trained offline by TD method to encode the interactions among the UAVs and between the UAVs and the environment and an online SINR mapping deep neural network (DNN) is designed and trained by supervised learning, to encode the influence and changes due to the jammer. Numerical results show that, without any information on the jammer, the proposed algorithm can achieve performance levels close to that of the ideal scenario with the perfect SINR map. Real time navigation for multi UAVs can be efficiently performed with high success rates, and collisions are avoided.


Quantitative form of Ball's Cube slicing in mathbb R n and equality cases in the min entropy power inequality. We prove a quantitative form of the celebrated Ball's theorem on cube slicing in mathbb R n and obtain, as a consequence, equality cases in the min entropy power inequality. Independently, we also give a quantitative form of Khintchine's inequality in the special case p 1 .


The Bridge Lemmas between Equivalent Fell Bundles and its Applications. In this paper, we prove that the induced representation theories of two equivalent Fell bundles are essentially identical and we apply our results to carry the induced representation theory and imprimitivity theorems of saturated Fell bundles to arbitrary Fell bundles.


Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search. Learning from mistakes is an effective learning approach widely used in human learning, where a learner pays greater focus on mistakes to circumvent them in the future to improve the overall learning outcomes. In this work, we aim to investigate how effectively we can leverage this exceptional learning ability to improve machine learning models. We propose a simple and effective multi level optimization framework called learning from mistakes using class weighting (LFM CW), inspired by mistake driven learning to train better machine learning models. In this formulation, the primary objective is to train a model to perform effectively on target tasks by using a re weighting technique. We learn the class weights by minimizing the validation loss of the model and re train the model with the synthetic data from the image generator weighted by class wise performance and real data. We apply our LFM CW framework with differential architecture search methods on image classification datasets such as CIFAR and ImageNet, where the results show that our proposed strategy achieves lower error rate than the baselines.


The Fitting height of finite groups with a fixed point free automorphism satisfying an identity. Motivated by classic theorems of Thompson and Berger on the Fitting height of finite groups with a fixed point free automorphism of coprime order, we conjecture that, for every non zero polynomial f(x) a 0 a 1 x cdots a d x d in mathbb Z x , there is an integer k 0 with the following property. Let G be a finite (solvable) group with a fixed point free automorphism alpha satisfying gcd( G ,k) 1 and g a 0 cdot alpha(g) a 1 cdot alpha 2(g) a 2 cdots alpha d(g) a d g in G 1 . Then the Fitting height of G is at most the number of irreducible factors of f(x) . We confirm the conjecture for a large family of polynomials with explicit constants k .


Deep Semi Supervised Learning for Time Series Classification. While Semi supervised learning has gained much attention in computer vision on image data, yet limited research exists on its applicability in the time series domain. In this work, we investigate the transferability of state of the art deep semi supervised models from image to time series classification. We discuss the necessary model adaptations, in particular an appropriate model backbone architecture and the use of tailored data augmentation strategies. Based on these adaptations, we explore the potential of deep semi supervised learning in the context of time series classification by evaluating our methods on large public time series classification problems with varying amounts of labelled samples. We perform extensive comparisons under a decidedly realistic and appropriate evaluation scheme with a unified reimplementation of all algorithms considered, which is yet lacking in the field. We find that these transferred semi supervised models show significant performance gains over strong supervised, semi supervised and self supervised alternatives, especially for scenarios with very few labelled samples.


Ensemble Fine tuned mBERT for Translation Quality Estimation. Quality Estimation (QE) is an important component of the machine translation workflow as it assesses the quality of the translated output without consulting reference translations. In this paper, we discuss our submission to the WMT 2021 QE Shared Task. We participate in Task 2 sentence level sub task that challenge participants to predict the HTER score for sentence level post editing effort. Our proposed system is an ensemble of multilingual BERT (mBERT) based regression models, which are generated by fine tuning on different input settings. It demonstrates comparable performance with respect to the Pearson's correlation and beats the baseline system in MAE RMSE for several language pairs. In addition, we adapt our system for the zero shot setting by exploiting target language relevant language pairs and pseudo reference translations.


A stochastically perturbed fluid structure interaction problem modeled by a stochastic viscous wave equation. We study well posedness for fluid structure interaction driven by stochastic forcing. This is of particular interest in real life applications where forcing and or data have a strong stochastic component. The prototype model studied here is a stochastic viscous wave equation, which arises in modeling the interaction between Stokes flow and an elastic membrane. To account for stochastic perturbations, the viscous wave equation is perturbed by spacetime white noise scaled by a nonlinear Lipschitz function, which depends on the solution. We prove the existence of a unique function valued stochastic mild solution to the corresponding Cauchy problem in spatial dimensions one and two. Additionally, we show that up to a modification, the stochastic mild solution is alpha H " o lder continuous for almost every realization of the solution's sample path, where alpha in 0,1) for spatial dimension n 1 , and alpha in 0,1 2) for spatial dimension n 2 . This result contrasts the known results for the heat and wave equations perturbed by spacetime white noise, including the damped wave equation perturbed by spacetime white noise, for which a function valued mild solution exists only in spatial dimension one and not higher. Our results show that dissipation due to fluid viscosity, which is in the form of the Dirichlet to Neumann operator applied to the time derivative of the membrane displacement, sufficiently regularizes the roughness of white noise in the stochastic viscous wave equation to allow the stochastic mild solution to exist even in dimension two, which is the physical dimension of the problem. To the best of our knowledge, this is the first result on well posedness for a stochastically perturbed fluid structure interaction problem.


The Heisenberg category of a category. Starting with a k linear or DG category admitting a (homotopy) Serre functor, we construct a k linear or DG 2 category categorifying the Heisenberg algebra of the numerical K group of the original category. We also define a 2 categorical analogue of the Fock space representation of the Heisenberg algebra. Our construction generalises and unifies various categorical Heisenberg algebra actions appearing in the literature. In particular, we give a full categorical enhancement of the action on derived categories of symmetric quotient stacks introduced by Krug, which itself categorifies a Heisenberg algebra action proposed by Grojnowski.


Time Evolving to Space Evolving Thermal Instability of a Porous Medium Flow. The Rayleigh B 'enard instability of the stationary throughflow in a horizontal porous layer, also known as Prats' problem, is here analysed in a fresh new perspective. In fact, the classical analysis of the linear instability, carried out by employing space periodic Fourier modes, is here reconsidered by focussing on the effects of time periodic and space evolving modes. The basic stationary flow is assumed to be perturbed by a localised source of perturbation which is steady periodic in time. Then, the streamwise development of such perturbations is monitored in order to detect their possible amplification or decay in the direction of propagation. Accordingly, the spatial stability instability threshold is determined. The study is carried out by employing a Fourier transform formalism, where the transformed variable is time.


Ionization induced plasma grating and its applications in strong field ionization measurements. An ionization induced plasma grating can be formed by spatially selective ionization of gases by the interference of two intersecting ultra short laser pulses. The density modulation of a plasma grating can approach unity since the plasma is produced only where the two pulses constructively interfere and ionization does not occur in destructive interference regions. Such a large density modulation leads to efficient Thomson scattering of a second ultra short probe pulse once the Bragg condition is satisfied. By measuring the scattering efficiency, it is possible to determine the absolute electron density in the plasma grating and thereby deduce the ionization degree for a given neutral gas density. In this paper, we demonstrate the usefulness of this concept by showing two applications ionization degree measurement of strong field ionization of atoms and molecules and characterization of extremely low density gas jets. The former application is of particular interest for ionization physics studies in dense gases where the collision of the ionized electron with neighboring neutrals may become important sometimes referred to as many body ionization, and the latter is useful for plasma based acceleration that requires extremely low density plasmas.


Laser induced thermal source for cold atoms. We demonstrate a simple and compact approach to laser cool and trap atoms based on laser induced thermal ablation (LITA) of a pure solid granule. A rapid thermalisation of the granule leads to a fast recovery of the ultra high vacuum condition required for a long trapping lifetime of the cold gas. We give a proof of concept of the technique, performing a magneto optical trap on the 461 nm 1S 0 rightarrow 1P 1 transition of strontium. We get up to 3.5 million of cold strontium 88 atoms with a trapping lifetime of more than 4 s. The lifetime is limited by the pressure of the strontium free residual background vapour. We also implement an original configuration of permanent magnets to create the quadruple magnetic field of the magneto optical trap. The LITA technique can be generalized to other atomic elements such as transition metals and lanthanide atoms, and shows a strong potential for applications in quantum technologies ranging from quantum computing to precision measurements such as outdoor inertial sensing.


Sentinel node approach to monitoring online COVID 19 misinformation. Understanding how different online communities engage with COVID 19 misinformation is critical for public health response, as misinformation confined to a small, isolated community of users poses a different public health risk than misinformation being consumed by a large population spanning many diverse communities. Here we take a longitudinal approach that leverages tools from network science to study COVID 19 misinformation on Twitter. Our approach provides a means to examine the breadth of misinformation engagement using modest data needs and computational resources. We identify influential accounts from different Twitter communities discussing COVID 19, and follow these sentinel nodes' longitudinally from July 2020 to January 2021. We characterize sentinel nodes in terms of a linked media preference score, and use a standardized similarity score to examine alignment of tweets within and between communities. We find that media preference is strongly correlated with the amount of misinformation propagated by sentinel nodes. Engagement with sensationalist misinformation topics is largely confined to a cluster of sentinel nodes that includes influential conspiracy theorist accounts, while misinformation relating to COVID 19 severity generated widespread engagement across multiple communities. Our findings indicate that misinformation downplaying COVID 19 severity is of particular concern for public health response.


Activity stabilization in a population model of working memory by sinusoidal and noisy inputs. According to mechanistic theories of working memory (WM), information is retained as persistent spiking activity of cortical neural networks. Yet, how this activity is related to changes in the oscillatory profile observed during WM tasks remains an open issue. We explore joint effects of input gamma band oscillations and noise on the dynamics of several firing rate models of WM. The considered models have a metastable active regime, i.e. they demonstrate long lasting transient post stimulus firing rate elevation. We start from a single excitatory inhibitory circuit and demonstrate that either gamma band or noise input could stabilize the active regime, thus supporting WM retention. We then consider a system of two circuits with excitatory intercoupling. We find that fast coupling allows for better stabilization by common noise compared to independent noise and stronger amplification of this effect by in phase gamma inputs compared to anti phase inputs. Finally, we consider a multi circuit system comprised of two clusters, each containing a group of circuits receiving a common noise input and a group of circuits receiving independent noise. Each cluster is associated with its own local gamma generator, so all its circuits receive gamma band input in the same phase. We find that gamma band input differentially stabilizes the activity of the "common noise" groups compared to the "independent noise" groups. If the inter cluster connections are fast, this effect is more pronounced when the gamma band input is delivered to the clusters in the same phase rather than in the anti phase. Assuming that the common noise comes from a large scale distributed WM representation, our results demonstrate that local gamma oscillations can stabilize the activity of the corresponding parts of this representation, with stronger effect for fast long range connections and synchronized gamma oscillations.


Efficient Four Component Dirac Coulomb Gaunt Hartree Fock in Pauli Spinor Representation. Four component Dirac Hartree Fock is an accurate mean field method for treating molecular systems where relativistic effects are important. However, the computational cost and complexity of the two electron interaction makes this method less common, even though we can consider the Dirac Hartree Fock Hamiltonian the "ground truth" of electronic structure, barring explicit quantum electrodynamical effects. Being able to calculate these effects is then vital to the design of lower scaling methods for accurate predictions in computational spectroscopy and properties of heavy element complexes that must include relativistic effects for even qualitative accuracy. In this work, we present a Pauli quaternion formalism of maximal component and spin separation for computing the Dirac Coulomb Gaunt Hartree Fock ground state, with a minimal floating point operation count algorithm. This approach also allows one to explicitly separate different spin physics from the two body interactions, such as spin free, spin orbit, and the spin spin contributions. Additionally, we use this formalism to examine relativistic trends in the periodic table, and analyze the basis set dependence of atomic gold and gold dimer systems.


Rethinking Network Pruning under the Pre train and Fine tune Paradigm. Transformer based pre trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective and prevalent, these models are usually prohibitively large for resource limited deployment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain then finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as remarkable as the pruning results in the literature of convolutional neural networks (CNNs). In particular, common wisdom in pruning CNN states that sparse pruning technique compresses a model more than that obtained by reducing number of channels and layers (Elsen et al., 2020 Zhu and Gupta, 2017), while existing works on sparse pruning of BERT yields inferior results than its small dense counterparts such as TinyBERT (Jiao et al., 2020). In this work, we aim to fill this gap by studying how knowledge are transferred and lost during the pre train, fine tune, and pruning process, and proposing a knowledge aware sparse pruning process that achieves significantly superior results than existing literature. We show for the first time that sparse pruning compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20 times weight FLOPs compression and neglectable loss in prediction accuracy.