Syntax is a powerful abstraction for language understanding. Many downstream tasks require segmenting input text into meaningful constituent chunks (e.g., noun phrases or entities); more generally, models for learning semantic representations of text benefit from integrating syntax in the form of parse trees (e.g., tree-LSTMs). Supervised parsers have traditionally been used to obtain these trees, but lately interest has increased in unsupervised methods that induce syntactic representations directly from unlabeled text. To this end, we propose the deep inside-outside recursive autoencoder (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Unlike many prior approaches, DIORA does not rely on supervision from auxiliary downstream tasks and is thus not constrained to particular domains. Furthermore, competing approaches do not learn explicit phrase representations along with tree structures, which limits their applicability to phrase-based tasks. Extensive experiments on unsupervised parsing, segmentation, and phrase clustering demonstrate the efficacy of our method. DIORA achieves the state of the art in unsupervised parsing (46.9 F1) on the benchmark WSJ dataset. Syntax in the form of parse trees is an essential component of many natural language processing tasks. Constituent spans taken from a parse tree are useful for tasks such as relation extraction BID0 and semantic role labeling BID1 , while the full parse itself can be used to build higher-quality systems for machine translation BID2 ) and text classification BID3 . Supervised parsers trained on datasets such as the Penn Treebank BID4 are traditionally used to obtain these trees; however, these datasets are generally small and restricted to the newswire domain. For out-of-domain applications, it is generally infeasible to create new treebanks, as syntactic annotation is expensive and time-consuming.Motivated by these limitations, we propose a method that extracts both shallow parses (i.e., noun phrases or entities) and full syntactic trees from any domain or language automatically without any training data. In addition to just producing the parse, we want our model to build representations for internal constituents that obey syntactic and semantic regularities, as we can then easily inject these representations into downstream tasks. Our model extends existing work on latent tree chart parsers BID5 BID6 BID7 BID8 , which build up representations for all internal nodes in the tree (cells in the chart) generated by a soft weighting over all possible sub-trees (Section 2).In previous work, the representation at the root node is used as a sentence encoding and trained to optimize some downstream task, typically natural language inference. Unfortunately , this method requires sentence level annotations to train the model. Worse still, analysis on the trees learned by these models show that they are actually quite poor at capturing syntax that in any way resembles linguistic theory BID9 . To address these issues, we incorporate the inside-outside algorithm BID10 BID11 ) into a latent tree chart parser. The bottom-up inside step is equivalent to the forward-pass of previous latent tree chart parsers BID7 . However, these inside representations are encoded by looking only within the current subtree, completely ignoring outside context. Thus, we perform an additional top-down outside calculation for each node in the tree incorporating external context into sub-tree representations. Finally, we train This news raised hopes for further interest-rate cuts .This news raised hopes for further interest-rate cuts .Figure 1: Example parse trees. Top PRPN-LM prediction, bottom DIORA prediction. DIORA correctly chunks the span 'raised hopes for further interest-rate cuts'. the outside representations of leaves to reconstruct the initial input, which results in a completely unsupervised autoencoder-like objective.Recently, BID12 proposed Parsing-Reading-Predict Networks (PRPN), an RNN based language model with an additional module for inferring syntactic distance. After training, this syntax module can be decomposed to recover a parse BID13 ) via a complex mechanism that involves modeling a distribution over possible syntactic structures with a stick-breaking process. Like DIORA, this model can be trained in a completely unsupervised manner. However, it has no mechanism of explicitly modeling phrases, and span representations can only be generated by post-hoc heuristics. Additionally, finding the most probable tree in DIORA is much simpler than in PRPN, as we can just run the CKY algorithm.To probe different properties of our model, we run experiments on unsupervised parsing, segmentation, and phrase representations. DIORA sets the state-of-the-art for unsupervised parsing on the WSJ dataset, has a greater recall on a more constituent types than PRPN, and demonstrates strong clustering of phrase representations. Latent tree models have been shown to perform particularly poorly on attachments at the beginning and end of the sequence BID9 . To address this, we incorporate a post-processing heuristic (+PP in Table 2 ). We see that PRPN-UP and DIORA benefit much more than PRPN-LM from this heuristic. This is consistent with qualitative analysis showing that DIORA and PRPN-UP incorrectly attach trailing punctuation much more than PRPN-LM. This heuristic simply attaches trailing punctuation to the root of the tree, regardless of its predicted attachment. We find this to be extremely effective, increasing our state-of-the-art WSJ parsing results by by over 3 absolute F1 points.On the MultiNLI dataset, PRPN-LM is the top performing model without using the PP heuristic and DIORA outperforms PRPN-UP. Afterwards, PRPN-UP surpasses DIORA. However, it is worth noting that this is not actually a gold standard evaluation and instead evaluates the ability to replicate the output of a trained parser . Table 2 : Unsupervised Parsing. † indicates trained to optimize NLI task.We use the max unlabeled binary F1 across runs for PRPN-UP 2 , PRPN-LM, and DIORA. F1 was calculated using the parse trees provided by BID13 and all results in the upper portion of the table were copied from BID13 . +PP refers to post-processing heuristic to remove trailing punctuation explained in Section 3.1. In Table 2 we see the breakdown of constituent recall across the 10 most common types. We see that PRPN-UP has the highest recall for the most common type noun-phrase, but drops in every other category. DIORA achieves the highest recall across the most types and is the only model to perform effectively on verb-phrases. However, DIORA performs poorly relative to PRPN at prepositional phrases. Table 3 : Segment recall from WSJ seperated by phrase type. The 10 most frequent phrase types are shown. Highest value in each row is bolded. In this work we presented DIORA, a completely unsupervised method for inducing syntactic trees and segmentations over text. We showed that an auto encoder language modeling objective on top of inside-outside representations of latent tree chart parsers allows us to effectively learn syntactic Third-quarter shipments slipped 7 % from the year-ago period and 17 % from this year 's second quarter Third-quarter shipments slipped 7 % from the year-ago period and 17 % from this year 's second quarter Mr. Hoelzer did n't return phone calls seeking comment on the judge 's decision Mr. Hoelzer did n't return phone calls seeking comment on the judge 's decisionThe earthquake caused many streets to buckle and crack making them impassibleThe earthquake caused many streets to buckle and crack making them impassible Figure 3 : Pairs of example parses for the same sentence from two different models. For each pair, the top is the output of PRPN-LM and bottom was produced by DIORA. Bolden token pairs or spans indicate a parse error by PRPN that was correctly attached by DIORA. Some punctuation was removed for clarity of printed trees. structure of language. In experiments on unsupervised parsing, chunking, and phrase representations we show our model is comparable to or outperforms current baselines, achieving the state-of-the-art performance on unsupervised parsing for the WSJ dataset. .Future work can improve the current method by training larger models over much larger corpora including other domains and languages. While the current model seems to focus primarily on syntax, extra unsupervised objectives or light supervision could be injected into the learning procedure to encourage a more thorough capturing of semantics.
In this work we propose deep inside-outside recursive auto-encoders(DIORA)  a  fully  unsupervised  method  of  discovering  syntax  while  simultaneously learning representations for discovered constituents.  A neural latent tree model trained with an auto-encoding objective that achieves state of the art on unsupervised constituency parsing and captures syntactic structure better than other latent tree models. The paper proposes a model for unsupervised dependency parsing (latent tree induction) that is based on a combination of the inside-outside algorithm with neural modeling (recursive auto-encoders). 