A Survey of Arabic Dialogues Understanding for Spontaneous Dialogues and Instant Message
---
paper_title: Critical Survey of the Freely Available Arabic Corpora

paper_content:

The availability of corpora is a major factor in building natural language processing applications. However, the costs of acquiring corpora can prevent some researchers from going further in their endeavours. The ease of access to freely available corpora is urgent needed in the NLP research community especially for language such as Arabic. Currently, there is not easy was to access to a comprehensive and updated list of freely available Arabic corpora. We present in this paper, the results of a recent survey conducted to identify the list of the freely available Arabic corpora and language resources. Our preliminary results showed an initial list of 66 sources. We presents our findings in the various categories studied and we provided the direct links to get the data when possible.
---
paper_title: Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech

paper_content:

We describe a statistical approach for modeling dialogue acts in conversational speech, i.e., speech-act-like units such as STATEMENT, QUESTION, BACKCHANNEL, AGREEMENT, DISAGREEMENT, and APOLOGY. Our model detects and predicts dialogue acts based on lexical, collocational, and prosodic cues, as well as on the discourse coherence of the dialogue act sequence. The dialogue model is based on treating the discourse structure of a conversation as a hidden Markov model and the individual dialogue acts as observations emanating from the model states. Constraints on the likely sequence of dialogue acts are modeled via a dialogue act n-gram. The statistical dialogue grammar is combined with word n-grams, decision trees, and neural networks modeling the idiosyncratic lexical and prosodic manifestations of each dialogue act. We develop a probabilistic integration of speech recognition with dialogue modeling, to improve both speech recognition and dialogue act classification accuracy. Models are trained and evaluated using a large hand-labeled database of 1,155 conversations from the Switchboard corpus of spontaneous human-to-human telephone speech. We achieved good dialogue act labeling accuracy (65% based on errorful, automatically recognized words and prosody, and 71% based on word transcripts, compared to a chance baseline accuracy of 35% and human accuracy of 84%) and a small reduction in word recognition error.
---
paper_title: Discriminative framework for spoken tunisian dialect understanding

paper_content:

In this paper, we propose to evaluate the performance of a discriminative model to semantically label spoken Tunisian dialect turns which are not segmented into utterances. We evaluate discriminative algorithm based on Conditional Random Fields (CRF). We check the performance of the CRF model to concept labeling on raw data in Tunisian dialect which are not analyzed in advance. We compared its performance with different types of preprocessing data until arriving to well treated data. CRF model showed the ability to ameliorate the accuracy of labeling task for spoken language understanding of not segmented and not treated speech in Tunisian dialect.
---
paper_title: Arabic Dialect Processing Tutorial

paper_content:

Language exists in a natural continuum, both historically and geographically. The term language as opposed to dialect is only an expression of power and dominance of one group/ideology over another. In the Arab world, politics (Arab nationalism) and religion (Islam) are what shape the perception of the distinction between the Arabic language and an Arabic dialect. This power relationship is similar to others that exist between languages and their dialects. However, the high degree of difference between standard Arabic and its dialects and the fact that standard Arabic is not any Arab's native language sets the Arabic linguistic situation apart.
---
paper_title: Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech

paper_content:

We describe a statistical approach for modeling dialogue acts in conversational speech, i.e., speech-act-like units such as STATEMENT, QUESTION, BACKCHANNEL, AGREEMENT, DISAGREEMENT, and APOLOGY. Our model detects and predicts dialogue acts based on lexical, collocational, and prosodic cues, as well as on the discourse coherence of the dialogue act sequence. The dialogue model is based on treating the discourse structure of a conversation as a hidden Markov model and the individual dialogue acts as observations emanating from the model states. Constraints on the likely sequence of dialogue acts are modeled via a dialogue act n-gram. The statistical dialogue grammar is combined with word n-grams, decision trees, and neural networks modeling the idiosyncratic lexical and prosodic manifestations of each dialogue act. We develop a probabilistic integration of speech recognition with dialogue modeling, to improve both speech recognition and dialogue act classification accuracy. Models are trained and evaluated using a large hand-labeled database of 1,155 conversations from the Switchboard corpus of spontaneous human-to-human telephone speech. We achieved good dialogue act labeling accuracy (65% based on errorful, automatically recognized words and prosody, and 71% based on word transcripts, compared to a chance baseline accuracy of 35% and human accuracy of 84%) and a small reduction in word recognition error.
---
paper_title: Discriminative framework for spoken tunisian dialect understanding

paper_content:

In this paper, we propose to evaluate the performance of a discriminative model to semantically label spoken Tunisian dialect turns which are not segmented into utterances. We evaluate discriminative algorithm based on Conditional Random Fields (CRF). We check the performance of the CRF model to concept labeling on raw data in Tunisian dialect which are not analyzed in advance. We compared its performance with different types of preprocessing data until arriving to well treated data. CRF model showed the ability to ameliorate the accuracy of labeling task for spoken language understanding of not segmented and not treated speech in Tunisian dialect.
---
paper_title: The Hcrc Map Task Corpus

paper_content:

This paper describes a corpus of unscripted, task-oriented dialogues which has been designed, digitally recorded, and transcribed to support the study of spontaneous speech on many levels. The corpus uses the Map Task (Brown, Anderson, Yule, and Shillcock, 1983) in which speakers must collaborate verbally to reproduce on one participant's map a route printed on the other's. In all, the corpus includes four conversations from each of 64 young adults and manipulates the following variables: familiarity of speakers, eye contact between speakers, matching between landmarks on the participants' maps, opportunities for contrastive stress, and phonological characteristics of landmark names. The motivations for the design are set out and basic corpus statistics are presented.
---
paper_title: Critical Survey of the Freely Available Arabic Corpora

paper_content:

The availability of corpora is a major factor in building natural language processing applications. However, the costs of acquiring corpora can prevent some researchers from going further in their endeavours. The ease of access to freely available corpora is urgent needed in the NLP research community especially for language such as Arabic. Currently, there is not easy was to access to a comprehensive and updated list of freely available Arabic corpora. We present in this paper, the results of a recent survey conducted to identify the list of the freely available Arabic corpora and language resources. Our preliminary results showed an initial list of 66 sources. We presents our findings in the various categories studied and we provided the direct links to get the data when possible.
---
paper_title: Discriminative framework for spoken tunisian dialect understanding

paper_content:

In this paper, we propose to evaluate the performance of a discriminative model to semantically label spoken Tunisian dialect turns which are not segmented into utterances. We evaluate discriminative algorithm based on Conditional Random Fields (CRF). We check the performance of the CRF model to concept labeling on raw data in Tunisian dialect which are not analyzed in advance. We compared its performance with different types of preprocessing data until arriving to well treated data. CRF model showed the ability to ameliorate the accuracy of labeling task for spoken language understanding of not segmented and not treated speech in Tunisian dialect.
---
paper_title: Automatic linguistic segmentation of conversational speech

paper_content:

As speech recognition moves toward more unconstrained domains such as conversational speech, we encounter a need to be able to segment (or resegment) waveforms and recognizer output into linguistically meaningful units such a sentences. Toward this end, we present a simple automatic segmenter of transcripts based on N-gram language modeling. We also study the relevance of several word-level features for segmentation performance. Using only word-level information, we achieve 85% recall and 70% precision on linguistic boundary detection.
---
paper_title: Data-Driven Strategies For An Automated Dialogue System

paper_content:

We present a prototype natural-language problem-solving application for a financial services call center, developed as part of the Amities multilingual human-computer dialogue project. Our automated dialogue system, based on empirical evidence from real call-center conversations, features a data-driven approach that allows for mixed system/customer initiative and spontaneous conversation. Preliminary evaluation results indicate efficient dialogues and high user satisfaction, with performance comparable to or better than that of current conversational travel information systems.
---
paper_title: Transformation-Based-Error-Driven Learning And Natural Language Processing: A Case Study In Part-Of-Speech Tagging

paper_content:

Recently, there has been a rebirth of empiricism in the field of natural language processing. Manual encoding of linguistic information is being challenged by automated corpus-based learning as a method of providing a natural language processing system with linguistic knowledge. Although corpus-based approaches have been successful in many different areas of natural language processing, it is often the case that these methods capture the linguistic information they are modelling indirectly in large opaque tables of statistics. This can make it difficult to analyze, understand and improve the ability of these approaches to model underlying linguistic behavior. In this paper, we will describe a simple rule-based approach to automated learning of linguistic knowledge. This approach has been shown for a number of tasks to capture information in a clearer and more direct fashion without a compromise in performance. We present a detailed case study of this learning method applied to part-of-speech tagging.
---
paper_title: Interactive Problem Solving And Dialogue In The ATIS Domain

paper_content:

This paper describes the present status of the discourse and dialogue models within the MIT ATIS system, extended to support the notion of booking a flight. The discourse model includes not only the resolution of explicit anaphoric references, but also indirect and direct references to information mentioned earlier in the conversation, such as a direct reference to an entry in a previously displayed table or an indirect reference to a date, as in "the following Thursday." The system keeps a history table containing objects such as flights and dates, represented as semantic frames, as well as the active ticket, previously booked tickets, and previously displayed tables. During flight reservations scenarios, the system monitors the state of the ticket (which is displayed to the user), making sure that all information is complete (by querying the user) before allowing a booking. It may even initiate calls to the database to provide additional unsolicited information as appropriate. We have collected several dialogues of subjects using the system to make reservations, and from these, we are learning how to design better dialogue models.
---
paper_title: Discriminative framework for spoken tunisian dialect understanding

paper_content:

In this paper, we propose to evaluate the performance of a discriminative model to semantically label spoken Tunisian dialect turns which are not segmented into utterances. We evaluate discriminative algorithm based on Conditional Random Fields (CRF). We check the performance of the CRF model to concept labeling on raw data in Tunisian dialect which are not analyzed in advance. We compared its performance with different types of preprocessing data until arriving to well treated data. CRF model showed the ability to ameliorate the accuracy of labeling task for spoken language understanding of not segmented and not treated speech in Tunisian dialect.
---
paper_title: User's utterance classification using machine learning for Arabic Conversational Agents

paper_content:

This paper presents a novel technique for the classification of Arabic sentences as Dialogue Acts, based on structural information contained in Arabic function words. It focuses on classifying questions and non-questions utterances as they are used in Conversational Agents. The proposed technique extracts function words features by replacing them with numeric tokens and replacing each content word with a standard numeric token. The Decision Tree has been chosen for this work to extract the classification rules. Experiments provide evidence for highly effective classification. The extracted classification rules will be embedded into a Conversational Agent called ArabChat in order to classify Arabic utterances before further processing on these utterances. This paper presents a complement work for the ArabChat to improve its performance by differentiating among question-based and non question-based utterances.
---

Title: A Survey of Arabic Dialogues Understanding for Spontaneous Dialogues and Instant Message

Section 1: Concepts and Terminologies
Description 1: This section presents the concepts related to language understanding and used in this paper.

Section 2: Dialogue Act
Description 2: This section discusses the terminology of dialogue acts based on speech act theory and its applications in dialogue systems.

Section 3: Turn vs Utterance
Description 3: This section explains the difference between turns and utterances in natural conversation and in spoken dialogue systems.

Section 4: Dialectal Arabic
Description 4: This section describes the different categories of Arabic language, specifically Classic Arabic, Modern Standard Arabic, and dialectal Arabic, and their characteristics.

Section 5: Language Understanding Component
Description 5: This section presents recent research for the four parts of building a language understanding component for Arabic dialogue systems.

Section 6: Dialogue Acts Annotation Schema
Description 6: This section elaborates on the importance and application of dialogue acts annotation schema in building annotated dialogue corpora and dialogue management systems.

Section 7: Arabic Dialogue Acts Corpora
Description 7: This section discusses the use of corpora in NLP research, specifically focusing on Arabic dialogue acts corpora and their properties.

Section 8: Arabic Dialogue Segmentation
Description 8: This section explains the segmentation process for dividing dialogues into meaningful units and various approaches used for segmentation in Arabic dialogues.

Section 9: Arabic Dialogue Acts Classification
Description 9: This section explores different methods and approaches for classifying dialogue acts in Arabic, including both shallow and deeper linguistic analysis methods.

Section 10: Conclusion and Future Work
Description 10: This section provides the conclusions of the survey and suggests directions for future research in the field.