---
abstract: 'Talaia is a platform for monitoring social media and digital press. A configurable crawler gathers content with respect to user defined domains or topics. Crawled data is processed by means of the EliXa Sentiment Analysis system. A Django powered interface provides data visualization for a user-based analysis of the data. This paper presents the architecture of the system and describes in detail its different components. To prove the validity of the approach, two real use cases are accounted for: one in the cultural domain and one in the political domain. Evaluation for the sentiment analysis task in both scenarios is also provided, showing the capacity for domain adaptation.'
address:
- Elhuyar Foundation
- 'IXA NLP Group, University of the Basque Country UPV/EHU'
author:
- 'Iñaki San Vicente\*, Xabier Saralegi'
- Rodrigo Agerri
bibliography:
- 'talaia\_eaai.bib'
title: Real Time Monitoring of Social Media and Digital Press
---

Sentiment Analysis ,Social Media Analysis ,Crawling ,Natural Language Processing ,Digital Media Monitoring

Introduction {#isv2018:intro}
============

The Internet is a very rich source of user-generated information. As knowledge management technologies have evolved, many organizations have turned their eyes to such information, as a way of obtaining global feedback on their activities [@chen2012business]. Some studies [@oconnor_tweets_2010; @ceron2015using] have pointed out that such systems could perform as well as traditional polling systems, but at a much lower cost.

Talaia is a platform for monitoring the impact of topics specified by the user in social media and digital press. The process starts when the user configures the system to find information related to a domain or topic. Talaia provides real time information on the topic and graphic visualizations to help users interpreting the data. Such technology has various applications areas, such as:

-   Monitoring events: Following public events in real time harvesting people’s opinions and media news [@sutton2009social; @yu2015world].

-   Analyze citizen or electors voice: Tracking the opinions citizens convey with respect to public services or trends during electoral campaigns [@ceron2015using].

-   Marketing and brand management: Measuring the impact of marketing campaigns in a digital environment [@ahmed2018measuring].

-   Business Intelligence: Fast and efficient visualization of the information extracted from social media offers companies the possibility to analyse opinions about their products or services [@he2013social; @mostafa2013more].

-   Security: Detection of social conflicts, crimes, and cyberbullying [@Xuetal2012sabullying; @dadvar2013improving].

Talaia consists of three main modules: (i) a crawler collecting the data; (ii) a data analysis module; and (iii) a Graphical User Interface (GUI) providing interpretation of the data analyzed. Figure \[isv2018:fig:arch\] describes the architecture. Its main features are the following:

-   Monitoring and automatic analysis: Definition of the domain/topic by means of term taxonomies. Continuous monitoring of various mention sources including social media and digital press.

-   Multilingual extraction of mentions and opinions relevant to the topics monitored, by means of Natural Language Processing (NLP) techniques.

-   Result exploration: Intuitive GUI to visualize and analyse the results. Advanced statistics and filters, such as per language results, impact of the topics or author statistics.

-   Control of the monitoring process through the user interface: update search terms or review and correct gathered mentions.

![image](architecture_talaia.eps){width="60.00000%"}

This paper focuses on those processes monitoring user satisfaction with respect to a topic, and that is why we pay special attention to the Sentiment Analysis (SA) module. Nevertheless, Talaia is capable of performing further data analysis tasks involving user profiling, in order to get the most out of the data. Specifically, geolocalization, user community identification and gender detection have been implemented. Section \[isv2018:da:profiling\] provides more details.

The rest of the paper is organized as follows. Section \[isv2018:soa\] discusses previous work, focusing on social media and SA. Both the academic and industrial points of view are taken into account. The third section describes in detail the modules composing Talaia. Section \[isv2018:success\] presents two success cases where the platform has been used for monitoring different events. Section \[isv2018:eval\] provides evaluation and results on the SA task for both scenarios. The last section draws some conclusions and future directions.

Background {#isv2018:soa}
==========

Social Media Analysis {#isv2018:soa:SocialMedia}
---------------------

Social media are becoming the primary environment for producing, spreading and consuming information. Enormous quantities of user generated content are produced constantly. Even traditional media spread their news and get a large amount of traffic trough social media. Monitoring events or topics in such an environment is however a challenging task. That is where data mining and Natural Language Processing (NLP) become essential. We have to be able to collect large scale data, but also to extract the relevant information. Tracking a topic over an extended time period means that the information flow grows and fades over time. Also a topic may evolve in terms of the vocabulary used, and thus “topic detection and tracking” (TDT) [@allan1998topic] techniques become relevant to maintain a successful monitoring.

Several systems have been proposed in the literature to explore events. Trend Miner [@preoctiuc2013temporal] extracts multilingual terms from social media, groups and visualizes them in temporal series. Social Sensor [@aiello2013sensing] and Twitcident [@abel2012twitcident] may be the most similar systems to ours. The first one focuses on tracking topic or events predefined by the user. The second makes user defined searches related to crisis management. LRA[^1] aims to discovering and tracking crisis situations based on crowdsourced information. ReDites [@osborne2014real] detects an tracks topics in a fully automated way.

Detecting terms that represent a domain or topic semantically has been traditionally addressed by statistical models such as Latent Dirichlet Association (LDA) [@blei2003latent]. Classical LDA models are applied over static document collections. In order to extract terms from dynamic collections, the most common approach is to follow a two step strategy [@shamma2011peaks] consisting of detecting emerging terms and grouping them in clusters thereby defining a domain.

@nguyen2016hot predict emerging terms by means of word co-occurrence distributional models, comparing the terms in an specific time window against the whole collection. @abilhoa2014keyword use a graph-based representation of the document collection. @aiello2013sensing propose $df$-$idf_t$ (Document Frequency - Inverse Document Frequency), a variation of $tf$-$idf$ that includes the temporal factor. @kim2016exploring combine neural networks and sequence labelling in order to extract relevant terms from conversations. @miao2017cost propose to reduce the cost of predicting emerging topics by finding a small group of representative users and predict the emerging topics from their social media activity.

There is also the problem of the scope of the event or topic to be tracked. An event might be tracked at global level (e.g. Football World Cup), but most events are local or regional at most. Two issues arise at this point. Firstly, how to restrict the data gathered to a specific region, and, secondly, how to cope with multilingual data. Some authors tackle the problem by automatically geolocating tweets while others try focus on user locations. See @jurgens2015geolocation [@zubiaga2017towards] for reviews of previous approaches. Our approach is to geolocate users rather than tweets, in order to construct a census of Twitter users in a region. We developed a SVM classifier similar to @zubiaga2017towards using follower and friend information as features.

Sentiment Analysis {#isv2018:soa:SA}
------------------

In the last years microblogging sites such as Twitter have attracted the attention of many researchers with diverse objectives such as stock market prediction [@bollen_twitter_2010; @oliveira2017impact], polling estimation [@oconnor_tweets_2010; @ceron2015using] or analysis of crisis situations [@griffith2016analysis; @shaikh2017tweet; @ozturk2018sentiment]. The growing number of SA related shared tasks (e.g., SemEval Aspect based SA and Twitter SA shared tasks) or the commercial platforms for reputation management (see section \[isv2018:soa:indust\]) are proof of the interest from both academic and market worlds.

The particularities of its language make it hard to analyze tweets. User mentions, hashtags, the growing presence of emojis, ungrammatical sentences, vocabulary variations and other phenomena pose a great challenge for traditional NLP tools [@foster_hardtoparse2011; @liu_recognizing_2011]. @brody_cool2011 deal with the word lengthening phenomenon, which is especially important for sentiment analysis because it usually expresses emphasis of the message. Hashtag decomposition (e.g.,*\#GameOfThrones = ‘Game Of Thrones’*) [@BrunRoux_F14-2015; @Belainine_W16-3915] or matching Out Of Vocabulary (OOV) forms and acronyms to their standard vocabulary forms (e.g., [*‘imo = in my opinion’*]{}) [@han_lexical_2011; @liu_broad-coverage_2012; @alegria2014tweetnorm] are other addressed issues. International benchmarking initiatives such as the TweetNorm shared task[@tweetnorm_jlre_2015] or the WNUT[^2] workshop series are proof of the interest to solve this task.

Once texts are normalized, sentiment analysis can be performed. Several ruled-based systems to polarity classification have been proposed [@hu_mining_2004; @thelwall2017heart; @taboada2011lexicon]. Nevertheless, we will focus on Machine Learning (ML) based approaches which are the most widespread. Support Vector Machines (SVM) and Logistic Regression algorithms have been the very popular for polarity classification as various international shared tasks [@roman_tass2014; @semeval-2014_4; @semeval-2014_9] show. Typical features of those systems include sentiment word/lemma ngram features, POS tags [@barbosa_robust_2010], Sentiment Lexicons [@kouloumpis_twitter_2011], emoticons [@oconnor_tweets_2010], discourse information [@somasundaran_supervised_2009] or, more recently, word embeddings [@mikolov2013distributed] and clusters [@elixa-semeval2015].

From 2015 onwards, the academic world has shifted to Deep Learning (DL) approaches, as @Semeval2016-Twitter-task4 confirm. Long-Short Term Memory (LSTM) Recurrent Neural Networks (RNN)[@DaiLe2015NIPS; @johnson2016supervised] and Convolutional Neural Networks (CNN) are the preferred choices. @severyn2015 use a single layer CNN, first to construct word embeddings and then to train the classifier. @Deriu2017 propose a two phase training method: first they train a neural network with large amounts of weakly supervised data collected from Twitter. The network is initialized with word embeddings learned by means of word2vec [@mikolov2013distributed] from very large corpora collected from twitter. Second, the weights learned in the first step are transferred to a second neural network trained over the actual annotated data, to learn the final classifier. A two convolutional layer CNN is used for both training phases. A very similar approach is followed by @Cliche2017 which achieves top results in SemEval [@Semeval2017-twitter-task4]. @ulmfit2018 follow a similar three step approach with a more complex network topology obtaining state of the art results for various tasks, including sentiment analysis.

A common problem of supervised approaches, specially of DL, is the need of large amounts of labeled data for training. The common practice in the literature is to gather weakly supervised datasets following the emoticon heuristic[^3] [@go_twitter_2009]. This is feasible for major languages, but it is a very difficult (if possible at all) and time costly task for non major languages such as Basque.

Our system is closest to @barbosa_robust_2010 and @kouloumpis_twitter_2011 because it combines polarity lexicons with machine learning for labelling sentiment of tweets. This strategy has proven to be a successful approach in previous shared tasks [@saralegi_tass_2012; @mohammad_semeval_2013].

Industrial Solutions {#isv2018:soa:indust}
--------------------

We can find various commercial solutions in the market. We are particularly interested in systems that provide an integral solution of the monitoring process, leaving out tools that only approach specific phases of the surveillance process, or solutions that offer bare NLP processing chains which require further development to achieve a working social media monitor. Table \[isv2018:soa:indust:comptable\] in Annex offers a detailed comparative of the tools analyzed. We focus our analysis on the sources where information is gathered on, their tracking capabilities, the processing of multilingual information, and the data visualization.

**Iconoce[^4]** is a system oriented to reputation management, offering various features such as measuring impact of campaigns, or reputation monitoring. Although it also can monitor social media (Twitter and Facebook) its strength lies on the analysis of digital press. Multilingual information can be gathered but no linguistic processing is performed (lemmatization or crosslingual searches). It has 3 separated search engines for authors, mentions and comments. A customizable dashboard offers various visualizations and data aggregations (e.g., salient term and topics, influencer, sentiment or trends). Periodical reports and alerts in the face of tendency changes are provided. As a distinctive feature, it offers a personalized press archive based on the customer configuration.

In a similar way, **INNGUMA[^5]** is a tool providing business intelligence services. They put their main effort in the crawling step. Rather than offering to the user results over analysed data, the tool is designed for a group of customers to analyze the data collaboratively. Customers are provided with a search engine (more or less powerful depending on the pricing plan), and interface where they can store and share their findings.

**Lexalitycs[^6]** and **Meaning Cloud[^7]** are text analytics enterprises. Their strength is the data analysis part rather than the monitoring of many sources. Both systems are built upon robust NLP chains. Document classification, entity extraction and aspect based sentiment analysis are performed. Sentiment Analysis is approached by means of rule-based systems based on lexicons and deep linguistic analysis, offering the possibility of custom domain adaptations. Both Lexalitics and Meaning Cloud lack a result visualization interface, limiting their outputs to Excel plugins, leaving the full analysis of the data into the user’s hands.

**Websays[^8]** monitors a wide range of sources including news, Blogs/RSS, Forums, Facebook, Twitter, LinkedIn, Instagram, Foursquare, Pinterest, Youtube, Vimeo, Reviews (Tripadvisor, Booking,...). The user is able to configure the crawling using keywords. Negative words are also allowed in order to effectively restrict the search to the desired domain. The system is able to process data in several languages, but they report to be most effective with European languages (Spanish, English, French, Italian, and Catalan). SA is performed by combining ML algorithms and human validation, so the statistical models may learn from corrected data. The user may navigate through results using a dashboard that offers multiple filtering options. Graphs, salient terms, trending topics, influencers, sentiment and trends are provided, as well as periodical alerts and reports. The interface offers the possibility to manually edit and correct the results.

Following the same concept of Websays, **Keyhole[^9]** is a monitoring and analytics tool that provides trends, insights, and analysis (including sentiment) of hashtags, keywords, or accounts on Twitter and Instagram. It reports supporting data processing in a number of languages, but no details are given on the technology. User can also track web mentions, but two separate monitoring processes must be setup.

**Lynguo[^10]** is also in the same group of Websays and Keyhole. It claims to provide support in 24 languages, although it reports full processing chain for Spanish and English[^11]. NLP is done by means of “a range of linguistic tools to cover and combine in real time the different lexical, morphological and semantic processing layers, with machine learning and deep learning models, and software architectures”[^12]. SA includes lexicons, customizable by the user. Monitoring is configured specifying keywords and users, allowing for negative ones as well. Lynguo is also able to geolocate comments.

**Ubermetrics[^13]** is one of the few platforms that monitors multimedia sources including Youtube and Vimeo, but also TV and Radio sources. According to their reports, it processes data in 40 languages. Its visualization dashboard offers customizable graphs based on multiple search criteria. Ubermetrics main objective is analyzing virality (impact) of the mentions and author profiling.

**Snaptrends[^14]** monitors social media (Twitter, Facebook, Instagram, and Pinterest). Multilingual data is handled by means of MT (80 languages to English). It uses a proprietary NLP chain for processing English data, including sentiment analysis and relevant term extraction. The main feature for filtering large volumes of information is a geolocation-based search engine, combined with keyword based searches and other filters such as data sources. With respect to visualization, it has various data aggregations, such as influencer rankings or sentiment evolution across time by geographical area. Snaptrend makes an special effort in visualizing specific data, generating mention mosaics and timelines in real time.

Talaia shares features with many of the aforementioned commercial solutions, yet it also possess its own characteristics. With a more robust text analysis than Iconoce and INNGUMA, and a more advanced interpretation of the data than MeaningCloud and Lexalitics, Talaia is closer to tools such as Websays and Lynguo. Having keywords organized in a taxonomy allows us to provide deeper data analysis and aggregations. Moreover, Talaia is built using open source software with a strong academical background and tested against well known benchmarks. Talaia’s performance is thus, verifiable.

Data Collection {#isv2018:dc}
===============

The first step of a monitoring system such as Talaia is the collection of information. The Multi Source Monitor (MSM) system[^15] is currently able to monitor Twitter, syndication feeds and also multimedia sources such as television or radio programs. Support for other social media such as Youtube, Facebook, etc. is under development.

MSM is a keyword based crawler, which works on a set of keywords defined by the user. Rather than a list of unconnected terms, Talaia is designed to work over a hierarchy, which allows a better organization of the data for the analysis step. In this way, keywords are defined as belonging to a specific category in the taxonomy. One handicap of crawling using a keyword-based strategy is that it is often difficult to define unambiguous terms that do not capture noisy messages. In order to minimize this situation, MSM implements a number of features:

1.  [**Regular expressions**]{} are used to define keywords. This allows to differentiate between common words and proper names, or full words and affixes (e.g., *podemos* ‘we can’ vs. *Podemos* political party). These phenomena are specially frequent in social media, where language rules are often ignored.

2.  [**Language specific keywords**]{}. A word that is a very good keyword in a language can be a source of noise in another, e.g. *mendia*, ‘mountain’ in Spanish, is unambiguously referring to ‘Idoia Mendia’, a Basque politician, in our context, while in Basque it is clearly ambiguous.

3.  [**Anchor terms**]{} usually define the general topic (e.g. election campaign) to monitor. If the user specifies that a keyword requires an anchor, then in order to accept a message containing that keyword the message must also contain at least one anchor term. Anchor terms may be keywords or not.

4.  [**Long paragraphs are split**]{} before looking for keywords in the case of messages coming from news sites. First, it looks if any keyword appears in a candidate article. If so, it looks for keywords sentence by sentence, and those sentences are considered as the message unit.

Language Identification (LID) {#isv2018:dc:crawler:lid .unnumbered}
-----------------------------

LID is indispensable in order to apply the corresponding NLP analysis. LID is integrated into the crawling process as part of the MSM system. There are two main reasons for that. First, it allows us implement the aforementioned “language specific keyword” feature. Second, having the language identified in the first place gives us flexibility for applying the subsequent NLP tools. At the moment language identification is implemented using the library Optimaize[^16], combined with source specific optimizations (social media vs. feeds).

Data Analysis {#isv2018:da}
=============

The data analysis is mainly performed by EliXa [@elixa-semeval2015] which integrates the following processes, each of them further detailed in the next sections.

EliXa[^17] is a supervised Sentiment Analysis system. It was developed as a modular platform which allows to easily conduct experiments by replacing the modules or adding new features. It was first tested in the ABSA 2015 shared task at SemEval workshop [@semeval-2015-absa]. EliXa currently offers resources and models for 4 languages: Basque, Spanish, English and French. Its implementation is easily adaptable to other languages, requiring a polarity lexicon and/or a training dataset for each new language.

Normalization {#isv2018:da:pre}
-------------

To address the particular characteristics of tweets, EliXa integrates a microtext normalization module which is applied to social media messages, based on @saralegi_tweetnorm_2013. The normalizer is based on heuristic rules, such as standardizing URLs, normalizing character repetitions or dividing long words (e.g. \#AVeryLongDay $\rightarrow$ ‘a very long day’). Also Out Of Vocabulary (OOV) term normalization is addressed by means of language specific frequency lists based on Twitter corpora.

Furthermore, EliXa’s normalization component also includes various specific functionalities related to SA:

-   Emoticons are normalized into a 7 sentiment scale: *smiley, crying, shock, mute, angry, kiss, sadness*.

-   Expressions that are meaningful for detecting SA such as interjections and onomatopoeia are marked.

Those normalized terms must be included in the polarity lexicons in order to have a greater impact in the sentiment analysis classification. Table \[isv2018:da:pre:rsrc-table\] presents the resources provided for normalization according to their use. Word form dictionaries are composed of word forms extracted from corpora. When applying microtext normalization, candidates are compared to forms in the dict in order to discard noisy candidates. For example, if we were to normalize “happppy”, we would know the that the correct normalization is “happy” by looking at theses dictionaries.

OOV dictionaries are composed of “OOV - standard form” pairs. These resources are valuable to normalize slang and commonly used abbreviations. In order to produce such dictionaries word form frequency lists were generated from Twitter corpora, and after pruning standard dictionary forms, the most frequent $n$ forms were manually reviewed and manually translated[^18]. When available, dictionaries were completed using precompiled lists existing in the Web.

Emoticon lexicon is a dictionary of regular expression matching a number of emoticons to their corresponding sentiment in the aforementioned scale.

Lastly, stopword lemma lists are used to discard most frequent lemmas when extracting n-gram features from texts. We adapted this lists to SA requirements by removing some lemmas, because of their relevance to polarity classification (e.g., no, good, ...).

  ------------------------ ----------------------------------------------------- --------- --------- -------- ---------
  [**Resource**]{}         [**Use**]{}                                                                        
                                                                                    eu        es        en       fr
  Word form dictionaries   text normalization (e.g. 4ever$\rightarrow$forever)    122,085   556,501   67,811   453,037
  OOV dictionaries         text normalization (e.g. 4ever$\rightarrow$forever)      63       7,823     223       279
  Emoticon lexicon         Polarity tagging                                                                   
  Stopword lemma lists     Polarity tagging feature extraction                      56        46        75       100
  ------------------------ ----------------------------------------------------- --------- --------- -------- ---------

NLP pre-processing {#isv2018:da:ixa-pipes}
------------------

EliXa currently performs tokenization, lemmatization and POS tagging prior to sentiment analysis classification. No entity recognition is applied; entities are matched only if they are defined as keywords. Although EliXa is able to work with corpora preprocessed with other taggers, its default NLP processing is made by means of IXA pipes [@agerri2014ixapipes] which is integrated as a library.

Sentiment Analysis {#isv2018:da:elixa}
------------------

EliXa’s core feature is its polarity classifier, which implements a multiclass Support Vector Machine (SVM) algorithm [@hall_weka_2009] combining the information extracted from polarity lexicons with linguistic features obtained from the NLP pre-processing step. Main features include polarity values from general and domain specific polarity lexicons, lemma and POS tag ngrams and positivity and negativity counts based on polarity lexicons. Features representing other linguistic phenomena such as treatment of negation, locutions or punctuation marks are also included. Finally, there are some social media specific features, such as the proportion of capitalized symbols (which often is used to increase the intensity of the message) or emoticon information.

EliXa currently provides ready to use polarity classification models, although one of its strengths is that new models can easily be trained if training data is available for a new domain.

User profiling {#isv2018:da:profiling}
--------------

Talaia is also capable of providing deeper analysis of the data, by means of user profiling. Specifically, geolocation, gender detection and user community identification are implemented.

Opinions gathered are geolocated. Geolocation may be done using two different approaches: (i) building a census of twitter users on a region or (ii) trying to geolocate the origin of the users detected. Approach (i) is most suitable when monitoring is done for a specific region, and high precision is required from geolocation. Details on this approach are given in section \[isv2018:success:pol\].

Approach (ii) allows Talaia to analyze the differences in opinions with respect to a topic that may arise between regions or countries. Geolocation is done by exploiting social media information from both messages and authors. If a message is geolocated, its information is used straightforwardly. Otherwise, user profile information is analyzed. The task is challenging, because users do not provide such information always (40%-45% of the users in our datasets have location information), or they define fictitious locations (e.g., ‘Middle earth’, ‘In a galaxy far, far away...’; 14%). Several geocoding APIs[^19] are queried, and results are then weighted, because APIs show divergent results when feeding fictitious locations. The weighted system obtains 82% accuracy for those users containing location information in their profile. Roughly, we are able to geolocate correctly around 32% of the users that appear in a monitoring process.

Gender detection is another important factor in many social science studies. A supervised gender classifier is implemented to infer user gender, based on features extracted from academic papers [@kokkos2014robust; @rangel2017overview]. User gender detection is based on classifying messages, no user profile information is used.

Data Visualization {#isv2018:da:vis}
==================

The GUI has been developed using the Django Web Application framework[^20]. This interface provides data analysis visualizations and manages the communication with both the crawler and EliXa.

Talaia implements a number of visualizations which may be customized depending on the needs of the specific use case at hand[^21]. The main visualizations include popularity, sympathy and antipathy comparison, evolution of mentions across time, most recent mentions, most widespread mentions, most active users in social media and news sources, and most frequent topics. All those visualizations include interactions that provide further analysis such as looking at the specific data regarding an specific party or candidate, or filtering the data according to various criteria such as language, time period, data source or author influence. All graph visualizations are implemented using d3.js[^22] javascript library.

The interface also has management capabilities which allows to manually review the automatic sentiment labelling. Keyword hierarchy and new website sources can be also set up through the interface. These functionalities ease the process of creating training datasets and adapting Talaia to new domains.

One of the main challenges the interface has to face is how to access data and maintain adequate time responses as the amounts of data gathered escalate. This depends to a great extent on the database optimization, but also on the number of visualizations offered by default. Talaia relies on a Mysql database. The interface does not access data from the actual tables, but rather from a joint view which is refreshed periodically. This reduces the time response from minutes to seconds [^23]

Success Cases {#isv2018:success}
=============

In this section we present two real use cases where Talaia has been applied, and use them for evaluation purposes. The first one focuses on tracking cultural events. The second one analyses citizen opinions with respect to political parties and candidates during an electoral campaign.

Both monitoring processes presented here ran on their own dedicated servers. We provide details on hardware specifications and volumes of data processed in the following subsections. As a measure of the performance capabilities of our system, the largest monitoring process we have carried out until now gathered 24M tweets per month, with an average of 700K tweets processed per day and a maximum of 1.35M tweets in a single day. Talaia ran on a server with two Intel Xeon 4 core processors (E5530) at 2.4 GHz and 16GB RAM. MySQL databases are locally stored in the server. The crawler and the text analyzers all ran locally, but no interface was implemented in this case.

Cultural Domain {#isv2018:success:cult}
---------------

Talaia was first applied in the Behagunea[^24] project. The objective of the project involved tracking the social media impact of cultural events and projects carried out (more than 500) in the framework of the Donostia European Capital of Culture (DSS2016) year during 2016. The project included monitoring opinions in press and social media in four languages: Basque, French and Spanish as coexisting languages in the different Basque speaking territories and English as international language.

Domain adapted polarity models were created. Since events related to DSS2016 were already programmed during 2015, a previous crawl was carried out in order to build datasets. Those datasets were manually annotated for polarity in a three category scale (positive, negative, neutral). Section \[isv2018:eval:datasets\] gives more details about the various language and domain specific datasets. Polarity classification models for the cultural domain were trained using those datasets and are distributed as part of EliXa. Section \[isv2018:eval:cult\] gives details related to those classifiers.

Talaia ran on an Amazon AWS t4.large dedicated instance[^25]. The crawler, the text analyzers and the interface all run locally. A total amount of 166K tweets and press mentions were gathered, with a maximum of 6.6K mentions in a single day. The interface was public and offered real-time results refreshed each 15 minutes. We can see from the volume of the data, that this was a low latency monitoring. Even if there were a lot of events to track, the local nature of most of them explains the little impact they have in social media.

Political Domain {#isv2018:success:pol}
----------------

Talaia was used to track citizen opinions during the electoral Basque electoral campaign of September 2016. The crawling was carried out during the election campaign period, starting on September 8th (23:59pm) and finishing on September 23th (23:59pm). It offers useful insights for political analysis such as sympathy rankings, the evolution of the opinions over time, most relevant messages, etc.

The system ran on a server with a Intel Xeon 4 core processor (E5530) at 2.4 GHz and 16GB RAM. MySQL databases are locally stored in the server. The crawler, text analyzers and the interface all run locally. A total amount of 4.25M tweets and press mentions were gathered, with an average of 125K mentions per day, and a maximum of 433K mentions in a single day. The interface was public and offered real-time results.

The crawler was configured to find mentions talking about the main political parties present on the campaign and their respective candidates (only main candidates monitored, i.e., those opting to be *Lehendakari*, ‘head of the government’).

Regarding social media, Twitter was monitored. Since we are talking about monitoring an event happening on a regional scope, two main restrictions were applied: only mentions written in Basque and Spanish were crawled, because those are the two official languages in the region. The second restriction was to constrain mentions to users from the specific geographical area of the Basque Country. The task was then to discard noisy messages, that do not belong to citizens involved in the election, but were likely to be talking about it. In this case, for example the crawling process was likely to capture many mentions from other regions in Spain.

In order to solve this problem we created a census of Twitter users of the Basque Country. We developed a five step algorithm:

1.  We gathered geolocated tweets from the Basque Country area for a certain period of time.

2.  Authors of those tweets were manually tagged with binary labels, as belonging to the required geographical area or not. Let’s call this dataset $D_{geo}$.

3.  Taking users tagged as Basque citizens from the previous step, we extended our dataset by retrieving up to the first 5,000 followers and friends from each user using the twitter following API[^26]. We compute the frequency of coocurrence for each of the candidates[^27], and manually label the most frequent 10,000 candidates. Let’s call this dataset $D_{geo+ff-manual}$.

4.  We train a binary SVM classifier with a linear kernel over $D_{geo+ff-manual}$. Features of the classifier are the number of followers and friends a user has and the relative number of followers and friends (with respect to the total number of follower and friends). The classifier obtains 96% accuracy in a 4-fold cross validation.

5.  Repeat step (ii) with the users labelled as Basque in $D_{geo+ff-manual}$, but this time label the most frequent 20,000 candidates with the classifier trained in step (iv). Our final census consists of 23,195 user ids.

As for the news sources, a list of 30 sources was manually compiled, including TV, printed media and radio stations, all of them with working within the regional scope.

Evaluation {#isv2018:eval}
==========

For the evaluation of Talaia, we evaluate the performance of Elixa’s polarity classifier for the two aforementioned domains. In all cases the L2-loss SVM implementation of the LIBLINEAR [@fan2008liblinear] toolkit was used as classification algorithm within Weka [@hall_weka_2009] data mining software. Experiments with polynomial kernels were also conducted (degrees 2-5) but we found no improvement at the expense of much longer training times. All classifiers presented in the following sections were evaluated by 10-fold cross validation. The Complexity parameter was optimized ($C=0.1$).

For the sake of comparison, all the systems presented from here onwards have been trained using the following set of features:

-   1-gram word forms with frequency $>=2$ and document frequency (df) $>=2$.

-   POS tag 1-gram features.

-   Polarity lemmas included in language dependent polarity lexicons. Default lexicons provided with EliXa were used (see Table \[isv2018:eval:tabLexicons\] for details).

-   Sentence length.

-   Upper case ratio: percentage of the capital letters with respect to the total number of characters in a sentence.

Microtext normalization features (URL standardization, OOV normalization and emoticon mapping) are applied before extracting the features of each sentence.

  [**Language**]{}   [**Lexicon**]{}                       [**\#neg. entries**]{}   [**\#pos. entries**]{}   [**Total entries**]{}
  ------------------ ------------------------------------- ------------------------ ------------------------ -----------------------
  eu                 $ElhPolar_{eu}$ [@SANVICENTE16.468]   742                      499                      1,241
  es                 $ElhPolar_{es}$ [@Saralegi2013b]      3,314                    1,903                    5.217
  en                 $EliXa_{en}$ [@elixa-semeval2015]     6,123                    3,992                    10,115
  fr                 Feel[@abdaoui2017feel]                5.717                    8,430                    14,147

Datasets {#isv2018:eval:datasets}
--------

Table \[isv2018:eval:datasets:table-cult\] presents the statistics and class distributions of the datasets gathered and annotated in order to build the polarity classifiers for each language in the cultural domain. All annotations were done manually. Polarity was annotated at mention level. Because of the level of specificity reached when defining the keyword taxonomy, we rarely find a mention referring to more than one entity or event. Statistics show that corpora in all languages have a similar distribution, with a high number of neutral mentions, and a larger presence of positive opinions than negative ones.

   [**Language**]{}   [**Total size**]{}   **\#pos**   **\#neg**   **\#neu**
  ------------------ -------------------- ----------- ----------- -----------
          eu                 2937             931         408        1598
          es                 4754            1487        1303        1964
          en                12,273           4,654       1,837       5,782
          fr                11,071           3,459       2,618       4,994

Table \[isv2018:eval:datasets:table-pol\] shows the characteristics of the political domain datasets. In this case, each tweet was annotated with respect to a number of entities appearing in the tweet. Annotators were asked to annotate the polarity of a tweet from the perspective of each of the entities detected in a tweet, that is, a tweet may contain more than one polarity annotation. Example \[isv2018:eval:datasets:example\] shows a real case where a tweet was given two different annotations, one for each entity (negative expressions underlined, positive ones in bold). In fact, the numbers in table \[isv2018:eval:datasets:table-pol\] give 1.3 and 1.24 average annotations per tweet for Basque and Spanish, respectively.

erabat ados, lotsagarria. Aukera ona aurrera begiratu ta **@ehbildu**|ren euskara arloko proposamena martxan jartzeko \#herriakordioa [[^28]]{} \[isv2018:eval:datasets:example\]

Annotating tweets in the political domain proved to be a rather challenging task. Sarcasm is often present, interpellations to a person are frequent even if they are not the target of the opinion, an opinion may be present but in an implicit manner, or a third party negative opinion may be expressed towards an entity but the author may defend it against the expressed opinion. The full annotation guidelines can be consulted in Annex .

Annotation was carried out in real time during the period of the electoral campaign. We established three shifts a day to annotate messages gathered until then. Three annotators took part in the process. Because of the limited resources and the volume of messages crawled daily, each tweet was annotated by a single annotator.

![Distribution of mentions in Spanish with respect to the political parties. From left to right, parties are sorted according the percentage of negative opinions received, with respect to the total amount of mentions received.[]{data-label="talaia:fig:neg_es"}](bec2016_neg_eu.eps){width="\linewidth"}

![Distribution of mentions in Spanish with respect to the political parties. From left to right, parties are sorted according the percentage of negative opinions received, with respect to the total amount of mentions received.[]{data-label="talaia:fig:neg_es"}](bec2016_neg_es.eps){width="\linewidth"}

Political domain datasets show very different distributions across languages. While the Basque dataset seems to follow the same pattern seen in the cultural domain, the Spanish dataset has a very high number of negative opinions. Analyzing some result samples, we realized that there are various phenomena that could explain this behaviour. First, much more debate and criticism takes place in Spanish compared to Basque where the tendency is to write a lot more supportive messages. Those not associated with Basque nationalist ideologies mainly communicate in Spanish. A clear example is that the left party EH Bildu has very few negative mentions in Basque (See figure \[talaia:fig:neg\_eu\]). Also the fact that the right wing parties such as *Partido Popular* (PP) and *Ciudadanos* (Cs) receive almost no attention in Basque is a symptom of the little engagement they show in this language.

Second, left wing people were more active in Twitter (in this specific campaign), with right wing parties concentrating the largest amount of negative mentions (see figures \[talaia:fig:neg\_es\]).

Lastly, there is the effect of negative campaigning [@skaperdas_grofman_1995], which is more pronounced in Spanish, because as we already said there is much more debate than in Basque. Figure \[talaia:fig:neg\_es\] also aligns with studies of negative campaigning in multi-party scenarios [@walter2014choosing; @haselmayer2017sentiment], being the front-runner *Partido Nacionalista Vasco* (PNV) and its previous government partner *Partido Socialista de Euskadi* (PSE) those who receive the greatest amount of negative mentions.

   [**Language**]{}   [**\#Tweets**]{}   [**\#Annotations**]{}   **\#pos**   **\#neg**   **\#neu**
  ------------------ ------------------ ----------------------- ----------- ----------- -----------
          eu               9,418                11,692             3,974       3,185       4,533
          es               15,550               20,278             3,788       7,601       8,889

Results {#isv2018:eval:cult}
-------

Table \[isv2018:eval:table\] shows the performance of the various multilingual classifiers trained.

Reported results are in general higher for the cultural domain, even if the datasets are smaller in comparison. Basque and Spanish classifier obtain results above 70%. English and French achieve lower results. Positive mentions present the greatest challenge for English. The main reason for this is the lack of positive training examples. Neutral mentions perform very good in all languages. After analyzing a random sample, we conclude that neutral mentions are of an homogeneous nature, mainly containing agenda events or promotion messages. If we add this to the fact that neutral is the class with the highest number of examples, it seems logical that our classifiers find highly representative features for this class.

   [**Language**]{}   **\#features**   **acc**   **fpos**   **fneg**   **fneu**
  ------------------ ---------------- --------- ---------- ---------- ----------
                                                                      
          eu              4,777         74.02     0.658      0.635      0.803
          es              10,037        73.03     0.683      0.756      0.744
          en              24,183        70.43     0.715      0.530      0.743
          fr              23,779        66.17     0.600      0.617      0.721
                                                                      
          eu              9,394         69.88     0.714      0.702      0.683
          es              15,751        67.05     0.545      0.693      0.700

Regarding the political domain, if we compare Basque and Spanish classifiers, their performance drops around 4% with respect to the results in the cultural domain. Results are not directly comparable, because political domain classifiers are evaluated over entity level tags. Also, political data is more challenging in terms of the linguistic phenomena used. We have detected a fair amount of messages containing sarcasm or opinion ambiguity towards targets.

In the case of Basque, the most sensible drop happens with neutral mentions. After analyzing a random sample we found that they are more heterogeneous than those in the cultural domain. They do contain agenda and promotion messages, but also many third party statements (candidate x says “...”) or messages that interpellate parties and candidates over hot topics in the campaign (e.g. “@DanielMaeztu @ehbildu @PodemosEuskadi\_ obra gelditzea onuragarria liteke ekonimia arloan 4.000 miloi gastatu eta gero?”[[^29]]{}). Many neutral messages contain personal opinions not involving any of our predefined target entities, even if they are interpellated. These phenomena make neutral class harder to represent.

Regarding Spanish, performance for positive mentions is significantly lower. Error analysis shows that incorrectly classified instances do not fall into a single category (42% negative, 58% neutral). Analysing the errors, we find two main reasons. First, many positive mentions are incorrectly classified as negative because their content is mainly negative (e.g. “ @AgirreGarita La diferencia es clara, PNV apoyando el desahucio y EHBILDU al desahuciado. NO SEAS COMPLICE, no votes a quien desahucia.” [[^30]]{}). Our strategy for assigning message level polarity to all entities involved in a mention is prone to this type of errors. Second, as we saw for Basque, neutral mentions also contain polar expressions or opinions, making it harder to distinguish them from actual positive or negative messages.

Conclusion and Future Work {#isv2018:conc}
==========================

We have presented Talaia, a real time monitor of social media and digital press. Talaia is able to extract information related to an specific topic and analyse it by means of natural language processing technologies. Two success cases and the resources generated from those cases have been described. In that sense, we have shown the ability to adapt our system to different domains and languages.

Talaia is still under development. The short term objectives include work on optimizing the information extraction process. Specifically, extracting keywords from the data downloaded up to a certain point would allow us automatically adapt the system to new terms, without losing information because the keyword hierarchy is outdated or the topic is poorly defined.

Another important point is the adaptation of our sentiment analysis model to new domains. In that sense experiments are being carried out in order to minimize the domain adaptation process, both in terms of data collection and annotation effort.

Multilinguality is one of the main challenges of such a system. Currently the system is able to process data in 4 languages, and we are working to extend it to new languages.

Furthermore, data analysis may include further processing other than sentiment analysis. Geolocation based analysis, user community detection and other useful tasks for user profiling (e.g. gender detection) are the focus of our ongoing work.

All the software behind the platform including the crawler, data processing chain and interface is publicly available under the GNU GPLv3 license.

Acknowledgements {#acknowledgements .unnumbered}
================

This work has been supported by the following projects: Elkarola project (Elkartek grant No. IE-14-382), and Tuner project (MINECO/FEDER grant No. TIN2015-65308-C5-1-R).

Comparative of commercial Social Media Monitors {#isv2018:annex:commercial}
===============================================

  --------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------ ---------------------------------------------------------------------------------------------------------------------------------------
  **Platform**    **Data Sources**                                                                                                                                                                                                                                                    **Crawling**                                                                             **Data Processing**                                                                                                                                                                                                                                                                            **Search**                                                                                                               **Navigation**
  Iconoce         Digital press, blogs, videos, social media (Facebook, Twitter, Linked-in?)                                                                                                                                                                                          Personalized, subject to agreement                                                       no                                                                                                                                                                                                                                                                                             Personalized archive 3 separate search engines (mentions, comment, authors) no lemmatization - no crosslingual.          Graphs (aggregations?), salient terms, salient topics, Influencers, alerts, reports
  INNGUMA         Rss multimedia, Deep Web, Twitter, Facebook, Linkedin, possibility to include external documentation manually                                                                                                                                                       Custom filters, not clear if filtering is done at a post-crawling stage                  MT, No mention of text processing. No SA                                                                                                                                                                                                                                                       Semantic search (techniques not especified). Index cards and documents. Information is tagged manually.                  Reports, content creation, social media management. Multilingual GUI.
  Meaning Cloud   Digital news, blogs, Twitter, satisfaction surveys (customer provided), phone survey transcriptions,                                                                                                                                                                                                                                                         5 languages (Es, En, Fr, Pt, It). Language identification, Clustering for topic detection. Lemmatization, pos tagging, parsing, NERC, GATE API SA: Ruled-based. Sentiment Lexicons + rules. Irony and subjectivity detection. Entity polarity detected using manually compiled dictionaries.   no                                                                                                                       No Dashboard, visualizations or data aggregatios. Excel plugin or API access
  Snap-trends     Social Media (Twitter, Facebook, Instagram, Google+,...)                                                                                                                                                                                                                                                                                                     MT from 80 languages. Propietary linguistic processing. Topic (trends) detection.Propietary sentiment analysis.                                                                                                                                                                                Geolocation based search engine, mutiple criteria: social network, search terms, geolocation. Previous search feature.   Agreggations, interactive visualization, temporal trends.
  Websays         News, Blogs/RSS, Forums, Facebook, Twitter, Google+, LinkedIn, Instagram, Foursquare, Pinterest, Youtube, Vimeo, Reviews (Tripadvisor, Booking,...)                                                                                                                 Keyword based, accepts also negative keywords.                                           Multilingual data processing, no specific data about the coverage SA: AI (ML) + human validation                                                                                                                                                                                               Multiple search criteria, filter-based.                                                                                  Graphs, salient terms, trending topics, influencers, sentiment, trends. Alerts and reports.
  Lynguo          Facebook, Twitter, Instagram, YouTube, online media, blogs and forums.                                                                                                                                                                                              Keyword based, accepts also negative keywords and accounts.                              Es,En. SA: ML + Lexicons + rules. Polarity and emotions. Aspect based SA                                                                                                                                                                                                                                                                                                                                                Customizable dashboard. Several default aggregations and possibility to generate custom visualizations. Alerts and periodical reports
  Keyhole         Twitter, Instagram, web sources.                                                                                                                                                                                                                                    Social media and web sources are configured and monitored separately. Keywords, users.   13 languages. SA: Polarity                                                                                                                                                                                                                                                                                                                                                                                              Influencers, timeline, trends, sentiment, aggregations.
  Uber-metrics    Blogs, forums, academic/scientific journals, digital press, Instagram, Tumblr, Google+, Facebook, Twitter, YouTube, Vimeo, Flickr, and Foursquare. With Ubermetrics you can even capture comments from YouTube, Facebook, and major online news sources. TV/Radio   Customizable “search agents”. Keyword based                                              40 languages. Propietary data processing.                                                                                                                                                                                                                                                      Detailed search based on multiple criteria included in visualization dashboard                                           Dashboard, alerts, reports.
  --------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------ ---------------------------------------------------------------------------------------------------------------------------------------

  : Comparison of commercial social media monitoring platforms.[]{data-label="isv2018:soa:indust:comptable"}

Polarity annotation guidelines {#isv2018:annex:guidelines}
==============================

We present the guidelines provided to the annotators for marking entity level polarity, including ambiguous cases and the solutions proposed for each of them:

-   Neutral: There is no clear opinion or sentiment respect to the target party or candidate from the holder. Mentions referring to objective facts fall into this category as well, even if the fact may be considered positive or negative (e.g. *“El PNV consigue grupo en el senado”*[[^31]]{} ).

-   Positive: The mention includes a positive assessment from the holder with respect to the target (e.g. *“Urkullu ha sido un buen lehendakari.”*[[^32]]{}).

-   Negative: The mention includes a negative assessment from the holder with respect to the target (e.g. *“Urkullu ha sido un lehendakari mediocre.”*[[^33]]{}).

-   ambiguous cases:

    1.  Subjectivity is not explicit: *“Cataluña desobedece constantemente la Ley, PNV pide acercamiento de presos, Ribo da los pasos hacia el nacionalismo y Rajoy en SanXenso”*[[^34]]{}. Main target in the example is “Rajoy” but author expresses a negative opinion towards PNV. Annotators were ask to interpret the implicit subjectivity according to the holder.

    2.  The holder expresses the opinion of a third party: *“Podemos cree que Urkullu tiene miedo y por eso adelantará las elecciones - EcoDiario.es &lt;URL&gt;”*[[^35]]{}.The mention expresses a negative opinion from Podemos towards Urkullu. Annotators were asked to annotate it as negative towards Urkullu if they could certify that the holder agreed with the opinion from Podemos, or netural otherways.

    3.  There are two (or more) references to a single target, expressing different polarities: *“PNV tendrá grupo propio en el Senado tras la cesión de cuatro asientos por parte del PP y mantiene su “no a Rajoy” ”*[[^36]]{}. The following criteria were applied: N+P=NEU, N+NEU=N, P+NEU=P.

    4.  The polarity of the message and the polarity towards the target are different: *“El tercer precandidato de \#Podemos llama a desalojar al PNV &lt;URL&gt;”*[[^37]]{}. In those cases, polarity towards the target should be annotated. In the example, message polarity would be neutral, but polarity towards “PNV” would be negative. Thus message would be marked as negative.

    5.  Irony/sarcasm: *“¿Las cambiamos por Calle Arnaldo Otegi o Paseo de Juana Chaos? Al fin y al cabo, son hombres de paz... &lt;URL&gt;”*[[^38]]{}. Annotators were asked to interpret irony. The previous example would be thus negative towards the target Arnaldo Otegi.

    6.  The holder is condemning a negative stance against the target: *“@eldiarionorte cada vez se os ve más el plumero. Panfleto anti bildu. Cuando la salud de los zubietarras empeore, vais y se lo contáis.”*[[^39]]{}. Annotators were asked to interpret the intention of the holder. If the notice a clear intention of defending the target the it should be regarded as positive.

    7.  The target captured is not the main focus of the opinion: *“@CristinaSegui\_ Subió impuestos,no hace nada contra los nacionalistas y les da dinero,no ilegaliza a Bildu y stá implicado en lo de Bárcenas”*[[^40]]{}. Annotators were asked to mark the polarity towards the target, regardless of the main focus of the opinion.

[^1]: <https://www.lracrisistracker.com>

[^2]: <http://noisy-text.github.io/2018/>

[^3]: Collect tweets containing the “:)” emoticon and regard them as positive, and likewise for the “:(” emoticon.

[^4]: <http://info.iconoce.com/>

[^5]: <https://www.innguma.com>

[^6]: <https://www.lexalytics.com>

[^7]: <https://www.meaningcloud.com/>

[^8]: <https://websays.com/>

[^9]: <https://keyhole.co/>

[^10]: <http://lynguo.iic.uam.es/>

[^11]: <http://www.iic.uam.es/en/big-data-services/digital-environment/lynguo-en/>

[^12]: <http://www.iic.uam.es/en/big-data-services/customer-intelligence-environment/natural-language-processing/>

[^13]: <https://www.ubermetrics-technologies.com/>

[^14]: <http://snaptrends.com/>

[^15]: <http://github.com/Elhuyar/MSM>

[^16]: https://github.com/optimaize/language-detector

[^17]: https://github.com/Elhuyar/Elixa

[^18]: $n$ varies depending on the size of the input corpus. We reviewed up to 1,500 candidates.

[^19]: OpenStreetMap Nominatim (<https://wiki.openstreetmap.org/wiki/Nominatim>) and Google Geocoding API <https://developers.google.com/maps/documentation/geocoding/intro>).

[^20]: <https://www.djangoproject.com/>

[^21]: Existing demos and installations implement different visualizations. Visualizations for the use cases described in this paper can be seen at <http://talaia.elhuyar.eus/demo_eae2016> and <http://behagune.elhuyar.eus/>

[^22]: <https://d3js.org/>

[^23]: Queries retrieving a million results could take up to 7 minutes (depending on the visualizations required)), while using a joint view takes 40 seconds for the same configuration.

[^24]: <http://behagune.elhuyar.eus>

[^25]: Especications are 2 vCPUs, 8GB RAM, 100GB EBS storage disk. More information at [https:
        //aws.amazon.com/ec2/instance-types/](https:
        //aws.amazon.com/ec2/instance-types/)

[^26]: <https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-show>

[^27]: The number of times a candidate appears as a follower or friend of another candidate.

[^28]: English translation: [@pnvgasteiz totally agrees, shameful. Good chance to look forward and apply the proposal of @ehbildu in the field of Basque \#herriakordioa ]{}

[^29]: English translation: [@DanielMaeztu @ehbildu @PodemosEuskadi\_ stopping the construction would be beneficial after spending 4,000 millions?]{}

[^30]: English translation: [@AgirreGarita The difference is that PNV is in favour of evictions and EHBILDU is with the evicted ones. DO NOT BE AND ACCOMPLICE, don’t vote to those who practice evictions.]{}

[^31]: English translation: [PNV gets its own group in the senate]{}

[^32]: English translation: [Urkullu has been a good president]{}

[^33]: English translation: [Urkullu has been a mediocre president]{}

[^34]: English translation: [Catalunya constantly disobeys the law, PNV asks for the rapprochement of prisoners, Ribo makes steps towards nationalism and (meanwhile) Rajoy is in SanXenso.]{}

[^35]: English translation: [Podemos thinks Urkullu is scared and that’s why he will call the election early - EcoDiario.es &lt;URL&gt;.]{}

[^36]: English translation: [PNV will have its own group in the senate thanks to PP handing over four seats, and they still maintain the “No to Rajoy”.]{}

[^37]: English translation: [The number three shortlisted candidate of \#Podemos calls on the people for throwing PNV out &lt;URL&gt;]{}

[^38]: English translation: [What if we change the name of the street to Arnaldo Otegi St. or Paseo de Juana Chaos? After all, they are men of peace... &lt;URL&gt;]{}

[^39]: English translation: [@eldiarionorte it is increasingly clear what you are up to. Anti Bildu pamphlet. When the people in Zubieta lose their health go and tell them.]{}

[^40]: English translation: [@CristinaSegui\_ He increased taxes, he does nothing against nationalists and gives them money, he does not ban Bildu and he is involved in the Bárcenas affair.]{}
