Presenters

Reinhard Schunck is Professor of Sociology at the University of Wuppertal. He works primarily in the field of social stratification and inequality, concentrating on migration and family related processes, and has a focus on quantitative methods.

Nora Huth-Stöckle is a doctoral student and works at the University of Wuppertal. Her research interests comprise intergroup relations, educational inequality, and quantitative methods.

Contents

1. How to use web log and related data for social research
2. How to obtain web log and related data
3. How to handle “big data”
4. About the presenter
5. Further reading
6. References

How to use web log and related data for social research

Web logs (“browsing histories”) are records of app use and search queries. They are highly interesting data sources for research in the social sciences as they offer detailed insights into human behavior. However, only a few studies have used such data in the social sciences so far. Examples of such studies include Stephens-Davidowitz (2014), who studied racial animus in the 2012 U.S. presidential election, Peterson, Goel, and Iyengar (2018) and Flaxman, Goel, and Rao (2016), who analyzed filter bubbles, echo chambers and partisan polarization in the U.S. and Guess, Nyhan, and Reifler (2020) who studied the spread of fake news in the U.S. In the Netherlands, Möller et al. (2019) analyzed online news engagement based on three different modes of news use. In Israel, Dvir-Girsman (2017) documented that audience homophily is higher among individuals with more extreme ideology and that it is associated with ideological polarization and intolerance. Bach et al. (2019) showed for Germany that online and mobile device activities predict voting behavior and political preferences to a limited degree only. Chancellor and Counts (2018) show that internet search data can be used to estimate employment demand in the U.S.

In other disciplines, similar data have been used, for example, to predict influenza activity (Ginsberg et al. 2009, but see @lazer2014parable) and to show that users who search for relatively harmless symptoms easily end up searching for serious diseases (White and Horvitz 2009). Another study demonstrates how concerns about pregnancy and childbirth change over the course of pregnancy (Fourney, White, and Horvitz 2015). Furthermore, several papers show how a variety of user attributes such as socio-demographics can be inferred from web logs, search queries and app records (see Hinds and Joinson 2018 for a recent overview). However, many of those studies have been conducted by researchers from computer science. While they often rely on search query data obtained from search engine providers like Bing and Google, they typically focus more on the technical aspects and on the evaluation of the performance of the underlying algorithms. As social scientists, however, we often focus on the theory-driven development of models and the testing of hypotheses about human and societal behavior. Thus, this blog post will focus on the latter.

One challenge that researchers face when designing studies that rely on web logs, records of app use, and search queries is how to get access to such data. Several of the studies mentioned above use large amounts of search query data from search engines like Google or Bing (Stephens-Davidowitz 2014; Chancellor and Counts 2018; White and Horvitz 2009; Fourney, White, and Horvitz 2015). These data are, however, usually only available if one teams up with researchers from the respective companies. Another way to obtain data is through commercial providers who keep opt-in panels of users who occasionally answer survey questions in exchange for money. Researchers can pay those vendors in order to get access to their panels and ask participants survey questions. In addition, several of these providers also offer web log and mobile device use records from users who (in exchange for additional pay) agreed to having their online mobile activities monitored. In this blog post, we will mainly focus on this latter way of obtaining data. Before we talk about this topic in more detail, we will briefly summarize what we mean when we speak of web logs, records of app use, and search queries.

To get a better understanding of such data, the table below shows a collection of a few artificial web logs made up for this blog post. Typically, we observe a person identifier (first column) for the person whose records we observe. Second, we have a URL (Uniform Resource Locator) column which tells us which URL this person visited, when (column “Timestamp”) and for how long (“Duration of use”; here, in seconds). The most interesting information in this table is the URL column. Even without a detailed understanding of the specific form of a URL, we can easily see that, in the first row, Person 1 visited a web address that seems to inform her about the weather in Mannheim. In addition, we observe when she visited this address and how much time she spent there. The second row tells us that this person likely sent (or received) a message to (from) Peter Mustermann. We cannot, however, observe the content of the message (which we also should not, given obvious privacy reasons). The third row shows that Person 1 then visited the Facebook page of the CDU. The fourth row shows that she watched a video on YouTube. If we accessed the web address (or programmed a scraping tool), we could also learn what the video was about. From the visit in the fifth row, we learn that this person read an article about the state of the German economy on DIE ZEIT, a German newspaper.

With respect to Person 2, we can also observe what items they searched on Amazon (sixth row) or which tweet they saw (seventh row). We also learn that they searched for the voting advice application “Wahl-O-Mat” (ninth row). From the URL in row 9, we can extract the exact words a person entered into a search engine (the “search queries”). From row 10, we also know that Person 2 then actually used this tool. Thus, we already see that we can learn a lot about users’ behavior, their interests and preferences.

We observe similar records regarding app use on mobile devices, as shown in the next table. Here, we observe the name of an app instead of a URL. Unfortunately, we cannot observe what users do inside the apps they use. That is, we do not know which articles they read when they open a newspaper app or which videos they watch on YouTube or Netflix. Yet, if somebody frequently opens newspaper apps, we might infer that this person may be interested in politics. Moreover, users who use an app called “Period Tracker” are likely female and we may even learn when they have their period by observing temporal variations in usage of this app. Similarly, somebody who uses an app that informs them about prayer times is likely religious. These are just a few examples that show how observing users’ web logs or app use can reveal information that users may perceive as sensitive personal information (see Bach et al. 2019 for a study on this topic).

It is important to note that participants of commercial panels can usually pause data collection on their devices temporarily. This might happen if they do not feel comfortable having their activities recorded, e.g. when they do online banking or watch movies from illegal streams. However, our own data collections show large amounts of potentially sensitive information (such as adult content, gambling, and illegal streaming), which suggests that users do not make use of this possibility very often.

How to obtain web log and related data

As mentioned earlier, the easiest way to obtain web log and app use data is through commercial vendors who operate online access panels (“non-probability panels”). So far, there seem to be only a handful of providers, such as respondi AG (Germany, UK and France, for example), YouGov (UK and US, amongst others) and netquest (mostly Spain and Latin America). For an overview of providers, see for example, Wakoopa Hub. Since such data have not been available for long, we do not know much about the quality of such data (for an exception see Revilla, Ochoa, and Loewe 2017).. In addition to accessing web logs and app use data, researchers can collect survey data for the same individuals. That is, by asking users questions we can enrich the web log data with important information about users’ socio-demographic characteristics, their voting behavior, or their political preferences. This information is usually not directly observable from the web logs and app use records, but likely makes the data much more valuable for social scientific research purposes. However, we should always keep in mind that due to the opt-in nature of these access panels, we need to be careful when making statements about the representativeness of our findings. That is, because most of our statistical estimators (e.g., for variances) rely on true probability sampling, we need to adjust our models and estimators, e.g. by using weights (for more information on non-probability sampling and non-probability panels see, e.g., Mercer et al. 2017; Cornesse et al. 2020).

Another way to obtain data on users’ online activities is through Google Trends. Briefly speaking, this service allows the estimation of the popularity of search queries in Google Search across various regions and languages. Several R packages in R allow users to automatically extract data from Google Trends – see, for instance, gtrendsR. Further details on Google Trends can be found here. A major drawback of these data is that they do not come with additional information about users and only provide information at aggregate levels. That is, one can learn only about the popularity of a specific search query compared to the popularity of other search queries, which is arguably much less informative than individual-level web logs.

A third way to obtain web log, app, and search query data is through developing one’s own research app. A few projects using this approach were launched in recent years. One of the most prominent examples is the IAB-SMART project (for details, see Kreuter et al. 2019). Researchers of the Institute for Employment Research (IAB) in Nuremberg, Germany, developed an app that has since been downloaded by several hundred respondents of the IAB’s panel survey “Labour Market and Social Security”. The app collects information on respondents’ smartphone activities and can access even more information than those described above (including geolocation and address books). While this approach is likely the most elaborate, it is also the most difficult and most costly one to implement: Researchers need to program their own tools and recruit participants on their own or piggyback on an existing study.

How to handle “big data”

Finally, a note on useful tools and prerequisites for analyzing web logs and records of smartphone use. First of all, the amount of data can quickly exceed the computational power of a standard desktop computer. Four months of web log data used in Bach et al. (2019), for example, contained about 38 million observations. Working with data of this size, researchers may have to consider using remote computing services like Digital Ocean, Amazon AWS or Microsoft Azure, which offer computational resources for little money through virtual servers. Second, understanding URL contents by observing single URLs is straightforward. Analyzing thousands of URLs, however, requires text mining and natural language processing (NLP) techniques if one wants, for example, to select only those URLs that point to news articles. Moreover, in addition to analyzing the title of a news article (which can often be observed from the URL alone), one might also want to analyze the whole content of the article. In such cases, in addition to being able to automatically extract the topic of an article through NLP techniques, knowing how to scrape website contents will likely also be helpful. Some useful materials are linked below.

About the presenter

Ruben Bach is a postdoctoral researcher at the University of Mannheim, focusing on social science quantitative research methods. His interests include topics related to big data in the social sciences, machine learning, causal inference, and survey research.

Further reading

• Meyer, Cosima and Cornelius Puschmann. 2019. Advancing Text Mining with R and quanteda. Methods Bites: Blog of the MZES Social Science Data Lab. Blog Post Tutorial.
• Munzert, Simon. 2016. Three easy-to-learn tools to scrape data from the Web with R. MZES Social Science Data Lab. Workshop Materials.

Contents

1. Shiny Apps and Data Presentation
2. The Structure of Shiny Apps
3. Developing Shiny Apps
   1. Building a Shiny App from Scratch
   2. Building the UI
   3. Implementing the Server Logic
   4. Rendering Plots
4. Deployment
   1. Deployment using shinyapps.io
   2. Deployment using Shiny Server
5. Conclusion

Shiny Apps and Data Presentation

Data are usually collected in a raw format and, no matter how well processed and analyzed, results should be presented in a plain and simple way in order to make them accessible for everyone. Unfortunately, the potential of effective data presentation is all too often not fully exploited. Yet, presentation is crucial as it illustrates the actual outcome of research. If not effectively visualized, we waste a great opportunity to build credibility, attract and sustain the interest of readers, and make large amounts of information easily accessible. Here, we introduce Shiny Apps as a powerful and highly flexible tool for interactive data presentation in the world wide web.

Shiny is an R package that offers cost- and programming-free tools for building web applications using R. It was developed by Joe Chang to serve as a reactive web framework for R that allows calculations, display of R objects, and the presentation of results. Since Shiny Apps come with an extensive back-end setup, users do not need extensive web development skills to build and host standalone apps on a homepage. However, for those keen on bringing their apps to perfection, Shiny Apps allows for CSS, HTML and JavaScript extensions.Shiny Apps can be used either for data presentation, as a communication tool for results, or even as an interactive analytical tool.

In what follows, we introduce the Shiny environment and guide readers through the development of Shiny Apps. Using the famous Kaggle titanic data set, we draw the distinction between the front-end ui.R and back-end server.R, which are required to build Shiny apps. Following this, we introduce important concepts and features to build an interactive app, including control widgets, reactivity, and rendering.

The Structure of Shiny Apps

Shiny applications have two components. The front-end builds the webpage that is actually shown to the user. As already mentioned, the HTML page is written by Shiny itself and includes layout, appearance, and design features. In Shiny terminology, this is called the ui, which stands for user interface. The ui file contains R functions that are then translated into an HTML file. The other component is the back-end, which includes the code for producing the app’s contents (e.g. functions or data import, management, and analysis). Here, we create the objects that are later shown on the front-end. In Shiny terminology this is called the server.

Developing Shiny Apps

## Packages
pkgs <- c("shiny",
          "tidyverse",
          "titanic",
          "plotly")

## Install uninstalled packages
lapply(pkgs[!(pkgs %in% installed.packages())], install.packages)

## Load all packages to library
lapply(pkgs, library, character.only = TRUE)

## Warning: package 'shiny' was built under R version 3.6.3

## Warning: package 'ggplot2' was built under R version 3.6.3

## Warning: package 'tidyr' was built under R version 3.6.3

## Warning: package 'plotly' was built under R version 3.6.3

ui <- fluidPage()
server <- function(input, output) {
}

runExample("01_hello")
# To show other Apps, please type runExample(NA) and choose another example

ui <- fluidPage(titlePanel("Title of my Shiny App"),
                sidebarLayout(
                  sidebarPanel("My input goes here"),
                  mainPanel("The results go here")
                ))

# The most simple structure of the UI:
ui <- fluidPage(
  titlePanel("Visualizing the Titanic data set"),
  sidebarLayout(
    sidebarPanel(
      checkboxInput("checkbox",
                    "Activate all filters",
                    FALSE),
      radioButtons(
        "buttons",
        "Did the passenger survive?",
        choices = c("did not survive" = 0, "did survive" = 1),
        selected = 0
      ),
      selectInput("selector",
                  "Select the class of the passenger",
                  choices = c(1, 2, 3)),
      sliderInput(
        "slider",
        "Pick a range of fare (in $)",
        min = 0,
        max = 550,
        value = c(10, 50),
        pre = "$"
      ),
      selectInput(
        "selectVars",
        "Select the variable to be displayed",
        choices = c(
          "Age" = "Age",
          "# of Siblings/Spouses on board" = "SibSp",
          "# of Parents/Children on board" = "Parch"
        )
      )
    ),
    mainPanel(plotlyOutput("greatPlot"),
              textOutput("goodText"))
  )
)

output$greatPlot <- renderPlot({
  plot(rnorm(input$slider[1]))
})

# Back-end
# The server function takes input and output as arguments
# input: Input provided by the user to specify a certain appearance, data 
#        or result
# output: The output created by the functions specified in the back end

server <- function(input, output) {
  # Create a reactive variable to use it for different objects interactively
  filtered <- reactive({
    titanic_train %>%
      filter(
        Fare >= input$slider[1],
        Fare <= input$slider[2],
        Pclass == input$selector,
        Survived == input$buttons
      )
  })
  
  # Plot the variables of the titanic_train data set
  # Using if-conditions to specify the variables being plotted
  output$greatPlot <- renderPlotly({
    if (input$checkbox == TRUE) {
      x <- filtered()[, input$selectVars]
      plotHist <- ggplot(filtered(), aes(x)) +
        geom_histogram()
    } else {
      x <- titanic_train[, input$selectVars]
      plotHist <- ggplot(titanic_train, aes(x)) +
        geom_histogram()
    }
    
    ggplotly(plotHist)
  })
  
  output$goodText <- renderText({
    if (input$checkbox == TRUE) {
      x <- nrow(filtered()[filtered()$Survived == 1, ])
      text <-
        paste0("Using your filter ", x, " passengers have survived")
      return(text)
    } else {
      x <- nrow(titanic_train[titanic_train$Survived == 1, ])
      text <- paste0("In total ", x, " passengers have survived")
      return(text)
    }
  })
}

Deployment

sudo apt-get install r-base

sudo apt-get -y install libcurl4-gnutls-dev 
sudo apt-get -y install libxml2-dev libssl-dev

install.packages("shiny")

wget https://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.5.6.875-amd64.deb
sudo gdebi shiny-server-1.5.6.875-amd64.deb

sudo ufw allow 3838

Conclusion

Data presentation is crucial for accessible scientific research. Even with accurate and comprehensive data that supports relevant findings, failure to present the data in an easily accessible way can result in wasted opportunities. Shiny Apps are a powerful and free-of-cost tool that allows scholars to provide readers with easy-to-use, interactive, and comprehensive insights into their research. For instance, scholars can use Shiny Apps to program an interactive online appendix and thereby offer readers full control to compare findings under different specifications of measurement and modeling.

However, the advantages of creating a functional web-application exclusively with R has some limitations. Considering the design and the placement of inputs (such as sliders) or outputs (such as graphics and tables), Shiny is limited. When combining Shiny with HTML, CSS, and JavaScript, however, it offers a good way to program professional web-apps. Shiny is particularly great for fast prototyping and fairly easy to use with little experience in programming. It comes with many different charting libraries and captures feedback and comments in a structured manner. It therefore offers an exciting toolbox that can be a valuable addition to scientific research.

About the Presenter

Konstantin Gavras is a Ph.D. candidate at the Graduate School of Economic and Social Sciences in Political Science, research associate at the Chair of Political Psychology at the University of Mannheim and doctoral researcher for the MZES project “Fighting together, moving apart? European common defence and shared security in an age of Brexit and Trump”. His research interests comprise the intersection of Social Psychology and Political Behavior, focusing on the behavioral consequences and conditions underlying political attitudes regarding both domestic and foreign policies.

Overview

1. What is quanteda?
2. How do we use quanteda?
3. Classification
   1. Known categories
      1. Dictionaries
      2. Supervised machine learning
         1. Naive Bayes (NB)
   2. Unknown categories
      1. Unsupervised machine learning
         1. Latent semantic analysis (LSA)
         2. Latent Dirichlet Allocation (LDA)
         3. Structural topic models (STM)
4. Further readings

This blog post is based on this report and on Cornelius’ post on topic models in R.

What is quanteda?

In order to analyze text data, R has several packages available. In this blog post we focus on quanteda. quanteda is one of the most popular R packages for the quantitative analysis of textual data that is fully-featured and allows the user to easily perform natural language processing tasks. It was originally developed by Ken Benoit and other contributors. It offers an extensive documentation and is regularly updated. quanteda is most useful for preparing data that can then be further analyzed using unsupervised/supervised machine learning or other techniques. A combination with tidyverse leads to a more transparent code structure and offers a mere variety of useful areas that could not be addressed within the limited time of the workshop (e.g., scaling models, part-of-speech (POS) tagging, named entities, word embeddings, etc.).

There are also similar R packages such as tm, tidytext, and koRpus. tm has simpler grammer but slightly fewer features, tidytext is very closely integrated with dplyr and well-documented, and koRpus is good for tasks such as part-of-speech (POS) tagging).

How do we use quanteda?

Most analyses in quanteda require three steps:

_{1. Import the data}

The data that we usually use for text analysis is available in text formats (e.g., .txt or .csv files).

_{2. Build a corpus}

After reading in the data, we need to generate a corpus. A corpus is a type of dataset that is used in text analysis. It contains “a collection of text or speech material that has been brought together according to a certain set of predetermined criteria” (Shmelova et al. 2019, p. 33). These criteria are usually set by the researchers and are in concordance with the guiding question. For instance, if you are interested in analyzing speeches in the UN General Debate, these predetermined criteria are the time and scope conditions of these debates (speeches by countries at different points in time).

_{3. Calculate a document-feature matrix (DFM)}

Another essential component for text analysis is a document-feature matrix (DFM); also called document-term matrix (DTM). These two terms are synonyms but quanteda refers to a DFM whereas others will refer to DTM. It describes how frequently terms occur in the corpus by counting single terms. To generate a DFM, we first split the text into its single terms (tokens). We then count how frequently each term (token) occurs in each document.

The following graphic describes visually how we turn raw text into a vector-space representation that is easily accessible and analyzable with quantitative statistical tools. It also visualizes how we can think of a DFM. The rows represent the documents that are part of the corpus and the columns show the different terms (tokens). The values in the cells indicate how frequently these terms (tokens) are used across the documents.

Figure 1: Model of a DFM

Important things to remember about DFMs:

• A corpus is positional (string of words) and a DFM is non-positional (bag of words). Put differently, the order of the words matters in a corpus whereas a DFM does not have information on the position of words.
• A token is each individual word in a text (but it could also be a sentence, paragraph, or character). This is why we call creating a “bag of words” also tokenizing text. In a nutshell, a DFM is a very efficient way of organizing the frequency of features/tokens but does not contain any information on their position. In our example, the features of a text are represented by the columns of a DFM and aggregate the frequency of each token.
• In most projects you want one corpus to contain all your data and generate many DFMs from that.
• The rows of a DFM can contain any unit on which you can aggregate documents. In the example above, we used the single documents as the unit. It may also well be more fine-grained with sub-documents or more aggregated with a larger collection of documents.
• The columns of a DFM are any unit on which you can aggregate features. Features are extracted from the texts and quantitatively measurable. Features can be words, parts of the text, content categories, word counts, etc. In the example above, we used single words such as “united”, “nations”, and “peace”.

To showcase the three steps introduced above, we are using the UN General Debate data by Mikhaylov, Baturo, and Dasandi dataset. There is also a pre-processed version of the dataset accessible with quanteda.corpora.

# Install package quanteda.corpora
devtools::install_github("quanteda/quanteda.corpora")

# Load the dataset
quanteda.corpora::data_corpus_ungd2017

We will, however, mainly rely on the original dataset throughout the following explanations to match closely the regular workflow of textual data in R. If you want to replicate the steps, please download the data here and unzip the zip file. Your global_path should direct you to the text file folders.

In a first step, we need to load the necessary packages and read in the data.

# Load all required packages
library(tidyverse)        # Also loads dplyr, ggplot2, and haven
library(quanteda)         # For NLP
library(readtext)         # To read .txt files
library(stm)              # For structural topic models
library(stminsights)      # For visual exploration of STM
library(wordcloud)        # To generate wordclouds
library(gsl)              # Required for the topicmodels package
library(topicmodels)      # For topicmodels
library(caret)            # For machine learning

# Download data here: 
# https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/0TJX8Y 
# and unzip the zip file

# Read in data (.txt files)
global_path <- "path/to/folder/UN-data/"

# We load the data (.txt files) from all subfolders (readtext can handle 
# this without specification)  and store them in the main UNGDspeeches 
# dataframe. Beyond the speech text, this data also includes the 
# meta-data from the text filenames and add variables for the country, 
# UN session, and year.
# The code is based on https://github.com/quanteda/quanteda.corpora/issues/6
# and https://github.com/sjankin/UnitedNations/blob/master/files/UNGD_analysis_example.Rmd 

# For the purpose of this blog post, we use the data from all sessions.
UNGDspeeches <- readtext(
  paste0(global_path, "*/*.txt"),
  docvarsfrom = "filenames",
  docvarnames = c("country", "session", "year"),
  dvsep = "_",
  encoding = "UTF-8"
)

We can then proceed and generate a corpus.

mycorpus <- corpus(UNGDspeeches)

# Assigns a unique identifier to each text
docvars(mycorpus, "Textno") <-
  sprintf("%02d", 1:ndoc(mycorpus)) 

As we can see (by calling the object mycorpus), the corpus consists of 8,093 documents.

mycorpus

Corpus consisting of 8,093 documents and 4 docvars.

With this data, we can already generate first descriptive statistics.

# Save statistics in "mycorpus.stats"
mycorpus.stats <- summary(mycorpus)

# And print the statistics of the first 10 observations
head(mycorpus.stats, n = 10)

#               Text Types Tokens Sentences country session year Textno
# 1  ALB_25_1970.txt  1727   9077       256     ALB      25 1970     01
# 2  ARG_25_1970.txt  1425   5192       218     ARG      25 1970     02
# 3  AUS_25_1970.txt  1611   5688       270     AUS      25 1970     03
# 4  AUT_25_1970.txt  1340   4717       164     AUT      25 1970     04
# 5  BEL_25_1970.txt  1289   4783       207     BEL      25 1970     05
# 6  BLR_25_1970.txt  1427   6138       204     BLR      25 1970     06
# 7  BOL_25_1970.txt  1559   5612       225     BOL      25 1970     07
# 8  BRA_25_1970.txt  1333   4422       154     BRA      25 1970     08
# 9  CAN_25_1970.txt   728   1887        97     CAN      25 1970     09
# 10 CMR_25_1970.txt   928   3144       106     CMR      25 1970     10

In a next step, we can also calculate the document-feature matrix. To do so, first we need to generate tokens (tokens()) and can also already pre-process the data. This includes removing the numbers (remove_numbers), punctuations (remove_punct), symbols (remove_symbols), and urls beginning with http(s) (remove_url).

An earlier version of this blog post used remove_hyphens to remove hyphens as well as remove_twitterto remove symbols such as @ and #. Both commands are either deprecated or defunctional. To remove hyphens, it is recommended to use split_hyphens instead. For twitter symbols there is no new command. Quanteda’s manual recommends to use ``an alternative tokenizer, including non-quanteda options’’.

We further include the docvars from our corpus (include_docvars).

# Preprocess the text

# Create tokens
token <-
  tokens(
    mycorpus,
    split_hyphens = TRUE,
    remove_numbers = TRUE,
    remove_punct = TRUE,
    remove_symbols = TRUE,
    remove_url = TRUE,
    include_docvars = TRUE
  )

Since the pre-1994 documents were scanned with OCR scanners, several tokens with combinations of digits and characters were introduced. We clean them manually following this guideline.

# Clean tokens created by OCR
token_ungd <- tokens_select(
  token,
  c("[\\d-]", "[[:punct:]]", "^.{1,2}$"),
  selection = "remove",
  valuetype = "regex",
  verbose = TRUE
)

In the next step, we then create the document-feature matrix. We lower and stem the words (tolower and stem) and remove common stop words (remove=stopwords()). Stopwords are words that appear in texts but do not give the text a substantial meaning (e.g., “the”, “a”, or “for”). Since the language of all documents is English, we only remove English stopwords here. quanteda can also deal with stopwords from other languages (for more information see here).

mydfm <- dfm(token_ungd,
             tolower = TRUE,
             stem = TRUE,
             remove = stopwords("english")
             )

We can also trim the text with dfm_trim. Using the command and its respective specifications, we filter words that appear less than 7.5% and more than 90%. This rather conservative approach is possible because we have a sufficiently large corpus.

mydfm.trim <-
  dfm_trim(
    mydfm,
    min_docfreq = 0.075,
    # min 7.5%
    max_docfreq = 0.90,
    #  max 90%
    docfreq_type = "prop"
  ) 

To get a look at the DFM, we now print their first 5 observations and first 10 features:

# And print the results of the first 10 observations and first 10 features in a DFM
head(dfm_sort(mydfm.trim, decreasing = TRUE, margin = "both"),
     n = 10,
     nf = 10) 

Document-feature matrix of: 5 documents, 10 features (4.0% sparse) and 4 docvars.
                 features
docs              problem session conflict council africa global resolut hope south situat
  CUB_34_1979.txt      36       8        1       0     13      3      10    8    10     23
  IRL_39_1984.txt      41       9       18      11     14      5      16   21    19      9
  PAN_37_1982.txt      14      12       12       8     11      2       6   11    20     10
  BFA_29_1974.txt      25      17        1       4     15      0      10   20     6      9
  GRC_43_1988.txt      27       9       13      14     10      2      10   10    11      5

The sparsity gives us information about the proportion of cells that have zero counts.

Classification

A next step can involve the classification of the text. The article by Grimmer and Stewart (2013) provides a good overview for this step. The upcoming section follows their structure. Classification sorts texts into categories. The following picture is leaned on the figure by Grimmer and Stewart (2013, 268) and illustrates a possible structure of classification.

Figure 2: Overview of classification (own illustration, based on Grimmer and Stewart (2013, 268))

A researcher usually faces one of the following situations: The categories are known beforehand or the categories are unknown. If the researcher knows the categories, s/he can use automated methods to minimize the workload that is associated with the categorization of the texts. Throughout the workshop, two methods were presented: a dictionary method and a supervised method. If the researcher does not know the categories, s/he is likely to resort to unsupervised machine learning. The following section provides illustrative examples for both methods.

Known categories

Known categories: Dictionaries

Dictionaries contain lists of words that correspond to different categories. If we apply a dictionary approach, we count how often words that are associated with different categories are represented in each document. These dictionaries help us to classify (or categorize) the speeches based on the frequency of the words that they contain. Popular dictionaries are sentiment dictionaries (such as Bing, Afinn or LIWC) or LexiCoder.

We use the “LexiCoder Policy Agenda” dictionary that can be accessed here in a .lcd format. The “LexiCoder Policy Agenda” dictionary captures major topics from the comparative Policy Agenda project and is currently available in Dutch and English.

To read in the dictionary, we use quanteda’s built-in function dictionary().

# load the dictionary with quanteda's built-in function
dict <- dictionary(file = "policy_agendas_english.lcd")

We apply this dictionary to filter the share of each country’s speeches on immigration, international affair and defence.

mydfm.un <- dfm(mydfm.trim, groups = "country", dictionary = dict)

un.topics.pa <- convert(mydfm.un, "data.frame") %>%
  dplyr::rename(country = doc_id) %>%
  select(country, immigration, intl_affairs, defence) %>%
  tidyr::gather(immigration:defence, key = "Topic", value = "Share") %>%
  group_by(country) %>%
  mutate(Share = Share / sum(Share)) %>%
  mutate(Topic = haven::as_factor(Topic))

In a next step, we can visualize the results with ggplot. This gives us a first impression of the distribution of the topics in the 2018 UN General Debate across countries.

un.topics.pa %>%
  ggplot(aes(country, Share, colour = Topic, fill = Topic)) +
  geom_bar(stat = "identity") +
  scale_colour_brewer(palette = "Set1") +
  scale_fill_brewer(palette = "Pastel1") +
  ggtitle("Distribution of PA topics in the UN General Debate corpus") +
  xlab("") +
  ylab("Topic share (%)") +
  theme(axis.text.x = element_blank(),
        axis.ticks.x = element_blank())

Figure 3: Distribution of PA topics in the UN General Debate corpus

We observe a relatively high share for both defence and international affairs whereas immigration receives fewer attention in the speeches.

Known categories: Supervised machine learning - Naive Bayes (NB)

We now turn to supervised machine learning. Similar to the dictionary approach explained above, this method also requires some pre-existing classifications. But in contrast to a dictionary, we now divide the data into a training and a test dataset. This follows the general logic of machine learning algorithms. The training data already contains the classifications and trains the algorithm (e.g., our Naive Bayes classifier) to predict the class of our speech based on the features that are given. A Naive Bayes classifier now calculates the probability for each class based on the features. It eventually goes for the class with the highest probability and selects this class as the corresponding category. It is based on the Bayes theorem for conditional probability. It can be formally written as: \[ P(A | B) = \frac{P(A) * P(B | A)}{P(B)}\] In plain words, the probability of A is conditional on B.

Bayes’ theorem

• \(A\) and \(B\) are events
• \(P(A)\) and \(P(B)\) is the probability of observing \(A\) and \(B\) (respectively) independent from each other
• \(P(A) \neq 0\) and \(P(B) \neq 0\)
• \(P(A|B)\) is the conditional probability that \(A\) occurs when \(B\) is true \[P(A|B) = \frac{P(A \cap B)}{P(B)}, if P(B) \neq 0\]
• \(P(B|A)\) is the conditional probability that \(B\) occurs when \(A\) is true \[P(B|A) = \frac{P(B \cap A)}{P(A)}, if P(A) \neq 0\]
• And we also have the joint probability of $ P(A B) = P(B A)$ because

\[ \Longrightarrow P(A \cap B) = P(A|B)P(B) = P(B|A)P(A)\] \[ \Longrightarrow P(A | B) = \frac{P(A \cap B)}{P(B)}\] \[ \Longrightarrow P(A | B) = \frac{\frac{P(A \cap B)}{P(B)}*P(A)}{P(B)}\] \[ \Longrightarrow P(A | B) = \frac{P(B|A)*P(A)}{P(B)}\]

Why is Naive Bayes “naive”? Naive Bayes is “naive” because of its strong independence assumptions. It assumes that all features are equally important and that all features are independent. If you think of n-grams and compare unigrams and bigrams, you can intuitively understand why the last assumption is a strong assumption. A unigram counts each word as a gram (“I” “like” “walking” “in” “the” “sun”) whereas a bigram counts two words as a gram (“I like” “like walking” “walking in” “in the” “the sun”).

However, even when the assumptions are not fully met, Naive Bayes still performs well.

A Naive Bayes is a relatively simple classification algorithm because it does not require much time and working capacity of your machine. To use a Naive Bayes classifier, we rely on quanteda’s built-in function textmodel_nb.

To perform the Naive Bayes estimation, we proceed with the following steps:

<sub>1. We set up training and test data based on the corpus.
2. Based on these two datasets, we generate a DFM.
3. We train the algorithm by feeding in the training data and eventually use the test data for performance.
4. We then check the performance (accuracy) of our results.
5. And compare it with a random prediction.</sub>

For this example, we use the pre-labeled dataset that is used for the algorithm newsmap by Kohei Watanabe. The dataset contains information on the geographical location of newspaper articles. We introduce this new dataset as Naive Bayes – a supervised machine learning algorithm – requires pre-labeled data.

We first load the dataset. To do so, we follow Kohei Watanabe’s description here, download the corpus of Yahoo News from 2014, and follow the subsequent processing steps he describes.

# load data
load("../newspaper.RData")

# transform variables
pred_data$text <- as.character(pred_data$text)
pred_data$country <- as.character(pred_data$country)

For simplicity, we keep only the USA, Great Britain, France, Brazil, and Japan.

pred_data <- pred_data %>%
  dplyr::filter(country %in% c("us", "gb", "fr", "br", "jp")) %>%
                  dplyr::select(text, country)

head(pred_data)

row	text	country
1	’08 French champ Ivanovic loses to Safarova in 3rd. PARIS (AP) - Former French Open champion Ana Ivanovic lost in the third round Saturday, beaten 6-3, 6-3 by 23rd-seeded Lucie Safarova of the Czech Republic.	fr
2	Up to USD1,000 a day to care for child migrants. More than 57,000 unaccompanied children, mostly from Central America, have been caught entering the country illegally since last October, and President Barack Obama has asked for USD3.7 billion in emergency funding to address what he has called an ‘urgent humanitarian solution.’ ‘One of the figures that sticks in everybody’s mind is we’re paying about USD250 to USD1,000 per child,’ Senator Jeff Flake told reporters, citing figures presented at a closed-door briefing by Homeland Security Secretary Jeh Johnson. Federal authorities are struggling to find more cost-effective housing, medical care, counseling and legal services for the undocumented minors. The base cost per bed was USD250 per day, including other services, Senator Dianne Feinstein said, without providing details.	us
3	1,400 gay weddings in England, Wales in first three months. Just over 1,400 gay couples tied the knot in the three months after same-sex marriage was allowed in England and Wales, figures out Thursday showed. The Office for National Statistics said 1,409 marriages took place between March 29 and June 30. ‘The novelty and significance of marriage becoming available led to an initial rush among same-sex couples wanting to be among the very first to assume the same rights and protection afforded to heterosexual couples,’ said James Brown, a partner at law firm JMW Solicitors. The figures will likely surge from December once civil partnerships can be converted into marriages.	gb
4	1 dead after fan fighting in Brazil. SAO PAULO (AP) - Police say a 21-year-old man died after a confrontation between rival football fan groups in Brazil on Sunday.	br
5	1 dead as plane with French tourists crashes in US. PAGE, Arizona (AP) - Authorities say a small plane carrying French tourists crashed while trying to land at an airport in Arizona, and one person was killed and another hospitalized.	fr
6	1 US theory is someone diverted missing plane. WASHINGTON (AP) - A U.S. official says investigators are examining the possibility that someone caused the disappearance of a Malaysia Airlines jet with 239 people on board, and that it may have been ‘an act of piracy.’	us

We pre-process the data again. Our final corpus thus includes the newspaper headlines by country.

data_corpus <- corpus(pred_data, text_field = "text")

In a first step, we need to define our training and our test dataset. Based on these two datasets, we generate a DFM. This code is based on Cornelius code and quanteda’s example. To do so, we apply similar general data pre-processing steps as discussed above.

# Set a seed for replication purposes
set.seed(68159)

# Generate random 10,000 numbers without replacement
training_id <- sample(1:29542, 10000, replace = FALSE)

# Create docvar with ID
docvars(data_corpus, "id_numeric") <- 1:ndoc(data_corpus)

# Get training set
dfmat_training <-
  corpus_subset(data_corpus, id_numeric %in% training_id) %>%
  dfm(stem = TRUE)

# Get test set (documents not in training_id)
dfmat_test <-
  corpus_subset(data_corpus,!id_numeric %in% training_id) %>%
  dfm(stem = TRUE)

We can now check the distribution of the countries across the two DFMs:

print(prop.table(table(docvars(
  dfmat_training, "country"
))) * 100)

   br    fr    gb    jp    us 
13.34 14.37 33.65 11.10 27.54 

print(prop.table(table(docvars(
  dfmat_test, "country"
))) * 100)

   br       fr       gb       jp       us 
13.70893 15.13151 33.02630 10.93542 27.19783 

As we can see, the countries are equally distributed across both DFMs.

In a next step, we train the Naive Bayes classifier. Going back to the formula stated above, we know that A is conditional on B.

\[ P(A | B) = \frac{P(A) * P(B | A)}{P(B)}\]

A is what we want to know (the country that is mainly addressed in each text) and B is what we see (the text). We can now proceed and replace A and B with the respective terms. This leads us to the next equation:

\[ P(Country | Text) = \frac{P(Country) * P(Text | Country)}{P(Text)}\]

We can then proceed and train our algorithm using quanteda’s built-in function textmodel_nb.

# Train naive Bayes
# The function takes a DFM as the first argument 
model.NB <-
  textmodel_nb(dfmat_training, docvars(dfmat_training, "country"), prior = "docfreq")

# The prior indicates an assumed distribution. 
# Here we choose how frequently the categories occur in our data.

dfmat_matched <-
  dfm_match(dfmat_test, features = featnames(dfmat_training))

The command summary(model.NB) gives us the results of our prediction. Click unfold to see the results.

Code: summary(model.NB)

summary(model.NB)

Call:
textmodel_nb.dfm(x = dfmat_training, y = docvars(dfmat_training, 
    "country"), prior = "docfreq")

Class Priors:
(showing first 5 elements)
    br     fr     gb     jp     us 
0.1334 0.1437 0.3365 0.1110 0.2754 

Estimated Feature Scores:
         '     08   french   champ  ivanov    lose      to safarova     in     3rd      .     pari
br 0.08949 0.1679 0.007791 0.07636 0.08195 0.08707 0.09989   0.1474 0.1214 0.34322 0.1176 0.008863
fr 0.14212 0.2778 0.940061 0.37912 0.20345 0.16469 0.13803   0.3659 0.1473 0.24345 0.1296 0.947706
gb 0.43332 0.1532 0.032008 0.24397 0.14963 0.49395 0.35845   0.1346 0.3540 0.17905 0.3321 0.032363
jp 0.06507 0.1184 0.004396 0.10771 0.28900 0.06580 0.10287   0.1040 0.1007 0.06916 0.1059 0.002778
us 0.27000 0.2827 0.015743 0.19285 0.27598 0.18850 0.30077   0.2482 0.2766 0.16512 0.3148 0.008290
         (      ap       )      -  former    open champion     ana    lost    the   third   round
br 0.16080 0.24091 0.16100 0.1611 0.09034 0.17116  0.12708 0.08142 0.08045 0.1101 0.08964 0.30833
fr 0.14221 0.14820 0.14219 0.1363 0.17312 0.18636  0.24110 0.13475 0.16643 0.1350 0.17178 0.19535
gb 0.34860 0.28303 0.34870 0.3403 0.45720 0.36654  0.51497 0.07433 0.38923 0.3603 0.39194 0.24189
jp 0.09123 0.08125 0.09126 0.0922 0.04583 0.07443  0.05684 0.22969 0.07943 0.1005 0.07653 0.04757
us 0.25715 0.24661 0.25685 0.2702 0.23351 0.20152  0.06002 0.47982 0.28446 0.2940 0.27010 0.20686
   saturday      ,  beaten     6-3      by 23rd-seed
br  0.14287 0.1078 0.19323 0.21226 0.09731    0.1955
fr  0.19921 0.1384 0.20558 0.40985 0.12900    0.3236
gb  0.44845 0.3437 0.45359 0.21530 0.33256    0.1785
jp  0.09248 0.1099 0.07787 0.08317 0.11636    0.1379
us  0.11699 0.3001 0.06972 0.07942 0.32477    0.1646

To better understand how well we did, we can also generate two frequency tables for right and wrong predictions.

prop.table(table(predict(model.NB) == docvars(dfmat_training, "country"))) * 100

FALSE  TRUE 
 2.71 97.29  

To check if this result indicates a good performance, we compare it with a random result. We randomize the list of countries (and keep the overall frequency distribution of our countries constant) to allow our random algorithm a legitimate chance for a correct classification.

prop.table(table(sample(predict(model.NB)) == docvars(dfmat_training, "country"))) * 100

FALSE  TRUE 
76.63 23.37  

As we can see from the the summarized table below, our Naive Bayes classifier clearly outperforms a random algorithm.

	Naive Bayes	Random
False	2.71	76.63
True	97.29	23.37

We are likely to increase our accuracy even more by pre-processing our text data.

A confusion matrix helps us assess how well our algorithm performed. It shows us the prediction for all five countries in contrast to the actual class that is given by our data. For example, we predict 2580 articles as belonging to Great Britain that actually belong to Great Britain. However, we also predict 39 articles as British articles while they are actually French. Overall, when we look at the diagonal, we see that most predictions correctly classify the articles and that our algorithm performs well.

actual_class <- docvars(dfmat_matched, "country")
predicted_class <- predict(model.NB, newdata = dfmat_matched)
tab_class <- table(actual_class, predicted_class)
tab_class

             predicted_class
actual_class   br   fr   gb   jp   us
          br 2580    6   50    6   37
          fr   39 2697  124   11   86
          gb   24   52 6072   24  282
          jp   33    9   31 1945  119
          us   56   32  194   66 4967

We store our confusion matrix in an object because we need it later to visualize the results.

confusion <- confusionMatrix(tab_class, mode = "everything")

Confusion Matrix and Statistics

            predicted_class
actual_class   br   fr   gb   jp   us
          br 2580    6   50    6   37
          fr   39 2697  124   11   86
          gb   24   52 6072   24  282
          jp   33    9   31 1945  119
          us   56   32  194   66 4967

Overall Statistics
                                          
               Accuracy : 0.9344          
                 95% CI : (0.9309, 0.9379)
    No Information Rate : 0.3311          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.914           
                                          
 Mcnemar's Test P-Value : < 2.2e-16       

Statistics by Class:

                     Class: br Class: fr Class: gb Class: jp Class: us
Sensitivity             0.9444    0.9646    0.9383   0.94786    0.9046
Specificity             0.9941    0.9845    0.9708   0.98902    0.9752
Pos Pred Value          0.9630    0.9121    0.9408   0.91015    0.9345
Neg Pred Value          0.9910    0.9940    0.9695   0.99385    0.9632
Precision               0.9630    0.9121    0.9408   0.91015    0.9345
Recall                  0.9444    0.9646    0.9383   0.94786    0.9046
F1                      0.9536    0.9376    0.9396   0.92862    0.9193
Prevalence              0.1398    0.1431    0.3311   0.10500    0.2810
Detection Rate          0.1320    0.1380    0.3107   0.09953    0.2542
Detection Prevalence    0.1371    0.1513    0.3303   0.10935    0.2720
Balanced Accuracy       0.9692    0.9745    0.9546   0.96844    0.9399

To display our confusion matrix visually, we could either produce a heatmap or a confusion matrix.

We first plot a heatmap using the code below.

Code: Heatmap using ggplot2

# Save confusion matrix as data frame
confusion.data <- as.data.frame(confusion[["table"]])

# Reverse the order
level_order_y <-
  factor(confusion.data$actual_class,
         level = c('us', 'jp', 'gb', 'fr', 'br'))

ggplot(confusion.data,
       aes(x = predicted_class, y = level_order_y, fill = Freq)) +
  xlab("Predicted class") +
  ylab("Actual class") +
  geom_tile() + theme_bw() + coord_equal() +
  scale_fill_distiller(palette = "Blues", direction = 1) +
  scale_x_discrete(labels = c("Brazil", "France", "Great \n Britain", "Japan", "USA")) +
  scale_y_discrete(labels = c("USA", "Japan", "Great \n Britain", "France", "Brazil"))

Figure 4: Contingency table

To plot the following confusion matrix, we need to slightly adjust the code from this post on data visualization by Richard Traunmüller.

Code: Confusion matrix

# Generate a data matrix from the confusion table
dat <- data.matrix(confusion$table)

# Change order of column names
order.columns <- c(5, 4, 3, 2, 1) 
dat <- dat[order.columns,]

par(mgp = c(1.5, .3, 0))

# Plot
plot(
  0,
  0,
  # Type of plotting symbol
  pch = "",
  # Range of x-axis
  xlim = c(0.5, 5.5),
  # Range of y-axis
  ylim = c(0.5, 6.5),
  # Suppresses both x and y axes
  axes = FALSE,
  # Label of x-axis
  xlab = "Predicted class",
  # Label of y-axis
  ylab = "Actual class",
)

# Write a for-loop that adds the bubbles to the plot
for (i in 1:dim(dat)[1]) {
  symbols(
    c(1:dim(dat)[2]),
    rep(i, dim(dat)[2]),
    circle = sqrt(dat[i,] / 9000 / pi),
    add = TRUE,
    inches = FALSE,
    fg = brewer.pal(sqrt(dat[i,] / 9000 / pi), "Blues"),
    bg = brewer.pal(sqrt(dat[i,] / 9000 / pi), "Blues")
  )
}

axis(
  1,
  col = "white",
  col.axis = "black",
  at = c(1:5),
  label = colnames(dat)
)
axis(
  2,
  at = c(1:5),
  label = rownames(dat),
  las = 1,
  col.axis = "black",
  col = "white"
)

# Add numbers to plot
for (i in 1:5) {
  text(c(1:5), rep(i, 5), dat[i,], cex = 0.8)
}

Figure 5: Contingency table

Both figures show us that our prediction performs well for all countries but it performs particularly well for the USA and Great Britain. Darker colors show a higher frequency in both plots, the contingency table also indicates a greater frequency with the size of the bubbles.

Unknown categories

Unknown categories: Unsupervised machine learning - Latent semantic analysis (LSA)

The next section addresses how to analyze texts with unknown categories. Latent Semantic Analysis (LSA) evaluates documents and seeks to find the underlying meaning or concept of these documents. If each word only had one meaning, LSA would have an easy job. However, oftentimes, words are ambiguous, have multiple meanings or are synonyms. One example from our corpus is “may” - it could be a verb, a noun for a month, or a name. To overcome this problem, LSA essentially compares how often words appear together in one document and then compares this across all other documents. By grouping words with other words, we try to identify those words that are semantically related and eventually also get the true meaning of ambiguous words. More technically, LSA is a useful technique for aligning feature distributions to an n-dimensional space. This is achieved via singular value decomposition (SVD). This decomposition allows us to decompose both a quadratic and a rectangular matrix. LSA can (among other things) be used to compare similarity of documents/documents grouped by some variable.

What are the major assumptions and simplifications that LSA has?

1. Documents are non-positional (“bag of words”). The “bag of words” approach assumes that the order of the words does not matter. What matters is only the frequency of the single words.
2. Concepts are understood as patterns of words where certain words often go together in similar documents.
3. Words only have one meaning given the contexts surrounding the patterns of words.

For the next example, we go back to the UN General Assembly speech data set.

corpus.un.sample <- corpus_sample(mycorpus, size = 500)

We again follow the cleaning steps described above.

Code: Data pre-processing steps

# Create tokens
token_sample <-
  tokens(
    split_hyphens = TRUE,
    corpus.un.sample,
    remove_numbers = TRUE,
    remove_punct = TRUE,
    remove_symbols = TRUE,
    remove_url = TRUE,
    include_docvars = TRUE
  )

# Clean tokens created by OCR
token_ungd_sample <- tokens_select(
  token_sample,
  c("[\\d-]", "[[:punct:]]", "^.{1,2}$"),
  selection = "remove",
  valuetype = "regex",
  verbose = TRUE
)

dfmat <- dfm(token_ungd_sample,
             tolower = TRUE,
             stem = TRUE,
             remove = stopwords("english")
             )

In an earlier version of this post, the function textmodel_lsa was already implemented in quanteda. It is now part of the package quanteda.textmodels. After loading the package, we can run the textmodel_lsa command.

# Load the package
library(quanteda.textmodels)

# Run the textmodel_lsa command
mylsa <- quanteda.textmodels::textmodel_lsa(dfmat, nd = 10)

One interesting question would be: How similar are the USA and Russia? Each dot represents a country-year observation. The USA are colored blue, Russia is colored red, and all other countries are grey.

# We need the "stringr" package for the following command
sources <-
  str_remove_all(rownames(mylsa$docs), "[0-9///'._txt]") 

sources.color <- rep("gray", times = length(sources))
sources.color[sources %in% "USA"] <- "blue"
sources.color[sources %in% "RUS"] <- "red"

plot(
  mylsa$docs[, 4:5],
  col = alpha(sources.color, 0.3),
  pch = 19,
  xlab = "Dimension 4",
  ylab = "Dimension 5",
  main = "LSA dimensions by subcorpus"
)

Figure 6: Distribution of PA topics in the UN General Debate corpus

On Dimension 5 we do not really observe a difference between documents from the US and Russia while we do see a topical divide on Dimension 4.

# create an LSA space; return its truncated representation in the low-rank space
tmod <- quanteda.textmodels::textmodel_lsa(dfmat[1:10, ])

# matrix in low_rank LSA space
tmod$matrix_low_rank[, 1:5]

                        forti         third session general assembl
GNQ_43_1988.txt  4.000000e+00  6.000000e+00      10      12       9
UZB_64_2009.txt -6.306067e-14 -1.748601e-15       2       1       1
PRT_50_1995.txt  2.000000e+00 -2.962543e-13       5       8       4
VCT_70_2015.txt -4.979003e-14 -1.879096e-13       3       1       6
GUY_67_2012.txt -7.959952e-14 -1.302430e-14       3       6       2
MDG_34_1979.txt -1.627830e-13  7.000000e+00      14      13       6
CHL_60_2005.txt -1.064461e-13  2.000000e+00       5       3       4
GTM_68_2013.txt -1.319652e-13  5.002986e-14       6       7      11
GIN_64_2009.txt -1.256287e-13  1.000000e+00       5       5       2
LBN_59_2004.txt -9.672818e-14  1.000000e+00       2       4       5

We now fold the queries into the space generated by dfmat[1:10,] and return its truncated versions of its representation in the new low-rank space. For more information on this, see Deerwester et al. (1990) and Rosario (2000).

pred <- predict(tmod, newdata = dfmat[1:10, ])
pred$docs_newspace

10 x 10 Matrix of class "dgeMatrix"
                      [,1]        [,2]        [,3]        [,4]         [,5]        [,6]
GNQ_43_1988.txt -0.3587033  0.04626331 -0.40268736  0.75368464 -0.198343099  0.15890426
UZB_64_2009.txt -0.1304653  0.04983523 -0.06010142 -0.07259054  0.057245087 -0.03112114
PRT_50_1995.txt -0.5120486  0.45389699  0.69697128  0.03247785 -0.071841805  0.18464906
VCT_70_2015.txt -0.2241820  0.09965473 -0.36666157 -0.45680821 -0.462552853  0.28818827
GUY_67_2012.txt -0.2905706  0.05927260 -0.29307648 -0.35382563  0.366086659  0.47269539
MDG_34_1979.txt -0.4919563 -0.83849245  0.20130813 -0.06146108 -0.038858129 -0.05927530
CHL_60_2005.txt -0.2663441  0.20190586 -0.13824724 -0.23099427 -0.006964881 -0.67210030
GTM_68_2013.txt -0.2074862  0.05127196 -0.13858784 -0.11930716  0.021618854 -0.33022555
GIN_64_2009.txt -0.2738299  0.12184387 -0.19025526  0.13644592  0.659835350 -0.13044787
LBN_59_2004.txt -0.1625876  0.12082658 -0.11439537  0.04220718 -0.408458883 -0.22780319
                       [,7]        [,8]         [,9]        [,10]
GNQ_43_1988.txt  0.13239767  0.10535490  0.203828010 -0.065029344
UZB_64_2009.txt -0.21573385 -0.08282906 -0.130474714 -0.947070625
PRT_50_1995.txt  0.07183401  0.00858728 -0.005558773  0.020944555
VCT_70_2015.txt  0.35837271 -0.38370263 -0.152115117  0.029859727
GUY_67_2012.txt -0.34813601  0.38234056  0.254633069  0.106243837
MDG_34_1979.txt -0.01837620 -0.06738449  0.012211891  0.023580112
CHL_60_2005.txt  0.04450401 -0.07554420  0.596888043  0.009695998
GTM_68_2013.txt  0.35028911  0.66063039 -0.502174774 -0.007009133
GIN_64_2009.txt  0.03757466 -0.49043679 -0.361551806  0.174062011
LBN_59_2004.txt -0.74478679 -0.03668237 -0.337785293  0.234975826

Unknown categories: Unsupervised machine learning - Latent Dirichlet Allocation (LDA)

Both Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) belong to topic modelling. Topic models find patterns of words that appear together and group them into topics. The researcher decides on the number of topics and the algorithms then discover the main topics of the texts without prior information, training sets or human annotations.

LDA is a Bayesian mixture model for discrete data where topics are assumed to be uncorrelated. It is a model that describes how the documents in a dataset were created. We assign an arbitrary number of topics (K) where each topic is a distribution over a fixed vocabulary. Each document is considered as a collection of words, one for each of K topics. It also follows the “bag of words” approach that considers each word in a document separately.

To calculate the LDA models, we need to load the package topicmodels. If you use this package for the first time on your machine, you need to execute a specific sequence of commands, detailed in the code below.

Code: Installing the package topicmodels

# 1) Install GSL
# We first need to make sure that GSL (e.g. 'brew install gsl' in the terminal) is installed on our machine

# 2) Install gsl
# Then we proceed and choose either of the following commands:
# A)
install.packages("gsl")
# or B)
install.packages(
  "https://cran.rstudio.com/src/contrib/gsl_2.1-6.tar.gz",
  repos = NULL,
  method = "libcurl"
)

# 3) Load the gsl package
library(gsl)

# 4) Install topicmodels
# Here we choose again either of the following commands:
# A)
install.packages("topicmodels")
# or B)
install.packages(
  "https://cran.r-project.org/src/contrib/topicmodels_0.2-8.tar.gz",
  repos = NULL,
  method = "libcurl"
)

# 5) Load the topicmodels package
library(topicmodels)

# Now you are all set for the following models.

For the LDA, we again first trim our DFM. As above, the command dfm_trim trimms the text. This allows us to filter words that appear less than 7.5% and more than 90%.

mydfm.un.trim <-
  dfm_trim(
    mydfm,
    min_docfreq = 0.075,
    # min 7.5%
    max_docfreq = 0.90,
    # max 90%
    docfreq_type = "prop"
  ) 

We then assign an arbitrary topic number and convert the trimmed DFM to a topicmodels object.

# Assign an arbitrary number of topics
topic.count <- 15

# Convert the trimmed DFM to a topicmodels object
dfm2topicmodels <- convert(mydfm.un.trim, to = "topicmodels")

Eventually, we can calculate the LDA model with quanteda’s LDA() command.

lda.model <- LDA(dfm2topicmodels, topic.count)

Output for lda.model

lda.model

A LDA_VEM topic model with 15 topics.

as.data.frame(terms(lda.model, 6))

topic1	topic2	topic3	topic4	topic5	topic6	topic7	topic8	topic9	topic10
nuclear	cooper	problem	trade	america	arab	arab	global	oper	war
weapon	council	interest	problem	american	small	palestinian	cooper	negoti	today
republ	reform	hope	product	latin	nuclear	israel	social	conflict	now
soviet	global	possibl	per	respect	issu	territori	order	problem	power
relat	effect	now	economi	law	pacif	isra	futur	confer	live
disarma	process	even	increas	principl	weapon	resolut	common	south	mani

topic11	topic12	topic13	topic14	topic15
global	terror	african	deleg	africa
sustain	council	africa	session	south
climat	terrorist	republ	problem	independ
chang	iraq	conflict	concern	african
challeng	resolut	democrat	great	namibia
goal	law	elect	republ	struggl

How similar are the fifteen topics? This question is particularly interesting because it allows us to (possibly) cluster homogeneous topics. To get a better idea of our LDA model and about the similarity among the different topics, we can plot our results using the following chunck of code. dist and hclust are standard R commands that allow us to calculate the similarity.

lda.similarity <- as.data.frame(lda.model@beta) %>%
  scale() %>%
  dist(method = "euclidean") %>%
  hclust(method = "ward.D2")

par(mar = c(0, 4, 4, 2))
plot(lda.similarity,
     main = "LDA topic similarity by features",
     xlab = "",
     sub = "")

Figure 7: LDA topic similarity by features

The plot is called dendogram and visualizes a hierarchial clustering. The x-axis gives you the topics and the clusters of these topics. Put differently, it gives you information on the smilarity of the topics. On the y-axis, we see the dissmilarity (or distance) between our fifteen topics.

Unknown categories: Unsupervised machine learning - Structural topic models (STM)

The structural topic models (STM) are a popular extension of the standard LDA models. The STM allows to include metadata (the information about each document) into the topicmodel and it offers an alternative initialization mechanism (“Spectral”). For STMs, the covariates can be used in priors. The stm vignette provides a good overview how to use a STM. The package includes estimation algorithms and tools for every stage of the workflow. A particularly large emphasis is on a number of diagnostic functions that are integrated into the R package.

The package stiminsights is very useful for visual exploration. It allows the user to process interactive validation, interpretation and visualization of one or several Structural Topic Models (stm).

We again trim our dfm with the command dfm_trim.

mydfm.un.trim <-
  dfm_trim(
    mydfm,
    min_docfreq = 0.075,
    # min 7.5%
    max_docfreq = 0.90,
    # max 90%
    docfreq_type = "prop"
  ) 

We then assign the number of topics arbitrarily.

topic.count <- 25 # Assigns the number of topics

And eventually convert the DFM (with convert()) and calculate the STM (with stm()).

# Calculate the STM 
dfm2stm <- convert(mydfm.un.trim, to = "stm")

model.stm <- stm(
  dfm2stm$documents,
  dfm2stm$vocab,
  K = topic.count,
  data = dfm2stm$meta,
  init.type = "Spectral"
)

To get a first insight, we print the terms that appear in each topic.

as.data.frame(t(labelTopics(model.stm, n = 10)$prob))

V1	V2	V3	V4	V5	V6	V7	V8	V9
council	lebanon	soviet	global	african	nuclear	european	trade	african
reform	arab	union	sustain	africa	weapon	europ	economi	situat
effect	resolut	relat	challeng	republ	treati	cooper	product	guinea
activ	problem	militari	chang	democrat	disarma	union	industri	africa
cooper	territori	forc	climat	millennium	arm	conflict	resourc	particular
strengthen	palestinian	republ	goal	commit	test	process	increas	hope
convent	withdraw	arm	commit	poverti	non	stabil	market	concern
role	principl	war	agenda	particular	prolifer	contribut	price	solut
confer	solut	socialist	respons	like	pakistan	bosnia	global	session
contribut	war	polici	address	summit	india	respect	financi	problem

V10	V11	V12	V13	V14	V15	V16	V17	V18
council	problem	cooper	republ	power	africa	order	problem	israel
iraq	now	stabil	korea	great	south	principl	session	arab
resolut	oper	dialogu	democrat	viet	debt	social	oper	palestinian
law	negoti	terror	china	deleg	deleg	univers	confer	isra
aggress	hope	sudan	south	coloni	hope	cultur	solut	palestin
charter	agreement	call	korean	territori	process	today	resolut	east
violat	mani	process	asia	nam	environ	life	cyprus	middl
iraqi	last	promot	north	republ	confer	respect	concern	resolut
islam	way	commit	east	sea	welcom	societi	negoti	occupi
iran	possibl	issu	cooper	charter	conflict	becom	disarma	territori

V19	V20	V21	V22	V23	V24	V25
island	today	per	terror	america	conflict	africa
small	war	cent	afghanistan	american	refuge	south
pacif	live	social	afghan	latin	war	independ
caribbean	want	educ	cooper	central	somalia	african
chang	terror	poverti	terrorist	respect	assist	struggl
ocean	mani	health	problem	social	humanitarian	regim
issu	let	drug	global	process	mani	namibia
sea	now	democraci	asia	solut	elect	apartheid
climat	everi	programm	central	possibl	situat	coloni
call	know	million	issu	express	forc	racist

The following plot allows us to intuitively get information on the share of the different topics at the overall corpus.

plot(
  model.stm,
  type = "summary",
  text.cex = 0.5,
  main = "STM topic shares",
  xlab = "Share estimation"
)

Figure 8: STM topic shares

Using the package stm, we can now visualize the different words of a topic with a wordcloud. Since topic 4 has the highest share, we use it for the next visualization. The location of the words is randomized and changes each time we plot the wordcloud while the size of the words is relative to their frequency and remains the same.

stm::cloud(model.stm,
           topic = 4,
           scale = c(2.25, .5))

Figure 9: Wordcloud with stm

If we want, we can also put several different topics in visually perspective using the following lines of code:

plot(model.stm,
     type = "perspectives",
     topics = c(4, 5),
     main = "Putting two different topics in perspective")

Figure 10: Wordcloud using stm – Perspective plots

The perspective plot visualizes the combination of two topics (here topic 4 and topic 5). The size of the words is again relative to their frequency (within the combination of the two topics). The x-axis shows the dregree that specific words align with Topic 4 or Topic 5. Global is closely aligned with Topic 4 whereas commit is more central in both topics.        

Further readings

• quanteda: Quantitative Analysis of Textual Data
• Benoit, K., & Nulty, P.. 2016. Quanteda: Quantitative Analysis of Textual Data.
• Benoit, K., Watanabe, K., Wang, H., Nulty, P., Obeng, A., Müller, S., & Matsuo, A. (2018). quanteda: An R package for the quantitative analysis of textual data. Journal of Open Source Software, 3(30), 774.
• Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6), 391-407. Chicago
• Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267-297.
• Mueller, S. Quanteda Cheat Sheet.
• Puschmann, C. Inhaltsanalyse mit R.
• Puschmann, C. Automatisierte Inhaltsanalyse mit R.
• Puschmann, C. Automatisierte Inhaltsanalyse mit R. Überwachtes maschinelles Lernen.
• Roberts, M. E., Stewart, B. M., & Tingley, D. (2014). stm: R package for structural topic models. R package, 1, 12.
• Rosario, B. (2000). Latent semantic indexing: An overview. Technical Report. INFOSYS, 240, 1-16.
• Welbers, K., Van Atteveldt, W., & Benoit, K. (2017). Text analysis in R. Communication Methods and Measures, 11(4), 245-265.

About the presenter

Cornelius Puschmann is professor of media and communication at ZeMKI, University of Bremen and an affiliate researcher at the Leibniz Institute for Media Research in Hamburg. His research interests center on online hate speech, the role of algorithms for the selection of media content, and methodological aspects of computational social science.

Contents

1. Wikipedia for Political Research
2. Collecting and Analyzing Wikipedia Data
   1. Application 1: Using Pageviews to Measure Public Attention
   2. Application 2: Using Article Links to Create a Network Graph of German MPs
   3. Application 3: Using Clickstream Data to Analyze Referral Patterns
3. Collecting Data via Wikidata Queries
4. legislatoR
   1. Application 1: Social Media Adoption Rates
   2. Application 2: Public Attention to Members of the German Bundestag
5. Conclusion
6. About the Presenter
7. References

Wikipedia for Political Research

According to its website, “Wikipedia […] is a multilingual, web-based, free-content encyclopedia project supported by the Wikimedia Foundation and based on a model of openly editable content”. As of July 2019, it comprises more than 48 million articles and is ranked sixth in the list of the most frequently visited websites.

Wikipedia harbors numerous types of data. These include both article contents as well as meta information such as pageviews, clickstreams, links and backlinks, or edits and revision histories. Additionally, Wikipedia’s sibling, the collaboratively edited document-oriented data base Wikidata, provides access to over 58 million data items (as of July 2019). Given the broad collection of articles on politicians and institutions from all over the world, Wikipedia offers tremendous potential for (comparative) political research.

In what follows, we will introduce the functionalities of various R packages, including WikipediR, WikidataR, and pageviews. In doing so, we will showcase how to connect to Wikipedia and Wikidata APIs, how to efficiently access and parse content, and how to process the retrieved data in order to address various questions of substantive interest. We will also provide an overview of the legislatoR package, a fully relational individual-level data package that comprises political, sociodemographic, and Wikipedia-related data on elected politicians from various consolidated democracies.

## Packages
pkgs <- c(
  "devtools",
  "ggnetwork",
  "igraph",
  "intergraph",
  "tidyverse",
  "rvest",
  "devtools",
  "magrittr",
  "plotly",
  "RColorBrewer",
  "colorspace",
  "lubridate",
  "networkD3",
  "pageviews",
  "readr",
  "wikipediatrend",
  "WikipediR",
  "WikidataR"
)

## Install uninstalled packages
lapply(pkgs[!(pkgs %in% installed.packages())], install.packages)

## Load all packages to library
lapply(pkgs, library, character.only = TRUE)

## legislatoR
devtools::install_github("saschagobel/legislatoR")
library(legislatoR)

Collecting and Analyzing Wikipedia Data

# get pageviews
trump_views <-
  article_pageviews(
    project = "en.wikipedia",
    article = "Donald Trump",
    user_type = "user",
    start = "2015070100",
    end = "2017050100"
  )
head(trump_views)

clinton_views <-
  article_pageviews(
    project = "en.wikipedia",
    article = "Hillary Clinton",
    user_type = "user",
    start = "2015070100",
    end = "2017050100"
  )

# Plot pageviews
plot(ymd(trump_views$date), trump_views$views, col = "red", type = "l", xlab="Time", ylab="Pageviews")
lines(ymd(clinton_views$date), clinton_views$views, col = "blue")
legend("topleft", legend=c("Trump","Clinton"), cex=.8,col=c("red","blue"), lty=1) 

## step 1: get info about legislators
dat <- semi_join(
  x = get_core(legislature = "deu"),
  y = filter(get_political(legislature = "deu"), session == 19),
  by = "pageid"
)

## step 2: get page links (max 500 links)
if (!file.exists("studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")) {
  links_list <- list()
  for (i in 1:nrow(dat)) {
    links <-
      page_links(
        "de",
        "wikipedia",
        page = dat$wikititle[i],
        clean_response = TRUE,
        limit = 500,
        namespaces = 0
      )
    links_list[[i]] <- lapply(links[[1]]$links, "[", 2) %>% unlist
  }
  save(links_list, file = "studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")
} else{
  load("studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")
}

## step 3: identify links between MPs
# loop preparation
connections <- data.frame(from = NULL, to = NULL)
# loop
for (i in seq_along(dat$wikititle)) {
  links_in_pslinks <-
    seq_along(dat$wikititle)[str_replace_all(dat$wikititle, "_", " ") %in%
                               links_list[[i]]]
  links_in_pslinks <- links_in_pslinks[links_in_pslinks != i]
  connections <-
    rbind(connections,
          data.frame(
            from = rep(i - 1, length(links_in_pslinks)), # -1 for zero-indexing
            to = links_in_pslinks - 1 # here too
            )
          )
}

# results
names(connections) <- c("from", "to")

# make symmetrical
connections <- rbind(connections,
                     data.frame(from = connections$to,
                                to = connections$from))
connections <- connections[!duplicated(connections), ]


## step 4: visualize connections
connections$value <- 1
nodesDF <- data.frame(name = dat$name, group = 1)

network_out <-
  forceNetwork(
    Links = connections,
    Nodes = nodesDF,
    Source = "from",
    Target = "to",
    Value = "value",
    NodeID = "name",
    Group = "group",
    zoom = TRUE,
    opacityNoHover = 3,
    height = 360,
    width = 636
  )

nodesDF$id <- as.numeric(rownames(nodesDF)) - 1
connections_df <-
  merge(connections,
        nodesDF,
        by.x = "to",
        by.y = "id",
        all = TRUE)
to_count_df <- count(connections_df, name)
arrange(to_count_df, desc(n))

## # A tibble: 712 x 2
##    name                     n
##    <fct>                <int>
##  1 Angela Merkel           59
##  2 Andrea Nahles           40
##  3 Heiko Maas              38
##  4 Katarina Barley         38
##  5 Peter Altmaier          38
##  6 Wolfgang Schäuble       38
##  7 Wolfgang Kubicki        37
##  8 Hans-Peter Friedrich    34
##  9 Hermann Gröhe           34
## 10 Ursula von der Leyen    33
## # ... with 702 more rows

# retrieve article titles of interest
enf <- "Europe_of_Nations_and_Freedom"
efdd <- "Europe_of_Freedom_and_Direct_Democracy"

enf_parties <- c(
  "Freedom_Party_of_Austria",
  "Vlaams_Belang",
  "National_Rally_(France)",
  "The_Blue_Party_(Germany)",
  "Lega_Nord",
  "Party_for_Freedom",
  "Congress_of_the_New_Right"
)

efdd_parties <- c(
  "Svobodní",
  "The_Patriots_(France)",
  "Debout_la_France",
  "Alternative_for_Germany",
  "Five_Star_Movement",
  "Order_and_Justice",
  "Liberty_(Poland)",
  "Brexit_Party",
  "Social_Democratic_Party_(UK,_1990–present)",
  "Libertarian_Party_(UK)"
)

articles <- c(enf, efdd, enf_parties, efdd_parties)

# import raw clickstream data
cs <-
  read.table(
    "clickstream-enwiki-2019-05.tsv",
    header = FALSE,
    col.names = c("prev", "curr", "type", "n"),
    fill = TRUE,
    stringsAsFactors = FALSE
  )
cs$n <- as.integer(cs$n)

# subset
cs <- subset(cs, prev %in% articles & curr %in% articles)

# assign categories
cs <- cs %>%
  mutate(
    curr_cat = ifelse(
      curr == enf,
      "ENF Group",
      ifelse(
        curr == efdd,
        "EFDD Group",
        ifelse(curr %in% enf_parties, "ENF Parties",
               "EFDD Parties")
      )
    ),
    prev_cat = ifelse(
      prev == enf,
      "ENF Group",
      ifelse(
        prev == efdd,
        "EFDD Group",
        ifelse(prev %in% enf_parties, "ENF Parties",
               "EFDD Parties")
      )
    )
  )

# summarize data
cs_sum <-  cs %>%
  group_by(curr_cat, prev_cat) %>%
  summarize(n = sum(n)) %>%
  arrange(prev_cat)

# Sankey diagram using plotly
labels <- c(unique(cs_sum$prev_cat), unique(cs_sum$curr_cat))
colors <- ifelse(grepl("EFDD", labels), "#24B9B9", "#2B3856")
sankey_plot <- plot_ly(
  type = "sankey",
  orientation = "h",
  
  node = list(
    label = labels,
    color = colors,
    pad = 15,
    thickness = 15,
    line = list(color = "black",
                width = 0.5)
  ),
  
  link = list(
    source = as.numeric(as.factor(cs_sum$prev_cat)) - 1L,
    target = as.numeric(as.factor(cs_sum$curr_cat)) + 3L,
    value =  cs_sum$n
  ),
  
  height = 340,
  width = 600
) %>%
  layout(font = list(size = 10))

# construct edges
cs_edge <-
  cs %>%
  group_by(prev, curr, prev_cat, curr_cat) %>%
  dplyr::summarise(weight = sum(n)) %>%
  arrange(curr)

# get list of unique articles to construct as nodes
cs_node <- 
  gather(cs_edge,
         `prev`,
         `curr`,
         key = "where",
         value = "article") %>%
  ungroup() %>%
  select(article) %>%
  distinct(article)
names(cs_node) <- c("node")
cs_node$category <-
  ifelse(cs_node$node == enf,
         "ENF Group",
         ifelse(
           cs_node$node == efdd,
           "EFDD Group",
           ifelse(cs_node$node %in% enf_parties, "ENF Parties",
                  "EFDD Parties")
         )
  )

# generate graph
set.seed(3)
cs_net <-
  graph.data.frame(cs_edge,
                   vertices = cs_node,
                   directed = F)
cs_net <-
  igraph::simplify(cs_net, remove.multiple = T, remove.loops = T)

# generate colors based on category
V(cs_net)$color <- 
  ifelse(grepl("EFDD", V(cs_net)$category), "#24B9B9", "#2B3856")

# compute node degrees (#links) and use that to set node size
deg <- igraph::degree(cs_net, mode = "all")
V(cs_net)$size <- deg / 10

# set labels
V(cs_net)$label <- NA
V(cs_net)$label.cex = 0.5
V(cs_net)$label = ifelse(igraph::degree(cs_net) > 5, V(cs_net)$label, NA)
cs_hc_labels <- as.vector(cs_node$node)

# set edge width based on weight
E(cs_net)$width <- log(E(cs_net)$weight) / 5
E(cs_net)$edge.color <- "gray80"

# transform the network to a ggplot friendly format
# (required to generate interactive graph embedded in blog post)
gg_cs_net <-
  ggnetwork(
    cs_net,
    layout = "fruchtermanreingold",
    weights = "weight",
    niter = 50000,
    arrow.gap = 0
  )

cs_plot <- ggplot(gg_cs_net, aes(x = x, y = y, xend = xend, yend = yend)) +
  geom_edges(aes(color = edge.color), size = 0.4, alpha = 0.25) +
  geom_nodes(aes(color = color, size = size)) +
  geom_nodetext(aes(color = color, label = vertex.names, cex = 0.6)) +
  guides(size=FALSE) +
  theme_blank() +
  theme(legend.position = "none")

Collecting Data via Wikidata Queries

Wikidata is a collaboratively edited knowledge base with over 58 million entries as of July 2019. It harbors various types of database items, including text, numerical quantities, coordinates, and images. There are no language editions, but individual entries can have values in different languages.

Wikidata allows users to submit queries using SPARQL, a query language for data stored in RDF (Resource Description Framework) format (see this link). Click here for a brief introduction to SPARQL. While basic queries can be used to answer mundane questions (e.g. “what is the capital city of every member of the European Union, and how many inhabitants live there?”), a targeted combination of related queries can be used for systematic data collection.

Instead of submitting explicit SPARQL queries, the example below uses the WikidataR package to combine various queries in order to collect data on the candidates in the 2019 leadership election of the UK Conservative Party. Suppose we want to retrieve the following information on each candidate:

• name
• sex
• date of birth
• political experience
• education
• official website URL
• Twitter accout
• Facebook account

In Wikidata, entries are stored as items with a unique item ID that starts with “Q”. For instance, the item 2019 Conservative Party (UK) leadership election is stored as “Q30325756”. Items are characterized by a number of statements or claims. Claims start with “P” and detail an item’s properties. For instance, the claim “candidate” is stored as “P726”. Claims have values, which may once again be items. For example, the values of claim “P726” (candidate) of item “Q30325756” (2019 UK Conservative Party leadership election) are 10 items: one entry for each of the 10 candidates running in the leadership election. Take, for example, winning candidate Boris Johnson, who is listed as a candidate under claim “P726”. In turn, the entry on Boris Johnson is stored as item “Q180589”. This item is characterized by numerous claims, including “P1559” (name in native language), “P21” (sex or gender), “P569” (date of birth), “P39” (positions held), “P69” (educated at), “P856” (official website), “P2002” (Twitter username), and “P2013” (Facebook ID).

In order to collect the data for all 10 candidates in the 2019 Conservative Party leadership election, the code chunk below implements the following steps:

1. We retrieve item “Q30325756”, i.e., the entry for 2019 Conservative Party (UK) leadership election
2. We extract claims “P726” of the above item to retrieve the item IDs of all 10 candidates, which we store in the object candidates
3. We save the IDs of the claims of interest, stored in the object claims
4. We then use some nested sapplycommands to do the following:
   • Retrieve the item (entry) for each candidate
   • Extract the eight claims from each candidate item
   • Process the informational value of each extracted claim, depending on whether the claim value is
     • an atomic object (such as web site URLs)
     • a textual object with auxiliary information (such as names, which come with language information)
     • a time/date (such as date of birth)
     • yet another item (such as previous positions, where each position has an own data base entry)

# get item based on item id
uk_item <- get_item("Q30325756", language = "en")

# extract candidates
candidates <- extract_claims(uk_item, claims = "P726")
candidates <- candidates[[1]][[1]]$mainsnak$datavalue$value$id

# collect the following attributes ("claims") for each candidate
claims <- c("P1559", "P21", "P569", "P39", "P69", "P856", "P2002", "P2013")
names(claims) <- c("nam", "sex", "dob", "exp", "edu", "web", "twi", "fbk")
claims

##     nam     sex     dob     exp     edu     web     twi     fbk 
## "P1559"   "P21"  "P569"   "P39"   "P69"  "P856" "P2002" "P2013"

# retrieve data
uk_data <-
  sapply(candidates,
         function (item) {
           tmp_item <- get_item(item, language = "en")
           sapply(claims,
                  function(claim) {
                    tmp_claim <- extract_claims(tmp_item, claim)[[1]][[1]]
                    if (any(is.na(tmp_claim))) {
                      return(NA)
                    } else {
                      tmp_claim <- tmp_claim$mainsnak$datavalue$value
                      if (is.atomic(tmp_claim)) {
                        return(tmp_claim)
                      } else if ("text" %in% names(tmp_claim)) {
                        return(tmp_claim$text)
                      } else if ("time" %in% names(tmp_claim)) {
                        tmp_claim <- as.Date(substr(tmp_claim$time, 2, 11))
                        return(tmp_claim)
                      } else if ("id" %in% names(tmp_claim)) {
                        tmp_claim <- tmp_claim$id
                        tmp_claim <- 
                          sapply(tmp_claim, 
                                 get_item, 
                                 language = "en",
                                 simplify = FALSE,
                                 USE.NAMES = TRUE)
                        tmp_claim <-
                          sapply(tmp_claim,
                                 function (x) {
                                   x[[1]]$labels$en$value
                                 })
                        return(tmp_claim)
                      }
                    }
                  },
                  simplify = FALSE,
                  USE.NAMES = TRUE)
         },
         simplify = FALSE,
         USE.NAMES = TRUE
  )

The retrieved data are stored in a nested list. At the upper level of the list, we have the ten candidates, named with their respective item IDs. Nested within each of the ten upper-level elements, we have the values of the eight claims, named with the labels we specified above. Claim values may either be atomic (such as date of birth) or vectors (such as “positions held”, which may have multiple entries). Below, we can see the retrieved data for the first candidate on the list, winning candidate Boris Johnson.

## $nam
## [1] "Boris Johnson"
## 
## $sex
## Q6581097 
##   "male" 
## 
## $dob
## [1] "1964-06-19"
## 
## $exp
##                                                    Q38931 
##                                         "Mayor of London" 
##                                                  Q1371091 
## "Secretary of State for Foreign and Commonwealth Affairs" 
##                                                 Q28841847 
##       "Member of the Privy Council of the United Kingdom" 
##                                                 Q30524710 
##     "Member of the 57th Parliament of the United Kingdom" 
##                                                 Q30524718 
##     "Member of the 56th Parliament of the United Kingdom" 
##                                                 Q35647955 
##     "Member of the 54th Parliament of the United Kingdom" 
##                                                 Q35921591 
##     "Member of the 53rd Parliament of the United Kingdom" 
##                                                    Q14211 
##                    "Prime Minister of the United Kingdom" 
##                                                  Q3303456 
##                        "Leader of the Conservative Party" 
##                                                 Q77685926 
##     "Member of the 58th Parliament of the United Kingdom" 
##                                                   Q609884 
##                              "First Lord of the Treasury" 
##                                                  Q3315116 
##                          "Minister for the Civil Service" 
##                                                 Q65988624 
##                                  "Minister for the Union" 
## 
## $edu
##                       Q192088                       Q805285 
##                "Eton College"             "Balliol College" 
##                      Q4804780                      Q5413121 
##        "Ashdown House School" "European School, Brussels I" 
## 
## $web
## [1] "http://www.boris-johnson.com"
## 
## $twi
## [1] "BorisJohnson"
## 
## $fbk
## [1] "borisjohnson"

legislatoR

legislatoR is a joint project of Sascha Göbel and Simon Munzert. It offers a comprehensive relational individual-level database that provides political, sociodemographic, and other Wikipedia-related data on members of various national parliaments, including the all sessions of the Austrian Nationalrat, the German Bundestag, the Irish Dáil, the French Assemblée, and the United States Congress (House and Senate). It currently comprises data of 42,534 elected representatives and holds information for a wide variety of variables, including:

• sociodemographics (Core)
• basic political variables (Political)
• records of individual Wikipedia data, including full revision histories (History)
• daily user traffic on individual Wikipedia biographies (Traffic)
• social media handles and website URLs (Social)
• URLs to individual Wikipedia portraits (Portraits)
• information on public offices held by MPs (Offices)
• MPs’ occupations (Professions)
• IDs that link politicians to other files, databases and websites (IDs)

The figure below, taken from Göbel and Munzert (2019), illustrates the data structure:

The package provides a relational database. This means that all data sets can be joined with the core data set via one of two keys: the Wikipedia page ID or the Wikidata ID, which uniquely identify individual politicians.

legislatoR services the increasing demand for micro-level data on political elites among political scientists, political analysts, and journalists and offers an accessible and rich collection of data on past and present politicians. The inclusion of Wikipedia and other web data allows for the inclusion of detailed information on politicians’ biographies.

To install the current developmental version from GitHub, we use the devtools package. After installing and loading legislatoR, we can use the ls() command to explore the full functionality of the package.

## Install from GitHub
devtools::install_github("saschagobel/legislatoR")
library(legislatoR)

## View functionality
ls("package:legislatoR")

## Get social media adoption rates
# get data: Germany
dat_ger <- right_join(
  x = get_core(legislature = "deu"),
  y = filter(
    get_political(legislature = "deu"),
    as.numeric(session) == max(as.numeric(session))
  ),
  by = "pageid"
)
dat_ger <- left_join(x = dat_ger,
                     y = get_social(legislature = "deu"),
                     by = "wikidataid")
dat_ger$legislature <- "Germany"
dat_ger_sum <- dat_ger %>%
  dplyr::summarize(
    twitter = mean(not(is.na(twitter)), na.rm = TRUE),
    facebook = mean(not(is.na(facebook)), na.rm = TRUE),
    website = mean(not(is.na(website)), na.rm = TRUE),
    session_start = ymd(first(session_start)),
    session_end = ymd(first(session_end)),
    legislature = first(legislature)
  )

##     twitter  facebook  website session_start session_end legislature
## 1 0.7591036 0.6918768 0.640056    2017-10-24  2021-10-24     Germany

Application 2: Public Attention to Members of the German Bundestag

In the second application, we use pageviews to identify peaks in public attention for MPs over time. This is particularly interesting in the context of politically significant events. For instance, we may want to know about public attention to parliamentarians following scandals, around elections, or during election campaigns. The code below illustrates this logic by averaging daily pageviews across all Wikipedia articles on members of the German Bundestag between July 2015 and December 2017.

Code: Plotting Average Daily Pageviews for German MPs

## Visualize average pageviews data of German MPs
# get data
ger_traffic <- right_join(
  x = get_traffic(legislature = "deu"),
  y = filter(
    get_political(legislature = "deu"),
    session_end >= as.Date("2015-07-01")
  ),
  by = "pageid"
)
ger_traffic <- left_join(x = ger_traffic,
                         y = get_core(legislature = "deu"),
                         by = "pageid")
ger_traffic <-
  dplyr::select(ger_traffic, pageid, date, traffic, session, party, name)

# aggregate data
ger_traffic$date <- ymd(ger_traffic$date)
ger_traffic_date <- group_by(ger_traffic, date)
ger_traffic_legislators <- group_by(ger_traffic, pageid)
ger_traffic_sum <-
  summarize(ger_traffic_date, mean = mean(traffic, na.rm = TRUE))
ger_traffic_sum <- mutate(
  ger_traffic_sum,
  mean_l1 = lag(mean, 1),
  mean_f1 = lead(mean, 1),
  peak = (mean >= 1.8 * mean_l1 &
            mean > 180)
)

# identify peaks
ger_traffic_peaks <- filter(ger_traffic_sum, peak == TRUE)
ger_traffic_peaks_df <-
  filter(ger_traffic, date %in% ger_traffic_peaks$date)
ger_traffic_peaks_group <-
  group_by(ger_traffic_peaks_df, date) %>% 
  dplyr::arrange(desc(traffic)) %>% 
  filter(row_number() == 1)
ger_traffic_peaks_group <- arrange(ger_traffic_peaks_group, date)
events_vec <-
  c(
    "deceased",
    "drug affair",
    "bullying affair",
    "candidacy for presidency",
    "deceased",
    "???",
    "chancellorchip announcement",
    "elected president",
    "???",
    "policy success",
    "TV debate",
    "general election",
    "threat to resign",
    "elected speaker of parliament"
  )

# plot
par(oma = c(0, 0, 0, 0))
par(mar = c(0, 4, 0, .5))
par(yaxs = "i", xaxs = "i", bty = "n")
layout(matrix(c(1, 1, 3, 2, 2, 3), 2, 3, byrow = TRUE),
       heights = c(1, 2, 3),
       widths = c(5, 5, 1.8))
# names labels
plot(
  ymd(ger_traffic_sum$date),
  rep(0, length(ger_traffic_sum$date)),
  xlim = c(ymd("2015-07-01"), ymd("2018-01-01")),
  xaxt = "n",
  ylim = c(0, 8),
  yaxt = "n",
  xlab = "",
  ylab = "",
  cex = 0
)
text(
  ger_traffic_peaks_group$date,
  0,
  ger_traffic_peaks_group$name,
  cex = .75,
  srt = 90,
  adj = c(0, 0)
)
# pageviews time series
par(mar = c(2, 4, 0, .5))
plot(
  ymd(ger_traffic_sum$date),
  ger_traffic_sum$mean,
  type = "l",
  ylim = c(0, 1.25 * max(ger_traffic_sum$mean)),
  xlim = c(ymd("2015-07-01"), ymd("2018-01-01")),
  xaxt = "n",
  yaxt = "n",
  xlab = "",
  ylab = "mean(pageviews)",
  col = "white"
)
abline(h = seq(0, 1.5 * max(ger_traffic_sum$mean), 250), col = "lightgrey")
lines(ymd(ger_traffic_sum$date), ger_traffic_sum$mean, lwd = .5)
dates <- seq(ymd("2015-07-01"), ymd("2018-01-01"), by = 1)
axis(1, dates[day(dates) == 1 &
                month(dates) %in% c(1, 4, 7, 10)], labels = FALSE)
axis(1,
     dates[day(dates) == 1 &
             month(dates) %in% c(1)],
     lwd = 0,
     lwd.ticks = 3,
     labels = FALSE)
axis(1,
     dates[day(dates) == 1 &
             month(dates) %in% c(7)],
     labels = as.character(year(dates[day(dates) == 15 &
                                        month(dates) %in% c(7)])),
     tick = F,
     lwd = 0)
axis(2, seq(0, 1.5 * max(ger_traffic_sum$mean), 250), las = 2)
# events labels in time series
for (i in seq_along(events_vec)) {
  text(ger_traffic_peaks_group$date[i],
       ger_traffic_sum$mean[ger_traffic_sum$peak == TRUE][i] + 80,
       i,
       cex = .8)
  points(
    ger_traffic_peaks_group$date[i],
    ger_traffic_sum$mean[ger_traffic_sum$peak == TRUE][i] + 80,
    pch = 1,
    cex = 2.2
  )
}
# election date
# events labels explained
par(mar = c(0, 0, 0, 0))
plot(
  0,
  0,
  xlim = c(0, 5),
  ylim = c(0, 10),
  xaxt = "n",
  yaxt = "n",
  xlab = "",
  ylab = "",
  cex = 0
)
positions <-
  data.frame(
    events_xpos = 0.45,
    events_ypos = seq(6.5, (6.5 - .5 * length(events_vec)),-.5),
    text_xpos = .5
  )
text(0, 7, "Events", pos = 4, cex = .75)
for (i in seq_along(events_vec)) {
  text(positions$events_xpos[i], positions$events_ypos[i], i, cex = .8)
  points(positions$events_xpos[i],
         positions$events_ypos[i],
         pch = 1,
         cex = 2.2)
  text(
    positions$text_xpos[i],
    positions$events_ypos[i],
    events_vec[i],
    pos = 4,
    cex = .75
  )
}

The plot shows 14 notable spikes in daily pageviews. We identify which politician’s article generated the most traffic during each of these events. Furthermore, we add a legend that lists salient political events that likely caused these spikes in attention to MPs Wikipedia entries. For example, spike 14 marks the day on which Wolfgang Schäuble (CDU) was elected speaker of the parliament.

Conclusion

Collecting and analyzing Wikipedia data is relatively easy and entirely free. It enables researchers to use and analyze an enormous body of data that offers valuable information for research in political science and beyond. Tools that facilitate the collection, processing, and analysis of Wikipedia data advance rapidly, broadening the realm of possibilities for scientific research. Political science research is increasingly picking up on these developments, as is evident in recent contributions (Munzert 2015; Göbel and Munzert 2018; Shi et al. 2019) and softwares (such as legislatoR).

However, using Wikipedia data may also come with limitations and pitfalls. As entries can be read and edited by both humans and machines, the accuracy of contents and the validity of metadata are not guaranteed. With respect to the latter, Wikidata adds provenance information to all the data. These can be used to evaluate the validity of the data in question for applied research. Researchers should also keep in mind that Wikipedia data highly depends on user-driven creation, editing, and use of contents. This may not only lead to systematic selection bias due to data availability but also induce problems of equivalence of data points (e.g., articles on historical political figures likely receive fewer views and edits than articles on active politicians for reasons unrelated to their legislative activity or real-world importance).

These caveats are however all but exclusive to Wikipedia data. They merely underline that Wikipedia data is no exception when it comes to the general necessity of thoroughly scrutinizing and critically assessing the suitability of any given data for addressing substantive research questions.

About the Presenter

Simon Munzert is a lecturer in Political Data Science at the Hertie School of Governance in Berlin, Germany. A former member of the MZES Data and Methods Unit, Simon founded the Social Science Data Lab in 2016. His research focuses on public opinion, political representation, and the role of new media for political processes.

A tour of Quantitative Text Analysis

The workshop started with a few basics: While QTA can be efficient and cheap, it always fails to rely on a correct model of language. It does not free us from reading texts, and validation is key. We learned about the basic distinction between classification (organizing text into categories) and scaling (estimation positions of actors), and its supervised (where hand-coded or other external data is available) and unsupervised (without such data) variants.

(Pre-)processing text

We relied on the R package quanteda developed by Ken Benoit and collaborators, which takes QTA by storm, at least for those working in R. Together with the readtext package, it easily allows to get your text data into R, create a so-called “corpus” of texts with the actual content as well as meta-information, and perform various tasks of corpus and text processing (subsetting a corpus, creating a document-feature matrix (dfm), stopword removal) as well as analysis (scaling, classification). A large and increasing number of extras is also available, such as ways to assess text similarity (function textsta_simil()) and lexical diversity (textstat_lexdiv()). Some of these features are demonstrated in an example below. Also see this overview by quanteda for a full list of functions and the preText package for advise on evaluating pre-processing specifications.

A small coalition corpus

For a few examples from the workshop, consider a small set of three documents: The coalition agreement between the CDU/CSU and the SPD as well as the respective election manifestos from the 2017 Bundestag election. The corpus is created by:

library(readtext)
library(quanteda)
text <- readtext(paste0(wd, "coalition/*.txt"),
                 docvarsfrom = "filenames",
                 docvarnames = "Party")
text$text <- gsub("\n", " ", text$text)
coalitioncorpus <- corpus(text, docid_field = "doc_id")
coalitioncorpus$metadata$source <- "[directory] on [system] by [user]"
summary(coalitioncorpus)

## Corpus consisting of 3 documents:
## 
##           Text Types Tokens Sentences     Party
##     cducsu.txt  4738  26004      1288    cducsu
##  coalition.txt 11660  93214      3763 coalition
##        spd.txt  7650  50298      2402       spd
## 
## Source: [directory] on [system] by [user]
## Created: Wed Nov 11 15:06:41 2020
## Notes:

The code relies on the two packages, readtext and quanteda, to create a data frame with the text files, using their names for a document-level variable called “Party”. The gsub() command removes whitespace, and corpus() turns the data frame into a corpus, which is a special case of a data frame containing texts, some meta-information and document-level variables, all optimized to perform a variety of quantitative text analysis operations using quanteda.

Further document-level variables are added via:

docvars(coalitioncorpus, "Year") <- 2017
docvars(coalitioncorpus, "Party_regex") <- 
  sub("[\\.].*", "", names(texts(coalitioncorpus)))
docvars(coalitioncorpus)

##                   Party Year Party_regex
## cducsu.txt       cducsu 2017      cducsu
## coalition.txt coalition 2017   coalition
## spd.txt             spd 2017         spd

Note that this uses a regular expression (regex) to alternatively retrieve the party names from the filenames after creating the corpus. For specific analyses, we want to know the distribution of words across documents and create a document-feature matrix (dfm):

dfm_coal <- dfm(
  coalitioncorpus,
  remove = c(stopwords("german"),
             "dass",
             "sowie",
             "insbesondere"),
  remove_punct = TRUE,
  stem = FALSE
)
dfm_coal[, 1:8]

## Document-feature matrix of: 3 documents, 8 features (16.7% sparse).
## 3 x 8 sparse Matrix of class "dfm"
##                features
## docs            gutes land zeit deutschland liebens lebenswertes gut
##   cducsu.txt        6   48   11         147       1            1  16
##   coalition.txt     1   39   14         195       0            0  13
##   spd.txt           6   44   33          97       0            0  17
##                features
## docs            wohnen
##   cducsu.txt         1
##   coalition.txt     10
##   spd.txt            6

Creating a dfm induces a bag-of-words assumption. This means that the order in which words appear is ignored. A dfm is a means of information reduction and the most efficient way of storing text as data, but allows only analyses under this assumption. We quickly glance at the similarity (function textstat_simil()) and lexical diversity (function textstat_lexdiv()) of texts:

simil <- textstat_simil(dfm_coal,
                        margin = "documents",
                        method = "correlation")
simil 

## textstat_simil object; method = "correlation"
##               cducsu.txt coalition.txt spd.txt
## cducsu.txt         1.000         0.968   0.975
## coalition.txt      0.968         1.000   0.986
## spd.txt            0.975         0.986   1.000

textstat_lexdiv(dfm_coal)[, 1:2]

##        document       TTR
## 1    cducsu.txt 0.3302084
## 2 coalition.txt 0.2487993
## 3       spd.txt 0.2802713

From this, we learn that the SPD manifesto has more similarity to the coalition agreement than that of the CDU/CSU, a notion which somewhat resembles the assessment of the 2017 German coalition. Also, the lexical diversity of the manifestos, measured in terms of types (different words) per token (total words), appears to be higher than the coalition agreement, especially for the CDU/CSU.

Sentiment analysis using a dictionary

For sentiment analyis, existing dictionaries are available. It is important to note that these do not necessarily fit the research question at hand. In this example, the German “LIWC” (linguistic inquiry and word count) dictionary is used, but alternatives exist, such as “Lexicoder” for political text. LIWC features the categories “anger”, “posemo” (positive emotion) and “religion”. After obtaining the dictionary and applying it while creating a dfm from the corpus, the share of the texts in the respective categories is displayed. The results indicate that the coalition agreement features less positive emotions as compared to the manifestos and that the SPD manifesto is the most “angry” text, while the CDU/CSU speaks most about religion.

# Create dictionary
liwcdict <- dictionary(file = paste0(wd, "German_LIWC2001_Dictionary.dic"),
                       format = "LIWC")

# Create dfm
liwcdfm <- dfm(
  coalitioncorpus,
  remove = c(stopwords("german")),
  remove_punct = TRUE,
  stem = FALSE,
  dictionary = liwcdict
)

# Subset and calculate percentage
liwcsub <-
  dfm_select(liwcdfm,
             pattern = c("Anger", "Posemo", "Relig"),
             selection = "keep")

liwcsub <- convert(liwcsub, to = "data.frame")
liwcsub$sum <- apply(dfm_coal, FUN = sum, 1)
liwcparties <- data.frame(
  docs = liwcsub$document,
  ShareAnger = liwcsub$Anger / liwcsub$sum,
  SharePosemo = liwcsub$Posemo / liwcsub$sum,
  ShareRelig = liwcsub$Relig / liwcsub$sum
)

liwcparties 

##            docs  ShareAnger SharePosemo  ShareRelig
## 1    cducsu.txt 0.004314995  0.06414959 0.005609493
## 2 coalition.txt 0.004666188  0.05050462 0.004074997
## 3       spd.txt 0.005550042  0.05697063 0.003454993

LDA topic modelling

LDA stands for Latent Dirichlet allocation. In a nutshell, the method represents texts as a mixture of topics, and simultaneously topics as mixtures of words. Fixing the number of topics to \(k = 5\), and using the topicmodels package, the command lda() delivers posterior probabilities of the topics for each document.

library(topicmodels)

# Preparation
dfm_coal <-
  dfm(
    coalitioncorpus,
    remove = c(
      stopwords("german"),
      "dass",
      "sowie",
      "insbesondere",
      "b",
      "z",
      "a",
      "u"
    ),
    remove_punct = TRUE,
    remove_numbers = TRUE,
    stem = FALSE
  )

dfm_coal <- dfm_wordstem(dfm_coal, language = "german")

# Define parameters
burnin <- 1000
iter <- 500
keep <- 50
seed <- 2010
ntopics <- 5

# Run LDA with 5 topics
ldaOut <- LDA(
  dfm_coal,
  k = ntopics,
  method = "Gibbs",
  control = list(
    burnin = burnin,
    iter = iter,
    keep = keep,
    seed = seed,
    verbose = FALSE
  )
)
# Posterior probabilities of the topics for each document
k <- posterior(ldaOut)

Interpretation is the difficult part. Each document can be expressed as a mixture of topics, and notwithstanding the precise meaning of the topics, we learn that all texts share content referring to topic 1, while the CDU/CSU manifesto also features topic 2 and the SPD manifesto topic 3 to some extent.

k$topics

##                       1          2          3          4          5
## cducsu.txt    0.5815482 0.02729874 0.36122496 0.01144536 0.01848272
## coalition.txt 0.6950774 0.03700023 0.06163624 0.10570834 0.10057777
## spd.txt       0.6755292 0.17646590 0.11404313 0.01837605 0.01558576

Wordfish Scaling

Wordfish scaling derives latent positions from texts based on a bag-of-words assumption. Here is an example relying on a set of Swiss manifestos, using only the sections on immigration. In preparation, a dfm is created while removing stopwords, stemming the remaining words and removing punctuation.

manifestos <- readtext(paste0(wd, "manifestos/*.txt"))
manifestocorpus <- corpus(manifestos)
dfm_manifesto <-
  dfm(
    manifestocorpus,
    remove = c(
      "gruen*",
      "sp",
      "sozialdemokrat*",
      "cvp",
      "fdp",
      "svp",
      "fuer",
      "dass",
      "koennen",
      "koennte",
      "ueber",
      "waehrend",
      "wuerde",
      "wuerden",
      "schweiz*",
      "partei*",
      stopwords("german")
    ),
    valuetype = "glob",
    stem = FALSE,
    remove_punct = TRUE
  )
dfm_manifesto <- dfm_wordstem(dfm_manifesto, language = "german")

The function textmodel_wordfish() computes the Wordfish model, originally decribed in an AJPS article by Slapin and Proksch in 2009. It assumes that the distribution of words across texts follows a Poisson distribution, and can be modeled by document and word fixed effects as well as word-specific weights and document positions. The model is a variant of unsupervised scaling, only requiring the relative location of two texts on the latent dimension. Here, a text of the Swiss People’s Party (SVP) is assumed to be right to that of the Social Democratic Party of Switzerland (SPS). The results make some sense, with the other manifestos aligning as expected on what could be interpreted as a anti-immigration dimension.

wf <- textmodel_wordfish(dfm_manifesto,
                         dir = c(13, 19),
                         dispersion = "poisson")
textplot_scale1d(wf)

Or, with some improvements to the plot:

library(ggplot2)

# Save document scores and confidence intervals in data frame
wfdata <- as.data.frame(predict(wf, interval = "confidence"))

# Add document variables
wfdata$docs <- rownames(wfdata)
wfdata$electionyear <- substr(wfdata$docs, 5, 8)
wfdata$party <- as.factor(substr(wfdata$docs, 1, 3))
wfdata$party <-
  factor(wfdata$party, levels = c("gps", "sps", "cvp", "fdp", "svp"))

ggplot(wfdata) +
  geom_pointrange(
    aes(
      x = electionyear,
      y = fit.fit,
      ymin = fit.lwr,
      ymax = fit.upr,
      group = party,
      color = party
    ),
    size = 0.5
  ) +
  geom_line(aes(
    x = electionyear,
    y = fit.fit,
    group = party,
    color = party
  )) +
  theme_bw() +
  labs(title = "Wordfish analysis",
       y = "Document position",
       x = "Electionyear") +
  scale_color_manual(values = c("green3",
                                "red1",
                                "darkorange1",
                                "dodgerblue4",
                                "springgreen4"))

Further readings

• An introductory article to QTA in R, especially relying on quanteda: Welbers, Kasper, Wouter Van Atteveldt and Kenneth Benoit (2017): Text Analysis in R, Communication Methods and Measures 11(4): 245-65.
• An application of the methods by the workshop host and co-authors: Schwarz, Daniel, Denise Traber and Kenneth Benoit (2017): Estimating Intra-Party Preferences: Comparing Speeches to Votes. Political Science Research and Methods 5(2): 379-396.

About the presenter

Denise Traber is a Senior Research Fellow at the University of Lucerne, Switzerland, where she heads an Ambizione research grant project on “The divided people: polarization of political attitudes in Europe” funded by the Swiss National Science Foundation. She has a strong interest in quantitative text analysis, co-organizes the “Zurich Summer School for Women in Political Methodology” and has published the article “Estimating Intra-Party Preferences: Comparing Speeches to Votes” in Political Science Research and Methods in 2017, jointly with Daniel Schwarz and Ken Benoit.

About Twitter

Twitter is an online news and social networking service, also used for micro-blogging. In everyday use, it mostly serves as a platform for publicly sharing short texts – often along with media content and/or links – in the form of so-called “tweets”. Twitter has approx. 326 Million monthly active users who send about 500 Million tweets each day (see this fact sheet).

Twitter in Social Science Research

Analyzing tweets and social interaction on Twitter can help to answer social science research questions, especially in communication research and political science. Contrary to Facebook (API depreciation/shut-down in April 2018) and Instagram (API depreciation/shut-down in December 2018), Twitter data is (easily) accessible for researchers.

According to the Web of Science database, there are 2,598 journal articles published since 2009 with the word “Twitter” in their titles.

Figure 1: Number of articles by year

Figure 2: Number of articles by discipline

Collecting Twitter Data

# Install Packages
install.packages("rtweet")
install.packages("ggmap")
install.packages("igraph")
install.packages("ggraph")
install.packages("tidytext")
install.packages("ggplot2")
install.packages("dplyr")
install.packages("readr")

# Set Working Directory
setwd("/PATH")

# Open Libraries
library(rtweet)

# Speficy Authentification Token's provided in your Twitter App
create_token(
  app = "APPNAME",
  consumer_key = "consumer_key",
  consumer_secret = "consumer_secret",
  access_token = "access_token",
  access_secret = "access_secret"
)

# Collect a 'random' sample of Tweets for 10 seconds
stream_tweets(
  q = "",
  timeout = 10,
  file_name = "sample.json",
  parse = FALSE
)
sample <- parse_stream("sample.json")
save(sample, file = "sample_live.Rda")

# Collect Tweets that contain specific keywords for 30 seconds
stream_tweets(
  q = "trump, donald trump",
  timeout = 30,
  file_name = "trump.json",
  parse = FALSE
)
trump <- parse_stream("trump.json")
save(trump, file = "trump_live.Rda")

# Collect Tweets from a specific location
stream_tweets(
  c(13.0883,52.3383,13.7612,52.6755), 
  timeout = 180,
  file_name = "berlin.json",
  parse = FALSE
)
berlin <- parse_stream("berlin.json")
save(berlin, file = "berlin_live.Rda")

# Dimensions of the data frame
dim(sample)

## [1] 359  88

# Variables in the data frame
names(sample)

##  [1] "user_id"                 "status_id"              
##  [3] "created_at"              "screen_name"            
##  [5] "text"                    "source"                 
##  [7] "display_text_width"      "reply_to_status_id"     
##  [9] "reply_to_user_id"        "reply_to_screen_name"   
## [11] "is_quote"                "is_retweet"             
## [13] "favorite_count"          "retweet_count"          
## [15] "hashtags"                "symbols"                
## [17] "urls_url"                "urls_t.co"              
## [19] "urls_expanded_url"       "media_url"              
## [21] "media_t.co"              "media_expanded_url"     
## [23] "media_type"              "ext_media_url"          
## [25] "ext_media_t.co"          "ext_media_expanded_url" 
## [27] "ext_media_type"          "mentions_user_id"       
## [29] "mentions_screen_name"    "lang"                   
## [31] "quoted_status_id"        "quoted_text"            
## [33] "quoted_created_at"       "quoted_source"          
## [35] "quoted_favorite_count"   "quoted_retweet_count"   
## [37] "quoted_user_id"          "quoted_screen_name"     
## [39] "quoted_name"             "quoted_followers_count" 
## [41] "quoted_friends_count"    "quoted_statuses_count"  
## [43] "quoted_location"         "quoted_description"     
## [45] "quoted_verified"         "retweet_status_id"      
## [47] "retweet_text"            "retweet_created_at"     
## [49] "retweet_source"          "retweet_favorite_count" 
## [51] "retweet_retweet_count"   "retweet_user_id"        
## [53] "retweet_screen_name"     "retweet_name"           
## [55] "retweet_followers_count" "retweet_friends_count"  
## [57] "retweet_statuses_count"  "retweet_location"       
## [59] "retweet_description"     "retweet_verified"       
## [61] "place_url"               "place_name"             
## [63] "place_full_name"         "place_type"             
## [65] "country"                 "country_code"           
## [67] "geo_coords"              "coords_coords"          
## [69] "bbox_coords"             "status_url"             
## [71] "name"                    "location"               
## [73] "description"             "url"                    
## [75] "protected"               "followers_count"        
## [77] "friends_count"           "listed_count"           
## [79] "statuses_count"          "favourites_count"       
## [81] "account_created_at"      "verified"               
## [83] "profile_url"             "profile_expanded_url"   
## [85] "account_lang"            "profile_banner_url"     
## [87] "profile_background_url"  "profile_image_url"

# An anonymized portion of the data frame, only tweets in English
sample$user_id <- seq(1, nrow(sample), 1)
sample$screen_name <- paste("name", seq(1, nrow(sample), 1), sep = "_")
sample <- subset(sample, lang == "en")
sample[1:5, c("user_id", "created_at", "screen_name", "text", "is_quote", 
              "is_retweet")]

## # A tibble: 5 x 6
##   user_id created_at          screen_name text          is_quote is_retweet
##     <dbl> <dttm>              <chr>       <chr>         <lgl>    <lgl>     
## 1       1 2019-04-03 10:22:32 name_1      just see..<U+2764><U+FE0F>~ FALSE    FALSE     
## 2       4 2019-04-03 10:22:33 name_4      I’m slowy gi~ FALSE    TRUE      
## 3      15 2019-04-03 10:22:33 name_15     "\"I didn't ~ TRUE     FALSE     
## 4      20 2019-04-03 10:22:33 name_20     Hmm. I’ma de~ FALSE    TRUE      
## 5      28 2019-04-03 10:22:33 name_28     I still hear~ TRUE     TRUE

Analyzing Twitter Data

We have now collected Tweets and meta-information based on selected keywords and/or regional parameters. This begs the question what to do with the raw data. Some common research interests include:

• Content Analysis: What kind of topics are users talking about?
• Sentiment Analysis: What kind of opinions, attitudes, and emotions towards objects are users communicating?
• Network Analysis: Who is related to whom? Who are important users?
• Geospatial Analysis: Where are users/Tweets coming from?

In the following, we present two quick examples that showcase possible avenues for the analysis of Twitter data.

Example 1: Prepping Tweets for Text Analysis

Quantitative Text Analysis (QTA) typically relies on pre-processed textual data (see this blog post for our SSDL workshop on QTA by Denise Traber). The code chunk below illustrates how a collection of Tweets can easily be prepared for QTA techniques such as content or sentiment analysis.

In this example, we use our 30 second sample of Tweets containing the key words “donald trump” and/or “trump”, which we stored as trump_live.RDa (see Collecting Tweets Using the rtweet Package above). Tweet contents are stored in the variable trump$text. Using the tidytextand dplyr packages, we then process the Tweets into a text corpus as follows:

1. Remove URLs from all tweets using gsub()
2. Remove punctuation, convert to lowercase, and seperate all words using unnest_tokens()
3. Remove stop words by first loading a list of stop words from the tidytext package via data("stop_words") and then removing these words from our tweets via anti_join(stop_words)

Code: Preparing Data for Content Analysis

# Open Libraries
library(tidytext)
library(dplyr)

# Data Cleaning
# Delete Links in the Tweets
trump$text <- gsub("http.*", "", trump$text)
trump$text <- gsub("https.*", "", trump$text)
trump$text <- gsub("&amp;", "&", trump$text)

# Remove punctuation, convert to lowercase, seperate all words
trump_clean <- trump %>%
  dplyr::select(text) %>%
  unnest_tokens(word, text)

# Load list of stop words - from the tidytext package
data("stop_words")

# Remove stop words from your list of words
cleaned_tweet_words <- trump_clean %>%
  anti_join(stop_words)

Following this, we can calculate the word counts in our Tweet collection and visualize the 15 most frequent words:

Code: Plotting the 15 Most Frequent Words

# Plot the top 15 words
cleaned_tweet_words %>%
  count(word, sort = TRUE) %>%
  top_n(15) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(x = word, y = n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip() +
  labs(y = "Count",
       x = "Unique words",
       title = "Count of unique words found in tweets",
       subtitle = "Stop words removed from the list")

## Selecting by n

Figure 3: Top 15 Most Frequent Words

Example 2: Analyzing Locations

Provided that users share their exact geo-information, we can locate their tweets on geographical maps. For example, in our 3 minute sample of Tweets from berlin (see Collecting Tweets Using the rtweet Package above), this information was available for only two Tweets.

The code below demonstrates how one can view these Tweets mapped onto their exact geographical locations. We first pre-process the data, extracting numerical geo-coordinates for longitude and latitude where this information is available. We then subset the data frame to observations where these information exist. Lastly, using the same geo-coordinates as in the data collection, we set up an empty rectangular map of Berlin. Note that the last step involves accessing the Google Maps API, for which you will have to register separately (see, e.g., this post at R-bloggers).

Code: Analyzing Locations - Setup

library(ggmap)
library(dplyr)

# Seperate Geo-Information (Lat/Long) Into Two Variables
berlin <- tidyr::separate(data = berlin,
                          col = geo_coords,
                          into = c("Latitude", "Longitude"),
                          sep = ",",
                          remove = FALSE)

# Remove Parentheses
berlin$Latitude <- stringr::str_replace_all(berlin$Latitude, "[c(]", "")
berlin$Longitude <- stringr::str_replace_all(berlin$Longitude, "[)]", "")

# Store as numeric
berlin$Latitude <- as.numeric(berlin$Latitude)
berlin$Longitude <- as.numeric(berlin$Longitude)

# Keep only those tweets where geo information is available
berlin <- subset(berlin, !is.na(Latitude) & !is.na(Longitude))

# Set up empty map
berlin_map <- get_map(location = c(lon = mean(c(13.0883, 13.7612)),
                                   lat = mean(c(52.3383, 52.6755))),
                      zoom = 10,
                      maptype = "terrain",
                      source = "google")

Building up on this, we can then map our collected tweets to see from where they were sent: Berlin-Mitte and Potsdam.

Code: Analyzing Locations - Map

tweet_map <- ggmap(berlin_map)
tweet_map + geom_point(data = berlin,
                       aes(x = Longitude,
                           y = Latitude),
                           color = "red",
                           size = 4,
                           alpha = .5)

Figure 4: GPS Coordinates of 2 Tweets

Potential Issues and Challenges

Conclusion: Twitter in the Social Sciences

Collecting Twitter data is comparatively easy and cheap. However, we usually know close to nothing about the users whom we collect tweets from. Thus, the research potential for social science projects is very limited when we are interested in questions that address ‘outside-social-media’ phenomena.

Arguably, we benefit most from Twitter data when we treat it as auxiliary or proxy information, or when we use it in combination with other data sources. Potential applications along those lines include:

• Variation in fear of crime across different regions
• Health monitoring over time
• Monitoring highly discussed topics in realtime
• Measuring existing stereotypes towards minorities

Further Reading

• Workshop on using rtweet by its developer Michael W. Kearney
• rtweet documentation
• Test mining of tweets
• Using Shiny and Leaflet
• Rieder, Y. & S. Kühne, 2018: Geospatial Analysis of Social Media Data - A Practical Framework and Applications. In: Stuetzer, C.M., Welker, M. & M. Egger (Eds.), Computational Social Science in the Age of Big Data. Concepts, Methodologies, Tools, and Applications. DGOF Schriftenreihe, Köln: Herbert van Halem Verlag. URL: http://www.halem-verlag.de/computational-social-science-in-the-age-of-big-data/.

About the Presenter

Simon Kühne is a post-doc at Bielefeld University. He holds a BA in Sociology and an MA in Survey Methodology from the University of Duisburg-Essen and a PhD in Sociology from Humboldt University of Berlin. His research focuses on survey methodology, social media and online data, and social inequality.

Overview

1. What is visual inference?
2. Potential challenges and how to overcome them
3. Practical applications: How do we reveal the “true” data graphically? A step-by-step guide
   1. Maps
   2. Scatter plot
   3. Group comparisons
4. Further readings

What is visual inference?

Visual inference uses our ability to detect graphical anomalies. The idea of formal testing remains the same in visual inference – with one exception: The test statistic is now a graphical display which is compared to a “reference distribution” of plots showing the null. Put differently, we plot both the “true pattern of the data” and additional random plots of our data. By comparing both, we should be able to identify the true data – if the pattern is not based on randomness. This approach can be applied to various (research) situations – some of them are described in the “Practical applications” section.

Potential challenges and how to overcome them

Major concerns related to exploratory data analysis are its seemingly informal approach to data analysis and the potential over-interpretation of patterns. Richard provides a line-up protocol how to best overcome these concerns:

<sub>1. Identify the question the plot is trying to answer or the pattern it is intended to show.
2. Formulate a null hypothesis (usually this will be \(H_0\): “There is no pattern in the plot.”)
3. Generate and visualize a null datasets (e.g., permutations of variable values, random simulations)</sub>

The following examples illustrate this procedure and explain the steps in detail.

Practical applications: How do we reveal the “true” data graphically? A step-by-step guide

To reveal the “true” data, we may use several visual approaches. In the following, we present three different examples: 1) maps, 2) scatter plots, and 3) group comparisons. The underlying logic follows the line-up protocol described above. To produce the visual inference, we always apply the following steps:

<sub>1. Identify the question: ‘Is there a visual pattern?’
2. Formulate a null hypothesis: ‘There is no visual pattern.’
3. Generate null datasets: Just randomly permute one variable column and plot the data.
4. Add the “true” data: Add the true data to the null datasets.
5. Visual inference: Is there a visual difference between the randomly permuted data and the “true” data?</sub>

# Read all required packages
library(maps)
library(mapdata)
library(RColorBrewer)

# Read data
data <- readRDS("sub_data.rds")

# Code interviewer behavior by color
data$col <-
  ifelse(data$status == "No Contact", "maroon3", "darkolivegreen2")

# Generate random plot placement
placement <- sample((1:20), 20)
layout(matrix(placement, 4, 5))

# Generate 19 null plots
par(mar = c(.01, .01, .01, .01), oma = c(0, 0, 0, 0))
for(i in 1:19) {
  # Randomize the order
  random <- sample(c(1:15591), 15591)
  # Plot
  map(
    # Refer to dataset
    database = "worldHires",
    fill = F,
    col = "darkgrey",
    # Range of x-axis
    xlim = c(6, 15),
    # Range of y-axis
    ylim = c(47.3, 55)
    ) 
  points(
    # Refer to data
    data$g_lon,
    data$g_lat,
    cex = .1,
    # Type of plotting symbol
    pch = 19,
    col = data$col[random]
    )
}

# Add the true plot
map(
  database = "worldHires",
  fill = F,
  col = "darkgrey",
  # Range of x-axis
  xlim = c(6, 15),
  # Range of y-axis
  ylim = c(47.3, 55)
  ) 
points(
  # Refer to data
  data$g_lon,
  data$g_lat,
  cex = .1,
  # Type of plotting symbol
  pch = 19,
  col = data$col
  )

# Reveal the true plot
box(col = "red", # Draw a box in red
    lty = 2, # Defines line type
    lwd = 2) # Defines line width
    which(placement == 20) # Defines the place of the box

# Read required package
library(foreign) # Necessary to load datasets in other formats (such as .dta)

# Read the data
slop <- read.dta("slop_2009_agg_example.dta")

# Generate a random plot placement
placement <- sample((1:20), 20)
layout(matrix(placement, 4, 5))

# Plot 19 null plots
par(mar = c(.1, .1, .1, .1))
for(i in 1:19) {
  # Plot random scatter plots of the data
  random <- sample(c(1:dim(slop)[1]), dim(slop)[1])
  plot(slop$mkath[random],
  slop$cdu,
  axes = F,
  ann = F,
  cex = .4)
  # Plot a box with grey lines
  box(bty = "l", 
  col = "grey")
}

# Add true plot
plot(slop$mkath,
     slop$cdu,
     axes = F,
     ann = F,
     cex = .4)
box(bty = "l", # Plot a box with grey lines
    col = "grey")

# Reveal true plot
box(col = "red", # Plot a box with red dashed lines
    lty = 2,
    lwd = 2)
which(placement == 20) # Define the position of the box

abline(lm(slop$cdu ~ slop$mkath[random]))

# Generate a random plot placement
placement <- sample((1:20), 20)
layout(matrix(placement, 4, 5))

# Plot 19 null plots
par(mar = c(.1, .1, .1, .1))
for(i in 1:19) {
  # Plot random scatter plots of the data
  random <- sample(c(1:dim(slop)[1]), dim(slop)[1])
  plot(slop$mkath[random],
  slop$cdu,
  axes = F,
  ann = F,
  cex = .4)
  # Add the abline to the plots
  abline(lm(slop$cdu ~ slop$mkath[random]))
  # Plot a box with grey lines
  box(bty = "l",
  col = "grey")
}
# Add true plot
plot(slop$mkath,
     slop$cdu,
     axes = F,
     ann = F,
     cex = .4)
abline(lm(slop$cdu ~ slop$mkath)) # Add the abline to the plot
box(bty = "l", # Plot a box with grey lines
    col = "grey")

# Reveal true plot
box(col = "red", # Plot a box with red dashed lines
    lty = 2,
    lwd = 2)
which(placement == 20) # Define the position of the box

# Generate random plot placement
placement <- sample((1:20), 20)
layout(matrix(placement, 4, 5))

# Plot 19 Null Plots
par(mar = c(.1, .1, .1, .1))
for (i in 1:19) {
  random <- sample(c(1:dim(slop)[1]), dim(slop)[1])
  
  plot(
    slop$bayern[random],
    slop$cdu,
    axes = F,
    ann = F,
    cex = .4,
    xlim = c(-1, 2)
    )
  points(1,
         mean(slop$cdu[slop$bayern[random] == 1]),
         pch = "-",
         col = "purple4",
         cex = 3)
  points(0,
         mean(slop$cdu[slop$bayern[random] == 0]),
         pch = "-",
         col = "darkolivegreen2",
         cex = 3)
  box(bty = "l", col = "grey")
  
}

# Add true plot
plot(
  slop$bayern,
  slop$cdu,
  axes = F,
  ann = F,
  cex = .4,
  xlim = c(-1, 2)
  )
points(1,
       mean(slop$cdu[slop$bayern == 1]),
       pch = "-",
       col = "purple4",
       cex = 3)
points(0,
       mean(slop$cdu[slop$bayern == 0]),
       pch = "-",
       col = "darkolivegreen2",
       cex = 3)
box(bty = "l", col = "grey")

# Reveal True Plot
box(col = "red", lty = 2, lwd = 2)
which(placement == 20)

Further readings

• Buja, A., Cook, D., Hofmann, H., Lawrence, M., Lee, E. K., Swayne, D. F., & Wickham, H. (2009). Statistical inference for exploratory data analysis and model diagnostics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906), 4361-4383.
• Few, S. (2009). Now you see it: simple visualization techniques for quantitative analysis. Analytics Press.
• Gelman, A. (2003). A Bayesian formulation of exploratory data analysis and goodness‐of‐fit testing. International Statistical Review, 71(2), 369-382.
• Gelman, A. (2004). Exploratory data analysis for complex models. Journal of Computational and Graphical Statistics, 13(4), 755-779.
• Traunmüller, R. Visual statistical inference for political research.
• Wickham, H., Cook, D., Hofmann, H., & Buja, A. (2010). Graphical inference for infovis. IEEE Transactions on Visualization and Computer Graphics, 16(6), 973-979.

About the presenter

Richard Traunmüller is a Visiting Associate Professor of Political Science at the University of Mannheim and currently on leave from Goethe University Frankfurt, where he is an Assistant Professor of Empirical Democracy Research. He has a strong interest in Bayesian analysis, data visualization, and survey experiments. He studies challenges that arise from deep-seated societal change: global migration and religious diversity, free speech in the digital age, as well as the legacies of civil war and sexual violence.