Known categories: Dictionaries

Dictionaries contain lists of words that correspond to different categories. If we apply a dictionary approach, we count how often words that are associated with different categories are represented in each document. These dictionaries help us to classify (or categorize) the speeches based on the frequency of the words that they contain. Popular dictionaries are sentiment dictionaries (such as Bing, Afinn or LIWC) or LexiCoder.

We use the “LexiCoder Policy Agenda” dictionary that can be accessed here in a .lcd format. The “LexiCoder Policy Agenda” dictionary captures major topics from the comparative Policy Agenda project and is currently available in Dutch and English.

To read in the dictionary, we use quanteda’s built-in function dictionary().

# load the dictionary with quanteda's built-in function
dict <- dictionary(file = "policy_agendas_english.lcd")

We apply this dictionary to filter the share of each country’s speeches on immigration, international affair and defence.

mydfm.un <- dfm(mydfm.trim, groups = "country", dictionary = dict)

un.topics.pa <- convert(mydfm.un, "data.frame") %>%
  dplyr::rename(country = doc_id) %>%
  select(country, immigration, intl_affairs, defence) %>%
  tidyr::gather(immigration:defence, key = "Topic", value = "Share") %>%
  group_by(country) %>%
  mutate(Share = Share / sum(Share)) %>%
  mutate(Topic = haven::as_factor(Topic))

In a next step, we can visualize the results with ggplot. This gives us a first impression of the distribution of the topics in the 2018 UN General Debate across countries.

un.topics.pa %>%
  ggplot(aes(country, Share, colour = Topic, fill = Topic)) +
  geom_bar(stat = "identity") +
  scale_colour_brewer(palette = "Set1") +
  scale_fill_brewer(palette = "Pastel1") +
  ggtitle("Distribution of PA topics in the UN General Debate corpus") +
  xlab("") +
  ylab("Topic share (%)") +
  theme(axis.text.x = element_blank(),
        axis.ticks.x = element_blank())

Figure 3: Distribution of PA topics in the UN General Debate corpus

We observe a relatively high share for both defence and international affairs whereas immigration receives fewer attention in the speeches.

Known categories: Supervised machine learning - Naive Bayes (NB)

We now turn to supervised machine learning. Similar to the dictionary approach explained above, this method also requires some pre-existing classifications. But in contrast to a dictionary, we now divide the data into a training and a test dataset. This follows the general logic of machine learning algorithms. The training data already contains the classifications and trains the algorithm (e.g., our Naive Bayes classifier) to predict the class of our speech based on the features that are given. A Naive Bayes classifier now calculates the probability for each class based on the features. It eventually goes for the class with the highest probability and selects this class as the corresponding category. It is based on the Bayes theorem for conditional probability. It can be formally written as: \[ P(A | B) = \frac{P(A) * P(B | A)}{P(B)}\] In plain words, the probability of A is conditional on B.

Why is Naive Bayes “naive”? Naive Bayes is “naive” because of its strong independence assumptions. It assumes that all features are equally important and that all features are independent. If you think of n-grams and compare unigrams and bigrams, you can intuitively understand why the last assumption is a strong assumption. A unigram counts each word as a gram (“I” “like” “walking” “in” “the” “sun”) whereas a bigram counts two words as a gram (“I like” “like walking” “walking in” “in the” “the sun”).

However, even when the assumptions are not fully met, Naive Bayes still performs well.

A Naive Bayes is a relatively simple classification algorithm because it does not require much time and working capacity of your machine. To use a Naive Bayes classifier, we rely on quanteda’s built-in function textmodel_nb.

To perform the Naive Bayes estimation, we proceed with the following steps:

<sub>1. We set up training and test data based on the corpus.
2. Based on these two datasets, we generate a DFM.
3. We train the algorithm by feeding in the training data and eventually use the test data for performance.
4. We then check the performance (accuracy) of our results.
5. And compare it with a random prediction.</sub>

For this example, we use the pre-labeled dataset that is used for the algorithm newsmap by Kohei Watanabe. The dataset contains information on the geographical location of newspaper articles. We introduce this new dataset as Naive Bayes – a supervised machine learning algorithm – requires pre-labeled data.

We first load the dataset. To do so, we follow Kohei Watanabe’s description here, download the corpus of Yahoo News from 2014, and follow the subsequent processing steps he describes.

# load data
load("../newspaper.RData")

# transform variables
pred_data$text <- as.character(pred_data$text)
pred_data$country <- as.character(pred_data$country)

For simplicity, we keep only the USA, Great Britain, France, Brazil, and Japan.

pred_data <- pred_data %>%
  dplyr::filter(country %in% c("us", "gb", "fr", "br", "jp")) %>%
                  dplyr::select(text, country)

head(pred_data)

We pre-process the data again. Our final corpus thus includes the newspaper headlines by country.

data_corpus <- corpus(pred_data, text_field = "text")

In a first step, we need to define our training and our test dataset. Based on these two datasets, we generate a DFM. This code is based on Cornelius code and quanteda’s example. To do so, we apply similar general data pre-processing steps as discussed above.

# Set a seed for replication purposes
set.seed(68159)

# Generate random 10,000 numbers without replacement
training_id <- sample(1:29542, 10000, replace = FALSE)

# Create docvar with ID
docvars(data_corpus, "id_numeric") <- 1:ndoc(data_corpus)

# Get training set
dfmat_training <-
  corpus_subset(data_corpus, id_numeric %in% training_id) %>%
  dfm(stem = TRUE)

# Get test set (documents not in training_id)
dfmat_test <-
  corpus_subset(data_corpus,!id_numeric %in% training_id) %>%
  dfm(stem = TRUE)

We can now check the distribution of the countries across the two DFMs:

print(prop.table(table(docvars(
  dfmat_training, "country"
))) * 100)

   br    fr    gb    jp    us 
13.34 14.37 33.65 11.10 27.54 

print(prop.table(table(docvars(
  dfmat_test, "country"
))) * 100)

   br       fr       gb       jp       us 
13.70893 15.13151 33.02630 10.93542 27.19783 

As we can see, the countries are equally distributed across both DFMs.

In a next step, we train the Naive Bayes classifier. Going back to the formula stated above, we know that A is conditional on B.

\[ P(A | B) = \frac{P(A) * P(B | A)}{P(B)}\]

A is what we want to know (the country that is mainly addressed in each text) and B is what we see (the text). We can now proceed and replace A and B with the respective terms. This leads us to the next equation:

\[ P(Country | Text) = \frac{P(Country) * P(Text | Country)}{P(Text)}\]

We can then proceed and train our algorithm using quanteda’s built-in function textmodel_nb.

# Train naive Bayes
# The function takes a DFM as the first argument 
model.NB <-
  textmodel_nb(dfmat_training, docvars(dfmat_training, "country"), prior = "docfreq")

# The prior indicates an assumed distribution. 
# Here we choose how frequently the categories occur in our data.

dfmat_matched <-
  dfm_match(dfmat_test, features = featnames(dfmat_training))

The command summary(model.NB) gives us the results of our prediction. Click unfold to see the results.

summary(model.NB)

Call:
textmodel_nb.dfm(x = dfmat_training, y = docvars(dfmat_training, 
    "country"), prior = "docfreq")

Class Priors:
(showing first 5 elements)
    br     fr     gb     jp     us 
0.1334 0.1437 0.3365 0.1110 0.2754 

Estimated Feature Scores:
         '     08   french   champ  ivanov    lose      to safarova     in     3rd      .     pari
br 0.08949 0.1679 0.007791 0.07636 0.08195 0.08707 0.09989   0.1474 0.1214 0.34322 0.1176 0.008863
fr 0.14212 0.2778 0.940061 0.37912 0.20345 0.16469 0.13803   0.3659 0.1473 0.24345 0.1296 0.947706
gb 0.43332 0.1532 0.032008 0.24397 0.14963 0.49395 0.35845   0.1346 0.3540 0.17905 0.3321 0.032363
jp 0.06507 0.1184 0.004396 0.10771 0.28900 0.06580 0.10287   0.1040 0.1007 0.06916 0.1059 0.002778
us 0.27000 0.2827 0.015743 0.19285 0.27598 0.18850 0.30077   0.2482 0.2766 0.16512 0.3148 0.008290
         (      ap       )      -  former    open champion     ana    lost    the   third   round
br 0.16080 0.24091 0.16100 0.1611 0.09034 0.17116  0.12708 0.08142 0.08045 0.1101 0.08964 0.30833
fr 0.14221 0.14820 0.14219 0.1363 0.17312 0.18636  0.24110 0.13475 0.16643 0.1350 0.17178 0.19535
gb 0.34860 0.28303 0.34870 0.3403 0.45720 0.36654  0.51497 0.07433 0.38923 0.3603 0.39194 0.24189
jp 0.09123 0.08125 0.09126 0.0922 0.04583 0.07443  0.05684 0.22969 0.07943 0.1005 0.07653 0.04757
us 0.25715 0.24661 0.25685 0.2702 0.23351 0.20152  0.06002 0.47982 0.28446 0.2940 0.27010 0.20686
   saturday      ,  beaten     6-3      by 23rd-seed
br  0.14287 0.1078 0.19323 0.21226 0.09731    0.1955
fr  0.19921 0.1384 0.20558 0.40985 0.12900    0.3236
gb  0.44845 0.3437 0.45359 0.21530 0.33256    0.1785
jp  0.09248 0.1099 0.07787 0.08317 0.11636    0.1379
us  0.11699 0.3001 0.06972 0.07942 0.32477    0.1646

To better understand how well we did, we can also generate two frequency tables for right and wrong predictions.

prop.table(table(predict(model.NB) == docvars(dfmat_training, "country"))) * 100

FALSE  TRUE 
 2.71 97.29  

To check if this result indicates a good performance, we compare it with a random result. We randomize the list of countries (and keep the overall frequency distribution of our countries constant) to allow our random algorithm a legitimate chance for a correct classification.

prop.table(table(sample(predict(model.NB)) == docvars(dfmat_training, "country"))) * 100

FALSE  TRUE 
76.63 23.37  

As we can see from the the summarized table below, our Naive Bayes classifier clearly outperforms a random algorithm.

We are likely to increase our accuracy even more by pre-processing our text data.

A confusion matrix helps us assess how well our algorithm performed. It shows us the prediction for all five countries in contrast to the actual class that is given by our data. For example, we predict 2580 articles as belonging to Great Britain that actually belong to Great Britain. However, we also predict 39 articles as British articles while they are actually French. Overall, when we look at the diagonal, we see that most predictions correctly classify the articles and that our algorithm performs well.

actual_class <- docvars(dfmat_matched, "country")
predicted_class <- predict(model.NB, newdata = dfmat_matched)
tab_class <- table(actual_class, predicted_class)
tab_class

             predicted_class
actual_class   br   fr   gb   jp   us
          br 2580    6   50    6   37
          fr   39 2697  124   11   86
          gb   24   52 6072   24  282
          jp   33    9   31 1945  119
          us   56   32  194   66 4967

We store our confusion matrix in an object because we need it later to visualize the results.

confusion <- confusionMatrix(tab_class, mode = "everything")

Confusion Matrix and Statistics

            predicted_class
actual_class   br   fr   gb   jp   us
          br 2580    6   50    6   37
          fr   39 2697  124   11   86
          gb   24   52 6072   24  282
          jp   33    9   31 1945  119
          us   56   32  194   66 4967

Overall Statistics
                                          
               Accuracy : 0.9344          
                 95% CI : (0.9309, 0.9379)
    No Information Rate : 0.3311          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.914           
                                          
 Mcnemar's Test P-Value : < 2.2e-16       

Statistics by Class:

                     Class: br Class: fr Class: gb Class: jp Class: us
Sensitivity             0.9444    0.9646    0.9383   0.94786    0.9046
Specificity             0.9941    0.9845    0.9708   0.98902    0.9752
Pos Pred Value          0.9630    0.9121    0.9408   0.91015    0.9345
Neg Pred Value          0.9910    0.9940    0.9695   0.99385    0.9632
Precision               0.9630    0.9121    0.9408   0.91015    0.9345
Recall                  0.9444    0.9646    0.9383   0.94786    0.9046
F1                      0.9536    0.9376    0.9396   0.92862    0.9193
Prevalence              0.1398    0.1431    0.3311   0.10500    0.2810
Detection Rate          0.1320    0.1380    0.3107   0.09953    0.2542
Detection Prevalence    0.1371    0.1513    0.3303   0.10935    0.2720
Balanced Accuracy       0.9692    0.9745    0.9546   0.96844    0.9399

To display our confusion matrix visually, we could either produce a heatmap or a confusion matrix.

We first plot a heatmap using the code below.

# Save confusion matrix as data frame
confusion.data <- as.data.frame(confusion[["table"]])

# Reverse the order
level_order_y <-
  factor(confusion.data$actual_class,
         level = c('us', 'jp', 'gb', 'fr', 'br'))

ggplot(confusion.data,
       aes(x = predicted_class, y = level_order_y, fill = Freq)) +
  xlab("Predicted class") +
  ylab("Actual class") +
  geom_tile() + theme_bw() + coord_equal() +
  scale_fill_distiller(palette = "Blues", direction = 1) +
  scale_x_discrete(labels = c("Brazil", "France", "Great \n Britain", "Japan", "USA")) +
  scale_y_discrete(labels = c("USA", "Japan", "Great \n Britain", "France", "Brazil"))

Figure 4: Contingency table

To plot the following confusion matrix, we need to slightly adjust the code from this post on data visualization by Richard Traunmüller.

# Generate a data matrix from the confusion table
dat <- data.matrix(confusion$table)

# Change order of column names
order.columns <- c(5, 4, 3, 2, 1) 
dat <- dat[order.columns,]

par(mgp = c(1.5, .3, 0))

# Plot
plot(
  0,
  0,
  # Type of plotting symbol
  pch = "",
  # Range of x-axis
  xlim = c(0.5, 5.5),
  # Range of y-axis
  ylim = c(0.5, 6.5),
  # Suppresses both x and y axes
  axes = FALSE,
  # Label of x-axis
  xlab = "Predicted class",
  # Label of y-axis
  ylab = "Actual class",
)

# Write a for-loop that adds the bubbles to the plot
for (i in 1:dim(dat)[1]) {
  symbols(
    c(1:dim(dat)[2]),
    rep(i, dim(dat)[2]),
    circle = sqrt(dat[i,] / 9000 / pi),
    add = TRUE,
    inches = FALSE,
    fg = brewer.pal(sqrt(dat[i,] / 9000 / pi), "Blues"),
    bg = brewer.pal(sqrt(dat[i,] / 9000 / pi), "Blues")
  )
}

axis(
  1,
  col = "white",
  col.axis = "black",
  at = c(1:5),
  label = colnames(dat)
)
axis(
  2,
  at = c(1:5),
  label = rownames(dat),
  las = 1,
  col.axis = "black",
  col = "white"
)

# Add numbers to plot
for (i in 1:5) {
  text(c(1:5), rep(i, 5), dat[i,], cex = 0.8)
}

Figure 5: Contingency table

Both figures show us that our prediction performs well for all countries but it performs particularly well for the USA and Great Britain. Darker colors show a higher frequency in both plots, the contingency table also indicates a greater frequency with the size of the bubbles.

Unknown categories: Unsupervised machine learning - Latent semantic analysis (LSA)

The next section addresses how to analyze texts with unknown categories. Latent Semantic Analysis (LSA) evaluates documents and seeks to find the underlying meaning or concept of these documents. If each word only had one meaning, LSA would have an easy job. However, oftentimes, words are ambiguous, have multiple meanings or are synonyms. One example from our corpus is “may” - it could be a verb, a noun for a month, or a name. To overcome this problem, LSA essentially compares how often words appear together in one document and then compares this across all other documents. By grouping words with other words, we try to identify those words that are semantically related and eventually also get the true meaning of ambiguous words. More technically, LSA is a useful technique for aligning feature distributions to an n-dimensional space. This is achieved via singular value decomposition (SVD). This decomposition allows us to decompose both a quadratic and a rectangular matrix. LSA can (among other things) be used to compare similarity of documents/documents grouped by some variable.

What are the major assumptions and simplifications that LSA has?

1. Documents are non-positional (“bag of words”). The “bag of words” approach assumes that the order of the words does not matter. What matters is only the frequency of the single words.
2. Concepts are understood as patterns of words where certain words often go together in similar documents.
3. Words only have one meaning given the contexts surrounding the patterns of words.

For the next example, we go back to the UN General Assembly speech data set.

corpus.un.sample <- corpus_sample(mycorpus, size = 500)

We again follow the cleaning steps described above.

# Create tokens
token_sample <-
  tokens(
    split_hyphens = TRUE,
    corpus.un.sample,
    remove_numbers = TRUE,
    remove_punct = TRUE,
    remove_symbols = TRUE,
    remove_url = TRUE,
    include_docvars = TRUE
  )

# Clean tokens created by OCR
token_ungd_sample <- tokens_select(
  token_sample,
  c("[\\d-]", "[[:punct:]]", "^.{1,2}$"),
  selection = "remove",
  valuetype = "regex",
  verbose = TRUE
)

dfmat <- dfm(token_ungd_sample,
             tolower = TRUE,
             stem = TRUE,
             remove = stopwords("english")
             )

In an earlier version of this post, the function textmodel_lsa was already implemented in quanteda. It is now part of the package quanteda.textmodels. After loading the package, we can run the textmodel_lsa command.

# Load the package
library(quanteda.textmodels)

# Run the textmodel_lsa command
mylsa <- quanteda.textmodels::textmodel_lsa(dfmat, nd = 10)

One interesting question would be: How similar are the USA and Russia? Each dot represents a country-year observation. The USA are colored blue, Russia is colored red, and all other countries are grey.

# We need the "stringr" package for the following command
sources <-
  str_remove_all(rownames(mylsa$docs), "[0-9///'._txt]") 

sources.color <- rep("gray", times = length(sources))
sources.color[sources %in% "USA"] <- "blue"
sources.color[sources %in% "RUS"] <- "red"

plot(
  mylsa$docs[, 4:5],
  col = alpha(sources.color, 0.3),
  pch = 19,
  xlab = "Dimension 4",
  ylab = "Dimension 5",
  main = "LSA dimensions by subcorpus"
)

Figure 6: Distribution of PA topics in the UN General Debate corpus

On Dimension 5 we do not really observe a difference between documents from the US and Russia while we do see a topical divide on Dimension 4.

# create an LSA space; return its truncated representation in the low-rank space
tmod <- quanteda.textmodels::textmodel_lsa(dfmat[1:10, ])

# matrix in low_rank LSA space
tmod$matrix_low_rank[, 1:5]

                        forti         third session general assembl
GNQ_43_1988.txt  4.000000e+00  6.000000e+00      10      12       9
UZB_64_2009.txt -6.306067e-14 -1.748601e-15       2       1       1
PRT_50_1995.txt  2.000000e+00 -2.962543e-13       5       8       4
VCT_70_2015.txt -4.979003e-14 -1.879096e-13       3       1       6
GUY_67_2012.txt -7.959952e-14 -1.302430e-14       3       6       2
MDG_34_1979.txt -1.627830e-13  7.000000e+00      14      13       6
CHL_60_2005.txt -1.064461e-13  2.000000e+00       5       3       4
GTM_68_2013.txt -1.319652e-13  5.002986e-14       6       7      11
GIN_64_2009.txt -1.256287e-13  1.000000e+00       5       5       2
LBN_59_2004.txt -9.672818e-14  1.000000e+00       2       4       5

We now fold the queries into the space generated by dfmat[1:10,] and return its truncated versions of its representation in the new low-rank space. For more information on this, see Deerwester et al. (1990) and Rosario (2000).

pred <- predict(tmod, newdata = dfmat[1:10, ])
pred$docs_newspace

10 x 10 Matrix of class "dgeMatrix"
                      [,1]        [,2]        [,3]        [,4]         [,5]        [,6]
GNQ_43_1988.txt -0.3587033  0.04626331 -0.40268736  0.75368464 -0.198343099  0.15890426
UZB_64_2009.txt -0.1304653  0.04983523 -0.06010142 -0.07259054  0.057245087 -0.03112114
PRT_50_1995.txt -0.5120486  0.45389699  0.69697128  0.03247785 -0.071841805  0.18464906
VCT_70_2015.txt -0.2241820  0.09965473 -0.36666157 -0.45680821 -0.462552853  0.28818827
GUY_67_2012.txt -0.2905706  0.05927260 -0.29307648 -0.35382563  0.366086659  0.47269539
MDG_34_1979.txt -0.4919563 -0.83849245  0.20130813 -0.06146108 -0.038858129 -0.05927530
CHL_60_2005.txt -0.2663441  0.20190586 -0.13824724 -0.23099427 -0.006964881 -0.67210030
GTM_68_2013.txt -0.2074862  0.05127196 -0.13858784 -0.11930716  0.021618854 -0.33022555
GIN_64_2009.txt -0.2738299  0.12184387 -0.19025526  0.13644592  0.659835350 -0.13044787
LBN_59_2004.txt -0.1625876  0.12082658 -0.11439537  0.04220718 -0.408458883 -0.22780319
                       [,7]        [,8]         [,9]        [,10]
GNQ_43_1988.txt  0.13239767  0.10535490  0.203828010 -0.065029344
UZB_64_2009.txt -0.21573385 -0.08282906 -0.130474714 -0.947070625
PRT_50_1995.txt  0.07183401  0.00858728 -0.005558773  0.020944555
VCT_70_2015.txt  0.35837271 -0.38370263 -0.152115117  0.029859727
GUY_67_2012.txt -0.34813601  0.38234056  0.254633069  0.106243837
MDG_34_1979.txt -0.01837620 -0.06738449  0.012211891  0.023580112
CHL_60_2005.txt  0.04450401 -0.07554420  0.596888043  0.009695998
GTM_68_2013.txt  0.35028911  0.66063039 -0.502174774 -0.007009133
GIN_64_2009.txt  0.03757466 -0.49043679 -0.361551806  0.174062011
LBN_59_2004.txt -0.74478679 -0.03668237 -0.337785293  0.234975826

Unknown categories: Unsupervised machine learning - Latent Dirichlet Allocation (LDA)

Both Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) belong to topic modelling. Topic models find patterns of words that appear together and group them into topics. The researcher decides on the number of topics and the algorithms then discover the main topics of the texts without prior information, training sets or human annotations.

LDA is a Bayesian mixture model for discrete data where topics are assumed to be uncorrelated. It is a model that describes how the documents in a dataset were created. We assign an arbitrary number of topics (K) where each topic is a distribution over a fixed vocabulary. Each document is considered as a collection of words, one for each of K topics. It also follows the “bag of words” approach that considers each word in a document separately.

To calculate the LDA models, we need to load the package topicmodels. If you use this package for the first time on your machine, you need to execute a specific sequence of commands, detailed in the code below.

# 1) Install GSL
# We first need to make sure that GSL (e.g. 'brew install gsl' in the terminal) is installed on our machine

# 2) Install gsl
# Then we proceed and choose either of the following commands:
# A)
install.packages("gsl")
# or B)
install.packages(
  "https://cran.rstudio.com/src/contrib/gsl_2.1-6.tar.gz",
  repos = NULL,
  method = "libcurl"
)

# 3) Load the gsl package
library(gsl)

# 4) Install topicmodels
# Here we choose again either of the following commands:
# A)
install.packages("topicmodels")
# or B)
install.packages(
  "https://cran.r-project.org/src/contrib/topicmodels_0.2-8.tar.gz",
  repos = NULL,
  method = "libcurl"
)

# 5) Load the topicmodels package
library(topicmodels)

# Now you are all set for the following models.

For the LDA, we again first trim our DFM. As above, the command dfm_trim trimms the text. This allows us to filter words that appear less than 7.5% and more than 90%.

mydfm.un.trim <-
  dfm_trim(
    mydfm,
    min_docfreq = 0.075,
    # min 7.5%
    max_docfreq = 0.90,
    # max 90%
    docfreq_type = "prop"
  ) 

We then assign an arbitrary topic number and convert the trimmed DFM to a topicmodels object.

# Assign an arbitrary number of topics
topic.count <- 15

# Convert the trimmed DFM to a topicmodels object
dfm2topicmodels <- convert(mydfm.un.trim, to = "topicmodels")

Eventually, we can calculate the LDA model with quanteda’s LDA() command.

lda.model <- LDA(dfm2topicmodels, topic.count)

lda.model

A LDA_VEM topic model with 15 topics.

as.data.frame(terms(lda.model, 6))

How similar are the fifteen topics? This question is particularly interesting because it allows us to (possibly) cluster homogeneous topics. To get a better idea of our LDA model and about the similarity among the different topics, we can plot our results using the following chunck of code. dist and hclust are standard R commands that allow us to calculate the similarity.

lda.similarity <- as.data.frame(lda.model@beta) %>%
  scale() %>%
  dist(method = "euclidean") %>%
  hclust(method = "ward.D2")

par(mar = c(0, 4, 4, 2))
plot(lda.similarity,
     main = "LDA topic similarity by features",
     xlab = "",
     sub = "")

Figure 7: LDA topic similarity by features

The plot is called dendogram and visualizes a hierarchial clustering. The x-axis gives you the topics and the clusters of these topics. Put differently, it gives you information on the smilarity of the topics. On the y-axis, we see the dissmilarity (or distance) between our fifteen topics.

Unknown categories: Unsupervised machine learning - Structural topic models (STM)

The structural topic models (STM) are a popular extension of the standard LDA models. The STM allows to include metadata (the information about each document) into the topicmodel and it offers an alternative initialization mechanism (“Spectral”). For STMs, the covariates can be used in priors. The stm vignette provides a good overview how to use a STM. The package includes estimation algorithms and tools for every stage of the workflow. A particularly large emphasis is on a number of diagnostic functions that are integrated into the R package.

The package stiminsights is very useful for visual exploration. It allows the user to process interactive validation, interpretation and visualization of one or several Structural Topic Models (stm).

We again trim our dfm with the command dfm_trim.

mydfm.un.trim <-
  dfm_trim(
    mydfm,
    min_docfreq = 0.075,
    # min 7.5%
    max_docfreq = 0.90,
    # max 90%
    docfreq_type = "prop"
  ) 

We then assign the number of topics arbitrarily.

topic.count <- 25 # Assigns the number of topics

And eventually convert the DFM (with convert()) and calculate the STM (with stm()).

# Calculate the STM 
dfm2stm <- convert(mydfm.un.trim, to = "stm")

model.stm <- stm(
  dfm2stm$documents,
  dfm2stm$vocab,
  K = topic.count,
  data = dfm2stm$meta,
  init.type = "Spectral"
)

To get a first insight, we print the terms that appear in each topic.

as.data.frame(t(labelTopics(model.stm, n = 10)$prob))

The following plot allows us to intuitively get information on the share of the different topics at the overall corpus.

plot(
  model.stm,
  type = "summary",
  text.cex = 0.5,
  main = "STM topic shares",
  xlab = "Share estimation"
)

Figure 8: STM topic shares

Using the package stm, we can now visualize the different words of a topic with a wordcloud. Since topic 4 has the highest share, we use it for the next visualization. The location of the words is randomized and changes each time we plot the wordcloud while the size of the words is relative to their frequency and remains the same.

stm::cloud(model.stm,
           topic = 4,
           scale = c(2.25, .5))

Figure 9: Wordcloud with stm

If we want, we can also put several different topics in visually perspective using the following lines of code:

plot(model.stm,
     type = "perspectives",
     topics = c(4, 5),
     main = "Putting two different topics in perspective")

Figure 10: Wordcloud using stm – Perspective plots

The perspective plot visualizes the combination of two topics (here topic 4 and topic 5). The size of the words is again relative to their frequency (within the combination of the two topics). The x-axis shows the dregree that specific words align with Topic 4 or Topic 5. Global is closely aligned with Topic 4 whereas commit is more central in both topics.