Topic Modeling with Newspaper Archives
Marijn Koolen
Koninklijke Bibliotheek, The Hague, 19/01/2015
Programme
- What are topic models?
- Why use topic models?
- How do topic models work?
- Working with topic models
Part 1: What Are Topic Models?
a general overview
Topic Models
- Representing topics in collection of documents
- Use statistics to find topics represented by groups of words
- Document is a mix of topics
- Topic is a mix of words
- Documents and words can be directly observed
- topics are latent
Illustration 1
Illustration 2
Illustration 3
Assumptions
- Two documents with the same topics will have overlap in words
- not literally true, but probabilistically true
- Single document can consist of many topics
- but to different degrees
- Three elements: words, topics, documents
- topics are formed by a selection of words
- documents are formed by a selection of topics
Statistical Modeling
- Given a collection of documents (text or otherwise), the modeling process does two things:
- create word probability distribution for topics
- create topic probability distribution for documents
- Both are purely based on frequency and co-occurrence of words
Probability and Intuition
Wikipedia: Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: “dog” and “bone” will appear more often in documents about dogs, “cat” and “meow” will appear in documents about cats, and “the” and “is” will appear equally in both. A document typically concerns multiple topics in different proportions; thus, in a document that is 10% about cats and 90% about dogs, there would probably be about 9 times more dog words than cat words.
Topic Modelling Software
- Mallet:
- popular in Digital Humanities community
- topic modelling, classification, information extraction
- topic-modeling-tool: GUI for Mallet, lacks many features
- Other options:
- GenSim (Python library)
- Stanford Topic Modeling Toolbox
Part 2: Why Use Topic Models?
relevance for research
Suggestive Patterns
- Overcomes problems of keyword search
- search with whole dictionary
- but words weighted by topical importance
- Gives insight in topical nature of collection
- Advantages:
- Great for finding suggestive patterns
- Disadvantages:
- Topics can be hard to interpret
Topic Modelling in Digital Humanities
- Popular in Digital Humanities, especially in Historical Sciences
- each DH conf. has sessions and workshops on topic modelling
- Cameron Blevins: Martha Ballard’s diary
- Robert K. Nelson: Mining the Dispatch
- Matthew Jockers: blog
Sub-Topic Modelling
- Tangherlini & Leonard use sub-topic modelling (STM)
- Use sub-corpus topics to 'trawl' in larger corpus
- generate topics on sub-corpus
- score docs in larger corpus
- more focus and control on topics
Experimenting with STM
- Tangherlini & Leonard (2013) ran 3 experiments
- model topics on Darwin's books, look for topics in Danish literature
- model topics on genre subset, look for unknown works of that genres
- model topics on folklore, look for influences in other literature
Topic Modelling In Research
- It’s only small aspect of larger methodology
- Other steps:
- selecting and gathering data
- preparing data
- using models to find patterns
- close reading to understand patterns
KB Newspaper Archive
- Over 80 million articles
- organised via formal metadata
- date, newspaper title, article type
- How organised in terms of topics?
- for intance, topics during the Interbellum?
Examples with KB Newspaper Archive
- Sampled collection:
- 100,000 articles containing: groote oorlog OR wereldoorlog OR europeesche oorlog OR 1914-1918
- Constrained to period 1918-1940
- Topics modelled by Mallet:
Researcher’s Responsibility
- If you use results of topic modelling, should investigate these questions:
- What data was used as input (where from)?
- How was the data prepared for use?
- Which software was used?
- What parameters were used for modelling?
- And ultimately, how do these aspects affect interpretation of the models and their application?
- This is part of methodology!
Familiarity with Corpus
- understanding topic requires understanding the corpus
- modelling topics in British history texts is hard if you know little about British history
- inexact science: checking usefulness of topics requires corpus knowledge and creativity
Lies, Damned Lies and Statistics
- Many pretty pictures based on topic modelling
- what do they mean?
- Word distributions often seem incoherent
- yet most informative topics often perform badly
- limited use as evidence, great for discovery (Ramsay)
- example of labelling topics
Mixing Languages
- E.g. non-English texts in mostly English corpus
- models language instead of topics
- Three topics modelled on 64,000 song lyrics:
- baby like come oh yeah let know gonna m go never get one na re hey love ll wanna man
- get like baby know let go ll got gonna love back girl feel away want oh gotta time take hey
- na que de y like la m get el te re tu en mi ang yo un ya sa es
Beyond Text
- In fact, you can use other data points than words
- Ben Schmidt uses lat/long coordinates of Whaling ships to model topics
- Topic model of coordinates can be plotted on a map for easy inspection
Part 3: How Do Topic Models Work?
the technical details
Two Parts of Technicalities
- Lots of statistics
- don’t panic, we’ll only briefly touch upon this!
- Lots of transformations
- most of these steps are easy to understand
- but also important to understand
- need to be aware of them to gain control
- Do experiments to get feel for what’s meaningful
Computational Thinking
- Think through task in computational steps
- breaking down complex problems
Transformations
- Four major steps:
- Text > preprocessing > Words
- Words > indexing > Numbers
- Numbers > modeling > Topics
- Topics > analysing > Compositions
- We’ll encounter specifc transformations on the way
1. Preprocessing Text
- What do the newspaper articles look like?
2. Indexing Text
- From words to vectors to indexes
2a. Text as Vectors
- text is linear sequence of words
- can be represented as
- or as a vector of words
- ‘Easy’ to see which texts have overlap in words
2b. Inverted Index
- Term-document index:
- word lists which texts it appears in
- index becomes rows of text vectors
- inverted index
- interesting aside: search engine use this for quick lookup!
Parsing & Units of Data
- Usually words as units
- can be anything, but features need high enough frequency of units
- trigrams and longer phrases often too sparse
- Bag of words:
- ignores word order, syntax, sentence or paragraph boundaries
- same with other kinds of data (colours, objects, melodies)
Stopwords
- Function words and other frequent words carry little meaning in modelled topics
- dominate the inverted index
- remove them to focus on meaningful terms
- But which words are stopwords?
- standard list Snowball Dutch stopword list
- domain dependent: make your own
3. Modeling
- Topics are latent
- Reduce high-dimensional term vector space to low-dimensional 'latent' topic space
- Topics represented by prob. dist. over words
- Texts represented by prob. dist. over topics
- Established models:
- LSA: Latent Semantic Analysis
- LDA: Latent Dirichlet Allocation
- Mallet uses LDA
Semantic Relatedness
- Two words co-occurring in a text
- signal that they are related
- document frequency determines strength of signal
- co-occurrence index
- Easy to see which words are related
Frequency vs. Importance
- How can statistics help identify important words?
- TF-IDF
- TF: Term Frequency
- terms more frequently in document are more important
- IDF: Inverted Document Frequency
- terms in fewer documents are more specific
- TF * IDF indicates importance of term relative to the document
Mallet & LDA
- Mallet uses Latent Dirichlet Allocation
- Diri-what?
- generative model
- iterative sampling to establish topics, word-topic dist. and topic-document dist.
- After so many iterations, distributions are stable
- done: topics are modelled
Considerations 1/2
- Number of documents:
- at least 1000, preferably more
- Selection of documents:
- type dimension: which article types? which newspaper(s)?
- topical dimension: filtered by keywords?
- temporal dimension: beware language change!
- geographical dimension: beware language differences!
Considerations 2/2
- Number of topics
- depends on number of documents
- below 10,000 documents: 20-100 topics
- 10,000 and more: 100-500 topics
- Domain specific stopwords
- Units of analysis: words, n-grams, phrases
Part 4: Working with Topic Models
hands on experience
Exercises
- Using Mallet
- making topic models
- Comparing models
- understanding impact of parameters
- Analysing probability distributions
- exploring topics in collections
Exercise 1: Using Mallet
- download sample articles
- Open a terminal/console/command prompt
- move to mallet directory
- run mallet on KB data
- with/without stopword removal
- inspect topic models
- inspect document-topic composition
Exercise 2: Comparing Models
- Compare variant models to understand impact of parameters: what makes one variant better than another?
- few or many stopwords
- few or many documents
- articles vs. paragraphs
- Pick your favourite and add meaningful labels to topics
Exercise 3: Analyse Probabilities
- Take one topic, compare topic probabilities over time
- Aggregating data over time units
- Take one topic, compare topic probabilities across newspapers
- Take one year, compare topic probabilities across topics
Further reading:
- Several tutorials by Scott Weingart:
- The Digital Humanities Contribution to Topic Modeling
- Topic Modeling for Humanists: A Guided Tour with links to more than you want to know
- Topic Modeling and Network Analysis
- More detail: On Topic Modeling
- Jockers - The LDA buffet is now open
- Topic Modeling in the Humanities: an Overview
Wrap Up
- What, why and how of topic modelling
- Good way to explore data sets, overcome keyword search limitations
- only a small step in research methodology