MALLET

Can digital tools always make our research more innovative—or sometimes, do they just get in the way?

At DiSC and the Beck Center, we've been mulling over the question of whether the push to "go digital" with a project is always a good thing. Accordingly, we were curious to see what we'd find if we used a bunch of digital text analysis programs, and then we simply read the texts. Would the digital programs offer new insights and save us time? Or would they clutter up an otherwise straighforward textual analysis?

We tested several free, open source tools by using them to analyze  57 sermons given after Lincoln's assassination. The sermons, which are digitized and housed on the Beck Center's website, present the perfect opportunity to experiment with digital analysis.

Did the digital tools pass the test? We arrived at a typical humanities answer: yes and no. You can find detailed results on the project site, which we're calling "Lincoln Logarithms: Finding Meaning in Sermons." The tools we employed— Voyant, Viewshare, PaperMachines, and MALLET—quickly offered us some potential research questions and highlighted places and subjects that we might look at more closely.

Perhaps most enticingly, the programs "read" the 57 sermons (comprised of 1672 pages and 481,575 words) in mere seconds. Reading them was more tedious; in fact, no one on our team made it through all of them. In this case, I think that actually reading the texts—even just a portion of them—was more valuable than looking at the digital tools' output.

Digital tools can help us hone in on what questions to ask. They are a way to help us arrive at questions and results, but they aren't results.

Topic Modeling output for a Zotero collection using Paper Machines

  Share

Paper Machines, the add-on that integrates a range of text analysis tools into Zotero, has generated quite a buzz in the short period of time since its release. For those of us that store notes, citation information, PDFs, and article links in huge Zotero libraries, Paper Machines has the potential to be a game-changer in terms of how we visualize our research.

Because Paper Machines is so new, it's being updated with added functionality every few days. I'll provide step-by step documentation for how to use specific components of Paper Machines in Part II of this post. For now, I'll discuss whether or not Paper Machines might be a good fit for your research, the tools that it offers, and how it might help your work.

Paper Machines provides a broad range of text analysis tools, but it's not meant for everyone's research. You'll probably benefit most from Paper Machines if you:

1.  Already use Zotero to manage your sources. Paper Machines draws on a number of open source tools available elsewhere on the web. If you want to visualize your data but aren't already comfortable using Zotero, you might want to look elsewhere.
2. Have a relatively large or robust Zotero library. At the time of this posting, Paper Machines incorporates the full text of Web snapshots and OCR'd PDF files into its text analysis, as well as the title, place, date, and subcollection of a source. The option to include notes, tags, and links to live websites will be available shortly.
3.  Are collaborating on a Zotero library with a group. Paper Machines is very good at helping you figure out the contents of a collection. If you're working on a collection with multiple group members, it's a quick way to visualize what kinds of material your collaborators are adding.

What kinds of analysis tools does Paper Machines employ?

• A word cloud with the option to filter out commonly used words.
• Phase nets, which allow you to visualize relationships between common words in your text (for example, x and y; x is y)
• A Geoparser, which uses location information to produce beautiful visualizations of the places mentioned in your texts.
• DBpedia Annotation, which produces a visualization of what people, places, and things are mentioned in your texts.
• MALLET-based topic modeling, which generates visualizations based on commonly occurring topics in your texts. The author offers some additional information about information about Paper Machines' use of topic modeling here.

What can Paper Machines help you do?

• Assess the contents of a collection. Looking through the Paper Machines results is a helpful way to get to know the contents of a group library or to get reacquainted with a collection that you haven't used for a while.
• Identify gaps in your material. Reviewing the MALLET output for a specific collection in my Zotero library (canonical works in US history) I noticed a surge in books about women's labor history (which MALLET identified using the terms women, labor, work, and activism) during the 1980s. I also noticed a lack of items in my library about these topics since 2000.
• Compare collections. Analyzing two collections with Paper Machines makes similarities and differences evident. Using topic modeling, for example, I could see what subjects came up most frequently in the two collections and if they coincided. The word cloud function is the easiest way to identify concurrent terms and subjects at a glance.
• Find patterns in your collections. Using the "Phrase Net" function, I conducted an "x is y" analysis on one of my collections. I was surprised to see that "democracy is necessary" and "Cold War is necessary" were recurring phrases a number of sources.

The Geoparser links texts in a Zotero collection to the places that they mention.

Additional examples of Paper Machines visualizations are available on the developer's site. The add-on is available for Firefox and Zotero Standalone, and visualizations  can also be saved as html files. While the occasional error or puzzling result is inevitable early on, the creator of Paper Machines is constantly tweaking the interface in response to feedback from users.

  Share

Beck Center Grad Fellow Sara Palmer tells us how MALLET works and what you can (and cannot) do with Topic Modeling

This past summer I have enjoyed working with a tool called MALLET (MAchine Learning for LanguagE Toolkit) that can generate a series of potential topics in a corpus of texts.  Created by Andrew McCallum at University of Massachussetts, it is an open source toolkit and that is fairly easy to download and install.  While the setup instructions on the website are minimal, Shawn Graham’s has written a helpful overview of how to get started.  

Topic modeling is based on the idea that within a set of related texts, certain words will occur near each other with statistically significant frequency. MALLET works on documents like its acronymic namesake by shattering texts into an array of words.  It then applies a statistical method called Latent Dirichlet Allocation to put related words into clusters. 

These related groups of words, or ‘word bags’ as they are often called, can be interpreted as making up a topic.  For example, the group of words ‘needle,’ ‘stitching,’ and ‘fabric’ could reasonably be labeled as the topic ‘sewing.’  Of course, scholars familiar with a corpus can readily name a list of topics but what if you’re working with a large set of texts and don’t have time to read all of them carefully?  MALLET allows you to quickly obtain a general idea of what topics may be present.  

Perhaps the most prominent example of topic modeling with MALLET is Cameron Blevins’s work on the diary of Martha Ballard, an American midwife who wrote daily entries for 27 years.  Blevins’s topic model identified key themes in the diary such as ‘death,’ ‘shopping’ and ‘gardening’ and graphed their prominence over the course of Ballard’s life.  These charts often corresponded with known biographical details.  For example, the topic ‘emotion’ peaked between 1803 and 1804 when her husband was imprisoned for debt and her son was indicted for fraud.   

The field of DH itself has also been put under the MALLET “macroscope.”  Elijah Meeks’s work visualizes the network of associated ideas in the self-definitions of digital humanists while Matt Jockers analyzes the work performed in the field by culling themes from the blog posts of 117 digital humanists on a single day (March 18, 2010).  The Maryland Institute of Technology in the Humanities (MITH) has an excellent overview of how topic modeling has been used in the humanities.  Other prominent topic modeling projects featured on the MITH blog include Travis Brown’s work on Austen and Byron and Jeff Drouin’s work on Proust. 

Interest in topic modeling has grown at DiSC since it hosted a talk by Robert Nelson from the University of Richmond this past January.  In his research on the American Civil War, Nelson ran MALLET on the issues of Richmond Daily Dispatch newspaper from 1860-1865.  He was able to graph and contextualize topics such as fugitive slave ads and military recruitment, the results of which are illustrated beautifully on the project’s website.  For each topic Nelson has a separate page displaying the graphs of its prevalence over time, the most prominent key words, as well as the articles, ranked by compositional percentage, that most clearly exemplify the topic.  

These two components, keywords and compositional percentage, can be respectively derived from the MALLET output files ‘topic-keys’ and ‘doc-topics.’  At the Beck Center we wanted to see how this output might be used for understanding the journal Southern Changes, published monthly by the Atlanta-based Southern Regional Council from 1979 through 2003.  Comprised of 110 issues containing 978 articles, it is considerably smaller than the data set Nelson worked with but still large enough to make it a good fit for MALLET’s method.  We hoped to identify key terms with which to catalogue the collection and also to generate topical groups of articles that could be featured on the website.  The results were encouraging as I found the themes suggested by the topic keyword lists to roughly correspond with the subjects provided by Allen Tullos, editor of Southern Changes from 1982 through 2003.  Tullos writes:

 “The articles in Southern Changes range across many subjects: racial justice and the freedom struggle, voting rights, educational opportunity, economic democracy, social equality and inclusion, women's rights, environmental justice, critical regional studies, regional-global issues, and popular culture.”

After discovering a ten-topic run of MALLET to be a bit broad, I opted for 20 topics.  In THIS TABLE I have labeled the topic keyword lists with my overall impressions in black.  The generally correspondent Tullos subjects—labeled in green—are included for most of the MALLET topics.  This is, of course, a very rough sketch but it does appear to suggest correlation between MALLET data and expert knowledge.

As a model built on sampling and probability, MALLET naturally does not generate the same exact output each time it is run.  The topics and keywords do vary but are generally similar from one run to the next.  Below is a table comparing the keywords from the ‘Voting Rights’ topic in the previous example to those produced in five additional runs.  The original run was based on 10,000 iterations but for the sake of time, I used 1,000 iterations on the additional trials.  VIEW THOSE RESULTS HERE

While the topic-keys outline the word composition of the topics, the doc-topics indicates the topic composition of each document.  It simply lists, in descending order, the ID number of each topic and the percentage of the document that it makes up.  CLICK HERE for a table of the documents in which ‘Voting Rights’ comprises over 40% of the composition.

From this one can get a clearer picture of what kind of content is picked up by MALLET for the ‘Voting Rights’ topic and see that the topic is most prevalent in the journal around the 1980 and 2000 election cycles. 

In sum, MALLET did what we wanted it to do: we obtained key terms and a general overview of topics present.  My own attempts to graph the topics over time did not illustrate trends with nearly as much clarity as I would have hoped.  However, my objective is not to prove that MALLET is some kind of magic bullet, ripping through a corpus to the heart of its intelligibility.  Rather, I found that the output, when interpreted cautiously, can offer an excellent starting point for more detailed content analysis.