Don't Forget the Good with the Bad and the Ugly: Topic Modeling a Policing Commission Listening Session

Last updated on May 14, 2021 8 min read police, topic modeling, text analysis

Following the summer of civil unrest in 2020, one capital city in the western US (like many other cities around the country) formed a Racial Equity in Policing Commission. According to the city’s administration, the commission was formed to examine the police department’s policies, culture, and budget. As part of the commission’s activities, in January 2021, it held a public ‘listening session’ in which residents could provide their perspectives on policing in the city.

I listened to approximately 20 minutes of the almost two-hour listening session. During those 20 minutes, I was somewhat surprised by what I heard. Like many other areas in the country, the city in question experienced several riots and hundreds of protests in the latter half of 2020. Further, there has been sustained attention to the profession of policing, both locally and nationally, with a large portion of the surrounding rhetoric being exceptionally negative. Negative rhetoric regarding policing has not been uncommon within the commission. However, in the short 20 minutes I listened to, more than half of the comments were generally positive about police and asking for an increased police presence.

Upon waking in the morning and looking at the news online, I saw the headline reporting the listening session: Common threads emerge in public’s comments. What were those “common threads” that the news agency decided to report? They were 1) accountability, 2) crisis intervention and disability training, and 3) consistency in policing. These were not the mainline topics that I walked away with the night before. Specifically, I noticed the lack of acknowledgment of the positive comments I had heard. However, I realized perhaps I simply listened to an anomalous portion of the listening session. Before jumping to conclusions about what was reported, I should empirically test whether or not the news agency did indeed correctly identify the “common threads” within the listening session. Below are the results.

Getting the Data

The first thing I needed to do was obtain all of the comments made during the listening session. Luckily, the commission’s meetings are public and posted on YouTube. I submitted the listening session video to an online transcription service for a nominal fee, and within 20 minutes, I had the transcript for the listening session.

Analyzing the Data

For the particular question at hand (the common themes expressed in the listening session), topic modeling is ideal. I prefer using structural topic modeling (STM) for several reasons I do not take the time to explain in this post. If you are interested in learning more about STM, you can read more about it in this paper I co-authored with Ian T. Adams. In a nutshell, STM identifies underlying latent variables through an unsupervised machine learning process in which themes are inferred from the distribution of words across topics. The topics are generated inductively from the data, utilizing a probabilistic approximation of Bayesian inference. In essence, STM counts words, determines the probability of words appearing with other words within and across documents (i.e., comments), and estimates the best fit for latent themes based upon those probabilities.

Descriptives

A total of 22 individuals gave comments during the listening session, although it was reported over 1,000 members of the public attended the virtual meeting. The shortest statement consisted of 130 words, with the most extended comment composed of 895 words. The median comment was 284 words long, with a mean of 375 words.

Determining the Number of Topics

A data driven-driven approach was used to determine the number of appropriate topics. Measures of the held-out likelihood, residuals analysis, and semantic coherence were examined for models ranging from 2 to 10 topics. Based on these measures, the models containing 2 to 4 topics were retained. While these measures are efficient and interpretable, they cannot replace human judgment. Accordingly, I performed a manual examination of semantic coherence and the exclusivity of words to topics within the remaining model, finding the four-topic model to have the best goodness of fit.

Were There Common Threads?

While the news story suggested that there were three common threads, the STM analysis identified four. As can be seen below, while all of the four topics are correlated, they are distinct.

The four topics are relatively evenly dispersed across all transcribed comments (23% - 30%)

And if each comment is placed in a discrete topic category based on the percentage of topic it encompasses, there is a fairly even distribution.

In sum, we can say with a reasonably high degree of certainty that there were four distinct themes among the comments given in the listening session.

What were the Themes?

Now we turn to the heart of the question: Do the themes identified in the news story align with the analysis? To answer this question, I manually explore the four estimated topics. This was accomplished by reviewing comments that were estimated to be highly associated with each topic. Through several close readings of the comments, an underlying theme for each topic was identified. Below I briefly describe each theme and provide comment excerpts demonstrating that theme.

Topic 1 — Police Procedures

Comments highly associated with Topic 1 appear to mostly convey concerns with police procedures:

I’m sure that there is a technical legal explanation for why it might be okay to be taking these items from the homeless folks. But as a civilian, it’s hard for me to understand…that sort of a policy…Do you have any documentation that you’re returning the gear to them?

[The PD] should also ensure equitable language access, including alternative means of communication for all who encounter police or enter the criminal justice system.

I’d like to see that manual updated to reflect this new [hate crime] legislation…for the benefit of all.

Topic 2 — Appreciation

Comments highly associated with Topic 2 appear either to 1) call for people to be more gracious and understanding with officers, 2) praise the work of officers, or 3) call for more police:

Before we point fingers, we should perhaps open our ears and listen to what our police officers have to go through first.

The female supervisor of internal affairs…I have a very positive message about them…she seemed to be…willing to listen to me.

A great concern with the [PD] is the reduction of staffing numbers. We’ve seen this year…a mass exodus. If we don’t understand why this happens, then we can only expect this problem to repeat itself at another time, an extremely difficult time in history when we need our public safety servants the most.

Topic 3 — Inconsistency

Comments highly associated with Topic 3 appear to mostly express concern with inconsistency in police response (or the broader criminal justice system):

There were two incidents that happened, but one was handled differently, and it affected me.

We need to be able to depend upon our police force to be there for us and help us. We don’t need to be in fear of calling them and wondering, will they show up? Are they going to help or are they going to make matters worse?

If the police are still serving, you know, an oppressive purpose against minorities then a minority being the one serving that group, this wouldn’t help, you know.

Topic 4 — Displeasure

Comments highly associated with Topic 4 appear to mostly be expressing displeasure:

There’s been a lot of issues. People are trying to break in my apartment and, um, I reported so many different times over it…and for a while, things were ignored.

The cop gave me a ticket for following too close…It’s like he was using a power move…it just wasn’t fair at all.

I still do not have a driver’s license and it is because of my fear of the police force.

Our daughter was murdered two years ago…It was carelessly investigated by the police department and he was not even arrested until seven months after the actual murder.

Conclusion

To conclude, I return to the puzzle I set out to answer. Are the themes that the news reported as the “common threads” of the listening session accurate (accountability, crisis intervention and disability training, and consistency in policing)? When one analyzes the transcribed comments systematically and empirically, the answer is a resounding “sort of.” There certainly was a thread of comments addressing consistency in policing. Further, one may interpret some of the topics as having to deal with accountability, which can bleed over into the need for crisis intervention training. However, the news story’s themes were not the overarching themes identified in the above analysis: police procedures, appreciation, inconsistency, and displeasure. Perhaps most noticeable was the exclusion of the ‘appreciation’ theme. It is telling that during a listening session for a commission created to address policing issues (perceived or real), there was still a clear theme calling for understanding officers, thanking officers, and calling for more police/expressing concern about recent turnover. If ‘common threads’ are going to be identified and communicated to the community at large, all of the common threads should be identified and expressed. That is, one should not ignore the good when pointing out the bad and the ugly.

police topic modeling text analysis