Matt Waite

A rant about teaching journalism in 2026

Matt Waite — Tue, 16 Jun 2026 05:00:00 GMT

I need to get this out of my head.

Trying to figure out what to teach in a journalism class in 2026 feels like being torn apart from the insides. And I can already feel the people thinking It’s Not That Hard, Just Teach The Basics blah blah blah sure. Easy for you to say. You’re either not teaching journalism in 2026 or you’re willfully blind as to what is going on right now.

Faculty are under enormous pressure from university leadership and voices in industry that students must have AI skills when they leave. When asked what those are – what do they mean by AI skills? – the answer we get is Yes. AI skills. The kids need the AI skills. AI. Skills. Yes.

They don’t know.

And they’re quick to tell you that you’re a useless dinosaur if you don’t teach them these magical skills that they can’t describe.

Worse, the journalism world is being fractured into two warring tribes that are disingenuously arguing right past each other. Everyone is screaming, no one is listening, and everyone is full of shit in their own way. Trying to engage with everyone’s arguments on their merits is enough to drive you crazy.

Oversimplifying and being as disingenuous to all as they are to each others arguments, here are the two tribes fighting with each other: The Stochastic Parrot/Plagiarism Machine is Evil And I Will Never Touch It Group vs the Make The Meat Bags Talk To Other Meat Bags And AI Will Lead Us To Glory Group. There’s a third group – the Supernerds Making Micro Tools Only 3 People Need/Understand That Can’t Be Trusted Without Massive Human Oversight Group but they’re not fighting this fight.

Just by writing this, I’m going to get shit from all of them. Oh, you’re actually using AI in your class? Poison! Heretic! Earth/Job killer! Oh, you’re not allowing your students to use AI? Lol dinosaur so out of step and irrelevant! Just quit already! Oh, you’re worried that if you let students use AI you’ll actually make them dumber? Or you’ll be the one that puts them on a path to a delusional spiral? Doomer.

Further frustrating me is that every argument contains danger. Every suggestion carries a risk of actually harming students. Go one way too far and you make them two letter grades dumber than you got them – which, to be clear, is not the gig. Go too far the other way and you hurt their chances of finding a place in a new world that includes an alien brain on call 24/7 that isn’t going to magically go away.

No one knows where the lines are. There is no manual for this. No one knows that you can do This, but Not This. The best we’ve got is Try Some Stuff, See What Happens.

And to be clear, I do think there are useful things that AI can do for journalism. And I’m quite sure there are things that will absolutely harm the profession. More on those in future posts, but I’m struggling to find why we shouldn’t use the thing that enables things we couldn’t do before if it serves the mission.

But here is the problem that keeps me awake at night: How do you teach someone how to do something with AI when they don’t know what success looks like in the first place? Someone who struggles with critical reading. Someone so riven with anxiety about failure they’ll accept anything that appears to help.

My spiral with all of this started with a February column by Chris Quinn, editor at the Cleveland Plain Dealer where he bagged on a recent grad for rejecting their job offer after the kid learned that they’ll be the one feeding AI so it can write stories. The headline: Journalism schools are teaching fear of the future. The kid wanted more out of a journalism career than feeding an AI quotes. Good for them. Not a reason to write a letter from the editor crapping on the kid and every j-school without knowing anything about what we do.

That sparked a lot of conversation online and in company Slacks, and that created even more journalism about journalism.

“It’s bots vs reporters at the AP” according to Semafor. This quote from the story sets the whole thing up:

“Because local newsrooms are so strapped, they are turning for assistance on the news making process in every direction. Advance Publications got there first, others will follow,” AP Senior Product Manager for AI Aimee Rinehart wrote in internal company Slack messages first shared with Semafor, referring to the Plain Dealer’s parent company. “Resistance is futile.”

The thing being “resisted”: the idea that people would go do the reporting and AI would write the story. Or at the very least, write the first version of the story.

Here is where this gets into a giant no-one-is-listening mess for me:

If you’re in the no, never let AI write a story camp, you’re wrong. There are all kinds of stories that this could totally work for. They are not the stories you are thinking of. The stories you aspire to write? AI is bad at them. The stories you don’t want to write, or don’t have to think very much while writing them? They’re not bad at those. Notice I didn’t say good. I said not bad. And they’re really, really fast at them, which has it’s own value.
If you’re in the absolutely yes, we’ll barely have to write anything again camp, you’re wrong. The set of stories this will work on is small, formulaic and low value. I’ve written hundreds of these stories in my career because I was cheap and it needed to be done. The city council will consider an ordinance that does xyz at its Tuesday meeting. Food trucks and puppies in costumes will star in this weekend’s local festival. The Department of Transportation said Jones Street will be closed for the next week as crews work on the sewer. That’s the level we’re talking here.
If you’re in the why are we even thinking about this, we’ve never done something like this before camp, you’re wrong. I’m sure even older examples exist, but earlier this year my former employer, the Tampa Bay Times, took some flak for publishing AI generated real estate stories. Online commenters lamented in all the ways they lament. But I automated Tampa Bay real estate stories … in 2008 … no AI required. It was just a templated thing using public data. No one lamented then. No one feared I was taking a reporter’s job then. See for yourself what it did and ask yourself: would you get enough value out of that if you made a person do it? And yet thousands of people used it regularly.

A long time ago, I once complained to an older reporter working on a Big Story that I wanted to be working on a Big Story and all I was doing were stories that ended sentences with police said. And he told me that to build the Taj Mahal, you’ve got to build a lot of shacks. You don’t go from shack to mansion in one step. And I hated that answer at 22. But he was right.

What AI presents is the industrialization of my metaphorical shack. Push some buttons, poof, shack. Need more shacks? Push more buttons. Dozens of shacks. More content! Just what everyone wants! But if AI could write Pulitzer winners or stories that grab attention in an attention starved world by pushing a few more buttons, Anthropic/OpenAI/Google wouldn’t have free accounts marketed to college kids. The marketing – and pricing – would be very different. The marketing now? Automated volume. More Stuff for the Thing! Volume as a value in and of itself.

The student journalist of 2026 did not grow up reading a print newspaper at home like generations of students before them. That student who walks into my classroom in 2026 often has very little to zero experience with a news product. Journalism faculty at other institutions that I’ve talked to all report the same thing: We have journalism majors who have not and do not read news. Or watch it. There is nothing for them to emulate.

So please, tell me: how am I supposed to prepare a student for a future where AI can write a simple story for them if they just tell it to do that when they do not know what that story looks like. I do. And the reason I can tap a few words into a chat window and get a reasonable facsimile of what I used to do on the night cops desk in the 90s is because I did that. I know in my bones exactly what that looks like and what it takes to get it. An 18-year old kid doesn’t.

Add in that they know they don’t know, and this tool will do something that looks like a story, so the temptation to use it is enormous, and so is the chance of a generation of student journalists learning how to write like AI because of their reliance on it. A bonus dollop? They are, as a group, unable or unwilling to read. Close, careful, critical reading? Exceptionally bad at it, generally speaking. The bulk of them are the worst human for the human in the loop.

How do you make that student labor to build shacks now, knowing a machine will do it for them today and maybe after graduation, without them ever having set foot in a shack? How can I get them to understand the value of shackbuilding – and to not fear but embrace all the mistakes that come with that – all so they might have a chance to build mansions one day?

Teaching a one-hour throwaway news apps class

Matt Waite — Thu, 30 Apr 2026 05:00:00 GMT

What you will need: * A Github Account (if you don’t already have one) * Google Antigravity * Some data in csv format.

Step 1: Setup

On the first page you see, you should see a big blue Open Folder button. Click that.

For most of you, you’re going to need to create a new folder and that new folder must be named [githubusername].github.io where githubusername is your github username that you just created. It must match it exactly. So, for example, my github user name is mattwaite, so my folder is called mattwaite.github.io. It has to match your username and watch the spelling of github. A common mistake is to misspell github in the name.
If you already have a Github Pages repository, go find that folder (that’s your username.github.io folder).

Step 2: If you don’t have one, make a homepage

For people who don’t already have a Github Pages account and have nothing in here, go over to the agent window and write “Make me a nice looking homepage that says Coming Soon. This will be hosted on github pages.”

Step 3: Make a folder for your throwaway web app

Inside of your username.github.io folder, create a folder using Antigravity.

You should name this folder dataname-search where dataname is a one word description of your data. If your data is election data, make it election-search. If it’s Groundhog Day predictions, make it predictions-search. The reason you use lowercase letters and dashes for spaces is because this folder name will be part of the url.

Click and highlight your folder after you have created it.

Step 4: Put your data into the dataname-search folder

That’s it. Just move it into that folder. You can drag the file from your file explorer (Finder on Mac, File Explorer on Windows) into the folder in Antigravity and it will move it.

Step 5: Time to prompt

With your folder created and your csv inside of the folder, it’s time to tell Antigravity what you want. Your prompt should: * Say that you are creating a web search of this data in your dataname-search folder. * That the page is hosted on Github Pages so it must be static files. * That you intend to iframe the page into a Content Management System elsewhere. * What columns you want people to be able to search. * How you want the data to be displayed, correcting any column names that won’t make sense to the general public. For example, if your column is called Jurisdiction_Name, you should tell Gemini you just want that to be Name. ### Step 6: Review and revise

What do you think? Don’t like something? Tell Gemini to change it.

Step 7: Commit and push to Github

Click the Source Control button. In the box that says Messages, say what you are committing. In this case, it’s likely something like Adding dataname search. It does not need to be verbose. It’s a note to yourself what this is so that you can know at a glance later what it is if you were to go diving through your own git commits. When done click commit.

Parsing PDFs with Antigravity

Matt Waite — Mon, 24 Nov 2025 06:00:00 GMT

Among the wisest things I’ve ever seen written about AI is “I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes.”

The same can be said for journalism. I want AI to do the chores so I can do the journalism. Time I’m not manually pulling apart PDFs is time I can spend talking to people.

Last week, Google launched their much anticipated Gemini 3 model, and much is being said and written about it. One very interesting thing they did was launch a Visual Studio Code version of their own called Antigravity. It’s a development environment with an “agentic coding surface” added as one of the primary interfaces to it.

I’ll be honest, when I first read about it, I was pretty meh. So it’s Claude Code (which you can plug into Visual Studio Code!) or Open AI’s Codex, but for Google this time. Okay. Fine. But I started seeing some overreaction to it online, and it made me curious.

I’m ready to say we may be looking at a truly impactful data journalism tool here. I don’t want to fall into the same trap and overreact by saying it’s as big as the spreadsheet or the search engine, but I’m also not saying it isn’t.

In short, it has astonishing potential as a data journalism tool.

What has me so excited? My employer’s salary “database” which is a 1,957 page PDF. A PDF formatted in such a way as to make parsing it a practical impossibility. And before you ask: state law says if they “publish” data in this form, then they don’t have to give it to you in a different form. Many of us have asked.

What makes it so hard are people who get paid out of multiple budget pots. Take, for example, me. I have one job according to the university: professor of practice. I get paid out of one account. My entry in the pdf is one line. Easy. Colleagues of mine might have multiple jobs. Some administrators in my college are half administrators (pot one), half professors (pot two). But they also have endowed positions, so they get paid from a third pot. In the PDF, they’re on four lines. Pots one, two and three and a fourth that is the total. But only the first line gets all of the data. The rest? Blank.

Three people. Three salaries. Eight lines of data.

Notice also that names and where they get paid from blur together. Notice how the length of the alternate funds also overlap the name and position columns. There is no reason this data needs to look like this, but the university considers this as being responsive to a public records request. Want to analyze this data? Want to compare it across time? Compare administrators to faculty? Good luck.

That is, until Gemini 3 and Antigravity came along.

After an afternoon of messing around with Antigravity to fiddle with the design of this website, I decided to just try something. I had been messing with DeepseekOCR, an open weights model that you can run on your own hardware that is very good at finding tables in PDFs and converting those to markdown tables. I was very impressed. But I wondered how well Antigravity/Gemini 3 would do with this pdf.

Answer: Gobsmacked. Gob. Smacked.

I put the PDF and a screenshot of the first page in a folder, connected that folder to Antigravity, and wrote this half-assed prompt from the couch.

I am attempting to extract structured data from a frustratingly formatted PDF. What I need at the end is a csv file that has the data contained in the screenshot. I can handle the intricacies of the data after I get the structured version. Can you take a look at the screenshot first to see if you can extract the tabular structure?

I didn’t even ask it to do the whole PDF. I just wanted the screenshot of the first page. That’s it. What it did was devise an implementation plan, wrote a walkthrough of what it did as it was doing it, then wrote a Python script using pdfplumber that extracted the data out of all 1,957 pages, and then wrote a cleaner script to fix some formatting weirdness. It took my prompt, worked for about 10 minutes and spit out a csv file that was orders of magnitude better than anything I had managed myself in years of on-and-off messing with this file.

All I did was stare at it as it kept trying things and checking them, improving the code using random selections of data to check if it was all working. And then, it finished. I couldn’t believe it worked, so I opened it and was blown away with what it did.

It wasn’t perfect, however. In fact, it assumed that those extra rows where people got paid from other pots were a mistake and it filtered them out.

So I went back to the prompt:

I’ve been doing some of my own spot checking and there is a basic assumption at the beginning that is not correct. That assumption is lines 34-38 of the clean_salary_data.py. Specifically:

# Basic check: First column should be Cost Element (6 digits) # OR sometimes it’s empty if it’s a continuation? No, looking at the data, most data rows have it. # Let’s look at row 26: 512100,“Batman, Renee”,F,…

first_col = row[0].strip()

There are entries where the next line, which does not contain a cost object *is* a continuation. This data is university salary data, and how they show professors with endowed chairs, for example, is to put their faculty job on one line with a cost object, then the next line without one is their endowed chair line and at the bottom is a total line for that person. Not capturing the next few lines is causing some issues with accuracy. Can we capture those? For the vast majority of data, your method works extraordinarily well. It’s just not working for the few who have multiple salary inputs.

About 8 minutes later, I had an astonishingly good – much better though still not perfect - version of this data.

What’s wrong with it?

It didn’t want to make assumptions about spacing, so it left odd spacing that is an artifact of the PDF. So some people are Ma tt Waite or Profe ssor. The overlapping columns are an issue I’m likely going to have to contend with manually. I’m going to have to fill in blank columns and total up people to get to one row one person.

The output from Antigravity.

But I can’t stress this enough: this is light years beyond any tool I’ve been able to throw at this in years of trying. Every NICAR, I throw some new tool at it and leave disappointed. This is the first time my gob has been smacked by something an AI is doing.

Could I have written this code? Sure. I’ve even tried using pdfplumber to do it and didn’t have the same results. It would have taken me much longer, and frankly that’s probably enough to get me to go away. I’ve got papers to grade and students upset with me about how long it’s taking.

Can’t say this enough: astonished at what this might mean for freeing journalists up to do journalism instead of un-screwing up government PDFs.

Want to see all of the output? It’s on GitHub.

An R + LLM starter kit

Matt Waite — Fri, 07 Mar 2025 06:00:00 GMT

I’ve written before, I am at best an enthusiastic amateur when it comes to AI, LLMs and R. But I’m braver/dumber than most, so for a talk I’m giving to NE-RUG – the Nebraska R Users Group – and to the NICAR data journalism conference, I’ve got some resources and some code to share.

R libraries

ellmer: From the folks who brought you the tidyverse comes ellmer, a library that supports talking to a large number of LLMs. To talk to the big commercial LLMs, you’ll need API keys, and that usually means having an API budget. But what I like about ellmer is that it also talks to locally hosted models as well. More about that later.

chores: Built on top of ellmer, chores is a neat way to make tools inside of R Studio that leverage LLMs to accomplish tasks. Examples are to help with certain kinds of code tasks, or help with explanations of what an error message means.

External resources

Ollama: A cross-platform system of downloading, managing and running open-source LLMs on your local machine. With Ollama, you can run Meta’s Llama3 or DeepSeek’s R1 locally, using them to accomplish tasks without incurring costs. A rough rule of thumb is that you can run models slightly smaller than the amount of RAM you have. For example, my computer has 16GB of RAM, so I can run 14 billion parameter models (albeit somewhat slowly).

Basics of ellmer

ellmer at first is very simple. You can just start a chat and … chat.

library(ellmer)
library(tidyverse)
library(glue)

chat <- chat_gemini()

chat$chat("Tell me three jokes about journalists")

Here are three jokes about journalists:

1.  What's a journalist's favorite kind of fish?
    A **red herring**!

2.  Why did the journalist break up with the thesaurus?
    It kept giving him too many synonyms for "scoop"!

3.  A journalist dies and goes to heaven. St. Peter meets him at the gates. 
"Welcome!" says St. Peter. "Before you come in, I just have one question for 
you."
    The journalist immediately pulls out his notebook and pen, eyes gleaming. 
"Fantastic!" he replies. "My first question for *you* is..."

Structured data output

Most models can be used to extract structure from sentences.

chat <- chat_gemini()

data <- chat$extract_data(
  "My name is Matt and I'm a 49 year old Journalism major",
  type = type_object(
    age = type_number(),
    name = type_string(),
    major = type_string()
  )
)

as.data.frame(data)

  age name      major
1  49 Matt Journalism

Something more like data journalism

What if we wanted to normalize some messy, messy data. In Nebraska, the Department of Corrections publishes a live dataset of incarcerated people that you can download. You can ask and answer all kinds of questions from it – demographics, etc. But what you can’t do is figure out what is holding the most people, or how many people are there at least in part because of drug charges. Why? Because the charges they are serving time for are not normalized. Here’s an example of what they look like:

POS CNTRL SUB-METHAMPHETAMINE
MURDER 1ST DEGREE
POSSESSION OF METHAMPHETAMINE
POS CNTRL SUB (METH)
POSSESS CONTR SUBSTANCE-METH
DELIVERY OF METHAMPHETAMINE
POSS W/ INTENT DIST METH

Can an LLM help us with this? Yes!*

chat <- chat_gemini()

chat$extract_data(
  "POSS W/ INTENT DIST METH",
  type = type_object(
    methamphetamine_related = type_boolean(),
    drug_possession_related = type_boolean(),
    drug_distribution_related = type_boolean(),
    fully_spelled_out_no_abbreviations = type_string()
  )
)

$methamphetamine_related
[1] TRUE

$drug_possession_related
[1] TRUE

$drug_distribution_related
[1] TRUE

$fully_spelled_out_no_abbreviations
[1] "Possession with intent to distribute methamphetamine"

* you have to ruthlessly check this. It’s good, but it is not perfect.

Why am I only doing one here? Because the free tier of Gemini limits you to about 5 queries a minute. But it wouldn’t be that hard to write a function that runs through your list of charges and inserts a pause after each one to ensure you stay in the free tier.

What else can be done?

Nuclear powered population maps

Here is a basic population map of Nebraska with one-year change values in it. We’ve all made this chart before. It’s simple, but it gets the job done.

I’ve written a lot more about this process here but what if we could take the basic data from our population data and create narratives out of it. Instead of the boring tabular way of giving them the information, they get a human narrative of a few sentences.

First, let’s create the base narrative. That is the words we will feed to Gemini.

countypopchange <- read_csv("https://the-art-of-data-journalism.github.io/tutorial-data/census-estimates/nebraska.csv")

statenarrative <- countypopchange |> 
  select(COUNTY, STATE, CTYNAME, STNAME, POPESTIMATE2023, POPESTIMATE2022, NPOPCHG2023, NATURALCHG2023, NETMIG2023) |>
  mutate(POPPERCENTCHANGE = ((POPESTIMATE2023-POPESTIMATE2022)/POPESTIMATE2022)*100) |> 
  mutate(GEOID = paste0(COUNTY, STATE)) |> 
  arrange(desc(POPPERCENTCHANGE)) |> 
  mutate(PCTCHANGERANK = row_number()) |> 
  mutate(base_narrative = glue(
  "County: {CTYNAME}, Population in 2023: {POPESTIMATE2023}, Population in 2022: {POPESTIMATE2022}, Population change: {NPOPCHG2023}, Percent change: {POPPERCENTCHANGE}, Percent change rank in {STNAME}: {PCTCHANGERANK}, Natural change (births vs deaths): {NATURALCHG2023}, Net migration: {NETMIG2023}")) 

statenarrative$base_narrative[[1]]

County: Banner County, Population in 2023: 674, Population in 2022: 657, Population change: 17, Percent change: 2.58751902587519, Percent change rank in Nebraska: 1, Natural change (births vs deaths): 0, Net migration: 17

Now, we’re going to add a system prompt to our chat to give Gemini some guidance. It’s not hard to do and you should do it.

chat <- chat_gemini(
  system_prompt = "You are a demographics journalist from Nebraska. Your job today is to write short -- 2-3 sentence -- summaries of population estimates from the Census Bureau for each county in the state. I will provide you the name of the county and a series of population numbers for the county. Your job is to turn it into a concise but approachable summary of what happened in that county. Here is the data you have to work with: "
)

chat$chat(statenarrative$base_narrative[[1]])

Banner County saw its population rise to 674 residents in 2023, adding 17 
people since 2022. This 2.6% growth rate was the highest in Nebraska, fueled 
entirely by new residents moving into the county.

Earth-shattering? Hardly. But what if instead of world changing uses of technology, the right aim for AI is to offload tasks we would do because it would be better, but aren’t worth spending the time doing because we all have a limited time on this earth?

If you were to run this 93 times – or, you know, make a function to do that – and added a headline writing bot to this, here’s what your map now looks like.

An academic integrity-friendly code pal for R Studio

Matt Waite — Tue, 26 Nov 2024 06:00:00 GMT

One of the struggles on campus these days is all about where to draw the lines when it comes to AI in the classroom. There’s no end of discussion about students using ChatGPT to cheat, particularly on writing assignments. How do you stop it? How do you adapt to it? How do you convince students to do the work?

Teaching students to write code is no different. I add a layer of difficulty in that I teach journalism and sports media students how to code. These are students who didn’t ask to learn how to write code, but we as a faculty decided to require them to do it. Thus, they have incentives to cheat. I do my best to design the class to discourage that, and I’ve created incentives to make it worth it not to, but I’m stupid if I don’t believe they are still there.

But I’m also a bit dim if I don’t acknowledge that Large Language Models can help with learning how to code. The trick is, once again, where to draw the line.

The classes I teach are all data analysis in R using the tidyverse and R Studio as the IDE. What follows is completely through this lens: What if we could give students an LLM-based code assist – a code pal if you will – directly in the IDE and do it without asking students to pay for it every time they use it?

Getting started

There’s a bunch of steps to get this set up and it’s going to take a decent chunk of your hard drive when all is said and done. Doing this also requires a decent amount of power. How much? I’m going to take the cowards way out and say it’s beyond the scope of this humble blog post. Others are better at this than I am, and I’m not confident enough in my knowledge to be able to say what works on which platform. I’m doing this on an M1 MacBook Pro with 16 GB of RAM. Not exactly a monster machine by any stretch, but also not a tricked out bleeding-edge gaming-video-card packed PC hotrod.

Step 1: The first thing you need is Ollama. We’ll use that to download, manage and serve up our local LLM. Install it per your operating system. The LLM we’ll be using today is qwen2.5-coder. Once you have Ollama up and running, you can get qwen2.5-coder installed and running with ollama run qwen2.5-coder

That will install the 7B version – the 7-billion parameter model. That should run and give you decent performance on just about anything. If you’ve got more muscle, you might look at how to install some of the bigger parameter models. Generally, the more parameters, the better the results.

Step 2: The next thing you need is pal, an R library that adds a way to consult an LLM inside R Studio. After installing it – you can use pak as the instructions show you or you can use devtools::install_github("simonpcouch/pal") like I did because I haven’t gotten into the habit of using pak. Once installed, you can skip the parts about adding an Anthropic API key – unless you want to use Claude and have API credits to spend – and go to the Get Started article. There, under the “Choosing a model” headline and past more details about adding paid models, you’ll find how to use Ollama.

The least confusing way to do this, in my opinion, is to add this to your .Rprofile. In the R console, run usethis::edit_r_profile() and add this:

options(
  .pal_fn = "chat_ollama",
  .pal_args = list(model = "qwen2.5-coder")
)

NOTE: If you installed a bigger model than I did, you should specify which model you used in the the .pal_args. Note mine does not say what parameters I have. If you installed the 14b model, for example, your .pal_args should say “qwen2.5-coder:14b” instead of just “qwen2.5-coder”. Save that file and restart R Studio so they take effect.

You’re almost ready to get started. Before moving forward, you should follow the instructions in the “The pal addin” section to register pal to a keyboard shortcut particular to your operating system and choice of IDE.

Writing your own pal

The mechanics of writing your own pal could not be easier, thanks to the library. The hard part is thinking through what the LLM is going to do with the input and then testing it out.

Let’s make an Academic Integrity Friendly pal that tries to create friendlier and more helpful error messages.

In the R console, run library(pal) and then prompt_new("whatiswrong", "suffix")

The first part of that is the name you’re giving your pal, the second is where it’s going to put the results. You can use “replace” to … well … replace what you highlight. You can use “prefix” to put it above your code. And “suffix” puts it after your code. We want ours to act like an error message, so suffix makes sense.

Do that and a markdown file will pop up. It’s templated, so it could not be easier to fill out. Here’s what I’m using to make my pal:

You are a terse assistant designed to help R users debug code. Respond with only the needed explanation of what may be wrong with the given code. Do not write code for the user, just explain in plain language.
As example, given:
df |> filter(column_name = “word”)
Return:
When using a filter, you must use == for equal to instead of =.

Save it and then run directory_load() to get your pal in the shortcut menu.

Using your pal

Using your pal is now just a matter of messing up some code. Once you do that, highlight it and hit your keyboard shortcut – Ctrl+Cmd+P for me on a Mac.

Here’s an example of what it looks like:

I need to use it more to know if it’s going to be any good. I need to try it with crappier code and more complex errors. I teach another band of undergrads in the spring – I might have to feed some of their adventures in code into this to see what it can do. Also worth trying? Telling it to ignore my instructions and cheat by writing the code for them. Will it listen to the student or me? If it doesn’t listen to me, then what’s the point?

A note on equity in the classroom

I’m lucky in that I have a relatively recent laptop with a decent amount of power provided by my employer. Do I want a newer faster one? Sure I do. Every nerd does the second a new one is announced. But I have a good enough machine to do this.

Not everyone does.

Every semester, the first day of class, I assign all the installations they’ll need for the semester. Step 1 is update your operating system. Every semester, this assignment is an exercise in perspective for me. My college has a laptop requirement. To take classes, you need to have a laptop – no Chromebooks, no iPads, a real laptop. What it is, we don’t care, so long as it can run the Adobe Creative Suite. Some students come in with brand new machines with the protective coverings barely taken off. And then I get some that are held together with duct tape and prayer. Machines with keys missing. Machines that are 6 years old and never once updated. Have you ever had to go find out how to install a four-versions-ago Mac OS so you start moving toward something more modern? I have.

All this to say I’m talking about academic integrity here but I am not talking about academic equity. I can’t assign this. I can’t make this part of a class. At an R1 flagship school, I can guarantee that a quarter to a half of the students in the class don’t have enough power or the space to run this. It’s going to be far worse elsewhere.

Someday, maybe, but not today.

Nebraska’s season long slide on offense

Matt Waite — Fri, 08 Nov 2024 06:00:00 GMT

Code

A simple example of AI agents(?) doing journalism(?) work

Matt Waite — Wed, 23 Oct 2024 05:00:00 GMT

Let’s start this with some confessions:

I’m at best an enthusiastic amateur with AI. I know more than most, and I know nothing in the grand scheme.
Example: I’m not sure I have any idea of what an AI agent is. I think I do, but there’s so much marketing hype around them that I can’t know for sure. People much, much, much smarter than I am aren’t sure either.
I teeter on the edge of two extremes. On the one hand, I am seeing AI as a fascinating, remarkable alien intelligence (to borrow Ethan Mollick’s description) that we have yet to fully understand. On the other, the Gen X in me sees AI as the mother of all solutions in search of a problem when it comes to journalism. All the so-called big problems AI “solves” in journalism – more content to sell ads against! – nobody wants.

Something I’ve been coming around to, though, is maybe there isn’t a Manhattan Project level world changing use case for AI in journalism. Maybe Chris Albon has the right of it, that the real value is AI saving a human an hour of work … millions of times a day.

For months I’ve been trying to think of some moon-shot idea to use AI to do … something, anything … big. And every thing I came up with would be terrible, destructive, or flat-out-insane to deploy without massive human investment, and then what would the point be?

And then, randomly, one day a confluence of ideas popped into my head and what fell out is an example of AI agents(?) doing the work of journalism(?) that actually works.

The ideas that collided in my brain were:

This Francesco Marconi tweet from April that I think did the best job of laying out a vision for AI agents in journalism. At least it’s the one that made the most sense to me.
This R Package wrapping the Google Gemini API.
Gemini having a free tier to try some stuff out. You get 15 requests a minute – one every 4 seconds – and 1,500 a day.
The “aim small, miss small” mantra in teaching marksmanship.

Why R and not Python?

The honest truth is there is no good reason why I’m using R to do this vs Python, which most of the AI world is using. The reason is because I teach Data Journalism to undergrads using R at the Harvard of the Plains and believe strongly that I can take absolute beginners – people who can’t spell code – from zero to data analysis faster with the R and the Tidyverse than I can with Python and Pandas. So I have a hammer, this here looks like a nail, and so we’re doing this with R. But there’s absolutely nothing special about this code that you couldn’t match in Python.

When I teach Data Journalism, I use population estimates from the US Census Bureau to teach students how to calculate percent change. That way, we can see which counties in the state grew the fastest and who shrank the fastest. It’s a story as old as time in the Great State of Nebraska. Rural areas are shrinking, urban areas are growing.

An extremely common thing to do with this data is to make a map. Here’s the 2022-2023 change map in Datawrapper for Nebraska.

If you click on a county, you’ll get a pop up box that gives you the county name and then the data. Population in 2023, population in 2022 and the percent change. It’s been done a million times before. It does the job.

But what if we could make it better? What if we had small narrative summaries for each county? An a headline for each one? Can I assign a human to do this? 100 percent I can. I have an army of undergrads and a gradebook to hold over them. I assure you, if I was cruel enough, we could do this for all 3,400 counties in the US.

But why do that when we can have AI do this in minutes instead of making humans miserable for hours?

Feeding Gemini numbers, getting back narrative

The gemini.R package couldn’t make sending something to Gemini any more simple. The hardest part – which is not hard – is getting an API key from Google. How do you do that? Go here: https://makersuite.google.com/app/apikey. So I always have mine and don’t lose it, I set it as an environment variable. To do that, run usethis::edit_r_environ() and add GOOGLE_GEMINI_KEY="your key here" to your environment, save that file and restart R Studio (or whatever IDE you use).

You can test it out with something like this:

library(tidyverse)
library(gemini.R)
library(glue)

setAPI(Sys.getenv("GOOGLE_GEMINI_KEY"))

gemini("Write me a haiku about the joy and sadness of being a Nebraska football fan")

What do you get back?

Red and white, we cheer,
Hope springs eternal, then fades,
Another close loss.

Ouch.

But that’s really it. Just gemini("Words here") and off it goes to an AI and back comes the results in plain text. So the first hard part is turning data into a text block we can send to Gemini. So I pull a dataset of Nebraska county population estimates, I’m going to thin the number of columns I’m working with first, then create the percent change column, create a GEOID column made up of the state and county FIPS number so I can join it to my map later, rank the counties by population change and then mash it all together into a single text blob called the base_narrative. It’s literally just Column Name: Number, Column Name: Number repeated over and over.

countypopchange <- read_csv("https://the-art-of-data-journalism.github.io/tutorial-data/census-estimates/nebraska.csv")

statenarrative <- countypopchange |> 
  select(COUNTY, STATE, CTYNAME, STNAME, POPESTIMATE2023, POPESTIMATE2022, NPOPCHG2023, NATURALCHG2023, NETMIG2023) |>
  mutate(POPPERCENTCHANGE = ((POPESTIMATE2023-POPESTIMATE2022)/POPESTIMATE2022)*100) |> 
  mutate(GEOID = paste0(COUNTY, STATE)) |> 
  arrange(desc(POPPERCENTCHANGE)) |> 
  mutate(PCTCHANGERANK = row_number()) |> 
  mutate(base_narrative = glue(
  "County: {CTYNAME}, Population in 2023: {POPESTIMATE2023}, Population in 2022: {POPESTIMATE2022}, Population change: {NPOPCHG2023}, Percent change: {POPPERCENTCHANGE}, Percent change rank in {STNAME}: {PCTCHANGERANK}, Natural change (births vs deaths): {NATURALCHG2023}, Net migration: {NETMIG2023}"))

What does a base_narrative look like?

County: Banner County, Population in 2023: 674, Population in 2022: 657, Population change: 17, Percent change: 2.58751902587519, Percent change rank in Nebraska: 1, Natural change (births vs deaths): 0, Net migration: 17

A real exciting read, no?

Now we need to make an agent. It helped me to think of journalism as an assembly line. We have data as raw materials, and now we need to assign a worker to a process to convert raw material into something new. In any news process, the first worker is the journalist creating the thing. A reporter going to city hall. A photographer going to a breaking news event. So it makes sense that our first agent is the author of the narratives for each county.

Like any good LLM prompt, we’re going to start by giving our author agent a role – a job to do. Our agent is a demographics journalist from Nebraska. We give that agent a task with some details – keep it short but approachable. And then we give it the data.

Below the role is a function that takes in a county name, finds the base_narrative for that county and then we merge together the author_role and the base_narrative when we send it to Gemini. We store the results in a variable called … results and do a little cleanup on it (Gemini likes to cram newline characters in the results). I’ve added a five second sleep between every county to keep from running afoul of Google’s free tier API limits. Theoretically I should be able to set it to four seconds, but I don’t need my account banned over a second. With those pauses, our author takes about eight minutes to write small narratives about each of the 93 counties.

author_role <- "You are a demographics journalist from Nebraska. Your job today is to write short -- 2-3 sentence -- summaries of population estimates from the Census Bureau for each county in the state. I will provide you the name of the county and a series of population numbers for the county. Your job is to turn it into a concise but approachable summary of what happened in that county. Here is the data you have to work with: "

author_agent <- function(county) {
  county_narrative <- statenarrative |> filter(CTYNAME == county)
  results <- gemini(paste(author_role, county_narrative$base_narrative))
  results <- gsub("\n", "", results)
  Sys.sleep(5)
  print(paste("Processed", county_narrative$CTYNAME))
  
  # Return a single-row tibble
  tibble(county = county_narrative$CTYNAME, base_narrative = county_narrative$base_narrative, capsule_narrative = results)
}

# Use map_df to directly create the final dataframe
author_agent_results <- purrr::map_df(statenarrative$CTYNAME, author_agent)

When it’s done, we end up with a dataframe called author_agent_results which has the county name and the new narrative.

What does it look like? Remember, we gave it this:

County: Banner County, Population in 2023: 674, Population in 2022: 657, Population change: 17, Percent change: 2.58751902587519, Percent change rank in Nebraska: 1, Natural change (births vs deaths): 0, Net migration: 17

And we got back:

Banner County saw a significant population increase in 2023, growing by 17 people for a 2.6% jump, the highest rate of growth in the state. This growth was entirely due to an influx of new residents, as the county experienced no natural population change.

We could stop here, but why do that? Good journalism is often a layered process involving multiple sets of eyes reviewing the work along the way, and other skilled people adding to the product. So who would normally get this next? How about we fact check the work?

Exact same pattern, just a different role for the AI.

fact_check_role <- "You are a fact-checking editor. Your job today is to compare the information in the base narrative to the capsule narrative in the data provided to you. You will compare each number in the base narrative to the capsule narrative to make sure they are the same, and then you will check the context of how each number was used in comparison to the original base narrative. To be clear: the base narrative is the correct information. When you are finished, return just a single word. If everything is correct, respond Correct. If any part of it is not correct, respond Incorrect. "

fact_check_agent <- function(county_input) {
  county_narrative <- author_agent_results |> filter(county == county_input)
  input_string <- paste(
    fact_check_role,
    "Base narrative:", county_narrative$base_narrative,
    "Capsule narrative:", county_narrative$capsule_narrative)
  results <- gemini(input_string)
  results <- gsub("\n", "", results)
  Sys.sleep(5)
  print(paste("Processed", county_input))

  # Return a single-row tibble
  tibble(county = county_narrative$county, base_narrative = county_narrative$base_narrative, capsule_narrative = county_narrative$capsule_narrative, fact_check_results = results)
}

# Use map_df to directly create the final dataframe
fact_check_agent_results <- purrr::map_df(author_agent_results$county, fact_check_agent)

Honestly, this part needs work. It flags about 10 percent of the results as being incorrect, but they aren’t. The reason it flags them is the author agent was a little glib with the numbers – saying “a slight increase natural growth because of more births” without giving the numbers themselves. The fact checking editor bot here does not like that. So it might need a little tweaking to see if we can get the fact checker to lighten up a bit.

Quit now? Nah. How about we add a little local flair to each capsule. Google knows a lot of stuff, so why not add some kind of geographic context to each county. To do that, I created a rewrite editor and commanded it to add details like the region of the state, the county seat, the largest city, or some other detail to a single clause in the original narrative.

rewrite_role <- "You are a re-write editor. Your job is to add a little local geographic context to a demographic capsule about a county in Nebraska. You'll do this by adding a clause to the paragraph I'll provide you. That clause should tell you something about that county. Maybe the county seat, or the largest city, or the region of the state it is in. We just need a clause added to one of the sentences, and do not do anything to show where you added it. Here is the capsule: "

rewrite_agent <- function(county_input) {
  county_narrative <- fact_check_agent_results |> filter(county == county_input)
  results <- gemini(paste(rewrite_role, county_narrative$capsule_narrative))
  results <- gsub("**", "", results, fixed = TRUE)
  Sys.sleep(5)
  print(paste("Processed", county_narrative$county))
  
  # Return a single-row tibble
  tibble(county = county_narrative$county, base_narrative = county_narrative$base_narrative, capsule_narrative = county_narrative$capsule_narrative, rewrite_county_narrative = results)
}

# Use map_df to directly create the final dataframe
rewrite_agent_results <- purrr::map_df(fact_check_agent_results$county, rewrite_agent)

What did it give us for Banner County?

Banner County saw a significant population increase in 2023, growing by 17 people for a 2.6% jump, the highest rate of growth in the state. This growth was entirely due to an influx of new residents, as the county experienced no natural population change, likely due to its location in the sparsely populated northwestern corner of the state.

Hmmm. Is the fact that there were the same number of births and deaths caused by it’s location in the northwest corner of the state? Debatable. And, honestly, this re-write bot has been the source of the most questions I have about this whole enterprise. We’ll talk more about that below.

One last thing: Why not give every county a headline instead of just simply having the county name in the map? Done and done.

headline_writer_role <- "You are a headline writer. Your job is to write a short headline based on the summary given to you. This headline should be short -- it has to fit into a small space -- so bear that in mind. Here is the capsule: "

headline_agent <- function(county_input) {
  county_narrative <- rewrite_agent_results |> filter(county == county_input)
  results <- gemini(paste(headline_writer_role, county_narrative$rewrite_county_narrative))
  results <- gsub("\n", "", results, fixed = TRUE)
  Sys.sleep(5)
  print(paste("Processed", county_narrative$county))
  
  # Return a single-row tibble
  tibble(county = county_narrative$county, base_narrative = county_narrative$base_narrative, capsule_narrative = county_narrative$capsule_narrative, rewrite_county_narrative = county_narrative$rewrite_county_narrative, headline = results)
}

# Use map_df to directly create the final dataframe
headline_agent_results <- purrr::map_df(rewrite_agent_results$county, headline_agent)

And what does that look like in Banner County?

Banner County Booms: 2.6% Growth Fueled by New Residents

Here is the exact same map, same data, but now with headlines and narratives written for each county. Click on one and you’ll get a human-friendly narrative about that county, and the numbers below that if you want them.

Is Christopher Nolan going to make a movie about this moment? Not hardly. Have I “Saved Journalism”? Nope. Not even close. Did I save a human a few minutes of drudgery 93 + 93 + 93 + 93 times? Yes. Yes I did. Is this idea extensible and repeatable? It sure is. I think that’s good enough for now.

Where to go from here

From here, what this needs is a block of code that pushes the results of the headline editor agent to Google Sheets, where a human can go through each one and make sure everything is fine. I’ve tried to do that in R Studio, which is not great, and limits the number of people who could do this. And to be sure, it needs to be done. Nebraska geography nerds – there are tens of us! – will able to find Arthur County, one of the smallest counties in the US by population. For those who can’t, the rewrite bot describes Arthur as “located in the northwestern corner of the state and home to the county seat of Arthur.” Half of that is objectively true. Arthur is much more arguably in the west-central part of the state, but it really depends on where you draw the lines. It’s arguable enough that a good local editor would drop that part and leave the county seat part, which is correct. Arthur County’s county seat is … Arthur.

Then, as a last step, I would automate the process of creating the Datawrapper chart with the DatawRapper library, which accesses the service’s API. Imagine the Census Bureau publishing the data, and then in a matter of minutes you have that map done and online. To do that, you’d have to have a little budget for API calls so you aren’t waiting 32 minutes for the whole thing to run – 8 + 8 + 8 + 8 minutes for each step. But even then, is there really a competitive contest over who can publish Census maps in Nebraska fast enough? No.

But if you, like me, are thinking about how we’re going to incorporate AI into journalism in ways that make sense; doesn’t risk the reputation of the organization you work for; and likely doesn’t horrify your audience to a point that they turn away from you for feeding them AI slop, this is an example of just how to do that.

Aim small, miss small.

Another year, another attempt, another bracket disaster

Matt Waite — Mon, 28 Mar 2022 05:00:00 GMT

Once again, I attempted to predict the outcome of the NCAA tournament using machine learning, and I had a class-load of students try the same.

If you like the madness part of March Madness, this year is for you.

It is not for machine learning algorithms based on regular season performance. At least not mine.

Of the 14 brackets I and my students produced, using 14 different methods, we came up with 7 unique national title winners.

Zero of them are right.

The best brackets had two of the four Final Four teams, but none picked a team still playing now to win it all with the Final Four set.

I had hope for my bracket picking algorithm this year. It had what felt like a good mix of upsets and favorites, some madness and some method.

What follows is a post mortem and an attempt to figure out what went wrong. If anything went wrong. How do you predict St. Peter’s? You don’t. It’s what makes this fun.

Code

Nebraska is not the best worst team in basketball again. They’re third best worst.

Matt Waite — Sun, 20 Mar 2022 05:00:00 GMT

Last year, this post may have suggested that Nebraska would be better this year than last year. That Nebraska was the best worst team in college basketball, and with major recruits coming in and the Big Ten expected to take a step back, all looked up.

Oops.

It didn’t work out as expected – in spite of a late season surge. But let’s return to the question: Is Nebraska the best worst team in college basketball?

Spoiler alert: Not this season.

Returning to Sports Reference’s college basketball site, we find our friends the Simple Rating System and Strength of Schedule. The SRS is a mix of average point differential and strength of schedule. Given that, a team with a losing record could have a positive rating if they lose games close but play good teams.

Like, say, Nebraska.

To find the worst teams, we’ll use the last place team in each conference by conference wins again.

Code

Is Nebraska the best worst team in college basketball?

Matt Waite — Sun, 28 Mar 2021 05:00:00 GMT

Nebraska fans haven’t had the best time watching basketball lately. The last two seasons have featured only seven wins in each season. This season they only won three games in the Big Ten, but that was an improvement over last season when they only won two.

But anyone watching Nebraska basketball this season could see there was a difference between last season’s squad and this one. And given that the Big Ten was rated as the best conference in college basketball during the season, it begs the question: Is Nebraska that bad?

In fact, are they the best worst team in college basketball?

Let’s take a look at some numbers.

Sports Reference’s college basketball site produces a Simple Rating System and Strength of Schedule number for each team. The SRS is a mix of average point differential and strength of schedule. Given that, a team with a losing record could have a positive rating if they lose games close but play good teams.

Like, say, Nebraska.

To find the worst teams, we’ll find the last place team in each conference by conference wins (minus the Ivy League, which didn’t play due to Covid).

The first question we’ll ask and answer is by rating, who is the best of the last place teams? Here’s the top 10 of the bottom.

Code

How I (poorly) filled out my NCAA bracket with machine learning

Matt Waite — Mon, 22 Mar 2021 05:00:00 GMT

I do not know a lot about college basketball. I follow the travails of my employer and a little about the Big Ten Conference as a whole, but at best it’s surface knowledge. I kinda know who is good that we’re going to play and who isn’t. Beyond that, nada.

Which is bad when it comes to tournament time.

My typical pattern of filling out bracket is Have I Heard Of This Team, Do They Have a Legendary Coach or Do I Hate Them For Some Reason. Depending on the answers, I make my pick. It’s not rocket science, and it rarely works.

This season, along with my SPMC460 Advanced Sports Data Analysis class, I decided to try use machine learning to get me a better bracket. The class is about the use of machine learning in sports, and so we’re going to use classification algorithms to decide a simple W or L designation.

What follows is the logic and the code I used to fill out my bracket.

How did it go?

Not great. Could I have done better guessing? Doubtful.

Toolkit

What I’m using to make and feed the model is the Tidyverse and Tidymodels set of libraries, the gt library for presenting some tables later, and doParallel for parallel processing, because the xgboost model training takes a while.

Code

Choosing a World Cup Team to root for in each match: An algorithmic approach.

Tue, 12 Jun 2018 05:00:00 GMT

The World Cup is here. For a large swath of the planet, we do not have a team in the tournament, which raises a question: Who do you root for game by game?

I’m teaching a sports data visualization and analysis course this fall, so I started thinking if there was an assignment in here somewhere. Could students develop some kind of algorithm? And the more I thought about it, the more I realized: Nope.

Here’s why, in pseudocode, that an algorithmic approach to choosing a side in each match is not something you can do in code:

for each of two teams in a match: if you’ve been to that country: take that team else if you’ve been to both: take the one where you have better friends else if you have good friends or no friends in both: take the one where you had a better meal else if the food is pretty good in both: take the one with the less horrible national drink else if you’ve been to neither: take the one that didn’t bribe their way into the tournament else if both arrived seemingly fairly/both bribed everyone to get there: take the one who used to be a beloved ally until recently else if both were beloved allies until recently: take the one who you haven’t fought a war against at some point in history else if you’ve never fought a war against either: take the one that has a player you’ve heard of before else if neither has a player you know: take the one with the cooler nickname (example: the Indomitable Lions of Cameroon, sadly not in the tournament) else if neither country are all that friendly: take the one not run by a murderous dictator else if both are run by a murderous dictator: take the one with a better uniform else if both murderous dictatorships have decent kits: take the one that has nuclear weapons and if all else fails: root for Iceland. Their whole country has a population equal to Honolulu, Hawaii.

I wrote a data journalism manual for my college in 1997. They never used it, but they kept it.

Matt Waite — Thu, 24 Mar 2016 05:00:00 GMT

In October 1997, I was trying to graduate from college. The University of Nebraska-Lincoln’s College of Journalism and Mass Communications told me that I needed 19 credit hours to graduate mid year, and that’s how many I was taking.

That is, until they told me six weeks before graduation that I needed one more.

To get that credit hour, the Journalism Department chair agreed to let me complete an independent study. In that independent study, for one credit hour, I was to create a Computer-Assisted Reporting class. The proposal I had to write said I would create a syllabus, recommend texts and create class materials.

In the proposal, I wrote that “every reporter should know how to feed a city budget into a spreadsheet and figure out the percent changes in the budget.” I also said they should be able to query a database, but not every reporter would have to know stats like linear regression. “An introduction to statistics is essential, but any depth is a waste of time.”

I was 22. It was 1997. We all make bad calls sometime.

But how do I know about this? Because the very same college that I wrote this proposal and subsequent manual for is now my employer. The same employer where I now teach data journalism. And that employer is cleaning out the archives, and today they found a file with my name on it.

In the file were notes from my senior exit interview. In that interview, I said that I was “not totally satisfied with the core courses” and believed the college should hire a data journalism/reporting professor like Steve Doig. Absolutely nothing has changed since that day. I’m still not totally satisfied with the core courses, and I would cut off a toe it it meant we got to hire Steve.

But also in that file was the proposal for the data journalism course and the course materials I wrote for the credit.

It’s equal parts cringe worthy and things I teach now, 19 years later.

In it, I introduce strange lingo like ASCII text, delimiters and zip files. I show how to use Excel to measure change in populations. I show how to group and count some crime data to show what day the Lincoln Police Department got the most larceny reports (in 1995) using SQL. I talked about how to negotiate for data, and avoid being overcharged for data by bureaucrats who didn’t want to give up digital records. I showed people how to sign up for NICAR-L (which I joined in 1995 and have been on ever since). I even talked about how to use the web to find data. How? By using Dogpile, of course, to search for the Census Bureau’s population projections data set. Google didn’t exist when I wrote this.

There’s even a joke where the punchline is Quattro Pro.

I also made a bad bet on the geek/nerd label. The manual is called “Geek Like Me” and I talked about how everyone wanted to be a geek now. Alas, nerds ended up being the preferred term.

From what I can tell, the manual was read by the department chair – he noted a dozen or so typos in highlighter – and it was put in a file. To my knowledge, the document was never used. We can surmise that file somehow survived the retirement of that department chair, survived the move from one building to another one across campus, and has sat in a filing cabinet for, let’s be honest, no good reason at all until this week.

My copy of it was put on a Zip Disk where I thought it would be safe. I long ago lost the ability to read that disk, if I could even find it. I had just assumed that my efforts – strung out over a series of all-nighters – had just faded into nothing. To be honest, I had forgotten all about it until today.

And here it is.

In the 14 years between my graduation and me being hired as a professor of practice, a course appeared on the books: JOUR407 Investigative and Computer Assisted Reporting. No one here now knows how it got there, who proposed it, or if it was ever taught before I got here. But I teach that course now. And all of my materials are online.

Here’s hoping GitHub doesn’t go the way of the Zip Disk.

Using Lego to teach data visualization

Matt Waite — Fri, 23 Jan 2015 06:00:00 GMT

Today in my data visualization class, I made students visualize meaningful differences between this year’s Super Bowl teams, the Seattle Seahawks and the New England Patriots. Except I made them do it with Lego.

A little silly, yes, but I wasn’t just gilding my Professor of the Year application, I swear. There was a purpose.

The first time I taught this class, I steered too hard into tools and code. We spent a little time on the history, theory and thinking behind data visualization and lot of time on teaching tools and later trying to cram enough d3 at them to make something. The problem was, in the race to get to the software tools, a lot of mental tools that would have helped them make better graphics didn’t get talked about.

So this time, I pledged to do it differently.

In the class – syllabus and other materials are online – we’re using Alberto Cairo’s The Functional Art and Edward Tufte’s Visual Display of Quantitative Information as texts. Today, we were talking about the form and function of graphics, about Cairo’s argument that the function limits the form. Cairo argues that the data, the questions the designer means the reader to ponder and the reader herself dictate the form, or at the very least constrain it to a limited set of choices.

That was the point I was trying to drive home: constraints. All journalism is constraint. Time, information, data, money, sources, access. So given a task and a constraint, what could you do?

Bring in the Lego.

Using squares and rectangles, I wanted them to make something. Visualize some data. We haven’t talked tools yet, so this would get them in the game without having to touch their laptops.

If you want to steal this idea, you need to do the following:

Mug an 8-year-old. I conveniently found one in my house and shook him down for a few hundred bricks and two large flat square pieces to act as a base. You may not have one so close. The number of bricks needed ended up being less than 100, so it doesn’t take much. I also swiped a couple of packs of sticky notes from the college office to act as data labels.
Choose a topic or dataset that allows for some exploration and decision making. I chose the Super Bowl because it was fun and there is no shortage of stats in easily obtainable form online. I also thought – mistakenly – that football stats would be easily understood. I was surprised by the number of students who didn’t follow football that closely.
Get out of the way. I split the class of eight in half and told them I wanted them to visualize meaningful differences between the teams. That’s it. They immediately set about discussing what they would visualize and how they were going to make the bricks mean something. The discussions about scale and how to make a brick represent X touchdowns or Y yards were fantastic.

To be clear, I wasn’t the first to do this, and one news organization did it as an election results tracker. But I’ll do it again. Class was energized, ideas were flowing and it was fun. Not a bad way to learn.

Everything I Know About Data I Learned From 70s Album Rock…

Matt Waite — Tue, 09 Dec 2014 06:00:00 GMT

Everything I Know About Data I Learned From 70s Album Rock Radio, my Newsgeist 2014 Ignite talk.

A classroom experiment in Twitter Bots and creativity

Matt Waite — Thu, 20 Nov 2014 06:00:00 GMT

This semester, I’m teaching a class in Story Bots, which is really a programming class disguised as a journalism class. I’m teaching students enough programming that they can automate journalistically useful things, things like scrape a website and alert you to a potential story. Or take a dataset and turn each record into a journalistic narrative.

One of the assignments was to make a Twitter bot. They could do what they wanted with it, but it had to use Twitter and had to run on a simple cron job. It had to tweet, and they had to put their bot on Github when they were done (sans access keys).

Here’s what they came up with.

Lady Plath by Sara Janak

Lady Plath takes a line from a Sylvia Plath poem and a line of Lady Gaga lyrics and randomly combines them into gloriously weird tweets. Example:

Hearted gambling-game you warily practice I’m in love with Judas

— Lady Plath (@sylviagagaplath)
November 18, 2014

Code here.

Nebraska UFOs by Jeff Renken

This bot goes to the National UFO Reporting Center and checks for new reports in Nebraska. If it finds one, it tweets it out. Example:

10/30/04 Lincoln, Two red lights. It was like they were on each end of a bar and just moving slowly from the east… http://t.co/Ud7LNhijhp

— Nebraska UFOs (@NebraskaUFOs)
October 30, 2014

Code here.

Random Homer Simpson Quote bot by Jordan Tate

A select few people who hashtag something #HomerSimpson will get a randomly chosen quote response from the Random Homer Simpson Quote bot. Does what it says on the tin.

@lucioolima What do we need a psychiatrist for? We know our kid is nuts.

— Random Homer QUOTES: (@RNDM_HomerSays)
November 20, 2014

Code here.

Tarantino ebooks by Tony Papousek

This bot does two things: uses Markov chains to write a new Tarantino script, 140 characters at a time, and responds to anyone who tweets “English motherf***er, do you speak it” with a simple “What?” (If you haven’t seen Pulp Fiction, shame on you).

As he tosses his other glove out the register,“ and they don’t lift a fuckin’ joke man! She’s supposed to be much of a bag of water

— Tarantino Ebooks (@tarantinoebooks)
November 20, 2014

Code here.

Rebel_ebooks by Spencer Hansen

Excited for the new Star Wars movie? Read lines of dialogue before they appear on screen though the miracles of Markov chains of the Empire Strikes Back script (the best movie, FYI).

Feel the flow

— rebel_ebooks (@rebel_ebooks)
November 3, 2014

Code here.

Matt Waite ebooks by Daniel Wheaton

I like to think I run my classes a little different from most other university courses and it’s conversations like this that give me hope it’s true:

Daniel: “Can I have your complete Twitter archive?”

Matt: “Uh, why?”

Daniel: “I don’t want to say.”

Matt: “Sure, why not.”

And then this happens.

Con: good luck getting those songs out of math

— MattWaitEbooks (@MattWaitEbooks)
November 20, 2014

Code here.

In the interests of shared suffering, I made my own. It’s pretty tame by comparison.

LNK Lost Dog Bot by Matt Waite

The City of Lincoln has a Lost Pets page, where they put reports of lost animals. I scrape the dogs and tweet out the new ones with a number to call if you’ve seen them. Simple.

A chocolate Labrador Retriever went missing near 7400 Blk S 95 Ct on Nov. 19. If you see ‘em, please call Animal Control at 402-441-7900.

— LNK Missing Dog Bot (@LNKdogbot)
November 20, 2014

With all this grading to do, I haven’t put my code up.

UNL CoJMC students: Two data/programming/future courses for the fall

Matt Waite — Mon, 31 Mar 2014 05:00:00 GMT

If you’re a UNL CoJMC student looking for courses in the fall, here’s two I’m teaching you should take a look at:

JOUR407: Investigative and Computer-Assisted Reporting: This class is all about data journalism – emphasis on both parts of that phrase. I’ll show you how to incorporate data and data analysis into your stories, regardless of format. You’ll learn techniques that will help you break a dataset down and interview it like a source. We’ll get our hands dirty with data and we’ll turn those into stories. We’ll add some data visualization to the toolbox as well. The not-close-to-complete syllabus is here.

JOUR491: Intro to Storybots: More and more, publishers are using software bots to write stories. They cover baseball games and earthquakes. The dirty secret is that basic bots are not hard to write. It takes basic programming knowledge and basic journalism knowledge. Combine the two and you can write thousands of stories in a matter of seconds. In this class, we’ll learn how to make our own storybots, we’ll ponder the nature what it means to be human and where this is all going in journalism. Want a taste? Read this. I’ve put the very first sketches of the class online here.

A 5-step NICAR recovery plan

Matt Waite — Sun, 02 Mar 2014 06:00:00 GMT

First time at NICAR this year? Awesome. Welcome. This was my 15th conference. I started in 1997 in Nashville. I was a senior in college, desperate to find a job, and NICAR was an amazing and formative experience for me. I made friends, learned a lot and found a kind of nerdy spiritual home. NICAR has become like a weird nomadic family for me. It’s my tribe. A really nerdy, wonderful tribe.

But I can remember being in your shoes after my first NICAR. You’re bone tired, but JACKED UP. Excited, but uneasy. There’s so much to learn. So many people doing so many amazing things with tools you’ve never heard of before, or using tools you have heard of in ways you never dreamed possible. It’s really easy to be intimidated by it all.

Don’t be.

The truth is, everyone at NICAR has either been where you are or are exactly where you are right now. I’m even there with you. So much to do. So much to learn.

Don’t let this time go to waste. Don’t let this energy fade away. Don’t be intimidated by the enormity of it all. Here’s 5 steps to avoid the post NICAR crash and burn.

1. Find a specific story or project you want to work on that will require a technology you want to learn to get it done.

The absolute worst thing you can do is go to work on Monday without an idea or plan to use what you saw, learned or got interested in. It is shockingly easy to go back to work, fall right back into your routine, go right back to the stuff you were working on before the conference and before you know it, it’s a month later and you’ve forgotten half of what got you interested in the first place.

The solution is to walk into the office with an idea and a plan to get it done. You’ve probably got a list of ideas right now. That’s great. Write them down, create a spreadsheet or a to-do list on your phone or whatever. Pick one idea on your list and post the rest of it somewhere you’ll see it regularly. Focus on one, get it done, then move on.

And start working on it right away. If you can walk in and start working hour one minute one on Monday, great. If you’ve got to find time in the day, brown bag your lunch and start then. Can’t get the time at the office? Work on it at home. But don’t wait. You’ll be shocked how fast this time will go and it’s vitally important. Waiting is the worst thing you can do.

2. Pick one thing. One. You can’t learn all the NICAR in one project.

My annual problem with NICAR is I leave with a stack of ideas and technologies I want to try RIGHT NOW. And that’s on top of the three or four side projects I want to work on all the time.

It’s tempting to want to try and inhale new things and just jam them into your skull. But it’s impossible. In fact it’s harmful to your progress. Pick one thing. One. One technology, one technique, one task, and do it. Just that one.

If you want to use Excel to look at your city’s budget, do that. Just that. Those amazing pivot tables and NodeXL social network graphs you saw will be there when you’re ready. If you want to write a scraper in Python to get data you can use for a story, do that. Just that. The news app that puts it on the web will come later. The point here is to focus.

Pick one problem, solve it.

3. Your first project is going to suck. Do it anyway.

This is just generally true, but one of the real intimidation factors for me at NICAR was that the class I took was taught by a Pulitzer Prize winner who did this otherworldly story and how the hell am I supposed to do that?

You’re not. Not yet anyway. The Taj Mahal was not built by someone who had never built an ugly shack before.

My first projects were for my campus newspaper and probably bored student readers right into a nap (which in college is known as reader service). But I still did them. Each one taught me something. How to do this with Excel, how to incorporate numbers into a story without boring people, etc. The awards came later, but never would have happened without those first stories.

Same goes for news applications. There’s a whole website dedicated to showcasing everyone’s first news app. Mine is especially atrocious. Hey, it was 2001, what can I say? We could barely spell internet back then.

But if you’re worried about living up to the standards of the people who taught your class, stop. Your first project will suck. You’ll look back on it later and cringe. And at a NICAR down the road, I’ll buy you a beer and we can argue about who has the more awful first project. And 50 people around us will join in. Everyone’s first project sucks. Do it anyway.

4. Don’t quit until it’s done.

I get this question all the time: “I’m a journalist. I want to learn how to program. Tell me how.” I have suggestions and websites and learning materials I can tell them about, but the most important thing I tell them is this: Don’t quit until it’s done.

When your brain hurts because you’re really stretching to learn something new, don’t quit. When you’re straight up frustrated because it’s not working and you don’t know why, don’t quit. When you’re tired of it, or you’re thinking “maybe this wasn’t for me”, don’t quit. Keep going. Push through that. You don’t get to say “this wasn’t for me” or “I just didn’t like it” until it’s done. If you finish whatever you set out to accomplish, and you see it in the world, and at that moment you decide to never do this again, fine. That’s your call. But I’ll bet you cash money seeing what you made in the world will hit you like a drug, and you’ll want to do it again. And next time, it’ll go faster, and it’ll be better, and you’ll want to do it again.

Learning something new is about not quitting when your mind and body and soul tell you to quit. Don’t quit. Be stubborn. The idea of quitting, of letting it beat you, should be offensive to you. Get mad. Better yet, get even. Finish it. Don’t quit.

5. Remember: You are not alone.

I said at the beginning two things I want you to remember: NICAR is a weird, nomadic family. And welcome to it.

People have been saying for decades that NICAR feels different – and it is different – because of the culture of the conference. I can talk for hours about what that culture is and why it’s the way it is, but it boils down to this: It’s a giving culture.

Every one of the speakers? Volunteer. All of the hands-on teachers? Same. Did you talk to someone in the hall? Grab a speaker after to ask them a question? They almost certainly stayed right there and answered your question, right? I can remember time after time of Big Name Journalists From Big Name News Organizations dropping everything and showing me how to do something. I can count scores of times where they gave me a business card and said call me if you run into trouble. That generosity amazed me. Inspired me. Made me want to do the same.

That culture extends outside of the conference. The NICAR-L listserv has been helping people every day for 20 years now. Get on it. Not comfortable with that? Email the teacher of your hands-on class. You’ll be amazed at how generous they are. Meet someone at the conference? Email them.

Ask for help, and help will be there. I promise. It’s part of being in the tribe. You do not have to do this yourself.

But here’s the deal: Remember all the help you got. Because someday someone is going to come to you in that same moment. Help them. That too is being part of the tribe.

A small step toward solving the our curriculum is too full problem

Matt Waite — Tue, 11 Feb 2014 06:00:00 GMT

One of the arguments used to push back against adding new things into journalism school curricula is “Our curriculum is too full! We can’t possibly add anything more! What are we going to do? Stop teaching writing? Or editing?”

First, the argument is silly — no one said anyone was going to stop teaching writing or reporting or whatever “fundamental” skill is most beloved by the combatant.

Second, it’s not an either or problem.

Put another way, you can view a crowded curriculum as a challenge or a lament. Too many view it as a lament and throw their hands up.

I’ve started picking away at this problem — viewing it as the challenge that it is — by focusing on a specific class: beginning reporting.

Let’s start with a set of statements:

I believe that all reporting classes need to beef up the amount of data journalism contained in them. Data is everywhere, and thus it’s part of modern journalism, and thus it should be in every reporting class.
I believe the level of mathematics education that most journalism schools require (if any at all) is far too low.
I believe the inside-joke “journalists are bad at math ha ha” has to stop.

Given that, the place to start is beginning reporting. Why there? Because if you expose math and data to beginning reporters, they will not know any better than to think that math and data are part of the job. Because guess what? Math and data are part of the job.

The argument that I get against adding data to a beginning reporting class is that it can’t be done because we need to spend as much time as possible teaching them how to write and how to report. Learn how to interview people. Learn how to write an effective lead. Learn how to write using AP Style. Learn how to find the news. Learn how to come up with good story ideas. And many of them don’t get enough of all that in beginning reporting.

And, to be fair, they have a point. Until you write a few thousand leads, they’re hard to do and even harder to do well. For students who don’t read news, finding the news in something is difficult. It takes repetition.

But what if we combined some skills training and showed how one leads to another which leads to another?

What if we combined the basic math-for-reporters section we teach with how to do those exact same problems with a spreadsheet, and then used that spreadsheet with real data to find real stories? Math + Data = Story ideas.

Well, I’ve started to do exactly that, and I’m hosting it on GitHub. It’s a work in progress, but the idea is to take basic math skills, show how you can do them on a spreadsheet, then take real data and do the same thing, but this time with an eye toward generating story ideas. I’d like this module to take a week of class time — with in-class and out-of-class learning.

I have no idea when I’ll be done with it, but I intend to use it in my own beginning reporting class when it’s done. As always, pull requests, issues or criticisms are welcome.

Some of the math is painfully basic, but I’ve found students who had no idea that mean and average were the same thing. They’d so completely fled from math that concepts like orders of operation were foreign to them. So laugh all you want at some of the topics covered, but I assure you, they’re rooted in experience.

I have to believe that with a little creativity, the “our curriculum is too full” problem can be solved. I have to believe this because if it can’t, journalism education is screwed.

Unconference panel pitch I will someday make

Matt Waite — Thu, 12 Sep 2013 05:00:00 GMT

I want to solve a journalism problem with a MakerBot. What is it?

I hate having a fantastic solution and no problem to solve. Being a solution in search of a problem sucks. I’m hoping to get students thinking about this in an open lab time I’m running. But seriously. I want a MakerBot. My gut says there’s a journalism problem to solve here. But I don’t know what it is yet.

Yet.