Week 3 · Section 2 · Lab

Train a neural network, then question it

On Wednesday you saw how machines learn from examples. Today you'll train a small neural network yourself, watch it overfit, and then test a fraud-flagging model the way an auditor would. Everything runs in this page. There's nothing to install.

Your name

Goes on your submission. Stored only in this browser.

Partner's name (optional)

Pairs can share a laptop, but each person submits their own answers.

0 of 9 items done

Part 1 · 12 minTrain a network to predict which bills get out of committee. Find the right size, then watch it overfit.
Part 2 · 12 minTrain a model to flag benefit claims for fraud checks. Then check who gets flagged.
Part 3 · 4 minAnswer Wednesday's question in your own words, then submit on Canvas before you leave.

Recap: how machines learn

The five ideas from Session 3 that you'll use today. Your TA will go over them at the start of section; come back here if you need a reminder.

1 · Where machine learning fits

AI systems that mimic human reasoning and decisions

Machine learning learns rules or patterns from data

Deep learning machine learning with many-layered neural networks

Generative AI learns what data look like, to make new content

LLMs generative AI built on transformers, trained on huge amounts of text

2 · Programming vs. machine learning

ProgrammingRules (a person writes them) + Data → Answers

Machine learningData + Answers (labels) → A rule learned from the examples

3 · Three kinds of learning

Try it: which kind of learning is each task?

Predict whether a bill will pass, from 10,000 past bills marked passed or failed.

Every example comes with the answer (passed or failed). That's supervised learning, and it's what you'll do in Part 1.

Group legislators who vote alike, with no labels given.

There are no answers to learn from, only patterns to find. That's unsupervised learning.

Improve a chatbot using people's thumbs-up and thumbs-down on its answers.

The model acts, then gets a reward or penalty. That's reinforcement learning. You'll see it again in the post-training session.

Flag benefit claims for review, learning from past claims marked correct or incorrect.

Supervised again: the past claims are labeled. But who wrote those labels? Keep that question in mind for Part 2.

4 · Overfitting: training error vs. new-data error

A flexible enough model can memorize its training examples, noise and all, so its training error keeps falling. What matters is the error on examples it has never seen: the test error. That falls at first, then rises once the model starts memorizing. This is overfitting. That's why we always hold back some data to test on.

5 · Neurons and gradient descent

A neuron takes a weighted sum of its inputs, then bends it. A network stacks layers of neurons. The weights are the numbers it learns. It learns them by gradient descent:

Start with random weights.
Measure the error (the loss) on the training examples.
Change every weight a little in the direction that lowers the error.
Repeat. One pass through all the training examples is one epoch.

Part 1 · Will the bill get out of committee?

Each dot is a bill in a made-up legislature. The network sees two numbers about each bill:

Sponsor's ideology (left to right): how far the sponsor is from the chamber's median member.
Majority-party cosponsors (bottom to top): what share of the bill's cosponsors come from the majority party.

Its job is to predict whether the bill got out of committee or died in committee. The background shows the network's guess for every possible bill. Test accuracy is the share of 1,000 new bills it gets right.

The data are simulated. In this made-up chamber, moderate sponsors need less majority support, and sponsors far from the median need a lot. The network doesn't know that rule; it has to learn it from the dots.

← left · sponsor's ideology · right → majority cosponsors →

Show the 1,000 test bills (faint)

Training data

Hidden layers

Neurons per layer

Speed

Changing the data or the network starts over from new random weights. Training stops on its own at epoch 3,000.

Epoch: 0
Training loss: –
Test loss: –
Training accuracy: –
Test accuracy: –
Lowest test loss: –

training loss test loss lowest test loss

Recorded runs go into your submission automatically.

#	Data	Network	Epoch	Train acc.	Test acc.	Train loss	Test loss	Lowest test loss at

1A · One neuron

Data: 200 bills. Network: 1 layer, 1 neuron. Train to about epoch 500, pause, and record. What shape is the boundary between the colors? Why can't this network follow the curve in the dots?
1B · Just big enough

Keep the 200 bills. Add neurons (and layers, if you like) until test accuracy reaches 88%. Give each network about 1,000 epochs: some get stuck for a while before they improve. What's the smallest network that gets there? Record it. How did training accuracy compare with test accuracy?
1C · Too much, too little data

Switch to 25 bills, messy (a small sample with some coding errors). Network: 2 layers of 8 neurons. Train to at least epoch 1,500 and record. What did the two loss lines do? About when was the test loss lowest? If this model were going to be used for real, when would you have stopped training, and why?

Part 2 · Which benefit claims get flagged?

A fictional benefits agency wants a model to pick which claims get a fraud check. It can check 10% of claims. The model learns from 2,000 past claims, each marked in the agency's records as fraud or not fraud. It's then tested on 2,000 new claims, and the 10% with the highest risk scores are flagged.

The agency can give the model up to four inputs:

Input	What it is
Income irregularity	Sudden jumps in reported income. Always used.
Earlier corrections	How many times the claim was corrected before. Always used.
Postcode	The share of non-citizens living in the claimant's postcode.
Citizenship	Whether the claimant is a citizen.

Loosely modeled on the Dutch childcare-benefits case from lecture. All claims are simulated by this page.

Inputs the model may use

Income irregularity Earlier corrections Postcode Citizenship

Network: 1 hidden layer of 6 neurons, 300 epochs.

What the agency can see: its own records

	Citizens	Non-citizens
Share of claims flagged
Flagged claims that the records call fraud
Test loss on the records	–	–

What only a simulation can see: the truth

This page made up the claims, so it knows which ones really were fraud. In the simulation, real fraud depends only on income irregularity and earlier corrections, the same way for everyone. But past investigators checked non-citizens' claims far more often, so the records find more of their fraud, and a few honest non-citizens were wrongly recorded as fraud.

	All claims	Citizens	Non-citizens
Flagged claims that really were fraud
Honest claimants who were flagged

Records the inputs and every number above, including the truth.

#	Inputs	Flagged: cit. / non-cit.	Records call fraud	Test loss	Really fraud	Honest flagged: cit. / non-cit.

2A · Judge it by the records

Train and record three models: (1) neither box checked, (2) postcode only, (3) postcode and citizenship. Don't open the truth yet. Judged only by what the agency can see, which model looks best? Are the differences between them large?
2B · Judge it by the truth

Now press Show the truth. Look at honest claimants who were flagged in each group, across your three models. What happened when you dropped citizenship but kept postcode? Why?
2C · No truth button

A real agency never gets a "Show the truth" button. Name one thing it could do to find out whether its model treats groups unfairly.

Part 3 · Back to Wednesday's question

Today's question

Is a model that predicts well good enough to use in policymaking?

Answer in 3 to 5 sentences, in your own words. Use at least one thing you saw today (in Part 1 or Part 2), and say what "predicts well" was measured against.

Submit before you leave

Press Download my submission. You get a text file with your name, recorded runs and answers.
On Canvas, open Section 2 lab and upload the file. (Or press Copy and paste it into the text box.)
Do this before section ends. Missing items? Submit what you have; see the assignment page.

Preview

Going further (optional, not graded)

The whole neural network on this page is about 100 lines of plain JavaScript, in nn.js. Save it into a folder, open that folder in your coding agent, and ask:

Ask your agent

This folder has nn.js, a small neural network used in my political science section. I'm not a programmer. Walk me through the step() function in plain language: where is the error measured, where does "change every weight a little" happen, and what does the learning rate do? Point to line numbers. Don't change the file.

Remember last week's rule: the agent's explanation is a claim, not proof. If something sounds off, ask it to show you the line.

Words you'll need

Training data: The examples the model learns from.
Test data: Examples held back to check the model on cases it has never seen.
Label: The answer attached to an example (passed / failed, fraud / not fraud). Someone, or some process, produced it.
Loss: The model's error, as one number. Lower is better. Here: cross-entropy, which punishes confident wrong answers hardest.
Accuracy: The share of examples the model gets right.
Weight: A number inside the network that it adjusts while learning.
Epoch: One pass through all the training examples.
Overfitting: Fitting the training data so closely, noise included, that the model does worse on new data.
Proxy: An input that stands in for another one, like postcode standing in for citizenship.