2 · Overfitting, and choosing a model

In Lesson 1 you trained one model and took its score at face value. This lesson asks the two questions a professional asks next: how complex should a model be, and which model should you deploy?

You will watch a decision tree reach a perfect score on its training data and then do worse than guessing on new learners. You will use cross-validation to choose a model without peeking at the test set. Then the ethics lab compares five models on fairness — and the answer to “does a better model treat people more fairly?” is not the one you expect.

⏳ The model-comparison cell trains a 100-tree random forest, so it takes a few seconds in the browser.

The notebook above runs entirely in your browser — nothing is installed and nothing leaves your device. Your changes are kept in this browser. First load takes a few seconds while Python starts.

Open it full-screen ↗↺ Reset this notebook

Checkpoint

• What does it look like when a model overfits — and why did the unlimited tree score worse than always predicting “pass”?
• Why must the test set play no part in choosing your model?
• Three models were within about a point of each other on accuracy. Were they also tied on fairness?
• For learners without a laptop, how did every trained model compare with simply predicting “pass” for everyone? What does that tell you?
• The EU AI Act asks for an “appropriate level of accuracy”. Whose accuracy?

Next: Capstone: audit and document a model.