Overfitting, Regularization & Occam's Razor

1. Data Fitting Lab: From Underfitting to Overfitting

True Function

Observed Data

Model Prediction Diagnosis: —

—

—

(Complexity)3

10^-2

1.8

Sample Size (Months)12

Hint: With 12 data points, an 11th-degree polynomial can pass through every point perfectly.

View & Complexity ()

How to Use:

Feel Complexity: Set "Regularization Strength λ" to its minimum, then slowly drag the "Polynomial Degree" slider from 0 to 11.
Observe Overfitting: As the degree increases, the "Training MSE" drops, but the curve becomes wildly distorted, causing the "Generalization MSE" to skyrocket.
Feel Regularization: At a high degree, increasing λ will "pull" the curve back to a smoother, simpler shape.
Understand Occam's Razor: Among multiple explanations, choose the simplest one (e.g., a 3rd-degree polynomial), as it often generalizes better.

True function on this page:
f(x) = 15 + 10·sin(2π·(x-3)/12)

The core of overfitting is ignoring "complexity". The answers to the following questions are rooted in psychology, not pure mathematics.

—

—

Hint: View the full sequence

The full sequence for n=1 to 8 is: 1, 2, 4, 8, 16, 31, 57, 99

Cramming for Exams: Memorizing specific details from past exams, treating exceptions as rules, and failing to adapt to new question formats.
Stereotypes: Forming complex, rigid opinions about a group based on limited encounters, ignoring individual variation.
Historical Explanations: Using a perfect theory to explain every past event as "inevitable," yet failing spectacularly when predicting the future.

Learning without thought → Underfitting: The model is too simple to capture the trend.
Thought without learning → Overfitting: Inventing complex theories that perform poorly on new data.
Allowing for error is a necessary cost: Chasing a "perfect explanation" of the past often sacrifices the ability to predict the future.