One lucky split: R² 0.906 from 40 rows
How much of the 0.906 is the model and how much is the random split?
Short answer
Across 1,000 random splits of the book's Advertising data, the same model scores between 0.75 and 0.96. Repeated cross-validation settles at 0.886.
The split lottery
- Reported in 2022
- 0.906
- one split, 40 test rows
- Same seed, book data
- 0.899
- Range over 1,000 seeds
- 0.75 to 0.96
- Repeated 5-fold CV
- 0.886
- sd 0.041
With 200 rows, an 80/20 split tests the model on just 40 markets. Which 40 is decided by the random seed, and the seed changes the score more than any modelling choice in the notebook. Re-running the identical model with 1,000 different seeds gives the spread below.
Test R² of the same linear regression over 1,000 random 80/20 splits
Nine in ten splits score between 0.81 and 0.93. Seed 42, the one the notebook used, sits at the 57th percentile on the book's data.
Show the numbers as a table
| Bin | Count |
|---|---|
| 0.74 to 0.75 | 1 |
| 0.75 to 0.76 | 3 |
| 0.76 to 0.77 | 4 |
| 0.77 to 0.78 | 1 |
| 0.78 to 0.79 | 6 |
| 0.79 to 0.80 | 17 |
| 0.80 to 0.81 | 13 |
| 0.81 to 0.82 | 15 |
| 0.82 to 0.83 | 15 |
| 0.83 to 0.84 | 35 |
| 0.84 to 0.85 | 62 |
| 0.85 to 0.86 | 61 |
| 0.86 to 0.87 | 68 |
| 0.87 to 0.88 | 71 |
| 0.88 to 0.89 | 99 |
| 0.89 to 0.90 | 109 |
| 0.90 to 0.91 | 137 |
| 0.91 to 0.92 | 125 |
| 0.92 to 0.93 | 94 |
| 0.93 to 0.94 | 52 |
| 0.94 to 0.95 | 9 |
| 0.95 to 0.96 | 3 |
Repeated cross-validation tests every market once per repeat and averages over 100 test folds, so it gives a steadier 0.886. The model is good. The third decimal place of a single split was never meaningful.
What the model says about each channel
| Budget | Coefficient | Plain reading |
|---|---|---|
| TV | 0.0458 | about 46 more units per extra $1,000 |
| Radio | 0.1885 | about 189 more units per extra $1,000 |
| Newspaper | -0.0010 | no measurable effect |
Newspaper adds nothing once TV and radio are known. Dropping it leaves cross-validated R² at 0.888, against 0.886 with it.
Notebook 14 drew the TV-only line on the same book data. Its intercept of 7.03 and slope of 0.0475 match the book, and TV alone explains 61% of the variation in sales. Adding radio is what takes the model from good to very good.
Intervals and a paired test
| Measure | Estimate | 95% interval | Method | Based on |
|---|---|---|---|---|
| R², 2022 split on the book's data | Estimate 0.899 | 95% 0.839 to 0.929 | Percentile bootstrap | 40 test markets |
| R², 5-fold × 20 | Estimate 0.886 | 95% 0.845 to 0.928 | Corrected resampled t | 100 folds |
Corrected resampled t
Dropping newspaper, on the same 100 folds
- Folds
- 100
- Mean difference
- +0.001
- 95% interval
- −0.001 to +0.003
- Corrected t
- p = 0.23
Without newspaper the model scores slightly higher on average, by an amount the folds cannot tell from zero. Newspaper budgets add nothing measurable once TV and radio are known.
Split seed 42 (the 2022 notebook's), split lottery seeds 0 to 999, cross-validation seed 0, bootstrap seed 2026 with 10,000 resamples. How each interval and test works is set out on the methods page.