Audits
I picked the notebooks where a little more rigour could change the conclusion, and where the data is licence-clear and downloadable without an account. Six are re-run by a script. Two are worked out from the numbers the 2022 notebooks printed, one of them checked by a local script that publishes only counts. Cards marked Interactive have a tool you can try in the browser.
01 · 05 · 09 · 28 · 32
Conclusion flipsOn a time-ordered split the Apple LSTM misses the close by 38.7% on average, against 1.7% for 'tomorrow equals today'. The linear model in notebook 05 also loses to 'no change' (0.50% versus 0.36%).
Read the audit
36
Conclusion flipsThe 2022 call scored the predictions against themselves, which returns 1.0 for any model. Scored against the real prices, the same tree reaches R² 0.85, with a typical miss of $2,372.
Read the audit
27
InteractiveConclusion flipsThe tree's 0.98 implies 33 of 40 right on the unseen rows, about 82.5%. Logistic regression and the passive-aggressive model both scored exactly 257 of 400, which is what predicting 'no purchase' for everyone would score.
Read the audit
02
InteractiveCan't tell from 202299.97% is 381 mistakes on 1,272,524 test rows. That is consistent with catching every fraud, or with missing almost a quarter of them, and the features include balances the dataset's author says must not be used.
Read the audit
24
InteractiveWeaker than it looked'Always unsafe' already scores 59.7%. The best model reaches 68.8%, and about 32% of the samples it calls safe are not.
Read the audit
03 · 14
Weaker than it lookedAcross 1,000 random splits of the book's Advertising data, the same model scores between 0.75 and 0.96. Repeated cross-validation settles at 0.886.
Read the audit
23 · 15
InteractiveWeaker than it lookedThe 1-nearest-neighbour model got 30 of 30 test flowers right, but the 95% interval runs from 88.4% to 100%. Over 1,000 other splits only 23% were perfect.
Read the audit
37
InteractiveHolds upYes. It scores 98.4% against 86.1% for 'always ham', catching 96% of spam with 93% precision. Ten-fold cross-validation agrees.
Read the audit
Many notebooks use data from the tutorial author's GitHub repository with no stated licence, or sensitive text such as tweets and mental-health posts. Re-running those would mean republishing or processing data I cannot share, so their pages describe what happened without recomputing anything.