Stock forecasts against 'tomorrow equals today'
Do the 2022 price models beat the simplest possible forecast once the test days come after the training days?
Short answer
On a time-ordered split the Apple LSTM misses the close by 38.7% on average, against 1.7% for 'tomorrow equals today'. The linear model in notebook 05 also loses to 'no change' (0.50% versus 0.36%).
Five forecasts, one question
Every price model in the log was judged in 2022 by a score that never asked the obvious question: would “tomorrow's price equals today's price” have done just as well? That flat forecast costs nothing and is famously hard to beat for prices. Here each model is tested the way a forecast is used, on days after the ones it learnt from.
| Notebook | Model MAPE | Flat MAPE | Beats flat? |
|---|---|---|---|
| 01 LSTM, Apple | 38.69% | 1.67% | No |
| 09 LSTM, Netflix | 14.57% | 2.17% | No |
| 05 Linear, five days ahead | 0.50% | 0.36% | No |
| 28 AutoTS, Tata Motors | 4.87% | 2.70% | No |
| 32 AutoTS, Apple | 3.01% | 2.18% | No |
MAPE is the average absolute error as a percentage of the true price. The flat forecast repeats the last known close. For 28 and 32 the numbers are averages over four five-day windows, so treat them as indicative.
01 and 09: the LSTMs
In 2022 both notebooks fed the same day's open, high, low and volume into the network and asked for that day's close, after shuffling the days. That set-up has two problems. The high and low already bracket the close, so it is not a forecast at all. And shuffling lets the network learn from days on both sides of each test day. The test rows were never scored, so I scored them now, then repeated the training with the last 20% of days held out in order.
The re-run reproduces the 2022 training closely: Apple's loss ends at 3.31 against 3.62 in 2022.
01 AAPL, 3,447 trading days
- LSTM
- 1.00%
- Yesterday's close
- 1.29%
- (High + low) / 2same inputs, no training
- 0.59%
- LSTM
- 38.69%
- Yesterday's close
- 1.67%
- (High + low) / 2
- 0.80%
AAPL: absolute error as a share of the close, 20-day rolling mean
Test period 2020-02-05 to 2022-10-28. The network never saw a close above $81.09 during training, and 89% of the test days were above that.
- LSTM (time-ordered)
- Yesterday's close
Show the numbers as a table
| Date | LSTM (time-ordered) | Yesterday's close |
|---|---|---|
| 2020-02-05 | 0.6% | 0.8% |
| 2020-02-10 | 1.0% | 1.0% |
| 2020-02-12 | 1.0% | 1.2% |
| 2020-02-18 | 1.0% | 1.1% |
| 2020-02-21 | 1.0% | 1.2% |
| 2020-02-25 | 1.2% | 1.6% |
| 2020-02-28 | 1.4% | 1.9% |
| 2020-03-04 | 1.5% | 2.4% |
| 2020-03-06 | 1.6% | 2.5% |
| 2020-03-11 | 1.9% | 3.3% |
| 2020-03-16 | 2.4% | 5.0% |
| 2020-03-18 | 2.6% | 5.2% |
| 2020-03-23 | 3.3% | 5.2% |
| 2020-03-26 | 3.2% | 5.4% |
| 2020-03-30 | 3.5% | 5.3% |
| 2020-04-02 | 3.8% | 5.1% |
| 2020-04-07 | 4.1% | 4.8% |
| 2020-04-09 | 3.8% | 4.2% |
| 2020-04-15 | 3.6% | 3.1% |
| 2020-04-20 | 3.2% | 2.9% |
| 2020-04-22 | 3.4% | 2.6% |
| 2020-04-27 | 3.1% | 2.3% |
| 2020-04-30 | 2.8% | 2.2% |
| 2020-05-04 | 2.6% | 2.2% |
| 2020-05-07 | 2.3% | 1.8% |
| 2020-05-12 | 2.1% | 1.6% |
| 2020-05-14 | 2.0% | 1.6% |
| 2020-05-19 | 1.5% | 1.5% |
| 2020-05-21 | 1.2% | 1.5% |
| 2020-05-27 | 0.9% | 1.3% |
| 2020-06-01 | 0.7% | 1.0% |
| 2020-06-03 | 0.7% | 0.9% |
| 2020-06-08 | 0.9% | 1.0% |
| 2020-06-11 | 1.4% | 1.3% |
| 2020-06-15 | 1.7% | 1.3% |
| 2020-06-18 | 2.4% | 1.2% |
| 2020-06-23 | 3.3% | 1.4% |
| 2020-06-25 | 4.0% | 1.5% |
| 2020-06-30 | 4.9% | 1.7% |
| 2020-07-06 | 5.9% | 1.7% |
| 2020-07-08 | 6.5% | 1.6% |
| 2020-07-13 | 7.6% | 1.3% |
| 2020-07-16 | 8.7% | 1.2% |
| 2020-07-20 | 9.4% | 1.3% |
| 2020-07-23 | 10.0% | 1.3% |
| 2020-07-28 | 10.3% | 1.2% |
| 2020-07-30 | 10.7% | 1.3% |
| 2020-08-04 | 13.2% | 1.8% |
| 2020-08-07 | 16.0% | 1.9% |
| 2020-08-11 | 17.8% | 2.1% |
| 2020-08-14 | 20.8% | 2.2% |
| 2020-08-19 | 23.9% | 2.1% |
| 2020-08-21 | 26.7% | 2.2% |
| 2020-08-26 | 31.2% | 2.1% |
| 2020-08-31 | 34.1% | 1.6% |
| 2020-09-02 | 35.7% | 1.9% |
| 2020-09-08 | 36.3% | 2.3% |
| 2020-09-11 | 36.6% | 2.3% |
| 2020-09-15 | 36.7% | 2.5% |
| 2020-09-18 | 35.8% | 2.7% |
| 2020-09-23 | 34.0% | 2.8% |
| 2020-09-25 | 32.8% | 2.9% |
| 2020-09-30 | 31.5% | 2.8% |
| 2020-10-05 | 30.7% | 2.6% |
| 2020-10-07 | 30.6% | 2.3% |
| 2020-10-12 | 31.2% | 2.3% |
| 2020-10-15 | 32.1% | 2.2% |
| 2020-10-19 | 33.0% | 2.1% |
| 2020-10-22 | 34.1% | 1.9% |
| 2020-10-27 | 34.3% | 1.7% |
| 2020-10-29 | 34.1% | 2.0% |
| 2020-11-03 | 33.4% | 1.9% |
| 2020-11-06 | 33.6% | 2.1% |
| 2020-11-10 | 33.2% | 1.8% |
| 2020-11-13 | 33.1% | 1.8% |
| 2020-11-18 | 33.3% | 1.7% |
| 2020-11-20 | 33.5% | 1.7% |
| 2020-11-25 | 33.6% | 1.7% |
| 2020-11-30 | 34.1% | 1.3% |
| 2020-12-03 | 35.5% | 1.2% |
| 2020-12-08 | 36.0% | 1.1% |
| 2020-12-10 | 36.3% | 1.1% |
| 2020-12-15 | 36.7% | 1.3% |
| 2020-12-18 | 37.4% | 1.3% |
| 2020-12-22 | 38.3% | 1.3% |
| 2020-12-28 | 39.7% | 1.4% |
| 2020-12-31 | 40.7% | 1.3% |
| 2021-01-05 | 41.1% | 1.5% |
| 2021-01-08 | 41.6% | 1.6% |
| 2021-01-13 | 42.1% | 1.7% |
| 2021-01-15 | 42.1% | 1.6% |
| 2021-01-21 | 42.5% | 1.8% |
| 2021-01-26 | 43.1% | 1.8% |
| 2021-01-28 | 43.3% | 1.8% |
| 2021-02-02 | 43.4% | 1.9% |
| 2021-02-05 | 43.9% | 1.7% |
| 2021-02-09 | 44.2% | 1.6% |
| 2021-02-12 | 44.6% | 1.4% |
| 2021-02-18 | 44.7% | 1.4% |
| 2021-02-22 | 44.3% | 1.3% |
| 2021-02-25 | 43.0% | 1.3% |
| 2021-03-02 | 42.2% | 1.3% |
| 2021-03-04 | 41.5% | 1.4% |
| 2021-03-09 | 40.2% | 1.7% |
| 2021-03-12 | 39.1% | 1.8% |
| 2021-03-16 | 38.6% | 1.9% |
| 2021-03-19 | 37.9% | 2.0% |
| 2021-03-24 | 37.6% | 2.1% |
| 2021-03-26 | 37.6% | 1.9% |
| 2021-03-31 | 37.3% | 1.6% |
| 2021-04-06 | 37.8% | 1.4% |
| 2021-04-08 | 38.3% | 1.3% |
| 2021-04-13 | 39.1% | 1.4% |
| 2021-04-16 | 39.9% | 1.3% |
| 2021-04-20 | 40.5% | 1.2% |
| 2021-04-23 | 41.5% | 1.2% |
| 2021-04-28 | 42.5% | 1.2% |
| 2021-04-30 | 43.0% | 1.2% |
| 2021-05-05 | 43.2% | 1.2% |
| 2021-05-10 | 43.1% | 1.2% |
| 2021-05-12 | 42.6% | 1.1% |
| 2021-05-17 | 42.0% | 1.2% |
| 2021-05-20 | 41.5% | 1.2% |
| 2021-05-24 | 41.1% | 1.3% |
| 2021-05-27 | 40.5% | 1.3% |
| 2021-06-02 | 40.1% | 1.1% |
| 2021-06-04 | 39.9% | 1.2% |
| 2021-06-09 | 39.8% | 1.0% |
| 2021-06-14 | 40.0% | 0.9% |
| 2021-06-16 | 40.2% | 0.9% |
| 2021-06-21 | 40.7% | 0.9% |
| 2021-06-23 | 41.0% | 0.9% |
| 2021-06-28 | 41.6% | 0.8% |
| 2021-07-01 | 42.5% | 0.8% |
| 2021-07-06 | 43.2% | 0.9% |
| 2021-07-09 | 44.4% | 1.0% |
| 2021-07-14 | 45.5% | 1.0% |
| 2021-07-16 | 46.2% | 1.0% |
| 2021-07-21 | 47.0% | 1.1% |
| 2021-07-26 | 47.9% | 1.2% |
| 2021-07-28 | 48.3% | 1.2% |
| 2021-08-02 | 48.8% | 1.1% |
| 2021-08-05 | 49.0% | 1.0% |
| 2021-08-09 | 49.0% | 1.0% |
| 2021-08-12 | 49.0% | 0.9% |
| 2021-08-17 | 49.3% | 0.7% |
| 2021-08-19 | 49.3% | 0.7% |
| 2021-08-24 | 49.3% | 0.7% |
| 2021-08-27 | 49.5% | 0.7% |
| 2021-08-31 | 49.7% | 0.8% |
| 2021-09-03 | 50.0% | 0.9% |
| 2021-09-09 | 50.5% | 1.0% |
| 2021-09-13 | 50.5% | 1.1% |
| 2021-09-16 | 50.5% | 0.9% |
| 2021-09-21 | 50.2% | 1.0% |
| 2021-09-23 | 50.1% | 1.1% |
| 2021-09-28 | 49.9% | 1.1% |
| 2021-10-01 | 49.3% | 1.1% |
| 2021-10-05 | 48.8% | 1.2% |
| 2021-10-08 | 48.3% | 1.0% |
| 2021-10-13 | 47.9% | 1.0% |
| 2021-10-15 | 47.8% | 1.0% |
| 2021-10-20 | 48.0% | 1.0% |
| 2021-10-25 | 48.1% | 0.9% |
| 2021-10-27 | 48.4% | 0.8% |
| 2021-11-01 | 48.9% | 0.8% |
| 2021-11-04 | 49.4% | 0.8% |
| 2021-11-08 | 49.7% | 0.8% |
| 2021-11-11 | 50.0% | 0.8% |
| 2021-11-16 | 50.2% | 0.7% |
| 2021-11-18 | 50.4% | 0.9% |
| 2021-11-23 | 51.0% | 0.9% |
| 2021-11-29 | 51.4% | 1.0% |
| 2021-12-01 | 51.8% | 1.1% |
| 2021-12-06 | 52.4% | 1.2% |
| 2021-12-09 | 53.4% | 1.4% |
| 2021-12-13 | 54.2% | 1.6% |
| 2021-12-16 | 55.1% | 1.8% |
| 2021-12-21 | 55.6% | 1.8% |
| 2021-12-23 | 56.0% | 1.8% |
| 2021-12-29 | 56.7% | 1.5% |
| 2022-01-03 | 57.3% | 1.6% |
| 2022-01-05 | 57.5% | 1.5% |
| 2022-01-10 | 57.4% | 1.4% |
| 2022-01-12 | 57.4% | 1.3% |
| 2022-01-18 | 57.3% | 1.1% |
| 2022-01-21 | 57.0% | 1.2% |
| 2022-01-25 | 56.5% | 1.1% |
| 2022-01-28 | 55.9% | 1.4% |
| 2022-02-02 | 55.7% | 1.4% |
| 2022-02-04 | 55.7% | 1.2% |
| 2022-02-09 | 55.8% | 1.3% |
| 2022-02-14 | 55.6% | 1.4% |
| 2022-02-16 | 55.7% | 1.3% |
| 2022-02-22 | 55.9% | 1.4% |
| 2022-02-25 | 56.0% | 1.6% |
| 2022-03-01 | 55.8% | 1.2% |
| 2022-03-04 | 55.5% | 1.3% |
| 2022-03-09 | 54.9% | 1.5% |
| 2022-03-11 | 54.5% | 1.6% |
| 2022-03-16 | 53.8% | 1.8% |
| 2022-03-21 | 53.6% | 1.9% |
| 2022-03-23 | 53.8% | 1.8% |
| 2022-03-28 | 54.2% | 1.8% |
| 2022-03-31 | 54.6% | 1.8% |
| 2022-04-04 | 55.0% | 1.7% |
| 2022-04-07 | 55.6% | 1.6% |
| 2022-04-12 | 56.2% | 1.4% |
| 2022-04-14 | 56.4% | 1.5% |
| 2022-04-20 | 56.4% | 1.3% |
| 2022-04-25 | 56.1% | 1.3% |
| 2022-04-27 | 55.5% | 1.4% |
| 2022-05-02 | 54.9% | 1.7% |
| 2022-05-05 | 54.3% | 1.9% |
| 2022-05-09 | 53.8% | 2.0% |
| 2022-05-12 | 52.9% | 2.3% |
| 2022-05-17 | 52.0% | 2.4% |
| 2022-05-19 | 51.1% | 2.8% |
| 2022-05-24 | 50.1% | 2.7% |
| 2022-05-27 | 49.3% | 2.6% |
| 2022-06-01 | 49.0% | 2.6% |
| 2022-06-06 | 48.5% | 2.4% |
| 2022-06-09 | 48.2% | 2.1% |
| 2022-06-13 | 47.8% | 2.3% |
| 2022-06-16 | 47.0% | 2.1% |
| 2022-06-22 | 46.7% | 2.0% |
| 2022-06-24 | 46.7% | 2.1% |
| 2022-06-29 | 46.3% | 2.0% |
| 2022-07-05 | 45.7% | 2.0% |
| 2022-07-07 | 45.6% | 2.1% |
| 2022-07-12 | 45.8% | 1.8% |
| 2022-07-15 | 46.7% | 1.6% |
| 2022-07-19 | 47.5% | 1.6% |
| 2022-07-22 | 48.5% | 1.5% |
| 2022-07-26 | 48.9% | 1.4% |
| 2022-07-29 | 50.0% | 1.5% |
| 2022-08-03 | 51.0% | 1.5% |
| 2022-08-05 | 51.6% | 1.4% |
| 2022-08-10 | 52.6% | 1.4% |
| 2022-08-15 | 53.6% | 1.3% |
| 2022-08-17 | 54.2% | 1.2% |
| 2022-08-22 | 54.9% | 1.2% |
| 2022-08-25 | 55.5% | 1.1% |
| 2022-08-29 | 55.5% | 1.2% |
| 2022-09-01 | 55.2% | 1.1% |
| 2022-09-07 | 54.8% | 1.2% |
| 2022-09-09 | 54.4% | 1.2% |
| 2022-09-14 | 53.8% | 1.6% |
| 2022-09-19 | 52.9% | 1.7% |
| 2022-09-21 | 52.6% | 1.8% |
| 2022-09-26 | 51.9% | 1.6% |
| 2022-09-29 | 51.3% | 1.8% |
| 2022-10-03 | 50.8% | 2.0% |
| 2022-10-06 | 50.3% | 2.0% |
| 2022-10-11 | 49.4% | 1.7% |
| 2022-10-13 | 48.9% | 1.7% |
| 2022-10-18 | 48.2% | 1.8% |
| 2022-10-21 | 47.8% | 1.8% |
| 2022-10-25 | 47.8% | 1.9% |
| 2022-10-28 | 48.2% | 2.0% |
09 NFLX, 3,449 trading days
- LSTM
- 4.09%
- Yesterday's close
- 2.04%
- (High + low) / 2same inputs, no training
- 0.99%
- LSTM
- 14.57%
- Yesterday's close
- 2.17%
- (High + low) / 2
- 1.02%
NFLX: absolute error as a share of the close, 20-day rolling mean
Test period 2020-02-07 to 2022-11-01. The network never saw a close above $418.97 during training, and 64% of the test days were above that.
- LSTM (time-ordered)
- Yesterday's close
Show the numbers as a table
| Date | LSTM (time-ordered) | Yesterday's close |
|---|---|---|
| 2020-02-07 | 0.2% | 0.0% |
| 2020-02-12 | 1.1% | 0.9% |
| 2020-02-14 | 1.3% | 0.7% |
| 2020-02-20 | 1.8% | 0.7% |
| 2020-02-25 | 1.6% | 1.1% |
| 2020-02-27 | 1.6% | 1.5% |
| 2020-03-03 | 1.7% | 1.6% |
| 2020-03-06 | 1.7% | 1.8% |
| 2020-03-10 | 1.8% | 2.3% |
| 2020-03-13 | 2.0% | 3.2% |
| 2020-03-18 | 2.3% | 4.1% |
| 2020-03-20 | 2.3% | 4.3% |
| 2020-03-25 | 2.3% | 4.4% |
| 2020-03-30 | 2.3% | 4.7% |
| 2020-04-01 | 2.2% | 4.5% |
| 2020-04-06 | 2.4% | 4.4% |
| 2020-04-09 | 2.3% | 3.6% |
| 2020-04-14 | 2.5% | 3.2% |
| 2020-04-17 | 2.9% | 3.0% |
| 2020-04-22 | 3.8% | 2.9% |
| 2020-04-24 | 4.3% | 2.5% |
| 2020-04-29 | 4.9% | 2.5% |
| 2020-05-04 | 5.8% | 2.5% |
| 2020-05-06 | 6.3% | 2.3% |
| 2020-05-11 | 7.1% | 2.0% |
| 2020-05-14 | 7.2% | 1.7% |
| 2020-05-18 | 7.5% | 1.5% |
| 2020-05-21 | 7.9% | 1.5% |
| 2020-05-26 | 7.8% | 1.7% |
| 2020-05-29 | 7.9% | 1.5% |
| 2020-06-03 | 8.0% | 1.4% |
| 2020-06-05 | 7.8% | 1.4% |
| 2020-06-10 | 7.8% | 1.4% |
| 2020-06-15 | 7.3% | 1.4% |
| 2020-06-17 | 7.2% | 1.7% |
| 2020-06-22 | 7.6% | 1.6% |
| 2020-06-25 | 8.5% | 1.5% |
| 2020-06-29 | 8.8% | 1.7% |
| 2020-07-02 | 9.9% | 2.0% |
| 2020-07-08 | 11.4% | 2.0% |
| 2020-07-10 | 12.9% | 2.3% |
| 2020-07-15 | 15.1% | 2.3% |
| 2020-07-20 | 16.5% | 2.6% |
| 2020-07-22 | 16.8% | 2.5% |
| 2020-07-27 | 17.5% | 2.4% |
| 2020-07-30 | 18.0% | 2.1% |
| 2020-08-03 | 18.1% | 1.9% |
| 2020-08-06 | 18.3% | 2.0% |
| 2020-08-11 | 17.0% | 1.9% |
| 2020-08-13 | 16.3% | 2.0% |
| 2020-08-18 | 16.0% | 1.5% |
| 2020-08-21 | 16.2% | 1.6% |
| 2020-08-25 | 16.2% | 1.5% |
| 2020-08-28 | 17.3% | 2.1% |
| 2020-09-02 | 18.3% | 2.2% |
| 2020-09-04 | 18.6% | 2.3% |
| 2020-09-10 | 19.1% | 2.3% |
| 2020-09-15 | 19.2% | 2.5% |
| 2020-09-17 | 19.0% | 2.6% |
| 2020-09-22 | 18.8% | 2.6% |
| 2020-09-25 | 17.7% | 2.2% |
| 2020-09-29 | 17.1% | 2.2% |
| 2020-10-02 | 16.4% | 2.2% |
| 2020-10-07 | 16.6% | 2.6% |
| 2020-10-09 | 17.5% | 2.5% |
| 2020-10-14 | 18.9% | 2.3% |
| 2020-10-19 | 20.3% | 2.1% |
| 2020-10-21 | 20.7% | 2.3% |
| 2020-10-26 | 20.8% | 2.1% |
| 2020-10-29 | 20.5% | 2.0% |
| 2020-11-02 | 20.0% | 2.0% |
| 2020-11-05 | 19.4% | 1.8% |
| 2020-11-10 | 18.1% | 2.2% |
| 2020-11-12 | 17.3% | 2.2% |
| 2020-11-17 | 16.1% | 2.2% |
| 2020-11-20 | 16.0% | 1.8% |
| 2020-11-24 | 15.9% | 2.0% |
| 2020-11-30 | 15.9% | 1.6% |
| 2020-12-02 | 16.2% | 1.6% |
| 2020-12-07 | 16.1% | 1.6% |
| 2020-12-10 | 16.6% | 1.2% |
| 2020-12-14 | 17.0% | 1.3% |
| 2020-12-17 | 18.2% | 1.4% |
| 2020-12-22 | 19.3% | 1.3% |
| 2020-12-24 | 19.8% | 1.3% |
| 2020-12-30 | 20.5% | 1.3% |
| 2021-01-05 | 21.2% | 1.6% |
| 2021-01-07 | 21.0% | 1.7% |
| 2021-01-12 | 21.1% | 1.6% |
| 2021-01-15 | 20.6% | 1.5% |
| 2021-01-20 | 20.7% | 2.2% |
| 2021-01-25 | 21.6% | 2.3% |
| 2021-01-28 | 22.1% | 2.7% |
| 2021-02-01 | 22.1% | 2.6% |
| 2021-02-04 | 22.8% | 2.5% |
| 2021-02-09 | 23.9% | 2.4% |
| 2021-02-11 | 24.8% | 2.3% |
| 2021-02-17 | 26.0% | 2.2% |
| 2021-02-22 | 25.3% | 1.5% |
| 2021-02-24 | 25.1% | 1.6% |
| 2021-03-01 | 25.4% | 1.2% |
| 2021-03-04 | 25.1% | 1.4% |
| 2021-03-08 | 24.4% | 1.5% |
| 2021-03-11 | 23.4% | 1.7% |
| 2021-03-16 | 22.6% | 1.7% |
| 2021-03-18 | 22.1% | 1.8% |
| 2021-03-23 | 21.7% | 1.9% |
| 2021-03-26 | 20.9% | 2.1% |
| 2021-03-30 | 20.4% | 2.0% |
| 2021-04-05 | 20.7% | 1.8% |
| 2021-04-08 | 21.8% | 1.6% |
| 2021-04-12 | 22.3% | 1.4% |
| 2021-04-15 | 22.8% | 1.5% |
| 2021-04-20 | 23.7% | 1.3% |
| 2021-04-22 | 23.4% | 1.5% |
| 2021-04-27 | 23.3% | 1.3% |
| 2021-04-30 | 23.0% | 1.1% |
| 2021-05-04 | 22.4% | 1.2% |
| 2021-05-07 | 21.2% | 1.2% |
| 2021-05-12 | 19.8% | 1.4% |
| 2021-05-14 | 18.9% | 1.4% |
| 2021-05-19 | 17.8% | 1.0% |
| 2021-05-24 | 17.6% | 1.1% |
| 2021-05-26 | 17.5% | 1.1% |
| 2021-06-01 | 17.3% | 1.1% |
| 2021-06-04 | 17.2% | 1.1% |
| 2021-06-08 | 17.1% | 0.9% |
| 2021-06-11 | 17.1% | 0.7% |
| 2021-06-16 | 17.2% | 0.8% |
| 2021-06-18 | 17.3% | 0.7% |
| 2021-06-23 | 17.4% | 0.8% |
| 2021-06-25 | 17.8% | 0.9% |
| 2021-06-30 | 18.5% | 1.0% |
| 2021-07-06 | 19.5% | 1.0% |
| 2021-07-08 | 20.2% | 1.0% |
| 2021-07-13 | 21.3% | 0.9% |
| 2021-07-16 | 22.4% | 1.0% |
| 2021-07-20 | 22.9% | 1.0% |
| 2021-07-23 | 22.9% | 1.0% |
| 2021-07-28 | 22.6% | 0.9% |
| 2021-07-30 | 22.4% | 0.9% |
| 2021-08-04 | 21.9% | 0.9% |
| 2021-08-09 | 21.6% | 0.9% |
| 2021-08-11 | 21.2% | 0.9% |
| 2021-08-16 | 20.7% | 0.8% |
| 2021-08-19 | 20.9% | 0.8% |
| 2021-08-23 | 21.4% | 0.8% |
| 2021-08-26 | 22.1% | 0.8% |
| 2021-08-31 | 23.2% | 0.9% |
| 2021-09-02 | 24.1% | 0.9% |
| 2021-09-08 | 25.7% | 1.0% |
| 2021-09-13 | 27.3% | 1.1% |
| 2021-09-15 | 28.2% | 1.2% |
| 2021-09-20 | 29.1% | 1.1% |
| 2021-09-23 | 29.8% | 1.2% |
| 2021-09-27 | 30.3% | 1.1% |
| 2021-09-30 | 30.6% | 1.2% |
| 2021-10-05 | 30.8% | 1.3% |
| 2021-10-07 | 30.9% | 1.3% |
| 2021-10-12 | 31.3% | 1.2% |
| 2021-10-15 | 31.7% | 1.2% |
| 2021-10-19 | 32.2% | 1.2% |
| 2021-10-22 | 32.6% | 1.4% |
| 2021-10-27 | 33.2% | 1.3% |
| 2021-10-29 | 33.6% | 1.4% |
| 2021-11-03 | 34.0% | 1.2% |
| 2021-11-08 | 34.2% | 1.5% |
| 2021-11-10 | 34.3% | 1.5% |
| 2021-11-15 | 34.6% | 1.7% |
| 2021-11-18 | 35.0% | 1.5% |
| 2021-11-22 | 34.9% | 1.5% |
| 2021-11-26 | 34.8% | 1.5% |
| 2021-12-01 | 34.4% | 1.7% |
| 2021-12-03 | 34.0% | 1.6% |
| 2021-12-08 | 33.8% | 1.5% |
| 2021-12-13 | 33.4% | 1.4% |
| 2021-12-15 | 33.0% | 1.4% |
| 2021-12-20 | 32.3% | 1.5% |
| 2021-12-23 | 32.0% | 1.5% |
| 2021-12-28 | 31.8% | 1.4% |
| 2021-12-31 | 31.8% | 1.2% |
| 2022-01-05 | 31.5% | 1.1% |
| 2022-01-07 | 30.8% | 1.2% |
| 2022-01-12 | 29.7% | 1.2% |
| 2022-01-14 | 28.7% | 1.2% |
| 2022-01-20 | 27.0% | 1.3% |
| 2022-01-25 | 22.8% | 3.0% |
| 2022-01-27 | 19.7% | 3.4% |
| 2022-02-01 | 16.3% | 4.2% |
| 2022-02-04 | 12.9% | 4.5% |
| 2022-02-08 | 10.9% | 4.4% |
| 2022-02-11 | 8.2% | 4.6% |
| 2022-02-16 | 5.7% | 4.7% |
| 2022-02-18 | 4.7% | 3.4% |
| 2022-02-24 | 4.7% | 3.5% |
| 2022-03-01 | 4.5% | 2.8% |
| 2022-03-03 | 3.6% | 2.4% |
| 2022-03-08 | 3.0% | 2.3% |
| 2022-03-11 | 2.3% | 2.6% |
| 2022-03-15 | 2.1% | 2.7% |
| 2022-03-18 | 2.0% | 2.8% |
| 2022-03-23 | 2.1% | 2.7% |
| 2022-03-25 | 1.7% | 2.5% |
| 2022-03-30 | 1.6% | 2.6% |
| 2022-04-04 | 1.9% | 2.5% |
| 2022-04-06 | 1.9% | 2.5% |
| 2022-04-11 | 1.9% | 2.3% |
| 2022-04-14 | 1.7% | 2.0% |
| 2022-04-19 | 1.7% | 2.0% |
| 2022-04-22 | 1.5% | 4.8% |
| 2022-04-27 | 1.4% | 5.2% |
| 2022-04-29 | 1.6% | 5.5% |
| 2022-05-04 | 1.6% | 5.4% |
| 2022-05-09 | 1.8% | 5.9% |
| 2022-05-11 | 2.0% | 6.2% |
| 2022-05-16 | 2.3% | 6.6% |
| 2022-05-19 | 2.2% | 4.2% |
| 2022-05-23 | 2.2% | 4.1% |
| 2022-05-26 | 2.0% | 3.8% |
| 2022-06-01 | 1.9% | 3.6% |
| 2022-06-03 | 1.8% | 3.5% |
| 2022-06-08 | 1.7% | 3.1% |
| 2022-06-13 | 1.5% | 3.1% |
| 2022-06-15 | 1.4% | 3.4% |
| 2022-06-21 | 1.5% | 3.2% |
| 2022-06-24 | 1.4% | 3.3% |
| 2022-06-28 | 1.5% | 3.4% |
| 2022-07-01 | 1.3% | 3.2% |
| 2022-07-07 | 1.4% | 3.3% |
| 2022-07-11 | 1.4% | 3.3% |
| 2022-07-14 | 1.4% | 2.7% |
| 2022-07-19 | 1.6% | 2.8% |
| 2022-07-21 | 1.8% | 3.0% |
| 2022-07-26 | 2.0% | 2.8% |
| 2022-07-28 | 1.9% | 2.8% |
| 2022-08-02 | 2.1% | 2.6% |
| 2022-08-05 | 2.3% | 2.6% |
| 2022-08-09 | 2.3% | 2.5% |
| 2022-08-12 | 2.7% | 2.4% |
| 2022-08-17 | 3.0% | 2.0% |
| 2022-08-19 | 3.1% | 1.9% |
| 2022-08-24 | 3.3% | 1.9% |
| 2022-08-29 | 3.6% | 2.2% |
| 2022-08-31 | 3.7% | 2.1% |
| 2022-09-06 | 3.7% | 2.3% |
| 2022-09-09 | 3.4% | 2.3% |
| 2022-09-13 | 3.3% | 2.6% |
| 2022-09-16 | 2.8% | 2.8% |
| 2022-09-21 | 3.0% | 2.6% |
| 2022-09-23 | 3.1% | 2.6% |
| 2022-09-28 | 2.9% | 2.8% |
| 2022-10-03 | 3.1% | 2.7% |
| 2022-10-05 | 3.2% | 2.5% |
| 2022-10-10 | 3.3% | 2.8% |
| 2022-10-13 | 3.4% | 2.7% |
| 2022-10-17 | 3.6% | 2.9% |
| 2022-10-20 | 3.4% | 3.5% |
| 2022-10-25 | 3.2% | 3.9% |
| 2022-10-27 | 3.0% | 3.5% |
| 2022-11-01 | 2.5% | 3.5% |
On the shuffled split the Apple LSTM looks respectable, but simply averaging the day's high and low beats it with no training at all, and with four other seeds it does not even beat yesterday's close (see the seed table). On the time-ordered split it falls apart: prices in the test years sat far above anything in the training years, and a network trained on raw prices cannot extrapolate. Netflix shows the same pattern, and with this seed its LSTM loses to yesterday's close even on the shuffled split.
Apple: training loss by epoch, 2022 against the re-run (shuffled split)
Epoch 1 is left off so the rest is readable (229.6 in 2022, 237.9 now).
- 2022 run
- Re-run
Show the numbers as a table
| Epoch | 2022 run | Re-run |
|---|---|---|
| 2 | 10.41 | 14.76 |
| 3 | 10.51 | 11.62 |
| 4 | 8.77 | 8.30 |
| 5 | 7.80 | 10.56 |
| 6 | 9.17 | 11.08 |
| 7 | 7.17 | 7.27 |
| 8 | 5.98 | 7.86 |
| 9 | 5.67 | 6.75 |
| 10 | 6.84 | 8.45 |
| 11 | 7.09 | 5.19 |
| 12 | 6.24 | 5.39 |
| 13 | 4.54 | 5.26 |
| 14 | 4.67 | 5.74 |
| 15 | 4.73 | 5.66 |
| 16 | 5.55 | 5.05 |
| 17 | 5.51 | 5.07 |
| 18 | 6.29 | 4.82 |
| 19 | 5.04 | 5.37 |
| 20 | 4.42 | 5.03 |
| 21 | 5.38 | 4.45 |
| 22 | 4.99 | 5.29 |
| 23 | 5.38 | 4.43 |
| 24 | 4.23 | 3.73 |
| 25 | 4.31 | 3.94 |
| 26 | 5.40 | 3.81 |
| 27 | 4.06 | 4.21 |
| 28 | 4.43 | 3.22 |
| 29 | 4.16 | 3.84 |
| 30 | 3.62 | 3.31 |
05: a straight line, five days ahead
Notebook 05 scaled the closing price and fitted a line from today's value to the value five rows later, on shuffled rows. The script reproduces its R² of 0.6391 exactly. On those same shuffled rows the flat forecast scores 0.608, so the line was only slightly ahead.
Trained on the first 205 days and tested on the last 52, the line's R² drops to -0.008 and the flat forecast wins clearly, with an R² of 0.412.
Absolute error as a share of the price, last 52 days
Test period 2020-11-09 to 2021-01-19. Mean 0.50% for the line, 0.36% for the flat forecast.
- Linear regression
- Price in five days = today
Show the numbers as a table
| Date | Linear regression | Price in five days = today |
|---|---|---|
| 2020-11-09 | 0.47% | 0.68% |
| 2020-11-10 | 0.40% | 0.62% |
| 2020-11-11 | 0.11% | 0.27% |
| 2020-11-12 | 0.29% | 0.20% |
| 2020-11-13 | 1.06% | 1.03% |
| 2020-11-16 | 0.55% | 0.44% |
| 2020-11-17 | 0.36% | 0.23% |
| 2020-11-18 | 0.81% | 0.70% |
| 2020-11-19 | 0.99% | 0.87% |
| 2020-11-20 | 0.26% | 0.07% |
| 2020-11-23 | 0.48% | 0.30% |
| 2020-11-24 | 0.46% | 0.30% |
| 2020-11-25 | 0.64% | 0.42% |
| 2020-11-26 | 0.42% | 0.16% |
| 2020-11-27 | 0.45% | 0.25% |
| 2020-11-30 | 0.43% | 0.20% |
| 2020-12-01 | 0.21% | 0.01% |
| 2020-12-02 | 0.19% | 0.10% |
| 2020-12-03 | 0.08% | 0.37% |
| 2020-12-04 | 0.43% | 0.19% |
| 2020-12-07 | 0.33% | 0.07% |
| 2020-12-08 | 0.42% | 0.22% |
| 2020-12-09 | 0.23% | 0.05% |
| 2020-12-10 | 0.66% | 0.43% |
| 2020-12-11 | 0.54% | 0.27% |
| 2020-12-14 | 0.45% | 0.18% |
| 2020-12-15 | 0.14% | 0.10% |
| 2020-12-16 | 0.42% | 0.69% |
| 2020-12-17 | 0.01% | 0.30% |
| 2020-12-18 | 0.28% | 0.04% |
| 2020-12-21 | 0.18% | 0.12% |
| 2020-12-22 | 0.76% | 0.53% |
| 2020-12-23 | 1.27% | 1.11% |
| 2020-12-24 | 1.21% | 0.97% |
| 2020-12-25 | 0.95% | 0.64% |
| 2020-12-28 | 1.10% | 0.82% |
| 2020-12-29 | 0.62% | 0.31% |
| 2020-12-30 | 0.44% | 0.11% |
| 2020-12-31 | 0.09% | 0.31% |
| 2021-01-01 | 0.08% | 0.49% |
| 2021-01-04 | 0.03% | 0.38% |
| 2021-01-05 | 0.10% | 0.26% |
| 2021-01-06 | 0.64% | 0.29% |
| 2021-01-07 | 0.61% | 0.26% |
| 2021-01-08 | 0.82% | 0.49% |
| 2021-01-11 | 0.64% | 0.29% |
| 2021-01-12 | 0.76% | 0.44% |
| 2021-01-13 | 0.36% | 0.04% |
| 2021-01-14 | 0.75% | 0.36% |
| 2021-01-15 | 0.59% | 0.18% |
| 2021-01-18 | 0.62% | 0.22% |
| 2021-01-19 | 0.69% | 0.30% |
28 and 32: AutoTS
AutoTS searches hundreds of model templates and returns the best ensemble, which sounds like it should win. I repeated the 2022 call on four rolling five-day windows at the end of each bundled year. Each fit took several minutes on a laptop, about 65 minutes for the ten fits in this section.
28 Tata Motors
| Window, 2022 | AutoTS MAPE | Flat MAPE | Closer |
|---|---|---|---|
| 28 Oct – 3 Nov | 4.89% | 2.14% | Flat |
| 4 Nov – 10 Nov | 5.51% | 5.17% | Flat |
| 11 Nov – 17 Nov | 5.38% | 2.22% | Flat |
| 18 Nov – 25 Nov | 3.69% | 1.27% | Flat |
| Mean of 4 | 4.87% | 2.70% | Flat |
AutoTS was closer in 0 of the 4 windows.
32 Apple
| Window, 2022 | AutoTS MAPE | Flat MAPE | Closer |
|---|---|---|---|
| 28 Oct – 3 Nov | 4.20% | 4.18% | Flat |
| 4 Nov – 10 Nov | 3.75% | 1.85% | Flat |
| 11 Nov – 17 Nov | 3.17% | 1.76% | Flat |
| 18 Nov – 25 Nov | 0.93% | 0.91% | Flat |
| Mean of 4 | 3.01% | 2.18% | Flat |
AutoTS was closer in 0 of the 4 windows.
For scale: across every possible start date in the Tata Motors year, a flat forecast misses by 3.15% on average over five days, and Apple's by 2.85%.
Tata Motors: the 2022 forecast against the same call today (change from the last close)
Same data, same settings, AutoTS 0.6.21 instead of 0.5.1. The 2022 forecast can no longer be checked because the ADR was delisted in 2023.
- 2022 forecast
- Re-run today
Show the numbers as a table
| Date | 2022 forecast | Re-run today |
|---|---|---|
| Day 1 | 0.28% | -0.07% |
| Day 2 | 0.53% | 0.92% |
| Day 3 | 0.77% | 1.14% |
| Day 4 | 0.99% | 2.33% |
| Day 5 | 1.20% | 2.36% |
Notebook 32 never finished in 2022. Finishing it now and checking against the five trading days that followed (2022-11-28 to 2022-12-02), AutoTS missed by 1.94% on average against 1.60% for the flat forecast. One week proves nothing either way, which is the point of backtesting on several windows.
Apple, the week after the data ends: change from the last close
- What happened
- AutoTS
- Flat forecast
Show the numbers as a table
| Date | What happened | AutoTS | Flat forecast |
|---|---|---|---|
| Day 1 | -2.63% | -0.28% | 0.00% |
| Day 2 | -4.69% | -3.61% | 0.00% |
| Day 3 | -0.05% | -2.10% | 0.00% |
| Day 4 | 0.14% | -2.06% | 0.00% |
| Day 5 | -0.20% | -2.13% | 0.00% |
Is the gap bigger than the noise?
Two average errors can differ by chance, and the errors of nearby days move together. So each model is compared with the flat forecast day by day, on the same days, with the Diebold-Mariano test: a positive statistic means the model's errors are larger. The intervals come from resampling blocks of consecutive days, which keeps that day-to-day dependence.
| Forecast | Model MAPE, 95% | Flat MAPE, 95% | Mean gap (pts), 95% | Paired test, two-sided |
|---|---|---|---|---|
| 01 LSTM, Apple, time-ordered690 days, h = 1 | 38.69%34.83% to 42.59% | 1.67%1.47% to 1.88% | +37.03+33.06 to +41.00 | p < 0.0001DM +53.72 |
| 09 LSTM, Netflix, time-ordered690 days, h = 1 | 14.57%12.34% to 17.13% | 2.17%1.90% to 2.50% | +12.39+9.98 to +15.12 | p < 0.0001DM +27.28 |
| 05 Linear, five days ahead52 days, h = 5 | 0.50%0.38% to 0.63% | 0.36%0.27% to 0.46% | +0.14+0.06 to +0.22 | p = 0.008DM +2.77 |
| 28 AutoTS, Tata Motors4 windows of 5 days; ranges, not intervals | 4.87%3.69% to 5.51% | 2.70%1.27% to 5.17% | +2.17+0.34 to +3.16 | p = 0.125sign test, 0 of 4 won |
| 32 AutoTS, Apple4 windows of 5 days; ranges, not intervals | 3.01%0.93% to 4.20% | 2.18%0.91% to 4.18% | +0.84+0.02 to +1.90 | p = 0.125sign test, 0 of 4 won |
| 01 LSTM, Apple, shuffled (2022)689 days, h = 1 | 1.00%0.91% to 1.07% | 1.29%1.16% to 1.42% | −0.29−0.41 to −0.18 | p < 0.0001DM −6.76 |
| 09 LSTM, Netflix, shuffled (2022)689 days, h = 1 | 4.09%2.83% to 5.27% | 2.04%1.85% to 2.24% | +2.05+0.81 to +3.20 | p < 0.0001DM +8.75 |
Every time-ordered comparison but AutoTS is decisive. AutoTS lost in all four windows for both shares, but four windows are thin evidence: an exact sign test on four windows cannot go below p = 0.125. Its 20 days are four forecasts of one to five days ahead each, not 20 comparable days, so neither the Diebold-Mariano test nor a bootstrap interval is used there. The direction is consistent; the proof is not there. The one comparison a model wins, Apple's LSTM on shuffled days, is the one where it was given the same day's high and low, and it wins only with the seed the audit happened to use.
One run of an LSTM is not a result. Re-running the script with the same seed reproduced Apple to every digit but not Netflix, so every fit was repeated with four more seeds. On the time-ordered split the spread is wide and every seed loses to the flat forecast by a wide margin. On the shuffled split the verdict depends on the seed: Apple's LSTM beats yesterday's close with seed 42 and with no other, and Netflix's with seed 1 and no other. Bold marks a seed that beats the flat forecast.
| LSTM MAPE | Seed 42 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Flat |
|---|---|---|---|---|---|---|
| 01 Apple, time-ordered | 38.69% | 36.73% | 38.79% | 35.86% | 43.28% | 1.67% |
| 09 Netflix, time-ordered | 14.57% | 10.95% | 16.42% | 14.11% | 15.35% | 2.17% |
| 01 Apple, shuffled | 1.00% | 2.27% | 2.26% | 1.30% | 1.45% | 1.29% |
| 09 Netflix, shuffled | 4.09% | 1.93% | 3.04% | 2.40% | 4.68% | 2.04% |
Mean gap is model minus flat forecast, in percentage points of absolute error, with its 95% block-bootstrap interval. The Diebold-Mariano (DM) test uses the Harvey, Leybourne and Newbold correction, with h = 5 for the five-day forecast in 05; block-bootstrap intervals use blocks of 5 days or more, seed 2026, 10,000 resamples. AutoTS rows give the lowest and highest of the four windows instead of an interval. The LSTMs in the first table were trained with seed 42 and TensorFlow's deterministic ops; the seed table adds seeds 1, 2, 3, 4. The AutoTS windows need no seed. How each interval and test works is set out on the methods page.
What I take from it
None of the five price models earned its complexity. The ones that looked good were tested on shuffled days or used information from the day being predicted. When I test them on later days, the flat forecast is as good or better.
That is the normal result for daily prices, and it is why every forecasting exercise should start by writing down the naive baseline and the time-ordered split before any model is trained.