Skip to content
Edition
Late EditionM.S. Computer Science · 3.97 GPA
The Sreekaran Reddy Portfolio
← Back to Projects
Time-Series Forecasting Case Study

The Forecast That Was Too Good To Be True

A thirty-five-year-old forecasting method, two eager machine-learning rookies, a leak that almost fooled everyone, and the honest scoreboard that came after.

There’s a desk at every bank that nobody throws a party for.

The person who sits there doesn’t launch products. Doesn’t demo anything flashy in front of the board. Every quarter, they answer one question and go home: how many of these loans are going to go bad next quarter?

For years, that desk had one occupant. Everyone just called it ARIMA.

Who’s who

  • ARIMA The VeteranA classical model. Thirty-five years on the desk.
  • Xander XGBoostThe rookie who builds thousands of trees.
  • Gigi LightGBMThe other rookie. Says little, builds fast.
  • Surya Runs the deskGives everyone the same fair test.

Chapter 1

The Veteran

ARIMA was old. Not “outdated” old. Reliable old. The kind of old that makes younger models nervous, because it’s still right more often than they’d like.

ARIMA starts with three order choices. p, d, q: autoregressive order, differencing order, and moving-average order. These define the model structure; its coefficients still have to be fitted to the data.

A first-timer

That’s it?

ARIMA

That sets my structure. I use past values and forecast errors, difference the series when needed, and fit the coefficients.

ARIMA watched five accounts, quarter after quarter, for thirty-five years:

  • All Loans
  • Credit Card
  • Business
  • Mortgage
  • CRE

Every quarter, the drill was the same. Look at everything known so far. Guess the next four quarters. Wait. Get graded. Do it again, a little more history each time, sliding forward one quarter at a time. Fifty-nine times per account, for thirty-five years of quarterly data. That’s called a walk-forward backtest, and it’s a brutal, honest way to be graded: no peeking at the future, ever, at any point.

Nobody expected ARIMA to be a genius. It was graded against two lazy alternatives: just guess it’s the same as last quarter (simple-naive), and just guess it’s the same as this time last year (seasonal-naive). Beating a lazy guess isn’t supposed to be hard.

The reported seasonal-naive-scaled scores were below 1 for all five series. The simple-naive comparison was mixed: only Business and CRE had scores below 1. ARIMA did not lead both benchmark comparisons across every series.

AccountReported seasonal-naive-scaled error
CRE0.135
Business0.236
All Loans0.269
Credit Card0.325
Mortgage0.348

These are aggregate backtest scores for each loan category. All five were below 1.0 on the reported seasonal-naive scale; that does not mean ARIMA beat the benchmark in every individual quarter.

Nobody clapped. That’s just what the job looked like, every quarter, for years.

Then, one Tuesday, two new hires arrived.

Chapter 2

The Rookies

Their names were Xander and Gigi. Everyone had heard of them. Xander went by his full name when he wanted to sound serious: XGBoost. Gigi’s was LightGBM, but nobody called her that unless something had gone wrong.

They didn’t work like ARIMA. ARIMA looked at a few numbers and wrote an equation. Xander and Gigi built trees: thousands of tiny yes/no questions, stacked on top of each other, voting on an answer.

“Was last quarter above 4%? If yes, was the quarter before that also rising? If yes…” and so on, hundreds of times, hundreds of trees.

They’d read about themselves winning competitions. Real ones. Big ones. They were, by reputation, the future.

Surya, who ran the desk now, decided to give them a fair shot: the exact same walk-forward drill ARIMA had been doing for years. No shortcuts, no head start. Same five accounts. Same fifty-nine graded quarters each.

There was one immediate problem: Xander and Gigi couldn’t read a raw number the way ARIMA could. So Surya gave them a study guide instead: what the account looked like 1, 2, 3, and 4 quarters ago, what it looked like 8 quarters ago (two years, worth checking for slower patterns too), and how choppy the last year had been.

Surya

That’s all you get. No looking anywhere else. Same information ARIMA has, just handed to you differently.

Xander cracked his knuckles. Gigi didn’t say anything. She just started building trees.

The first results came back.

They were stunning. Numbers so good Surya actually sat back in the chair.

They were also, it would turn out, a lie.

Chapter 3

Too Good to Be True

The tell came from somewhere small: a routine experiment to see if Xander and Gigi could get even better with some tuning. Just twenty tries each at adjusting their own settings, evaluated fairly, on data they’d never touched.

One run came back with a number so low it didn’t look like a forecast error anymore. It looked like a typo.

Surya stared at it for a long moment.

Nobody predicts loan delinquency rates that well. Not from five numbers and a rolling average. Not in five tuning attempts. Not ever, really.

The rule

When a result looks too good to be true, it’s not a discovery. It’s a leak.

So Surya went looking. And found it.

Here’s what had happened. Xander and Gigi weren’t just handed “what happened 1, 2, 3, 4, 8 quarters ago.” They were handed a study guide with the answer key stapled in by accident, right at the edge, where nobody had checked.

Every training example was supposed to teach them: given the past, guess four quarters ahead. But near the very end of each training window, a few examples’ “four quarters ahead” landed inside the quarters they were about to be tested on that same round. A handful of times, for a handful of quarters, they weren’t predicting the test. They were copying it: their own upcoming answer, quietly sitting in their own homework.

Nobody had done this on purpose. It was one line of logic, quietly wrong, in exactly the kind of place these things always hide: a boundary condition, off by a few rows, at the edge of a sliding window.

Surya fixed it. Rebuilt the study guide so it stopped at exactly the right line, every time. Not one row of the future allowed to leak backward, ever. Then ran everything again, from scratch.

The stunning numbers disappeared. What was left was the truth.

Chapter 4

The Honest Scoreboard

AccountARIMAXander (XGBoost)Gigi (LightGBM)
All Loans0.2690.6010.510
Credit Card0.3250.7700.833
Business0.2360.6600.643
Mortgage0.3481.6631.490
CRE0.1350.5370.515

ARIMA won. Every account. Not close.

Xander didn’t say anything for a while.

Xander

Maybe it’s just how the coin landed. Maybe on a different week, we’d have taken one.

So Surya brought in a referee whose entire job is answering exactly that question: the Diebold-Mariano test, which looks at fifty-nine graded quarters and asks: is this gap real, or did ARIMA just get lucky?

Across the series, models, and horizons, 27 of 40 comparisons had reported p<0.05, including all ten one-quarter-ahead comparisons. Fewer crossed that threshold at longer horizons. These are individual test results; an adjustment across the 40 tests is not documented here, so the count should not be read as proof that chance has been ruled out.

But there wasn’t a single one of the forty match-ups where the numbers leaned toward Xander or Gigi. Not once. Just some where the lead wasn’t big enough yet to call it, officially, beyond doubt.

Chapter 5

Two Things the Rookies Tried

Xander and Gigi didn’t quit. They tried two things.

First, they stopped competing separately and pooled what they knew. Instead of Xander learning Mortgage alone from its own 142 quarters, he learned from all five accounts at once, roughly 700 examples, telling them apart with a simple tag on each one: this one’s Mortgage, this one’s CRE. More to learn from. Shared patterns across accounts that move together.

It helped. Genuinely. On four of the five accounts, the pooled version beat the solo version by a wide margin.

It still didn’t beat ARIMA. Not once.

Second, they asked for a tutor. Twenty rounds of careful, honest tuning, searched for on the early three-quarters of their training history, tested only on the untouched final quarter of it, so the tutor couldn’t quietly cheat either.

The tutor helped Xander and Gigi in four of ten tries. In the other six, tutoring actually made things worse.

Gigi

That’s… not encouraging.

Surya

It’s honest, though. You don’t have enough homework for a tutor to matter much. You’d need a lot more of it before fine-tuning starts paying off reliably. That’s not a flaw in you. It’s arithmetic.

Chapter 6

Asking Them What They Were Actually Looking At

There was one more thing Surya wanted to know. Not how well Xander and Gigi were guessing: what they were even looking at when they guessed.

Surya used SHAP to attribute the tree models’ predictions to input features relative to a reference. This describes model behavior, not human-like reasoning or causal effects.

The answer came back the same way, ten times out of ten, every account, both Xander and Gigi:

What they were looking at

Last quarter’s number. It accounted for 47–59% of the reported aggregate SHAP importance across the ten model/series combinations, not that share of every individual prediction.

Xander looked almost embarrassed.

Xander

That’s… barely a model. That’s just looking at yesterday.

Surya

It’s not just that. You also glanced at the quarter before, and the one before that, and a two-year-back check, and the recent choppiness. But yes. Mostly, you were looking at yesterday.

ARIMA also uses past observations, but these SHAP results describe Xander and Gigi. They do not measure ARIMA’s feature contributions or prove that all three models learned the same relationship.

The attribution gave Surya a useful clue about the tree models: the most recent quarter mattered most in this summary. Explaining the accuracy gap would require additional controlled comparisons.

Chapter 7

The Verdict

One last thing needed checking: not who guessed closer, but who was honest about how sure they were.

ARIMA said “I’m 95% confident the real number lands in this range,” quarter after quarter. Checked against fifty-nine real outcomes, it was right 98% to 100% of the time. If anything, a little too modest, a bit wider than it needed to be.

Xander and Gigi tried to build their own version of that confidence, from their own track record of past misses. Checked the same way, their “95% confident” was actually right only 75% to 83% of the time.

That’s not a small gap. That’s a prediction interval that lies about itself, and it lies hardest exactly when a bank can least afford it. The two-decade run of data made clear that both the mess and the model’s uncertainty about it get bigger right when a downturn is already happening. That’s the one moment a “95% confident” range needs to actually mean it. Xander and Gigi’s didn’t, not yet.

So here’s the honest version of the story, the one that doesn’t flatter anybody on purpose.

ARIMA outperformed the two tree models in the reported comparison. A small dataset may favor a simpler model, but that is a possible explanation, not something the SHAP analysis proves. Feature attribution cannot rule out training, tuning, or feature-design limitations.

Bottom line

The takeaway is to judge a model by its backtest, then treat explanations for its performance as hypotheses. New macroeconomic inputs, different features, or different model choices are candidates for further tests, not guaranteed improvements.

A full walk-forward backtest, complete with a caught-and-fixed data leakage bug, Diebold-Mariano significance testing, conformal prediction intervals, and SHAP interpretability. Full methodology, code, and results in the technical write-up and on GitHub.