June 16, 2022
Forecasting is used everywhere in business, politics, policy, and science…and yet we don’t take it seriously enough. Sure, there’s the M-series forecasting competition and there’s Hyndman & Athanasopoulos and related textbooks as well as a bunch of papers per year. But these are mostly domain experts in other fields applying whatever they can to their specific problems. Forecasting in all of these cases is a means to an end, not an end in and of itself.
Such an approach is for cowards and unambitious people. It’s why when forecasting tries to get serious as a field it fails (like with all of those timeseries foundation models, what’s the point to them?). It always seems to fail to push the envelope.. Actually, what exactly is the envelope here? What is forecasting trying to solve in the end? Just some short horizon MAPE with large error bands? Get real.
Let’s get serious here. What if we gave Forecasting (capital F) it’s proper place? What do we need to do to make it like Physics, Biology, Statistics, or AI? Let me suggest four things:
A unifying theory of everything. Replicating the human mind in silico. Reverse engineering causality. These are epoch encompassing goals that serve fill multiple crucial roles in a discipline. They captivate minds and propagate narratives. They serve as a war cry, a banner to rally on. They foster competition to get to a holy grail. They enrich the human spirit.
A single, unifying ambition is perhaps the most important thing a field can have and how it lives or dies. For Forecasting, it can’t just be “predicting the future”. That’s too vague and not even remotely relatable. Yeah I mean of course it would be nice to know the future, so what?
No, it needs to be something more. Something like “A framework to tame fate” or “Predicting the unpredictable” or “Ending the tyranny of surprise”. When prompted with e.g. these goals your mind already thinks in different directions, and I can bet you that those directions are not auto-regressions or exogenous variables, but about a purer concepts yielding richer frameworks. Importantly, solid goals like these allow you to cheat. Remember when AI used to be concerned about the bias/variance tradeoff, until it was sort of thrown out the window with double descent? It’s because the end goal of AI is to build a mind, and if that necessitates throwing apparent statistical limitations via empirical ruthlessness then so be it.
All models in all fields need to pass tests of generalization. For Biology, for example, you must show what parts of a model are evolution invariant (or covariant). AI has its own tests of generalization based on out of distribution performance that differ slightly from the Statistics view.
Similar generalization value judgements would be needed in Forecasting, and probably something beyond the “does this backtest well” or “does this do good on the hold out set”. It would have to follow the spirit of our Unifying Ambition. It could be, for example, that we judge the quality of models by how many classes in the “forecasting complexity family” they can cover (I don’t think “forecasting complexity families” are a thing, but if you’ve done forecasting you know what I’m talking about).
Notions of generalization are necessary but not sufficient for general theories to work. An additional ingredient of deeper invariants is required, like the conservation principles and the predictive reach of theories. In AI, things like “the bitter lesson” come to mind, or in Biology the fitness ansatz behind natural selection. For Forecasting, this would likely means son deep statements relating system complexity and the size of the horizon, or maybe going even beyond the models and how they are used downstream as decision processes.
Finally, no field is complete without explicit and implicit self-propagating norms that dictate how to consume the models that the field creates (within and without the field itself). Collaborative science and the application of its fruits is in a way an exercise of scaled social engineering, where in the end the narratives that the models convey are more important than the models themselves. This is particularly true in Forecasting, and something that is likely necessary to be made explicit. The Forecaster must not only produce correct models of reality but should have tools to convince those who consume the forecasts to act in ways aligned to the predictions. This is the final and toughest ingredient of “taming fate”, and one that is not generally acknowledged even in “proper” fields.