We’re familiar with application of the scientific method in certain industries, such as Pharmaceuticals. When a new drug is introduced, we expect that its safety and efficacy has been demonstrated through appropriately controlled experiments.
For example, to test a new cold remedy we would find 100 people with colds, randomly give half the new drug and half a placebo, and see whether there is any difference in outcomes. If those given the new drug get more immediate relief from the discomfort and recover faster, we may be able to conclude that the drug actually works – that it is an effective treatment for colds.
In conducting such an experiment, the scientist begins with a null hypothesis:
H0: The drug has no effect.
Through the controlled experiment, we determine whether there is sufficient evidence to reject the null hypothesis and infer that the drug does have an effect (which can be either positive or negative).
The Null Hypothesis for Forecasting
In forecasting we are fond of elaborate systems and processes, with more touch points and human engagement. We tend to believe that the more sophisticated our models and the more engaging our processes, this will result in better forecasts. But do we ever pause to test this belief?
If we approach business forecasting like a scientist, we would ask whether any of our forecasting efforts are having a beneficial effect. Do our statistical models result in a better forecast? Do our analyst overrides make it better still? Are other participants (like sales, marketing, or finance) providing further improvement?
We would start with appropriate null hypotheses such as:
H0: The statistical model has no effect on forecast accuracy.
H0: The analyst override of the statistical forecast has no effect on forecast accuracy.
H0: Input from the sales force has no effect on forecast accuracy.
But “no effect” compared to what? Is there a placebo for forecasting?
Fortunately for those of us who want to let science get in the way of our forecasting, there is a placebo… something referred to as the naïve forecast. A naïve forecast is something simple to compute, essentially a free alternative to implementing a forecasting systems and process. The two standard examples are:
Random Walk – using the last known value as the forecast. (If we sold 12 units in May, our forecast for June would be 12.)
Seasonal Random Walk – using the known value from a year ago as the forecast. (If we sold 15 in June of 2011, then we would forecast 15 for June of 2012.)
If we did nothing – had no forecasting software or forecasting process – and just used the naïve forecast, we would achieve some level of accuracy, say X%. So the questions become, does our statistical forecasting software do better than that? Do overrides to the statistical forecast make it even better? Does input from the sales force provide further improvement?
Forecast Value Added (FVA) analysis is the name for this kind of approach to evaluating forecasting performance. FVA is defined as:
The change in a forecasting performance metric (such as MAPE, or bias, or whatever metric you are using) that can be attributed to a particular step or participant in the forecasting process.
FVA is essentially an exercise in hypothesis testing – are the steps in your process having an effect (either positive or negative). The objective of FVA analysis is to identify those steps or participants that have no effect on forecasting performance – or may even be making it worse! By eliminating the non-value adding activities (or redirecting those resources to more productive activities outside of forecasting), you reduce the resources committed to forecasting and potentially achieve better forecasts.
So should we implement a consensus process, or CPFR, or allow executive management final say over the forecast? I don’t know– they all sound like good enough ideas! But until we apply FVA analysis and put them to the test, it isn’t safe to make that assumption.