Interviewer: Michael Gilliland, SAS

This month our interview is with Jonathon Karelse, a recognized Demand Planning and S&OP thought leader, frequent speaker, moderator, and panelist at IBF and supply chain events. Jonathon is graduate of the MIT Sloan School of Management’s Executive Program in Value Chain and Operations Management. He was the youngest ever Executive at Yokohama Tire, where he implemented a successful demand-driven global planning process, and served as Business Unit Director for the Company’s Canadian Consumer products division. He has been published in various trade and academic journals including the Journal of Business Forecasting (JBF).

Through NorthFind Partners – the company he co-founded – Jonathon developed operations strategies and enterprise demand planning for some of the world’s most successful manufacturers and distributors.

In the interview with Jonathon, we’ll be discussing the application of FVA analysis with his clients.

Mike: Jonathon, what forecasting performance metric(s) are you currently using?

Jonathon: I currently use wMAPE (by 12 trailing months of Gross Profit $), wBias (by 6 trailing months of COGS$), and for FVA we use RMSE as the basis for comparison. All our key metrics are weighted by profit because we want to remain focused on parts and actions that will impact EBIT, not just academically trying to improve accuracy everywhere. With limited time and resources, companies should focus on what moves the needle.

Mike : Glad to hear you are measuring Bias because it is often overlooked. What are your findings?

Jonathon: Most companies as expected tend to have positive bias over time, which analysts find easier to systematically correct than variability in error rates. But in some companies with perennially bad supply, they actually compound the problem with negative bias as Sales have been programmed not to expect full deliveries, and they don’t want sales targets tied to a number they think can’t be built.

Mike : Are you comparing performance to a naïve model?

Jonathon: Yes. We often use a seasonal random walk, though increasingly Simple Exponential Smoothing or Moving Average. A random walk is almost too naïve.

Mike : What are the steps in the typical forecasting process with your clients?

Jonathon: The general process steps are Statistical Forecast, Analyst Adjustment, Customer Forecasts, Sales and Marketing Inputs, and Final Demand Planner forecast.

Mike : What FVA comparisons are you measuring?

Jonathon: We make all the pairwise comparisons between all the steps, and also to the naïve model.

Mike : Do your clients adopt FVA as a key performance indicator?

Jonathon: Absolutely. Every client we’ve been engaged with finds this KPI really resonates, and is often the key area that executive management looks at. It is also the key metric used for root cause and corrective action.

Mike : Have there been any process changes as the result of FVA findings?

Jonathon: We’ve seen many changes implemented as the result of FVA findings. A couple of examples are how they handle collaborative discussions with customers, and changing the way forecasters work with sales for inputs.

For example, for a major transportation client the tribal knowledge was that customer forecasts were an indispensable element of the demand planning process. Over 180 customer forecasts dropped into MRP directly; some of them on EDI signals that turned out to be unmonitored at both ends. FVA revealed that these inputs, though intuitively beneficial, were systemically impairing our ability to forecast customer requirements. This allowed us to go back to key customers and engage in collaborative discussions focused on process and data improvement.

At another client, a major global manufacturer of electronics components, nearly 800 Sales Engineer inputs were systematically gathered every month. Who better than the Sales Engineers to understand the requirements of their customers? Well…FVA showed us that the Pareto rule was alive and well here, and only a handful of the SEs were giving us input that demonstrably improved Forecast Accuracy. By paring out hundreds of non-value added inputs, we saved hundreds of hours of time, and improved the overall Forecast Accuracy.

Mike : Have you done any volatility analysis (e.g. “comet chart”), or otherwise attempted to assess “forecastability”?

Jonathon: Yes, this is another standard for us. We often start with a histogram of a company’s Coefficient of Variation (CV) by SKU before getting into the deep dive. It gives us a sense of whether more heavy lifting up front will be required from Sales judgmental inputs (lots of high CV parts from project based business, for instance) or statistical models (most applicable to lower, more stable CV items).

We use comet charts to validate the break point on part segmentation (or forecastability) matrices, subject to required service and inventory levels. We also use comet charts for diagnostic insights into the health of the planning process by looking at parts that fall outside the control limits of expected Forecast Accuracy vs CV.

I should note this CV is calculated on the basis of deseasonalized data, since leaving it seasonal would create false positives in terms of variability; or at least unfairly suggest that fewer parts will respond well to statistical analysis. So by the time we begin looking at FVA, we have a good idea from the histogram whether initially we are going to see a good bump versus a naïve baseline. In businesses with tons of volatility, the baseline is often pretty tough to outperform.

Mike : Does your clients set forecasting performance goals (e.g. wMAPE < 20%)?

Jonathon: We strongly discourage those goals. We focus on continuous error reduction, continued improvement in FVA, and on keeping Bias volatility as low as possible.

Mike : What FVA analyses / reports do you use?

Jonathon: Here is an example from a client in transportation. They utilize a modified version of the “stairstep” report, based on RMSE (rather than the more commonly used MAPE or wMAPE). They love metrics and waterfall their FVA reports by monthly lag out to Lag 9. (Creating lagged versions is probably not something most companies need to do!)



Mike : Have you developed other new ways to conduct FVA related analyses, or report the results?

Jonathon: I believe we are the only group using the comet chart as a diagnostic tool as we are, and I also believe we are the only group tying a comet chart to the inventory/service optimization curve.

Mike : Can you point to any specific results / benefits / improvements from using FVA at your organization?

Jonathon: Much faster root cause and corrective action, and as a result, much faster improvements.

Mike : Anything else you’d like to say about FVA? Including advice for other companies considering the application of FVA?

Jonathon: FVA is easy! If you aren’t using it, you are missing a critical indicator of your organization’s forecasting performance.


Willing to share your experiences with FVA?   Please contact the IBF at to arrange an interview for the blog series.