We compute the forecast accuracy measures for this period. Sometimes, different accuracy measures will lead to different results as to which forecast method is best. To take a non-seasonal example, consider the Google stock price. The following graph shows the observations ending on 6 Dec , along with forecasts of the next 40 days obtained from three different methods.
In this procedure, there are a series of test sets, each consisting of a single observation. The corresponding training set consists only of observations that occurred prior to the observation that forms the test set. Thus, no future observations can be used in constructing the forecast.
Since it is not possible to obtain a reliable forecast based on a small training set, the earliest observations are not considered as test sets. The following diagram illustrates the series of training and test sets, where the blue observations form the training sets, and the red observations form the test sets. The forecast accuracy is computed by averaging over the test sets. With time series forecasting, one-step forecasts may not be as relevant as multi-step forecasts. In this case, the cross-validation procedure based on a rolling forecasting origin can be modified to allow multi-step errors to be used.
Then the corresponding diagram is shown below. Time series cross-validation is implemented with the tsCV function. A good way to choose the best forecasting model is to find the model with the smallest RMSE computed using time series cross-validation. The ugliness of the above R code makes this a good opportunity to introduce some alternative ways of stringing R functions together. In the above code, we are nesting functions within functions within functions, so you have to read the code from the inside out, making it difficult to understand what is being computed.
And secondly, forecast error is normally not something they worry about. These groups are major drivers of forecast error in the supply chain forecast, but they often cannot be told they have a high error, and their eyes tend to glaze over at reviewing forecast error reports. All of this is work to get a person to just a basic understanding of the forecast error. It is curious how many times. The fact that the person does not realize this tells you how little they know about forecast error measurement.
When reporting average error across many items or SKUs, it does not make sense to do so without weighing the error, or the forecast error on an items with sales of 20 items will count the same as the highest volume SKU that has yearly sales of , items. One of the major problems of forecast error measurement is that forecasting systems do not have a weighted forecast. So they can only tell users the forecast error of a specific product location combination.
This is very typical of forecasting applications. There are multiple forecast error measurements. However, in this screenshot, an aggregation is selected. This means the forecast error measurement is not functional. You can read more about how we use causal models to forecast the impact of promotions here. Some of these are known well in advance, such as holidays or local festivals.
One-off events typically require manual planning, but for recurring events, such as Easter, for which past data is available, forecasting can be highly automated. Some external factors naturally take us by surprise, such as a specific product taking off in social media.
In Finland, this happened recently with cauliflower, for which demand doubled in response to a social media campaign initiated by a few concerned citizens who wanted to help farmers move an exceptionally large crop. Even when the information becomes available only after important business decisions have been made, it is important to use the information to cleanse the data used for forecasting to avoid errors in future forecasts.
Does your forecast accuracy behave in a predictable way? It is often more important to understand in which situations and for which products forecasts can be expected to be good or bad, rather than to pour vast resources into perfecting forecasts that are by their nature unreliable.
Understanding when forecast accuracy is likely to be low, makes it possible to do a risk analysis of the consequences of over- and under forecasting and to make business decisions accordingly. As the products have limited shelf-life, the manufacturer does not want to risk potentially very inflated forecasts driving up inventory just in case, rather they make sure they have production capacity, raw materials and packaging supplies to be able to deal with a situation where the original forecast turns out to be too low.
The need for predictable forecast behavior is also the reason why we apply extreme care when taking new forecasting methods, such as different machine learning algorithms into use. For example, when testing different variants of machine learning on promotion data, we discarded one approach that was on average slightly more accurate than some others, but significantly less robust and more difficult for the average demand planner to understand.
Occasional extreme forecast errors can be very detrimental to your performance, when the planning process has been set up to tolerate a certain level of uncertainty. If demand changes in ways that cannot be explained or demand is affected by factors for which information is not available early enough to impact business decisions, you simply must find ways of making the process less dependent on forecast accuracy.
We already mentioned weather as one external factor having an impact on demand. In the short-term, weather forecasts can be used to drive replenishment to stores you can read more about how to use machine learning to benefit from weather data in your forecasting here. However, long-term weather forecasts are still too uncertain to provide value in demand planning that needs to be done months ahead of sales.
In very weather-dependent businesses, such as winter sports gear, our recommendation is to make a business decision concerning what inventory levels to go for. For high-margin items, the business impact of losing sales due to stock-outs is usually worse than the impact of needing to resort to clearance sales to get rid of excess stock, which is why it may make sense to plan in accordance with favorable weather.
For low-margin items, rebates may quickly turn products unprofitable, which is why it may be wiser to have a more cautious inventory plan. In any case, setting your operations up so that final decisions on where to position stock are made as late as possible allow for collecting more information and improving forecast accuracy.
In practice, this can mean holding back a proportion of inventory at your distribution centers to be allocated to the regions that have the most favorable conditions and the best chance of selling the goods at full price. You can read more about managing seasonal products here. Depending on the chosen metric, level of aggregation and forecasting horizon, you can get very different results on forecast accuracy for the exact same data set.
To be able to analyze forecasts and track the development of forecasts accuracy over time, it is necessary to understand the basic characteristics of the most commonly used forecast accuracy metrics. There is probably an infinite number of forecast accuracy metrics, but most of them are variations of the following three: forecast bias, mean average deviation MAD , and mean average percentage error MAPE.
We will have a closer look at these next. Do not let the simple appearance of these metrics fool you. After explaining the basics, we will delve into the intricacies of how the metrics are calculated in practice and show how simple and completely justifiable changes in the calculation logic has the power of radically altering the forecast accuracy results.
Forecast bias is the difference between forecast and sales. If the forecast over-estimates sales, the forecast bias is considered positive.
If the forecast under-estimates sales, the forecast bias is considered negative. In many cases it is useful to know if demand is systematically over- or under-estimated. For example, even if a slight forecast bias would not have notable effect on store replenishment, it can lead to over- or under-supply at the central warehouse or distribution centers if this kind of systematic error concerns many stores. A word of caution: When looking at aggregations over several products or long periods of time, the bias metric does not give you much information on the quality of the detailed forecasts.
The bias metric only tells you whether the overall forecast was good or not. It can easily disguise very large errors. You can find an example of this in Table 1. Mean absolute deviation MAD is another commonly used forecasting metric. This metric shows how large an error, on average, you have in your forecast. However, as the MAD metric gives you the average error in units, it is not very useful for comparisons.
An average error of 1, units may be very large when looking at a product that sells only 5, units per period, but marginal for an item that sells , units in the same time. Basically, it tells you by how many percentage points your forecasts are off, on average. This is probably the single most commonly used forecasting metric in demand planning. As the MAPE calculations gives equal weight to all items, be it products or time periods, it quickly gives you very large error percentages if you include lots of slow-sellers in the data set, as relative errors amongst slow sellers can appear rather large even when the absolute errors are not see Table 2 for an example of this.
In fact, a typical problem when using the MAPE metric for slow-sellers on the day-level are sales being zero, making it impossible to calculate a MAPE score. Measuring forecast accuracy is not only about selecting the right metric or metrics. There are a few more things to consider when deciding how you should calculate your forecast accuracy:.
Measuring accuracy or measuring error: This may seem obvious, but we will mention it anyway, as over the years we have seen some very smart people get confused over this. Aggregating data or aggregating metrics: One of the biggest factors affecting what results your forecast accuracy metrics produce is the selected level of aggregation in terms of number of products or over time.
As discussed earlier, forecast accuracies are typically better when viewed on the aggregated level. However, when measuring forecast accuracy at aggregate levels, you also need to be careful about how you perform the calculations. As we will demonstrate below, it can make a huge difference whether you apply the metrics to aggregated data or calculate averages of the detailed metrics. In the example see Table 3 , we have a group of three products, their sales and forecasts from a single week as well as their respective MAPEs.
The bottom row shows sales, forecasts, and the MAPE calculated at a product group level, based on the aggregated numbers. Which number is correct? The answer is that both are, but they should be used in different situations and never be compared to one another. The same dynamics are at play when aggregating over periods of time. The data in the previous examples were on a weekly level, but the results would look quite different if we calculated the MAPE for each weekday separately and then took the average of those metrics.
The planners must review the forecasting model assignments periodically, so that a better baseline forecast can be generated, which can later be complimented further by other inputs from sales, marketing or logistics. I have seen many planners who do not know how to view such results.
Olivehorse is helping clients using SAP APO with custom enhancements that can be used to overcome these limitations and bring an improvement in your statistical forecasting approach.
Call us to know more about it! Are you using the tool in the right way? Let us know in the comments area! Olivehorse Consulting is unique in being focused entirely on supply chain planning, execution and management, whether it be leveraging SAP or Anaplan technologies.
Building on this foundation we continue to expand our team attracting, developing and retaining talented, consultants passionate about delivering practical solutions to supply chain problems. Do you have a passion for providing practical solutions to complex supply chain problems, in a fast-moving, customer-centric environment?
If so, we'd love to hear from you as we are always looking to develop our team. Forecast Error measures are widely used in the following two scenarios: To evaluate the statistical forecast model fit by allowing several forecast models to compete against each other. Understanding the Misunderstanding Often, Planners can't understand why tools sometimes produce strange statistical forecast results!
Figure 1: Error Measure for Automatic Model Selection 2 Hence a sensible approach would be to segment the data on variability and volume, and then use a forecasting approach with the appropriate error measure.
0コメント