Most Predictive Analytics discussions focus on methods that can be used for better quality prediction in a particular context. Realizing that the possibility of perfect prediction is a near impossibility, practitioners looking to support their futuristic initiatives wonder, what is a suitable model for their use.
In other words, if all prediction models are imperfect (have leakage) how much of this imperfection can be tolerated and yet better decisions can be taken with model output. This paper is an attempt to provide a simplified approach to this practical problem of evaluating model performance taking account of the decision context.
Two scenarios are discussed; a) a classification problem often used for profiling customers into segments and, b) a volume forecasting problem. In both cases, the leakage is defined (misclassification or uncertainty band) and their impact (adverse) on the subsequent decision is identified. Contextual dimensions that have an impact on the quality of the decision and the scope to alleviate the problem are also discussed.