Machine learning train/test splits and leakage: are we measuring real generalization?
Supervised learning fundamentals through a practical dataset: labels, leakage, train/test isolation and meaningful evaluation.
Define the prediction target
In supervised learning, each example contains input features and a known target. Suppose we predict a future reservation cancellation: booking lead time and booking channel may be available, but a refund record created after cancellation is not. Including it creates data leakage and an unrealistic accuracy score. The model appears strong because it has seen information that would not be available at prediction time.
Separate training, validation and testing
Parameters are learned from the training data, while model selection and threshold tuning happen on validation data. Keep the test set for the final estimate. Repeatedly tuning features against test scores undermines independence. In time-dependent problems, chronological splits are often safer than random shuffles because future outcomes must not leak into past predictions.
Choose a metric for the decision
When cancellations are rare, predicting 'no cancellation' for everyone can achieve deceptively high accuracy. Precision, recall, F1 and sometimes precision-recall curves better express false-alarm and missed-event costs. Report class balance, the confusion matrix and the chosen threshold rather than one flattering score.
Monitor performance after deployment
Data sources, customer behavior and seasons change. A model that performs well on its first test may degrade later. Monitor feature distributions, label availability and real-world error costs. Plan retraining with fresh data, but also create new independent evaluations instead of repeatedly reusing the original test set.
Comments
0 real commentsNo real reader comments yet.
✎ Write a comment