TRENDING

TECHNOLOGY NEWS FROM AROUND THE WORLD

→
Machine learning train/test splits and leakage: are we measuring real generalization?
Artificial Intelligence · 2026Photo: Unsplash / Unsplash ↗
Artificial Intelligence

Machine learning train/test splits and leakage: are we measuring real generalization?

Supervised learning fundamentals through a practical dataset: labels, leakage, train/test isolation and meaningful evaluation.

Yazılımcının GünlüğüPublished: Nov 16, 20263 min read

Define the prediction target

In supervised learning, each example contains input features and a known target. Suppose we predict a future reservation cancellation: booking lead time and booking channel may be available, but a refund record created after cancellation is not. Including it creates data leakage and an unrealistic accuracy score. The model appears strong because it has seen information that would not be available at prediction time.

Separate training, validation and testing

Parameters are learned from the training data, while model selection and threshold tuning happen on validation data. Keep the test set for the final estimate. Repeatedly tuning features against test scores undermines independence. In time-dependent problems, chronological splits are often safer than random shuffles because future outcomes must not leak into past predictions.

Conceptual workflow — original editorial illustration

Choose a metric for the decision

When cancellations are rare, predicting 'no cancellation' for everyone can achieve deceptively high accuracy. Precision, recall, F1 and sometimes precision-recall curves better express false-alarm and missed-event costs. Report class balance, the confusion matrix and the chosen threshold rather than one flattering score.

Monitor performance after deployment

Data sources, customer behavior and seasons change. A model that performs well on its first test may degrade later. Monitor feature distributions, label availability and real-world error costs. Plan retraining with fresh data, but also create new independent evaluations instead of repeatedly reusing the original test set.

0 reads

Comments

0 real comments

No real reader comments yet.

✎ Write a comment

Comments are reviewed before publication.