Certain group, pioneer in leveraging the value of their data, developed a risk prediction model for defaults, using data from the last 3 years, and quickly managed to reduce the incidence from 3% to less than 1% of sales, all while undergoing significant expansion. Two years later, not only could they not afford to do away with their credit coverage policy, but the incidence had risen again, and the system was performing worse and worse. No one had told them about “model drift.”
What happened? Simply put, their context has continued to evolve while their model remained static. Probably, by rejecting operations due to default risk, their customer base had slowly changed, in addition to the larger customer base due to their expansion, with new practices and behavior patterns, so the data they now feed into the model has nothing to do with the data used in its original training. Mathematically speaking: P(x|y)≠P(x|y’), in other words, the probability of default (x) given the data patterns from before (y) is different from the probability of default given the data patterns from now (y’).
This particular type of degradation is known as “data drift,” and it can be prevented by periodically retraining the model with the new data and giving more weight to the most recent data, and less relevance to the old data, so it is adapted to the new context without losing all knowledge from the past. In fact, the last stage of the CRISP-ML methodology is continuous model monitoring and evaluation, retraining it, or even reparametrizing it when it loses accuracy.
Other times, models degrade because changes occur in the variables to be predicted, the rules of the game change, or the criteria change. For example, if we used to not consider a customer as delinquent if they paid after 180 days, and now we decide to comply with the law and shorten the payment deadlines to 60 days or less, our model will no longer be valid, and it needs to be retrained. This type of degradation is known as “concept drift.”
Machine learning models do not forget, but it can happen that, like it happens to humans, what they know can lose value over time. Retrain or perish.
Image: Socrates Robot generated by Dall-E