My colleagues and I are developing an NLP tool for an enterprise customer. In production usage, the tool is going to be applied to a large stream of newswire documents. One current problem is that our training and testing datasets are static, i.e. based on a document sample that doesn’t change over time. With time passing, it might be that the tool’s output in production gradually becomes irrelevant but no one gets alerted about this.

Could anyone here share any experience on continuous, longitudinal quality monitoring for NLP (and, more broadly, ML)? Assuming worse-than-human performance of the model, are there any better options than just to have a dedicated person who would regularly label a sample of recent predictions and raise a flag if anything goes wrong?