So I’ve been pretty interested in this paper A Closer Look at Memorization in Deep Networks and particularly the first experiment they did where they showed that certain data points are consistently fit in the first epoch of training whereas other data points consistently take longer epochs to fit.

But I haven’t seen any discussions anywhere about why that would be the case? Like what is it about these data points that allows them to be easily fit in the first epoch? How can we formalize this notion of “simpleness”?

My first thought is that the “simple” data are just the ones which have a gradient direction that is close to the averaged gradient direction for a given minibatch?

Anyone aware of any work specifically expanding on these questions?

Unfortunately I don’t have anyone in my lab to discuss these things with so I just resort to the next best place lol.