I had been struggling with implementing a somewhat niche paper for months now, and I only now was able to reproduce the result because the author published a follow-up paper that fixed some egregious errors in the original one (some benign typos, some things you simply couldn’t guess). I had contacted the author about the original paper but never heard back (I suspect because he had become aware of the errors).

I think requiring a reference implementation would also go a long way of leveling the playing field between small contributors and the big guns like Google, Amazon etc. Those big guys like to hide behind their proprietary data (“not our problem if you don’t have the data to reproduce our results”), I think it would make for better science if anybody can just clone a repo, download some data and get the result.

Sorry, this was mostly a rant born from the frustration of seeing massively erroneous papers in journals and conferences that slip past peer review because the reviewers have no way of checking the veracity of the claims.