Skip to main content


Learn About Our Meetup

5000+ Members



Join our meetup, learn, connect, share, and get to know your Toronto AI community. 



Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.



Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[D] Tests for comparing predictive accuracy of regression models

I’m trying to compare the predictive accuracy of few regression models. For simplicity, let’s say that I have a polynomial of degree 6 and a GAM model with many knots. One simple approach would be to compare the RMSE and/or MAE. In this case, the GAM model has a lower RMSE and MAE than those of the polynomial model, but the difference is small. Now based on the RMSE and MAE values, I should choose the GAM model, but the small difference is making me question whether it makes sense to take the GAM model over the simpler polynomial model.

Searching around, I found that one can use the Diebold-Mariano (or the similar HLN) test to compare the predictive accuracy of two forecasts in time series. The DM/HLN tests determine whether there is any significant difference between the forecasts. However, I think it would not be appropriate to use the DM/HLN-test in my case, since the tests compute autocovariance at lags in order to derive the test-statistic and that would make little sense in the context of non-time series forecasts.

Are there any similar tests that can be used for non-time series forecasts?

submitted by /u/Megaslaking
[link] [comments]