Learn About Our Meetup

4500+ Members

[D] Tests for comparing predictive accuracy of regression models

I’m trying to compare the predictive accuracy of few regression models. For simplicity, let’s say that I have a polynomial of degree 6 and a GAM model with many knots. One simple approach would be to compare the RMSE and/or MAE. In this case, the GAM model has a lower RMSE and MAE than those of the polynomial model, but the difference is small. Now based on the RMSE and MAE values, I should choose the GAM model, but the small difference is making me question whether it makes sense to take the GAM model over the simpler polynomial model.

Searching around, I found that one can use the Diebold-Mariano (or the similar HLN) test to compare the predictive accuracy of two forecasts in time series. The DM/HLN tests determine whether there is any significant difference between the forecasts. However, I think it would not be appropriate to use the DM/HLN-test in my case, since the tests compute autocovariance at lags in order to derive the test-statistic and that would make little sense in the context of non-time series forecasts.

Are there any similar tests that can be used for non-time series forecasts?

submitted by /u/Megaslaking
[link] [comments]

Next Meetup




Plug yourself into AI and don't miss a beat


Toronto AI is a social and collaborative hub to unite AI innovators of Toronto and surrounding areas. We explore AI technologies in digital art and music, healthcare, marketing, fintech, vr, robotics and more. Toronto AI was founded by Dave MacDonald and Patrick O'Mara.