I’m trying to compare the predictive accuracy of few regression models. For simplicity, let’s say that I have a polynomial of degree 6 and a GAM model with many knots. One simple approach would be to compare the RMSE and/or MAE. In this case, the GAM model has a lower RMSE and MAE than those of the polynomial model, but the difference is small. Now based on the RMSE and MAE values, I should choose the GAM model, but the small difference is making me question whether it makes sense to take the GAM model over the simpler polynomial model.

Searching around, I found that one can use the Diebold-Mariano (or the similar HLN) test to compare the predictive accuracy of two forecasts in time series. The DM/HLN tests determine whether there is any significant difference between the forecasts. However, I think it would not be appropriate to use the DM/HLN-test in my case, since the tests compute autocovariance at lags in order to derive the test-statistic and that would make little sense in the context of non-time series forecasts.

Are there any similar tests that can be used for non-time series forecasts?