What an unruly mess some hyperparameter configurations are… In many open source deep learning codebases, the hyperparameters are treated as global variables but it’s nothing new that global variables should be avoided. Yet, here we are.

3 years ago, I started as a junior deep learning engineer at Apple and I developed a similar approach to this one: https://www.reddit.com/r/MachineLearning/comments/e5jvhq/p_how_to_get_rid_of_boilerplate_ifstatements_and/. My team had to abondon it though because…. The solution required redundant boilerplate. Using YAML files was annoying too because YAML has little support for variables and YAML has no support for lambdas or Python objects. Lastly, it wasn’t an easy process to modularize YAML files as compared to Python functions.

Anywho, the above solutions just didn’t work.

3 year later, after tinkering and working at different companies as a deep learning engineer, I came up with this approach: https://github.com/PetrochukM/HParams. Here’s what it looks like:

Example Code

The approach has a couple of benefits:

• There are no global variables.
• The approach lends it’s self to automatic checks that ensure:
  • That no hyperparameters are overwritten.
  • All declared hyperparameters are set and used.
  • The hyperparameter type is correct.
• It enables you to configure external functions.
• The various hyperparameter dependencies are visable and intuitive.
• The hyperparameters are easy to export and track.
• It’s easy to add-on a CLI, like so: foo@bar:~$ u/ --torch.optim.adam.Adam.__init__ 'HParams(lr=0.1,betas=(0.999,0.99))'

Anywho, let me know what you think!

Lastly, it’s kinda cool, that similar approaches to mine were discovered and implemented by the AllenNLP library and Google’s gin-config. Does that mean I’m doing something right?