Blog

Learn About Our Meetup

4500+ Members

[P] Simple hyperparameter management through dependency injection

[P] Simple hyperparameter management through dependency injection

What an unruly mess some hyperparameter configurations are… In many open source deep learning codebases, the hyperparameters are treated as global variables but it’s nothing new that global variables should be avoided. Yet, here we are.

3 years ago, I started as a junior deep learning engineer at Apple and I developed a similar approach to this one: https://www.reddit.com/r/MachineLearning/comments/e5jvhq/p_how_to_get_rid_of_boilerplate_ifstatements_and/. My team had to abondon it though because…. The solution required redundant boilerplate. Using YAML files was annoying too because YAML has little support for variables and YAML has no support for lambdas or Python objects. Lastly, it wasn’t an easy process to modularize YAML files as compared to Python functions.

Anywho, the above solutions just didn’t work.

3 year later, after tinkering and working at different companies as a deep learning engineer, I came up with this approach: https://github.com/PetrochukM/HParams. Here’s what it looks like:

Example Code

The approach has a couple of benefits:

  • There are no global variables.
  • The approach lends it’s self to automatic checks that ensure:
    • That no hyperparameters are overwritten.
    • All declared hyperparameters are set and used.
    • The hyperparameter type is correct.
  • It enables you to configure external functions.
  • The various hyperparameter dependencies are visable and intuitive.
  • The hyperparameters are easy to export and track.
  • It’s easy to add-on a CLI, like so: foo@bar:~$ u/ --torch.optim.adam.Adam.__init__ 'HParams(lr=0.1,betas=(0.999,0.99))'

Anywho, let me know what you think!

Lastly, it’s kinda cool, that similar approaches to mine were discovered and implemented by the AllenNLP library and Google’s gin-config. Does that mean I’m doing something right?

submitted by /u/Deepblue129
[link] [comments]

Next Meetup

 

Days
:
Hours
:
Minutes
:
Seconds

 

Plug yourself into AI and don't miss a beat

 


Toronto AI is a social and collaborative hub to unite AI innovators of Toronto and surrounding areas. We explore AI technologies in digital art and music, healthcare, marketing, fintech, vr, robotics and more. Toronto AI was founded by Dave MacDonald and Patrick O'Mara.