https://youtu.be/YZ-_E7A3V2w

Excellent talk from Neurips retrospectives workshop – one of the most interesting ones I’ve seen.

One question I had related to the last question: When I originally heard about the adjoint sensitivity method used for the backwards pass of Neural ODE’s, I was curious about the fact that the backwards pass is essentially untethered from the forwards pass. Would it make sense to write a ODE solver specifically for the backwards pass that is able to make use of the forwards pass checkpoints? For example, you could try to enforce that ypur backwards pass doesn’t stray too far from for your forwards pass. I don’t know enough about ODE solvers to know if this makes sense.