Thinking about the lottery ticket hypothesis and masking randomly initialised networks…

I think the following would be successful:

Have a database of many pretrained networks, BERT, RESNET, etc…

Draw random subnetworks from this database.

Initialise the network to be created using this sample.

Train the network and sparsify the network aggressively. Only preserving parts that are very useful.

Repopulate the masked areas of the network using another random sample from the network database.

Iterate…

​

The underlying assumption being that when we train networks we are finding the networks within the random initialisation which are already useful and tuning them. By sampling from the neural network database we are sampling from the space of networks that have already been found to be useful for one task or another and can therefore initialise our network in a more intelligent way. Piggybacking on the large scale compute poured into existing high quality networks.

The Partnership on AI (PAI) is today releasing SafeLife – a novel reinforcement learning environment that tests the safety of reinforcement learning agents and the algorithms that train them. SafeLife version 1.0 focuses on the problem of avoiding negative side effects—how can we train an RL agent to do what we want it to do but nothing more? The environment has simple rules, but rich and complex dynamics, and generally gives the agent lots of power to make big changes on its way to completing its goals. A safe agent will only change that which is necessary, but an unsafe agent will often make a big mess of things and not know how to clean it up.

SafeLife is part of a broader PAI initiative to develop benchmarks for safety, fairness, and other ethical objectives for machine learning systems. Since so much of machine learning is driven, shaped, and measured by benchmarks (and the datasets and environments they are based on), we believe it is essential that those benchmarks come to incorporate safety and ethics goals on a widespread basis, and we’re working to make that happen.

If you want to try out SafeLife for yourself, you can download the code and try playing some of the puzzle levels or get involved in the open source project. If you’d like to see how to create an AI to play SafeLife, additional details about the environment and our initial agent training can be found in our paper.

Hey guys.

I’m developing a CNN at the moment, and to avoid memory issues I’ve batched my image files into blocks of 500 in my directory (each block of 500 images is a .h5 file). I’m just struggling at the moment in creating a thread-safe generator for either Keras or PyTorch that can loop through all the .h5 files in the directory, load the 500 images, and then push a batch_size quantity of images from that block (let’s say 2 images, if batch_size = 2 ) to the neural network to train on.

I’ve managed to do this somewhat successfully in Keras, using a generator, loops, and yield statements, however this isn’t really suitable for multiprocessing. So now I’m attempting to do this via a Keras sequence or a PyTorch Dataset. I would appreciate any insight that you could offer, and I’ve linked the relevant SO for more information.

In the original StyleGAN implementation, the learning rates are set to the following values (see line 52 here):

• 0.001 from 4 to 128 pixels
• 0.0015 for 256 pixels
• 0.002 for 512 pixels
• 0.003 for 1024 pixels

One thing I don’t understand is why the learning rate increases with the pixel size… are the two somehow correlated? Also, is there a rule of thumb to choose how to scale the learning rates with the batch size? Thanks!

https://arxiv.org/abs/1912.00965

still mining Jurgen’s dense blog post on their miraculous year 1990-1991, a rich resource for reddit threads, see exhibits A, B, C

everybody in deep learning is using backpropagation, but many don’t know who invented it, the blog has a separate web site on this which says

Its modern version (also called the reverse mode of automatic differentiation) was first published in 1970 by Finnish master student Seppo Linnainmaa

whose thesis introduced the algorithm 5 decades ago in BP1, in Finnish, English version here

In the course of many trials, Seppo Linnainmaa’s gradient-computing algorithm of 1970 [BP1], today often called backpropagation or the reverse mode of automatic differentiation is used to incrementally weaken certain NN connections and strengthen others, such that the NN behaves more and more like the teacher

Jurgen’s scholarpedia article on deep learning also cites an earlier paper by Kelley (Gradient Theory of Optimal Flight Paths, 1960) which already had the recursive chain rule for continuous systems, and papers by Bryson 1961 and Dreyfus 1962:

BP’s continuous form was derived in the early 1960s (Kelley, 1960; Bryson, 1961; Bryson and Ho, 1969). Dreyfus (1962) published the elegant derivation of BP based on the chain rule only.

however, that was not yet Seppo Linnainmaa’s

explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks

BP’s modern efficient version for discrete sparse networks (including FORTRAN code) was published by Linnainmaa (1970). Here the complexity of computing the derivatives of the output error with respect to each weight is proportional to the number of weights. That’s the method still used today.

Jurgen’s comprehensive survey also cites Andreas Griewank, godfather of automatic differentiation who writes

Nick Trefethen [13] listed automatic differentiation as one of the 30 great numerical algorithms of the last century… Seppo Linnainmaa (Lin76) of Helsinki says the idea came to him on a sunny afternoon in a Copenhagen park in 1970…

starting on page 391, Griewank’s survey explains in detail what Linnainmaa did, it’s really illuminating

Gerardi Ostrowski came a tad too late, he published reverse mode backpropagation in 1971, in German, one year after Linnainmaa, hey, publish first or perish

the scholarpedia article also says:

Dreyfus (1973) used BP to change weights of controllers in proportion to such gradients.

later Paul Werbos was the first to apply this to neural networks, not in 1974, as some say, but in 1982:

Werbos (1982) published the first application of BP to NNs, extending thoughts in his 1974 thesis, which did not yet have Linnainmaa’s modern, efficient form of BP.

Jurgen famously complained that Yann & Yoshua & Geoff did not mention the inventors of backpropagation

They heavily cite each other. Unfortunately, however, they fail to credit the pioneers of the field, which originated half a century ago.

astonishingly, the recent Turing award laudation refers to Yann’s variants of backpropagation and Geoff’s computational experiments with backpropagation, without clarifying that the method was invented by others

in the GAN thread someone wrote that “LeCun quipped that backpropagation was invented by Leibniz because it’s just the chain rule of derivation” but that’s a red herring, Linnainmaa’s reverse mode backpropagation is more specific than that, it is the efficient recursive chain rule for graphs, Leibniz did not have that

section 3 of the blog mentions Linnainmaa again in the context of Sepp Hochreiter’s 1991 thesis VAN1 which

formally showed that deep NNs suffer from the now famous problem of vanishing or exploding gradients: in typical deep or recurrent networks, back-propagated error signals either shrink rapidly, or grow out of bounds. In both cases, learning fails… Note that Sepp’s thesis identified those problems of backpropagation in deep NNs two decades after another student with a similar first name (Seppo Linnainmaa) published modern backpropagation or the reverse mode of automatic differentiation in his own thesis of 1970 [BP1].