Good luck!

https://datascience.stackexchange.com/questions/49695/why-not-turn-momentum-update-equation-into-exponentially-weighted-moving-average

In Pytorch, the update equation of SGD with (non-Nesterov) momentum is m^[i+1] = β m^[i] + g L(w^[i+1]), where g means gradient, β is the momentum coefficient, m^[i] is the momentum at iteration i, L is the loss function, w^[i] is the value of weights at iteration i.

If we are starting with m^[0] = 0, then for all i > 0 m^[i] = sum({ β^j g L(w^[i-j]) | j∈{0, …, i-1} }).

Now, let’s write down the formulas for exponentially weighted moving average of gradients (which we’ll denote as a^[i]) to show that one is equivalent to the other multiplicated by a constant. We will make a non-traditional assumption that a^[0] = 0. It doesn’t matter, because as i goes to infinity, the contribution of the zeroth term goes to zero.

a^[i+1] = β a^[i] + (1-β) g L(w^[i+1]) We can rewrite it as a^[i] = (1 – β) sum({ β^j g L(w^[i-j]) | j ∈ {0, …, i-1} }).

Notice that ∀ β ∈ [0, 1) it holds that (1 – β) m^[i] = a^[i].

It seems to me that we should change the update equation of momentum SGD to the equation of exponentially weighted moving average of gradients, i.e. add the 1 – β coefficient to the gradient term. Here’s why:

1. It decouples learning rate from momentum coefficient. Currently, larger momentum coefficient increases the effective learning rate (i.e. by how much the weights are updated). Suppose we are in an ideal scenario, when for all iterations i, j we have g L(w^[i]) = g L(w^[j]) = g L, then lim_{i→∞} m^[i] = g L / (1 – β). For β = 0.9 this value equals 10 g L. For β = 0.99 this value equals 100 g L. In contrast, if we use exponentially weighted moving average formula, for all β the analagous limit would equal just g L. I concede that this is an unrealistic scenario, and in real problems gradients at steps i, i+1, i+2, … , i+k somewhat cancel each other out, but still I think it’s a good point.
2. Weighted moving average is a somewhat well known concept, while momentum isn’t.

I am interested to hear, what reasons are there not to change the update formula? And if you think this is a good change, how should the authors of deep learning libraries proceed?

I want to start learning machine learning where do I start?

Hi, I’m working with a large dataset of 1-D data (think time series) with wildly varying values (decidedly non-Normal.) My goal is to train an autoregressive generative model like WaveNet using this data. I’ll need to normalize all series to the range [0, 1] so I can then quantize the data to 256 possible values for the softmax output of the WaveNet. I’ve run into a few problems and haven’t found much help on the internet (mostly searching for time series normalization, standardization, etc.)

A quick run through my current process:

• Divide each series by their median value in a small window where the signal is (or should be) ~0
• At this point, the distribution of values in each series is roughly log-normal so I’m taking the log and then standardizing each series individually by subtracting its mean and dividing by its standard deviation
• If I now normalize the entire dataset based on the max/min (i.e. data <- (data – data.min())/(data.max() – data.min()), most series are squeezed into a range like [0.4, 0.6] due to massive outlying max/min values. I’ve tried scaling each series by their respective min/max but again, due to some series having massive outliers this skews their scale compared to the rest of the dataset.

Is my best option to just cull the outliers? Or am I missing a step somewhere? I’ve visualized a few of the outliers, and they are valid data. I’ve also tried median-stacking nearest neighbor series to tame some of the volatility but am not sure where to go from here. Leaving it as is will increase the effective quantization noise in my data since instead of being spread over 256 values, most series only span ~100 values in the discrete space. Any help would be much appreciated!

This is a place to share machine learning research papers, journals, and articles that you’re reading this week. If it relates to what you’re researching, by all means elaborate and give us your insight, otherwise it could just be an interesting paper you’ve read.

Please try to provide some insight from your understanding and please don’t post things which are present in wiki.

Preferably you should link the arxiv page (not the PDF, you can easily access the PDF from the summary page but not the other way around) or any other pertinent links.

Previous weeks :

Most upvoted papers two weeks ago:

/u/spoiltForChoice: https://lear.inrialpes.fr/pubs/2011/JDS11/jegou_searching_with_quantization.pdf

/u/Moseyic: VAE

/u/ToolTechSoftware: https://accu.org/index.php/journals/2639

Besides that, there are no rules, have fun.

Hi,

I’m pretty new to the whole ML thing, but I’ve a decent grasp on how it works under the hood. I’ve been mulling over a few projects I want to try, using TensorFlow in particular, it seems to be the easiest one to use (that I’ve discovered, anyway).

I just wanted to inquire here about the limits of something like TF, such as the maximum number of inputs, the complexity of an output, compute time etc.

When I say complexity of an output, I’m referring to say, having the program spit out a full engineering diagram, blueprints, measurements, 3D models and things along those lines. I’m not expecting it to be easy, but I’d like to get a rough idea of how much work I can have the machine do for me.

Thanks, and I hope this isn’t some totally off the wall, absolute misunderstanding of the technology!

Some recently hired data scientists at work are suggesting that we start using tools like Algorithmia and FastScore for deploying machine learning models. Some of them are talking about “One click model deployment”. I took a look at the websites for these products, but I’m still having a hard time figuring out what they can do that is so special and why exactly do they make life easier for a data scientists compared to running your model on Google Cloud ML or Sagemaker?

A small group of humans is winning consistently against OpenAI Five. There seem to be a few reproducible strategies that keep beating the bot. Can someone describe what those strategies are for someone that hasn’t played DoTA?

Link https://arena.openai.com/#/results

I’ve read about microsofts IntelliCode that aims to guide developers with better code completion and I want to start something similar for the Rust programming language as project both to learn ml and to use it at the end. I know that means a lot of work and the result will probably be in a whole different league than what microsoft has build.

​

From what I’ve read syntax checks can be implemented with autoencoders and LSTM networks, but I don’t know how to suggestion for code corrections.

LSTM networks are used for autocompletion and text generation, but I would need to implement a context aware network that gets code from before and behind the cursor. I guess I would need to tokenize the different syntax elements like natural language in NLTK is processed?

Right now this is mostly guess work for me. I would highly appreciate tipps on the network type and design and information about related work.