The authors use a classic Armijo line-search approach in the context of SGD to automatically tune the line search parameter in training the neural networks. They’re also able to prove convergence results on minimizing convex and non-convex objective functions satisfying certain growth conditions. An aside, but as an optimization-head myself, it’s nice to see some of the traditional optimization ideas make their way into an ML context.

​

https://arxiv.org/pdf/1905.09997.pdf

I implemented dilated convolution Seq2Seq, based architecture from Convolution Seq2Seq, tested on 100k English-Malay translation dataset, and I beat that model in term of word position. 80% to train, 20% to test.

This result after 20 epochs only,

1. Attention is All you need, train accuracy 19.09% test accuracy 20.38%

2. BiRNN Seq2Seq Luong Attention, Beam decoder, train accuracy 45.2% test accuracy 37.26%

3. Convolution Encoder Decoder, train accuracy 35.89% test accuracy 30.65%

4. Dilated Convolution Encoder Decoder, train accuracy 82.3% test accuracy 56.72%

5. Dilated Convolution Encoder Decoder Self-Attention, train accuracy 60.76% test accuracy 36.59%

Source code here

To make sure the translation work, I implemented beamsearch from tensor2tensor on no 4.

Feel free to use it for future research, and let me know if got better or bad results!

So I’m looking for work that studies how normalization and whitening can be included in the input layer for a time series model. I know batch normalization is related but I don’t think that explicitly looks at time series data. Anyone have recommendations?

Does anyone have a benchmark/comparison of a neuroevolution algorithm for machine learning such as NEAT and a more traditional deep learning approach? Preferably an interactive one that features many different kinds of tasks and tracks the performance of different sizes of models as they train. I have done some experiments with genetic algorithms in the past and would like to see if they provide any long-term advantages over gradient-descent based methods at any type of problem. Any help would be appreciated. (Repost because forgot to tag post the first time.)

like

https://www.researchgate.net/profile/Mehmet_Hacibeyoglu/publication/328405250/figure/fig1/AS:683792754806784@1540040305691/The-basic-structure-of-convolution-neural-networks.jpg

15 samples generated after fine-tuning GPT-2 and no cherry-picking. Some might be interesting to read and some don’t make sense at all. All of them are far from patenting a product like Oclean Air Electric Toothbrush. Our motivation is trying to see whether GPT-2 can generate new ideas for innovation.

Calculating pruned weights after pruning them is easy, just nonzero_params/all_params * 100%

However I can’t find papers/method to calculate pruned weights via activation-mask based pruning,

Anyone here know how to calculate pruned weights due to pruned activations (masked activations)? This is the case where no weights are pruned directly, but instead weights are pruned by zeroing them by putting mask on activations before them ( because zero * zero = zero)

Unlike ordinary weight pruning, it involves going through zeros through matmul, conv, with strides, padding, and kernel size, etc.