Gradient Driven Adaptive Learning Rate (GDALR) is proposed to efficiently duplicate a model

http://sci-hub.tw/https://ieeexplore.ieee.org/document/8878726

https://www.youtube.com/watch?v=tw6CUVk4mn0

What the developers want to achieve is basicaly like there would be a human dungeon master in the game that reacts to your actions in the game.

probably many do not know this, I learned it by studying the references in section 19 of Jurgen’s very dense inaugural tweet

I knew AlexNet, the CUDA CNN by Alex Krizhevsky and Ilya Sutskever and Geoff Hinton which won ImageNet 2012, but prior to AlexNet, Jurgen’s team with his “outstanding Romanian postdoc Dan Ciresan … won 4 important computer vision competitions in a row between May 15, 2011, and September 10, 2012” with an earlier CUDA CNN, let me call this DanNet, the blog post on their miraculous year links to a summary of these contests

I saw a news article claiming that AlexNet started a deep learning revolution in 2012, but actually the references show that DanNet was the first superhuman CNN in 2011 and also won a medical imaging contest on images way bigger than AlexNet’s

the most cited DanNet paper is CVPR July 2012, 5 months before AlexNet at NIPS 2012, but earlier descriptions of DanNet appeared at IJCAI 2011 and IJCNN 2011

in his blog, Jurgen also cites CNN pioneers since Fukushima 1979, and GPU implementations of neural networks since Jung and Oh 2004

to be fair, AlexNet cites DanNet and admits that it is similar, however, it does not mention that DanNet won all those earlier challenges

ResNet beat AlexNet on ImageNet in 2015, but ResNet is actually a special case of the earlier highway networks, also invented in Jurgen’s lab, the “First Working Feedforward Networks With Over 100 Layers,” section 4 of The Blog links to an overview, he credits his students Rupesh Kumar Srivastava and Klaus Greff

there was a big reddit thread on section 5 of his blog, Jurgen’s GAN of 1990, and everybody knows LSTM, which won contests already in 2009, section 4 of The Blog, but I think many don’t know yet that his team also was first in the CUDA CNN game

Hi Reddit,

I am working on various NLP project for several years. And, I have used various pre-processing packages related with NLP, such as nltk, gluonnlp and torchtext. However, It’s not still clear which package is the best to process text.

– what NLP packages would you recommend me ?

Thank you so much !

Hi, how can I run gpt-2 with the new full model locally? And will I be able to generate very long (100,000 characters) outputs on a standard gaming PC? 16GB ram, gtx 1060 6GB , i7 4770k @4.8ghz

I’m looking for a good Neural Architecture Search implementation that I could use to learn a deep network for my custom dataset. The dataset is sequential (long sequences of ~40 unique tokens), and the task is just sequence modeling. I’m happy to use a fixed window size and one-hot encoding to turn the inputs into fixed-size matrices if that makes life easier (e.g. turning this into an image classification task). I’ve already used hand-crafted RNNs and attention-based models and achieved pretty high accuracy, but I’m hoping to try out NAS and see how it compares.

I wanted to ask here and see if anyone has suggestions first before I sink a large number of hours into any one package. Some packages I’ve used/heard of:

AutoKeras – Doesn’t seem to work with the sequential form of the data but does work with the fixed-size matrices by using the ImageClassifier. Having a bunch of problems though as 0.4 version is a little broken, and 1.0 version isn’t yet complete.

auto-sklearn – My understanding is it focuses on XGBoost rather than neural nets.

https://github.com/quark0/darts – I really like the idea/paper for this one, has anyone had experience using it for a custom dataset?

https://github.com/carpedm20/ENAS-pytorch – Same algorithm as AutoKeras (ENAS) but only RNNs are implemented so far. I see a lot of open issues on GitHub so a little apprehensive about committing time to this specific implementation.

Exploring Randomly Wired Neural Networks for Image Recognition (https://arxiv.org/abs/1904.01569) – I love this paper but can only find unofficial implementations on GitHub. Looking through them it seems like they don’t achieve quite as high results as in the paper. Has anyone seen an official (or best) implementation of this?

Microsoft NNI (https://github.com/microsoft/nni) – This seems a little bit intense and I haven’t had a chance to learn how to use it yet. It does seem to contain ENAS as one of the algorithms though, so this seems promising.

Is one of these the right tool for me to use? Are there some other implementations out there that would be better suited for my task?

Paper: https://arxiv.org/abs/1910.11933

Github repo: https://github.com/alecokas/BiLatticeRNN-Confidence

Abstract:

Recently, there has been growth in providers of speech transcription services enabling others to leverage technology they would not normally be able to use. As a result, speech-enabled solutions have become commonplace. Their success critically relies on the quality, accuracy, and reliability of the underlying speech transcription systems. Those black box systems, however, offer limited means for quality control as only word sequences are typically available. This paper examines this limited resource scenario for confidence estimation, a measure commonly used to assess transcription reliability. In particular, it explores what other sources of word and sub-word level information available in the transcription process could be used to improve confidence scores. To encode all such information this paper extends lattice recurrent neural networks to handle sub-words. Experimental results using the IARPA OpenKWS 2016 evaluation system show that the use of additional information yields significant gains in confidence estimation accuracy.

TL;DR:

This paper looks at how to improve confidence estimates for black-box automatic speech recognition systems.

I’m looking for a way to save, annotate, highlight, and write notes on recent papers I’ve read.

Is anyone aware of an online, browser based tool that would allow for this (E.G. upload a PDF and do aforementioned tasks)? I use a variety of devices, both personal and at work, and would prefer to avoid downloading software onto all, but would be open to any suggestions.

What do you all use?

I have been experimenting with classifying raw audio samples using temporal convolutions and RNNs. Recently I did an experiment where I just replaced all layers with Dense layers(Keras) and took out all other layers. Surprisingly it did quite well on validation and test data and trained very fast.

Why did this work? It’s only classifying one second of audio data sampled at 8khz. Can fully connected networks model sequential data ? Or is there something wrong with my experiment/setup?

TL;DR Can FCN classify sequential data ?

Hey guys,

which neural network would you prefer to use to implement this style transfer algorithm from that paper?

https://arxiv.org/pdf/1508.06576.pdf

I saw Tutorials e.g. from Tensorflow using the VGG19 net, but I find it too painful too train and since accuracy in classification is not that important because it is more a sort of artistic thing, I thing a network like SqueezeNet is a better fit and the final model is ridiculously light. I would like to hear your thoughts about which architecture you would prefer.