Hi everyone,

I’m working on this time-series regression problem and I’ve already gone through the following stages:

• prepared different datasets by adding first only the series itself, then moving averages, then sentiment data, etc;
• trained benchmarks: persistent models, linear regressions, ARIMA, …
• tried a variety of different deep learning architectures (MLPs, resnets, wavenets, lstm, etc.)

So, what happens is that no matter (i) how the dataset is built and (ii) how complex or fancy the architecture is but the model always end up ignoring the input and predicting as output at timestep t the input a timestep t-1, which is called a persistent model in literature and (it’s one of the benchmarks)

TL; DR:

Time series framing problem: DL models (of several architectures) end up totally ignoring the input and learn to give always the same prediction: ŷ(t) = x(t-1)

Q: How to address this issue? Is there a way to penalise this behaviour during the training?

http://standoutpublishing.com/Blog/archives/64-Introducing-Multitemporal-Synapses.html

http://standoutpublishing.com/g/The-Stability-Plasticity-Problem.html

I came across this patented idea/site/thing while looking for information on catastrophic forgetting, and now find myself perplexed on a couple different levels.

First, a bold claim from the site:

“Now, however, with the advent of multi-temporal synapses, the limitation has been eliminated for artificial neural networks as well. This problem has been solved..” Here problem refers to plasticity vs stability, aka catastrophic forgetting.

Perplexed point 1) What is this technique and how useful is it? I’ve read through the information on the site w/o checking the references cited and glanced at Netlab but nothing seems to have any actual application of this idea.

Perplexed point 2) This blog post is 9 years old, if it was worthwhile wouldn’t it have become wide spread by now?

Perplexed point 3) Since /something/ is patented, not sure what exactly, what does that mean legally for trying to understand or use aspects of this technique in tackling catastrophic forgetting?

There is a benchmark of desktop and laptop GPU cards for deep learning: AI Benchmark. You can run these tests yourself, see https://pypi.org/project/ai-benchmark/.

More detailed results here: http://ai-benchmark.com/ranking_cpus_and_gpus_detailed.html (TensorFlow training and inference times for: MobileNet-V2, Inception-V3, Inception-V4, Inc-ResNet-V2, ResNet-V2-50, ResNet-V2-152, VGG-16, SRCNN 9-5-5, VGG-19 Super-Res, ResNet-SRGAN, ResNet-DPED, U-Net, Nvidia-SPADE, ICNet, PSPNet, DeepLab, Pixel-RNN, LSTM, GNMT).

I found other useful benchmarks and tests:

• Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning
• RTX 2080 Ti Deep Learning Benchmarks with TensorFlow – 2019

Take note that some GPUs are good for games but not for deep learning (for games 1660 Ti would be good enough and much, much cheaper, vide this and that). For general benchmarks, I recommend UserBenchmark (my Lenovo Y740 with Nvidia RTX 2080 Max-Q here.)

For comparison of different cards between frameworks, see Performance in: Keras or PyTorch as your first deep learning framework (June 2018), based on Comparing Deep Learning Frameworks: A Rosetta Stone Approach.

Do you know any other good rankings, benchmarks, and tests? (There is MLPerf, but I guess due to the complication of the procedure, the amount of data is very small.)

I’m taking a machine learning class and I have a project, whose task is to implement an original supervised learning algorithm. It doesn’t have to be something new from scratch and it doesn’t need to be overcomplicated, because it’s a one week project. It can be a combination of two learning algorithms that can accurately classify a labeled data set, such as using genetic algorithms with artificial neural networks. It can also use part of an existing algorithm, as long as I add something substantial to it. The problem is I can’t think of a simple idea that is not already proposed in a published research paper.

To put things into perspective, the learning algorithms I’m familiar with are:

• Decision Trees
• KNN
• ANN
• SVM
• Linear and Logistic Regression
• Genetic Algorithm
• Clustering

As someone having trouble to concentrate on classic screens I often print the papers I need to read thoroughly but I feel guilty every time.

Has anyone tried e-readers and could give a feedback on it for reading papers ?

John Guerra conducts these types of analysis at various academic conferences over the years.

Here is a list of the 783 Twitter accounts most followed by the members of the NeurIPS2019 community (computed by identifying the 679 accounts tweeting using #NeurIPS2019 with at least 3 tweets, between 2019-12-02 and 2019-12-20).

https://johnguerra.co/viz/influentials/NeurIPS2019/