Came across the state of AI report and didn’t see it posted here so thought it’d be interesting to have a discussion about the recently released report (second yearly one by Nathan Benaich and Ian Hogarth).

https://www.slideshare.net/StateofAIReport/state-of-ai-report-2019-151804430

What stands out for you?Did they miss anything important?

​

Two things stand out for me so far:

1) The 2018 prediction they made that a major AI lab would go dark actually happend (MIRI lab and OpenAI -GPT2)

2) For me a as a fps CTF lover: the the Deepmind Quake III RL is wonderful to see. In 2019 they managed to include all the powerups which was a common criticism last year. (Updated blogpost here: https://deepmind.com/blog/capture-the-flag-science/)

I’m still going through the presentation myself so if I’ll see more interesting stuff I’ll add it to this thread.

https://www.youtube.com/watch?v=s7O3oCWZgjE

I’m nearing the end of my job interview process and I’m finally going to meet with the research lead soon. The role I am applying for is heavily involved with deep reinforcement learning R&D at a prestigious company. The title itself, if relevant, is ‘AI Engineer’. I feel very nervous and am wondering what kind of questions I should expect / prepare for. Do you guys have any advice on what I should read up on?

Say you want to add something to your model that you think might improve overall performance, e.g. some feature engineering or you decide to add to your training data, some new data you acquired. How do you make sure that this is actually increasing performance and that it is not just due to the randomness of the process ?

I kinda see how it goes for traditional ML with traditional IID assumption, fix a seed for anything that is random and just compare. But what about deep learning models ?

For instance, say you have your neural network tuned for some past state. Wouldn’t comparing past configuration with new configuration (with the added features or training data) on the same network be biased ? Maybe the feature engineering was relevant but because the network isn’t large enough, it is not able to process those additional features. Or maybe adding more data changed the loss surface and the learning rate/batch size tuned to the previous configuration is not well fitted to the new configuration ? Or maybe more data meant more updates per epoch (assuming the batch size is the same), so maybe we missed the optimal training state because we only look at the validation loss per epoch. So surely setting a seed for the randomness of the network training (weights initialization, shuffle after each epoch … ) is not enough.

I’ve thought of doing some sort of autoML/gridsearch to optimize on the learning rate/batch size for several seeds on the weights initialization and do some statistical significance on the results but this would take way too much time considering how many things I need to check. I feel like a statistical study on a given network (with hyperparameters fixed) for different weights initialization might not be relevant.

I’m asking this because whenever I change something on the preprocessing side (new feature, new data, different scaling …), or even weights initialization of the network, the “optimal” learning rate I find by hand tuning my network is never the same (and can differ a lot).

Any idea is welcome!

I finished Simple XLNet implementation with Pytorch Wrapper!

You can see How XLNet Architecture work in pre-training with small batch size(=1) example.

Also I added comment in code, so you can learn XLNet Architecture more easily

https://github.com/graykode/xlnet-Pytorch

AI Benchmark is an open source python library for evaluating AI performance of various hardware platforms, including CPUs, GPUs and TPUs. The benchmark is relying on TensorFlow machine learning library, and is providing a lightweight solution for assessing inference and training speed for key Deep Learning models, including:

​

1. MobileNet-V2
2. Inception-V3
3. Inception-V4
4. Inception-ResNet-V2
5. ResNet-V2-50
6. ResNet-V2-152
7. VGG-16
8. SRCNN 9-5-5
9. VGG-19
10. ResNet-SRGAN
11. ResNet-DPED
12. U-Net
13. Nvidia-SPADE
14. ICNet
15. PSPNet
16. DeepLab
17. Pixel-RNN
18. LSTM
19. GNMT

​

It is currently distributed as a Python pip package, installation instructions can be found at http://ai-benchmark.com/alpha.html and https://pypi.org/project/ai-benchmark/

​

Note: Fast installation [if TensorFlow is already installed]:

1. pip install ai-benchmark
2. Run benchmark using the following python code:
   1. from ai_benchmark import AIBenchmark
   2. results = AIBenchmark().run()

​

A detailed information about test setups: http://ai-benchmark.com/ranking_cpus_and_gpus_detailed.html

A short preliminary ranking is available here: http://ai-benchmark.com/ranking_cpus_and_gpus.html

​

A global ranking with the results of various hardware and software platforms, drivers / configs and TF builds should be available soon. Original post: http://ai-benchmark.com/alpha.html