Abstract: We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are nonconvex already in the single layer case. When replacing the last layer of the neural nets by our bi-temperature generalization of logistic loss, the training becomes more robust to noise. We visualize the effect of tuning the two temperatures in a simple setting and show the efficacy of our method on large data sets. Our methodology is based on Bregman divergences and is superior to a related two-temperature method using the Tsallis divergence.

https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html

We made a comparison of various Pandas functions for indexing, vectorization and filtering. We benchmarked and compared their performance. For example, we found out that using a vectorized function to transform data is 82000x faster than using a for-loop with iloc[]. Check out the blog post with more details on various Pandas optimizations here: https://medium.com/bigdatarepublic/advanced-pandas-optimize-speed-and-memory-a654b53be6c2

Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research

David Schlangen(Submitted on 28 Aug 2019)

“This paper introduces a new task and a new dataset”, “we improve the state of the art in X by Y” — it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.

https://arxiv.org/abs/1908.10747

This is an example of an image classifier built with Keras to identify presence of traffic objects.

Specifically, the VGG16 network was used as the pre-trained model, and the images themselves were manipulated using PIL for cropping purposes, and using Grad-CAM and cv2 to respectively generate a heatmap, and superimpose the heatmap on the relevant image.

Findings can be found here. Hope that you find this of use, and any feedback or tips greatly appreciated.

Hi,

I was trying to use a 2d relative position encoding in my transformer network and couldn’t find one in pytorch,

So I decided to change the tensor2tensor’s implementation into pytorch and added 3d and 1d support as well.

Also because of the heavy usage of attention in the field, I decided to implement that same function in cuda.

It is not a general purpose cuda kernel, and only works great in my settings (large batch size with relatively small patch size) but it might be worth it to check the performance on your settings (I’m getting 2.5x speed up in my forward and 1.5x on my backward)

One last thing, It also supports the B and D attention terms in the Transformer-XL paper.

here is the https://github.com/Separius/CudaRelativeAttention