Hi,

I was trying to use a 2d relative position encoding in my transformer network and couldn’t find one in pytorch,

So I decided to change the tensor2tensor’s implementation into pytorch and added 3d and 1d support as well.

Also because of the heavy usage of attention in the field, I decided to implement that same function in cuda.

It is not a general purpose cuda kernel, and only works great in my settings (large batch size with relatively small patch size) but it might be worth it to check the performance on your settings (I’m getting 2.5x speed up in my forward and 1.5x on my backward)

One last thing, It also supports the B and D attention terms in the Transformer-XL paper.

here is the https://github.com/Separius/CudaRelativeAttention

The title says it all. I thought I’d ask the community! Please do share your thoughts and unique perspectives. Also curious to hear about RL in general.

Few years ago, I had stumbled upon Michael Jordan’s book recommendations for budding researchers in the field. Since then, I have tried to go through many of them even though I am not currently enrolled as a researcher in any university.

Here is the link for the book recommendation.

https://honglangwang.wordpress.com/2014/12/30/machine-learning-books-suggested-by-michael-i-jordan-from-berkeley/amp/

If you check out the books, the recommended books give an idea as to how Prof. Jordan view the problem of Machine Learning and what approaches he prefers more than other.

Then it occured to me that I do not have similar book recommendations from other noted researchers to understand about what they think to be a good approach towards solving machine learning related problems.

So, I would like to call for help in crowdsourcing the book recommendations from noted researchers in this particular thread for the ease of all.

If they have previously mentioned somewhere, please link the post.

If they haven’t and you or anyone you know are working under them, it would be great if you could get them to answer the question:

What are some must read books for people who want to devote many decades of their lives to the field?

Thank you !

Hello. I’m currently working on my undergraduate capstone project and checking other schools’ featured projects for ideas. I was particularly interested in a couple of projects developed for stock price prediction using RNN:

• ML-based Investment Analytical Tool, from UBerkeley’s Master in Information and Data Science. Using stock prices from Yahoo, fundamental data from Intrino and news data from Google News they try to predict stock price evolution for some S&P 500 companies using LSTM networks.

• Machine Learning Engineer Nanodegree: Using only stock prices from Yahoo and also using LSTM and Stacked LSTM networks, they try to predict stock price evolution and also added an algorithm that recommends trades based on those predictions.

I’ve also checked some work based on Restricted Boltzmann Machines stacked over a Multi-layer Perceptron to classify stocks in “going up” and “going down” depending on whether the NN predicts them to go.

Having considered that, what do you think of the following idea: try to pull information from S&P 500 companies (both stock price evolution and fundamental data, as far back as I can) and try to come up with a good model using deep belief networks and deep learning networks to predict stock price evolution. I’d be creating a baseline model using logistic regression or multi-layer perceptron to compare performance.

Do you think it’s doable? Do you think it is an interesting project to carry on or would it be something that everyone knows leads nowhere because companies use Algorithm X and Model Y for this purpose? Do you think this can be done with limited hardware available (just a PC)?

Thank you.