I am trying to train a BERT model for a certain form of text classification and I realized that it might be useful to know whether a sentence is on the same line or on a newline in a pdf document. Is there any way to encode the newline distance in BERT? My idea was using a different customized positional encoding but I am not sure whether it is the correct approach, and if it is, what continuous function to use. Would love to hear any suggestions on this.

I think of something that’s modelled like a GAN but one side is a real instrument (like a midi controlled organ) as discrimination and the other network as generation.

In that way the network could learn from the real instrument, to generate perfect real sounding midi instruments.

Just an idea – some thoughts about the implementation?

I have a large corpus of source code, and space matters in certain languages (like Python). It seems that https://github.com/rsennrich/subword-nmt splits on space. Are there other packages that will split on ”?

I was actually looking to start my own project on forecasting cryptocurrency prices and stumbled upon this.
Their top AI is predicting at around 63%, i found it rather impressive for it to be predicting at such a high rate.

Does anyone have any idea on what they are doing? Was trying to dive further but the language is quite terrible.

​

Few days ago, in this post, I asked about a way to make BERT smaller. I got some interesting results and found some relevant papers. The basic idea is, given a relatively small labelled dataset, and another much bigger unlabelled set:

1. Train BERT on the labeled set
2. Predict values of the unlabelled set
3. Train a much smaller model using the now the labelled big set

​

I tried it with Logistic Regression and got some interesting results here:

https://towardsdatascience.com/distilling-bert-how-to-achieve-bert-performance-using-logistic-regression-69a7fc14249d?source=friends_link&sk=2e62337d0b44f56409640c27277b99ce