There are papers related to making multi-head attention in transformer look at different parts of inputs to reduce redundancy. do you try any? if so, what works well?

​

[1]https://arxiv.org/abs/1910.04500

[2]https://arxiv.org/abs/1808.05377

[3]https://arxiv.org/pdf/1703.03130.pdf

I am reading about node/graph embeddings. It seems that Neural Networks & the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications to generate embeddings from graph-data. However, when generating node embeddings learned from GNNs, I don’t seem to understand how edge information are captured. How do you incorporate edge information (if you have a lot of edge features) to generate graph/node embeddings?. Most of the techniques that I came across [1] [2] don’t consider edge information.

Do you have any recommendation of a paper/reference of a method that incorporate edge rich information to generate embeddings?

Gary Marcus share his thoughts on how we can solve the problem here:

​

https://thegradient.pub/an-epidemic-of-ai-misinformation/

From Neural Architecture Search: A Survey, first published in 2018:

“Moreover, common search spaces are also based on predefined building blocks, such as different kinds of convolutions and pooling, but do not allow identifying novel building blocks on this level; going beyond this limitation might substantially increase the power of NAS.”

I wonder now if somebody tried that since this was written or if anybody has some thoughts about the feasability of this idea.

In other words what could we expect to achieve sooner or later if our theoretical work stopped today and we just focused on collecting more data and adding more compute? Are there any types of problems that definitely not wouldn’t be solvable?

If you have any paper discussion this subject, feel free to share them.

Hi, I saw this video: https://www.youtube.com/watch?v=AhEVk7TLVjQ explaining the difference and I’m quite confused.

What is the actual basis for deciding whether to use pre-trained models or learn from scratch? My understanding is, if you don’t have resources you have to use pre-trained models. If you have enough resources you can learn from scratch with your own data.

1- Imputing a bunch of text and getting an output that tries to replicate it as close as possible 2- The same but with images

Hey all,

​

Not sure if this is the best place to post this, but I am looking to enhance my portfolio and ML skills more importantly through Kaggle competitions and side projects.

​

About me:

cornell grad with coursework experience in ML, did an internship dealing with NLP and CV applications. now working as a data engineer and doing a project dealing with using nlp learning from unstructured data.

currently following along with cs231 and 230 to improve my deep learning skills. Almost done with the andrew ng

​

looking for like minded individuals to partake in side projects or kaggle contests with!

In terms of performance, which would be better? The price is about the same

With the 2080Tis, I have the overhead of inter gpu communication, and have 44 GB of memory, 17408 cuda cores total, 2176 tensor cores total, 1545 Mhz speed

With the 2 Titans, I dont have as much overhead i suppose, have 48GB, only 9216 cuda cores total, 1152 tensor cores total, 1770 Mhz speed.

​

Which do I go with. Once again, the price is about the same