This is a throwaway account since I don’t wish to reveal my affiliation.

I am currently a PhD student at one of the top 10 groups in ML in the USA and am loving my studies and really wouldn’t want to be doing anything else at this current moment. Given this, it’s an extremely specific set of circumstances that led me to do a PhD. To be blunt, outside of top tier programs, I don’t see any benefits of doing a PhD. I have interned at a top company and every single employee I worked with had come from the same set of about 50 labs, with many publications in the same top 3 conferences. I have realized that there is a very small set of (extremely) high paying research jobs for a very small selected amount of outstanding PhD graduates. Publications in theoretical ML are also dominated by a very small amount of universities and groups and if you are not in one of these groups you are unlikely to ever come close to these conferences.

If you are offered a PhD in a group that does not have consistent excellence in the top tier ML conferences, you do not have a shot at these research scientist roles (where we hear about the 300k+ salaries in ML). Everyone else I have seen has ended up in very traditional data scientist roles that could have been easily gained out of a masters or undergraduate degree. These roles do not require the research skills that we spend 4-7 years learning during a PhD and these research skills could even be a detriment to most commercial applied ML roles (since most of them are engineering oriented). Work experience is also extremely valuable in these roles and can present opportunities for raises that double your salary in the equivalent time to complete a PhD.

​

I hear a lot of “I have a PhD/MSc in a related field” on this forum, but these credentials do not honestly carry any weight in getting the top tier research jobs that the media likes to discuss. Do a PhD because you would love to make major theoretical contributions and have a supervisor that has the ability and track record to guide you to those contributions. If the right formula exists, the money will come also.

​

As a side note, concerning PhD admission, I also hear here that publications are not essential. In the current year, almost every admitted student had a top tier first-author conference publication during undergraduate studies. It is absolutely essential without interning at your target lab for 1+ years before applying.

I have seen some conferences that I would like to attend or be a part and various deep learning summer schools. The issue is that most of these can be quite expensive, require you to travel and require you to apply to attend (i’m not sure if I would make the requirements to attend) and i’m only an undergrad student. Is it worth trying to attend or such things when you’r not an expert in the field or what can be an alternate if you’re not able to physically attend but would still like to learn from?

I’m currently working toward my thesis in ML (Graph NN to be specific) however my supervisor isn’t very knowledgeable in ML techniques as I’m in an adjacent field.

Does anyone have any advice on how I can make the most of this situation? I currently attend paper reading groups but most of the time I am working alone and have no one to consult or bounce ideas off.

After looking at this paper and the image generation quality : Generating Diverse High-Fidelity Images with VQ-VAE-2, I wondered if such models can be used for text generation. And I noticed that most of the similar papers discussing discrete latent space models, don’t talk about text at all, all the experiments seem to be done on images or audio domain.

What gives ? Intuitively I would think it makes sense for discrete latent space models to be quite useful for text considering it’s discrete nature. Can anyone explain this or provide links to papers that successfully generate quality text from a discrete latent space?

Building on StyleGAN, I generated a set of fake basketball players, starting from pics of NBA players. Now I would like to map the latent space using the number of points each player made in the last season. My idea is to adapt what Puzer did, mapping the latent space using the prolificacy of the different players.

​

How should I proceed after labeling the players images? I have no landmark model I can build on.

I am a little confused about how the actual training of models like ELMo and BERT can be achieved. In ELMo, the model predicts a representation of a word given its backward and forward context, while in BERT the encoder of the transformer model uses the attention mechanism over other words in the input to determine the representation for the masked word(s). In both models the representation is fed to a softmax layer over the vocabulary, correct? So say we have the two sentences “the bank of the river” and “the central bank of Germany”. The word “bank” should get different representations in the sentences because of the different contexts. However, if this representation is sent to the softmax layer, both would like the output to have the highest probability for the index of the word “bank” in the vocabulary. How is this achieved if the two representations are different? How can we condition to learn to create different contextual representation if we, in the end, still want to end up with the same word in the vocabulary from the softmax output? Should this not result in all representations being conditioned on the same thing regardless of context, i.e. the softmax output having the highest probability for the true target word?

I know this question might sound stupid, but let’s consider we have infinite amount of time for a simple classification task. What would be the minimal computational device able to perform such task? Has a net this “enefficient” been ever built?

Landing page: https://www.biorxiv.org/content/10.1101/620880v1

Paper: https://www.biorxiv.org/content/biorxiv/early/2019/04/28/620880.full.pdf

​

In this paper, the IGLOO structure is used as an alternative to RNNs for the biological task of predicting whether a RNA sequence codes for proteines or not. This approach improves upon standard methods. Contrary to GRUs and LSTMs, IGLOO can deal with a large number of time steps, i.e. more than 25,000 which is particularly suitable for biological data.

Why does MAE work much worse than CCE although it is noise-robust?

How to improve MAE against CCE to embrace noise-robustness and high generalisation performance?

​

To find an answer, please check and feel free to contact:

https://arxiv.org/abs/1903.12141 (Xinshao Wang, Elyor Kodirov, Yang Hua, Neil M. Robertson)

​

Abstract:

Label noise is inherent in many deep learning tasks when the training set becomes large. A typical approach to tackle noisy labels is using robust loss functions. Categorical cross entropy (CCE) is a successful loss function in many applications. However, CCE is also notorious for fitting samples with corrupted labels easily. In contrast, mean absolute error (MAE) is noise-tolerant theoretically, but it generally works much worse than CCE in practice. In this work, we have three main points. First, to explain why MAE generally performs much worse than CCE, we introduce a new understanding of them fundamentally by exposing their intrinsic sample weighting schemes from the perspective of every sample’s gradient magnitude with respect to logit vector. Consequently, we find that MAE’s differentiation degree over training examples is too small so that informative ones cannot contribute enough against the non-informative during training. Therefore, MAE generally underfits training data when noise rate is high. Second, based on our finding, we propose an improved MAE (IMAE), which inherits MAE’s good noise-robustness. Moreover, the differentiation degree over training data points is controllable so that IMAE addresses the underfitting problem of MAE. Third, the effectiveness of IMAE against CCE and MAE is evaluated empirically with extensive experiments, which focus on image classification under synthetic corrupted labels and video retrieval under real noisy labels.