Hi all,
for my dataset t-SNE produces beautiful clusters with some transition in between clusters and a plot that overall is just very exciting. PCA on the other hand just produces very boring results.

Now I’m aware that t-SNE will try much harder to cluster stuff than PCA, so I’m not sure what to make of it.

Can I somehow verify I’m not seeing artefacts that are based on the workings of t-SNE?

I can’t share the data, but here are some crudely drawn examples <3 https://imgur.com/a/7gQPrMA

Thanks!

From Paper is easy to understand that BERT input is composed by Token Embeddings, Positional Encode, Sentence Encode. The last two are well-defined in BERT paper and in “Attention is all you need”. But Token embeddings is not clear how are build. Reading on Internet I found different opinions. For sure tokenization is performed using WordPiece Tokens and it’s easy understand how it splits words. But once you have the token id how BERT converts it in a Embedding?

We build a theoretical framework for analyzing disentanglement in the weakly supervised regime. We provide new definitions for disentanglement (sorry) that can be measured in a weakly supervised manner, and use these definitions as the cornerstone for developing a calculus and theory of disentanglement. We then analyzed several weak supervision techniques and proved (and empirically demonstrated) their disentanglement guarantees (or lack thereof).

We hope that the concepts developed in this paper will help researchers frame their discussion and analysis of weakly supervised disentanglement in future work.

Paper: https://arxiv.org/abs/1910.09772

Cute gif: https://twitter.com/i/status/1187507675258486784

One month ago I post an introduction to what I was working on, and someone suggested I should “create *something*, anything that people can look at to try and understand “. So I have written an article now. This is the link.

This article proposes a self-supervising machine learning architecture which is actually a two-step model. The first step is to construct a causal representation model, and the second step is to promote intentions for it to complete tasks. The output cannot be supervised directly but it is action control signals sparsely coding information. In these situations, the end-to-end supervised learning is not applicable anymore.

Here is the former post a month ago.

tl;dr: Using location-relative attention mechanisms allows Tacotron-based TTS systems to generalize to very long utterances.

Abstract:
Despite the ability to produce human-level speech for in-domain text, attention-based end-to-end text-to-speech (TTS) systems suffer from text alignment failures that increase in frequency for out-of-domain text. We show that these failures can be addressed using simple location-relative attention mechanisms that do away with content-based query/key comparisons. We compare two families of attention mechanisms: location-relative GMM-based mechanisms and additive energy-based mechanisms. We suggest simple modifications to GMM-based attention that allow it to align quickly and consistently during training, and introduce a new location-relative attention mechanism to the additive energy-based family, called Dynamic Convolution Attention (DCA). We compare the various mechanisms in terms of alignment speed and consistency during training, naturalness, and ability to generalize to long utterances, and conclude that GMM attention and DCA can generalize to very long utterances, while preserving naturalness for shorter, in-domain utterances.

Paper: https://arxiv.org/abs/1910.10288

Audio Examples: https://google.github.io/tacotron/publications/location_relative_attention

Hi all,

How do CNNs understand “compositional semantic relationships” between their classes? The problem exists in the entire field, but I’m referencing this paper in particular: http://gandissect.csail.mit.edu/

In the 3rd paragraph of the introduction of the paper, Bau et al. say (emphasis mine):

To a human observer, a well-trained GAN appears to have learned facts about the objects in the image: for example, a door can appear on a building but not on a tree. We wish to understand how a GAN represents such structure. Do the objects emerge as pure pixel patterns without any explicit representation of objects such as doors and trees, or does the GAN contain internal variables that correspond to the objects that humans perceive?

From my (limited) understanding of CNNs, they take in the input image (HxWx3 channels) and pass it through a bunch of filters and maxpool layers. Each maxpool layer reduces the HxW of the matrix. Each filter layer increases the depth of matrix as we move away from the image and toward the “highest levels of representation.” In a sense, we’re abstracting toward higher and higher level features as the receptive field of our neurons increases and we’re able to model longer-distance relationships.

Finally, the last layer of the CNN is connected to the class output by a fully connected layer.

From what I’ve read, it seems like the field is a bit split on this. You have this paper saying

oh look! GANs (and thus CNNs) can understand relationships between classes because doors don’t appear in the sky

But others also say it’s an explicit shortcoming of the convolution function — that the spatial equivariance of convolution means it inherently cannot understand these relationships.

A CNN only looks for 2 eyes, 1 nose, and 1 mouth. It doesn’t care that the eyes are parallel and above the nose, or that the nose is above the mouth!

————

My take is that the CNN can understand broadly the correlations between classes because of the last, FC layer. As a result, it can understand that maybe standing is negatively correlated with beer, or that pens are correlated with paper. But, it can’t understand spatial relationships.

What do you guys think of this issue?

Disclaimer: this is my paper that I’ve been working on, if this sort of thing is not allowed on /r/ml please let me know.

arXiv page: https://arxiv.org/abs/1910.10294

Abstract: This paper presents a novel unifying framework of bilinear LSTMs that can represent and utilize the nonlinear interaction of the input features present in sequence datasets for achieving superior performance over a linear LSTM and yet not incur more parameters to be learned. To realize this, our unifying framework allows the expressivity of the linear vs. bilinear terms to be balanced by correspondingly trading off between the hidden state vector size vs. approximation quality of the weight matrix in the bilinear term so as to optimize the performance of our bilinear LSTM, while not incurring more parameters to be learned. We empirically evaluate the performance of our bilinear LSTM in several language-based sequence learning tasks to demonstrate its general applicability.

​

Comments: This approach is novel because it considers improvement through the use of bilinear neurons (essentially polynomial regression + nonlinearity) as a building block. This is typically not done in neural networks as it is typically accepted that linear neuron + nonlinearity is sufficient as a universal approximator. However, we find that performance improvement can be achieved without incurring additional learnable parameters if bilinear neurons are used. It should be noted that the original proof on the universal approximability of linear neurons (Cybenko, 1989) does not show that they are efficient.