Most networks in tutorials only show examples with 10-1000 output classes. What techniques can be used to build and train networks with a million or more output classes?

It seems to me that using the traditional approach would be horribly computationally expensive. Intuitively, I would expect that some kind of hierarchical approach should be taken. Any links to blogs or papers that show these techniques would be appreciated.

aideeptalk will livestream the expo and posters at NeurIPS 2019 on Twitch at twitch.tv/aideeptalk

To receive a notification when we go live, please follow us and enable notifications on our Twitch channel.

Follow us on twitter.com/aideeptalk for our schedule.

Please pass this on to those who can’t make it to NeurIPS

For more details see our website aideeptalk.com

I’m reading the Attention Is All You Need paper and it doesn’t seem to explain how exactly the Transformer is used to generate a translation. Here’s how I understand it so far (please correct if I’m wrong):

1. A sequence of k tokens comes in as one-hot vectors of length v – the vocab size. This is a (k x v) token matrix.
2. The tokens are embedded in d_m (model size, e.g. 512) dimensional space via multiplication by an Embedding matrix E of dim (v x d_m), yielding a (k x d_m).
3. Positional Encodings added, dim is still (k x d_m).

Encoding:

1. Encoder block takes in the (k x d_m) matrix and outputs another (k x d_m) matrix.
2. Repeat N times to get a final (k x d_m) matrix, i.e. the encoder output.

Now for decoding:

1. The decoder takes in a (p x d_m) matrix and adds positional encodings.
2. The (non-masked) multi-head attention function inside the decoder receives encoder’s (k x d_m) output as key K, and value V, and a (p x d_m) matrix as the query Q, yielding a (p x d_m) output.
3. The final output of the decoder is therefore (p x d_m).

Final output:

1. The (p x d_m) decoder output is mapped to (p x v) by a matrix multiply (Question: they say it’s “tied” to the embedding matrix E, so is this just E^T?).
2. Select the max of each value in the p rows (softmax), so you get p tokens out.

Suppose I want to translate the sequence “This attention paper is super confusing !” into German. Here k = 7, so my encoder outputs a (7 x 512) matrix. From here, can someone walk me through the steps of generating the translation?

Thanks for looking at my question and have an awesome day!

Paper: https://arxiv.org/abs/1810.12805

Video summary: https://www.youtube.com/watch?v=z89BTMQGVn

Earlier related work: https://arxiv.org/abs/1607.04917 (piecewise convexity)

I am not the author. This paper will be presented at NeurIPS this month and exposes some convexity results about piece-wise linear nns under the least squares loss – namely piecewise strong-convexity & the non-existance of differentiable local maxima. The approach is a spectral analysis of the Hessian and weights of the nn. The result is a relatively attractive convergence estimate for sgd.

I guess this provides some more motivation for studying techniques like ADMM which have convergence properties for some classes of piece-wise functions and can exploit lipschitz cts gradients. Nice work!

Hi,

I’m working on an indie game where you evolve teams of agents that each have a neural network, and then battle them against other players team (looks like this: https://youtu.be/EPekL1JMXEY).

I already have an implementation of sparse lstm-ish networks (1), but I’d like to optimize this further and wanted to see what people here have to suggest. Since it’s an evolution based game I don’t use backprop. It also needs to be fairly simple as it all runs on the GPU (which is why I can have simulations of thousands of agents on a single machine). And since it all runs on the GPU I’d prefer something that is fixed size, which is why I’ve stayed away from NEAT so far.

So my question is; what would be the best network for something like this?

(1) My current network works as follows; Each node has: a state (1 float), 12 indices, 12 weights, 2 bias values. Each index decides which other nodes it “reads” from, so a node can be connected to any other node (layers are therefore less important, they only decide the order of updates). 8 of the inputs are used for the next value of the state. 4 of the inputs are used as a write gate (-1 keep state -> 1 update state). There are some more details but that’s roughly it.

Hey,

We are considering to use Figure Eight in our company.

Has someone tried to use Figure Eight as an annotation plartform? What are the pros and cons? How is it compared to Amazon in terms of price, annotation quality etc?

Would love to hear your opinions.

Researchers who work with Evolutionary Algorithms, do you feel there are current libraries that satisfy your needs? Do you feel there is a need for a new library? What features do you feel are missing/you need?