I was reading Andrej Karpathy’s blog on RNN to get familiarised with working of RNN, both mathematically and intuitively. From my understanding, there are three sets of parameters to optimise.

1. Wxh – multiple with new input to give a hidden state
2. Whh – multiply with rolling hidden state to add to it the above hidden state
3. Why – multiple with the rolling hidden state to obtain the output

And we have the rolling hidden state (H) which accumulates all the information from the inputs. And we optimise on the loss calculated from the output to find the best set of above params

What I am not able to visualise and understand is the part in which the so-called sequential memory is stores ?

Is it stored in the vector H (the rolling hidden state) or the weight matrix Whh ?

In either case, could you also give some intuition on how it contains memory in the form of matrix / vector ?

Does anyone know if it’s possible to do adversarial attacks on YouTube Content ID to make it not detect or detect something else. The most information about the system I could find was 10 years old. See [here]( https://stackoverflow.com/questions/649116/youtube-content-identification-technology). The system is probably very different now.

​

Any information to help me get started would be awesome.

Hello. I am currently enrolled in CMU and very conversant in linear algebra, multivariable calculus, differential equations, and probability and stats. I was wondering if I should jump right into stanford’s cs 229 machine learning course or go through the introduction to elements of statistical learning lecture first. I haven’t learned R yet, but I do know a bit of python.

​

Which way would be the most effective? Any advice would be so thankful! 🙂

I just came across this recent post by Andrew Ng’s deeplearning.ai team, talking about the importance of initialization. The interactive demos are fantastic! I haven’t seen it shared anywhere, so thought I would do so here.

http://www.deeplearning.ai/ai-notes/initialization/

I have ANN Toolbox for Scilab and some script which I do not understand completely. I studied that at least 8 years ago but don’t remember much. What I need is help understanding the script and making few changes to it. The main thing I would like to see is correct chart for the computations. The code:

clear clc // data X = [1.00 2.00 3.00 2.00 3.00 4.00 3.00 4.00 5.00 4.00 5.00 6.00 5.00 6.00 7.00 6.00 7.00 8.00]; // libraries exec("ann_FF_init.sci"); exec("ann_FF_Mom_online.sci"); exec("ann_d_sum_of_sqr.sci"); exec("ann_d_log_activ.sci"); exec("ann_FF_run.sci"); exec("ann_log_activ.sci"); // k - predicition range? k=2; // data rows and cols count [rows,cols]=size(X); // normalize the learning data - what it does? makes values in range from 0.00 to 1.00? for i=1:cols X(:,i)=X(:,i)/sqrt(X(:,i)'*X(:,i)); end; // learning series, pair <U,Z>, automatic conversion of X which is a column - what happens here? U=[]; Z=[]; for i=1:rows-k-1 U=[U X(i:i+k-1)]; Z=[Z X(i+k)]; end; // NN structure [in_count, pattern_count]=size(X); [out_count, pattern_count]=size(Z); // neurons count in layers N=[in_count 10 7 out_count]; r=[0,1]; rb=[0,1]; // initialize W=ann_FF_init(N,r,rb); // learning parameters lp=[0.1 0.05 0.5 0.1]; lp=[0.1 0.05 0.5 0.1]; // epochs count T=12000; 500 for testing epochs=500; // learning; X/x = training, Z/t = output, N = architecture, W = init weights, lp = learning rate, epochs/T = iterations [W,sW]=ann_FF_Mom_online(X,Z,N,W,lp,epochs); // full run Y=ann_FF_run(X,N,W); // show data - which are what? //Z' //disp(Y); //(Y-Z)' // plot data - how to plot input? how to plot prediction correctly? //plot(X); //plot(Y,"r"); //plot(Z); 

The questions I have given in the code. Can anyone explain me the script?

Hi, im doing a project where I am classifying foreground and background elements using HOG features and SVM ,
I have a sliding window that goes over an image and extracts hog features, it compares each result to a ground truth image and stores the foreground and background in the correct array.

as the background is a lot larger i am currently taking the first x amount from background to match up with the foreground to make it even.

​

is this a good idea or should i run it with all the background data and make it unbalanced, or even more so, is it ok to keep repeating the foreground to match the background (i think this is a very bad / pointless idea)

Thanks in advance.

Hey folks,

I have been working on a customizable implementation of vanilla DC GAN. I scrapped icons dataset across the internet. Then converted those icons to MNIST like images and trained GAN. Here is link to my project.

https://github.com/NaxAlpha/xgan

And the follow up blog post:

https://medium.com/@NaxAlpha/this-icon-does-not-exist-an-application-of-gans-to-icon-generation-5442f0f867a

During training, I tried different architectures and batch sizes. Some results were as following:

– network learned to create hand-drawn icons after ~70 epochs but generator failed after ~200 epochs

Do you guys have any suggestions to improve results?

Thanks

Like the title says, looking at the cost of entry-level GPU instances on the major clouds:

AWS: p2.xlarge — 1 Tesla K80, 4 vCPUs, 61gb ram $0.900/hr

Azure: NC6 — 1 Tesla K80, 6 vCPUs, 56gb ram $0.900/hr

GCP: 1 Tesla K80, 6 vCPUs, 52gb ram $0.663/hr

Further, for training CNNs on the K80 I never exceed 4-5gb of memory usage or reasonable utilization of 4 vCPUs. Since GCP is the only cloud that gives me ability to finely tune specs I can even further decrease cost for ML applications. For example:

GCP: 1 Tesla K80, 4 vCPUs, 5gb ram $0.424/hr

When benchmarking resnet50, this cheaper configuration provides no performance decrease compared to the more expensive instance.

Perhaps spot instance pricing (low-priority for azure, preemptible for GCP) comes into play, where GCP is in the middle of the pack:

AWS: p2.xlarge — 1 Tesla K80, 4 vCPUs, 61gb ram $0.270/hr

Azure: NC6 — 1 Tesla K80, 6 vCPUs, 56gb ram $0.180/hr

GCP: 1 Tesla K80, 6 vCPUs, 52gb ram $0.236/hr

This kind of instance, however, does not work for every use case so the regular on-demand pricing difference is still significant.

This all leaves me wondering:

If you train your models on the cloud, which provider do you use?

Can you imagine any reasons/use cases/etc that might warrant picking a provider other than GCP?

What is GCPs business model? How can they make money selling for so much less? Is this a loss leader to gain market share?

So, I’ve been doing a deep dive into understanding Transformer (in the Neural Machine Translation context).I’ve found The Illustrated Transformer and The Annotated Transformer much help.

​

So, after great lengths I think I’ve gotten solid intuition on what the self-attention layer will learn. It will essentially learn a contextualized meaning for each word in the input function (correct me if I’m wrong here).

Then each of those “contextualized-meaning embeddings” are then put through the same 2 layer, fully connected feed-forward network – which has an output of the same size (512), with a much larger hidden layer.

The output is then feed into the next Transformer layer, with a batch norm, and a residual connection along for the ride (Going to try leave them out of this for a while if possible)

Do we have any idea what the that feed-forward neural network actually learns? What is it’s purpose?Or why the same feedforward is applied to each “contextualized word”? Is it sort of learning what might be important? (But then again, didn’t the WO matrix that took the multi-head attention matrices into a single matrix learn to do the same thing?)