If anyone is a subject expert or simply want to help with the project please send me a pull request or get in contact with me at d.snowatsymbolcomesherejbs.cam.ac.uk. Any help on this project would be greatly appreciated.

Its still very fresh so any ideas/feedback are welcome and certainly appreciated. See below for the industries/areas currently covered.

Link: https://github.com/firmai/industry-machine-learning

Does anyone have experience with finetuning a resnet / VGG model pretrained on imagenet but using a higher resolution for the inputs?

Any advice, papers or slides are most welcome!

Background:

I am currently working on binary classification of images and have had pretty good success by finetuning a resnet50 pretrained on imagenet (this is a pretty good guide https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html ). I have only a few hundred images of each class, so overfitting is a massive concern and thus unfreezing the lower parts of the resnet would not work.

However, my images are much higher resolution than imagenet – most tutorials I have found seem to rescale images to 244×244 which is what I am doing right now, but I think that downscaling the images a bit less might make the task easier to solve.

Keras resnet50 implementation accepts higher resolution images, but the model doesn’t improve during training at all.

Hey folks!

Just finished my project: “Quote Me AI” https://github.com/tomasrasymas/quote-me-ai . It’s bot that automatically generates quotes and puts them on image and posts that image to r/QuoteMeAI subreddit.

Take a look!

Any Feedback is welcome!!!

https://medium.com/@xlnet.team/a-fair-comparison-study-of-xlnet-and-bert-with-large-models-5a4257f59dc0

We are the authors of XLNet. We conducted a fair comparison study of XLNet and BERT with large models. In this study, we ensure that almost every possible hyperparameter is the same for the training recipes of both BERT and XLNet, using the same training data.

We have the following interesting observations among others:

1. Trained on the same data with an almost identical training recipe, XLNet outperforms BERT by a sizable margin on all the datasets.
2. The gains of training on 10x more data are smaller than the gains of switching from BERT to XLNet on 8 out of 11 benchmarks.

I frequently run into a problem where I’m dealing with a tensor in a neural network and one of the axes has a dimension of unknown size, which depends on properties of the input data. This can present problems when passing that tensor into fully connected layers, because those expect a tensor of a fixed predetermined size. One thing I’ve noticed is that attention layers seem to be pretty good at dealing with this problem. They can take an axis of an unknown size and “integrate” it out, giving importance to the most relevant entries in that axis. I usually see attention as giving importance to certain words or time stamps in an input series. Does this seem like a valid use for attention?

First of all, sorry if I do not get the terminology right, I am a newbie in machine learning.

I am training a neural network providing batches of numpy arrays, each array consisting of summary statistics (0’s, 1’s, and 2’s) belonging to one of three possible pure data categories, e.g.

input_batch1 = np.array([[0, 1, 1, 2, 1, ...], [1, 2, 2, 1, 0, ...], ... [0, 0, 1, 1, 1, ...]] 

where, in this case, the 1st array belongs to the category 2, the 2nd to the category 3, and the last to the category 1. I also provide the real one-hot probabilities p for calculating the loss, e.g. for the the previous input:

p_batch1 = np.array([[0, 1, 0], [0, 0, 1], ... [1, 0, 0]]) 

I am using the softmax activation function in the output layer, and I calculate the loss with cross-entropy:

-tf.reduce_sum(p_batch1 * tf.log(softmax_output)) 

However, I want to test it with summary statistics resulting from combinations of the 3 possible categories, so it predicts the proportions from each category. For this testing, I would provide “non-one-hot” probability distributions containing the proportions of the 3 categories that result in the input sum. stats., like

p_mixed = np.array([[0.2, 0.8, 0.0], [0.7, 0.2, 0.1], ... [0.2, 0.3, 0.5]]) 

where the first array specifies that the input data was the result of the combination of 20% of the category 1 and 80% of the category 2.

I understand that the proper loss function for this kind of probabilities should be the mean squared difference:

tf.reduce_mean(tf.squared_difference(softmax_output, p_mixed)) 

So my questions are:

1. Is it possible to use a different loss function when testing?
2. Can I train the network feeding it sum. stats. from pure categories and providing the real one-hot probabilities, using cross-entropy as the loss function, and then test its accuracy with mixed sum. stats and providing the real proportions of each category participating in it, evaluate it using mean squared difference as loss function, and expect it to have a good accuracy?
3. Should I expect better results if I already use the mean squared difference as the training loss function? Or it does not work well for one-hot probabilities?
4. Should I better off train it with sum. stats. from mixed categories? I’d rather use the pure categories for training, but I could do this if it really is the best practice.

It says like 1.5 Billion parameters or 345 Million, but what would they be for example? I am confused about this. Because to use the network you give it a sentence or two, I don’t understand where all these parameters are coming from.