Hi guys,

I’d like to share another PyTorch related project some of you (hopefully) will find helpful and interesting. Here is GitHub repository and here is documentation.

Also, if you have any suggestions/improvements/questions for this project I will be glad to answer/help, thanks.

Now, description (taken from project’s readme with minor adjustments):

What is it?

torchfunc is library revolving around PyTorch with a goal to help you with:

• Improving and analysing performance of your neural network (e.g. Tensor Cores compatibility)
• Record/analyse internal state of torch.nn.Module as data passes through it
• Do the above based on external conditions (using single Callable to specify it)
• Day-to-day neural network related duties (model size, seeding, performance measurements etc.)
• Get information about your host operating system, CUDA devices and others

Quick examples

Get instant performance tips about your module. All problems described by comments will be shown by torchfunc.performance.tips:

class Model(torch.nn.Module): def __init__(self): super().__init__() self.convolution = torch.nn.Sequential( torch.nn.Conv2d(1, 32, 3), torch.nn.ReLU(inplace=True), # Inplace may harm kernel fusion torch.nn.Conv2d(32, 128, 3, groups=32), # Depthwise is slower in PyTorch torch.nn.ReLU(inplace=True), # Same as before torch.nn.Conv2d(128, 250, 3), # Wrong output size for TensorCores ) self.classifier = torch.nn.Sequential( torch.nn.Linear(250, 64), # Wrong input size for TensorCores torch.nn.ReLU(), # Fine, no info about this layer torch.nn.Linear(64, 10), # Wrong output size for TensorCores ) def forward(self, inputs): convolved = torch.nn.AdaptiveAvgPool2d(1)(self.convolution(inputs)).flatten() return self.classifier(convolved) # All you have to do print(torchfunc.performance.tips(Model())) 

Seed globaly (including numpy and cuda), freeze weights, check inference time and model size:

# Inb4 MNIST, you can use any module with those functions model = torch.nn.Linear(784, 10) torchfunc.seed(0) frozen = torchfunc.module.freeze(model, bias=False) with torchfunc.Timer() as timer: frozen(torch.randn(32, 784) print(timer.checkpoint()) # Time since the beginning frozen(torch.randn(128, 784) print(timer.checkpoint()) # Since last checkpoint print(f"Overall time {timer}; Model size: {torchfunc.sizeof(frozen)}") 

Record and sum per-layer and per-neuron activation statistics as data passes through network:

# Still MNIST but any module can be put in it's place model = torch.nn.Sequential( torch.nn.Linear(784, 100), torch.nn.ReLU(), torch.nn.Linear(100, 50), torch.nn.ReLU(), torch.nn.Linear(50, 10), ) # Recorder which sums all inputs to layers recorder = torchfunc.hooks.recorders.ForwardPre(reduction=lambda x, y: x+y) # Record only for torch.nn.Linear recorder.children(model, types=(torch.nn.Linear,)) # Train your network normally (or pass data through it) ... # Activations of all neurons of first layer! print(recorder[1]) # You can also post-process this data easily with apply 

Installation

pip

Latest release:

pip install --user torchfunc

Nightly:

pip install --user torchfunc-nightly

One could also check the project out with Docker, see more info in README as this post is getting long I guess

BTW. There is also another project of mine torchdata revolving around data processing with PyTorch, might be of interest to some of you as well (it was announced one week ago here as well, but in case you missed).

It seems the Dirichlet distribution is pretty ubiquitous in its use as a prior for LDA and other types of topic models. However, in variational inference the Dirichlet doesn’t have a closed form reparameterization. What are the various properties that make the Dirichlet so successful, and are there any other distributions or methods that exhibit these properties without having to use the Dirichlet?

Just checking the recent activity and people have started posting missing citations. Now, some of them will be genuine, but some will be self-promotions. However, this year, as there are no anonymous comments, it is going to be quite a show, getting my popcorn ready.

Hi folks!

I wanted to share a project I have been working on. It is a recurrent neural network that predicts bytes which can be used to mimic e.g. lyrical content (such as books). It uses Adams gradient optimization algorithm and the only requirement for using it is GCC. No third parties, just a bunch of C source files needed to be compiled. I have included some pretrained models that might be fun to play around with.

I hope this can serve as an inspiration to anyone wanting to make their own stuff “from scratch”.

Feel free to come with constructive comments on what I can add/change or any such requests. Have a great Monday!

Here is a link to the GitHub repository:

LSTM implementation (GitHub)

Don’t forget to give the repository a star if you like it, it would be really appreciated!

https://github.com/moabitcoin/ig65m-pytorch

https://github.com/moabitcoin/ig65m-pytorch/releases

Hey folks, we ported the r(2+1)d video classification model (from CVPR 2018, see https://arxiv.org/abs/1711.11248) and the weights pre-trained by Facebook Research on over 65 million Instagram videos (from CVPR 2019, https://arxiv.org/abs/1905.00561) to PyTorch and released architecture, weights, tools for conversion, and feature extraction example.

The official Facebook Research codebase can be found at https://github.com/facebookresearch/vmz

These models and pre-trained weights are immensly powerful e.g. for fine-tuning on action recognition tasks or extracting features from 3d data such as videos. Think: resnet+imagenet but for videos.

We hope the PyTorch models and weights are useful for folks out there and are easier to use and work with compared to the goal driven, caffe2 based, research’y official code base.

The deep pockets of large tech companies are luring AI scientists and professors away from academia into business and for-profit endeavors. How will this affect scientific research and machine learning projects that do not necessarily aim to fatten the pockets of big tech? I discussed this in my latest column, would be happy to get your thoughts on it.

https://bdtechtalks.com/2019/09/26/artificial-intelligence-brain-drain/

Hey all, I have large binary classification dataset with follow counts 0: 200k 1: 500k

Now the labels are not true labels. I know the percentage of correct of each label group. By this I mean I know that 80% of 0 are correct and 85% of 1 are correct, I don’t know which.

Now I have tried the following:- ° Random first with class weight – massively overfit and if played around with max _depth parameter to reduce overfitting however I am unable to get good results. ° Tried oversampling like SMOTE etc but they take large amount of time.

Do you have any suggestions how to deal with imbalance and noisy labels?

Are there any such events in NYC? I’d love to meet more people in that space in my area

I was looking at some conference programs and noticed the same names appearing not once, twice, but thrice. Twice as first author, once as second.

I looked up that person and noticed that they had been publishing a lot in 2019 alone. How do they do this? Do they not need time to do actual research?

I’ve been working on some research, and I was going to submit it to a conference later this year, but just now someone else has done the same thing (on a different dataset). Can I still publish this? If not, how do these people that publish so much not have this happen all the time? I essentially have to throw away half a year of work because someone else submitted to a conference with an earlier deadline.

Sometimes I dread research…

We published our work, Neural Volumes, at SIGGRAPH this year. The goal of the paper is to create an animate-able 3D model of objects and scenes from calibrated multi-view video. You can check out the video in that link for a visual explanation of how the method works and to see some of the results. The formulation of the method is basically an autoencoder with a fixed-function rendering technique that renders a volume into an image.

We’ve just put the source code on Github today. Since calibrated multi-view video is relatively hard to come by, we’ve including the dry ice dataset as well as a pretrained model (in the v0.1 release). I’m happy to answer any questions about this work or talk about neural rendering in general.