I strongly believe that there must be alternatives to Google, Amazon or Microsoft for speech recognition. I like the direction in which Common Voice (Mozilla foundation) is headed, but I’m yet to see any production quality models coming out of that.

For the past few months, I’ve been training ASR models on a lot of speech data collected from various sources. Today, I’m releasing a couple of trained models as an open-source library called at16k. The idea is to provide the developer community with production quality models for speech to text conversion.

Check out the following links for more info: Github repo and PyPI project

For examples, BERT and GPT-2, both achieve sota across several NLP tasks.

The field of machine learning underwent massive changes in the 2010’s. At the beginning, the field saw diverse approaches applied to a variety of topics and data structures. Then Alexnet blew away the competition for the Imagenet challenge with his CNN, and the field was forever changed. However, there was a warming up phase. Caffe’s first release was in 2013. Tensorflow and Keras were first released in 2015. Pytorch in 2017. Before Caffe’s first release, groups were largely ill-equipped to do convolutional neural network research as they did not possess the resources to develop an efficient library that would run on the GPU. Research continued mostly the same from 2010 – 2014 while these libraries were being developed. Then the real explosion happened from 2014-2016, where paper submissions (and subsequent accepted papers) exploded at various conferences. And of course, as more people explored CNN based machine learning, the better the community got at designing them, so naturally, the scores on ILSVRC get better and better. Various other empirical benchmarks have been used in a standardized test fashion to explicitly compare methods (models).

Prior to the neural network revolution, and even a little after its beginning, it was still believed that there were some problems that are still unsolvable for learning machines, namely Go. The anomaly to these statistics is of course Deepmind and the like. They showed approximating the value function via a CNN is able to master seven classic Atari games. This was in 2013. In March of 2016, they beat the world champion of Go with a match score of 4-1. AlphaGo showcased the power of combining neural networks with traditional algorithms, which largely went against the tide of end-to-end approaches. In April of 2019, OpenAI 5 stomped the world champions in Dota 2. Deepmind showcased their StarCraft II AI at Blizzcon in November 2019, beating everyone, including the last world champion 4-0.

As the 2010’s wrap up, I encourage you to take a step back from your work and appreciate how far we’ve come. What was your favorite moment of this decade?

TL;DR I wrote this package and I find it useful in my projects

The problem

It’s common to specify script parameters in yaml config files. For example:

models: modelA: densenet121: pretrained: True memory_efficient: True modelB: resnext50_32x4d: pretrained: True 

Usually, the config file is loaded and then various if-statements or switches are used to instantiate objects etc:

if args.models["modelA"] == "densenet121": modelA = torchvision.models.densenet121(pretrained = args.pretrained) elif args.models["modelA"] == "googlenet": modelA = torchvision.models.googlenet(pretrained = args.pretrained) elif args.models["modelA"] == "resnet50": modelA = torchvision.models.resnet50(pretrained = args.pretrained) elif args.models["modelA"] == "inception_v3": modelA = torchvision.models.inception_v3(pretrained = args.pretrained) ... 

This is kind of annoying to do, and every time PyTorch adds new classes or functions that you want access to, you need to add new cases to your giant if-statement.

Use this package and get rid of those if-statements

So I wrote easy_module_attribute_getter.

With this package, the above if-statements get reduced to this:

from easy_module_attribute_getter import PytorchGetter pytorch_getter = PytorchGetter() models = pytorch_getter.get_multiple("model", args.models) 

“models” is a dictionary that maps from strings to the desired objects.

Automatically have yaml access to new classes when you upgrade PyTorch

The nice thing about this package is that if you upgrade to a new version of PyTorch which has 20 new classes, you don’t have to change anything. You automatically have access to all the new classes, and you can specify them in your yaml file.

Merge or override options at the command line

You can also merge or override complex config options (i.e. nested dictionaries) at the command line.

For example, if we want a modelC in addition to the 2 models specified in the yaml file:

python example.py --models {modelC: {googlenet: {pretrained: True, aux_logits: True}}} 

Now the models dictionary will contain modelA, modelB, and modelC. Or if we want to override the models option in the config file:

python example.py --models~OVERRIDE~ {modelC: {googlenet: {pretrained: True, aux_logits: True}}} 

Now the models dictionary will contain only modelC.

Transforms and optimizers

There are also some nice features specifically for transforms and optimizers which you can read about here.

Register your own modules

You can register your own modules so that you have direct yaml access to them as well.

Also note that the base class EasyModuleAttributeGetter, isn’t specific to PyTorch, but I wrote the child class PyTorchGetter since that’s what I’m using this package for.

I hope someone finds this useful!

Hello everyone,

we are researchers from TU-Berlin and UL FRI, and we are doing a research on how people interact with certain explainable AI techniques. We are currently in the process of gathering data and we need people to take part in our survey. If you have 15-20 minutes to spare to participate that would be extremely helpfull. All the details about the survey are explained in the survey itself.

The survey is reachable at this link: https://dss.vicos.si/. It is meant to be solved on a computer.

After we analyze the data we will obviously share the paper 🙂 Thank you in advance!

Hi everyone!

In my own ML projects, I’ve been using Python classes to define my settings such as batch size and learning rate. In many ways, it has been more powerful than Argparse, so I thought I’d make it into a package. Without any more fuzz, introducing Anyfig!! (Gihub repo)

Anyfig is a Python library for creating configurations (settings) at runtime. Anyfig utilizes Python classes which empowers the developer to put anything, from strings to custom objects in the config. Hence the name Any(con)fig.

The basics

1. Decorate a class with ‘@anyfig.config_class’.
2. Add config-parameters as attributes in the class
3. Call the ‘setup_config’ function to instantiate the config object

​

import anyfig import random @anyfig.config_class class FooConfig(): def __init__(self): # Config-parameters goes as attributes self.experiment_note = 'Changed some stuff' self.seed = random.randint(0, 80085) config = anyfig.setup_config(default_config='FooConfig') print(config) print(config.seed) 

At the moment, Anyfig supports inheritance, command line inputs, saving & loading and more ideas in the pipeline. Check the github for some more info or reach out to me.

The benefits might not be obvious at first glance so let me try to sell this.

1. Defining config-parameters at runtime offers dynamic settings. Leverage the power of Python. Also cleans up the code

​

{"loss_fn": "l1loss"} # Lives inside a .json file config = ~parse_jsonfile~ if config.loss_fn = 'l1loss': loss_fn = torch.nn.L1Loss() elif config.loss_fn = 'mse' loss_fn = torch.nn.MSELoss() 

Skip the if else statements. You can do those kinds of operations directly in the config

@anyfig.config_class class Train(): def __init__(self): self.loss_fn = torch.nn.L1Loss() 

1. Class inheritance avoids duplicated code. Good for biiig projects. How often have you started training with the wrong settings? Anyfig can help mitigate that problem.

​

A typical use case for Anyfig would be to have one config-class for training, one for laptop-debugging and one for prediction. Since many of the config-parameters are the same, you can choose one class as the parent class and have the other two inherit from the parent and overwrite selected config-parameters.

import anyfig @anyfig.config_class class Train(): def __init__(self): self.batch_size = 32 self.seed = random.randint(0, 80085) @anyfig.config_class class DebugTrain(Train): def __init__(self): super().__init__() self.seed = 0 @anyfig.config_class class Predict(Train): def __init__(self): super().__init__() self.batch_size = 1 

I’m currently looking for people who are willing to try this out / give feedback 🙂

Vision and Language Group, a deep learning group at IIT Roorkee, has written a summary of the ICCV 2019 best paper awardee ‘SinGAN’ which can be found here:

https://github.com/vlgiitr/papers_we_read/blob/master/summaries/singan.md

Feedback on the same is appreciated and do star the repo if you found it useful.

Hi, I have made an implementation of paper ‘vGraph: A Generative Model For Joint Community Detection and Node Representational Learning’ under NeurIPS Reproducibility Challenge 2019, which you can find here: https://github.com/aniket-agarwal1999/vGraph-Pytorch

Hope you all find it useful, feedback on the same would be appreciated.

Working on estimating position of minimum by modelling where linear trend of gradients interests zero, simple approximation (corr(g,theta)=1) leads to looking obvious:

learning rate ~ sqrt(var(theta)/var(g))

proportional to width of displacement of theta, and inversely proportional to width of displacement of gradients – assuming they are in line (corr(g,theta)=1), such learning rate would take us exactly to g=0 minimum of parabola in one step.

Adaptive variance estimation is just a matter of maintaining two exponential moving averages: of value and of value^2, hence we can e.g. cheaply do it coordinate-wise in SGD – getting 2nd order adaptation of learning rate independently for each coordinate (5th page here).

There is popular square root of mean gradient² in denominator (e.g. RMSprop, ADAM), but have anybody seen use of variance in SGD optimizers?