My wishlist:

-I really want to see doc2vec but with contextualized vectors (Bert, Elmo, etc) instead of word2vec. I think it’ll be a slam dunk. I don’t think I’ll ever get around to testing this. If anyone wants to do it, i’ll be happy to give some guidance if it’s needed.

-I would really like to see word2vec or glove tested with a context limited to other words within the same sentence as the target word. Or, perhaps extend the context to any word in the same paragraph. I was sort of planning on doing this, but lost some motivation with the rise of contextualized vectors. I think it would give some great insight though.

NeurIPS2019 paper, looks interesting:

https://arxiv.org/abs/1904.00243

​

Symmetry-Based Disentangled Representation Learning cannot only be based on static observations: agents should interact with the environment to discover its symmetries.

I’m not familiar with this line of research, but it seems like this could have significant implications on how models are trained, as many current benchmark datasets are static. I’d be interested in hearing thoughts from those more familiar with the method.

Hey everyone,

I’ve written a new blog post (https://rajatvd.github.io/Factor-Graphs/) on an awesome visualization tool that I recently came across — factor graphs. Initially, I encountered them in the context of message passing on graphical models, but soon realized that they are useful in more general contexts.

This is the first post in a series that covers the basics and mainly focuses on understanding how factor graphs work as a visualization tool, along with a cool example of a visual proof using them. In future posts, I plan to cover algorithms like message passing and belief propagation using this visualization framework.

I made the animations using manim, a math animation tool created by the amazing 3blue1brown. I built a small library, manimnx, on top of manim to help interface it with the graph package networkx. You can find the code for the animations in this github repo.

Feedback is welcome!

Are there embeddings for seq2seq tasks? I have a very large vocab and I want to feed a fixed-length vector rather than one-hot vector to the input of the neural network (and for the output)

Interesting research from DeepMind:

“Our new work on memory uses a neural network’s weights as fast and compressive associative storage. Reading from the memory is performed by approximate minimization of the energy modeled by the network.”

“Unlike classical associative memory models such as Hopfield networks, we are not limited in the expressivity of our energy model, and make use of the deep architectures with fully-connected, convolutional and recurrent layers.”

“For this to work, stored patterns must be local minima of the energy. We use recent advances in gradient-based meta-learning to write into the memory such that this requirement approximately holds.”

https://arxiv.org/abs/1910.02720

I’d like to annotate source code (mainly PHP/HTML/JS), highlight some parts of the docs and apply tags. Is anyone aware of this kind of tool?

We are open-sourcing pgANN – an ANN (approx nearest neighbor) approach with a PostgreSQL backend. The key differentiator between pgANN and the rest (FAISS, Annoy,NearPy etc) is:

1. this enables “online” learning i.e. doesn’t require retraining with every CRUD, and
2. works with extremely large datasets, since it’s not held in RAM like the others

We use it internally to QA images and find it consistently provides sub-second query performance with a few million rows of vectors on a 32GB.8 vcpu Ubuntu box and can reasonably be expected to scale-up with normal pgsql scaling techniques. We invite the community to give this a try and share feedback.

http://arxiv.org/abs/1910.08489

Hello, this is my first paper preprint:

Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data

Though it is not fully biased toward deep learning, or federated learning on a edge device, I made a new framework for learning a global model in a horizontally distributed setting, especially in a clinical field.

AFAIK, it is the first trial to apply Approximate Bayesian Computation(ABC) on federated learning.
(If not, please let me know!)

Without complicated perturbation techniques e.g. Differential Privacy, Homomorphic encryption, Hashing, etc., the proposed method can preserve privacy.

As I said in the paper, unless each local site reveals trained weights and the structure of Autoencoder, shrunk data CANNOT be recovered in the central server. Also not possible even if some local sites conspire against the other site to disclose the information.

• But this is my hypothesis and expectation, so I want to listen to some feedback or opinions on this.
  Is it really impossible to make leakage on data shrunk by local autoencdoer?

Plus, a global model can be learned in the central server with the minimal information (merely with a distance between local data (perturbed via Autoencoder) and generated data (same dim. with the perturbed one)).

Welcome and thank you in advance for any feedback and questions!

In recent years I have heard of key advances made in NLP by transformer models and BERT. Then there was this paper on neural ODEs by DeepMind.

I have been wanting to dig deeper into the details and understand the key ideas behind these hot topics. They are not directly relevant to my work (which is focused mainly on images/videos) but I still feel it is important as an ML engineer to keep myself up-to-date with key developments in diverse areas. However because it does not directly relate to my work, I find it hard to find the time to get a deeper understanding of these papers.

Has anyone found themselves in a similar situation? How do you deal with it?

I am currently preparing for ML Engineer interviews and wanted to learn a design methodology to crack the design rounds. I have looked for standard resources but couldn’t find any. So I thought of coming up with one on my own from my own experiences and material that I have read online. So below is an overview of the approach I have in my mind. Please let me know what you think of it.

​

1. Understand the use case.
   1. Figure out what can be solved deterministically
   2. Figure out what needs to be solved using ‘Data + Machine learning’
2. Pose the problem(1.b) as a math problem.
   1. Come up with an objective that you want to optimize.
3. Select the right data set.
   1. What data is needed. How is the labeling done/derived?
   2. How do you deal with bias, skewed classes,
4. Feature Engineering.
   1. Think about what(information) you need to solve the problem optimally.
   2. What aspects from the data could you use to replicate findings in 4.a?
   3. Transform data into features and a form more apt for model’s learning
5. Model selection:
   1. What model is best suited to solve the problem(2.a) at hand.
   2. Are there any off the shelf(direct or transfer learning) that would help.
6. Training:
   1. How will you train
   2. How will you handle: skewed classes and other problems associated with the dataset.
   3. Validation: what metrics, experiments do you conduct to validate the model’s learning.
7. Productionizing:
   1. How will the solution be deployed?
   2. Performance monitoring, feedback loop/retrain,
   3. Application scaling and model maintenance.