Are there embeddings for seq2seq tasks? I have a very large vocab and I want to feed a fixed-length vector rather than one-hot vector to the input of the neural network (and for the output)

Interesting research from DeepMind:

“Our new work on memory uses a neural network’s weights as fast and compressive associative storage. Reading from the memory is performed by approximate minimization of the energy modeled by the network.”

“Unlike classical associative memory models such as Hopfield networks, we are not limited in the expressivity of our energy model, and make use of the deep architectures with fully-connected, convolutional and recurrent layers.”

“For this to work, stored patterns must be local minima of the energy. We use recent advances in gradient-based meta-learning to write into the memory such that this requirement approximately holds.”

https://arxiv.org/abs/1910.02720

I’d like to annotate source code (mainly PHP/HTML/JS), highlight some parts of the docs and apply tags. Is anyone aware of this kind of tool?

We are open-sourcing pgANN – an ANN (approx nearest neighbor) approach with a PostgreSQL backend. The key differentiator between pgANN and the rest (FAISS, Annoy,NearPy etc) is:

1. this enables “online” learning i.e. doesn’t require retraining with every CRUD, and
2. works with extremely large datasets, since it’s not held in RAM like the others

We use it internally to QA images and find it consistently provides sub-second query performance with a few million rows of vectors on a 32GB.8 vcpu Ubuntu box and can reasonably be expected to scale-up with normal pgsql scaling techniques. We invite the community to give this a try and share feedback.

http://arxiv.org/abs/1910.08489

Hello, this is my first paper preprint:

Privacy-preserving Federated Bayesian Learning of a Generative Model for Imbalanced Classification of Clinical Data

Though it is not fully biased toward deep learning, or federated learning on a edge device, I made a new framework for learning a global model in a horizontally distributed setting, especially in a clinical field.

AFAIK, it is the first trial to apply Approximate Bayesian Computation(ABC) on federated learning.
(If not, please let me know!)

Without complicated perturbation techniques e.g. Differential Privacy, Homomorphic encryption, Hashing, etc., the proposed method can preserve privacy.

As I said in the paper, unless each local site reveals trained weights and the structure of Autoencoder, shrunk data CANNOT be recovered in the central server. Also not possible even if some local sites conspire against the other site to disclose the information.

• But this is my hypothesis and expectation, so I want to listen to some feedback or opinions on this.
  Is it really impossible to make leakage on data shrunk by local autoencdoer?

Plus, a global model can be learned in the central server with the minimal information (merely with a distance between local data (perturbed via Autoencoder) and generated data (same dim. with the perturbed one)).

Welcome and thank you in advance for any feedback and questions!

In recent years I have heard of key advances made in NLP by transformer models and BERT. Then there was this paper on neural ODEs by DeepMind.

I have been wanting to dig deeper into the details and understand the key ideas behind these hot topics. They are not directly relevant to my work (which is focused mainly on images/videos) but I still feel it is important as an ML engineer to keep myself up-to-date with key developments in diverse areas. However because it does not directly relate to my work, I find it hard to find the time to get a deeper understanding of these papers.

Has anyone found themselves in a similar situation? How do you deal with it?

I am currently preparing for ML Engineer interviews and wanted to learn a design methodology to crack the design rounds. I have looked for standard resources but couldn’t find any. So I thought of coming up with one on my own from my own experiences and material that I have read online. So below is an overview of the approach I have in my mind. Please let me know what you think of it.

​

1. Understand the use case.
   1. Figure out what can be solved deterministically
   2. Figure out what needs to be solved using ‘Data + Machine learning’
2. Pose the problem(1.b) as a math problem.
   1. Come up with an objective that you want to optimize.
3. Select the right data set.
   1. What data is needed. How is the labeling done/derived?
   2. How do you deal with bias, skewed classes,
4. Feature Engineering.
   1. Think about what(information) you need to solve the problem optimally.
   2. What aspects from the data could you use to replicate findings in 4.a?
   3. Transform data into features and a form more apt for model’s learning
5. Model selection:
   1. What model is best suited to solve the problem(2.a) at hand.
   2. Are there any off the shelf(direct or transfer learning) that would help.
6. Training:
   1. How will you train
   2. How will you handle: skewed classes and other problems associated with the dataset.
   3. Validation: what metrics, experiments do you conduct to validate the model’s learning.
7. Productionizing:
   1. How will the solution be deployed?
   2. Performance monitoring, feedback loop/retrain,
   3. Application scaling and model maintenance.