For beginners there’s PRML by Bishop and maybe Understanding Machine Learning by Shai² but for advanced readers or those interested in the deep learning and GAN research landscape (and how to apply it) there really isn’t anything good out there.

I personally don’t like Goodfellow’s Deep Learning book. I wish there was a good deep-dive out there but there just isn’t what I need.

I think Andrej Karpathy is a good writer, kind of wish he could throw something together!

Link to story

This post is not an ML research related post. I am posting this because I think it is important for the community to see how research is applied by authoritarian governments to achieve their goals. It is related to a few previous popular posts on this thread with high upvotes, which prompted me to post this story.

Previous related stories:

• Is machine learning’s killer app totalitarian surveillance and oppression?

• Using CV for surveillance and regression for threat scoring citizens in Xinjiang

• ICCV 19: The state of some ethically questionable papers

• Hikvision marketed ML surveillance camera that automatically identifies Uyghurs

• Working on an ethically questionnable project…

The story reports the details of a new leak of highly classified Chinese government documents reveals the operations manual for running the mass detention camps in Xinjiang and exposed the mechanics of the region’s system of mass surveillance.

The lead journalist‘s summary of findings

The China Cables represent the first leak of a classified Chinese government document revealing the inner workings of the detention camps, as well as the first leak of classified government documents unveiling the predictive policing system in Xinjiang.

The leak features classified intelligence briefings that reveal, in the government’s own words, how Xinjiang police essentially take orders from a massive “cybernetic brain” known as IJOP, which flags entire categories of people for investigation & detention.

These secret intelligence briefings reveal the scope and ambition of the government’s AI-powered policing platform, which purports to predict crimes based on computer-generated findings alone. The result? Arrest by algorithm.

The article describe methods used for algorithmic policing

The classified intelligence briefings reveal the scope and ambition of the government’s artificial-intelligence-powered policing platform, which purports to predict crimes based on these computer-generated findings alone. Experts say the platform, which is used in both policing and military contexts, demonstrates the power of technology to help drive industrial-scale human rights abuses.

“The Chinese have bought into a model of policing where they believe that through the collection of large-scale data run through artificial intelligence and machine learning that they can, in fact, predict ahead of time where possible incidents might take place, as well as identify possible populations that have the propensity to engage in anti-state anti-regime action,” said Mulvenon, the SOS International document expert and director of intelligence integration. “And then they are preemptively going after those people using that data.”

In addition to the predictive policing aspect of the article, there are side articles about the entire ML stack, including how mobile apps are used to target Uighurs, and also how the inmates are re-educated once inside the concentration camps. The documents reveal how every aspect of a detainee’s life is monitored and controlled.

Note: My motivation for posting this story is to raise ethical concerns and awareness in the research community. I do not want to heighten levels of racism towards the Chinese research community (not that it may matter, but I am Chinese). See this thread for some context about what I don’t want these discussions to become.

I am aware of the fact that the Chinese government’s policy is to integrate the state and the people as one, so accusing the party is perceived domestically as insulting the Chinese people, but I also believe that we as a research community is intelligent enough to be able to separate government, and those in power, from individual researchers. We as a community should keep in mind that there are many Chinese researchers (in mainland and abroad) who are not supportive of the actions of the CCP, but they may not be able to voice their concerns due to personal risk.

An open source book compiled by Chip Huyen. Feel free to contribute:

This booklet covers four main steps of designing a machine learning system:

1. Project setup

2. Data pipeline

3. Modeling: selecting, training, and debugging

4. Serving: testing, deploying, and maintaining

It comes with links to practical resources that explain each aspect in more details. It also suggests case studies written by machine learning engineers at major tech companies who have deployed machine learning systems to solve real-world problems.

At the end, the booklet contains 27 open-ended machine learning systems design questions that might come up in machine learning interviews. The answers for these questions will be published in the book Machine Learning Interviews.

project: https://github.com/chiphuyen/machine-learning-systems-design

PDF: https://github.com/chiphuyen/machine-learning-systems-design/blob/master/build/build1/consolidated.pdf

Sharing our recent blog post summarizing some of our recent work understanding the boundaries of the lottery ticket hypothesis. In particular, we make some progress towards the following questions:

• Do winning ticket initializations contain generic inductive biases or are they overfit to the particular dataset and optimizer used to generate them?
• Is the lottery ticket phenomenon limited to supervised image classification, or is it also present in other domains like RL and NLP?
• Can we begin to explain lottery tickets theoretically?

The blog post is below:

Understanding the generalization of “lottery tickets” in neural networks

And the papers covered can be found here:

One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP

Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension

I would like to know anyone’s opinion on this.

Recent work has identified that classification models implemented as neural networks are vulnerable to data-poisoning and Trojan attacks at training time.

Source: Attacks on Deep Reinforcement Learning Agents : https://arxiv.org/abs/1903.06638

​

1. Is it a real threat?
2. How the risk can be identified from someone that just uses the model without access to its source or training data (i.e. prepare a set of tests)?

AI asks fundamental questions about the nature of “intelligence”, but what about understanding life itself? Sina gives an overview of Artificial Life for AI people.

​

https://thegradient.pub/an-introduction-to-artificial-life-for-people-who-like-ai/

Are there any best practices on how one should track gradient norms during training? Surprisingly, I haven’t been able to find much reliable information on it, except the classical Glorot’s paper.

My current approach is to track 2-norm of weights raw gradients. However, I don’t have any practical intuition on which values should make me worried. Tracking the actual weight updates (e.g adjusted by Adam) makes make much more sense, but I haven’t seen anyone doing so.

A few words why am I concerned: I’m working on some exotic NN architecture for 3D, where different architecture choices implicate gradient behavior drastically, up to blow up.