Updating exponential moving average is a basic tool of SGD methods, starting with of gradient g in momentum method to extract local linear trend from the statistics.

Then e.g. Adagrad, ADAM family adds averages of g_i*g_i to strengthen underrepresented coordinates.

TONGA can be seen as another step: updates g_i*g_j averages to model (uncentered) covariance matrix of gradients for Newton-like step.

I wanted to propose a discussion about some other interesting/promising updated averages for SGD convergence e.g. met in literature?

For example updating 4 exponential moving averages: of g, x, gx, x² gives MSE fitted parabola in a given direction, estimated Hessian = Cov(g,x).Cov(x,x)^-1 in multiple directions (derivation). Analogously we could MSE fit e.g. in a single direction degree 3 polynomial if updating 6 averages: of g, x, gx, x^2, g*x^2, x^3.

Have you seen such additional updated averages in literature, especially of g*x? Is it worth e.g. to expand momentum method by such additional averages to model parabola in its direction for smarter step size?

Deep Learning on Healthcare (6): Regulations. I found quite interesting argument in twitter between influential people such as Hugh Harvey, Jeremy Howard and Luke Oakden-Rayner. So I decided to introduce this argument and discuss about the regulation for deep learning in healthcare and medicine for the last lecture theme.

​

Deep Learning on Healthcare (6)

Deep Learning on Healthcare (5)

Deep Learning on Healthcare (4)

Deep Learning on Healthcare (3)

Deep Learning on Healthcare (2)

Deep Learning on Healthcare (1)

This is a great demonstration of some of the different TTS models I’ve trained and how I can control style:

https://www.youtube.com/watch?v=SXTdnk7-2i0

These models were trained using my implementation of the papers “Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis” (https://arxiv.org/abs/1803.09017) and “Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron” (https://arxiv.org/abs/1803.09047).

I’ve been working on writing an article about AlphaStar for a while (cough, very late), but after last week’s events I decided to sit down and write about OpenAI’s 5 success.

​

There are a few areas I wish I had more knowledge to expand on:

• I wish I knew more about OpenAI’s Rapid to write about.
• Pros and Cons of PPO for Dota 2. I’d also like to know what didn’t work.
• Decision Tree of Starcraft 2 vs OpenAI. Relative to each of the games, OpenAI has solved more of Dota 2’s action space. However, Starcraft 2 seems to have a larger decision tree?
• OpenAI mentioned “surgery” when thinking of transfer learning, but there isn’t much information out there.

​

https://senrigan.io/blog/takeaways-from-openai-5

Very open to feedback and suggestions, thanks!

http://arxiv.org/abs/1904.09730

I’m not even sure if mode collapse is the correct term; neither the generator nor discriminator is learning anything when I pass both true/false samples to the discriminator. If instead I only show the discriminator true or false samples, the loss drops. I’ve seen mode collapse after a few epochs of training other GANs but never complete stagnation out of the gate. What might be going wrong here?

def generator(): neurons = 121 model = Sequential() # Input shape [batch_size,timestep,input_dim] model.add(LSTM(neurons,activation='tanh',recurrent_activation='hard_sigmoid',kernel_initializer='RandomUniform',return_sequences=True)) model.add(LSTM(neurons,activation='tanh',recurrent_activation='hard_sigmoid',kernel_initializer='RandomUniform',return_sequences=True)) model.add(Dense(1,activation=None)) return model def discriminator(): model = Sequential() # Input shape [batch_size,steps,channels] model.add(Conv1D(32,4,strides=2,activation=None,padding='same',input_shape=(None,1))) model.add(LeakyReLU()) model.add(Conv1D(64,4,strides=2,activation=None,padding='same')) model.add(LeakyReLU()) model.add(BatchNormalization()) model.add(Conv1D(128,4,strides=2,activation=None,padding='same')) model.add(LeakyReLU()) model.add(BatchNormalization()) model.add(Dense(128,activation='relu')) model.add(Dense(1,activation='sigmoid')) return model def generator_containing_discriminator(g, d): model = Sequential() model.add(g) d.trainable = False model.add(d) return model def g_loss_function(y_true,y_pred): l_bce = keras.losses.binary_crossentropy(y_tue,y_pred) l_norm = K.sqrt(K.square(y_true)-K.square(y_pred)) return l_bce+l_norm def train(X,Y,BATCH_SIZE): d_optim = SGD(lr=0.002) g_optim = SGD(lr=0.00004) g = generator() d = discriminator() gan = generator_containing_discriminator(g, d) g.compile(loss=g_loss_function, optimizer=g_optim) gan.compile(loss='binary_crossentropy',optimizer="SGD") d.trainable = True d.compile(loss='binary_crossentropy', optimizer=d_optim) num_batches = int(X.shape[0]/float(BATCH_SIZE)) for epoch in range(1000): for index in range(1,num_batches): # Prepare data startIdx = (index-1)*BATCH_SIZE endIdx = index*BATCH_SIZE inputs = X[startIdx:endIdx,:] targets = Y[startIdx:endIdx] # Generate predictions Y_pred = g.predict(inputs) # Build input and truth arrays for discriminator targets = targets.reshape(BATCH_SIZE,1,1) truth = np.vstack((np.ones((BATCH_SIZE,1,1)),np.zeros((BATCH_SIZE,1,1)))) d_loss = d.train_on_batch(np.vstack((targets,Y_pred)),truth) d.trainable = False # Test GAN g_truth = np.ones((BATCH_SIZE,1,1)) g_loss = gan.train_on_batch(inputs,g_truth) d.trainable = True print('Epoch {} | d_loss: {} | g_loss: {}'.format(epoch, d_loss,g_loss)) g.save_weights('generator',True) d.save_weights('discriminator',True) return d,g,gan 

Hi guys,

We have continue updating our open source project for packaging and deploying ML models to product (github.com/bentoml/bentoml), and we have create an easy way to deploy ML model as a serverless (www.serverless.com) project that you could easily deploy to AWS lambda and Google Cloud Function. We want to share with you guys about it and hear your feedback.

A little background of BentoML for those aren’t familiar with it. BentoML is a python library for packaging and deploying machine learning models. It provides high-level APIs for defining a ML service and packaging its artifacts, source code, dependencies, and configurations into a production-system-friendly format that is ready for deployment.

Feature highlights: * Multiple Distribution Format – Easily package your Machine Learning models into format that works best with your inference scenario: – Docker Image – deploy as containers running REST API Server – PyPI Package – integrate into your python applications seamlessly – CLI tool – put your model into Airflow DAG or CI/CD pipeline – Spark UDF – run batch serving on large dataset with Spark – Serverless Function – host your model with serverless cloud platforms

• Multiple Framework Support – BentoML supports a wide range of ML frameworks out-of-the-box including Tensorflow, PyTorch, Scikit-Learn, xgboost and can be easily extended to work with new or custom frameworks.

• Deploy Anywhere – BentoML bundled ML service can be easily deploy with platforms such as Docker, Kubernetes, Serverless, Airflow and Clipper, on cloud platforms including AWS Lambda/ECS/SageMaker, Gogole Cloud Functions, and Azure ML.

• Custom Runtime Backend – Easily integrate your python preprocessing code with high-performance deep learning model runtime backend (such as tensorflow-serving) to deploy low-latancy serving endpoint.

How to package machine learning model as serverless project with BentoML

It’s surprising easy, just with a single CLI command. After you finished training your model and saved it to file system with BentoML. All you need to do now is run bentoml build-serverless-archive command, for example:

 $bentoml build-serverless-archive /path_to_bentoml_archive /path_to_generated_serverless_project --platform=[aws-python, aws-python3, google-python] 

This will generate a serverless project at the specified directory. Let’s take a look of what files are generated.

 /path_to_generated_serverless_project - serverless.yml - requirements.txt - copy_of_bentoml_archive/ - handler.py/main.py (if platform is google-python, it will generate main.py) 

serverless.yml is the configuration file for serverless framework. It contains configuration to the cloud provider you are deploying to, and map out what events will trigger what function. BentoML automatically modifies this file to add your model prediction as a function event and update other info for you.

requirements.txt is a copy from your model archive, it includes all of the dependencies to run your model

handler.py/main.py is the file that contains your function code. BentoML fill this file’s function with your model archive class, you can make prediction with this file right away without any modifications.

copy_of_bentoml_archive: A copy your model archive. It will be bundle with other files for serverless deployment.

What’s next

After you generate this serverless project. If you have the default configuration for AWS or google. You can deploy it right away. Otherwise, you can update the serverless.yaml based on your own configurations.

Love to hear feedback from you guys on this.

Cheers

Bo

Edit: Styling

Hey everyone! I just created a new runtime for Open Neural Network Exchange (ONNX) models called ONNXCV.

Basically, you can inference ONNX models for realtime computer vision applications (i.e. image classification and object detection) without having to write boilerplate code. It is useful in the sense that one can focus on making deep learning models using any deep learning library (that converts models into the ONNX file format), without having to sacrifice time into building the actual inferencing program.

Let me know what you think and if I should continue to build upon this (or not).

Here is the code.