Skip to main content

Blog

Learn About Our Meetup

5000+ Members

GO >

MEETUPS

LEARN, CONNECT, SHARE

Join our meetup, learn, connect, share, and get to know your Toronto AI community. 

JOIN

JOB POSTINGS

INDEED POSTINGS

Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.

JOBS

CONTACT

CONNECT WITH US

Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

CONTACT

Category: Global

Sea of Green: NVIDIA Floods ISC with AI and HPC

Written on . Posted in NVIDIA.

The intersection of HPC and AI is extending the reach of science and accelerating the pace of innovation like never before. It’s driving discovery in scientific astrophysics, weather forecasting, energy exploration, molecular dynamics and many other fields.

That’s why over 3,000 people will flock to ISC High Performance 2019, in Frankfurt, Germany, next week. Attendees will descend on the annual supercomputing conference, running from June 16-20, for scores of talks, demos and workshops to explore the latest HPC breakthroughs.

Hear from NVIDIA Experts at ISC

GPUs are at the heart of accelerating HPC. That’s why you’ll find NVIDIA technology featured in a number of talks and workshops across the show.

Make sure not to miss:

Witness Groundbreaking Technology in Action

GPU computing is the most accessible and energy-efficient path forward for HPC and the data center.

At ISC, dozens of NVIDIA partners will demonstrate the importance of GPU acceleration through a range of exhibits and demos.

Look out for “NVIDIA partner” signs at booths including those from Dell EMC, HPE, Mellanox, Boston, One Stop Systems and Supermicro to discover GPU-powered demos. Across the show, you’ll also find:

  • The AI “Emoji” demo — Pass by one of these demo stations at our partners’ booths and get your emotion read in real time. The Emoji demo performs real-time face detection and can identify a whole range of emotions, including “neutral,” “happiness,” “surprise,” “sadness,” “anger,” “disgust,” “fear”  and “contempt.”
  • The Index Supernova demo — Large, 3D scientific simulations typically take about four months to create and generate over a terabyte of visualization data. With the NVIDIA IndeX SDK running on NGC, researchers can now view and interact with their data, make modifications and focus on the most pertinent parts of the data — all in real time.

Students Battle It Out in Cluster Challenge

For this year’s Student Cluster Competition, half of the teams have chosen to build based on NVIDIA V100 Tensor Core GPUs.

Over the course of three days, a total of 14 teams will have the chance to showcase systems of their own design and compete to achieve the highest performance across a series of standard HPC benchmarks and applications.

The winner will be announced on Wednesday, June 19, at 5:15 p.m. in Panorama 2.

Keep up to date on all things HPC and AI by following our social handles @NVIDIAEU and #ISC19.

 

 

Image courtesy of million-memories.com

The post Sea of Green: NVIDIA Floods ISC with AI and HPC appeared first on The Official NVIDIA Blog.

Amazon SageMaker Neo Enables Pioneer’s Machine Learning in Cars

Written on . Posted in Amazon.

Pioneer Corp is a Japanese multinational corporation specializing in digital entertainment products. Pioneer wanted to help their customers check road and traffic conditions through in-car navigation systems. They developed a real-time, image-sharing service to help drivers navigate. The solution analyzes photos, diverts traffic, and sends alerts based on the observed conditions.  Because the pictures are of public roadways, they also had to ensure privacy by blurring out faces and license plate numbers.

Pioneer built their image-sharing service using Amazon SageMaker Neo. Amazon SageMaker is a fully-managed service that provides the ability for developers to build, train, and deploy machine learning models at much less effort and lower cost. Amazon SageMaker Neo is a service that allows developers to train machine learning models once and run them anywhere in the cloud and at the edge. Amazon SageMaker Neo optimizes models to run up to twice as fast, with less than a tenth of the memory footprint, with no loss in accuracy.

You start with an ML model built using MXNet, TensorFlow, PyTorch, or XGBoost and trained using Amazon SageMaker. Then, choose your target hardware platform such as M4/M5/C4/C5 instances or edge devices. With a single click, Amazon SageMaker Neo compiles the trained model into an executable.

The compiler uses a neural network to discover and apply all of the specific performance optimizations to make your model run most efficiently on the target hardware platform. You can deploy the model to start making predictions in the cloud or at the edge.

At launch, Amazon SageMaker Neo was available in four AWS Regions: US East (N. Virginia), US West (Oregon), EU (Ireland), Asia Pacific (Seoul). As of May 2019, SageMaker Neo is now available in Asia Pacific (Tokyo), Japan.

Pioneer developed a machine learning model for real-time image detection and classification using data from cameras in cars. They detect many different kinds of images, such as license plates, people, street traffic, and road signs. The in-car cameras upload data to the cloud and run inference using Amazon SageMaker Neo. The results are sent back to the cars so drivers can be informed on the road.

Here’s how it works.

“We decided to use Amazon SageMaker, a fully managed service for machine learning,” said Ryunosuke Yamauchi, an AI Engineer at Pioneer. “We needed a fully managed service because we didn’t want to spend time managing GPU instances or integrating different applications. In addition, Amazon SageMaker offers hyperparameter optimization, which eliminates the need for time-consuming, manual hyperparameter tuning. Also, we choose Amazon SageMaker because it supports all leading frameworks such as MXNet GluonCV. That’s our preferred framework because it provides state-of-the-art pre-trained object detection models such as Yolo V3.”

To learn more about Amazon SageMaker Neo, see the Amazon SageMaker Neo webpage.

About the Authors

Satadal Bhattacharjee is Principal Product Manager with AWS AI. He leads the Machine Learning Engine PM team working on projects such as SageMaker Neo, AWS Deep Learning AMIs, and AWS Elastic Inference. For fun outside work, Satadal loves to hike, coach robotics teams, and spend time with his family and friends.

 

 

 

Kimberly Madia is a Principal Product Marketing Manager with AWS Machine Learning. Her goal is to make it easy for customers to build, train, and deploy machine learning models using Amazon SageMaker. For fun outside work, Kimberly likes to cook, read, and run on the San Francisco Bay Trail.

Building enterprise-grade, stable, smart bots using machine learning services from AWS

Written on . Posted in Amazon.

Abbott Laboratories has more data than its field team can decipher while on-site with other clients.  Their solution? Working with Smart Bots to build an enterprise-grade, reliable and stable chatbot called Maya, powered by AWS machine learning services like Amazon Lex, AWS Lambda, Amazon Comprehend, and Amazon SageMaker.

For context, Abbott Laboratories is a multinational healthcare company and a forerunner in India in its deployment of AI.  Maya serves Abbott’s 3000+ person field force in India, providing sales operations support and providing access to contextual information at employees’ fingertips.

The chatbot proves especially helpful while employees are in the field meeting doctors. Maya can handle the nitty-gritty of querying and fetching information from enterprise applications so that employees can focus on higher-order tasks.

Maya is integrated with the customer relationship management (CRM) system at Abbott. For each query, the bot gets authenticated on behalf of the user and retrieves the required information.

Amazon Lex enables the language model

Amazon Lex is core to the Maya solution, having been chosen after long discussions regarding the conversation flows and data access protocol from the backend system.

The team identified intents from the conversation flows. Maya today has more than 50 intents—including a “small talk” intent to make the bot more human-like—and close to 250 slots. Most of the intents revolve around data-related actions (for example, filter, compute, and so on). The small talk intent handles phrases like “thank you for your help.”

Lambda determined the response

All 50 intents are linked to a single Lambda function. The following steps are performed on all the requests that call the function.

  • Validate the slots based on business rules.
  • Call all the subscribed methods related to the newly filled slots.
  • Identify the next state.
  • Construct the response object.

Lambda acted as the right fit to implement the validation and state flow logic described above.

Session attributes handled context

The team used intent chaining to enhance the conversation flow, which they laud because it makes the bot smarter and streamlines bot management. For those less familiar with this concept, intent chaining facilitates shifting between multiple intents without losing the context. In Maya, context is stored as JSON in the session attributes. The Context object is structured as follows:

sessionAttributes: {
  "context": {
    "previous-context": {
        "primary-context": true,
        "intent-name":"intent-A",
        "slots": {
          "slot-name": "slot-value",
          ...
        },
        "context-variable-1": "value",
        "context-variable-2": "value"
    },
    "current-context": {
        "intent-name":"intent-B",
      "context-variable-3": "value",
      "context-variable-4": "value"
    }
  }
}

* Values in session attributes can only be a string, so the Context JSON object has to be stringified and then assigned.

In the above example, the flow was shifted from intent A to intent B (leaving intent A pending fulfillment). After the current intent (intent B) is fulfilled, the dialogue state goes back to intent A, retaining the previous state.

In real-world terms, this example is applicable in the healthcare space when a user wants to toggle between analysis of a large dataset and individual patient health records. For example, users may want to view the analysis for the causes, symptoms, and likelihood of various diseases.

Results and next steps

With the Maya chatbot deployed in the field, about a third of the queries that medical representatives raise are now answered by Maya rather than a human.

In the coming months, the team looks to further the use of the chatbot and also make it smarter. In particular, they’re looking at using Amazon SageMaker Reinforcement Learning with the Gym interface to facilitate ongoing training while engaging users. The thinking is to prompt a user with what it expects is next set of useful interactions, then reward or penalize the bot based on the relevance of its recommendations.

Amazon SageMaker is also core to a mother-bot architectural approach that is currently being tested. This mother bot is effectively the coordination point that can query the correct child bot to get an answer to the user. This ensemble of bots is expected to perform even better than a single bot handling all the intents. From a technical perspective, the mother bot is a classification algorithm implemented in Amazon SageMaker—a relatively easy task thanks to the streamlined workflow that Amazon SageMaker enables.

About the Author

Marisa Messina is on the AWS AI marketing team, where her job includes identifying the most innovative AWS-using customers and showcasing their inspiring stories. Prior to AWS, she worked on consumer-facing hardware and then university-facing cloud offerings at Microsoft. Outside of work, she enjoys exploring the Pacific Northwest hiking trails, cooking without recipes, and dancing in the rain.

 

 

 

Google at ICML 2019

Written on . Posted in Google.

Posted by Andrew Helton, Editor, Google AI Communications

Machine learning is a key strategic focus at Google, with highly active groups pursuing research in virtually all aspects of the field, including deep learning and more classical algorithms, exploring theory as well as application. We utilize scalable tools and architectures to build machine learning systems that enable us to solve deep scientific and engineering challenges in areas of language, speech, translation, music, visual processing and more.

As a leader in machine learning research, Google is proud to be a Sapphire Sponsor of the thirty-sixth International Conference on Machine Learning (ICML 2019), a premier annual event supported by the International Machine Learning Society taking place this week in Long Beach, CA. With nearly 200 Googlers attending the conference to present publications and host workshops, we look forward to our continued collaboration with the larger machine learning research community.

If you’re attending ICML 2019, we hope you’ll visit the Google booth to learn more about the exciting work, creativity and fun that goes into solving some of the field’s most interesting challenges, with researchers on hand to talk about Google Research Football Environment, AdaNet, Robotics at Google and much more. You can learn more about the Google research being presented at ICML 2019 in the list below (Google affiliations highlighted in blue).

ICML 2019 Committees
Board Members include: Andrew McCallum, Corinna Cortes, Hugo Larochelle, William Cohen (Emeritus)

Senior Area Chairs include: Charles Sutton, Claudio Gentile, Corinna Cortes, Kevin Murphy, Mehryar Mohri, Nati Srebro, Samy Bengio, Surya Ganguli

Area Chairs include: Jacob Abernethy, William Cohen, Dumitru Erhan, Cho-Jui Hsieh, Chelsea Finn, Sergey Levine, Manzil Zaheer, Sergei Vassilvitskii, Boqing Gong, Been Kim, Dale Schuurmans, Danny Tarlow, Dustin Tran, Hanie Sedghi, Honglak Lee, Jasper Snoek, Lihong Li, Minmin Chen, Mohammad Norouzi, Nicolas Le Roux, Phil Long, Sanmi Koyejo, Timnit Gebru, Vitaly Feldman, Satyen Kale, Katherine Heller, Hossein Mobahi, Amir Globerson, Ilya Tolstikhin, Marco Cuturi, Sebastian Nowozin, Amin Karbasi, Ohad Shamir, Graham Taylor

Accepted Publications
Learning to Groove with Inverse Sequence Transformations
Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, David Bamman

Metric-Optimized Example Weights
Sen Zhao, Mahdi Milani Fard, Harikrishna Narasimhan, Maya Gupta

HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving
Kshitij Bansal, Sarah Loos, Markus Rabe, Christian Szegedy, Stewart Wilcox

Learning to Clear the Market
Weiran Shen, Sebastien Lahaie, Renato Paes Leme

Shape Constraints for Set Functions
Andrew Cotter, Maya Gupta, Heinrich Jiang, Erez Louidor, James Muller, Tamann Narayan, Serena Wang, Tao Zhu

Self-Attention Generative Adversarial Networks
Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena

High-Fidelity Image Generation With Fewer Labels
Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, Sylvain Gelly

Learning Optimal Linear Regularizers
Matthew Streeter

DeepMDP: Learning Continuous Latent Space Models for Representation Learning
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G. Bellemare

kernelPSI: a Post-Selection Inference Framework for Nonlinear Variable Selection
Lotfi Slim, Clément Chatelain, Chloe-Agathe Azencott, Jean-Philippe Vert

Learning from a Learner
Alexis Jacq, Matthieu Geist, Ana Paiva, Olivier Pietquin

Rate Distortion For Model Compression:From Theory To Practice
Weihao Gao, Yu-Han Liu, Chong Wang, Sewoong Oh

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Behrooz Ghorbani, Shankar Krishnan, Ying Xiao

Graph Matching Networks for Learning the Similarity of Graph Structured Objects
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, Pushmeet Kohli

Subspace Robust Wasserstein Distances
François-Pierre Paty, Marco Cuturi

Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints
Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, Seungil You

The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
Daniel Park, Jascha Sohl-Dickstein, Quoc Le, Samuel Smith

A Theory of Regularized Markov Decision Processes
Matthieu Geist, Bruno Scherrer, Olivier Pietquin

Area Attention
Yang Li, Łukasz Kaiser, Samy Bengio, Si Si

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Mingxing Tan, Quoc Le

Static Automatic Batching In TensorFlow
Ashish Agarwal

The Evolved Transformer
David So, Quoc Le, Chen Liang

Policy Certificates: Towards Accountable Reinforcement Learning
Christoph Dann, Lihong Li, Wei Wei, Emma Brunskill

Self-similar Epochs: Value in Arrangement
Eliav Buchnik, Edith Cohen, Avinatan Hasidim, Yossi Matias

The Value Function Polytope in Reinforcement Learning
Robert Dadashi, Marc G. Bellemare, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans

Adversarial Examples Are a Natural Consequence of Test Error in Noise
Justin Gilmer, Nicolas Ford, Nicholas Carlini, Ekin Cubuk

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, Sergey Levine

Garbage In, Reward Out: Bootstrapping Exploration in Multi-Armed Bandits
Branislav Kveton, Csaba Szepesvari, Sharan Vaswani, Zheng Wen, Tor Lattimore, Mohammad Ghavamzadeh

Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition
Yao Qin, Nicholas Carlini, Garrison Cottrell, Ian Goodfellow, Colin Raffel

Direct Uncertainty Prediction for Medical Second Opinions
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, Jon Kleinberg

A Large-Scale Study on Regularization and Normalization in GANs
Karol Kurach, Mario Lučić, Xiaohua Zhai, Marcin Michalski, Sylvain Gelly

Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
Shanshan Wu, Alex Dimakis, Sujay Sanghavi, Felix Yu, Daniel Holtmann-Rice, Dmitry Storcheus, Afshin Rostamizadeh, Sanjiv Kumar

NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, Boqing Gong

Distributed Weighted Matching via Randomized Composable Coresets
Sepehr Assadi, Mohammad Hossein Bateni, Vahab Mirrokni

Monge blunts Bayes: Hardness Results for Adversarial Training
Zac Cranko, Aditya Menon, Richard Nock, Cheng Soon Ong, Zhan Shi, Christian Walder

Generalized Majorization-Minimization
Sobhan Naderi Parizi, Kun He, Reza Aghajani, Stan Sclaroff, Pedro Felzenszwalb

NAS-Bench-101: Towards Reproducible Neural Architecture Search
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, Frank Hutter

Variational Russian Roulette for Deep Bayesian Nonparametrics
Kai Xu, Akash Srivastava, Charles Sutton

Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Zhenxun Zhuang, Ashok Cutkosky, Francesco Orabona

Improved Parallel Algorithms for Density-Based Network Clustering
Mohsen Ghaffari, Silvio Lattanzi, Slobodan Mitrović

The Advantages of Multiple Classes for Reducing Overfitting from Test Set Reuse
Vitaly Feldman, Roy Frostig, Moritz Hardt

Submodular Streaming in All Its Glory: Tight Approximation, Minimum Memory and Low Adaptive Complexity
Ehsan Kazemi, Marko Mitrovic, Morteza Zadimoghaddam, Silvio Lattanzi, Amin Karbasi

Hiring Under Uncertainty
Manish Purohit, Sreenivas Gollapudi, Manish Raghavan

A Tree-Based Method for Fast Repeated Sampling of Determinantal Point Processes
Jennifer Gillenwater, Alex Kulesza, Zelda Mariet, Sergei Vassilvtiskii

Statistics and Samples in Distributional Reinforcement Learning
Mark Rowland, Robert Dadashi, Saurabh Kumar, Remi Munos, Marc G. Bellemare, Will Dabney

Provably Efficient Maximum Entropy Exploration
Elad Hazan, Sham Kakade, Karan Singh, Abby Van Soest

Active Learning with Disagreement Graphs
Corinna Cortes, Giulia DeSalvo,, Mehryar Mohri, Ningshan Zhang, Claudio Gentile

MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, Aram Galstyan

Understanding the Impact of Entropy on Policy Optimization
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, Dale Schuurmans

Matrix-Free Preconditioning in Online Learning
Ashok Cutkosky, Tamas Sarlos

State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations
Alex Lamb, Jonathan Binas, Anirudh Goyal, Sandeep Subramanian, Ioannis Mitliagkas, Yoshua Bengio, Michael Mozer

Online Convex Optimization in Adversarial Markov Decision Processes
Aviv Rosenberg, Yishay Mansour

Bounding User Contributions: A Bias-Variance Trade-off in Differential Privacy
Kareem Amin, Alex Kulesza, Andres Munoz Medina, Sergei Vassilvtiskii

Complementary-Label Learning for Arbitrary Losses and Models
Takashi Ishida, Gang Niu, Aditya Menon, Masashi Sugiyama

Learning Latent Dynamics for Planning from Pixels
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson

Unifying Orthogonal Monte Carlo Methods
Krzysztof Choromanski, Mark Rowland, Wenyu Chen, Adrian Weller

Differentially Private Learning of Geometric Concepts
Haim Kaplan, Yishay Mansour, Yossi Matias, Uri Stemmer

Online Learning with Sleeping Experts and Feedback Graphs
Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, Scott Yang

Adaptive Scale-Invariant Online Algorithms for Learning Linear Models
Michal Kempka, Wojciech Kotlowski, Manfred K. Warmuth

TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
Augustus Odena, Catherine Olsson, David Andersen, Ian Goodfellow

Online Control with Adversarial Disturbances
Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, Karan Singh

Adversarial Online Learning with Noise
Alon Resler, Yishay Mansour

Escaping Saddle Points with Adaptive Gradient Methods
Matthew Staib, Sashank Reddi, Satyen Kale, Sanjiv Kumar, Suvrit Sra

Fairness Risk Measures
Robert Williamson, Aditya Menon

DBSCAN++: Towards Fast and Scalable Density Clustering
Jennifer Jang, Heinrich Jiang

Learning Linear-Quadratic Regulators Efficiently with only √T Regret
Alon Cohen, Tomer Koren, Yishay Mansour

Understanding and correcting pathologies in the training of learned optimizers
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, Jascha Sohl-Dickstein

Parameter-Efficient Transfer Learning for NLP
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly

Efficient Full-Matrix Adaptive Regularization
Naman Agarwal, Brian Bullins, Xinyi Chen, Elad Hazan, Karan Singh, Cyril Zhang, Yi Zhang

Efficient On-Device Models Using Neural Projections
Sujith Ravi

Flexibly Fair Representation Learning by Disentanglement
Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, Richard Zemel

Recursive Sketches for Modular Deep Learning
Badih Ghazi, Rina Panigrahy, Joshua Wang

POLITEX: Regret Bounds for Policy Iteration Using Expert Prediction
Yasin Abbasi-Yadkori, Peter L. Bartlett, Kush Bhatia, Nevena Lazić, Csaba Szepesvári, Gellért Weisz

Anytime Online-to-Batch, Optimism and Acceleration
Ashok Cutkosky

Insertion Transformer: Flexible Sequence Generation via Insertion Operations
Mitchell Stern, William Chan, Jamie Kiros, Jakob Uszkoreit

Robust Inference via Generative Classifiers for Handling Noisy Labels
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, Jinwoo Shin

A Better k-means++ Algorithm via Local Search
Silvio Lattanzi, Christian Sohler

Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton

Learning to Generalize from Sparse and Underspecified Rewards
Rishabh Agarwal, Chen Liang, Dale Schuurmans, Mohammad Norouzi

MeanSum: A Neural Model for Unsupervised Multi-Document Abstractive Summarization
Eric Chu, Peter Liu

CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
Tom Kenter, Vincent Wan, Chun-An Chan, Rob Clark, Jakub Vit

Similarity of Neural Network Representations Revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton

Online Algorithms for Rent-Or-Buy with Expert Advice
Sreenivas Gollapudi, Debmalya Panigrahi

Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities
Octavian Ganea, Sylvain Gelly, Gary Becigneul, Aliaksei Severyn

Non-monotone Submodular Maximization with Nearly Optimal Adaptivity and Query Complexity
Matthew Fahrbach, Vahab Mirrokni, Morteza Zadimoghaddam

Agnostic Federated Learning
Mehryar Mohri, Gary Sivek, Ananda Theertha Suresh

Categorical Feature Compression via Submodular Optimization
Mohammad Hossein Bateni, Lin Chen, Hossein Esfandiari, Thomas Fu, Vahab Mirrokni, Afshin Rostamizadeh

Cross-Domain 3D Equivariant Image Embeddings
Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, Ameesh Makadia

Faster Algorithms for Binary Matrix Factorization
Ravi Kumar, Rina Panigrahy, Ali Rahimi, David Woodruff

On Variational Bounds of Mutual Information
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, George Tucker

Guided Evolutionary Strategies: Augmenting Random Search with Surrogate Gradients
Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, Jascha Sohl-Dickstein

Semi-Cyclic Stochastic Gradient Descent
Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, Kunal Talwar

Workshops
1st Workshop on Understanding and Improving Generalization in Deep Learning
Organizers Include: Dilip Krishnan, Hossein Mobahi
Invited Speaker: Chelsea Finn

Climate Change: How Can AI Help?
Invited Speaker: John Platt

Generative Modeling and Model-Based Reasoning for Robotics and AI
Organizers Include: Dumitru Erhan, Sergey Levine, Kimberly Stachenfeld
Invited Speaker: Chelsea Finn

Human In the Loop Learning (HILL)
Organizers Include: Been Kim

ICML 2019 Time Series Workshop
Organizers Include: Vitaly Kuznetsov

Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (ODML-CDNNR)
Organizers Include: Sujith Ravi, Zornitsa Kozareva

Negative Dependence: Theory and Applications in Machine Learning
Organizers Include: Jennifer Gillenwater, Alex Kulesza

Reinforcement Learning for Real Life
Organizers Include: Lihong Li
Invited Speaker: Craig Boutilier

Uncertainty and Robustness in Deep Learning
Organizers Include: Justin Gilmer

Theoretical Physics for Deep Learning
Organizers Include: Jaehoon Lee, Jeffrey Pennington, Yasaman Bahri

Workshop on the Security and Privacy of Machine Learning
Organizers Include: Nicolas Papernot
Invited Speaker: Been Kim

Exploration in Reinforcement Learning Workshop
Organizers Include: Benjamin Eysenbach, Surya Bhupatiraju, Shixiang Gu

ICML Workshop on Imitation, Intent, and Interaction (I3)
Organizers Include: Sergey Levine, Chelsea Finn
Invited Speaker: Pierre Sermanet

Identifying and Understanding Deep Learning Phenomena
Organizers Include: Hanie Sedghi, Samy Bengio, Kenji Hata, Maithra Raghu, Ali Rahimi, Ying Xiao

Workshop on Multi-Task and Lifelong Reinforcement Learning
Organizers Include: Sarath Chandar, Chelsea Finn
Invited Speakers: Karol Hausman, Sergey Levine

Workshop on Self-Supervised Learning
Organizers Include: Pierre Sermanet

Invertible Neural Networks and Normalizing Flows
Organizers Include: Rianne Van den Berg, Danilo J. Rezende
Invited Speakers: Eric Jang, Laurent Dinh

What’s the Difference Between Hardware and Software Accelerated Ray Tracing?

Written on . Posted in NVIDIA.

You don’t need specialized hardware to do ray tracing, but you want it.

Software-based ray tracing, of course, is decades old. And it looks great: movie makers have been using ray tracing for decades now.

But it’s now clear that specialized hardware — like the RT Cores built into NVIDIA’s Turing architecture — makes a huge difference if you’re doing ray tracing in real time. Games require real-time ray tracing.

Once considered the “holy grail” of graphics, real-time ray tracing brings the same techniques long used by movie makers to gamers and creators.

Thanks to a raft of new AAA games developers have introduced this year — and the introduction last year of NVIDIA GeForce RTX GPUs — this once wild idea is mainstream.

Millions are now firing up PCs that benefit from the RT Cores and Tensor Cores built into RTX. And they’re enjoying ray-tracing enhanced experiences many thought would be years, even decades, away.

Real-time ray tracing, however, is possible without dedicated hardware. That’s because — while ray tracing has been around since the 1970s — the real trend is much newer: GPU-accelerated ray tracing with dedicated cores.

The use of GPUs to accelerate ray-tracing algorithms gained fresh momentum last year with the introduction of Microsoft’s DirectX Raytracing (DXR) API. And that’s great news for gamers and creators.

Ray Tracing Isn’t New

So what is ray tracing? Look around you. The objects you’re seeing are illuminated by beams of light. Now follow the path of those beams backwards from your eye to the objects that light interacts with. That’s ray tracing.

It’s a technique first described by IBM’s Arthur Appel, in 1969, in “Some Techniques for Shading Machine Renderings of Solids.” Thanks to pioneers such as Turner Whitted, Lucasfilm’s Robert Cook, Thomas Porter and Loren Carpenter, CalTech’s Jim Kajiya, and a host of others, ray tracing is now the standard in the film and computer graphics industry for creating lifelike lighting and images.

However, until last year, almost all ray tracing was done offline. It’s very compute intensive. Even today, the effects you see at movie theaters require sprawling, CPU-equipped server farms. Gamers want to play interactive, real-time games. They won’t wait minutes or hours per frame.

GPUs, by contrast, can move much faster, thanks to the fact they rely on larger numbers of computing cores to get complex tasks done more quickly. And, traditionally, they’ve used another rendering technique, known as “rasterization,” to display three-dimensional objects on a two-dimensional screen.

With rasterization, objects on the screen are created from a mesh of virtual triangles, or polygons, that create 3D models of objects. In this virtual mesh, the corners of each triangle — known as vertices — intersect with the vertices of other triangles of different sizes and shapes. It’s fast and the results have gotten very good, even if it’s still not always as good as what ray tracing can do.

GPUs Take on Ray Tracing

But what if you used these GPUs — and their parallel processing capabilities — to accelerate ray tracing? This is where GPU-accelerated software ray tracing comes in. NVIDIA OptiX, introduced in 2009, targeted design professionals with GPU-accelerated ray tracing. Over the next decade, OptiX rode the steady advance in speed delivered by successive generations of NVIDIA GPUs.

By 2015, NVIDIA was demonstrating at SIGGRAPH how ray tracing could turn a CAD model into a photorealistic image — indistinguishable from a photograph — in seconds, speeding up the work of architects, product designers and graphic artists.

That approach — GPU-accelerated software ray tracing — was endorsed by Microsoft early last year, with the introduction of DXR, which enables full support of NVIDIA RTX ray-tracing software through Microsoft’s DXR API.

Delivering high-performance, real-time ray tracing required two innovations: dedicated ray-tracing hardware, RT Cores; and Tensor Cores for high-performance AI processing for advanced denoising, anti-aliasing and super resolution.

RT Cores accelerate ray tracing by speeding up the process of finding out where a ray intersects with the 3D geometry of a scene. These specialized cores accelerate a tree-based ray-tracing structure called a bounding volume hierarchy, or BVH, used to calculate where rays and the triangles that comprise a computer-generated image intersect.

Tensor Cores — first unveiled with NVIDIA’s Volta architecture aimed at enterprise and scientific computing in 2018 to accelerate AI algorithms — further accelerate graphically intense workloads. That’s through a special AI technique called NVIDIA DLSS, short for Deep Learning Super Sampling. RTX’s Tensor Cores make this possible.

Turing at Work

You can see how this works by comparing how quickly Turing and our previous generation Pascal architecture render a single frame of Metro Exodus.

Top: One frame of Metro Exodus rendered on Pascal, with the time in the middle spent on ray tracing.

On Turing, you can see several things happening. One is green, that’s our RT cores kicking in. As you can see, the same ray tracing done on Pascal GPU is done in one-fifth of the time on Turing.

Reinventing graphics, NVIDIA and our partners have been driving Turing to market through a stack of products that now range from the highest performance product, at $999, all the way down to an entry gamer, at $149. The RTX products, with RT Cores and Tensor Cores, start at $349.

Broad Support

There’s no question that real-time ray tracing is the next generation of gaming.

Some of the most important ecosystem partners have announced their support and are now opening the floodgates for real-time ray tracing in games.

Inside of Microsoft’s DirectX 12 multimedia programming interfaces is a ray-tracing component they call DirectX Raytracing (DXR). So every PC, if enabled by the GPU, is capable of accelerated ray tracing.

At the Game Developers Conference in March, we turned on DXR-accelerated ray tracing on our Pascal and Turing GTX GPUs.

To be sure, earlier GPU architectures, such as Pascal, were designed to accelerate DirectX 12. So on this hardware, these calculations are performed on the programmable shader cores, a resource shared with many other graphics functions of the GPU.

So while your mileage will vary — since there are many ways ray tracing can be implemented — Turing will consistently perform better when playing games that make use of ray-tracing effects.

And that performance advantage on the most popular games is only going to grow.

EA’s AAA engine Frostbite, supports ray tracing. Unity and Unreal, which together power 90 percent of the world’s games, now support Microsoft’s DirectX ray tracing in the engine.

Collectively, that opens up an easy path for thousands and thousands of game developers to implement ray tracing in their games.

All told, NVIDIA’s engaged somewhere in excess of 100 developers who are working on ray-traced games.

To date we have millions of gamers who are gaming on RTX hardware, GPU-accelerated hardware with RT Cores.

And — thanks to ray tracing — that number is growing every week.

The post What’s the Difference Between Hardware and Software Accelerated Ray Tracing? appeared first on The Official NVIDIA Blog.

Introducing Google Research Football: A Novel Reinforcement Learning Environment

Written on . Posted in Google.

Posted by Karol Kurach, Research Lead and Olivier Bachem, Research Scientist, Google Research, Zürich

The goal of reinforcement learning (RL) is to train smart agents that can interact with their environment and solve complex tasks, with real-world applications towards robotics, self-driving cars, and more. The rapid progress in this field has been fueled by making agents play games such as the iconic Atari console games, the ancient game of Go, or professionally played video games like Dota 2 or Starcraft 2, all of which provide challenging environments where new algorithms and ideas can be quickly tested in a safe and reproducible manner. The game of football is particularly challenging for RL, as it requires a natural balance between short-term control, learned concepts, such as passing, and high level strategy.

Today we are happy to announce the release of the Google Research Football Environment, a novel RL environment where agents aim to master the world’s most popular sport—football. Modeled after popular football video games, the Football Environment provides a physics based 3D football simulation where agents control either one or all football players on their team, learn how to pass between them, and manage to overcome their opponent’s defense in order to score goals. The Football Environment provides several crucial components: a highly-optimized game engine, a demanding set of research problems called Football Benchmarks, as well as the Football Academy, a set of progressively harder RL scenarios. In order to facilitate research, we have released a beta version of the underlying open-source code on Github.

Football Engine
The core of the Football Environment is an advanced football simulation, called Football Engine, which is based on a heavily modified version of Gameplay Football. Based on input actions for the two opposing teams, it simulates a match of football including goals, fouls, corner and penalty kicks, and offsides. The Football Engine is written in highly optimized C++ code, allowing it to be run on off-the-shelf machines, both with GPU and without GPU-based rendering enabled. This allows it to reach a performance of approximately 25 million steps per day on a single hexa-core machine.

The Football Engine is an advanced football simulation that supports all the major football rules such as kickoffs (top left), goals (top right), fouls, cards (bottom left), corner and penalty kicks (bottom right), and offside.

The Football Engine has additional features geared specifically towards RL. First, it allows learning from both different state representations, which contain semantic information such as the player’s locations, as well as learning from raw pixels. Second, to investigate the impact of randomness, it can be run in both a stochastic mode (enabled by default), in which there is randomness in both the environment and opponent AI actions, and in a deterministic mode, where there is no randomness. Third, the Football Engine is out of the box compatible with the widely used OpenAI Gym API. Finally, researchers can get a feeling for the game by playing against each other or their agents, using either keyboards or gamepads.

Football Benchmarks
With the Football Benchmarks, we propose a set of benchmark problems for RL research based on the Football Engine. The goal in these benchmarks is to play a “standard” game of football against a fixed rule-based opponent that was hand-engineered for this purpose. We provide three versions: the Football Easy Benchmark, the Football Medium Benchmark, and the Football Hard Benchmark, which only differ in the strength of the opponent.

As a reference, we provide benchmark results for two state-of-the-art reinforcement learning algorithms: DQN and IMPALA, which both can be run in multiple processes on a single machine or concurrently on many machines. We investigate both the setting where the only rewards provided to the algorithm are the goals scored and the setting where we provide additional rewards for moving the ball closer to the goal.

Our results indicate that the Football Benchmarks are interesting research problems of varying difficulties. In particular, the Football Easy Benchmark appears to be suitable for research on single-machine algorithms while the Football Hard Benchmark proves to be challenging even for massively distributed RL algorithms. Based on the nature of the environment and the difficulty of the benchmarks, we expect them to be useful for investigating current scientific challenges such as sample-efficient RL, sparse rewards, or model based RL.

The average goal difference of agent versus opponent at different difficulty levels for different baselines. The Easy opponent can be beaten by a DQN agent trained for 20 million steps, while the Medium and Hard opponents require a distributed algorithm such as IMPALA that is trained for 200 million steps.

Football Academy & Future Directions
As training agents for the full Football Benchmarks can be challenging, we also provide Football Academy, a diverse set of scenarios of varying difficulty. This allows researchers to get the ball rolling on new research ideas, allows testing of high-level concepts (such as passing), and provides a foundation to investigate curriculum learning research ideas, where agents learn from progressively harder scenarios. Examples of the Football Academy scenarios include settings where agents have to learn how to score against the empty goal, where they have to learn how to quickly pass between players, and where they have to learn how to execute a counter-attack. Using a simple API, researchers can further define their own scenarios and train agents to solve them.

Top: A successful policy that runs towards the goal (as required, since a number of opponents chase our player) and scores against the goal-keeper. Second: A beautiful way to drive and finish a counter-attack. Third: A simple way to solve a 2-vs-1 play. Bottom: The agent scores after a corner kick.

The Football Benchmarks and the Football Academy consider the standard RL setup, in which agents compete against a fixed opponent, i.e., where the opponent can be considered a part of the environment. Yet, in reality, football is a two-player game where two different teams compete and where one has to adapt to the actions and strategy of the opposing team. The Football Engine provides a unique opportunity for research into this setting and, once we complete our on-going effort to implement self-play, even more interesting research settings can be investigated.

Acknowledgments
This project was undertaken together with Anton Raichuk, Piotr Stańczyk, Michał Zając, Lasse Espeholt, Carlos Riquelme, Damien Vincent‎, Marcin Michalski, Olivier Bousquet‎ and Sylvain Gelly at Google Research, Zürich. We also wish to thank Lucas Beyer, Nal Kalchbrenner, Tim Salimans and the rest of the Google Brain team for helpful discussions, comments, technical help and code contributions. Finally, we would like to thank Bastiaan Konings Schuiling, who authored and open-sourced the original version of this game.

AWS DeepRacer League: The June race gets underway, as the first Virtual Circuit champion is crowned!

Written on . Posted in Amazon.

The AWS DeepRacer League is the world’s first global autonomous racing league, open to anyone. Developers of all skill levels can get hands on with machine learning in a fun and exciting way, racing for prizes and glory at 21 events globally and online via the DeepRacer console. The Virtual Circuit launched at the end of April, allowing developers to compete from anywhere in the world via the console – no car or track required – for a chance to top the leaderboard and score points in one of the 6 monthly competitions.

The rubber hits the road for the June race!

On June 3rd the Kumo Torakku challenge opened, and will be open for racing until June 30th, at midnight PST. Inspired by the Suzuka circuit in Japan, this track will help developers of all skill levels put their models to the test and advance their knowledge and practice of machine learning. All you need to do is log into the console, where you will be taken through a few quick and easy steps to get your model up and running and ready to race. With the AWS Free Tier you are covered for up to 10 hours of training (in your first 30 days of usage), so you can enter the AWS DeepRacer League at no cost to you.

Once you have learned the basics you will be able to immerse yourself inside the AWS DeepRacer online simulator and watch your model train, until it is ready for submission to the leaderboard. Will it make it round the hairpin, to get views of Mt Fuji? Will you optimize for speed or direction to get the model through the curves? Can you tune your model to take pole position? Get racing today, and don’t forget, if you compete in multiple online races you will score more points, and increase your chances to be eligible for one of the overall Virtual Circuit prizes!

AWS DeepRacer League is open to all and you don’t need the AWS DeepRacer car or to visit an in-person race for a chance to compete, with the virtual circuit you can participate in the race from the comfort of the console. Start your engines, the June race is on!

Watch a successful full lap of the Kumo Torakku, from the AWS DeepRacer 3D online simulator

The Suzuka circuit and the new Kumo Torakku virtual race track

What’s new in the Kumo Torakku?

Aside from enjoying the scenery, you will now have the ability to train your model at a maximum speed of 8 meters per second. But beware, the Kumo Torakku has tight corners and a car travelling at that speed may not be able to take the turns well. It may take time for your model to converge and training time could increase with more throttle, so you will have to experiment with speed in your reward function to help you to succeed. Get started today for your chance to win your expenses paid ticket and join the best of the best at re:Invent 2019.

Cheers to the London Loop winner!

And if that doesn’t inspire you, here’s a quick spotlight and celebration of the May race winner. After a month long race, the London Loop closed on Friday May 31st and the first champion of the virtual tournament was crowned. Karl, who works for the National Australia Bank (NAB) took home the top prize and will now be heading to re:Invent 2019 to join the race for the Championship Cup. At NAB, teams are encouraged to experiment with new concepts and technologies, and the team there have been on their machine learning journey with DeepRacer since it launched at re:Invent 2018. They have created their own DeepRacer community, hosted their own competition, and even saw a team member take third place at the AWS Summit in Sydney.

Karl was joined on the London Loop podium by his teammate Paul, who came third in the May race. Paul recently posted about their experience with the AWS DeepRacer League and you can check it out here. Also be on the lookout for part two where they will share more tips on how to compete to win. Karl, Paul and the rest of the NAB team made a combined 533 attempts to conquer the London Loop challenge. They worked hard on their models, tuning them over time and ultimately clinching the win, and they even said “the virtual league was much more fun than the real race!”

Congratulations to the team and here’s to more AWS DeepRacer success!

About the Author

Alexandra Bush is a Senior Product Marketing Manager for AWS AI. She is passionate about how technology impacts the world around us and enjoys being able to help make it accessible to all. Out of the office she loves to run, travel and stay active in the outdoors with family and friends.

 

 

 

Turning unstructured text into insights with Bewgle powered by AWS

Written on . Posted in Amazon.

Bewgle is an SAP.iO, Techstars-funded company that uses AWS services to surface insights from user-generated text and audio streams. Bewgle generates insights to help product managers to increase customer satisfaction and engagement with their various products—beauty, electronics, or anything in between.  By listening to the voices of their customers with the help of Bewgle powered by AWS, these product managers are able to drive increased sales for their products.

An average human can read only about 250 words per minute. To synthesize 1000 customer reviews would therefore take upwards of 8 hours. Analyzing the information from all those reviews—plus other text like forum posts and blog posts, as well as unstructured content like survey verbatims and audio streams—quickly becomes untenable.

This is exactly the kind of problem where AI can excel, specifically, the subset of machine learning (ML) called natural language processing (NLP). At the heart of Bewgle’s solution is an AI platform developed completely on AWS that analyzes millions of pieces of content, then extracts key topics and the sentiment behind them. What would otherwise take years can now be done in minutes with Amazon Machine Learning and the AWS tech stack as a whole.

Indeed, the Bewgle solution makes use of a breadth of AWS services. Bewgle’s data processing pipeline relies on AWS Lambda and Amazon DynamoDB, which form the core of the ML tasks involved:

  • Storing data for analysis at scale.
  • Cleaning up data.
  • Firing various processing functions dynamically to generate the analysis.

The team developed an innovative serverless ML workflow to scale the system and orchestrate various workflows in a loosely coupled way. This gave them tremendous agility and flexibility in evaluating and choosing various approaches independently, facilitating speedy innovation.

A typical workflow for Bewgle starts with Amazon SageMaker Ground Truth, which they use to collect and tag data at scale and on demand. The team lauds the high accuracy of the data tagging that Amazon SageMaker Ground Truth delivers. Bewgle co-founder Shantanu Shah explains, “It [Amazon SageMaker Ground Truth] enables efficiency for Bewgle as we no longer have to look for and manage human taggers, and it’s affordable too.”

Once the data tagging is complete, the Bewgle team turns to Amazon SageMaker to reason over it.  They appreciate using the familiar Jupyter Notebook interface to work with the data; they quickly and easily build and test multiple models.  The automatic hyperparameter tuning within Amazon SageMaker greatly speeds and facilitates what would otherwise be a significant effort for the Bewgle team and makes it possible to achieve a high level of accuracy and confidence.

The next step is model deployment, and Amazon SageMaker once again is the solution.  Deploying with Amazon SageMaker is helpful because, in Shah’s words, “Traffic bursts are not an issue as the scalability and redundancy are automatically taken care of.”   He adds, “Overall, [Amazon] SageMaker helps in every step of model building, tuning and serving and saves countless hours of effort for Bewgle.”

This end to end workflow is depicted in the below diagram.

To make the insights available to customers, they built an API using AWS Elastic Beanstalk. The API allows customers to consume the data in any format. A UI layer built on top of the API also allows the customers to view the data as a digest and a dashboard.  With this implementation, listening to user insights at scale becomes easy.  Bewgle users from R&D teams can be smarter in designing new products; product design teams can consider many factors that might otherwise be overlooked; and business development teams can analyze and compare competitor data when determining new features.

Customer support teams are another key user group for Bewgle. Traditional approaches to customer support center mostly or strictly on answering queries related to structured data that they already have (e.g., templatized emails).  Because verbatims (such as comments left by hotel guests) are unstructured data, they cannot contribute to answering customer support queries. Bewgle believes that converting this unstructured text data into structured data is a key to continuously enhancing customer service. Bewgle’s NLP algorithms continuously learn as the data increases, and their output is structured data that is usable by customer service teams. As a tangible example, consider a customer who notes in a feedback form for a product that they could not open the container to access it. The customer service team is able to take that insight and realize that the glue had hardened on a certain batch, making them impossible to open. As such, the company can avoid creating more disgruntled customers (and potentially losing revenue as a result) by removing that batch from the customer-ready pile.

The team is composed of ex-Googlers who founded Bewgle to solve the information overload problem.  The Bewgle crew finds that the AWS AI and ML services enable their workflow to include “less headache” and more impact. The ease of use, documentation, and broad popularity of the AWS tech stack makes it appealing, and the reason for Bewgle’s choice to use AWS as its primary AI/ML platform.

In particular, Shah notes, “Amazon SageMaker allows us to add tremendous flexibility. [Now] we can rapidly iterate on our models as a result and this directly impacts the strength of our company.”

As the awareness of unstructured data analysis, NLP, and AI techniques has grown, Bewgle has seen rapid growth in its business over the last year. Going forward, the team plans to further scale the technology to other verticals and expand to other geographies.

About the Author

Marisa Messina is on the AWS AI marketing team, where her job includes identifying the most innovative AWS-using customers and showcasing their inspiring stories. Prior to AWS, she worked on consumer-facing hardware and then university-facing cloud offerings at Microsoft. Outside of work, she enjoys exploring the Pacific Northwest hiking trails, cooking without recipes, and dancing in the rain.

 

 

 

An Inside Look at Google Earth Timelapse

Written on . Posted in Google.

Posted by Paul Dille, Senior Software Developer, Carnegie Mellon University CREATE Lab, and Chris Herwig, Geo Data Engineer, Google Earth Outreach

Six years ago, we first introduced Google Earth Timelapse, a global, zoomable time-lapse video that lets anyone explore our changing planet’s surface—from the global scale to the local scale. Earth Timelapse consists of 83 million multi-resolution overlapping video tiles, which are made interactively explorable through the open-source Time Machine client software developed at Carnegie Mellon University’s CREATE Lab. At its core, Google Earth Timelapse is an example of how organizing information can make it more accessible and useful, turning petabytes of satellite imagery into an interactive experience that shows the dynamic changes occurring across space and time.

In April, we introduced several updates to Timelapse, including two additional years of imagery to the time-series visualization, which now spans from 1984 to 2018, with visual upgrades that make exploring more accessible and intuitive. We are especially excited that this update includes support for mobile and tablet devices, which are quickly overtaking desktop computers as the dominant source of app traffic.

Building the Global Visualization
Making a planetary-sized time-lapse video required a significant amount of pixel crunching in Earth Engine, Google’s cloud platform for petabyte-scale geospatial analysis. The new release followed a process similar to what we did in 2013, but at a significantly greater scale—turning 15 million satellite images acquired over the last three and a half decades from the USGS/NASA Landsat and European Sentinel programs into 35 cloud-free 4-terapixel images of the planet—one for each year from 1984 to 2018.

At its native resolution, the Timelapse visualization is a 4 terapixel video (that’s four trillion pixels), which would take about 12 days to download on a 95 Mb/s internet connection. Most computers would have difficulty playing a video of this size, let alone with an interactive, zoomable interface. The problem is even more severe for a mobile device.

A solution was pioneered by Google Maps in 2004 with the map pyramiding technique. Before that time, navigating a map required the use of directional arrows to pan and zoom, with each step requiring the page to reload. The map pyramiding technique assembles the full map image displayed on-screen from tens of small 256×256 pixel non-overlapping image tiles in an array, with new tiles fetched as needed at an appropriate resolution as the user pans and zooms across the map.

A traditional Mercator map pyramid contains non-overlapping image tiles.

This works very well for maps made of static images, but less so for pyramids of video tiles, such as those used by Timelapse, since it requires a web browser to keep up to 16 videos in sync while interacting with the visualization. The solution is embodied in CREATE Lab’s open source Time Machine software: create much larger video tiles that can cover the entire screen and only show one whole-screen tile at a time. The tiles create a pyramid, where sibling tiles overlap with their neighbors to provide a seamless transition between tiles while panning and zooming. Though the overlapping tiles require the use of about 16x more videos, this pyramid structure enables the use of Timelapse on mobile devices by minimizing the amount of data required for visualization.

In our newest release, the global video pyramid consists of 83 million videos across 13 zoom levels, which required about 2 million CPU hours distributed across thousands of machines in Google Cloud to generate.

Earth Timelapse uses a pyramid of overlapping video tiles.

Time Travel, Wherever You Are
Prior to April’s update, ~30% of visitors to the Timelapse visualization were on mobile devices and didn’t actually experience the visualization; instead they saw a YouTube playlist of locations in Timelapse. Until recently, the hardware and CPUs for phones and tablets could not decode videos fast enough without significant delays when someone attempted to zoom in or pan across a video, making mobile exploration unpleasant, if not impossible. In addition, in order for the visualization to be smooth as you pan and zoom, each video that is loaded must sync to the previously playing video and begin playing automatically. But, until only recently, mobile browser vendors had disabled video autoplay at the browser level for bandwidth reasons.

Now that mobile browser vendors have re-enabled video autoplay, we are able to take advantage of current mobile hardware and CPU capabilities, while leveraging the pyramid mapping technique’s efficient use of data, to enable Timelapse on mobile.

Redesigning Timelapse for Exploration Across Devices
Timelapse is a tool for exploration, so we designed for immersiveness, devoting as much real estate as possible to the map. On the other hand, it’s not just a map, but a map of videos. So we kept controls visible, like pausing and restarting the timeline or choosing highlights, by leveraging Material Design with simple, clean lines and clear focal areas.

Navigate with Google Maps using the new “Maps Mode” toggle.

To explore, you need to know where you are or where somewhere else is, so the new interface includes a new “Maps Mode” toggle that lets the user navigate with Google Maps. We also built in scalability to the timeline element of the UI, so that new features added in the future, such as lengthening the time-lapse or adding options for different time increments, won’t break the design. The timeline also allows the user to go backwards in time—an interesting way to compare the present with the past.

For desktop browsers supporting WebGL, we also added a new WebGL viewer to the open source project, which loads and synchronizes multiple videos to fill the screen at optimal resolution. The aesthetic improvement of this is nontrivial, with >4x better resolution.

What’s next
We’re excited about the abundance of freely available, openly licensed satellite imagery and remote sensing data available, enabling new visualizations across time, space, and the visual and non-visual spectrum. We’ve found it’s often the data combined with supplemental layers, such as the World Database on Protected Areas (WDPA) boundaries, that can spark new insights. For example, seeing the visual connection between declining home ownership and shifts in the city of Pittsburgh’s racial makeup tells a story about inequality that numbers on a page simply cannot. Visual evidence can transcend language and cultural barriers and, we hope, generate productive conversations about our global challenges.

Acknowledgements
Randy Sargent, Senior Systems Scientist, Carnegie Mellon University CREATE Lab and the Google Earth Engine team