Over the past while, I’ve been working on a side project to forecast hotel cancellations on a weekly basis (original data and authors available here).

While the original intent of this research was to identify the drivers of such cancellations and predict whether customers would cancel using classification (i.e. cancelling customer = 1, non-cancelling customer = 0), I wanted to investigate whether time series forecasting could be a good addition to this study.

The first step was using pandas for data manipulation, i.e. sorting the cancellations by week and then summing up to get the total number of cancellations every week.

Following this, I decided to use both ARIMA and LSTM to predict future cancellations across the test set. This was done for two separate hotel datasets (H1 and H2).

Interestingly, I found that LSTM performed better on the more volatile dataset (H2), while ARIMA showed more forecast accuracy on the dataset with a smoother trend (H1).

Ultimately, doing this project reinforced to me that machine learning models like LSTM are just like any other model – they are not necessarily suitable for all situations, and one needs to understand the data they are working with before selecting the model.

If you’re interested in the findings, feel free to take a further look. It is a three-part study, but here are the relevant links below:

– LSTM Forecasts

– ARIMA Forecasts (first half of the article covers classification with SVM)

Hope you find this of use, and grateful for any feedback!

Just wanted to raise awareness and get discussion going if anyone wants to meet up at MLconf (next next Friday, November 8th). Talks will cover topics such as: NLP, Voice Agents, ML & Medical Research, ML & Quantum Computing, ML Models, Data Science for Good, etc.

If you’re not going to be in San Francisco then you can also check out past sessions (going back to 2012) here: https://mlconf.com/sessions/.

If you do want to be there I’d suggest going to eventbrite instead of their website since there’s a discount. Below are the speakers and the topic they’ll speak on (if I could find it):

2019 MLconf SF Speakers:

• Franziska Bell, Senior Data Science Manager on the Platform Team, Uber – Opening Remarks
• Anitha Kannan, Founding Member, Curai – AI for healthcare: Scaling Access and Quality of Care for Everyone
• Xavier Amatriain, CTO, Curai – AI for healthcare: Scaling Access and Quality of Care for Everyone
• Mihajlo Grbovic, Principal Machine Learning Scientist, Airbnb
• Josh Wills, Software Engineer, Slack – Data Labeling as Religious Experience
• Ted Willke, Sr. Principal Engineer, Intel
• Jekaterina Novikova, Director of Machine Learning, Winterlight Labs – Machine Learning Methods in Detecting Alzheimer’s Disease from Speech and Language
• Bradley Voytek, Associate Professor, UCSD – The Art of Parameterization
• June Andrews, AI Instruments, Stitch Fix – The Uncanny Valley of ML
• Sneha Rajana, Software Development Engineer, Amazon – Deep Learning Architectures for Semantic Relation Detection Tasks
• Noam Finkelstein, PhD Student, Johns Hopkins University – The Importance of Modeling Data Collection
• Anoop Deoras, Researcher, Netflix – Building an Incrementally Trained, Local Taste Aware, Global Deep Learned Recommender System Model
• Jamila Smith-Loud, User Researcher, Google
• Justin Armstrong, Senior Backend Engineer – Applied ML, Compology – Applying Computer Vision to Reduce Contamination in the Recycling Stream
• Igor Markov, Facebook/ Professor, University of Michigan
• Vinay Prabhu, Chief Scientist, UnifyID Inc – Project GaitNet: Ushering in the ImageNet moment for human Gait kinematics
• Meghanna Ravikumar, Machine Learning Engineer, SigOpt – Optimized Image Classification on the Cheap
• Martin Isaksson, Co-Founder, PerceptiLabs

Sponsors: PerceptiLabs, Oracle, Apple, Proofpoint, HiringSolved, SigOpt, Medium, Walmart Labs, Compology.

Personally, I’m most looking forward to the healthcare applications they’ll go over, but I’m also curious what “Data Labeling as a Religious Experience” means.

Since its formulation by Sir Isaac Newton, the problem of solving the equations of motion for three bodies under their own gravitational force has remained practically unsolved. Currently, the solution for a given initialization can only be found by performing laborious iterative calculations that have unpredictable and potentially infinite computational cost, due to the system’s chaotic nature. We show that an ensemble of solutions obtained using an arbitrarily precise numerical integrator can be used to train a deep artificial neural network (ANN) that, over a bounded time interval, provides accurate solutions at fixed computational cost and up to 100 million times faster than a state-of-the-art solver. Our results provide evidence that, for computationally challenging regions of phase-space, a trained ANN can replace existing numerical solvers, enabling fast and scalable simulations of many-body systems to shed light on outstanding phenomena such as the formation of black-hole binary systems or the origin of the core collapse in dense star clusters.

Paper: arXiv

Technology Review article: A neural net solves the three-body problem 100 million times faster

What are the best labs/PIs for Deep Learning (CV, NLP, Robotics included) right now?

Papers commonly report NLL scores, such as a value of around 3 for PixelCNN. I believe this is bits-per-dimension,

but, is it:

a) an average across all the datapoints in the test set, or
b) a sum across all datapoints, or
c) the best score on an individual datapoint?

Or maybe my question makes not sense.

Explaining furher In the case of PixelCNN, “datapoint” = image, so I believe the NLL of trained model can be evaluated by summing the logs of the conditional probabilities of each pixel (conditioned on the neighborhood in the pixelcnn scheme), plus the marginal probability for the first pixel. This gives the overall LL for a single image from the test set, but what about the other images.

I just finished a personal project whose goal is to predict the world championship ranking from previous scores in the season (for male and female single skaters).

The obvious way to rank skaters is to take their average score of the season and rank them from highest to lowest. However, one potential problem with this approach is that the scores are averaged over different events, and no two events are the same (think different judges, ice conditions, or event altitudes). Therefore, I came up with different ranking models that can somehow tease out the skater effect (how good a skater intrinsically) from the event effect (how does an event affect the score of a skater). The models themselves are essentially just simple linear models, but I’d never thought about using linear regression this way. I’ve also documented how my models perform over the baseline model, which is the average season score model mentioned above.

If you have any feedback or ideas on my project, please don’t hesitate to let me know!

PS. I’m in the process of cleaning up the code that I used for the analysis, and will soon add the link to the Github repo in the write-up. I’m writing a part 2 on even more complicated models to rank skater and will post them here when I’m done writing it.

Abstract: Emotion recognition in conversation (ERC) has received much attention, lately, from researchers due to its potential widespread applications in diverse areas, such as health-care, education, and human resources. In this paper, we present Dialogue Graph Convolutional Network (DialogueGCN), a graph neural network based approach to ERC. We leverage self and inter-speaker dependency of the interlocutors to model conversational context for emotion recognition. Through the graph network, DialogueGCN addresses context propagation issues present in the current RNN-based methods. We empirically show that this method alleviates such issues, while outperforming the current state of the art on a number of benchmark emotion classification datasets.

Paper: https://arxiv.org/abs/1908.11540 (EMNLP 2019)

Blog Post: https://towardsdatascience.com/emotion-recognition-using-graph-convolutional-networks-9f22f04b244e

I am the primary author of the paper. Feel free to ask questions.

I’m relatively new to ML and have been running my own finetuning of GPT2 for a while. I’m not sure if this is the best or most appropriate place to ask this, so please forgive me if not (and feel free to redirect me elsewhere). With many prompts, GPT2 seems to get stuck in an infinite loop repeating the input phrase or part of the input phrase, particularly when I pass it titles. It will also randomly spit out the endtext token, or say things like Join my mailing list! Read the comments blow at the start of an “article”.

​

Am I doing something wrong with finetuning, or is this just a thing that happens and that people discard?

Understanding searches better than ever before

If there’s one thing I’ve learned over the 15 years working on Google Search, it’s that people’s curiosity is endless. We see billions of searches every day, and 15 percent of those queries are ones we haven’t seen before–so we’ve built ways to return results for queries we can’t anticipate.

When people like you or I come to Search, we aren’t always quite sure about the best way to formulate a query. We might not know the right words to use, or how to spell something, because often times, we come to Search looking to learn–we don’t necessarily have the knowledge to begin with.

At its core, Search is about understanding language. It’s our job to figure out what you’re searching for and surface helpful information from the web, no matter how you spell or combine the words in your query. While we’ve continued to improve our language understanding capabilities over the years, we sometimes still don’t quite get it right, particularly with complex or conversational queries. In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question.

With the latest advancements from our research team in the science of language understanding–made possible by machine learning–we’re making a significant improvement to how we understand queries, representing the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search.

Applying BERT models to Search
Last year, we introduced and open-sourced a neural network-based technique for natural language processing (NLP) pre-training called Bidirectional Encoder Representations from Transformers, or as we call it–BERT, for short. This technology enables anyone to train their own state-of-the-art question answering system.

This breakthrough was the result of Google research on transformers: models that process words in relation to all the other words in a sentence, rather than one-by-one in order. BERT models can therefore consider the full context of a word by looking at the words that come before and after it—particularly useful for understanding the intent behind search queries.

But it’s not just advancements in software that can make this possible: we needed new hardware too. Some of the models we can build with BERT are so complex that they push the limits of what we can do using traditional hardware, so for the first time we’re using the latest Cloud TPUs to serve search results and get you more relevant information quickly.

Cracking your queries
So that’s a lot of technical details, but what does it all mean for you? Well, by applying BERT models to both ranking and featured snippets in Search, we’re able to do a much better job helping you find useful information. In fact, when it comes to ranking results, BERT will help Search better understand one in 10 searches in the U.S. in English, and we’ll bring this to more languages and locales over time.

Particularly for longer, more conversational queries, or searches where prepositions like “for” and “to” matter a lot to the meaning, Search will be able to understand the context of the words in your query. You can search in a way that feels natural for you.

To launch these improvements, we did a lot of testing to ensure that the changes actually are more helpful. Here are some of the examples that showed up our evaluation process that demonstrate BERT’s ability to understand the intent behind your search.

Here’s a search for “2019 brazil traveler to usa need a visa.” The word “to” and its relationship to the other words in the query are particularly important to understanding the meaning. It’s about a Brazilian traveling to the U.S., and not the other way around. Previously, our algorithms wouldn’t understand the importance of this connection, and we returned results about U.S. citizens traveling to Brazil. With BERT, Search is able to grasp this nuance and know that the very common word “to” actually matters a lot here, and we can provide a much more relevant result for this query.

Let’s look at another query: “do estheticians stand a lot at work.” Previously, our systems were taking an approach of matching keywords, matching the term “stand-alone” in the result with the word “stand” in the query. But that isn’t the right use of the word “stand” in context. Our BERT models, on the other hand, understand that “stand” is related to the concept of the physical demands of a job, and displays a more useful response.

Here are some other examples where BERT has helped us grasp the subtle nuances of language that computers don’t quite understand the way humans do.

Improving Search in more languages
We’re also applying BERT to make Search better for people across the world. A powerful characteristic of these systems is that they can take learnings from one language and apply them to others. So we can take models that learn from improvements in English (a language where the vast majority of web content exists) and apply them to other languages. This helps us better return relevant results in the many languages that Search is offered in.

For featured snippets, we’re using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi and Portuguese.

Search is not a solved problem
No matter what you’re looking for, or what language you speak, we hope you’re able to let go of some of your keyword-ese and search in a way that feels natural for you. But you’ll still stump Google from time to time. Even with BERT, we don’t always get it right. If you search for “what state is south of Nebraska,” BERT’s best guess is a community called “South Nebraska.” (If you’ve got a feeling it’s not in Kansas, you’re right.)

Language understanding remains an ongoing challenge, and it keeps us motivated to continue to improve Search. We’re always getting better and working to find the meaning in– and most helpful information for– every query you send our way.

Source