From developer to machine learning developer

The AWS DeepRacer League is the world’s first autonomous racing league open to developers of all skill levels and it kicked off last week in Santa Clara, California. Chris Miller was crowned our first champion of the 2019 season. Chris is the founder of Cloud Brigade, based in Santa Cruz, California, and he came to the AWS Summit specifically to learn more about machine learning.

At AWS, we are committed to putting machine learning in the hands of all developers of all skill levels, making their experiences with machine learning fun and easy. At Santa Clara, our top three finishers all built a model in one of the onsite workshops and had a lot of fun doing it.

Chris Miller achieved a winning lap time of 10.43 seconds, and will now be advancing the finals at re:Invent 2019 where he will race to win the AWS DeepRacer Championship Cup. Before he arrived at the AWS Summit, he had no experience with machine learning.

Chris says, “When I got here today, I had no experience with machine learning, but that’s exactly what I came here to learn and what a great way to learn machine learning.”

Rahul Shah from Fremont, California came in second place. He was pleasantly surprised by how successful his model was and had a lot of fun with AWS DeepRacer. Rahul has been working with machine learning for the past few years, but this was his first time working with reinforcement learning.

“Working on this was easy, and any developer would be able to have success. The DeepRacer event is a really fun and exciting thing to do at the AWS Summit,” Rahul said.

The third-place finisher was Adrian Sarno from San Mateo, California. Adrian is a data scientist and has been actively involved with machine learning for most of his career. Attending the workshop and participating in the league was his first experience with reinforcement learning and he was curious to learn this advanced ML technique. Adrian’s first attempt at building his model was not as successful as he wanted it to be. When he realized what was at stake, he took to his keyboard and retrained his model for 2 hours. Then he returned with a model that scored him a podium finish.

Adrian says, “It’s straightforward to work with the applications that have been put together.”

All of our participants are excited to experiment more and use the coming months to get more advanced models ready to compete at re:Invent 2019. There, they can use their new found skills to help them win the AWS DeepRacer Championship Cup. 

Heading to Paris to reach developers globally

And it doesn’t end there. The AWS DeepRacer League made its first international stop at the AWS Summit in Paris, France yesterday. Paris is fast becoming a hub for learning and research on artificial intelligence. The French government has plans to invest in Paris to help enable the AI ecosystem in France and the rest of Europe. Such an investment can encourage a large community of developers to learn with easy access to the tools they need to become machine learning developers just like Chris, Rahul, and Adrian.

Today, at the AWS Summit in Paris, the AWS DeepRacer League welcomed more developers to learn, build, and train models to compete. The podium was filled with developers who came to the Summit to participate in the league and each of them had spent time on their models at home before arriving. Positions changed throughout the afternoon as they learned more. In a tense final 60 minutes of racing, Arthur Pace from Paris, took home the Paris Summit Champion cup with a lap time of 13.87 seconds. Second place went to “JO” (Wajdi Fathallah), who attended a DeepRacer meet up before the AWS Summit and secured a 15.5 second lap. The third place finisher was Matthieu Rousseau (16.00 seconds). Matthieu worked on his model with fellow engineering student (and Paris Champion) Arthur Pace for the last 2 weeks in order to land on the podium!

Félicitations aux gagnants de l’#AWSDeepRacer de l’#AWSSummit Paris !
@jorjarthur
@WajdiFathallah
@rousseau_matt
pic.twitter.com/PETPJnMwRC

— AWSonAir (@AWSonAir) April 2, 2019

The 2019 developer journey continues

On April 10, the AWS DeepRacer League will be at the AWS Summit in Singapore. The Summit there offers an opportunity to get hands-on with AWS DeepRacer. There will be multiple workshops and hours of live racing. You can follow the action live on at www.deepracerleague.com. Coming soon is the AWS DeepRacer Virtual League. Get ready today by taking the digital training course for reinforcement learning and AWS DeepRacer.

Developers, start your engines! Your journey to becoming a machine learning developer begins with the AWS DeepRacer League.

About the Author

Alexandra Bush is a Senior Product Marketing Manager for AWS AI. She is passionate about how technology impacts the world around us and enjoys being able to help make it accessible to all. Out of the office she loves to run, travel and stay active in the outdoors with family and friends.

Amazon SageMaker Ground Truth helps you quickly build highly accurate training datasets for machine learning (ML). You can use your own workers, a choice of vendor-managed workforces that specialize in data labeling, or a public workforce powered by Amazon Mechanical Turk to provide the human-generated labels. To get high-quality labels, you must provide simple, concise, and clear instructions, especially when using a public workforce. Writing good instructions is the single most important action you can take to improve annotation quality. It’s worth investing the time to do it right.

This blog post shares best practices for creating highly effective instructions for a public workforce. There are two key points: reduce the cognitive load for the workers as much as possible, and experiment early in the process to fine-tune your instructions and save yourself trouble later on. You can experiment by labeling some of your data yourself and by submitting small jobs to the public workforce throughout the process.

The following screenshot shows an example of a Ground Truth bounding box labeling task with good instructions from the worker’s perspective. In this example task, we ask workers to draw boxes around flowers in images taken from the Google Open Images Dataset. The left side of image shows the short instructions that are constantly visible in a sidebar while the worker is annotating. They are clear, to the point, specialized to the task, and focused on example images.

The following figure shows an example of the full instructions that a worker can see by choosing View full instructions in the sidebar. They clarify ambiguities that could confuse the worker. By the end of this post, you’ll be able to create high-quality instructions for your own labeling job.

Our recommended workflow

The quickest way to create good instructions is use the tools provided by Ground Truth to annotate some of your own data. You can then use the results as examples in your instructions. To do this, you should take the following steps:

1. Select a small number of examples from your data.
2. Run a private job on Ground Truth to label your chosen examples.
3. Create the short instructions using your results. Focus on example images and small amounts of text.
4. Create the full instructions to clarify ambiguities in the task.
5. Run a small public job to test the instructions. Iterate on the results until you are satisfied.
6. Consider simplifying your task, and set a reasonable price.

Note: Running the private labeling jobs will cost $0.08 per example. For pricing details, see the Amazon SageMaker Ground Truth pricing page.

After you have produced high-quality instructions, you can send your full labeling job out to the public workforce. Let’s go over each step in the checklist.

Select a small number of examples from your data

Browse your dataset and select examples that capture the variety in your data. Choosing examples from the items you want to label (as opposed to generic examples) ensures the instructions will help annotators understand your specific task.

Here, we select images with different numbers of flowers of various shapes and sizes. The flowers in some of these images are hidden behind others or touch the edge of the frame. Choosing a variety of cases makes it easier to find good examples for creating the instructions. It also gives you insight into the difficulty of the task from the worker’s point of view.

Run a private job on Ground Truth to label your chosen examples

A previous blog post described how to run a labeling job using the AWS Management Console. You should follow the method described there to label the examples you chose from the previous section. You need to add the images you have selected to a manifest file, create a private work team with your own email address, and select one annotator per example. There’s no reason for you to label the same example multiple times.

Running this private job gives you perspective on what you want to accomplish with your labeling job, the difficulty of the task, and the tools the annotators will be using. Make a record of the examples that were difficult or ambiguous as you work. You will need these later to write the full instructions. In addition, you should consider timing yourself to gauge how much to pay the workers for your task.

The left figure shows a preview of the bounding box tool at work. Notice that the instructions on the left side of the image have not yet been created. The right figure demonstrates the results from the private labeling job.

Create the short instructions using your results

After you finish the private labeling job, you can find the results in the Amazon SageMaker console by going to Labeling jobs and selecting the name you gave the job. The annotated examples are at the bottom of the page. For image labeling tasks, the simplest way to extract the results is to zoom in on these annotated images and take screenshots.

Ideally, narrow your results to one or two exemplary “good” instances, then create one or two images with various bad annotations illustrating what you expect to be the most common sources of failure. You can do this by re-running the private labeling job and skipping all the other examples. Alternatively, you can combine examples of good and bad annotations in a single example image to help the workers quickly understand the task. One particularly inventive strategy is to use an animated GIF that alternates between good and bad examples. For the flower labeling instructions, we use the following images for the good and bad examples, respectively:

After you have selected the example instances and extracted the results, use your favorite image editing software (such as Google Drawings, GIMP, Keynote, or PowerPoint) to put the finishing touches on the figures for your instructions. For example, you might consider placing Xs over images representing incorrect annotations.

Upload your images to an Amazon S3 bucket

Upload the images to an Amazon S3 bucket and set the object permissions so that the images are publicly available. If your S3 bucket has the default permissions, you’ll have to first change the public access settings for the bucket to allow the images to be publicly available. We strongly recommend against making the entire bucket publicly accessible. To make it possible for the images to be public, go to the Amazon S3 console, select your bucket, and choose the Permissions tab. You should see something similar to the following image:

Choose Edit, then uncheck the first two boxes. Choose Save.

A confirmation dialog box appears. Type “confirm” in the appropriate field and choose Confirm to update the public access settings.

To finish uploading the image, return to your S3 bucket overview by choosing Overview. Then choose Upload, drag and drop the file into the dialog box, and then choose Upload in the dialog box. Finally, select the image name from the S3 bucket overview and choose Make public to make the image publically accessible from the internet.

If your bucket permissions have been set correctly, a message saying Success appears.

Finally, we recommend returning to the bucket permissons tab and re-checking the first box, Block new public ACLs and uploading public objects. This prevents you from accidentally making a different object public in the future.

Use the instruction-making tool to finish creating the instructions

Finally, go to the instruction-making tool in the job creation section of the Amazon SageMaker console, create your instructions, and link to the images you gathered in your S3 bucket. You can place your images in the short instructions by choosing the image icon in the instructions tool and entering the object URL, which you can find in the S3 bucket overview by selecting the image name.

After you have added the image, you’ll see a thumbnail in the instruction-making tool.

If you instead see a broken image link icon like the one on the right in the preceding figure, double-check that you have correctly set the bucket and object permissions by following the steps in the previous section.

Many workers will only read the short instructions, so make them count. Focus on your example images, with a small amount of explanatory text in simple English. Use short sentences. Remember, the annotators are not always fluent in English, and ambiguous instructions lead to ambiguous results. Your goal is to be as explicit as possible while keeping things simple.

Create the full instructions to clarify ambiguities in the task

After you have finished writing the short instructions, choose Additional instructions in the instruction-making tool to begin working on the full instructions. Here are some points to keep in mind:

• The full instructions should clarify ambiguities in your task. Often, annotators will only consult these if they are confused. Use your experience from the private job to anticipate sources of confusion.
• Try not to repeat the short instructions.
• Catching every edge case at the expense of having pages and pages of instructions is usually a mistake. In our experience, two or three additional good/bad example pairs should suffice, and further instructions yield diminishing returns.

The following figure shows the final instructions for the flower example.

Run a small public labeling job to test the instructions

After you complete the first draft of the instructions, you can create and submit a small public labeling job. Inspect the results, and look for common mistakes that aren’t addressed in the current version of the instructions. Workers often make mistakes that are different from the ones that you anticipate. It’s better to catch these early in the process than to run a large and expensive labeling job twice. You can continue to repeat this process until the results are satisfactory.

Consider simplifying your task and set a reasonable price

If your instructions are still too long, too complex, or are missing difficult examples from your data, think about how to split your task into several simpler ones. You might have noticed this image in our selection of examples:

Asking workers to label images like this for the same price as the other examples is a recipe for failure. In this case, you might first perform an image classification job to estimate the number of flowers in each image. Then, you can go back and subdivide the images with many flowers so no single image is too challenging.

As another example, consider a job that asks workers to label flowers, people, and dogs in each image. In this case you might get better results by launching three jobs, each focused on a single category. You can run these jobs in parallel or one after another and then combine the results.

As the final step in the process of creating the instructions, use your newly gained experience labeling the examples yourself to set a reasonable price for your tasks. The job creation section of the Amazon SageMaker console allows you to choose a payment for each labeled example using a drop-down menu:

You can use your records of the amount of time it took to complete the labeling jobs for the instructions together with the suggestions in the menu to select an appropriate reward.

Conclusion

Instructions specific to your data will always be superior to generic ones. Creating them might be time-consuming, but the workers will appreciate your effort. They want to complete your task as quickly as possible, and making their lives easier will improve your results.

Here are some resources if you would like to learn more about Ground Truth and making instructions for a public workforce:

• Amazon SageMaker Ground Truth – Build Highly Accurate Datasets and Reduce Labeling Costs by up to 70%
• Annotate data for less with Amazon SageMaker Ground Truth and automated data labeling
• Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
• Amazon Mechanical Turk Requester’s Best Practices Guide
• Mechanical Turk 101: How to use MTurk for tagging training data
• AWS Developer Forums: Amazon SageMaker

Disclosure regarding the Open Images Dataset V4

Open Images Dataset V4 is created by Google Inc. In some cases we have modified the images or the accompanying annotations. You can obtain the original images and annotations here. The annotations are licensed by Google Inc. under CC BY 4.0 license. The images are listed as having a CC BY 2.0 license. The following paper describes Open Images V4 in depth: from the data collection and annotation to detailed statistics about the data and evaluation of models trained on it.

A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982, 2018. (link to PDF)

About the Authors

Tristan McKinney is an applied scientist in the Amazon ML Solutions Lab. He recently completed his PhD in theoretical physics at Caltech where he studied effective field theory and its application to high-T_c superconductors. As his father was in the US Army, he lived all over the place when growing up, including Germany and Albania. In his spare time, Tristan loves to ski and play soccer.

Krzysztof Chalupka is an applied scientist in the Amazon ML Solutions Lab. He has a PhD in causal inference and computer vision from Caltech. At Amazon, he figures out ways in which computer vision and deep learning can augment human intelligence. His free time is filled with family. He also loves forests, woodworking, and books (trees in all forms).

 Fedor Zhdanov is a Machine Learning Scientist at Amazon. He works on developing Machine Learning algorithms and tools for our internal and external customers.

One of the problems that business owners commonly face is detecting when something unusual is happening in their business. Detecting unusual user activity or changes in daily traffic patterns are just some of the challenges. With an ever-increasing amount of data and metrics, detecting anomalies with the help of machine learning is a great way to proactively identify problems.

In this blog post we’ll explain how to build a serverless anomaly detection tool using Amazon SageMaker with Java. Amazon SageMaker makes it easy to train and host machine learning models, and the available built-in algorithms solve common business problems. To solve this particular business problem, we’ll use the Random Cut Forest (RCF) anomaly detection algorithm. Amazon Web Services offers a broad set of global cloud-based products to help organizations move faster, lower IT costs, and scale. We’ll demonstrate how these can be used to build a serverless anomaly detection tool. While Python is one of the most popular programming languages for tackling machine learning problems, many users build micro-services and serverless applications using Java and other JVM-based languages. By the end of this blog post you’ll be able to enable machine learning in your Java applications using Amazon SageMaker.

Throughout the blog post we will use Java code snippets to focus on particular aspects of the tool. You can find the code used to build and deploy this solution into your own AWS account here.

Problem overview

In our example, Alice is a Java developer who owns a video streaming platform that runs on top of multiple AWS services and serves thousands of customers. Alice sets up dashboards to track metrics that show how well her platform is performing. One of the most important metrics she looks at is the total number of active users of the platform, as shown in the following diagram.

This metric shows a general daily pattern of usage, but it also changes seasonally. A low number of active users, a high number of active users, and breaks of daily pattern are all considered anomalies. Alice is mostly interested in understanding the root cause for those anomalous datapoints. Currently, she doesn’t rely on automated tools for finding anomalies in the data. Instead, she goes through a manual process and spend a lot of time identifying spikes, dips, and breaks in periodicity. Fixed thresholds or threshold windows don’t work for her due to changing patterns and seasonality. She needs a better solution!

What can we do to make Alice’s life easier?

Solution architecture

To help Alice solve her anomaly detection problem we first need to identify all the building blocks for an anomaly detection tool:

• Amazon SageMaker– We’ll need Amazon SageMaker to easily build a model based on the historical metric data. Then, we’ll use it to find anomalous data points in current data (from the previous week). The Amazon SageMaker Random Cut Forest algorithm learns the trends in your data and after training can identify anomalies. For using your trained model to find anomalies, we can choose between two options: (1) We can host a model on an endpoint and run inference requests against that endpoint using HTTP requests. (2) We can use a batch transform job to bulk transform new metric data. We need to get results once a week, so the batch transform job seems like a better option. Hosting a model and then hitting an endpoint once a week would be a waste of resources.
• Amazon CloudWatch Events – We’ll use Amazon CloudWatch Events to schedule a recurring weekly event that triggers our weekly transformation job. The patterns in the underlying data will change over time, so it’s important to occasionally refresh the model we’re using. We will use another CloudWatch Events rule to run a training job once per month.
• Amazon CloudWatch Metrics– Alice stores all of her metrics in CloudWatch, which we’ll use as our data source. We’ll also publish our anomalous metric scores to CloudWatch from the batch transform job so Alice can easily view when anomalies occur.
• Amazon S3 –Amazon SageMaker uses Amazon S3 as an input data source for training and batch transform jobs. After we retrieve and preprocess CloudWatch data we will store it in S3 for our Amazon SageMaker jobs.
• AWS Step Functions– Getting data from CloudWatch, uploading it to S3, starting the training and batch transform jobs, and publishing the results back to CloudWatch are all steps that we need so that our anomaly detection tool works as expected. Instead of writing a new service to orchestrate this workflow, we’ll use serverless technologies to simplify the process, and we’ll automate the process using AWS Step Functions. We’ll use two state machines, one for training and one for batch inference, which will ensure that all of the described steps are being executed in the correct order and that any failures are handled gracefully.
• AWS Lambda– All the previously described actions will be executed as AWS Lambda functions, which will be triggered by the AWS Step Functions state machine. All of our Lambda functions use Java 8 and the AWS SDK. Note:  Some of the Lambda functions could potentially be replaced following recent release of Amazon SageMaker support for Amazon States Language. However, in this blog post we want to focus on the perspective of Java development to provide unified view on the subject.

The following diagram illustrates our architecture:

Training job state machine

The following diagram illustrates the training state machine:

1. The first Lambda function (“Store CloudWatch Metric Data in S3”) gets one-month worth of metric data from CloudWatch with a resolution of 5 minutes. The Lambda function creates a CSV file containing the timestamp and a value for each of the 5-minute data points, and uploads the file to the S3 bucket.
2. The second Lambda function (“Start SageMaker Training Job”) uses the S3 dataset created in the previous step to start an Amazon SageMaker training job. The creation of the job is executed in asynchronous fashion and the execution of the state machine continues.

   public class StartTrainingJobHandler {
   
       private static final String TRAINING_JOB_STATUS = "InProgress";
   
       private final AmazonSageMaker sagemaker;
   
       public StartTrainingJobHandler() {
           sagemaker = AmazonSageMakerClientBuilder.standard().build();
       }
   
       public StartTrainingJobOutput handleRequest(StartTrainingJobInput input, Context context) {
           StartTrainingJobConfig config = new StartTrainingJobConfig(
               input.getTimestamp(), input.getBucket(), input.getValuesKey());
           
           CreateTrainingJobRequest request = config.getTrainingJobRequest();
           sagemaker.createTrainingJob(request);
           
           return new StartTrainingJobOutput(
               input.getTimestamp(), request.getTrainingJobName(),
               TRAINING_JOB_STATUS, config.getModelOutputPath());
       }
   }

3. Wait until the Amazon SageMaker training job is finished. If the job failed, we report the job failure and finish the execution. If the job has completed successfully we move to the next state.

   public class CheckTrainingJobStatusHandler {
   
       private final AmazonSageMaker sagemaker;
   
       public CheckTrainingJobStatusHandler() {
           sagemaker = AmazonSageMakerClientBuilder.standard().build();
       }
   
       public StartTrainingJobOutput handleRequest(StartTrainingJobOutput input, Context context) {
           DescribeTrainingJobRequest request = new DescribeTrainingJobRequest()
               .withTrainingJobName(input.getTrainingJobName());
   
           DescribeTrainingJobResult result =        sagemaker.describeTrainingJob(request);
   
           input.setTrainingJobStatus(result.getTrainingJobStatus());
           return input;
       }
   }

4. The final Lambda function (“Create SageMaker Model”) creates an Amazon SageMaker model based on model output created in training job.

   public class CreateModelHandler {
   
       private final AmazonSageMaker sagemaker;
   
       public CreateModelHandler() {
           sagemaker = AmazonSageMakerClientBuilder.standard().build();
       }
   
       public CreateModelOutput handleRequest(CreateModelInput input, Context context) {
           ContainerDefinition containerDefinition = new ContainerDefinition()
               .withImage(RandomCutForestConfig.getAlgorithmImage())
               .withModelDataUrl(input.getModelOutputPath());
   
           CreateModelRequest request = new CreateModelRequest()
               .withExecutionRoleArn(Env.getSagemakerRoleArn())
               .withModelName(RandomCutForestConfig.ALGORITHM_NAME + "-" + input.getTimestamp())
               .withPrimaryContainer(containerDefinition);
   
           sagemaker.createModel(request);
   
           return new CreateModelOutput(request.getModelName());
       }
   }

Transform job state machine

The following diagram illustrates the transform job state machine:

The following steps are executed as part of transform job state machine:

1. We reuse same Lambda function as in the training step (“Store CloudWatch Metric Data in S3”), but we configure it to get only one week of data from CloudWatch.
2. The second Lambda function (“Start SageMaker Transform Job”) finds the models we have trained (created by training state machine), picks the latest one, and asynchronously starts the Amazon SageMaker batch transform job.

   public class StartTransformJobHandler {
   
       private static final String TRANSFORM_JOB_STATUS = "InProgress";
   
       private static final int LIST_MODELS_MAX_RESULTS = 1;
       private static final int LATEST_MODEL_INDEX = 0;
   
       private final AmazonSageMaker sagemaker;
   
       public StartTransformJobHandler() {
           sagemaker = AmazonSageMakerClientBuilder.standard().build();
       }
   
       public StartTransformJobOutput handleRequest(StartTransformJobInput input, Context context) {
           String modelName = getLatestModelName();
           return createSageMakerTransformJob(input, modelName);
       }
   
   
       private String getLatestModelName() {
           ListModelsRequest request = new ListModelsRequest()
                   .withNameContains(ALGORITHM_NAME)
                   .withMaxResults(LIST_MODELS_MAX_RESULTS)
                   .withSortBy(ModelSortKey.CreationTime)
                   .withSortOrder(OrderKey.Descending);
   
           ListModelsResult result = sagemaker.listModels(request);
           ModelSummary modelSummary = result.getModels().get(LATEST_MODEL_INDEX);
   
           return modelSummary.getModelName();
       }
   
           private StartTransformJobOutput createSageMakerTransformJob(StartTransformJobInput input, String modelName) {
           StartTransformJobConfig config = new StartTransformJobConfig(
               input.getTimestamp(), input.getBucket(), input.getValuesKey(), input.getValuesFile(), modelName);
           CreateTransformJobRequest request = config.getTransformJobRequest();
           
           sagemaker.createTransformJob(request);
           return new StartTransformJobOutput(input.getBucket(), input.getTimestamp(),
               input.getTimestampsKey(), config.getAnomalyScoresKey(),
               request.getTransformJobName(), TRANSFORM_JOB_STATUS);
       }
   }

3. Wait until the batch transform job finishes successfully.

   public class CheckTransformJobStatusHandler {
   
       private final AmazonSageMaker sagemaker;
   
       public CheckTransformJobStatusHandler() {
           sagemaker = AmazonSageMakerClientBuilder.standard().build();
       }
   
       public StartTransformJobOutput handleRequest(StartTransformJobOutput input, Context context) {
           DescribeTransformJobRequest request = new DescribeTransformJobRequest()
               .withTransformJobName(input.getTransformJobName());
   
           DescribeTransformJobResult result = sagemaker.describeTransformJob(request);
   
           input.setTransformJobStatus(result.getTransformJobStatus());
           return input;
       }
   }

4. The final Lambda function (“Publish Anomaly Score Metric to CloudWatch”) gets output scores from the batch transform job. It uses a simple, standard technique for classifying anomalies in which all anomaly scores outside three standard deviations from the mean score are considered anomalous. Finally, all the data points that have been labeled as anomalous are published to CloudWatch with a value of 1, and all the data points that haven’t been marked as anomalous are published with a value of 0. To know for which timestamp to publish the anomalous score metric, we use the input dataset.

   public class AnomalousDataUploadHandler {
   
       private final AmazonCloudWatch cloudWatch;
       private final S3FileManager s3FileManager;
   
       public AnomalousDataUploadHandler() {
           cloudWatch = AmazonCloudWatchClientBuilder.standard().build();
           s3FileManager = new S3FileManager();
       }
   
       public AnomalousDataUploadOutput handleRequest(AnomalousDataUploadInput input, Context context) throws IOException {
           List<Double> anomalyScores = getAnomalyScores(input.getBucket(), input.getAnomalyScoresKey());
   
           List<Integer> anomalyIndices = findAnomalousIndices(anomalyScores);
   
           List<Long> timestamps = getTimestamps(input.getBucket(), input.getTimestampsKey());
   
           return uploadAnomalousDataToCloudWatch(timestamps, anomalyIndices, anomalyScores.size());
       }
   
       private List<Integer> findAnomalousIndices(List<Double> anomalyScores) {
           double mean = getMean(anomalyScores);
           
           double std = getStd(anomalyScores, mean);
                   double scoreCutoff = mean + 2 * std;
           
           List<Integer> anomalousIndices = getAnomalousIndices(anomalyScores, scoreCutoff);
           
           return anomalousIndices;
       }
   
   	private List<Integer> getAnomalousIndices(List<Double> anomalyScores, double scoreCutoff) {
       	  return IntStream.range(0, anomalyScores.size())
           	      .filter(i -> anomalyScores.get(i) > scoreCutoff)
           	      .boxed().collect(Collectors.toList());
       }
   
   }

After both state machines have run, a new metric is available in the Amazon CloudWatch console. We can graph this new metric over the original metric to understand when anomalies happen. Now Alice can use the new metric to zoom in on specific points of interest in her original metric, and navigate to the Amazon CloudWatch Logs console for those data points.

Since Alice is storing anomalies in CloudWatch, she can use all of the rich alerting and monitoring functionality that is available so she can be notified automatically when something strange happens. Similarly, because she is using Amazon SageMaker s she can take the model and use it for online inference in the future if she wants to (for example, she can evaluate anomalies in near real time by making HTTP calls to a hosted endpoint).

Conclusion

In this blog post we showed you how to build an automated anomaly detection tool using Amazon SageMaker. We explained what services help us remove the undifferentiated heavy lifting to build the tool and how they all fit together to form a meaningful workflow. We also showcased one of the latest Amazon SageMaker releases, batch transform jobs, which is ideal for use cases that don’t require hosting a model for near real-time inference. All the Lambda functions were written using Java 8. It is our hope that this blog post, in combination with code examples, will help Java developers integrate Amazon SageMaker into their services and applications.

About the authors

Luka Krajcar is a Software Development Engineer on the AWS AI Labs team. He received his M.S. in Computer Science at the Faculty of Electrical Engineering and Computing at the University of Zagreb. Outside of work, Luka enjoys reading fiction, running, and video gaming.

Julio Delgado Mangas is a Software Development Engineer on the AWS AI Labs team. He has contributed to AWS services like Amazon CloudWatch and the Amazon QuickSight SPICE engine. Before joining Amazon, he was a research engineer on the Human Brain Project.

Laurence Rouesnel is the Algorithms & Platforms Group Manager in Amazon AI Labs. He leads a team of engineers and scientists working on deep learning and machine learning research and products. In his spare time, he is an avid traveler, and loves the outdoors whether it’s hiking, skiing, or windsurfing.

Chris Swierczewski is an Applied Scientist on the AWS AI Labs team, where he has contributed to the Amazon SageMaker Latent Dirichlet Allocation and the Amazon SageMaker Random Cut Forest algorithms. Before Amazon, Chris was a Ph.D. student in Applied Mathematics at the University of Washington. He likes to go hiking, backpacking, and camping with his wife and their dog, River.

Madhav Jha is an Applied Scientist on the AWS AI Labs team where he uses his background in sublinear algorithms to develop scalable machine learning algorithms. He is a theoretical computer scientist who enjoys coding. He is always up for coffee conversations on startups and technology.