Amazon Elastic Inference (EI) now supports the latest version of TensorFlow­–1.12. It provides EIPredictor, a new easy-to-use Python API function for deploying TensorFlow models using EI accelerators. You can now use this new Python API function within your inference scripts as an alternative to using TensorFlow Serving when running TensorFlow models with EI. EIPredictor allows for easy experimentation and lets you compare performance with and without EI. This blog post shows you how to use EIPredictor to deploy your models on EI.

Let me start with some background. Amazon Elastic Inference is a new capability we launched at re:Invent 2018. EI provides a new, significantly more cost-effective way to apply acceleration to your deep learning inference workloads than using standalone GPU instances. EI lets you attach accelerators to any Amazon SageMaker or Amazon EC2 instance type and provides you the low latency, high throughput benefits of GPU acceleration at a much lower cost (up to 75%). You can use EI to deploy TensorFlow, Apache MXNet, and ONNX models for inference.

Using TensorFlow Serving to run models on EI

At the launch of Amazon EI we introduced EI-enabled TensorFlow Serving, which provides an easy way to run your TensorFlow models with EI accelerators without having to make any code changes. Just start a model server with EI-enabled TensorFlow Serving with your trained TensorFlow SavedModel, and make calls to it. EI-enabled TensorFlow Serving uses the same API as normal TensorFlow Serving. The only difference is that the entry point is a different binary named AmazonEI_TensorFlow_Serving_v1.12_v1. Here is an example command that you can use to launch the server:

$ AmazonEI_TensorFlow_Serving_v1.12_v1 --model_name=ssdresnet --model_base_path=/tmp/ssd_resnet50_v1_coco --port=9000

You can find EI-enabled TensorFlow Serving in the AWS Deep Learning AMIs (here’s a tutorial), or you can download the package from this Amazon S3 bucket so you can build it into your own custom Amazon Machine Image (AMI) or Docker container. EI-enabled TensorFlow Serving extends TensorFlow’s high performance model serving system to work seamlessly with EI. It automates accelerator discovery, secures your inference requests over the network with TLS encryption, and restricts access with AWS Identity and Access Management (IAM) policies.

Using EIPredictor to run models on EI

EIPredictor is a simple Python function for performing inference on a pretrained model. It is a new API function available within EI-enabled TensorFlow.  It’s also available in the Deep Learning AMI and for download using Amazon S3. You can use EIPredictor in the following ways:

• You can use EIPredictor with a saved model or a frozen graph. It’s similar to TF predictor. Please see EI’s documentation for using EIPredictor with these model formats.
• You can disable usage of EI by using the use_ei flag which is defaulted to True. This is useful to see how your model performs with and without EI acceleration.
• EIPredictor can also be created from a TensorFlow Estimator. Given a trained Estimator, you first export a SavedModel. Refer to the SavedModel documentationfor more details. Example usage:

  saved_model_dir = estimator.export_savedmodel(my_export_dir, serving_input_fn)
  ei_predictor = EIPredictor(export_dir=saved_model_dir)
  //Once the EIPredictor is created, inference is done using the following:
  output_dict = ei_predictor(feed_dict)

The following code sample shows the available parameters for this function:

ei_predictor = EIPredictor(model_dir,
           signature_def_key=None,
           signature_def=None,
           input_names=None,
           output_names=None,
           tags=None,
           graph=None,
           config=None,
           use_ei=True)

output_dict = ei_predictor(feed_dict)

Example for running a model with EI Predictor

Here’s an example you can try for serving a ResNet using a Single Shot Detector (SSD) model using EI Predictor. This example assumes that you’ve launched an EC2 instance with an EI accelerator. We’re going to use the latest Deep Learning AMI here for this example.

1. The first step is to activate the TensorFlow Elastic Inference Note that this is specific to the Deep Learning AMI. You don’t need this step if you built the EI-enabled TensorFlow library with your own custom AMI. You can choose between the Python 2 and Python 3 TensorFlow EI environments. I’ll use Python 2 for this example:

   $ source activate amazonei_tensorflow_p27

2. Download the ResNet SSD model example from Amazon S3.

   $ curl -O https://s3-us-west-2.amazonaws.com/aws-tf-serving-ei-example/ssd_resnet.zip

3. Unzip the model. Again, you may skip this step if you already have the model.

   $ unzip ssd_resnet.zip -d /tmp

4. Download a picture of three dogs to your current directory.

   $ curl -O https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/3dogs.jpg

5. Now open a text editor, such as vim, and paste the following inference script. Save the file as ssd_resnet_predictor.py

   from __future__ import absolute_import
   from __future__ import division
   from __future__ import print_function
   
   import os
   import sys
   import numpy as np
   import tensorflow as tf
   import matplotlib.image as mpimg
   import time
   from tensorflow.contrib.ei.python.predictor.ei_predictor import EIPredictor
   
   tf.app.flags.DEFINE_string('image', '', 'path to image in JPEG format')
   FLAGS = tf.app.flags.FLAGS
   if(FLAGS.image == ''):
     print("Supply an Image using '--image [path/to/image]'")
     exit(1)
   coco_classes_txt = "https://raw.githubusercontent.com/amikelive/coco-labels/master/coco-labels-paper.txt"
   local_coco_classes_txt = "/tmp/coco-labels-paper.txt"
   # Downloading coco labels
   os.system("curl -o %s -O %s" % (local_coco_classes_txt, coco_classes_txt))
   # Setting default number of predictions
   NUM_PREDICTIONS = 20
   # Reading coco labels to a list
   with open(local_coco_classes_txt) as f:
     classes = ["No Class"] + [line.strip() for line in f.readlines()]
   
   
   def main(_):
     # Reading the test image given by the user
     img = mpimg.imread(FLAGS.image)
     # Setting batch size to 1
     img = np.expand_dims(img, axis=0)
     # Setting up EIPredictor Input
     ssd_resnet_input = {'inputs': img}
   
     print('Running SSD Resnet on EIPredictor using specified input and outputs')
     # This is the EIPredictor interface, using specified input and outputs
     eia_predictor = EIPredictor(
         # Model directory where the saved model is located
         model_dir='/tmp/ssd_resnet50_v1_coco/1/',
         # Specifying the inputs to the Predictor
         input_names={"inputs": "image_tensor:0"},
         # Specifying the output names to tensor for Predictor
         output_names={"detection_classes": "detection_classes:0", "num_detections": "num_detections:0",
                       "detection_boxes": "detection_boxes:0"},
     )
   
     pred = None
     # Iterating over the predictions. The first inference request can take saveral seconds to complete
     for curpred in range(NUM_PREDICTIONS):
       if(curpred == 0):
         print("The first inference request loads the model into the accelerator and can take several seconds to complete. Please standby!")
       # Start the timer
       start = time.time()
       # This is where the inference actually happens
       pred = eia_predictor(ssd_resnet_input)
       print("Inference %d took %f seconds" % (curpred, time.time()-start))
   
     # Getting the number of objects detected in the input image from the output of the predictor
     num_detections = int(pred["num_detections"])
     print("%d detection[s]" % (num_detections))
     # Getting the class ids from the output
     detection_classes = pred["detection_classes"][0][:num_detections]
     # Mapping the class ids to class names from the coco labels
     print([classes[int(i)] for i in detection_classes])
   
     print('Running SSD Resnet on EIPredictor using default Signature Def')
     # This is the EIPredictor interface using the default Signature Def
     eia_predictor = EIPredictor(
         # Model directory where the saved model is located
         model_dir='/tmp/ssd_resnet50_v1_coco/1/',
     )
   
     # Iterating over the predictions. The first inference request can take saveral seconds to complete
     for curpred in range(NUM_PREDICTIONS):
       if(curpred == 0):
         print("The first inference request loads the model into the accelerator and can take several seconds to complete. Please standby!")
       # Start the timer
       start = time.time()
       # This is where the inference actually happens
       pred = eia_predictor(ssd_resnet_input)
       print("Inference %d took %f seconds" % (curpred, time.time()-start))
   
     # Getting the number of objects detected in the input image from the output of the predictor
     num_detections = int(pred["num_detections"])
     print("%d detection[s]" % (num_detections))
     # Getting the class ids from the output
     detection_classes = pred["detection_classes"][0][:num_detections]
     # Mapping the class ids to class names from the coco labels
     print([classes[int(i)] for i in detection_classes])
   
   
   if __name__ == "__main__":
     tf.app.run()
   

6. Run the inference script.

   $ python ssd_resnet_predictor.py --image 3dogs.jpg

Conclusion

You now have two convenient ways, depending on your preference, to run your TensorFlow models on cost efficient accelerators. Give it a try and let us know what you think at amazon-ei-feedback@amazon.com

You can learn more about Elastic Inference here and see our documentation user guide. For instructions on using the Deep Learning AMI for EI check out the AWS Deep Learning AMI documentation.

About the Author

Dominic Divakaruni is the Product Manager for Amazon Elastic Inference. He builds services that help customers scale production machine learning applications. In this spare time he enjoys drumming with his son and working on cars.