Rustem Feyzkhanov · Writing

Making LLMs Scalable: Cloud Inference with AWS Fargate and Copilot

Rustem Feyzkhanov · August 7, 2023 · Read on AWS in Plain English ↗

In the dynamic field of artificial intelligence, Large Language Models (LLMs) are becoming increasingly prevalent. They’re vital for a variety of applications, such as intelligent personal assistants, mainline products, and even supplementary services. AWS, short for Amazon Web Services, offers a multitude of options to host and deploy these LLMs. It could be as straightforward as a single command deployment using SageMaker JumpStart, or slightly more intricate using the Elastic Compute Service and Application Load Balancer.

In this blog post, I aim to guide you through a quick and efficient deployment of these more sophisticated configurations utilizing a powerful tool from AWS, known as AWS Copilot. This is essentially a command-line interface designed specifically for containers, thereby simplifying the deployment and management of containerized applications.

Catch my full talk on this topic with a demo at LinkedIn or Youtube

The code for the demo is available on github.

Important parts of LLM inference

When deciding on LLM inference there are several parameters to keep in mind:

Model
Code
Container
Orchestration
Hardware

Model

Choosing a model is challenging since there are numerous LLMs that are available for usage. HuggingFace keeps an Open LLM dashboard with top-performing models and the LLM arena by LMSYS.ORG is a way to see the responses to the same prompt by different models.

Another thing to decide on is which optimizations to use. This guide by Sergei Savvov covers different optimizations for LLMs and how they work.

For my demo, I will be using distilled LLM by LaMini-LM (GitHub, huggingface, paper). These models were distilled from popular LLMs like T5 and trained on a large-scale dataset of 2.58M instructions.

Code and container

The important part here is to decide on the inference frameworks. There are several guides like this one or this one which compare different frameworks from a speed perspective as well as optimizations and ease of use.

For my demo, I will be using TGI framework by HuggingFace since it has the best support for different models and doesn’t require model conversion.

Orchestration and networking

The important part here is to decide how the cluster, container, and load balancer will be deployed. There are several ways to deploy containerized applications for inference — SageMaker Endpoint which is easy to deploy and manage, ECS+ASG+ALB which provides the most flexibility and AWS Lambda which would be the most scalable solution.

Get Rustem Feyzkhanov’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For my demo, I will be using the ECS+ALB+Fargate option. Usually, this kind of solution is hard to deploy, but with AWS Copilot it’s extremely easy to deploy and manage this kind of web server. Check out my previous blog post which covers Fargate and AWS Copilot in more detail. One thing to keep in mind — Fargate only has CPU options so even though we are not limited by memory (max is 120GB), we would still need to choose LLMs that are optimized for CPU.

Step-by-Step Deployment

1. Clone the Repository

First, clone the GitHub repository to your local machine:

git clone https://github.com/ryfeus/aws-inference-benchmark.git
cd copilot/transformers/aws-copilot-inference-service

This repository contains all the necessary code to deploy the deep learning model for LLM inference.

2. Clone the model from HuggingFace

Next, we need to clone the HuggingFace repo with the model. I chose LaMini-T5 for this example, but other models could be used as well. Keep in mind the CPU limitation though.

git lfs install
git clone https://huggingface.co/MBZUAI/LaMini-T5-223M.git
mv LaMini-T5-223M model

3. Code in the repo

The main files in the code repo are app.py with business logic, Dockerfile with docker image implementation and manifest.yml which is used for configuring cloud infrastructure

app.py — contains code for the transformers inference, healthcheck, and web request management.

from aiohttp import web
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from transformers import pipeline
import torch
import json

checkpoint = "./model/"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
base_model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint,device_map='auto',torch_dtype=torch.float32)
pipe = pipeline('text2text-generation',model=base_model, tokenizer=tokenizer, max_length=512, do_sample=True,
                temperature=0.3, top_p=0.95)

routes = web.RouteTableDef()


@routes.get('/')
async def healthcheck(request):
    return web.Response()


@routes.post('/predict')
async def predict(request):
    try:
        req = await request.json()
        instruction = req.get('instruction', 'What is the capital of Spain?')
        generated_text = pipe(instruction)
        response = ''
        for text in generated_text:
            response += text['generated_text']
        resp = {
            'result': response
        }
        return web.Response(text=json.dumps(resp))
    except Exception as e:
        return web.Response(text=str(e), status=500)


def create_app():
    app = web.Application()
    app.add_routes(routes)
    return app


async def create_gunicorn_app():
    return create_app()


if __name__ == '__main__':
    web.run_app(create_app(), port=8080)

Dockerfile — is used for building docker image which will be running on ECS cluster.

FROM python:3.11
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8080
CMD [ "python", "./app.py" ]

Copilot manifest — configuration file which defines different parameters of the cluster like CPU/Memory allocation, number of VMs, and available paths.

name: demo
type: Load Balanced Web Service

http:
  path: '/'
  healthcheck: '/'

# Configuration for your containers and service.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
  build: copilot/aws-copilot-sample-service/Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 8080

cpu: 1024       # Number of CPU units for the task.
memory: 2048    # Amount of memory in MiB used by the task.
platform: linux/x86_64  # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1       # Number of tasks that should be running in your service.
exec: true     # Enable running commands in your container.
network:
  connect: true # Enable Service Connect for intra-environment traffic between services.

4. Initialize the Environment and Deploy the Application

Next, initialize your AWS Copilot environment:

copilot env init

This command sets up the environment where your service will reside.

Now, you can deploy the application using:

copilot deploy

This command will take care of building the Docker image, pushing it to Amazon ECR, and deploying it to Amazon ECS/Fargate.

5. Test the Endpoint

After successful deployment, you can make a single prediction by making a POST request to your service. Replace <prefix> with the prefix of your endpoint:

curl -X POST -H "Content-Type: application/json" -d '{"instruction":"What are the main tour attractions in Rome?"}' http://<prefix>.us-east-1.elb.amazonaws.com/predict

The response should be:

{"result": "1. Colosseum 2. Pantheon 3. Roman Forum 4. Trevi Fountain 5. Vatican City 6. Roman Forum 7. Roman Forum 8. Roman Forum 9. Castel Sant'Angelo 10. Roman Forum"}

Conclusion

In this blog post, we showed how to deploy an LLM inference endpoint using AWS Copilot on AWS Fargate with the TGI framework. AWS Copilot simplifies the process of deploying your services, and AWS Fargate ensures that they run smoothly in a serverless environment. This stack is both flexible and easy to manage so we hope that this tutorial will help you in your journey of deploying scalable and robust LLMs to the cloud.