Scaling Amazon Elastic Kubernetes Service Workloads with KEDA and Amazon CloudWatch
Table of contents
In our previous post, Horizontal Pod Autoscaling with Kubernetes Event-Driven Autoscaler, we explored how to scale an application based on events (Azure Service Bus). Today, we aim to scale an application running in Amazon EKS based on a metric (requests per second) provided by Amazon CloudWatch.
Pre-requisites
Install Docker Desktop
Enable Kubernetes (the standalone version included in Docker Desktop)
An Amazon EKS Cluster with KEDA installed is required. We will use the option
operatoras theidentityOwnerfor our AWS CloudWatch scaler. Therefore, we must grant the KEDA operator the necessary IAM permissions to access CloudWatch. You can find an example of how to accomplish this here.Docker Desktop Kubernetes context configured to work with the Amazon EKS cluster.
An IAM User with programmatic access.
Install the AWS CLI.
Install Terraform CLI.
Install K6.
The Application
Run the following commands to set up the solution:
dotnet new web -o MyApi
dotnet new sln -n MyApi
dotnet sln add --in-root MyApi
dotnet add MyApi package AWSSDK.CloudWatch
dotnet add MyApi package AWSSDK.Extensions.NETCore.Setup
dotnet add MyApi package AWSSDK.SecurityToken
We will create an EventListener class to subscribe to the Microsoft.AspNetCore.Hostingsource. Then, the requests-per-second metric will be forwarded to a Channel responsible for sending it to AWS CloudWatch. Open the solution and create the RequestPerSecondChannelProducer.cs file with the following content:
using System.Diagnostics.Tracing;
using System.Threading.Channels;
public class RequestPerSecondChannelProducer : EventListener
{
private readonly Channel<string> _channel;
public RequestPerSecondChannelProducer(Channel<string> channel)
{
_channel = channel;
}
protected override void OnEventSourceCreated(EventSource source)
{
if (source.Name.Equals("Microsoft.AspNetCore.Hosting"))
{
EnableEvents(source, EventLevel.Verbose, EventKeywords.All, new Dictionary<string, string?>()
{
["EventCounterIntervalSec"] = "1"
});
}
}
protected override void OnEventWritten(EventWrittenEventArgs eventData)
{
if (eventData?.EventName!=null && eventData.EventName.Equals("EventCounters", StringComparison.InvariantCulture) && eventData.Payload!=null)
{
for (int i = 0; i < eventData.Payload.Count; i++)
{
var eventPayload = eventData.Payload[i] as IDictionary<string, object>;
if (eventPayload != null && eventPayload.Any(item=> item.Value!=null && item.Value.ToString()!.Equals("requests-per-second", StringComparison.InvariantCulture)))
{
var value = eventPayload.First(item => item.Key != null && item.Key.Equals("Increment", StringComparison.InvariantCulture));
if (value.Value != null)
{
_channel.Writer.TryWrite(value.Value.ToString()!);
}
}
}
}
}
}
Create a CloudWatchChannelConsumer.cs file as follows:
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using System.Threading.Channels;
public class CloudWatchChannelConsumer : BackgroundService
{
private readonly Channel<string> _channel;
private readonly IAmazonCloudWatch _amazonCloudWatch;
private readonly string _host;
private readonly ILogger<CloudWatchChannelConsumer> _logger;
public CloudWatchChannelConsumer(Channel<string> channel, IAmazonCloudWatch amazonCloudWatch, ILogger<CloudWatchChannelConsumer> logger)
{
_channel = channel;
_host = Environment.GetEnvironmentVariable("POD_NAME") ?? "local";
_logger = logger;
_amazonCloudWatch = amazonCloudWatch;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!_channel.Reader.Completion.IsCompleted && await _channel.Reader.WaitToReadAsync())
{
if (_channel.Reader.TryRead(out var value))
{
try
{
await _amazonCloudWatch.PutMetricDataAsync(new PutMetricDataRequest
{
Namespace = "MyApiNamespace",
MetricData =
[
new MetricDatum
{
MetricName = "RequestPerSecond",
Value = double.Parse(value),
Unit = StandardUnit.CountSecond,
TimestampUtc = DateTime.UtcNow,
Dimensions = new List<Dimension>
{
new() {
Name = "Host",
Value = _host
}
}
}
]
});
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to send CloudWatch Metric");
}
}
}
}
}
We do not recommend using the AWS SDK for .NET to send metrics to CloudWatch. Instead, the proper approach would be to use the OTEL Collector, as shown here. Open the Program.cs file and update the content as follows:
using Amazon.CloudWatch;
using System.Threading.Channels;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonCloudWatch>();
builder.Services.AddHostedService<CloudWatchChannelConsumer>();
var channel = Channel.CreateUnbounded<string>();
builder.Services.AddSingleton(channel);
var app = builder.Build();
app.MapGet("/", () => "Hello World!");
var client = app.Services.GetRequiredService<IAmazonCloudWatch>();
var listener = new RequestPerSecondChannelProducer(channel);
app.Run();
Terraform
We will use Terraform to create the necessary resources for running our application on the Kubernetes cluster. At the project level, create a main.tf file with the following content:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "5.31.0"
}
}
backend "local" {}
}
provider "aws" {
region = "<MY_REGION>"
profile = "<MY_AWS_PROFILE>"
max_retries = 2
}
locals {
repository_name = "myrepo"
cluster_name = "<MY_K8S_CLUSTER_NAME>"
role_name = "myrole"
namespace = "<MY_K8S_NAMESPASE>"
}
resource "aws_ecr_repository" "repository" {
name = local.repository_name
image_tag_mutability = "MUTABLE"
image_scanning_configuration {
scan_on_push = false
}
}
data "aws_iam_policy" "cloudwatch_policy" {
name = "CloudWatchFullAccess"
}
data "aws_eks_cluster" "cluster" {
name = local.cluster_name
}
module "iam_assumable_role_with_oidc" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "4.14.0"
oidc_subjects_with_wildcards = ["system:serviceaccount:${local.namespace}:*"]
create_role = true
role_name = local.role_name
provider_url = data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer
role_policy_arns = [
data.aws_iam_policy.cloudwatch_policy.arn,
]
number_of_role_policy_arns = 1
}
output "role_arn" {
value = module.iam_assumable_role_with_oidc.iam_role_arn
}
output "repository_url" {
value = aws_ecr_repository.repository.repository_url
}
We are creating an Amazon ECR repository to upload our application's image and an IAM Role for our Pod with sufficient permissions for AWS CloudWatch. Run the following commands to create the resources in AWS:
terraform init
terraform plan -out app.tfplan
terraform apply 'app.tfplan'
Docker Image
Create a Dockerfile with the following content:
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
COPY ["MyApi/MyApi.csproj", "MyApi/"]
RUN dotnet restore "MyApi/MyApi.csproj"
COPY . .
WORKDIR "/MyApi"
RUN dotnet build "MyApi.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "MyApi.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyApi.dll"]
Run the following command to upload the image to the Amazon ECR repository:
aws ecr get-login-password --region <MY_REGION> --profile <MY_AWS_PROFILE> | docker login --username AWS --password-stdin <MY_ACCOUNT_ID>.dkr.ecr.<MY_REGION>.amazonaws.com
docker build -t <MY_ACCOUNT_ID>.dkr.ecr.<MY_REGION>.amazonaws.com/myrepo:1.0 -f .\MyApi\Dockerfile .
docker push <MY_ACCOUNT_ID>.dkr.ecr.<MY_REGION>.amazonaws.com/myrepo:1.0
Kubernetes
To interact with the AWS CloudWatch API, we must assume an AWS IAM Role from our Pod through a Service Account. You can find more information here. Create a serviceaccount.yaml file with the following content:
apiVersion: v1
kind: ServiceAccount
metadata:
name: myapi-sa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<MY_ACCOUNT_ID>:role/myrole
Next, we will use the Service Account mentioned earlier. Create a deployment.yaml file with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapi-deployment
labels:
app: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
serviceAccountName: myapi-sa
containers:
- name: myapi-c
env:
- name: ASPNETCORE_ENVIRONMENT
value: Development
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: <MY_ACCOUNT_ID>.dkr.ecr.<MY_REGION>.amazonaws.com/myrepo:1.0
ports:
- name: http
containerPort: 8080
protocol: TCP
Our application will be accessible through a Service using a Load Balancer as its type. Create a service.yaml file containing the following content:
apiVersion: v1
kind: Service
metadata:
name: myapi-service
labels:
app: api
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: api
Finally, create a Scaled Object referencing the Deployment created earlier. Create a scaledobject.yaml file containing the following content:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: myapi-scaledobject
spec:
scaleTargetRef:
name: myapi-deployment
kind: Deployment
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: aws-cloudwatch
metadata:
namespace: MyApiNamespace
expression: SELECT AVG(RequestPerSecond) FROM MyApiNamespace
targetMetricValue: '2'
awsRegion: "<MY_REGION>"
identityOwner: operator
We use the expression property to specify a query that should remain below the one indicated by the targetMetricValue property. In our case, the query provides the average request-per-second for all the Pods in our Deployment. Execute the following commands to deploy the application to the cluster:
kubectl apply -f serviceaccount.yaml --namespace=<MY_K8S_NAMESPASE>
kubectl apply -f deployment.yaml --namespace=<MY_K8S_NAMESPASE>
kubectl apply -f service.yaml --namespace=<MY_K8S_NAMESPASE>
kubectl apply -f scaledobject.yaml --namespace=<MY_K8S_NAMESPASE>
Run kubectl get services --namespace=<MY_K8S_NAMESPACE> to see the URL assigned to our application. The output will look something like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myapi-service LoadBalancer 10.100.136.188 <MY_URL> 80:31119/TCP 18d
Tests
Create a load.js file with the following content:
import http from 'k6/http';
import { sleep } from 'k6';
export const options = {
vus: 16,
duration: '600s',
};
export default function () {
http.get('<MY_URL>');
sleep(1);
}
Execute the script using the command k6 run load.js. Increasing the load on the application will cause the request-per-second metric in AWS CloudWatch to rise to two or higher, triggering the deployment scaling.
Next, we can observe how the Deployment scales up to create more pods in response to the load on the application. Run kubectl get pods --namespace=<MY_K8S_NAMESPACE>:
NAME READY STATUS RESTARTS AGE
myapi-deployment-7bbdb8f8d9-44vds 1/1 Running 0 6m46s
myapi-deployment-7bbdb8f8d9-48567 1/1 Running 0 39h
myapi-deployment-7bbdb8f8d9-frpcf 1/1 Running 0 6m46s
In conclusion, scaling workloads in Amazon EKS can be effectively achieved using KEDA and Amazon CloudWatch. This approach enables an application to scale based on specific metrics, ensuring optimal resource utilization and improved application performance. You can find the code and scripts here. Thank you, and happy coding.