In a previous post, OpenTelemetry: The OTEL Collector, we saw how easy it's to use the OTEL Collector with a .NET application. Today we will see how to deploy those components in an Amazon EKS Cluster. To complete the post, we will require:
- An IAM User with programmatic access.
- Setup the AWS CLI locally.
- An Amazon EKS Cluster available.
Download the code located here. We will need to adjust it to support AWS X-Ray as a backend (here you can find the reason why). Add the following NuGet Package to the project:
Open the Program.cs
file and change the following section:
...
builder.Services.AddOpenTelemetryTracing(builder =>
{
builder
.AddXRayTraceId()
.AddOtlpExporter()
.AddSource("AnimeQuoteApi")
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
.AddHttpClientInstrumentation()
.AddAspNetCoreInstrumentation()
;
});
...
And that's it from a code perspective. Now, let's work on generating the Docker image. Go to the animequoteapi
folder and add a Dockerfile
file with the following content:
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["AnimeQuoteApi.csproj", "."]
RUN dotnet restore "AnimeQuoteApi.csproj"
COPY . .
WORKDIR "/src"
RUN dotnet build "AnimeQuoteApi.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "AnimeQuoteApi.csproj" -c Release -o /app/publish /p:UseAppHost=false
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "AnimeQuoteApi.dll"]
To store the image, we will use Amazon ECR as a container image registry. Go to the AWS Console and create a new repository (Amazon ECR -> Create Repository) named opentelemetry-api
:
To upload our image, run the following commands:
docker build -t [account-id].dkr.ecr.[region].amazonaws.com/opentelemetry-api:1.0 .
aws ecr get-login-password --region [region] | docker login --username AWS --password-stdin [account-id].dkr.ecr.[region].amazonaws.com
docker push [account-id].dkr.ecr.[region].amazonaws.com/opentelemetry-api:1.0
[region]
is the AWS Region for the database cluster. An example of a region is us-east-2.[account-id]
is the AWS account number. An example of an account number is 1234567890.
Go to the AWS Console and create a new AIM Policy (AIM-> Policies-> Create policy) named opentelemetry-policy
with the following json:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"xray:GetSamplingStatisticSummaries",
"logs:CreateLogStream",
"xray:PutTelemetryRecords",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"xray:GetSamplingRules",
"ssm:GetParameters",
"xray:GetSamplingTargets",
"logs:CreateLogGroup",
"logs:PutLogEvents",
"xray:PutTraceSegments"
],
"Resource": "*"
}
]
}
Then, create an AIM Role named opentelemetry-role
as we saw in the post How to assume an AWS IAM Role from an EKS Pod?. The final IAM Role should have a thrust policy like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::[account-id]:oidc-provider/oidc.eks.[region].amazonaws.com/id/[oidc-id]"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.[region].amazonaws.com/id/[oidc-id]:sub": "system:serviceaccount:[k8s-namespace]:*"
}
}
}
]
}
[oidc-id]
is part of the OpenID Connect provider URL in our EKS Cluster.[k8s-namespace]
is the namespace in our EKS Cluster where we will deploy the application.
Create a service-account.yaml
file with the following content:
apiVersion: v1
kind: ServiceAccount
metadata:
name: opentelemetry-sa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::[account-id]:role/opentelemetry-role
The OTEL Collector image that we will use is a distribution supported by AWS. Create a deployment.yaml
file with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: opentelemetry-deployment
labels:
app: api
spec:
replicas: 2
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
serviceAccountName: opentelemetry-sa
containers:
- name: api-container
env:
- name: ASPNETCORE_ENVIRONMENT
value: Development
image: 025381531841.dkr.ecr.us-east-2.amazonaws.com/opentelemetry-api:2.0
ports:
- name: http
containerPort: 80
protocol: TCP
- name: otel-container
image: amazon/aws-otel-collector:latest
env:
- name: AWS_REGION
value: us-east-2
imagePullPolicy: Always
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 250m
memory: 250Mi
And finally, create a service.yaml
file with the following content:
apiVersion: v1
kind: Service
metadata:
name: opentelemetry-service
labels:
app: api
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: api
Time to deploy everything to our EKS Cluster, run the following commands:
kubectl apply -f service-account.yml --namespace=[k8s-namespace]
kubectl apply -f deployment.yml --namespace=[k8s-namespace]
kubectl apply -f service.yml --namespace=[k8s-namespace]
Once it is completed, we can check the URL of our application with:
kubectl get services --namespace=[k8s-namespace]
Look for the EXTERNAL-IP
column, open the swagger page (add /swagger/index.html
add the end of the URL) in a browser and send a few requests to our API:
Go to the AWS Console to see the Service map graph (CloudWatch->X-Ray traces->Service map):
Go to Traces (CloudWatch->X-Ray traces->Traces):
And open a record to see all the details:
Go to Metrics (CloudWatch->Metrics->All metrics), and open the custom namespace AnimeQuoteApi
:
Select one metric:
In case we need to use our own OTEL Collector configuration file, we could create a ConfigMap with the following content:
apiVersion: v1
kind: ConfigMap
metadata:
name: opentelemetry-config
data:
config.yaml: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
batch:
exporters:
awsxray:
awsemf:
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
And attach it to our pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: opentelemetry-deployment
labels:
app: api
spec:
replicas: 2
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
serviceAccountName: opentelemetry-sa
containers:
- name: api-container
env:
- name: ASPNETCORE_ENVIRONMENT
value: Development
image: 025381531841.dkr.ecr.us-east-2.amazonaws.com/opentelemetry-api:2.0
ports:
- name: http
containerPort: 80
protocol: TCP
- name: otel-container
image: amazon/aws-otel-collector:latest
env:
- name: AWS_REGION
value: us-east-2
imagePullPolicy: Always
command:
- "/awscollector"
- "--config=/conf/config.yaml"
volumeMounts:
- name: otel-config-volume
mountPath: /conf/config.yaml
subPath: config.yaml
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 250m
memory: 250Mi
volumes:
- configMap:
name: opentelemetry-config
name: otel-config-volume
You can see all the code and scripts here. Thanks, and happy coding.