In the post Observability with OpenTelemetry in .NET, we saw how OpenTelemetry tries to provide a standardized way to instrument our applications. OpenTelemetry achieves that by providing us with a specification that can be implemented in multiple languages. But there is another component, the OTEL Collector:
The OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. This works with improved scalability and supports open-source observability data formats (e.g. Jaeger, Prometheus, Fluent Bit, etc.) sending to one or more open-source or commercial back-ends. The local Collector agent is the default location to which instrumentation libraries export their telemetry data.
In short, instead of sending our telemetry data to a specific backend, we could send it to OTEL Collector and from there to any backend. Additionally, the collector can take care of additional handling like retries, batching, encryption, or even sensitive data filtering. The OTEL Collector consists of three components that access telemetry data: receivers, processors, and exporters, which are used to build pipelines.
Receivers
A receiver is how data gets into the OpenTelemetry Collector. Generally, a receiver accepts data in a specified format, translates it into the internal format and passes it to processors and exporters defined in the applicable pipelines.
Here, you can find all the available receivers.
Exporters
An exporter is how data gets sent to different systems/back-ends. Generally, an exporter translates the internal format into another defined format.
Here, you can find all the available exporters.
Processors
Processors are used at various stages of a pipeline. Generally, a processor pre-processes data before it is exported (e.g. modify attributes or sample) or helps ensure that data makes it through a pipeline successfully (e.g. batch/retry).
Pipelines
A pipeline consists of a set of receivers, processors and exporters. Each receiver/processor/exporter must be defined in the configuration outside of the service section to be included in a pipeline. Pipelines can be of the following types:
- Traces: collects and processes trace data.
- Metrics: collects and processes metric data.
- Logs: collects and processes log data.
Let's clone or download the following code. Open the solution and add the following NuGet packages:
Edit the Program.cs file as follow:
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;
using AnimeQuoteApi;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.AddHttpClient<QuoteClient>(client=>
{
client.BaseAddress = new Uri("https://animechan.vercel.app");
});
builder.Services.AddOpenTelemetryTracing(builder =>
{
builder
.AddOtlpExporter()
.AddSource("AnimeQuoteApi")
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
.AddHttpClientInstrumentation()
.AddAspNetCoreInstrumentation()
;
});
builder.Services.AddOpenTelemetryMetrics(builder =>
{
builder
.AddRuntimeInstrumentation()
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
.AddOtlpExporter();
});
builder.Logging.ClearProviders();
builder.Logging.AddOpenTelemetry(builder =>
{
builder
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
.AddOtlpExporter();
});
var app = builder.Build();
if (app.Environment.IsDevelopment())
{
app.UseSwagger();
app.UseSwaggerUI();
}
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();
Run the following command to start Jeager:
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -e COLLECTOR_OTLP_ENABLED=false -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14250:14250 -p 14268:14268 -p 14269:14269 -p 9411:9411 jaegertracing/all-in-one:1.36
Edit the prometheus.yml file as follows:
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: "AnimeQuoteApi"
scrape_interval: 5s
static_configs:
- targets: ["host.docker.internal:8889"]
Run the following command to start Prometheus:
docker run -d --name prometheus -p 9090:9090 -v ${pwd}/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
Add an otel-collector-config.yaml file with the following content:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 15s
send_batch_size: 10
exporters:
jaeger:
endpoint: "host.docker.internal:14250"
tls:
insecure: true
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "default"
logging:
loglevel: debug
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [logging]
Add a docker-compose.yml file:
version: "3.4"
services:
aws-ot-collector:
image: otel/opentelemetry-collector-contrib:0.56.0
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "8888:8888"
- "8889:8889"
- "13133:13133"
- "4317:4317"
- "4318:4318"
And, run the following command to start the OTEL Collector:
docker-compose up
Run the application and send a couple of requests to start to see traces in Jeager (http://localhost:16686/search):
Metrics in Prometheus (http://localhost:9090/):
And logs in the console:
You can see all the code here. Thanks, and happy coding.
