OpenTelemetry: The OTEL Collector

OpenTelemetry: The OTEL Collector

In the post Observability with OpenTelemetry in .NET, we saw how OpenTelemetry tries to provide a standardized way to instrument our applications. OpenTelemetry achieves that by providing us with a specification that can be implemented in multiple languages. But there is another component, the OTEL Collector:

The OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. This works with improved scalability and supports open-source observability data formats (e.g. Jaeger, Prometheus, Fluent Bit, etc.) sending to one or more open-source or commercial back-ends. The local Collector agent is the default location to which instrumentation libraries export their telemetry data.

In short, instead of sending our telemetry data to a specific backend, we could send it to OTEL Collector and from there to any backend. Additionally, the collector can take care of additional handling like retries, batching, encryption, or even sensitive data filtering. The OTEL Collector consists of three components that access telemetry data: receivers, processors, and exporters, which are used to build pipelines.

Receivers

A receiver is how data gets into the OpenTelemetry Collector. Generally, a receiver accepts data in a specified format, translates it into the internal format and passes it to processors and exporters defined in the applicable pipelines.

Here, you can find all the available receivers.

Exporters

An exporter is how data gets sent to different systems/back-ends. Generally, an exporter translates the internal format into another defined format.

Here, you can find all the available exporters.

Processors

Processors are used at various stages of a pipeline. Generally, a processor pre-processes data before it is exported (e.g. modify attributes or sample) or helps ensure that data makes it through a pipeline successfully (e.g. batch/retry).

Pipelines

A pipeline consists of a set of receivers, processors and exporters. Each receiver/processor/exporter must be defined in the configuration outside of the service section to be included in a pipeline. Pipelines can be of the following types:

  • Traces: collects and processes trace data.
  • Metrics: collects and processes metric data.
  • Logs: collects and processes log data.

Let's clone or download the following code. Open the solution and add the following NuGet packages:

Edit the Program.cs file as follow:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;
using AnimeQuoteApi;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.AddHttpClient<QuoteClient>(client=>
{
    client.BaseAddress = new Uri("https://animechan.vercel.app");
});

builder.Services.AddOpenTelemetryTracing(builder =>
{
    builder
    .AddOtlpExporter()
    .AddSource("AnimeQuoteApi")
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
    .AddHttpClientInstrumentation()
    .AddAspNetCoreInstrumentation()
    ;
});

builder.Services.AddOpenTelemetryMetrics(builder =>
{
    builder
    .AddRuntimeInstrumentation()
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
    .AddOtlpExporter();
});

builder.Logging.ClearProviders();
builder.Logging.AddOpenTelemetry(builder =>
{
    builder
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AnimeQuoteApi"))
    .AddOtlpExporter();
});

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

Run the following command to start Jeager:

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -e COLLECTOR_OTLP_ENABLED=false -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14250:14250 -p 14268:14268 -p 14269:14269 -p 9411:9411  jaegertracing/all-in-one:1.36

Edit the prometheus.yml file as follows:

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: "AnimeQuoteApi"
    scrape_interval: 5s
    static_configs:
      - targets: ["host.docker.internal:8889"]

Run the following command to start Prometheus:

docker run -d --name prometheus -p 9090:9090 -v ${pwd}/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Add an otel-collector-config.yaml file with the following content:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 15s
    send_batch_size: 10

exporters:
  jaeger:
    endpoint: "host.docker.internal:14250"
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "default"
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Add a docker-compose.yml file:

version: "3.4"

services:
  aws-ot-collector:
    image: otel/opentelemetry-collector-contrib:0.56.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver

And, run the following command to start the OTEL Collector:

docker-compose up

Run the application and send a couple of requests to start to see traces in Jeager (localhost:16686/search):

jeager.PNG

Metrics in Prometheus (localhost:9090):

prometheus.PNG

And logs in the console:

console.PNG

You can see all the code here. Thanks, and happy coding.