.NET and Kubernetes: Collect dumps on crash

Collecting dumps after a crash is an invaluable practice for understanding why it occurred. For .NET applications, this can be achieved simply by specifying environment variables. However, how do we accomplish this when hosting our application in a Kubernetes cluster? It's particularly challenging when the dump is generated inside the container, which will be destroyed shortly after the crash. In this article, we will explore how Persistent Volumes can help address this challenge.

Pre-requisites

Install Docker Desktop
Enable Kubernetes (the standalone version included in Docker Desktop)

The Application

Run the following command to create a simple .NET API:

dotnet new web -o CrashApi
dotnet new sln -n CrashApi
dotnet sln add --in-root CrashApi

Navigate to the Program.cs file and update it as follows:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/", () => "Hello World!");
throw new Exception("Boom!!");
app.Run();

This application will crash as soon as it starts.

The Container Image

At the project level, we need a Dockerfile to containerize our application. It should include the following content:

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
COPY ["CrashApi/CrashApi.csproj", "CrashApi/"]
RUN dotnet restore "CrashApi/CrashApi.csproj"
COPY . .
WORKDIR "/CrashApi"
RUN dotnet build "CrashApi.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "CrashApi.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "CrashApi.dll"]

At the solution level, execute the following command to create the container image:

docker build -t raulnq/crashapi:1.0 -f .\CrashApi\Dockerfile .

The Persistent Volume

In the persistent volume, we will mount the local directory C:\Temp. Create a persistentvolume.yaml file at the solution level with the following content:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-persistent-volume
  labels:
    type: local
spec:
  storageClassName: hostpath
  capacity:
    storage: 256Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/run/desktop/mnt/host/c/Temp"
  persistentVolumeReclaimPolicy: Retain

Since we are using Docker Desktop for Windows, the storageClassName is set to hostpath, and the path is configured as /run/desktop/mnt/host/c/Temp. Generally, the path follows the pattern /run/desktop/mnt/host/MY_LOCAL_PATH. To create the Persistent Volume Claim, create a persistentvolumeclaim.yaml file with the following content:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-persistent-volume-claim
spec:
  storageClassName: hostpath
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 128Mi

The Deployment

Create a deployment.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: my-container
        image: raulnq/crashapi:1.0
        imagePullPolicy: IfNotPresent
        env:
        - name: DOTNET_DbgEnableMiniDump
          value: "1"
        - name: DOTNET_DbgMiniDumpType
          value: "1"
        - name: DOTNET_DbgMiniDumpName
          value: "/dumps/coredump.%p.dmp"
        volumeMounts:
        - mountPath: /dumps
          name: my-volume
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-persistent-volume-claim

Here, we mounted the previously created Persistent Volume Claim and set the following environment variables:

COMPlus_DbgEnableMiniDump or DOTNET_DbgEnableMiniDump: If set to 1, this enables core dump generation.
COMPlus_DbgMiniDumpType or DOTNET_DbgMiniDumpType: Specifies the type of dump to be collected (Mini, Heap, Triage, and Full).
COMPlus_DbgMiniDumpName or DOTNET_DbgMiniDumpName: Path to the file where the dump will be written. The %p acts as a wildcard for the PID of the dumped process.

The prefix DOTNET_ becomes functional, starting with .NET 7 on all platforms. Execute the following commands to deploy the application to the cluster:

kubectl apply -f .\persistentvolume.yaml
kubectl apply -f .\persistentvolumeclaim.yaml
kubectl apply -f .\deployment.yaml

The Dump

The command kubectl get pods will display our application:

NAME                              READY   STATUS             RESTARTS       AGE
my-deployments-5884c58f88-mgxvw   0/1     CrashLoopBackOff   4 (84s ago)    2m59s

Check the pod logs with the command kubectl logs my-deployments-5884c58f88-mgxvw:

Unhandled exception. System.Exception: Boom!!
   at Program.<Main>$(String[] args) in /CrashApi/Program.cs:line 4

To analyze the dump, we need a tool like dotnet-dump. Run the following command to install it:

dotnet tool install --global dotnet-dump

Finally, run dotnet-dump analyze C:\Temp\coredump.1.dmp to begin the analysis (you might need to delete our deployment first):

Loading core dump: C:\Temp\coredump.1.dmp ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.

As a final note, in a real environment, the persistent volume should be replaced with an appropriate type like csi. The final code can be found here. Thank you, and happy coding.