.NET and Kubernetes: Collect dumps on crash

.NET and Kubernetes: Collect dumps on crash

Collecting dumps after a crash is an invaluable practice for understanding why it occurred. For .NET applications, this can be achieved simply by specifying environment variables. However, how do we accomplish this when hosting our application in a Kubernetes cluster? It's particularly challenging when the dump is generated inside the container, which will be destroyed shortly after the crash. In this article, we will explore how Persistent Volumes can help address this challenge.


The Application

Run the following command to create a simple .NET API:

dotnet new web -o CrashApi
dotnet new sln -n CrashApi
dotnet sln add --in-root CrashApi

Navigate to the Program.cs file and update it as follows:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/", () => "Hello World!");
throw new Exception("Boom!!");

This application will crash as soon as it starts.

The Container Image

At the project level, we need a Dockerfile to containerize our application. It should include the following content:

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
COPY ["CrashApi/CrashApi.csproj", "CrashApi/"]
RUN dotnet restore "CrashApi/CrashApi.csproj"
COPY . .
WORKDIR "/CrashApi"
RUN dotnet build "CrashApi.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "CrashApi.csproj" -c Release -o /app/publish

FROM base AS final
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "CrashApi.dll"]

At the solution level, execute the following command to create the container image:

docker build -t raulnq/crashapi:1.0 -f .\CrashApi\Dockerfile .

The Persistent Volume

In the persistent volume, we will mount the local directory C:\Temp. Create a persistentvolume.yaml file at the solution level with the following content:

apiVersion: v1
kind: PersistentVolume
  name: my-persistent-volume
    type: local
  storageClassName: hostpath
    storage: 256Mi
    - ReadWriteMany
    path: "/run/desktop/mnt/host/c/Temp"
  persistentVolumeReclaimPolicy: Retain

Since we are using Docker Desktop for Windows, the storageClassName is set to hostpath, and the path is configured as /run/desktop/mnt/host/c/Temp. Generally, the path follows the pattern /run/desktop/mnt/host/MY_LOCAL_PATH. To create the Persistent Volume Claim, create a persistentvolumeclaim.yaml file with the following content:

apiVersion: v1
kind: PersistentVolumeClaim
  name: my-persistent-volume-claim
  storageClassName: hostpath
    - ReadWriteMany
      storage: 128Mi

The Deployment

Create a deployment.yaml file with the following content:

apiVersion: apps/v1
kind: Deployment
  name: my-deployments
  replicas: 1
      app: demo
        app: demo
      - name: my-container
        image: raulnq/crashapi:1.0
        imagePullPolicy: IfNotPresent
        - name: DOTNET_DbgEnableMiniDump
          value: "1"
        - name: DOTNET_DbgMiniDumpType
          value: "1"
        - name: DOTNET_DbgMiniDumpName
          value: "/dumps/coredump.%p.dmp"
        - mountPath: /dumps
          name: my-volume
      - name: my-volume
          claimName: my-persistent-volume-claim

Here, we mounted the previously created Persistent Volume Claim and set the following environment variables:

  • COMPlus_DbgEnableMiniDump or DOTNET_DbgEnableMiniDump: If set to 1, this enables core dump generation.

  • COMPlus_DbgMiniDumpType or DOTNET_DbgMiniDumpType: Specifies the type of dump to be collected (Mini, Heap, Triage, and Full).

  • COMPlus_DbgMiniDumpName or DOTNET_DbgMiniDumpName: Path to the file where the dump will be written. The %p acts as a wildcard for the PID of the dumped process.

The prefix DOTNET_ becomes functional, starting with .NET 7 on all platforms. Execute the following commands to deploy the application to the cluster:

kubectl apply -f .\persistentvolume.yaml
kubectl apply -f .\persistentvolumeclaim.yaml
kubectl apply -f .\deployment.yaml

The Dump

The command kubectl get pods will display our application:

NAME                              READY   STATUS             RESTARTS       AGE
my-deployments-5884c58f88-mgxvw   0/1     CrashLoopBackOff   4 (84s ago)    2m59s

Check the pod logs with the command kubectl logs my-deployments-5884c58f88-mgxvw:

Unhandled exception. System.Exception: Boom!!
   at Program.<Main>$(String[] args) in /CrashApi/Program.cs:line 4

To analyze the dump, we need a tool like dotnet-dump. Run the following command to install it:

dotnet tool install --global dotnet-dump

Finally, run dotnet-dump analyze C:\Temp\coredump.1.dmp to begin the analysis (you might need to delete our deployment first):

Loading core dump: C:\Temp\coredump.1.dmp ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.

As a final note, in a real environment, the persistent volume should be replaced with an appropriate type like csi. The final code can be found here. Thank you, and happy coding.