# Forward Client-Side Logs to Splunk

Client-side applications, such as React SPAs, generate valuable logs on their side: UI errors, unexpected flows, browser-specific issues, performance signals, and metrics about user interactions. However, centralizing these logs in enterprise monitoring systems like Splunk presents unique challenges:

* **Security constraints**: Client-side applications typically cannot send logs directly to Splunk due to security policies, credential exposure risks, and CORS restrictions.
    
* **Format compatibility**: When our backend uses [Serilog](https://serilog.net/) with [Serilog.Sinks.Splunk](https://github.com/serilog-contrib/serilog-sinks-splunk), maintaining a consistent log format across frontend and backend becomes critical for unified querying and alerting.
    
* **Network efficiency**: Batching, retry logic, and error handling require careful implementation to avoid losing logs or impacting application performance.
    

This article presents a practical solution: a TypeScript logging library that mirrors Serilog's format, combined with a .NET proxy that forwards logs to Splunk. We'll examine the implementation from the [ui-logger-to-proxy](https://github.com/raulnq/ui-logger-to-proxy) repository, explaining both the architectural decisions and the technical details.

## The Architecture

The solution consists of three components:

* **Client-side logger** (TypeScript): Captures logs in Serilog-compatible format.
    
* **.NET proxy API**: Receives logs from clients and forwards them to Splunk.
    
* **Splunk**: The final destination for centralized log storage and analysis.
    

## Understanding the Serilog Log Format

Before implementing the client-side logger, we need to understand the target format. Serilog.Sinks.Splunk produces JSON logs with this structure:

```json
{
  "Level": "Information",
  "RenderedMessage": "User logged in successfully",
  "MessageTemplate": "User {UserName} logged in successfully",
  "Properties": {
    "UserName": "johndoe@gmail.com",
    "SessionId": "abc123"
  },
  "ReleaseVersion": "10.0.0",
  "Timestamp": "2024-11-14T10:30:00.000Z"
}
```

Key fields explained:

* **Level**: Log severity (Verbose, Debug, Information, Warning, Error, Fatal).
    
* **MessageTemplate**: The template string with placeholders (e.g., `{UserName}`).
    
* **RenderedMessage**: The final message with placeholders replaced.
    
* **Properties**: Structured data extracted from the message template or added as a context.
    
* **And Other Custom Fields**.
    

The distinction between `MessageTemplate` and `RenderedMessage` is crucial. Splunk can index and query based on the template pattern, enabling queries like "show all login failures" regardless of the specific username.

## Client-Side Implementation

The TypeScript `BatchLogger` class provides the foundation for structured logging with Serilog compatibility:

```typescript
type LogLevel =
  | "Verbose"
  | "Debug"
  | "Information"
  | "Warning"
  | "Error"
  | "Fatal";

type LogEntry = {
  time: number;
  host: string;
  source: string;
  sourcetype: string;
  index: string;
  event: {
    Level: LogLevel;
    RenderedMessage: string;
    MessageTemplate: string;
    Properties: Record<string, any>;
    Exception?: string;
    [key: string]: any;
  };
};
```

This structure mirrors Serilog's internal format, ensuring seamless integration with existing Splunk configurations.

### **Configuration**

The `BatchLogger` constructor accepts a comprehensive configuration object that controls both behavior and performance characteristics:

```typescript
type LoggerConfig = {
  source: string;
  sourcetype: string;
  index: string;
  host: string;
  endpoint: string;
  batchSize: number;
  flushInterval: number;
  maxRetries: number;
  minimumLogLevel?: LogLevel;
  enrichment?: Record<string, any>;
};
```

#### **Configuration Parameters**

**Splunk Metadata:**

* `source`: Identifies the application generating logs (e.g., "my-app").
    
* `sourcetype`: Categorizes the log format (e.g., "ui", "json").
    
* `index`: Specifies the Splunk index for storage (e.g., "my-index").
    
* `host`: Logical hostname for the logs (e.g., "web-client").
    

**Network Configuration:**

* `endpoint`: URL of the .NET proxy server (e.g., "[http://localhost:5244/collector")](http://localhost:5244/collector).
    

**Performance Tuning:**

* `batchSize`: Number of log entries per batch (default: 10).
    
* `flushInterval`: Maximum time to hold logs before sending in milliseconds (default: 5000).
    
* `maxRetries`: Number of retry attempts for failed requests (default: 3).
    

**Log Level Filtering:**

* `minimumLogLevel`: Minimum log level to process (default: "Information").
    
    * Only logs at or above this level will be processed and sent.
        
    * Hierarchy: Verbose(0) &lt; Debug(1) &lt; Information(2) &lt; Warning(3) &lt; Error(4) &lt; Fatal(5).
        

**Global Enrichment:**

* `enrichment`: Key-value pairs added to all log entries (e.g., version, environment).
    

#### **Considerations**

**Batch Size:**

* **Small batches (5-10)**: Better for real-time monitoring, higher network overhead.
    
* **Large batches (20-50)**: More efficient network usage, potential memory pressure.
    
* **Very large batches (100+)**: Risk of losing many logs on failures.
    

**Flush Interval:**

* **Short intervals (1-3 seconds)**: Near real-time delivery, more network requests.
    
* **Medium intervals (5-10 seconds)**: Balanced performance and timeliness.
    
* **Long intervals (30+ seconds)**: Risk of log loss on page navigation.
    

**Retry Strategy:**

* **Few retries (1-2)**: Fast failure detection, potential log loss.
    
* **Moderate retries (3-5)**: Good balance for temporary network issues.
    
* **Many retries (10+)**: Risk of blocking the logging queue.
    

**Log Level Strategy:**

* **Verbose/Debug**: Development and troubleshooting scenarios only.
    
* **Information**: General application flow and user actions (production default).
    
* **Warning**: Potentially problematic situations that don't break functionality.
    
* **Error/Fatal**: Production environments focusing on actionable issues.
    

#### **Example**

```typescript
export const logger = new BatchLogger({
  source: "my-app",
  sourcetype: "ui",
  index: "my-index",
  host: "127.0.0.1",
  endpoint: "http://localhost:5244/collector",
  batchSize: 10,
  flushInterval: 5000,
  maxRetries: 3,
  minimumLogLevel: "Information",
  enrichment: {
    ReleaseVersion: "10.0.0",
    Environment: "Development"
  },
});
```

### **Template Rendering Engine**

The logger implements a simple but effective template rendering system:

```typescript
private renderMessage(template: string, properties: Record<string, any>): string {
  let message = template;
  for (const [key, value] of Object.entries(properties)) {
    const placeholder = `{${key}}`;
    message = message.replace(placeholder, String(value));
  }
  return message;
}
```

This approach maintains compatibility with Serilog's message template format, allowing developers to write familiar logging statements:

```typescript
logger.information("Processed {Count} items in {Duration}ms", {
  Count: 150,
  Duration: 2340,
});
```

### Log Level Filtering

The logger implements efficient log level filtering to reduce noise and improve performance:

```typescript
const LogLevelValues: Record<LogLevel, number> = {
  Verbose: 0,
  Debug: 1,
  Information: 2,
  Warning: 3,
  Error: 4,
  Fatal: 5,
};

private isEnabled(level: LogLevel): boolean {
  const minimumLogLevel = this.config.minimumLogLevel || 'Information';
  return LogLevelValues[level] >= LogLevelValues[minimumLogLevel];
}

private log(level: LogLevel, messageTemplate: string, properties?: Record<string, any>, error?: Error): void {
  // Early exit if log level is below minimum threshold
  if (!this.isEnabled(level)) return;
  
  // ... continue with log processing
}
```

**Key Benefits:**

* **Performance optimization**: Prevents unnecessary object creation and processing for filtered logs.
    
* **Centralized filtering**: Single point of control for all log level decisions.
    
* **Early exit**: Returns immediately without any overhead for filtered logs.
    

### **Batching Strategy**

Performance optimization is achieved through intelligent batching:

```typescript
private log(
  level: LogLevel,
  messageTemplate: string,
  properties?: Record<string, any>,
  error?: Error
): void {
  if (!this.isEnabled(level)) return;

  const props = properties || {};
  const renderedMessage = this.renderMessage(messageTemplate, props);
  const mergedProperties = {
    ...this.contextProperties,
    ...props,
  };
  const logEntry: LogEntry = {
    time: Date.now(),
    host: this.config.host,
    source: this.config.source,
    sourcetype: this.config.sourcetype,
    index: this.config.index,
    event: {
      Level: level,
      RenderedMessage: renderedMessage,
      MessageTemplate: messageTemplate,
      Properties: mergedProperties,
      ...this.config.enrichment,
    },
  };

  if (error) {
    logEntry.event.Exception = error.stack;
  }

  this.logQueue.push(logEntry);

  if (this.logQueue.length >= this.config.batchSize) {
    this.flush();
  } else if (!this.flushTimer) {
    this.flushTimer = window.setTimeout(
      () => this.flush(),
      this.config.flushInterval
    );
  }
}
```

The batching mechanism:

* **Size-based flushing**: Triggers when batch size is reached.
    
* **Time-based flushing**: Ensures logs aren't held indefinitely.
    
* **Event-based flushing**: Flushes on page unload and visibility changes.
    

### **Reliability Features**

The logger includes robust error handling and retry logic:

```typescript
public async flush(): Promise<void> {
    if (this.logQueue.length === 0 || this.isFlushing) return;

    this.isFlushing = true;
    const logsToSend = [...this.logQueue];
    this.logQueue = [];

    if (this.flushTimer) {
      clearTimeout(this.flushTimer);
      this.flushTimer = null;
    }

    let retries = 0;
    let success = false;

    while (retries < this.config.maxRetries && !success) {
      try {
        const controller = new AbortController();
        const timeoutId = setTimeout(() => controller.abort(), 10000);

        const response = await fetch(this.config.endpoint, {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
          },
          body: JSON.stringify(logsToSend),
          signal: controller.signal,
        });

        clearTimeout(timeoutId);

        if (response.ok) {
          success = true;
        } else {
          const shouldRetry = this.shouldRetryError(response.status);
          if (!shouldRetry) {
            console.error(
              `Non-retryable error ${response.status}: ${response.statusText}`
            );
            break;
          }
          throw new Error(`HTTP ${response.status}: ${response.statusText}`);
        }
      } catch (error: any) {
        retries++;
        if (error.name === 'AbortError') {
          console.error(
            `Request timeout (attempt ${retries}/${this.config.maxRetries})`
          );
        } else {
          console.error(
            `Failed to send logs (attempt ${retries}/${this.config.maxRetries}):`,
            error
          );
        }

        if (retries < this.config.maxRetries) {
          const delay = Math.pow(2, retries) * 1000;
          const jitter = Math.random() * 1000;
          await new Promise(resolve => setTimeout(resolve, delay + jitter));
        }
      }
    }
    if (!success) {
      console.warn('Failed to send logs after all retries. Logs:', logsToSend);
    }

    this.isFlushing = false;
  }
```

Key reliability features:

* **Exponential backoff**: Prevents overwhelming the server during failures.
    
* **Jitter**: Reduces thundering herd problems.
    
* **Request timeouts**: Prevents hanging requests.
    
* **Selective retry**: Avoids retrying non-recoverable errors.
    

### **Page Lifecycle Management**

Critical for single-page applications, the logger handles browser lifecycle events:

```typescript
constructor(config: LoggerConfig) {
  const defaults = {
    batchSize: 10,
    flushInterval: 5000,
    maxRetries: 3,
  };

  this.config = {
    ...defaults,
    ...config,
  };

  this.handleBeforeUnload = () => {
    this.flushSync();
  };
  this.handleVisibilityChange = () => {
    this.onVisibilityChange();
  };

  window.addEventListener('beforeunload', this.handleBeforeUnload);
  document.addEventListener('visibilitychange', this.handleVisibilityChange);
}

private onVisibilityChange(): void {
  if (document.hidden) {
    this.flush();
  }
}

public flushSync(): void {
  if (this.logQueue.length === 0) return;

  const logsToSend = [...this.logQueue];
  this.logQueue = [];

  if (this.flushTimer) {
    clearTimeout(this.flushTimer);
    this.flushTimer = null;
  }

  const blob = new Blob([JSON.stringify(logsToSend)], {
    type: 'application/json',
  });
  navigator.sendBeacon(this.config.endpoint, blob);
}
```

The `sendBeacon` API ensures log delivery even when users navigate away from the page, providing better log coverage for user journeys.

## **Server-Side Implementation**

### **Proxy Endpoint**

The .NET server provides a minimal proxy with CORS support:

```csharp
app.MapPost("/collector", (LogEntry[] logEntries) =>
{
    if (logEntries != null && logEntries.Length > 0)
    {
        foreach (var logEntry in logEntries)
        {
            var individualJson = JsonSerializer.Serialize(logEntry, jsonOptions);
            Log.Logger.ForwardToSplunk(individualJson);
        }
    }
    return Results.Ok(new { timestamp = DateTime.UtcNow });
});
```

This approach:

* **Accepts batched logs**: Reduces HTTP overhead.
    
* **Maintains structure**: Preserves client-generated log format.
    

### **Custom Splunk Formatter**

The key innovation is the `RawJsonFormatter` class:

```csharp
using Serilog.Events;
using Serilog.Formatting;

public class RawJsonFormatter : ITextFormatter
{
    public void Format(Serilog.Events.LogEvent logEvent, TextWriter output)
    {
        if (logEvent.Properties.TryGetValue("RawJson", out var rawJsonProperty) &&
            rawJsonProperty is ScalarValue scalarValue &&
            scalarValue.Value is string rawJson)
        {
            output.Write(rawJson);
        }
        else
        {
            throw new NotSupportedException("RawJsonFormatter only supports log events with RawJson property");
        }
    }
}
```

This formatter bypasses Serilog's standard JSON serialization, allowing client-generated JSON to pass through unchanged to Splunk. This preserves the exact structure and field names required for compatibility.

### **Serilog Extension**

The extension method simplifies the forwarding process:

```csharp
public static class SplunkJsonLoggerExtensions
{
    public static void ForwardToSplunk(this ILogger logger, string rawJson)
    {
        logger.Information("Raw JSON data received from client {@RawJson}", rawJson);
    }
}
```

By using Serilog's structured logging with the `@` operator, the raw JSON becomes a property that the custom formatter can extract. The `@` ensures that:

* **No double-encoding**: The JSON string is not serialized again.
    
* **Preserved structure**: The raw JSON maintains its exact format.
    
* **Direct access**: The formatter can extract the string property directly.
    

### **Configuration**

The Serilog configuration demonstrates the complete pipeline:

```csharp
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .WriteTo.EventCollector(
        "<SPLUNK_HOST>",// The Splunk host that is configured with an Event Collector
        "<EVENT_COLLECTOR_TOKEN>", //The token provided to authenticate to the Splunk Event Collector
        new RawJsonFormatter(), //The text formatter used to render log events into a JSON format
        "services/collector/event", //Splunk Event Collector uri
        LogEventLevel.Information, //The minimum log event level required in order to write an event to the sink.
        2,      // The interval in seconds that the queue should be instpected for batching
        100,    // The size of the batch
        1000    // Maximum number of events in the queue
    )
    .CreateLogger();
```

## Running Locally

This section provides a complete step-by-step guide to running the logging solution on our local development environment.

### Prerequisites

Ensure we have the following tools installed:

* **Node.js 24+**: Required for the React client application.
    
* **.NET 9 SDK or later**: Required for the [ASP.NET](http://ASP.NET) Core proxy server.
    
* **Docker and Docker Compose**: Required for running Splunk locally.
    
* **Git**: For cloning the repository.
    

### Clone the Repository

```bash
git clone https://github.com/raulnq/ui-logger-to-proxy.git
cd ui-logger-to-proxy
```

### Start Splunk Container

Start the Splunk container using Docker Compose:

```bash
docker-compose up -d splunk
```

**Wait for Splunk to initialize** (this typically takes 3-5 minutes). We can monitor the startup progress:

```bash
docker-compose logs -f splunk
```

Look for the message indicating Splunk has started successfully.

### Configure Splunk Index

1. **Access Splunk Web UI**: Navigate to [http://localhost:8000](http://localhost:8000)
    
2. **Login credentials**:
    
    * Username: `admin`
        
    * Password: `splunk123456.`
        
3. **Create Index**:
    
    * Go to **Settings** &gt; **Indexes**
        
    * Click **New Index**
        
    * Index Name: `my-index`
        
    * Click **Save**
        

### Start the .NET Proxy Server

Open a new terminal window and start the server:

```bash
cd server
dotnet restore
dotnet run
```

The server will start on [`http://localhost:5244`](http://localhost:5244). We should see output similar to:

```typescript
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5244
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
```

### Start the React Client

Open another terminal window and start the client:

```bash
cd client
npm install
npm run dev
```

The client will start on [`http://localhost:5173`](http://localhost:5173). We should see:

```typescript
  VITE v4.x.x  ready in xxx ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: use --host to expose
```

### Test the Complete Pipeline

1. **Open the React app**: Navigate to [http://localhost:5173](http://localhost:5173)
    
2. **Generate test logs**:
    
    * Enter a message in the text area
        
    * Click "Send Log Message"
        
3. **Verify server reception**: Check the .NET server console for log entries
    
4. **Verify Splunk indexing**:
    
    * Go to Splunk Web UI ([http://localhost:8000](http://localhost:8000))
        
    * Navigate to **Search & Reporting**
        
    * Search query: `index="my-index"`
        
    * We should see the log entries from the client.
        

Thanks and happy coding.