logwisp/doc/architecture.md

Architecture Overview

LogWisp implements a pipeline-based architecture for flexible log processing and distribution.

Core Concepts

Pipeline Model

Each pipeline operates independently with a source → filter → format → sink flow. Multiple pipelines can run concurrently within a single LogWisp instance, each processing different log streams with unique configurations.

Component Hierarchy

Service (Main Process)
├── Pipeline 1
│   ├── Plugin Sources (1 or more)
│   ├── Flow
│   │   ├── Heartbeat Generator (optional)
│   │   ├── Rate Limiter (optional)
│   │   ├── Filter Chain (optional)
│   │   └── Formatter (optional)
│   └── Plugin Sinks (1 or more)
├── Pipeline 2
│   └── [Similar structure]
└── Status Reporter (optional)

Data Flow

Processing Stages

1. Source Stage: Plugin sources monitor inputs and generate log entries
2. Flow - Rate Limiting: Optional pipeline-level rate control
3. Flow - Filtering: Pattern-based inclusion/exclusion
4. Flow - Formatting: Transform entries to desired output format with sanitization
5. Distribution: Fan-out to multiple plugin sinks

Entry Lifecycle

Log entries flow through the pipeline as core.LogEntry structures containing:

• Time: Entry timestamp
• Level: Log level (DEBUG, INFO, WARN, ERROR)
• Source: Origin identifier
• Message: Log content
• Fields: Additional metadata (JSON)
• RawSize: Original entry size

Buffering Strategy

Each component maintains internal buffers to handle burst traffic:

• Sources: Configurable buffer size (default 1000 entries)
• Sinks: Independent buffers per sink
• Network components: Additional TCP/HTTP buffers

"Sink dispatch uses non-blocking sends. When a sink's input buffer is full, the event is dropped for that sink only and counted in pipeline statistics (total_dropped_by_sink). The policy is uniform and not configurable; a slow sink does not stall the pipeline or other sinks."

Component Types

Sources (Input)

• File Source: File system directory monitoring with rotation detection
• Console Source: Standard input processing (stdin)
• Random Source: Generates random log entries for testing
• Null Source: Discards logs, used for testing

Sinks (Output)

• Console Sink: stdout/stderr output
• File Sink: Rotating file writer
• HTTP Sink: Server-Sent Events (SSE) streaming
• TCP Sink: TCP server for client connections
• Null Sink: Discards all received events

Processing Components

• Rate Limiter: Token bucket algorithm for flow control
• Filter Chain: Sequential pattern matching
• Formatters: Raw, JSON, or text transformation with sanitizer policies

Concurrency Model

Goroutine Architecture

• Each source runs in dedicated goroutines for monitoring
• Sinks operate independently with their own processing loops
• Network listeners use optimized event loops (gnet for TCP)
• Pipeline processing uses channel-based communication

Synchronization

• Atomic counters for statistics
• Read-write mutexes for configuration access
• Context-based cancellation for graceful shutdown
• Wait groups for coordinated startup/shutdown

Network Architecture

Connection Patterns

Chaining Design:

• Future plan

Monitoring Design:

• TCP Sink: Debugging interface
• HTTP Sink: Browser-based live monitoring

Protocol Support

• HTTP/1.1 and HTTP/2 for HTTP connections
• Raw TCP connections

Resource Management

Memory Management

• Bounded buffers prevent unbounded growth
• Automatic garbage collection via Go runtime
• Connection limits prevent resource exhaustion

File Management

• Automatic rotation based on size thresholds
• Retention policies for old log files
• Minimum disk space checks before writing

Connection Management

• HTTP sink silenty drops IPv6 connections (deliberate IPv4-only enforcement)

Note: Placeholder; below features are removed in restructuring and will be added in the future release.

• Per-IP connection limits
• Global connection caps
• Automatic reconnection with exponential backoff
• Keep-alive for persistent connections

Reliability Features

Fault Tolerance

• Panic recovery in pipeline processing
• Independent pipeline operation
• Sink failure isolation

Data Integrity

• Entry validation at ingestion
• Size limits for entries and batches
• Duplicate detection in file monitoring
• Position tracking for file reads

Performance Characteristics

Throughput

• Pipeline rate limiting: Configurable (default 1000 entries/second)
• Network throughput: Limited by network and sink capacity
• File monitoring: Sub-second detection (default 100ms interval)

Latency

• Entry processing: Sub-millisecond in-memory
• Network forwarding: Depends on batch configuration
• File detection: Configurable check interval

Scalability

• Horizontal: Multiple LogWisp instances with different configurations
• Vertical: Multiple pipelines per instance
• Fan-out: Multiple sinks per pipeline
• Fan-in: Multiple sources per pipeline