LogWisp - Simple Log Streaming

A lightweight log streaming service that monitors files and streams updates via Server-Sent Events (SSE).

Philosophy

LogWisp follows the Unix philosophy: do one thing and do it well. It monitors log files and streams them over HTTP/SSE. That's it.

Features

• Monitors multiple files and directories simultaneously
• Streams log updates in real-time via SSE
• Supports both plain text and JSON formatted logs
• Automatic file rotation detection
• Configurable rate limiting
• Environment variable support
• Simple TOML configuration
• Atomic configuration management

Quick Start

1. Build:

./build.sh

2. Run with defaults (monitors current directory):

./logwisp

3. View logs:

curl -N http://localhost:8080/stream

Configuration

LogWisp uses a three-level configuration hierarchy:

1. Environment variables (highest priority)
2. Configuration file (~/.config/logwisp.toml)
3. Default values (lowest priority)

Configuration File Location

Default: ~/.config/logwisp.toml

Override with environment variables:

• LOGWISP_CONFIG_DIR - Directory containing config file
• LOGWISP_CONFIG_FILE - Config filename (absolute or relative)

Examples:

# Use config from current directory
LOGWISP_CONFIG_DIR=. ./logwisp

# Use specific config file
LOGWISP_CONFIG_FILE=/etc/logwisp/prod.toml ./logwisp

# Use custom directory and filename
LOGWISP_CONFIG_DIR=/opt/configs LOGWISP_CONFIG_FILE=myapp.toml ./logwisp

Environment Variables

All configuration values can be overridden via environment variables:

Environment Variable	Config Path	Description
LOGWISP_PORT	port	HTTP listen port
LOGWISP_MONITOR_CHECK_INTERVAL_MS	monitor.check_interval_ms	File check interval
LOGWISP_MONITOR_TARGETS	monitor.targets	Comma-separated targets
LOGWISP_STREAM_BUFFER_SIZE	stream.buffer_size	Client buffer size
LOGWISP_STREAM_RATE_LIMIT_ENABLED	stream.rate_limit.enabled	Enable rate limiting
LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC	stream.rate_limit.requests_per_second	Rate limit
LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE	stream.rate_limit.burst_size	Burst size
LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL	stream.rate_limit.cleanup_interval_s	Cleanup interval

Monitor Targets Format

The LOGWISP_MONITOR_TARGETS environment variable uses a special format:

path:pattern:isfile,path2:pattern2:isfile

Examples:

# Monitor directory and specific file
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp

# Multiple directories
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp

Example Configuration

port = 8080

[monitor]
check_interval_ms = 100

# Monitor directory (all .log files)
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false

# Monitor specific file
[[monitor.targets]]
path = "/app/logs/app.log"
pattern = ""  # Ignored for files
is_file = true

# Monitor with specific pattern
[[monitor.targets]]
path = "/var/log/nginx"
pattern = "access*.log"
is_file = false

[stream]
buffer_size = 1000

[stream.rate_limit]
enabled = true
requests_per_second = 10
burst_size = 20
cleanup_interval_s = 60

Color Support

LogWisp can pass through ANSI color escape codes from monitored logs to SSE clients using the -c flag.

# Enable color pass-through
./logwisp -c

# Or via systemd
ExecStart=/opt/logwisp/bin/logwisp -c

How It Works

When color mode is enabled (-c flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream.

Example Log with Colors

Original log file content:

\033[31mERROR\033[0m: Database connection failed
\033[33mWARN\033[0m: High memory usage detected
\033[32mINFO\033[0m: Service started successfully

SSE output with -c:

{
  "time": "2024-01-01T12:00:00.123456Z",
  "source": "app.log",
  "message": "\u001b[31mERROR\u001b[0m: Database connection failed"
}

Client-Side Handling

Terminal Clients

For terminal-based clients (like curl), the escape codes will render as colors:

# This will show colored output in terminals that support ANSI codes
curl -N http://localhost:8080/stream | jq -r '.message'

Web Clients

For web-based clients, you'll need to convert ANSI codes to HTML:

// Example using ansi-to-html library
const AnsiToHtml = require('ansi-to-html');
const convert = new AnsiToHtml();

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  const html = convert.toHtml(data.message);
  document.getElementById('log').innerHTML += html + '<br>';
};

Custom Processing

# Python example with colorama
import json
import colorama
from colorama import init

init()  # Initialize colorama for Windows support

# Process SSE stream
for line in stream:
    if line.startswith('data: '):
        data = json.loads(line[6:])
        # Colorama will handle ANSI codes automatically
        print(data['message'])

Common ANSI Color Codes

Code	Color/Style
\033[0m	Reset
\033[1m	Bold
\033[31m	Red
\033[32m	Green
\033[33m	Yellow
\033[34m	Blue
\033[35m	Magenta
\033[36m	Cyan

Limitations

1. JSON Escaping: ANSI codes are JSON-escaped in the stream (e.g., \033 becomes \u001b)
2. Client Support: The client must support or convert ANSI codes
3. Performance: No significant impact, but slightly larger message sizes

Security Note

Color codes are passed through as-is. Ensure monitored logs come from trusted sources to avoid terminal escape sequence attacks.

Disabling Colors

To strip color codes instead of passing them through:

• Don't use the -c flag
• Or set up a preprocessing pipeline:

  tail -f colored.log | sed 's/\x1b\[[0-9;]*m//g' > plain.log
  

API

Endpoints

• GET /stream - Server-Sent Events stream of log entries
• GET /status - Service status information

Log Entry Format

{
  "time": "2024-01-01T12:00:00Z",
  "source": "app.log",
  "level": "error",
  "message": "Something went wrong",
  "fields": {"key": "value"}
}

Usage Examples

Basic Usage

# Start with defaults
./logwisp

# View logs
curl -N http://localhost:8080/stream

With Environment Variables

# Change port and add rate limiting
LOGWISP_PORT=9090 \
LOGWISP_STREAM_RATE_LIMIT_ENABLED=true \
LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \
./logwisp

Monitor Multiple Locations

# Via environment variable
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \
./logwisp

# Or via config file
cat > ~/.config/logwisp.toml << EOF
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false

[[monitor.targets]]
path = "/app/logs"
pattern = "*.json"
is_file = false

[[monitor.targets]]
path = "/tmp/debug.log"
is_file = true
EOF

Production Deployment

Example systemd service with environment overrides:

[Unit]
Description=LogWisp Log Streaming Service
After=network.target

[Service]
Type=simple
User=logwisp
ExecStart=/usr/local/bin/logwisp
Restart=always

# Environment overrides
Environment="LOGWISP_PORT=8080"
Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true"
Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100"
Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false"

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/
ReadWritePaths=/var/log

[Install]
WantedBy=multi-user.target

Rate Limiting

When enabled, rate limiting is applied per client IP address:

[stream.rate_limit]
enabled = true
requests_per_second = 10  # Sustained rate
burst_size = 20           # Allow bursts up to this size
cleanup_interval_s = 60   # Clean old clients every minute

Rate limiting uses the X-Forwarded-For header if present, falling back to RemoteAddr.

Building from Source

Requirements:

• Go 1.23 or later

go mod download
go build -o logwisp ./src/cmd/logwisp

File Rotation Detection

LogWisp automatically detects log file rotation by:

• Monitoring file inode changes (Linux/Unix)
• Detecting file size decrease
• Resetting read position when rotation is detected

Security Notes

1. No built-in authentication - Use a reverse proxy for auth
2. No TLS support - Use a reverse proxy for HTTPS
3. Path validation - Monitors only specified paths
4. Rate limiting - Optional but recommended for internet-facing deployments

Design Decisions

• Unix philosophy: Single purpose - stream logs
• No CLI arguments: Configuration via file and environment only
• SSE over WebSocket: Simpler, works everywhere
• Atomic config management: Using LixenWraith/config package
• Graceful shutdown: Proper cleanup on SIGINT/SIGTERM

License

BSD-3-Clause