lixen/logwisp

SHA256

Fork 0

Files

Lixen Wraith a3450a9589 v0.1.2 update readme and config, failed attempt to fix slow client

2025-07-01 16:34:19 -04:00

14 KiB

Raw Blame History

LogWisp - Simple Log Streaming

A lightweight log streaming service that monitors files and streams updates via Server-Sent Events (SSE).

Philosophy

LogWisp follows the Unix philosophy: do one thing and do it well. It monitors log files and streams them over HTTP/SSE. That's it.

Features

Monitors multiple files and directories simultaneously
Streams log updates in real-time via SSE
Supports both plain text and JSON formatted logs
Automatic file rotation detection
Configurable rate limiting
Environment variable support
Simple TOML configuration
Atomic configuration management
Optional ANSI color pass-through

Quick Start

Build:

./build.sh

Run with defaults (monitors current directory):

./logwisp

View logs:

curl -N http://localhost:8080/stream

Command Line Options

logwisp [OPTIONS] [TARGET...]

OPTIONS:
  -c, --color           Enable color pass-through for ANSI escape codes
  --config FILE         Config file path (default: ~/.config/logwisp.toml)
  --port PORT           HTTP port (default: 8080)
  --buffer-size SIZE    Stream buffer size (default: 1000)
  --check-interval MS   File check interval in ms (default: 100)
  --rate-limit          Enable rate limiting
  --rate-requests N     Rate limit requests/sec (default: 10)
  --rate-burst N        Rate limit burst size (default: 20)

TARGET:
  path[:pattern[:isfile]]   Path to monitor (file or directory)
                           pattern: glob pattern for directories (default: *.log)
                           isfile: true/false (auto-detected if omitted)

EXAMPLES:
  # Monitor current directory for *.log files
  logwisp

  # Monitor specific file with color support
  logwisp -c /var/log/app.log

  # Monitor multiple locations
  logwisp /var/log:*.log /app/logs:error*.log:false /tmp/debug.log::true

  # Custom port with rate limiting
  logwisp --port 9090 --rate-limit --rate-requests 100 --rate-burst 200

Configuration

LogWisp uses a three-level configuration hierarchy:

Command-line arguments (highest priority)
Environment variables
Configuration file (~/.config/logwisp.toml)
Default values (lowest priority)

Default Values

Setting	Default	Description
`port`	8080	HTTP listen port
`monitor.check_interval_ms`	100	File check interval (milliseconds)
`monitor.targets`	[{"path": "./", "pattern": "*.log", "is_file": false}]	Paths to monitor
`stream.buffer_size`	1000	Per-client event buffer size
`stream.rate_limit.enabled`	false	Enable rate limiting
`stream.rate_limit.requests_per_second`	10	Sustained request rate
`stream.rate_limit.burst_size`	20	Maximum burst size
`stream.rate_limit.cleanup_interval_s`	60	Client cleanup interval

Configuration File Location

Default: ~/.config/logwisp.toml

Override with environment variables:

LOGWISP_CONFIG_DIR - Directory containing config file
LOGWISP_CONFIG_FILE - Config filename (absolute or relative)

Examples:

# Use config from current directory
LOGWISP_CONFIG_DIR=. ./logwisp

# Use specific config file
LOGWISP_CONFIG_FILE=/etc/logwisp/prod.toml ./logwisp

# Use custom directory and filename
LOGWISP_CONFIG_DIR=/opt/configs LOGWISP_CONFIG_FILE=myapp.toml ./logwisp

Environment Variables

All configuration values can be overridden via environment variables:

Environment Variable	Config Path	Description
`LOGWISP_PORT`	`port`	HTTP listen port
`LOGWISP_MONITOR_CHECK_INTERVAL_MS`	`monitor.check_interval_ms`	File check interval
`LOGWISP_MONITOR_TARGETS`	`monitor.targets`	Comma-separated targets
`LOGWISP_STREAM_BUFFER_SIZE`	`stream.buffer_size`	Client buffer size
`LOGWISP_STREAM_RATE_LIMIT_ENABLED`	`stream.rate_limit.enabled`	Enable rate limiting
`LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC`	`stream.rate_limit.requests_per_second`	Rate limit
`LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE`	`stream.rate_limit.burst_size`	Burst size
`LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL_S`	`stream.rate_limit.cleanup_interval_s`	Cleanup interval

Monitor Targets Format

The LOGWISP_MONITOR_TARGETS environment variable uses a special format:

path:pattern:isfile,path2:pattern2:isfile

Examples:

# Monitor directory and specific file
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp

# Multiple directories
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp

Complete Configuration Example

# Port to listen on (default: 8080)
port = 8080

[monitor]
# How often to check for file changes in milliseconds (default: 100)
check_interval_ms = 100

# Paths to monitor
# Default: [{"path": "./", "pattern": "*.log", "is_file": false}]

# Monitor all .log files in current directory
[[monitor.targets]]
path = "./"
pattern = "*.log"
is_file = false

# Monitor specific file
[[monitor.targets]]
path = "/app/logs/app.log"
pattern = ""  # Ignored for files
is_file = true

# Monitor with specific pattern
[[monitor.targets]]
path = "/var/log/nginx"
pattern = "access*.log"
is_file = false

[stream]
# Buffer size for each client connection (default: 1000)
# Controls how many log entries can be queued per client
buffer_size = 1000

[stream.rate_limit]
# Enable rate limiting (default: false)
enabled = false

# Requests per second per client (default: 10)
# This is the sustained rate
requests_per_second = 10

# Burst size - max requests at once (default: 20)
# Allows temporary bursts above the sustained rate
burst_size = 20

# How often to clean up old client limiters in seconds (default: 60)
# Clients inactive for 2x this duration are removed
cleanup_interval_s = 60

Color Support

LogWisp can pass through ANSI color escape codes from monitored logs to SSE clients using the -c flag.

# Enable color pass-through
./logwisp -c

# Or via systemd
ExecStart=/opt/logwisp/bin/logwisp -c

How It Works

When color mode is enabled (-c flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream.

Example Log with Colors

Original log file content:

\033[31mERROR\033[0m: Database connection failed
\033[33mWARN\033[0m: High memory usage detected
\033[32mINFO\033[0m: Service started successfully

SSE output with -c:

{
  "time": "2024-01-01T12:00:00.123456Z",
  "source": "app.log",
  "message": "\u001b[31mERROR\u001b[0m: Database connection failed"
}

Client-Side Handling

Terminal Clients

For terminal-based clients (like curl), the escape codes will render as colors:

# This will show colored output in terminals that support ANSI codes
curl -N http://localhost:8080/stream | jq -r '.message'

Web Clients

For web-based clients, you'll need to convert ANSI codes to HTML:

// Example using ansi-to-html library
const AnsiToHtml = require('ansi-to-html');
const convert = new AnsiToHtml();

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  const html = convert.toHtml(data.message);
  document.getElementById('log').innerHTML += html + '<br>';
};

Custom Processing

# Python example with colorama
import json
import colorama
from colorama import init

init()  # Initialize colorama for Windows support

# Process SSE stream
for line in stream:
    if line.startswith('data: '):
        data = json.loads(line[6:])
        # Colorama will handle ANSI codes automatically
        print(data['message'])

Common ANSI Color Codes

Code	Color/Style
`\033[0m`	Reset
`\033[1m`	Bold
`\033[31m`	Red
`\033[32m`	Green
`\033[33m`	Yellow
`\033[34m`	Blue
`\033[35m`	Magenta
`\033[36m`	Cyan

Limitations

JSON Escaping: ANSI codes are JSON-escaped in the stream (e.g., \033 becomes \u001b)
Client Support: The client must support or convert ANSI codes
Performance: No significant impact, but slightly larger message sizes

Security Note

Color codes are passed through as-is. Ensure monitored logs come from trusted sources to avoid terminal escape sequence attacks.

Disabling Colors

To strip color codes instead of passing them through:

Don't use the -c flag

Or set up a preprocessing pipeline:

tail -f colored.log | sed 's/\x1b\[[0-9;]*m//g' > plain.log

API

Endpoints

GET /stream - Server-Sent Events stream of log entries
GET /status - Service status and configuration information

Log Entry Format

{
  "time": "2024-01-01T12:00:00.123456Z",
  "source": "app.log",
  "level": "error",
  "message": "Something went wrong",
  "fields": {"key": "value"}
}

SSE Event Types

Event	Description	Data Format
`connected`	Initial connection	`{"client_id": "123456789"}`
`data`	Log entry	JSON log entry
`disconnected`	Client disconnected	`{"reason": "slow_client"}`
`timeout`	Client timeout	`{"reason": "client_timeout"}`
`:`	Heartbeat (comment)	ISO timestamp

Status Response Format

{
  "service": "LogWisp",
  "version": "2.0.0",
  "port": 8080,
  "color_mode": false,
  "config": {
    "monitor": {
      "check_interval_ms": 100,
      "targets_count": 2
    },
    "stream": {
      "buffer_size": 1000,
      "rate_limit": {
        "enabled": true,
        "requests_per_second": 10,
        "burst_size": 20
      }
    }
  },
  "streamer": {
    "active_clients": 5,
    "buffer_size": 1000,
    "color_mode": false,
    "total_dropped": 42
  },
  "rate_limiter": "Active clients: 3"
}

Usage Examples

Basic Usage

# Start with defaults
./logwisp

# Monitor specific file
./logwisp /var/log/app.log

# View logs
curl -N http://localhost:8080/stream

With Environment Variables

# Change port and add rate limiting
LOGWISP_PORT=9090 \
LOGWISP_STREAM_RATE_LIMIT_ENABLED=true \
LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \
./logwisp

Monitor Multiple Locations

# Via command line
./logwisp /var/log:*.log /app/logs:*.json /tmp/debug.log

# Via environment variable
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \
./logwisp

# Or via config file
cat > ~/.config/logwisp.toml << EOF
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false

[[monitor.targets]]
path = "/app/logs"
pattern = "*.json"
is_file = false

[[monitor.targets]]
path = "/tmp/debug.log"
is_file = true
EOF

Production Deployment

Example systemd service with environment overrides:

[Unit]
Description=LogWisp Log Streaming Service
After=network.target

[Service]
Type=simple
User=logwisp
ExecStart=/usr/local/bin/logwisp -c
Restart=always
RestartSec=5

# Environment overrides
Environment="LOGWISP_PORT=8080"
Environment="LOGWISP_STREAM_BUFFER_SIZE=5000"
Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true"
Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100"
Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false"

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/
ReadWritePaths=/var/log /opt/app/logs

[Install]
WantedBy=multi-user.target

Performance Tuning

Buffer Size

The stream.buffer_size setting controls how many log entries can be queued per client:

Small buffers (100-500): Lower memory usage, clients skip entries during bursts
Default (1000): Good balance for most use cases
Large buffers (5000+): Handle burst traffic better, higher memory usage

When a client's buffer is full, new messages are skipped for that client until it catches up. The client remains connected and will receive future messages once buffer space is available.

Check Interval

The monitor.check_interval_ms setting controls file polling frequency:

Fast (10-50ms): Near real-time updates, higher CPU usage
Default (100ms): Good balance
Slow (500ms+): Lower CPU usage, more latency

Rate Limiting

When to enable rate limiting:

Internet-facing deployments
Shared environments
Protection against misbehaving clients

Rate limiting applies only to establishing SSE connections, not to individual messages. Once connected, clients receive all messages (subject to buffer capacity).

Troubleshooting

Client Missing Messages

If clients miss messages during bursts:

Check total_dropped and clients_with_drops in status endpoint
Increase stream.buffer_size to handle larger bursts
Messages are skipped when buffer is full, but clients stay connected

High Memory Usage

If memory usage is high:

Reduce stream.buffer_size
Enable rate limiting to limit concurrent connections
Each client uses buffer_size * avg_message_size memory

Browser Stops Receiving Updates

This shouldn't happen with the current implementation. If it does:

Check browser developer console for errors
Verify no proxy/firewall is timing out the connection
Ensure reverse proxy (if used) doesn't buffer SSE responses

File Rotation Detection

LogWisp automatically detects log file rotation using multiple methods:

Inode changes (Linux/Unix)
File size decrease
Modification time reset
Read position beyond file size

When rotation is detected, LogWisp:

Logs a rotation event
Resets read position to beginning
Continues streaming from new file

Security Notes

No built-in authentication - Use a reverse proxy for auth
No TLS support - Use a reverse proxy for HTTPS
Path validation - Only specified paths can be monitored
Directory traversal protection - Paths containing ".." are rejected
Rate limiting - Optional but recommended for public deployments
ANSI escape sequences - Only enable color mode for trusted log sources

Design Decisions

Unix philosophy: Single purpose - stream logs
SSE over WebSocket: Simpler, works everywhere, built-in reconnect
No database: Stateless operation, instant startup
Atomic config management: Using LixenWraith/config package
Graceful shutdown: Proper cleanup on SIGINT/SIGTERM
Platform agnostic: POSIX-compliant where possible

Building from Source

Requirements:

Go 1.23 or later

git clone https://github.com/yourusername/logwisp
cd logwisp
go mod download
go build -o logwisp ./src/cmd/logwisp

License

BSD-3-Clause

14 KiB Raw Blame History