# LogWisp - Simple Log Streaming A lightweight log streaming service that monitors files and streams updates via Server-Sent Events (SSE). ## Philosophy LogWisp follows the Unix philosophy: do one thing and do it well. It monitors log files and streams them over HTTP/SSE. That's it. ## Features - Monitors multiple files and directories simultaneously - Streams log updates in real-time via SSE - Supports both plain text and JSON formatted logs - Automatic file rotation detection - Configurable rate limiting - Environment variable support - Simple TOML configuration - Atomic configuration management - Optional ANSI color pass-through ## Quick Start 1. Build: ```bash ./build.sh ``` 2. Run with defaults (monitors current directory): ```bash ./logwisp ``` 3. View logs: ```bash curl -N http://localhost:8080/stream ``` ## Command Line Options ```bash logwisp [OPTIONS] [TARGET...] OPTIONS: -c, --color Enable color pass-through for ANSI escape codes --config FILE Config file path (default: ~/.config/logwisp.toml) --port PORT HTTP port (default: 8080) --buffer-size SIZE Stream buffer size (default: 1000) --check-interval MS File check interval in ms (default: 100) --rate-limit Enable rate limiting --rate-requests N Rate limit requests/sec (default: 10) --rate-burst N Rate limit burst size (default: 20) TARGET: path[:pattern[:isfile]] Path to monitor (file or directory) pattern: glob pattern for directories (default: *.log) isfile: true/false (auto-detected if omitted) EXAMPLES: # Monitor current directory for *.log files logwisp # Monitor specific file with color support logwisp -c /var/log/app.log # Monitor multiple locations logwisp /var/log:*.log /app/logs:error*.log:false /tmp/debug.log::true # Custom port with rate limiting logwisp --port 9090 --rate-limit --rate-requests 100 --rate-burst 200 ``` ## Configuration LogWisp uses a three-level configuration hierarchy: 1. **Command-line arguments** (highest priority) 2. **Environment variables** 3. **Configuration file** (~/.config/logwisp.toml) 4. **Default values** (lowest priority) ### Default Values | Setting | Default | Description | |---------|---------|-------------| | `port` | 8080 | HTTP listen port | | `monitor.check_interval_ms` | 100 | File check interval (milliseconds) | | `monitor.targets` | [{"path": "./", "pattern": "*.log", "is_file": false}] | Paths to monitor | | `stream.buffer_size` | 1000 | Per-client event buffer size | | `stream.rate_limit.enabled` | false | Enable rate limiting | | `stream.rate_limit.requests_per_second` | 10 | Sustained request rate | | `stream.rate_limit.burst_size` | 20 | Maximum burst size | | `stream.rate_limit.cleanup_interval_s` | 60 | Client cleanup interval | ### Configuration File Location Default: `~/.config/logwisp.toml` Override with environment variables: - `LOGWISP_CONFIG_DIR` - Directory containing config file - `LOGWISP_CONFIG_FILE` - Config filename (absolute or relative) Examples: ```bash # Use config from current directory LOGWISP_CONFIG_DIR=. ./logwisp # Use specific config file LOGWISP_CONFIG_FILE=/etc/logwisp/prod.toml ./logwisp # Use custom directory and filename LOGWISP_CONFIG_DIR=/opt/configs LOGWISP_CONFIG_FILE=myapp.toml ./logwisp ``` ### Environment Variables All configuration values can be overridden via environment variables: | Environment Variable | Config Path | Description | |---------------------|-------------|-------------| | `LOGWISP_PORT` | `port` | HTTP listen port | | `LOGWISP_MONITOR_CHECK_INTERVAL_MS` | `monitor.check_interval_ms` | File check interval | | `LOGWISP_MONITOR_TARGETS` | `monitor.targets` | Comma-separated targets | | `LOGWISP_STREAM_BUFFER_SIZE` | `stream.buffer_size` | Client buffer size | | `LOGWISP_STREAM_RATE_LIMIT_ENABLED` | `stream.rate_limit.enabled` | Enable rate limiting | | `LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC` | `stream.rate_limit.requests_per_second` | Rate limit | | `LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE` | `stream.rate_limit.burst_size` | Burst size | | `LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL_S` | `stream.rate_limit.cleanup_interval_s` | Cleanup interval | ### Monitor Targets Format The `LOGWISP_MONITOR_TARGETS` environment variable uses a special format: ``` path:pattern:isfile,path2:pattern2:isfile ``` Examples: ```bash # Monitor directory and specific file LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp # Multiple directories LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp ``` ### Complete Configuration Example ```toml # Port to listen on (default: 8080) port = 8080 [monitor] # How often to check for file changes in milliseconds (default: 100) check_interval_ms = 100 # Paths to monitor # Default: [{"path": "./", "pattern": "*.log", "is_file": false}] # Monitor all .log files in current directory [[monitor.targets]] path = "./" pattern = "*.log" is_file = false # Monitor specific file [[monitor.targets]] path = "/app/logs/app.log" pattern = "" # Ignored for files is_file = true # Monitor with specific pattern [[monitor.targets]] path = "/var/log/nginx" pattern = "access*.log" is_file = false [stream] # Buffer size for each client connection (default: 1000) # Controls how many log entries can be queued per client buffer_size = 1000 [stream.rate_limit] # Enable rate limiting (default: false) enabled = false # Requests per second per client (default: 10) # This is the sustained rate requests_per_second = 10 # Burst size - max requests at once (default: 20) # Allows temporary bursts above the sustained rate burst_size = 20 # How often to clean up old client limiters in seconds (default: 60) # Clients inactive for 2x this duration are removed cleanup_interval_s = 60 ``` ## Color Support LogWisp can pass through ANSI color escape codes from monitored logs to SSE clients using the `-c` flag. ```bash # Enable color pass-through ./logwisp -c # Or via systemd ExecStart=/opt/logwisp/bin/logwisp -c ``` ### How It Works When color mode is enabled (`-c` flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream. ### Example Log with Colors Original log file content: ``` \033[31mERROR\033[0m: Database connection failed \033[33mWARN\033[0m: High memory usage detected \033[32mINFO\033[0m: Service started successfully ``` SSE output with `-c`: ```json { "time": "2024-01-01T12:00:00.123456Z", "source": "app.log", "message": "\u001b[31mERROR\u001b[0m: Database connection failed" } ``` ### Client-Side Handling #### Terminal Clients For terminal-based clients (like curl), the escape codes will render as colors: ```bash # This will show colored output in terminals that support ANSI codes curl -N http://localhost:8080/stream | jq -r '.message' ``` #### Web Clients For web-based clients, you'll need to convert ANSI codes to HTML: ```javascript // Example using ansi-to-html library const AnsiToHtml = require('ansi-to-html'); const convert = new AnsiToHtml(); eventSource.onmessage = (event) => { const data = JSON.parse(event.data); const html = convert.toHtml(data.message); document.getElementById('log').innerHTML += html + '
'; }; ``` #### Custom Processing ```python # Python example with colorama import json import colorama from colorama import init init() # Initialize colorama for Windows support # Process SSE stream for line in stream: if line.startswith('data: '): data = json.loads(line[6:]) # Colorama will handle ANSI codes automatically print(data['message']) ``` ### Common ANSI Color Codes | Code | Color/Style | |------|-------------| | `\033[0m` | Reset | | `\033[1m` | Bold | | `\033[31m` | Red | | `\033[32m` | Green | | `\033[33m` | Yellow | | `\033[34m` | Blue | | `\033[35m` | Magenta | | `\033[36m` | Cyan | ### Limitations 1. **JSON Escaping**: ANSI codes are JSON-escaped in the stream (e.g., `\033` becomes `\u001b`) 2. **Client Support**: The client must support or convert ANSI codes 3. **Performance**: No significant impact, but slightly larger message sizes ### Security Note Color codes are passed through as-is. Ensure monitored logs come from trusted sources to avoid terminal escape sequence attacks. ### Disabling Colors To strip color codes instead of passing them through: - Don't use the `-c` flag - Or set up a preprocessing pipeline: ```bash tail -f colored.log | sed 's/\x1b\[[0-9;]*m//g' > plain.log ``` ## API ### Endpoints - `GET /stream` - Server-Sent Events stream of log entries - `GET /status` - Service status and configuration information ### Log Entry Format ```json { "time": "2024-01-01T12:00:00.123456Z", "source": "app.log", "level": "error", "message": "Something went wrong", "fields": {"key": "value"} } ``` ### SSE Event Types | Event | Description | Data Format | |-------|-------------|-------------| | `connected` | Initial connection | `{"client_id": "123456789"}` | | `data` | Log entry | JSON log entry | | `disconnected` | Client disconnected | `{"reason": "slow_client"}` | | `timeout` | Client timeout | `{"reason": "client_timeout"}` | | `:` | Heartbeat (comment) | ISO timestamp | ### Status Response Format ```json { "service": "LogWisp", "version": "2.0.0", "port": 8080, "color_mode": false, "config": { "monitor": { "check_interval_ms": 100, "targets_count": 2 }, "stream": { "buffer_size": 1000, "rate_limit": { "enabled": true, "requests_per_second": 10, "burst_size": 20 } } }, "streamer": { "active_clients": 5, "buffer_size": 1000, "color_mode": false, "total_dropped": 42 }, "rate_limiter": "Active clients: 3" } ``` ## Usage Examples ### Basic Usage ```bash # Start with defaults ./logwisp # Monitor specific file ./logwisp /var/log/app.log # View logs curl -N http://localhost:8080/stream ``` ### With Environment Variables ```bash # Change port and add rate limiting LOGWISP_PORT=9090 \ LOGWISP_STREAM_RATE_LIMIT_ENABLED=true \ LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \ ./logwisp ``` ### Monitor Multiple Locations ```bash # Via command line ./logwisp /var/log:*.log /app/logs:*.json /tmp/debug.log # Via environment variable LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \ ./logwisp # Or via config file cat > ~/.config/logwisp.toml << EOF [[monitor.targets]] path = "/var/log" pattern = "*.log" is_file = false [[monitor.targets]] path = "/app/logs" pattern = "*.json" is_file = false [[monitor.targets]] path = "/tmp/debug.log" is_file = true EOF ``` ### Production Deployment Example systemd service with environment overrides: ```ini [Unit] Description=LogWisp Log Streaming Service After=network.target [Service] Type=simple User=logwisp ExecStart=/usr/local/bin/logwisp -c Restart=always RestartSec=5 # Environment overrides Environment="LOGWISP_PORT=8080" Environment="LOGWISP_STREAM_BUFFER_SIZE=5000" Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true" Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100" Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false" # Security hardening NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=true ReadOnlyPaths=/ ReadWritePaths=/var/log /opt/app/logs [Install] WantedBy=multi-user.target ``` ## Performance Tuning ### Buffer Size The `stream.buffer_size` setting controls how many log entries can be queued per client: - **Small buffers (100-500)**: Lower memory usage, clients skip entries during bursts - **Default (1000)**: Good balance for most use cases - **Large buffers (5000+)**: Handle burst traffic better, higher memory usage When a client's buffer is full, new messages are skipped for that client until it catches up. The client remains connected and will receive future messages once buffer space is available. ### Check Interval The `monitor.check_interval_ms` setting controls file polling frequency: - **Fast (10-50ms)**: Near real-time updates, higher CPU usage - **Default (100ms)**: Good balance - **Slow (500ms+)**: Lower CPU usage, more latency ### Rate Limiting When to enable rate limiting: - Internet-facing deployments - Shared environments - Protection against misbehaving clients Rate limiting applies only to establishing SSE connections, not to individual messages. Once connected, clients receive all messages (subject to buffer capacity). ## Troubleshooting ### Client Missing Messages If clients miss messages during bursts: 1. Check `total_dropped` and `clients_with_drops` in status endpoint 2. Increase `stream.buffer_size` to handle larger bursts 3. Messages are skipped when buffer is full, but clients stay connected ### High Memory Usage If memory usage is high: 1. Reduce `stream.buffer_size` 2. Enable rate limiting to limit concurrent connections 3. Each client uses `buffer_size * avg_message_size` memory ### Browser Stops Receiving Updates This shouldn't happen with the current implementation. If it does: 1. Check browser developer console for errors 2. Verify no proxy/firewall is timing out the connection 3. Ensure reverse proxy (if used) doesn't buffer SSE responses ## File Rotation Detection LogWisp automatically detects log file rotation using multiple methods: - Inode changes (Linux/Unix) - File size decrease - Modification time reset - Read position beyond file size When rotation is detected, LogWisp: 1. Logs a rotation event 2. Resets read position to beginning 3. Continues streaming from new file ## Security Notes 1. **No built-in authentication** - Use a reverse proxy for auth 2. **No TLS support** - Use a reverse proxy for HTTPS 3. **Path validation** - Only specified paths can be monitored 4. **Directory traversal protection** - Paths containing ".." are rejected 5. **Rate limiting** - Optional but recommended for public deployments 6. **ANSI escape sequences** - Only enable color mode for trusted log sources ## Design Decisions - **Unix philosophy**: Single purpose - stream logs - **SSE over WebSocket**: Simpler, works everywhere, built-in reconnect - **No database**: Stateless operation, instant startup - **Atomic config management**: Using LixenWraith/config package - **Graceful shutdown**: Proper cleanup on SIGINT/SIGTERM - **Platform agnostic**: POSIX-compliant where possible ## Building from Source Requirements: - Go 1.23 or later ```bash git clone https://github.com/yourusername/logwisp cd logwisp go mod download go build -o logwisp ./src/cmd/logwisp ``` ## License BSD-3-Clause