log/README.md

Log

A high-performance, buffered, rotating file logger for Go applications, configured via the LixenWraith/config package. Designed for production-grade reliability with features like disk management, log retention, and lock-free asynchronous processing using atomic operations and channels.

Features

• Lock-free Asynchronous Logging: Non-blocking log operations with minimal application impact. Logs are sent via a buffered channel, processed by a dedicated background goroutine. Uses atomic operations for state management, avoiding mutexes in the hot path.
• External Configuration: Fully configured using github.com/LixenWraith/config, supporting both TOML files and CLI overrides with centralized management.
• Automatic File Rotation: Seamlessly rotates log files when they reach configurable size limits (max_size_mb), generating timestamped filenames.
• Comprehensive Disk Management:
  • Monitors total log directory size against configured limits (max_total_size_mb)
  • Enforces minimum free disk space requirements (min_disk_free_mb)
  • Automatically prunes oldest log files to maintain space constraints
  • Implements recovery behavior when disk space is exhausted
• Adaptive Resource Monitoring: Dynamically adjusts disk check frequency based on logging volume (enable_adaptive_interval, min_check_interval_ms, max_check_interval_ms), optimizing performance under varying loads.
• Reliable Buffer Management: Periodic buffer flushing with configurable intervals (flush_interval_ms). Detects and reports dropped logs during high-volume scenarios.
• Automated Log Retention: Time-based log file cleanup with configurable retention periods (retention_period_hrs, retention_check_mins).
• Structured Logging: Support for both human-readable text (txt) and machine-parseable (json) output formats with consistent field handling.
• Comprehensive Log Levels: Standard severity levels (Debug, Info, Warn, Error) with numeric values compatible with other logging systems.
• Function Call Tracing: Optional function call stack traces with configurable depth (trace_depth) for debugging complex execution flows.
• Clean API Design: Straightforward logging methods that don't require context.Context parameters.
• Graceful Shutdown: Managed termination with best-effort flushing to minimize log data loss during application shutdown.

Installation

go get github.com/LixenWraith/log
go get github.com/LixenWraith/config

Basic Usage

package main

import (
	"fmt"
	"os"
	"sync"
	"time"

	"github.com/LixenWraith/config" // External config package
	"github.com/LixenWraith/log"    // This logger package
)

const configFile = "app_config.toml"
const logConfigPath = "logging" // Base path for logger settings in TOML/config

// Example app_config.toml content:
/*
[logging]
  level = 0 # Info Level (0)
  directory = "./app_logs"
  format = "json"
  extension = "log"
  max_size_mb = 50
  flush_interval_ms = 100
  disk_check_interval_ms = 5000 # Check disk every 5s
  enable_adaptive_interval = true
  # Other settings will use defaults registered by log.Init
*/

func main() {
	// 1. Initialize the main config manager
	cfg := config.New()

	// Optional: Create a dummy config file if it doesn't exist
	if _, err := os.Stat(configFile); os.IsNotExist(err) {
		content := fmt.Sprintf("[%s]\n  level = 0\n  directory = \"./app_logs\"\n", logConfigPath)
		os.WriteFile(configFile, []byte(content), 0644)
	}

	// 2. Load configuration (e.g., from file and/or CLI)
	_, err := cfg.Load(configFile, os.Args[1:])
	if err != nil {
		fmt.Fprintf(os.Stderr, "Warning: Failed to load config file '%s': %v. Using defaults.\n", configFile, err)
	}

	// 3. Initialize the logger, passing the config instance and base path.
	// log.Init registers necessary keys (e.g., "logging.level") with cfg.
	err = log.Init(cfg, logConfigPath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Fatal: Failed to initialize logger: %v\n", err)
		os.Exit(1)
	}
	fmt.Println("Logger initialized.")

	// 4. Optionally save the merged config (defaults + file/CLI overrides)
	err = cfg.Save(configFile) // Save back to the file
	if err != nil {
		fmt.Fprintf(os.Stderr, "Warning: Failed to save config: %v\n", err)
	}

	// 5. Use the logger
	log.Info("Application started", "pid", os.Getpid())
	log.Debug("Debugging info", "value", 42) // Might be filtered by level

	// Example concurrent logging
	var wg sync.WaitGroup
	for i := 0; i < 3; i++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			log.Info("Goroutine task started", "goroutine_id", id)
			time.Sleep(time.Duration(id*10) * time.Millisecond)
			log.InfoTrace(1, "Goroutine task finished", "goroutine_id", id)
		}(i)
	}
	wg.Wait()

	// ... application logic ...

	// 6. Shutdown the logger gracefully before exit
	fmt.Println("Shutting down...")
	// Shutdown timeout is used internally for a brief wait, not a hard deadline for flushing.
	shutdownTimeout := 2 * time.Second
	err = log.Shutdown(shutdownTimeout) // Pass timeout (used for internal sleep)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Logger shutdown error: %v\n", err)
	}
	fmt.Println("Shutdown complete.")
}

Configuration

The log package is configured via keys registered with the config.Config instance passed to log.Init. log.Init expects these keys relative to the basePath argument.

Key (basePath + Key)	Type	Description	Default Value
level	int64	Minimum log level (-4=Debug, 0=Info, 4=Warn, 8=Error)	0 (Info)
name	string	Base name for log files	"log"
directory	string	Directory to store log files	"./logs"
format	string	Log file format ("txt", "json")	"txt"
extension	string	Log file extension (without dot)	"log"
show_timestamp	bool	Show timestamp in log entries	true
show_level	bool	Show log level in entries	true
buffer_size	int64	Channel buffer capacity for log records	1024
max_size_mb	int64	Max size (MB) per log file before rotation	10
max_total_size_mb	int64	Max total size (MB) of log directory (0=unlimited)	50
min_disk_free_mb	int64	Min required free disk space (MB) (0=unlimited)	100
flush_interval_ms	int64	Interval (ms) to force flush buffer to disk via timer	100
trace_depth	int64	Function call trace depth (0=disabled, 1-10)	0
retention_period_hrs	float64	Hours to keep log files (0=disabled)	0.0
retention_check_mins	float64	Minutes between retention checks via timer (if enabled)	60.0
disk_check_interval_ms	int64	Base interval (ms) for periodic disk space checks via timer	5000
enable_adaptive_interval	bool	Adjust disk check interval based on load (within min/max bounds)	true
enable_periodic_sync	bool	Periodic sync with disk based on flush interval	false
min_check_interval_ms	int64	Minimum interval (ms) for adaptive disk checks	100
max_check_interval_ms	int64	Maximum interval (ms) for adaptive disk checks	60000

Example TOML Configuration (app_config.toml)

# Main application settings
app_name = "My Service"

# Logger settings under the 'logging' base path
[logging]
  level = -4 # Debug
  directory = "/var/log/my_service"
  format = "json"
  extension = "log"
  max_size_mb = 100
  max_total_size_mb = 1024 # 1 GB total
  min_disk_free_mb = 512 # 512 MB free required
  flush_interval_ms = 100
  trace_depth = 2
  retention_period_hrs = 168.0 # 7 days (7 * 24)
  retention_check_mins = 60.0
  disk_check_interval_ms = 10000 # Check disk every 10 seconds
  enable_adaptive_interval = false # Disable adaptive checks

# Other application settings
[database]
  host = "db.example.com"

API Reference

Initialization

• Init(cfg *config.Config, basePath string) error Initializes or reconfigures the logger using settings from the provided config.Config instance under basePath. Registers required keys with defaults if not present. Thread-safe.
• InitWithDefaults(overrides ...string) error Initializes the logger using built-in defaults, applying optional overrides provided as "key=value" strings. Thread-safe.

Logging Functions

These methods accept ...any arguments, typically used as key-value pairs for structured logging (e.g., "user_id", 123, "status", "active"). All logging functions are non-blocking and use atomic operations for state checks.

• Debug(args ...any): Logs at Debug level (-4).
• Info(args ...any): Logs at Info level (0).
• Warn(args ...any): Logs at Warn level (4).
• Error(args ...any): Logs at Error level (8).

Trace Logging Functions

Temporarily enable function call tracing for a single log entry, regardless of the configured trace_depth.

• DebugTrace(depth int, args ...any): Logs Debug with trace.
• InfoTrace(depth int, args ...any): Logs Info with trace.
• WarnTrace(depth int, args ...any): Logs Warn with trace.
• ErrorTrace(depth int, args ...any): Logs Error with trace. (depth specifies the number of stack frames, 0-10).

Other Logging Variants

• Log(args ...any): Logs with timestamp only, no level (uses Info internally).
• Message(args ...any): Logs raw message without timestamp or level.
• LogTrace(depth int, args ...any): Logs with timestamp and trace, no level.

Shutdown and Control

• Shutdown(timeout time.Duration) error Gracefully shuts down the logger. Signals the processor to stop, waits briefly for pending logs to flush, then closes file handles. Returns error details if closing operations fail.

• Flush(timeout time.Duration) error Explicitly triggers a sync of the current log file buffer to disk and waits for completion or timeout.

Constants

• LevelDebug (-4), LevelInfo (0), LevelWarn (4), LevelError (8) (int64): Log level constants.
• FlagShowTimestamp, FlagShowLevel, FlagDefault: Record flag constants controlling output format.

Implementation Details

• Lock-Free Hot Path: Log methods (Info, Debug, etc.) operate without locks, using atomic operations to check logger state and non-blocking channel sends. Only initialization, reconfiguration, and shutdown use a mutex.

• Channel-Based Architecture: Log records flow through a buffered channel from producer methods to a single consumer goroutine, preventing contention and serializing file I/O operations.

• Adaptive Resource Management:

  • Disk checks run periodically via timer and reactively when write volume thresholds are crossed
  • Check frequency automatically adjusts based on logging rate when enable_adaptive_interval is enabled
  • Intelligently backs off during low activity and increases responsiveness during high volume

• File Management:

  • Log files are rotated when max_size_mb is exceeded, with new files named using timestamps
  • Oldest files (by modification time) are automatically pruned when space limits are approached
  • Files older than retention_period_hrs are periodically removed

• Recovery Behavior: When disk issues occur, the logger temporarily pauses new logs and attempts recovery on subsequent operations, logging one disk warning message to prevent error spam.

• Graceful Shutdown Flow:

  1. Sets atomic flags to prevent new logs
  2. Closes the active log channel to signal processor shutdown
  3. Waits briefly for processor to finish pending records
  4. Performs final sync and closes the file handle

Performance Considerations

• Non-blocking Design: The logger is designed to have minimal impact on application performance, with non-blocking log operations and buffered processing.

• Memory Efficiency: Uses a reusable buffer for serialization, avoiding unnecessary allocations when formatting log entries.

• Disk I/O Management: Batches writes and intelligently schedules disk operations to minimize I/O overhead while maintaining data safety.

• Concurrent Safety: Thread-safe through careful use of atomic operations, minimizing mutex usage to initialization and shutdown paths only.

Caveats & Limitations

• Log Loss Scenarios:

  • Buffer Saturation: Under extreme load, logs may be dropped if the internal buffer fills faster than records can be processed. A summary message will be logged once capacity is available again.
  • Shutdown Race: The Shutdown function provides a best-effort attempt to process remaining logs, but cannot guarantee all buffered logs will be written if the application terminates quickly.
  • Persistent Disk Issues: If disk space cannot be reclaimed through cleanup, logs will be dropped until the condition is resolved.

• Configuration Dependencies: Requires the github.com/LixenWraith/config package for advanced configuration management.

• Retention Accuracy: Log retention relies on file modification times, which could be affected by external file system operations.

• Reconfiguration Impact: Changing buffer size during runtime requires restarting the background processor, which may cause a brief period where logs could be dropped.

License

BSD-3-Clause