Log

A robust, buffered, rotating file logger for Go applications, configured via the LixenWraith/config package. Designed for performance and reliability with features like disk management, log retention, and asynchronous processing using atomic operations and channels.

Features

• Buffered Asynchronous Logging: Logs are sent non-blockingly to a buffered channel, processed by a dedicated background goroutine for minimal application impact. Uses atomic operations for state management, avoiding mutexes in the logging hot path.
• External Configuration: Fully configured using github.com/LixenWraith/config, allowing settings via TOML files and CLI overrides managed centrally.
• Automatic File Rotation: Rotates log files when they reach a configurable size (max_size_mb).
• Disk Space Management:
  • Monitors total log directory size against a limit (max_total_size_mb).
  • Monitors available disk space against a minimum requirement (min_disk_free_mb).
  • Automatically attempts to delete the oldest log files (by modification time) to stay within limits during periodic checks or when writes fail.
  • Temporarily pauses logging if space cannot be freed, logging an error message.
• Adaptive Disk Check Interval: Optionally adjusts the frequency of disk space checks based on logging load (enable_adaptive_interval, disk_check_interval_ms, min_check_interval_ms, max_check_interval_ms) to balance performance and responsiveness.
• Periodic Flushing: Automatically flushes the log buffer to disk at a configured interval (flush_interval_ms) using a timer.
• Log Retention: Automatically deletes log files older than a configured duration (retention_period_hrs), checked periodically via a timer (retention_check_mins). Relies on file modification time.
• Dropped Log Detection: If the internal buffer fills under high load, logs are dropped, and a summary message indicating the number of drops is logged later.
• Structured Logging: Supports both plain text (txt) and json output formats.
• Standard Log Levels: Provides Debug, Info, Warn, Error levels (values match slog).
• Function Call Tracing: Optionally include function call traces in logs with configurable depth (trace_depth) or enable temporarily via *Trace functions.
• Simplified API: Public logging functions (log.Info, log.Debug, etc.) do not require context.Context.
• Graceful Shutdown: log.Shutdown signals the background processor to stop by closing the log channel. It then waits for a brief, fixed duration (best-effort) before closing the file handle. Note: This is a best-effort flush; logs might be lost if flushing takes longer than the internal wait or if the application exits abruptly.

Installation

go get github.com/LixenWraith/log
go get github.com/LixenWraith/config

The config package has its own dependencies which will be fetched automatically.

Basic Usage

package main

import (
	"fmt"
	"os"
	"sync"
	"time"

	"github.com/LixenWraith/config" // External config package
	"github.com/LixenWraith/log"    // This logger package
)

const configFile = "app_config.toml"
const logConfigPath = "logging" // Base path for logger settings in TOML/config

// Example app_config.toml content:
/*
[logging]
  level = 0 # Info Level (0)
  directory = "./app_logs"
  format = "json"
  extension = "log"
  max_size_mb = 50
  flush_interval_ms = 100
  disk_check_interval_ms = 5000 # Example: Check disk every 5s
  enable_adaptive_interval = true
  # Other settings will use defaults registered by log.Init
*/

func main() {
	// 1. Initialize the main config manager
	cfg := config.New()

	// Optional: Create a dummy config file if it doesn't exist
	if _, err := os.Stat(configFile); os.IsNotExist(err) {
		content := fmt.Sprintf("[%s]\n  level = 0\n  directory = \"./app_logs\"\n", logConfigPath)
		os.WriteFile(configFile, []byte(content), 0644)
	}

	// 2. Load configuration (e.g., from file and/or CLI)
	_, err := cfg.Load(configFile, os.Args[1:])
	if err != nil {
		fmt.Fprintf(os.Stderr, "Warning: Failed to load config file '%s': %v. Using defaults.\n", configFile, err)
	}

	// 3. Initialize the logger, passing the config instance and base path.
	// log.Init registers necessary keys (e.g., "logging.level") with cfg.
	err = log.Init(cfg, logConfigPath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Fatal: Failed to initialize logger: %v\n", err)
		os.Exit(1)
	}
	fmt.Println("Logger initialized.")

	// 4. Optionally save the merged config (defaults + file/CLI overrides)
	err = cfg.Save(configFile) // Save back to the file
	if err != nil {
		fmt.Fprintf(os.Stderr, "Warning: Failed to save config: %v\n", err)
	}

	// 5. Use the logger
	log.Info("Application started", "pid", os.Getpid())
	log.Debug("Debugging info", "value", 42) // Might be filtered by level

	// Example concurrent logging
	var wg sync.WaitGroup
	for i := 0; i < 3; i++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			log.Info("Goroutine task started", "goroutine_id", id)
			time.Sleep(time.Duration(id*10) * time.Millisecond)
			log.InfoTrace(1, "Goroutine task finished", "goroutine_id", id)
		}(i)
	}
	wg.Wait()

	// ... application logic ...

	// 6. Shutdown the logger gracefully before exit
	fmt.Println("Shutting down...")
	// Shutdown timeout is used internally for a brief wait, not a hard deadline for flushing.
	shutdownTimeout := 2 * time.Second
	err = log.Shutdown(shutdownTimeout) // Pass timeout (used for internal sleep)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Logger shutdown error: %v\n", err)
	}
	fmt.Println("Shutdown complete.")
}

Configuration

The log package is configured via keys registered with the config.Config instance passed to log.Init. log.Init expects these keys relative to the basePath argument.

Key (basePath + Key)	Type	Description	Default Value (Registered by log.Init)
level	int64	Minimum log level (-4=Debug, 0=Info, 4=Warn, 8=Error)	0 (LevelInfo)
name	string	Base name for log files	"log"
directory	string	Directory to store log files	"./logs"
format	string	Log file format ("txt", "json")	"txt"
extension	string	Log file extension (e.g., "log", "app")	"log"
show_timestamp	bool	Show timestamp in log entries	true
show_level	bool	Show log level in entries	true
buffer_size	int64	Channel buffer capacity for log records	1024
max_size_mb	int64	Max size (MB) per log file before rotation	10
max_total_size_mb	int64	Max total size (MB) of log directory (0=unlimited)	50
min_disk_free_mb	int64	Min required free disk space (MB) (0=unlimited)	100
flush_interval_ms	int64	Interval (ms) to force flush buffer to disk via timer	100
trace_depth	int64	Function call trace depth (0=disabled, 1-10)	0
retention_period_hrs	float64	Hours to keep log files (0=disabled)	0.0
retention_check_mins	float64	Minutes between retention checks via timer (if enabled)	60.0
disk_check_interval_ms	int64	Base interval (ms) for periodic disk space checks via timer	5000
enable_adaptive_interval	bool	Adjust disk check interval based on load (within min/max bounds)	true
enable_periodic_sync	bool	Periodic sync with disk based on flush interval	false
min_check_interval_ms	int64	Minimum interval (ms) for adaptive disk checks	100
max_check_interval_ms	int64	Maximum interval (ms) for adaptive disk checks	60000

Example TOML (config.toml)

# Main application settings
app_name = "My Service"

# Logger settings under the 'logging' base path
[logging]
  level = -4 # Debug
  directory = "/var/log/my_service"
  format = "json"
  extension = "log"
  max_size_mb = 100
  max_total_size_mb = 1024 # 1 GB total
  min_disk_free_mb = 512 # 512 MB free required
  flush_interval_ms = 100
  trace_depth = 2
  retention_period_hrs = 168.0 # 7 days (7 * 24)
  retention_check_mins = 60.0
  disk_check_interval_ms = 10000 # Check disk every 10 seconds
  enable_adaptive_interval = false # Disable adaptive checks

# Other application settings
[database]
  host = "db.example.com"

Your application would then initialize the logger like this:

cfg := config.New()
cfg.Load("config.toml", os.Args[1:]) // Load from file & CLI
log.Init(cfg, "logging")             // Use "logging" as base path
cfg.Save("config.toml")              // Save merged config

API Reference

Initialization

• Init(cfg *config.Config, basePath string) error Initializes or reconfigures the logger using settings from the provided config.Config instance under basePath. Registers required keys with defaults if not present. Handles reconfiguration safely, potentially restarting the background processor goroutine (e.g., if buffer_size changes). Must be called before logging. Thread-safe.
• InitWithDefaults(overrides ...string) error Initializes or reconfigures the logger using built-in defaults, applying optional overrides provided as "key=value" strings. Useful for simple setups without a config file. Thread-safe.

Logging Functions

These functions accept ...any arguments, typically used as key-value pairs for structured logging (e.g., "user_id", 123, "status", "active"). They are non-blocking and read configuration/state using atomic operations.

• Debug(args ...any): Logs at Debug level.
• Info(args ...any): Logs at Info level.
• Warn(args ...any): Logs at Warn level.
• Error(args ...any): Logs at Error level.

Trace Logging Functions

Temporarily enable function call tracing for a single log entry.

• DebugTrace(depth int, args ...any): Logs Debug with trace.
• InfoTrace(depth int, args ...any): Logs Info with trace.
• WarnTrace(depth int, args ...any): Logs Warn with trace.
• ErrorTrace(depth int, args ...any): Logs Error with trace. (depth specifies the number of stack frames, 0-10).

Other Logging Variants

• Log(args ...any): Logs with timestamp, no level (uses Info internally), no trace.
• Message(args ...any): Logs raw message, no timestamp, no level, no trace.
• LogTrace(depth int, args ...any): Logs with timestamp and trace, no level.

Shutdown

• Shutdown(timeout time.Duration) error Attempts to gracefully shut down the logger. Sets atomic flags to prevent new logs, closes the internal log channel to signal the background processor, waits for a brief fixed duration (currently using the flush_interval_ms configuration value, timeout argument is used as a default if the interval is <= 0), and then closes the current log file. Returns nil on success or an error if file operations fail. Note: This provides a best-effort flush; logs might be lost if disk I/O is slow or the application exits too quickly after calling Shutdown.

Constants

• LevelDebug, LevelInfo, LevelWarn, LevelError (int64): Log level constants.

Implementation Details & Behavior

• Asynchronous Processing: Log calls (log.Info, etc.) are non-blocking. They format a logRecord and attempt a non-blocking send to an internal buffered channel (ActiveLogChannel). A single background goroutine (processLogs) reads from this channel, serializes the record (to TXT or JSON using a reusable buffer), and writes it to the current log file.
• Configuration Source: Relies on an initialized github.com/LixenWraith/config.Config instance passed to log.Init or uses internal defaults with InitWithDefaults. It registers expected keys with "log." prefix and retrieves values using the config package's type-specific accessors (Int64, String, Bool, Float64).
• State Management: Uses sync.Mutex (initMu) only to protect initialization and reconfiguration logic. Uses sync/atomic variables extensively for runtime state (IsInitialized, CurrentFile, CurrentSize, DroppedLogs), allowing lock-free reads in logging functions and the processor loop.
• Timers: Uses time.Ticker internally for:
  • Periodic buffer flushing (flush_interval_ms).
  • Periodic log retention checks (retention_check_mins).
  • Periodic and potentially adaptive disk space checks (disk_check_interval_ms, etc.).
• File Rotation: Triggered synchronously within processLogs when writing a record would exceed max_size_mb. The old file is closed, a new one is created with a timestamped name, and the atomic CurrentFile pointer and CurrentSize are updated.
• Disk/Retention Checks:
  • performDiskCheck is called periodically by a timer and reactively if writes fail or a byte threshold is crossed. It checks total size and free space limits. If limits are exceeded and forceCleanup is true (for periodic checks), it calls cleanOldLogs. If checks fail, DiskStatusOK is set to false, causing subsequent logs to be dropped until the condition resolves.
  • cleanOldLogs deletes the oldest files (by modification time, skipping the current file) until enough space is freed or no more files can be deleted.
  • cleanExpiredLogs is called periodically by a timer based on retention_check_mins. It deletes files whose modification time is older than retention_period_hrs.
• Shutdown Process:
  1. Shutdown sets atomic flags (ShutdownCalled, LoggerDisabled) to prevent new logs.
  2. It closes the current ActiveLogChannel (obtained via atomic load).
  3. It performs a fixed short sleep based on the configured flush_interval_ms as a best-effort attempt to allow the processor goroutine time to process remaining items in the channel buffer before the file is closed.
  4. The processLogs goroutine detects the closed channel, performs a final file sync, and exits.
  5. Shutdown performs final Sync and Close on the log file handle after the sleep.

Limitations, Caveats & Failure Modes

• Dependency: Requires github.com/LixenWraith/config for configuration via log.Init.
• Log Loss Scenarios:
  • Buffer Full: If the application generates logs faster than they can be written to disk, ActiveLogChannel fills up. Subsequent log calls will drop messages until space becomes available. A "Logs were dropped" message will be logged later. Increase buffer_size or reduce logging volume.
  • Shutdown: The Shutdown function uses a brief, fixed wait, not a guarantee that all logs are flushed. Logs remaining in the buffer or OS buffers after Shutdown returns might be lost, especially under heavy load or slow disk I/O. Ensure critical logs are flushed before shutdown if necessary (though this logger doesn't provide an explicit flush mechanism).
  • Application Exit: If the application exits abruptly before or during log.Shutdown, buffered logs will likely be lost.
  • Disk Full (Unrecoverable): If performDiskCheck detects low space and cleanOldLogs cannot free enough space (e.g., no old files to delete, permissions issues), DiskStatusOK is set to false. Subsequent logs are dropped until the condition resolves. An error message is logged to stderr once when this state is entered.
• Configuration Errors: log.Init or InitWithDefaults will return an error and fail if configuration values are invalid (e.g., negative max_size_mb, invalid format, bad override string) or if the config.Config instance is nil (for Init). The application must handle these errors.
• Cleanup Race Conditions: Under high load with frequent rotation/cleanup, benign "failed to remove old log file ... no such file or directory" errors might appear in stderr if multiple cleanup attempts target the same file.
• Retention Accuracy: Log retention is based on file modification time. External actions modifying old log files could interfere with accurate retention.
• Reconfiguration: Changing buffer_size restarts the background processor, involving closing the old channel and creating a new one. Logs sent during this brief transition might be dropped. Other configuration changes are applied live where possible via atomic updates.

License

BSD-3-Clause