e1.6.0 Documentation update.
This commit is contained in:
384
README.md
384
README.md
@ -1,59 +1,21 @@
|
||||
# Log
|
||||
|
||||
A high-performance, buffered, rotating file logger for Go applications, configured via
|
||||
the [lixenwraith/config](https://github.com/lixenwraith/config) package or simple overrides. Designed for
|
||||
production-grade reliability with features like disk management, log retention, and lock-free asynchronous processing
|
||||
using atomic operations and channels.
|
||||
[](https://golang.org)
|
||||
[](https://opensource.org/licenses/BSD-3-Clause)
|
||||
[](doc/)
|
||||
|
||||
**Note:** This logger requires creating an instance using `NewLogger()` and calling methods on that instance (e.g.,
|
||||
`l.Info(...)`). It does not use package-level logging functions.
|
||||
A high-performance, buffered, rotating file logger for Go applications with built-in disk management, operational monitoring, and framework compatibility adapters.
|
||||
|
||||
## Features
|
||||
## ✨ Key Features
|
||||
|
||||
- **Instance-Based API:** Create logger instances via `NewLogger()` and use methods like `l.Info()`, `l.Warn()`, etc.
|
||||
- **Lock-free Asynchronous Logging:** Non-blocking log operations with minimal application impact. Logs are sent via a
|
||||
buffered channel, processed by a dedicated background goroutine. Uses atomic operations for state management, avoiding
|
||||
mutexes in the hot path.
|
||||
- **External Configuration:** Fully configured using `github.com/lixenwraith/config`, supporting both TOML files and CLI
|
||||
overrides with centralized management. Also supports simple initialization with defaults and string overrides via
|
||||
`InitWithDefaults`.
|
||||
- **Automatic File Rotation:** Seamlessly rotates log files when they reach configurable size limits (`max_size_mb`),
|
||||
generating timestamped filenames.
|
||||
- **Comprehensive Disk Management:**
|
||||
- Monitors total log directory size against configured limits (`max_total_size_mb`)
|
||||
- Enforces minimum free disk space requirements (`min_disk_free_mb`)
|
||||
- Automatically prunes oldest log files to maintain space constraints
|
||||
- Implements recovery behavior when disk space is exhausted
|
||||
- **Adaptive Resource Monitoring:** Dynamically adjusts disk check frequency based on logging volume (
|
||||
`enable_adaptive_interval`, `min_check_interval_ms`, `max_check_interval_ms`), optimizing performance under varying
|
||||
loads.
|
||||
- **Operational Heartbeats:** Multi-level periodic statistics messages (process, disk, system) that bypass level
|
||||
filtering to ensure operational monitoring even with higher log levels.
|
||||
- **Reliable Buffer Management:** Periodic buffer flushing with configurable intervals (`flush_interval_ms`). Detects
|
||||
and reports dropped logs during high-volume scenarios.
|
||||
- **Automated Log Retention:** Time-based log file cleanup with configurable retention periods (`retention_period_hrs`,
|
||||
`retention_check_mins`).
|
||||
- **Structured Logging:** Support for both human-readable text (`txt`) and machine-parseable (`json`) output formats
|
||||
with consistent field handling.
|
||||
- **Comprehensive Log Levels:** Standard severity levels (Debug, Info, Warn, Error) with numeric values compatible with
|
||||
other logging systems.
|
||||
- **Function Call Tracing:** Optional function call stack traces with configurable depth (`trace_depth`) for debugging
|
||||
complex execution flows.
|
||||
- **Clean API Design:** Straightforward logging methods on the logger instance that don't require `context.Context`
|
||||
parameters.
|
||||
- **Graceful Shutdown:** Managed termination with best-effort flushing to minimize log data loss during application
|
||||
shutdown.
|
||||
- 🚀 **Lock-free async logging** with minimal application impact
|
||||
- 📁 **Automatic file rotation** and disk space management
|
||||
- 📊 **Operational heartbeats** for production monitoring
|
||||
- 🔄 **Hot reconfiguration** without data loss
|
||||
- 🎯 **Framework adapters** for gnet v2 and fasthttp
|
||||
- 🛡️ **Production-grade reliability** with graceful shutdown
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
go get github.com/lixenwraith/log
|
||||
go get github.com/lixenwraith/config
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
|
||||
This example shows minimal initialization using defaults with a single override, logging one message, and shutting down.
|
||||
## 🚀 Quick Start
|
||||
|
||||
```go
|
||||
package main
|
||||
@ -63,285 +25,83 @@ import (
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create and initialize logger
|
||||
logger := log.NewLogger()
|
||||
_ = logger.InitWithDefaults("directory=/var/log/myapp")
|
||||
logger.Info("Application starting", "pid", 12345)
|
||||
_ = logger.Shutdown()
|
||||
err := logger.InitWithDefaults("directory=/var/log/myapp")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
defer logger.Shutdown()
|
||||
|
||||
// Start logging
|
||||
logger.Info("Application started", "version", "1.0.0")
|
||||
logger.Debug("Debug information", "user_id", 12345)
|
||||
logger.Warn("Warning message", "threshold", 0.95)
|
||||
logger.Error("Error occurred", "code", 500)
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
## 📦 Installation
|
||||
|
||||
The `log` package can be configured in two ways:
|
||||
```bash
|
||||
go get github.com/lixenwraith/log
|
||||
```
|
||||
|
||||
1. Using a `config.Config` instance with the `Init` method
|
||||
2. Using simple string overrides with the `InitWithDefaults` method
|
||||
For configuration management support:
|
||||
```bash
|
||||
go get github.com/lixenwraith/config
|
||||
```
|
||||
|
||||
### Configuration via Init
|
||||
## 📚 Documentation
|
||||
|
||||
When using `(l *Logger) Init(cfg *config.Config, basePath string)`, the `basePath` argument defines the prefix for all configuration keys in the `config.Config` instance. For example, if `basePath` is set to `"logging"`, the logger will look for configuration keys like `"logging.level"`, `"logging.name"`, etc. This allows embedding logger configuration within a larger application configuration.
|
||||
- **[Getting Started](doc/getting-started.md)** - Installation and basic usage
|
||||
- **[Configuration Guide](doc/configuration.md)** - All configuration options
|
||||
- **[API Reference](doc/api-reference.md)** - Complete API documentation
|
||||
- **[Logging Guide](doc/logging-guide.md)** - Logging methods and best practices
|
||||
- **[Examples](doc/examples.md)** - Sample applications and use cases
|
||||
|
||||
### Advanced Topics
|
||||
|
||||
- **[Disk Management](doc/disk-management.md)** - File rotation and cleanup
|
||||
- **[Heartbeat Monitoring](doc/heartbeat-monitoring.md)** - Operational statistics
|
||||
- **[Performance Guide](doc/performance.md)** - Architecture and optimization
|
||||
- **[Compatibility Adapters](doc/compatibility-adapters.md)** - Framework integrations
|
||||
- **[Troubleshooting](doc/troubleshooting.md)** - Common issues and solutions
|
||||
|
||||
## 🎯 Framework Integration
|
||||
|
||||
The package includes adapters for some popular Go frameworks:
|
||||
|
||||
```go
|
||||
// Initialize config
|
||||
cfg := config.New()
|
||||
configExists, err := cfg.Load("app_config.toml", os.Args[1:])
|
||||
if err != nil {
|
||||
// Handle error
|
||||
}
|
||||
// gnet v2 integration
|
||||
adapter := compat.NewGnetAdapter(logger)
|
||||
gnet.Run(handler, "tcp://127.0.0.1:9000", gnet.WithLogger(adapter))
|
||||
|
||||
// Initialize logger with config and path prefix
|
||||
logger := log.NewLogger()
|
||||
err = logger.Init(cfg, "logging") // Look for keys under "logging."
|
||||
if err != nil {
|
||||
// Handle error
|
||||
}
|
||||
// fasthttp integration
|
||||
adapter := compat.NewFastHTTPAdapter(logger)
|
||||
server := &fasthttp.Server{Logger: adapter}
|
||||
```
|
||||
|
||||
Example TOML with `basePath = "logging"`:
|
||||
```toml
|
||||
[logging] # This matches the basePath
|
||||
level = -4 # Debug
|
||||
directory = "/var/log/my_service"
|
||||
format = "json"
|
||||
See [Compatibility Adapters](doc/compatibility-adapters.md) for detailed integration guides.
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
The logger uses a lock-free, channel-based architecture for high performance:
|
||||
|
||||
```
|
||||
Application → Log Methods → Buffered Channel → Background Processor → File/Console
|
||||
↓ ↓
|
||||
(non-blocking) (rotation, cleanup, monitoring)
|
||||
```
|
||||
|
||||
### Configuration via InitWithDefaults
|
||||
Learn more in the [Performance Guide](doc/performance.md).
|
||||
|
||||
When using `(l *Logger) InitWithDefaults(overrides ...string)`, the configuration keys are provided directly without a prefix, e.g., `"level=-4"`, `"directory=/var/log/my_service"`.
|
||||
## 🤝 Contributing
|
||||
|
||||
```go
|
||||
// Simple initialization with specific overrides
|
||||
logger := log.NewLogger()
|
||||
err := logger.InitWithDefaults("directory=/var/log/app", "level=-4")
|
||||
if err != nil {
|
||||
// Handle error
|
||||
}
|
||||
```
|
||||
Contributions and suggestions are welcome!
|
||||
There is no contribution policy, but if interested, please submit pull requests to the repository.
|
||||
Submit suggestions or issues at [issue tracker](https://github.com/lixenwraith/log/issues).
|
||||
|
||||
### Configuration Parameters
|
||||
|
||||
| Key (`basePath` + Key) | Type | Description | Default Value |
|
||||
|:---------------------------|:----------|:--------------------------------------------------------------------------|:--------------|
|
||||
| `level` | `int64` | Minimum log level (-4=Debug, 0=Info, 4=Warn, 8=Error) | `0` (Info) |
|
||||
| `name` | `string` | Base name for log files | `"log"` |
|
||||
| `directory` | `string` | Directory to store log files | `"./logs"` |
|
||||
| `format` | `string` | Log file format (`"txt"`, `"json"`) | `"txt"` |
|
||||
| `extension` | `string` | Log file extension (without dot) | `"log"` |
|
||||
| `show_timestamp` | `bool` | Show timestamp in log entries | `true` |
|
||||
| `show_level` | `bool` | Show log level in entries | `true` |
|
||||
| `buffer_size` | `int64` | Channel buffer capacity for log records | `1024` |
|
||||
| `max_size_mb` | `int64` | Max size (MB) per log file before rotation | `10` |
|
||||
| `max_total_size_mb` | `int64` | Max total size (MB) of log directory (0=unlimited) | `50` |
|
||||
| `min_disk_free_mb` | `int64` | Min required free disk space (MB) (0=unlimited) | `100` |
|
||||
| `flush_interval_ms` | `int64` | Interval (ms) to force flush buffer to disk via timer | `100` |
|
||||
| `trace_depth` | `int64` | Function call trace depth (0=disabled, 1-10) | `0` |
|
||||
| `retention_period_hrs` | `float64` | Hours to keep log files (0=disabled) | `0.0` |
|
||||
| `retention_check_mins` | `float64` | Minutes between retention checks via timer (if enabled) | `60.0` |
|
||||
| `disk_check_interval_ms` | `int64` | Base interval (ms) for periodic disk space checks via timer | `5000` |
|
||||
| `enable_adaptive_interval` | `bool` | Adjust disk check interval based on load (within min/max bounds) | `true` |
|
||||
| `enable_periodic_sync` | `bool` | Periodic sync with disk based on flush interval | `true` |
|
||||
| `min_check_interval_ms` | `int64` | Minimum interval (ms) for adaptive disk checks | `100` |
|
||||
| `max_check_interval_ms` | `int64` | Maximum interval (ms) for adaptive disk checks | `60000` |
|
||||
| `heartbeat_level` | `int64` | Heartbeat detail level (0=disabled, 1=proc, 2=proc+disk, 3=proc+disk+sys) | `0` |
|
||||
| `heartbeat_interval_s` | `int64` | Interval (s) between heartbeat messages | `60` |
|
||||
| `enable_stdout` | `bool` | Mirror all log output to stdout/stderr | `false` |
|
||||
| `stdout_target` | `string` | Target for console output (`"stdout"` or `"stderr"`) | `"stdout"` |
|
||||
| `disable_file` | `bool` | Disable file output entirely (console-only mode) | `false` |
|
||||
|
||||
## API Reference
|
||||
|
||||
**Note:** All logging and control functions are methods on a `*Logger` instance obtained via `NewLogger()`.
|
||||
|
||||
### Creation
|
||||
|
||||
- **`NewLogger() *Logger`**
|
||||
Creates a new, uninitialized logger instance with default configuration parameters registered internally.
|
||||
|
||||
### Initialization
|
||||
|
||||
- **`(l *Logger) Init(cfg *config.Config, basePath string) error`**
|
||||
Initializes the logger instance `l` using settings from the provided `config.Config` instance under `basePath`. Starts
|
||||
the background processing goroutine.
|
||||
- **`(l *Logger) InitWithDefaults(overrides ...string) error`**
|
||||
Initializes the logger instance `l` using built-in defaults, applying optional overrides provided as "key=value"
|
||||
strings (e.g., `"directory=/tmp/logs"`). Starts the background processing goroutine.
|
||||
|
||||
### Logging Functions
|
||||
|
||||
These methods accept `...any` arguments, typically used as key-value pairs for structured logging. They are called on an
|
||||
initialized `*Logger` instance (e.g., `l.Info(...)`).
|
||||
|
||||
- **`(l *Logger) Debug(args ...any)`**: Logs at Debug level (-4).
|
||||
- **`(l *Logger) Info(args ...any)`**: Logs at Info level (0).
|
||||
- **`(l *Logger) Warn(args ...any)`**: Logs at Warn level (4).
|
||||
- **`(l *Logger) Error(args ...any)`**: Logs at Error level (8).
|
||||
|
||||
### Trace Logging Functions
|
||||
|
||||
Temporarily enable function call tracing for a single log entry on an initialized `*Logger` instance.
|
||||
|
||||
- **`(l *Logger) DebugTrace(depth int, args ...any)`**: Logs Debug with trace.
|
||||
- **`(l *Logger) InfoTrace(depth int, args ...any)`**: Logs Info with trace.
|
||||
- **`(l *Logger) WarnTrace(depth int, args ...any)`**: Logs Warn with trace.
|
||||
- **`(l *Logger) ErrorTrace(depth int, args ...any)`**: Logs Error with trace.
|
||||
(`depth` specifies the number of stack frames, 0-10).
|
||||
|
||||
### Other Logging Variants
|
||||
|
||||
Called on an initialized `*Logger` instance.
|
||||
|
||||
- **`(l *Logger) Log(args ...any)`**: Logs with timestamp only, no level (uses Info internally).
|
||||
- **`(l *Logger) Message(args ...any)`**: Logs raw message without timestamp or level.
|
||||
- **`(l *Logger) LogTrace(depth int, args ...any)`**: Logs with timestamp and trace, no level.
|
||||
|
||||
### Shutdown and Control
|
||||
|
||||
Called on an initialized `*Logger` instance.
|
||||
|
||||
- **`(l *Logger) Shutdown(timeout time.Duration) error`**
|
||||
Gracefully shuts down the logger instance `l`. Signals the processor to stop, waits briefly for pending logs to flush,
|
||||
then closes file handles.
|
||||
|
||||
- **`(l *Logger) Flush(timeout time.Duration) error`**
|
||||
Explicitly triggers a sync of the current log file buffer to disk for instance `l` and waits for completion or
|
||||
timeout.
|
||||
|
||||
### Constants
|
||||
|
||||
- **`LevelDebug (-4)`, `LevelInfo (0)`, `LevelWarn (4)`, `LevelError (8)` (`int64`)**: Standard log level constants.
|
||||
- **`LevelProc (12)`, `LevelDisk (16)`, `LevelSys (20)` (`int64`)**: Heartbeat log level constants. These levels bypass
|
||||
the configured `level` filter.
|
||||
- **`FlagShowTimestamp`, `FlagShowLevel`, `FlagDefault`**: Record flag constants controlling output format.
|
||||
|
||||
### Console Output Configuration
|
||||
|
||||
The logger supports flexible output routing with options to mirror logs to stdout/stderr or disable file output entirely.
|
||||
|
||||
**Console-only logging** (no file output):
|
||||
```go
|
||||
logger := log.NewLogger()
|
||||
err := logger.InitWithDefaults(
|
||||
"enable_stdout=true",
|
||||
"disable_file=true",
|
||||
"level=-4", // Debug level
|
||||
)
|
||||
defer logger.Shutdown()
|
||||
|
||||
logger.Info("This goes only to console")
|
||||
```
|
||||
|
||||
**Dual output** (both file and console):
|
||||
```go
|
||||
gologger := log.NewLogger()
|
||||
err := logger.InitWithDefaults(
|
||||
"directory=/var/log/app",
|
||||
"enable_stdout=true",
|
||||
"stdout_target=stderr", // Use stderr to keep stdout clean
|
||||
)
|
||||
defer logger.Shutdown()
|
||||
|
||||
logger.Info("This goes to both file and stderr")
|
||||
```
|
||||
|
||||
**Dynamic configuration** (toggle at runtime):
|
||||
```go
|
||||
// Start with file-only logging
|
||||
logger := log.NewLogger()
|
||||
cfg := config.New()
|
||||
cfg.Load("app.toml", os.Args[1:])
|
||||
logger.Init(cfg, "logging")
|
||||
|
||||
// Later, enable console output dynamically
|
||||
cfg.Set("logging.enable_stdout", true)
|
||||
logger.Init(cfg, "logging") // Reconfigure
|
||||
|
||||
// Or disable file output
|
||||
cfg.Set("logging.disable_file", true)
|
||||
logger.Init(cfg, "logging") // Now console-only
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- **Lock-Free Hot Path:** Logging methods (`l.Info`, `l.Debug`, etc.) operate without locks, using atomic operations to
|
||||
check logger state and non-blocking channel sends. Only initialization, reconfiguration, and shutdown use a mutex
|
||||
internally.
|
||||
|
||||
- **Channel-Based Architecture:** Log records flow through a buffered channel from producer methods to a single consumer
|
||||
goroutine per logger instance, preventing contention and serializing file I/O operations.
|
||||
|
||||
- **Adaptive Resource Management:**
|
||||
- Disk checks run periodically via timer and reactively when write volume thresholds are crossed.
|
||||
- Check frequency automatically adjusts based on logging rate when `enable_adaptive_interval` is enabled.
|
||||
|
||||
- **Heartbeat Messages:**
|
||||
- Periodic operational statistics that bypass log level filtering.
|
||||
- Three levels of detail (`heartbeat_level`):
|
||||
- Level 1 (PROC): Logger metrics (uptime, processed/dropped logs)
|
||||
- Level 2 (DISK): Adds disk metrics (rotations, deletions, file counts, sizes)
|
||||
- Level 3 (SYS): Adds system metrics (memory usage, goroutine count, GC stats)
|
||||
- Ensures monitoring data is available regardless of the configured `level`.
|
||||
|
||||
- **File Management:**
|
||||
- Log files are rotated when `max_size_mb` is exceeded.
|
||||
- Oldest files are automatically pruned when space limits (`max_total_size_mb`, `min_disk_free_mb`) are approached.
|
||||
- Files older than `retention_period_hrs` are periodically removed.
|
||||
|
||||
- **Recovery Behavior:** When disk issues occur, the logger temporarily pauses new logs and attempts recovery on
|
||||
subsequent operations, logging one disk warning message to prevent error spam.
|
||||
|
||||
- **Graceful Shutdown Flow:**
|
||||
1. Sets atomic flags to prevent new logs on the specific instance.
|
||||
2. Closes the active log channel to signal processor shutdown for that instance.
|
||||
3. Waits briefly for the processor to finish pending records.
|
||||
4. Performs final sync and closes the file handle.
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Non-blocking Design:** The logger is designed to have minimal impact on application performance, with non-blocking
|
||||
log operations and buffered processing.
|
||||
|
||||
- **Memory Efficiency:** Uses a reusable buffer (`serializer`) per instance for serialization, avoiding unnecessary
|
||||
allocations when formatting log entries.
|
||||
|
||||
- **Disk I/O Management:** Batches writes and intelligently schedules disk operations to minimize I/O overhead while
|
||||
maintaining data safety.
|
||||
|
||||
- **Concurrent Safety:** Thread-safe through careful use of atomic operations and channel-based processing, minimizing
|
||||
mutex usage to initialization and shutdown paths only. Multiple `*Logger` instances operate independently.
|
||||
|
||||
## Heartbeat Usage Example
|
||||
|
||||
Heartbeats provide periodic operational statistics even when using higher log levels:
|
||||
|
||||
```go
|
||||
// Enable all heartbeat types with 30-second interval
|
||||
logger := log.NewLogger()
|
||||
err := logger.InitWithDefaults(
|
||||
"level=4", // Only show Warn and above for normal logs
|
||||
"heartbeat_level=3", // Enable all heartbeat types
|
||||
"heartbeat_interval_s=30" // 30-second interval
|
||||
)
|
||||
|
||||
// The PROC, DISK, and SYS heartbeat messages will appear every 30 seconds
|
||||
// even though regular Debug and Info logs are filtered out
|
||||
```
|
||||
|
||||
## Caveats & Limitations
|
||||
|
||||
- **Log Loss Scenarios:**
|
||||
- **Buffer Saturation:** Under extreme load, logs may be dropped if the internal buffer fills faster than records
|
||||
can be processed by the background goroutine. A summary message will be logged once capacity is available again.
|
||||
- **Shutdown Race:** The `Shutdown` function provides a best-effort attempt to process remaining logs, but cannot
|
||||
guarantee all buffered logs will be written if the application terminates abruptly or the timeout is too short.
|
||||
- **Persistent Disk Issues:** If disk space cannot be reclaimed through cleanup, logs will be dropped until the
|
||||
condition is resolved.
|
||||
|
||||
- **Configuration Dependencies:**
|
||||
For full configuration management (TOML file loading, CLI overrides, etc.), the `github.com/lixenwraith/config` package is required when using the `Init` method. For simpler initialization without this external dependency, use `InitWithDefaults`.
|
||||
|
||||
- **Retention Accuracy:** Log retention relies on file modification times, which could potentially be affected by
|
||||
external file system operations.
|
||||
|
||||
## License
|
||||
## 📄 License
|
||||
|
||||
BSD-3-Clause
|
||||
Reference in New Issue
Block a user