log/doc/formatting.md

Formatting and Sanitization

The logger package exports standalone formatter and sanitizer packages that can be used independently for text formatting and sanitization needs beyond logging.

Formatter Package

The formatter package provides buffered writing and formatting of log entries with support for txt, json, and raw output formats.

Standalone Usage

import (
    "time"
    "github.com/lixenwraith/log/formatter"
    "github.com/lixenwraith/log/sanitizer"
)

// Create formatter with optional sanitizer
s := sanitizer.New().Policy(sanitizer.PolicyTxt)
f := formatter.New(s)

// Configure formatter
f.Type("json").
  TimestampFormat(time.RFC3339).
  ShowLevel(true).
  ShowTimestamp(true)

// Format a log entry
data := f.Format(
    formatter.FlagDefault,
    time.Now(),
    0,  // Info level
    "", // No trace
    []any{"User logged in", "user_id", 42},
)

Formatter Methods and Concurrency

The formatter provides two classes of methods. You must understand the concurrency contract when using these standalone:

1. Buffered Methods (Single Goroutine Only) These methods reuse an internal buffer to prevent allocations. The returned byte slice is valid ONLY until the next buffered call. You must copy the result (bytes.Clone()) before retention or async hand-off.

• Format(flags int64, timestamp time.Time, level int64, trace string, args []any) []byte
• FormatWithOptions(format string, flags int64, timestamp time.Time, level int64, trace string, args []any) []byte
• FormatValue(v any) []byte
• FormatArgs(args ...any) []byte

2. Append Methods (Thread-Safe) These methods write to a caller-provided destination buffer. Once the formatter is configured, these are safe for concurrent use and are preferred for async sinks.

• AppendFormat(dst []byte, flags int64, timestamp time.Time, level int64, trace string, args []any) []byte
• AppendFormatWithOptions(dst []byte, format string, flags int64, timestamp time.Time, level int64, trace string, args []any) []byte
• AppendValue(dst []byte, v any) []byte
• AppendArgs(dst []byte, args ...any) []byte

Format Flags

Flags use an additive resolution system. No* flags suppress output, Show* flags force output, and if neither is specified, the configured default applies. (No* flags win on conflicts).

const (
    FlagRaw            int64 = 0b0001    // Bypass formatter and sanitizer completely
    FlagShowTimestamp  int64 = 0b0010    // Force include timestamp
    FlagShowLevel      int64 = 0b0100    // Force include level
    FlagStructuredJSON int64 = 0b1000    // Use structured JSON with message/fields
    FlagNoTimestamp    int64 = 0b010000  // Suppress timestamp
    FlagNoLevel        int64 = 0b100000  // Suppress level
    FlagDefault              = FlagShowTimestamp | FlagShowLevel
)

Note: FormatWithOptions and AppendFormatWithOptions bypass configured defaults entirely. Unset Show* bits in these methods mean the feature is off.

Level Constants

// Use formatter.LevelToString() to convert levels
formatter.LevelToString(0)  // "INFO"
formatter.LevelToString(4)  // "WARN"
formatter.LevelToString(8)  // "ERROR"

Sanitizer Package

The sanitizer package provides fluent and composable string sanitization based on configurable rules using bitwise filter flags and transforms.

Standalone Usage

import "github.com/lixenwraith/log/sanitizer"

// Create sanitizer with predefined policy
s := sanitizer.New().Policy(sanitizer.PolicyJSON)
clean := s.Sanitize("hello\nworld")  // "hello\\nworld"

// Custom rules
s = sanitizer.New().
    Rule(sanitizer.FilterControl, sanitizer.TransformHexEncode).
    Rule(sanitizer.FilterShellSpecial, sanitizer.TransformStrip)

clean = s.Sanitize("cmd; echo test")  // "cmd echo test"

Predefined Policies

const (
    PolicyRaw   PolicyPreset = "raw"   // No-op passthrough
    PolicyJSON  PolicyPreset = "json"  // JSON-safe strings
    PolicyTxt   PolicyPreset = "txt"   // Text file safe
    PolicyShell PolicyPreset = "shell" // Shell command safe
)

• PolicyRaw: Pass through all characters unchanged
• PolicyTxt: Hex-encode non-printable characters as <XX>
• PolicyJSON: Escape control characters with JSON-style backslashes
• PolicyShell: Strips shell metacharacters (` $ ; | & > < ( ) # ' " \ * ? [ ] { } ~ !), whitespace, and control characters. Note: Used for defense-in-depth logging, NOT for safely constructing executable shell commands.

Filter Flags

const (
    FilterNonPrintable uint64 = 1 << iota  // Non-printable runes
    FilterControl                          // Control characters
    FilterWhitespace                       // Whitespace characters
    FilterShellSpecial                     // Shell metacharacters
)

Transform Flags

const (
    TransformStrip      uint64 = 1 << iota  // Remove character
    TransformHexEncode                      // Encode as <XX>
    TransformJSONEscape                     // JSON backslash escape
)

Custom Rules

Combine filters and transforms for custom sanitization:

// Remove control characters, hex-encode non-printable
s := sanitizer.New().
    Rule(sanitizer.FilterControl, sanitizer.TransformStrip).
    Rule(sanitizer.FilterNonPrintable, sanitizer.TransformHexEncode)

// Apply multiple policies
s = sanitizer.New().
    Policy(sanitizer.PolicyTxt).
    Rule(sanitizer.FilterWhitespace, sanitizer.TransformJSONEscape)

Serializer

The sanitizer includes a Serializer for type-aware sanitization:

serializer := sanitizer.NewSerializer("json", s)

var buf []byte
serializer.WriteString(&buf, "hello\nworld")  // Adds quotes and escapes
serializer.WriteNumber(&buf, "123.45")        // No quotes for numbers
serializer.WriteBool(&buf, true)              // "true"
serializer.WriteNil(&buf)                     // "null"

JSON Escaping Layers

The sanitizer is a content transform; JSON string escaping is transport encoding applied afterward, unconditionally. Output is valid JSON for any sanitization policy. Multi-byte UTF-8 passes through unescaped.

• format=json + sanitization=raw: recommended; transport escaping only.
• format=json + sanitization=txt: non-printables appear as <XX> inside JSON strings.
• format=json + sanitization=json: redundant; produces visible \\n double escapes. Use raw instead.
• Structured JSON (FlagStructuredJSON) marshals the fields map via encoding/json and bypasses the sanitizer; validity is guaranteed, content-level sanitization is not applied to field values.

PolicyShell Scope

PolicyShell strips metacharacters, whitespace, and control characters as defense-in-depth for logged values. It is not sufficient for constructing shell commands from untrusted input; pass arguments via exec argv.

Hex Marker Integrity

PolicyTxt hex-encodes literal < as <3c>. Every < in sanitized output therefore starts a genuine marker; encoded sequences cannot be spoofed by input containing literal <XX> text.

Format Flags

Flag	Effect
FlagShowTimestamp / FlagShowLevel	Force display on
FlagNoTimestamp / FlagNoLevel	Force display off (wins over Show)
neither	Configured default applies (Format/AppendFormat only)

FormatWithOptions/AppendFormatWithOptions ignore configured defaults: unset Show bits mean off. Unknown format strings fall back to "txt".

Integration with Logger

The logger uses these packages internally but configuration remains simple:

logger := log.NewLogger()

// Configure sanitization policy
logger.ApplyConfigString(
    "format=json",
    "sanitization=json",  // Uses PolicyJSON
)

// Or with custom formatter (advanced)
s := sanitizer.New().Policy(sanitizer.PolicyShell)
customFormatter := formatter.New(s).Type("txt")
// Note: Direct formatter injection requires using lower-level APIs

Common Patterns

Security-Focused Sanitization

// For user input that will be logged
userInput := getUserInput()
s := sanitizer.New().
    Policy(sanitizer.PolicyShell).
    Rule(sanitizer.FilterControl, sanitizer.TransformStrip)

safeLogs := s.Sanitize(userInput)
logger.Info("User input", "data", safeLogs)

Custom Log Formatting

// Format logs for external system
f := formatter.New()
f.Type("json").ShowTimestamp(false).ShowLevel(false)

// Create custom log entry
entry := f.FormatArgs("action", "purchase", "amount", 99.99)
sendToExternalSystem(entry)

Multi-Target Output

// Different sanitization for different outputs
jsonSanitizer := sanitizer.New().Policy(sanitizer.PolicyJSON)
shellSanitizer := sanitizer.New().Policy(sanitizer.PolicyShell)

// For JSON API
jsonFormatter := formatter.New(jsonSanitizer).Type("json")
apiLog := jsonFormatter.Format(...)

// For shell script generation
txtFormatter := formatter.New(shellSanitizer).Type("txt")
scriptLog := txtFormatter.Format(...)

Performance Considerations

• Both packages use pre-allocated buffers for efficiency
• Sanitizer rules are applied in a single pass
• Formatter reuses internal buffers via Reset()
• No regex or reflection in hot paths

Ownership and Thread Safety

• Configuration (Type, ShowLevel, Rule, RuleFunc, Policy, ...) must complete before an instance is shared between goroutines.
• Formatter buffered methods (Format, FormatWithOptions, FormatValue, FormatArgs) reuse an internal buffer. The returned slice is valid only until the next buffered call. Copy (bytes.Clone) before retaining or handing off to async queues. Single goroutine only.
• Formatter append methods (AppendFormat, AppendFormatWithOptions, AppendValue, AppendArgs) write to a caller-provided buffer and are safe for concurrent use. Preferred for async sinks and multi-goroutine callers.
• Sanitizer is immutable after configuration; Sanitize/AppendSanitize are safe for concurrent use. Sanitize returns the input unchanged (allocation-free) when no rule matches.