logwisp/doc/formatters.md

Formatters

The formatter is the last flow stage. It turns a core.LogEntry into the byte payload that sinks write, applying a sanitizer policy on the way.

[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"
flags            = 0
timestamp_format = ""

One formatter serves the whole pipeline. Sinks receive an identical payload; there is no per-sink formatting. When you need two shapes of the same data, run two pipelines, or chain to a node that formats differently.

Omitting [pipelines.flow.format] entirely selects raw.

Options

Option	Type	Default	Description
type	string	raw	raw, txt (alias text), or json
sanitizer_policy	string	derived from type	raw, txt, json, or shell
flags	int64	0	Bitmask override; 0 selects a per-type default
timestamp_format	string	formatter default	Go reference layout, e.g. "2006-01-02T15:04:05Z07:00"

Types

raw

Passthrough. FlagRaw bypasses formatting and sanitization: the message reaches the sink exactly as the source produced it, with no timestamp, level or source prefix added. An entry that also carries fields gets the fields JSON appended verbatim after a single space — raw never drops data and never re-encodes it.

[pipelines.flow.format]
type = "raw"

Fastest option, and the right one when you are relaying text that is already in its final form. Note that it also bypasses sanitization, so control characters in the source data reach your sinks intact.

Byte-exact transport needs a source that does not split the line: the console source, or the file source with raw = true. Both put the whole line — newline included — in the message and leave fields empty. The file source's JSON branch splits a line into message and fields, so raw reassembles it as <msg> <fields> rather than reproducing the original object.

txt

Human-readable line output with a timestamp and level.

[pipelines.flow.format]
type             = "txt"
sanitizer_policy = "txt"
timestamp_format = "2006-01-02 15:04:05"

json

Structured output, the natural choice for downstream ingestion.

[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"

Output has the shape:

{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}

The exact key names and structure come from the lixenwraith/log formatter, not from LogWisp; they are stable for a given dependency version but are not part of LogWisp's own configuration surface.

Flags

flags is a bitmask passed to the underlying formatter. Leave it at 0 unless you need to override the defaults.

Value	Name	Effect
1	Raw	Bypass formatting and sanitization entirely
2	ShowTimestamp	Emit the timestamp
4	ShowLevel	Emit the level
8	StructuredJSON	Render attached fields as a JSON object
16	NoTimestamp	Suppress the timestamp
32	NoLevel	Suppress the level

With flags = 0 the formatter selects 1 for type = "raw" and 6 (timestamp + level) for every other type. 8 is added automatically whenever an entry carries parseable fields and 1 is not set; 1 always wins.

Examples: flags = 4 for level only, no timestamp; flags = 2 for timestamp only, no level.

Sanitizer Policies

The sanitizer runs before serialization and neutralizes control characters that would otherwise break framing or reach a terminal.

Policy	Behaviour	Use with
raw	No-op passthrough	type = "raw" where you control the data
txt	Escapes non-printable characters	File and console sinks
json	Escapes control characters for safe JSON embedding	type = "json", chain links
shell	Strips shell metacharacters, whitespace, and control characters	Data that will be passed to a command

When sanitizer_policy is omitted, the policy is derived from type: json for json, txt for txt/text, and raw for anything else — so the safe pairing is the default.

shell strips dangerous characters but is not sufficient to make a string safe for shell construction. Pass arguments through exec argv instead of building command lines.

To see a policy working, point a pipeline at the random source with special = true, which injects control bytes and multi-byte Unicode into every message.

Node Identity in Output

Entries that arrived over a chain link carry a Node label. The formatter renders it as a syslog-style prefix on the source field:

edge-01/app.log

Entries with no node label show the bare source. Node identity therefore appears inside the source field rather than as a separate output key — worth knowing when writing downstream parsers or grep patterns.

Structured Fields

When an entry carries Fields (raw JSON) and FlagRaw is not set, the formatter parses it and switches to structured rendering by adding the StructuredJSON flag automatically. Fields reach a pipeline in two ways: from the file source when a tailed line parses as JSON with a fields key, and from the heartbeat generator when include_stats = true.

Choosing a Configuration

Goal	Configuration
Maximum throughput, data already formatted	type = "raw"
Human reading in a terminal or file	type = "txt", sanitizer_policy = "txt"
Downstream ingestion (Loki, Elasticsearch, jq)	type = "json", sanitizer_policy = "json"
Compact console output	type = "txt", flags = 4
Untrusted log content	never raw; pick txt or json and set the matching policy

Formatting and Chain Links

Chain sinks (tcp_chain, http_chain) do not ship the formatted payload. They re-serialize the structured entry into the canonical chain encoding, which makes them independent of the local formatter.

The practical consequence: setting flow.format on an edge node changes only that node's own local sinks. The output shape seen by a human or a downstream system is decided on the node that owns the sink they read.

If an event ever reaches a chain sink without a structured entry, the sink wraps the formatted payload into a synthetic entry and counts it in synthesized. A non-zero synthesized count means something upstream lost structure.

Performance

Relative cost, cheapest first: raw (passthrough) → txt (line assembly) → json (serialization). Sanitization adds a scan of the message; the raw policy skips it.

The formatter holds a mutex because the underlying implementation reuses an internal buffer and is not goroutine-safe. It is the only shared serialization point in the hot path, and the reason a single pipeline formats entries one at a time.