logwisp/doc/formatters.md

Formatters

The formatter is the last flow stage. It turns a core.LogEntry into the byte payload that sinks write, applying a sanitizer policy on the way.

[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"
flags            = 0
timestamp_format = ""

One formatter serves the whole pipeline. Sinks receive an identical payload ending in a newline; under raw a message's own newlines stay, so a record can span lines. Apart from the http sink's SSE framing, only the console sink alters what it writes, escaping control characters on a terminal and painting level names (console). When you need two shapes of the same data, run two pipelines, or chain to a node that formats differently.

Omitting [pipelines.flow.format] entirely selects raw.

Options

Each as type and default:

• type (string, raw): raw, txt, or json.
• sanitizer_policy (string, derived from type): raw, txt, json, or shell.
• flags (int64, 0): bitmask override; 0 selects a per-type default.
• timestamp_format (string, time.RFC3339Nano): Go reference layout, e.g. "2006-01-02T15:04:05Z07:00".

Types

raw

Passthrough, the fastest type, for text already in its final form. FlagRaw bypasses formatting and sanitization: the message reaches the sink as the source produced it, control characters included, with no timestamp, level or source prefix added, and a newline appended unless the message ends in one. An entry that also carries fields gets the fields JSON appended verbatim after a single space, so FlagRaw (1) never drops data or re-encodes it. Heartbeats format without it, so in raw their include_stats fields print in Go's map syntax (heartbeat map[beat_count:0 type:heartbeat]), as any entry's fields do under a non-zero flags without 1.

[pipelines.flow.format]
type = "raw"

The closest to verbatim transport is the console source, or the file source with raw = true. The whole line, terminator removed, becomes the message and fields stays empty; raw writes it back with one \n. It is not byte-exact: empty lines are skipped, CRLF becomes LF, and a line over 1 MiB, or one the file source reads while it is still being written, continues in the next entry. The file source's JSON branch splits a line into message and fields, so raw writes <msg> <fields> rather than the original object.

txt

Human-readable lines, <time> <level> <source> <message>, any fields after the message. The message and the fields are each double-quoted when empty or holding whitespace, a non-printable character, a quote, or a shell or = character ($, ;, <, *, ...), with " and \ backslash-escaped inside; the time and source never are:

2026-01-02T15:04:05.123Z ERROR edge-01/app.log "connection refused"

[pipelines.flow.format]
type             = "txt"
sanitizer_policy = "txt"
timestamp_format = "2006-01-02 15:04:05"

json

Structured output, the natural choice for downstream ingestion.

[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"

Output has the shape:

{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}

level is DEBUG, INFO, WARN or ERROR, here and in txt. Only DEBUG, INFO, WARN, WARNING and ERROR, all upper or all lower case, are read as such; any other level, TRACE included, and an entry without one print as INFO. Filters see the level as the source stored it.

The key names and structure come from the lixenwraith/log formatter, not from LogWisp's configuration.

Flags

flags is a bitmask passed to the underlying formatter. Leave it at 0 unless you need to override the defaults.

• 1 Raw: bypass formatting and sanitization entirely.
• 2 ShowTimestamp: emit the timestamp.
• 4 ShowLevel: emit the level.
• 8 StructuredJSON: in json, the fields become a fields object beside message.
• 16 NoTimestamp: suppress the timestamp.
• 32 NoLevel: suppress the level.

With flags = 0 the formatter selects 1 for type = "raw" and 6 (timestamp + level) for every other type. 8 is added automatically whenever an entry's fields is a non-empty JSON object and 1 is not set (it shapes json only); 1 always wins.

txt and json show timestamp and level unless a suppress bit says otherwise (suppress wins over show), so 2 and 4 alone change nothing. Examples: flags = 16 for level only, no timestamp; flags = 32 for timestamp only, no level.

Sanitizer Policies

The sanitizer runs before serialization and neutralizes control characters that would otherwise break framing or reach a terminal.

Each policy, then what to use it with:

• raw: no-op passthrough.
  • type = "raw" where you control the data.
• txt: writes < and non-printable characters as <hex> of their bytes (<b> becomes <3c>b>).
  • File and console sinks.
• json: writes control characters as JSON escapes (\t, \u001b) into the text, which type = "json" quotes again: a reader that decodes the record gets the escape, never the character.
  • type = "json".
• shell: strips shell metacharacters, whitespace, and control characters.
  • Data that will be passed to a command.

Every policy but raw replaces invalid UTF-8 with U+FFFD. type = "json" is valid JSON under any policy, since its own quoting escapes control characters and replaces invalid UTF-8; with policy raw the decoded message keeps its control characters. With type = "raw" and flags = 0 no policy applies (1 bypasses it). When sanitizer_policy is omitted, it is the policy of the same name as type, so the safe pairing is the default.

shell strips dangerous characters but is not sufficient to make a string safe for shell construction. Pass arguments through exec argv instead of building command lines.

To see a policy working, point a pipeline at the random source with special = true and format raw or txt, which mix control and non-ASCII bytes, often invalid UTF-8, into its messages (random).

Node Identity in Output

Entries that arrived over a chain link carry a Node label, written as a syslog-style prefix on the source (trace in json): edge-01/app.log. Entries without one show the bare source. There is no separate node key, which downstream parsers and grep patterns need to know.

Structured Fields

When an entry's fields is a non-empty JSON object and 1 is not set, the formatter parses it and adds 8; without 1, any other fields value ([1,2], {}, "s") is dropped. Fields reach a pipeline from the file source when a tailed line parses as JSON with a fields key, from the tcp_chain and http_chain sources when the upstream entry carried them, and from the heartbeat when include_stats = true. In json output such an entry is {"time":…,"level":…,"trace":…,"message":…,"fields":{…}}; without fields the message sits in "fields":[…]. In txt the fields follow the message in Go's map syntax, keys sorted: "disk full" "map[dev:sda pct:99]".

Choosing a Configuration

By goal:

• Maximum throughput, data already formatted: type = "raw".
• Human reading in a terminal or file: type = "txt", sanitizer_policy = "txt".
• Downstream ingestion (Loki, Elasticsearch, jq): type = "json", sanitizer_policy = "json".
• Compact console output: type = "txt", flags = 16.
• Untrusted log content: never raw; txt or json, whose policies are the default.

Formatting and Chain Links

Chain sinks (tcp_chain, http_chain) do not ship the formatted payload: they send the structured entry in the chain encoding. flow.format on an edge node therefore changes only that node's own local sinks; the shape a human or a downstream system sees is decided on the node that owns the sink they read.

An event that reaches a chain sink without a structured entry is wrapped from its formatted payload and counted in synthesized; a non-zero count means something upstream lost structure.

Performance

Relative cost, cheapest first: raw (passthrough) → txt (line assembly) → json (serialization). Sanitization adds a scan of the message; the raw policy skips it.

The formatter reuses an internal buffer under a mutex, so a pipeline formats its entries one at a time.