v0.34.0 doc sweep and update

doc/formatters.md

@@ -12,8 +12,9 @@ timestamp_format = ""
 ```
 
 One formatter serves the whole pipeline. Sinks receive an identical payload,
-one record ending in exactly one newline; there is no per-sink formatting. When you need two shapes of the same data, run
-two pipelines, or chain to a node that formats differently.
+one record ending in exactly one newline; there is no per-sink formatting.
+When you need two shapes of the same data, run two pipelines, or chain to a
+node that formats differently.
 
 Omitting `[pipelines.flow.format]` entirely selects `raw`.
 
@@ -32,27 +33,25 @@ Each as type and default:
 
 ### raw
 
-Passthrough. `FlagRaw` bypasses formatting and sanitization: the message reaches
-the sink as the source produced it, with no timestamp, level or source prefix
-added, and a newline appended unless the message ends in one. An entry that also
-carries `fields` gets the fields JSON appended verbatim after a single space —
-`raw` never drops data and never re-encodes it.
+Passthrough, the fastest type, for text already in its final form. `FlagRaw`
+bypasses formatting and sanitization: the message reaches the sink as the
+source produced it, control characters included, with no timestamp, level or
+source prefix added, and a newline appended unless the message ends in one.
+An entry that also carries `fields` gets the fields JSON appended verbatim
+after a single space; `raw` never drops data and never re-encodes it.
 
 ```toml
 [pipelines.flow.format]
 type = "raw"
 ```
 
-Fastest option, and the right one when you are relaying text that is already in
-its final form. Note that it also bypasses sanitization, so control characters
-in the source data reach your sinks intact.
-
-Byte-exact transport needs a source that does not split the line: the `console`
-source, or the `file` source with `raw = true`. Both put the whole line, its
-terminator removed, in the message and leave `fields` empty; `raw` writes it
-back as one `\n`. The `file` source's
-JSON branch splits a line into message and fields, so `raw` reassembles it as
-`<msg> <fields>` rather than reproducing the original object.
+The closest to verbatim transport is the `console` source, or the `file`
+source with `raw = true`. The whole line, terminator removed, becomes the
+message and `fields` stays empty; `raw` writes it back with one `\n`. It is not
+byte-exact: blank lines are skipped, CRLF becomes LF, and a line over 1 MiB
+continues in the next entry. The `file` source's JSON branch splits a line
+into message and fields, so `raw` writes `<msg> <fields>` rather than the
+original object.
 
 ### txt
 
@@ -81,9 +80,13 @@ Output has the shape:
 {"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}
 ```
 
-The exact key names and structure come from the `lixenwraith/log` formatter, not
-from LogWisp; they are stable for a given dependency version but are not part of
-LogWisp's own configuration surface.
+`level` is DEBUG, INFO, WARN or ERROR, here and in `txt`. Only `DEBUG`,
+`INFO`, `WARN`, `WARNING` and `ERROR`, all upper or all lower case, are read as
+such; any other level, `TRACE` included, and an entry without one print as
+INFO. Filters see the level as the source stored it.
+
+The key names and structure come from the `lixenwraith/log` formatter, not
+from LogWisp's configuration.
 
 ## Flags
 
@@ -93,13 +96,15 @@ you need to override the defaults.
 - `1` Raw: bypass formatting and sanitization entirely.
 - `2` ShowTimestamp: emit the timestamp.
 - `4` ShowLevel: emit the level.
-- `8` StructuredJSON: render attached fields as a JSON object.
+- `8` StructuredJSON: in `json`, the fields become a `fields` object beside
+  `message`.
 - `16` NoTimestamp: suppress the timestamp.
 - `32` NoLevel: suppress the level.
 
 With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
-(timestamp + level) for every other type. `8` is added automatically whenever an
-entry carries parseable `fields` and `1` is not set; `1` always wins.
+(timestamp + level) for every other type. `8` is added automatically whenever
+an entry carries parseable `fields` and `1` is not set (it shapes `json` only);
+`1` always wins.
 
 `txt` and `json` show timestamp and level unless a suppress bit says otherwise
 (suppress wins over show), so `2` and `4` alone change nothing. Examples:
@@ -115,45 +120,43 @@ Each policy, then what to use it with:
 
 - `raw`: no-op passthrough.
   - `type = "raw"` where you control the data.
-- `txt`: escapes non-printable characters.
+- `txt`: writes `<` and non-printable characters as `<hex>` of their bytes
+  (`<b>` becomes `<3c>b>`).
   - File and console sinks.
 - `json`: escapes control characters for safe JSON embedding.
   - `type = "json"`.
 - `shell`: strips shell metacharacters, whitespace, and control characters.
   - Data that will be passed to a command.
 
-When `sanitizer_policy` is omitted, it is the policy of the same name as
-`type`, so the safe pairing is the default.
+Every policy but `raw` replaces invalid UTF-8 with U+FFFD. With
+`type = "raw"` and `flags = 0` no policy applies (`1` bypasses it). When
+`sanitizer_policy` is omitted, it is the policy of the same name as `type`, so
+the safe pairing is the default.
 
 > `shell` strips dangerous characters but is **not** sufficient to make a string
 > safe for shell construction. Pass arguments through `exec` argv instead of
 > building command lines.
 
 To see a policy working, point a pipeline at the `random` source with
-`special = true`, which injects control bytes and multi-byte Unicode into every
-message.
+`special = true` and `format` `raw` or `txt`, which mix control and non-ASCII
+bytes, often invalid UTF-8, into its messages ([random](sources.md#random)).
 
 ## Node Identity in Output
 
-Entries that arrived over a chain link carry a `Node` label. The formatter
-renders it as a syslog-style prefix on the source field (`trace` in `json`
-output):
-
-```
-edge-01/app.log
-```
-
-Entries with no node label show the bare source. Node identity therefore appears
-*inside* the source field rather than as a separate output key — worth knowing
-when writing downstream parsers or grep patterns.
+Entries that arrived over a chain link carry a `Node` label, written as a
+syslog-style prefix on the source (`trace` in `json`): `edge-01/app.log`.
+Entries without one show the bare source. There is no separate node key, which
+downstream parsers and grep patterns need to know.
 
 ## Structured Fields
 
 When an entry carries `Fields` (raw JSON) and `FlagRaw` is not set, the
-formatter parses it and switches to structured rendering by adding the
-`StructuredJSON` flag automatically. Fields reach a pipeline in two ways: from
-the `file` source when a tailed line parses as JSON with a `fields` key, and
-from the heartbeat generator when `include_stats = true`.
+formatter parses it and adds the `StructuredJSON` flag. Fields reach a pipeline
+from the `file` source when a tailed line parses as JSON with a `fields` key,
+from the `tcp_chain` and `http_chain` sources when the upstream entry carried
+them, and from the heartbeat when `include_stats = true`. In `json` output
+such an entry is `{"time":…,"level":…,"trace":…,"message":…,"fields":{…}}`;
+without fields the message sits in `"fields":[…]`.
 
 ## Choosing a Configuration
 
@@ -165,22 +168,19 @@ By goal:
 - Downstream ingestion (Loki, Elasticsearch, jq): `type = "json"`,
   `sanitizer_policy = "json"`.
 - Compact console output: `type = "txt"`, `flags = 16`.
-- Untrusted log content: never `raw`; pick `txt` or `json` and set the
-  matching policy.
+- Untrusted log content: never `raw`; `txt` or `json`, whose policies are the
+  default.
 
 ## Formatting and Chain Links
 
-Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload.
-They re-serialize the structured entry into the canonical chain encoding, which
-makes them independent of the local formatter.
+Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload:
+they send the structured entry in the chain encoding. `flow.format` on an edge
+node therefore changes only that node's own local sinks; the shape a human or
+a downstream system sees is decided on the node that owns the sink they read.
 
-The practical consequence: setting `flow.format` on an edge node changes only
-that node's own local sinks. The output shape seen by a human or a downstream
-system is decided on the node that owns the sink they read.
-
-If an event ever reaches a chain sink without a structured entry, the sink wraps
-the formatted payload into a synthetic entry and counts it in `synthesized`.
-A non-zero `synthesized` count means something upstream lost structure.
+An event that reaches a chain sink without a structured entry is wrapped from
+its formatted payload and counted in `synthesized`; a non-zero count means
+something upstream lost structure.
 
 ## Performance
 
@@ -188,7 +188,5 @@ Relative cost, cheapest first: `raw` (passthrough) → `txt` (line assembly) →
 `json` (serialization). Sanitization adds a scan of the message; the `raw`
 policy skips it.
 
-The formatter holds a mutex because the underlying implementation reuses an
-internal buffer and is not goroutine-safe. It is the only shared serialization
-point in the hot path, and the reason a single pipeline formats entries one at
-a time.
+The formatter reuses an internal buffer under a mutex, so a pipeline formats
+its entries one at a time.