logwisp/doc/chaining.md

Chaining

Chaining links LogWisp nodes together. An edge node forwards its entries to a relay or collector, which can filter, reformat, fan out, or forward them again. Unlike the tcp and http sinks — which emit formatted text for humans and generic clients — chain links carry the structured entry, so downstream nodes can filter and reformat as if the entries were local.

Topology

  edge-01                       relay                        consumers
  ┌───────────────┐             ┌────────────────────┐       ┌──────────────┐
  │ file source   │             │ tcp_chain source   │       │ browser (SSE)│
  │      ↓        │  TCP/TLS    │        ↓           │  ───► │ nc / telnet  │
  │ tcp_chain sink├────────────►│      flow          │       │ archive file │
  └───────────────┘   :15801    │        ↓           │       └──────────────┘
                                │ http sink, tcp sink│
  edge-02                       │ file sink          │
  ┌───────────────┐             │ http_chain sink ───┼──►  upstream collector
  │ file source   │  HTTP/TLS   │                    │
  │      ↓        ├────────────►│ http_chain source  │
  │ http_chain sink│   :15802   └────────────────────┘
  └───────────────┘

Both chain sources can feed a single pipeline (fan-in) whose sinks then fan the merged stream out. test/chain-test.sh builds the two-independent-pipelines variant; test/chain-aggregate-test.sh builds the fan-in variant. The edge and aggregator presets build an authenticated TLS link of either transport from a few keys.

Node Identity

Chained entries carry a node label identifying where they originated.

• A chain sink stamps node on any entry that does not already have one. The label comes from the node option, defaulting to os.Hostname().
• A chain source either honours the sender's label or overrides it, according to trust_node:
  • true (default): keep the label the sender declared; fall back to the remote address when absent.
  • false: always overwrite with the sender's remote address.

Relays preserve node, so a label survives any number of hops and identifies the original producer rather than the last relay.

With an auth block the source can instead bind the label to the sender's authenticated identity — its certificate identity under mtls, its username under scram. assert and force take the connection label over from trust_node; only force also overrides the per-entry labels. auth.node_binding:

• none: trust_node governs the connection label and the per-entry node field.
• assert: the connection label must equal the identity, or the peer is rejected; trust_node governs the per-entry node field.
• force (default with an auth block): the identity is the connection label and overwrites every per-entry node field.

Pick force at an ingest boundary you do not trust — it is the only setting where a compromised edge cannot mislabel its entries, including through the per-entry node field. Pick assert on a relay-to-relay hop, where the relay should prove its own identity but the origin labels it forwards must survive. See Security.

Formatters render node identity as a syslog-style prefix on the source field: edge-01/app.log. In JSON output the node therefore appears inside the source field, not as a separate top-level key.

trust_node = true with no auth block means any peer that can connect can claim any node label, including one belonging to another host. On an untrusted network set an auth block (mtls or scram) and keep node_binding = "force"; trust_node = false is the fallback when neither is an option.

Wire Protocol

Protocol version: 1. Both transports carry the same canonical entry encoding, and differ only in how the preamble and framing are expressed.

TCP transport

A persistent connection carrying newline-delimited JSON.

1. The dialer connects and, under TLS, completes the handshake.

2. The dialer immediately writes the hello preamble as one JSON line:

   {"logwisp":1,"node":"edge-01"}
   

3. The listener reads that line within hello_timeout_ms and rejects the connection if it is malformed or declares a different protocol version.

4. Every subsequent line is one JSON-encoded LogEntry.

Line size is bounded at 1 MiB. An oversized line is a protocol violation and terminates the connection, because the scanner cannot resynchronize afterwards.

Under auth.type = "scram" the hello also carries the SCRAM client-first message, and three more lines precede the entries, each one JSON object of at most 4 KiB:

dialer   {"logwisp":1,"node":"edge-01","scram":{"username":"edge-01","client_nonce":"…"}}
listener {"challenge":{"full_nonce":"…","salt":"…","argon_time":3,"argon_memory":65536,"argon_threads":4}}
dialer   {"proof":{"full_nonce":"…","client_proof":"…"},"binding":"<base64 SHA-256 of the server certificate>"}
listener {"final":{"server_signature":"…","username":"edge-01"}}

The whole exchange runs within hello_timeout_ms; the listener sends the final line only after node binding agrees, and a refusal at any step is one {"error":"…"} line before the connection closes. The field is additive, so the protocol version stays 1: a listener without scram answers a hello carrying it with {"error":"authentication not enabled"}, while an older binary ignores it and the dialer gives up after 10 s.

HTTP transport

Batches of NDJSON delivered by POST, with the preamble expressed as headers.

• Request headers
  • X-Logwisp-Protocol: the protocol version; must be 1.
  • X-Logwisp-Node: the origin node label.
  • Content-Type: application/x-ndjson.
  • Authorization: Bearer <token>, under scram.
• Response header
  • X-Logwisp-Accepted: the number of entries ingested.

Responses: 204 on success, 400 for a bad protocol version or a malformed body, 413 when the body cap is exceeded, 405 for a non-POST method, 401 for a missing or expired token, 403 for a refused certificate or node label.

Under scram the sink first logs in with two POST /auth requests carrying the same JSON as the TCP lines: the hello (200 and the challenge), then the proof (200 with the final message, token and expires_in in seconds). A refusal is {"error":"…"}: 401 for a failed proof, 429 when throttled, 503 when the listener is busy, 400 or 413 for a malformed or oversized body.

Entry encoding

{
  "time": "2026-01-02T15:04:05.123456789Z",
  "node": "edge-01",
  "source": "app.log",
  "level": "ERROR",
  "message": "connection refused",
  "fields": {"attempt": 3}
}

node, level, and fields are omitted when empty. A missing time is filled in at ingest.

Delivery Semantics

• tcp_chain: per line, each held across reconnects.
  • On failure it retries with exponential backoff plus ±20 % jitter until the line is written or the sink shuts down.
  • Back-pressure appears upstream as total_dropped_by_sink.
• http_chain: at least once per batch.
  • It retries transport errors, 408, 429, 5xx and, under scram, failed logins and 401.
  • It drops the batch on any other non-2xx, counted in dropped_batches.

http_chain batches can be delivered twice when a successful request's response is lost. There is no de-duplication downstream; design your consumers to tolerate it, or use tcp_chain where each line is written once per successful write.

Neither transport persists anything to disk. Entries buffered in memory during an outage are lost if the process exits.

Worked Example

Edge node — tail files, forward over mTLS:

[[pipelines]]
name = "edge"

[[pipelines.plugin_sources]]
id = "app"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "forward"
type = "tcp_chain"
[pipelines.plugin_sinks.config]
host = "relay.internal"
port = 15801
node = "edge-01"
[pipelines.plugin_sinks.config.tls]
enabled   = true
ca_file   = "/etc/logwisp/tls/ca.crt"
cert_file = "/etc/logwisp/tls/edge-01.crt"
key_file  = "/etc/logwisp/tls/edge-01.key"
[pipelines.plugin_sinks.config.auth]
type  = "mtls"
allow = ["relay.internal"]      # pin the relay, not just its hostname

Relay — ingest, keep errors only, archive and stream:

[[pipelines]]
name = "relay"

[[pipelines.flow.filters]]
type = "include"
patterns = ["ERROR", "FATAL"]

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "ingest"
type = "tcp_chain"
[pipelines.plugin_sources.config]
host = "0.0.0.0"
port = 15801
[pipelines.plugin_sources.config.tls]
enabled        = true
cert_file      = "/etc/logwisp/tls/relay.crt"
key_file       = "/etc/logwisp/tls/relay.key"
client_auth    = true
client_ca_file = "/etc/logwisp/tls/ca.crt"
[pipelines.plugin_sources.config.auth]
type         = "mtls"
allow        = ["edge-01", "edge-02"]
node_binding = "force"          # entries are labelled from the certificate

[[pipelines.plugin_sinks]]
id = "archive"
type = "file"
[pipelines.plugin_sinks.config]
directory = "/var/log/logwisp"
name = "errors"

[[pipelines.plugin_sinks]]
id = "live"
type = "http"
[pipelines.plugin_sinks.config]
host = "127.0.0.1"
port = 8080

Entries arriving on this relay are labelled edge-01 or edge-02 because that is what their certificates say, regardless of the node each edge configured. test/mtls-chain-test.sh builds exactly this shape against a throwaway PKI.

To use passwords instead of client certificates, replace both auth blocks: type = "scram" with username and password_file on the edge, and with a credentials_file on the relay. client_auth and the edge's certificate become optional, and entries are labelled with the username. See Password Authentication.

Operational Notes

• Formatting is a relay decision. Because chain links carry structured entries, the edge node's flow.format affects only its own local sinks. Set the output shape on the node that owns the human-facing sink.
• Filtering early saves bandwidth. A filter on the edge drops entries before they cross the network; a filter on the relay is easier to change centrally.
• Rate limits are per pipeline. An edge limit protects the link; a relay limit protects the relay from a noisy edge.
• Heartbeats traverse chain links as ordinary structured entries and keep otherwise-idle links and their sessions warm.
• Ports used by the bundled test scripts: 15801 tcp_chain ingest, 15802 http_chain ingest, 15803 tcp sink, 15804 http sink.
• Dial the listener's own literal when testing locally: a listener on 127.0.0.1 takes IPv4 only, and localhost may resolve to ::1.