Chaining

Chaining links LogWisp nodes together. An edge node forwards its entries to a relay or collector, which can filter, reformat, fan out, or forward them again. Unlike the tcp and http sinks — which emit formatted text for humans and generic clients — chain links carry the structured entry, so downstream nodes can filter and reformat as if the entries were local.

Topology

  edge-01                       relay                        consumers
  ┌───────────────┐             ┌────────────────────┐       ┌──────────────┐
  │ file source   │             │ tcp_chain source   │       │ browser (SSE)│
  │      ↓        │  TCP/TLS    │        ↓           │  ───► │ nc / telnet  │
  │ tcp_chain sink├────────────►│      flow          │       │ archive file │
  └───────────────┘   :15801    │        ↓           │       └──────────────┘
                                │ http sink, tcp sink│
  edge-02                       │ file sink          │
  ┌───────────────┐             │ http_chain sink ───┼──►  upstream collector
  │ file source   │  HTTP/TLS   │                    │
  │      ↓        ├────────────►│ http_chain source  │
  │ http_chain sink│   :15802   └────────────────────┘
  └───────────────┘

Both chain sources can feed a single pipeline (fan-in) whose sinks then fan the merged stream out. test/chain-test.sh builds the two-independent-pipelines variant; test/chain-aggregate-test.sh builds the fan-in variant.

Node Identity

Chained entries carry a node label identifying where they originated.

A chain sink stamps node on any entry that does not already have one. The label comes from the node option, defaulting to os.Hostname().
A chain source either honours the sender's label or overrides it, according to trust_node:

`trust_node`	Behaviour
`true` (default)	Keep the label the sender declared; fall back to the remote address when absent
`false`	Always overwrite with the sender's remote address

Relays preserve node, so a label survives any number of hops and identifies the original producer rather than the last relay.

Formatters render node identity as a syslog-style prefix on the source field: edge-01/app.log. In JSON output the node therefore appears inside the source field, not as a separate top-level key.

trust_node = true means an authenticated peer can claim any node label, including one belonging to another host. On an untrusted network use trust_node = false, or read the mTLS authentication plan, which proposes binding the label to the peer's certificate identity.

Wire Protocol

Protocol version: 1. Both transports carry the same canonical entry encoding, and differ only in how the preamble and framing are expressed.

TCP transport

A persistent connection carrying newline-delimited JSON.

The dialer connects and, under TLS, completes the handshake.
The dialer immediately writes the hello preamble as one JSON line:
```
{"logwisp":1,"node":"edge-01"}
```
The listener reads that line within hello_timeout_ms and rejects the connection if it is malformed or declares a different protocol version.
Every subsequent line is one JSON-encoded LogEntry.

Line size is bounded at 1 MiB. An oversized line is a protocol violation and terminates the connection, because the scanner cannot resynchronize afterwards.

HTTP transport

Batches of NDJSON delivered by POST, with the preamble expressed as headers.

Header	Direction	Meaning
`X-Logwisp-Protocol`	request	Protocol version; must be `1`
`X-Logwisp-Node`	request	Origin node label
`Content-Type`	request	`application/x-ndjson`
`X-Logwisp-Accepted`	response	Number of entries ingested

Responses: 204 on success, 400 for a bad protocol version or a malformed body, 413 when the body cap is exceeded, 405 for a non-POST method.

Entry encoding

{
  "time": "2026-01-02T15:04:05.123456789Z",
  "node": "edge-01",
  "source": "app.log",
  "level": "ERROR",
  "message": "connection refused",
  "fields": {"attempt": 3}
}

node, level, and fields are omitted when empty. A missing time is filled in at ingest.

Delivery Semantics

Transport	Guarantee	Failure behaviour
`tcp_chain`	Per-line, held across reconnects	Retries with exponential backoff plus ±20 % jitter until written or shutdown; back-pressure appears upstream as `total_dropped_by_sink`
`http_chain`	At-least-once per batch	Retries transport errors, `408`, `429`, `5xx`; drops on any other non-2xx (`dropped_batches`)

http_chain batches can be delivered twice when a successful request's response is lost. There is no de-duplication downstream; design your consumers to tolerate it, or use tcp_chain where each line is written once per successful write.

Neither transport persists anything to disk. Entries buffered in memory during an outage are lost if the process exits.

Worked Example

Edge node — tail files, forward over mTLS:

[[pipelines]]
name = "edge"

[[pipelines.plugin_sources]]
id = "app"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "forward"
type = "tcp_chain"
[pipelines.plugin_sinks.config]
host = "relay.internal"
port = 15801
node = "edge-01"
[pipelines.plugin_sinks.config.tls]
enabled   = true
ca_file   = "/etc/logwisp/tls/ca.crt"
cert_file = "/etc/logwisp/tls/edge-01.crt"
key_file  = "/etc/logwisp/tls/edge-01.key"

Relay — ingest, keep errors only, archive and stream:

[[pipelines]]
name = "relay"

[[pipelines.flow.filters]]
type = "include"
patterns = ["ERROR", "FATAL"]

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "ingest"
type = "tcp_chain"
[pipelines.plugin_sources.config]
host = "0.0.0.0"
port = 15801
trust_node = true
[pipelines.plugin_sources.config.tls]
enabled        = true
cert_file      = "/etc/logwisp/tls/relay.crt"
key_file       = "/etc/logwisp/tls/relay.key"
client_auth    = true
client_ca_file = "/etc/logwisp/tls/ca.crt"

[[pipelines.plugin_sinks]]
id = "archive"
type = "file"
[pipelines.plugin_sinks.config]
directory = "/var/log/logwisp"
name = "errors"

[[pipelines.plugin_sinks]]
id = "live"
type = "http"
[pipelines.plugin_sinks.config]
host = "127.0.0.1"
port = 8080

Operational Notes

Formatting is a relay decision. Because chain links carry structured entries, the edge node's flow.format affects only its own local sinks. Set the output shape on the node that owns the human-facing sink.
Filtering early saves bandwidth. A filter on the edge drops entries before they cross the network; a filter on the relay is easier to change centrally.
Rate limits are per pipeline. An edge limit protects the link; a relay limit protects the relay from a noisy edge.
Heartbeats traverse chain links as ordinary structured entries and keep otherwise-idle links and their sessions warm.
Ports used by the bundled test scripts: 15801 tcp_chain ingest, 15802 http_chain ingest, 15803 tcp sink, 15804 http sink.
Use 127.0.0.1, not localhost, when testing locally: all listeners and dialers are IPv4-only, and localhost may resolve to ::1.

7.7 KiB Raw Blame History