v0.18.2 resume rotated files instead of replay, keep quiet SSE alive, refuse HEAD on strem path

This commit is contained in:
2026-09-16 04:36:47 -04:00
parent 7b782a79ef
commit b8982e4e44
9 changed files with 260 additions and 20 deletions
+12 -5
View File
@@ -148,7 +148,7 @@ This endpoint is scoped to one sink, not to the whole process, and it is
| `dropped_entries` | source | Downstream cannot keep up with the source |
| `total_dropped` | flow | Rate limit or filters are discarding entries (often intended) |
| `total_dropped_by_sink` | pipeline | A sink's input queue is full |
| `dropped_writes` | tcp/http sink | A specific client is too slow |
| `dropped_writes` | tcp/http sink | A client's queue overflowed: either it is too slow, or one burst exceeded `client_buffer_size` |
| `rejected_conns` / `rejected_clients` | tcp/http sink, tcp_chain source | `max_connections` is being hit |
| `tls_handshake_errors` | tcp sink, tcp_chain source | Certificate or version mismatch, or scanning |
| `parse_errors` | chain source | Protocol or version skew upstream |
@@ -185,11 +185,18 @@ the filter stage logs several lines per entry evaluated.
### Buffers
Raise `buffer_size` when `total_dropped_by_sink` is climbing but the sink itself
is healthy — that is a burst-absorption problem. Raise `client_buffer_size` when
`dropped_writes` is climbing for network sinks; that is a slow-consumer or
startup-replay burst problem, and a bigger buffer only buys time. The HTTP
is healthy — that is a burst-absorption problem.
`dropped_writes` on a network sink has two causes that a counter alone does not
separate. A consumer slower than the sustained rate cannot be bought off with
buffer, and drops are the intended outcome. A burst the consumer would have
drained, arriving faster than it reads, is configuration: the sink queues a
whole burst while the client writes one frame at a time, so the part of a burst
above `client_buffer_size` is lost even to a loopback reader.
Where a `rate_limit` bounds the pipeline, its `burst` is that number — keep
`client_buffer_size` at or above it and the second cause disappears. The HTTP
status endpoint reports both queue bounds alongside the counters so an operator
can distinguish configuration from demand.
can tell which one is in play.
```toml
[pipelines.plugin_sinks.config]
+15 -4
View File
@@ -140,7 +140,9 @@ allow = ["viewer-01"]
**Behaviour**
- Only `GET` is routed to either path; anything else gets `405`.
- Only `GET` is routed to either path; anything else gets `405`, `HEAD` on
`stream_path` included — a stream is a body, and a client registered to have
its body discarded never reads and never leaves.
- With an `auth` block, one middleware gates **both** endpoints: an
unauthorized client gets `403` with no body detail, and the rejection is
logged at WARN and counted in `auth_rejected`. The authorized identity is
@@ -150,11 +152,20 @@ allow = ["viewer-01"]
- Payloads are framed per the SSE spec, one `data:` line per newline in the
payload, so multi-line entries stream correctly.
- The server sets no `WriteTimeout` (that would kill long-lived streams);
per-write deadlines come from `write_timeout_ms` via `http.ResponseController`.
per-write deadlines come from `write_timeout_ms` via `http.ResponseController`
and cover the connected frame, every payload, and the idle comment.
- A quiet stream emits an SSE comment every 15 s. It refreshes the client's
session and is how a peer that stopped reading is noticed.
- A client whose send queue is full has that event dropped
(`dropped_writes`); it is not disconnected.
(`dropped_writes`); it is not disconnected. A `dropped_writes` that rises while
no client is behind is a burst larger than `client_buffer_size`, not
backpressure: size the queue at or above whatever burst the pipeline's
`rate_limit` releases at once.
- A client is registered only once its connected frame has flushed, so the
broker never queues into a buffer whose reader has not started.
- Clients whose session has been idle-expired by the session manager are
evicted by the broker.
evicted by the broker. With the idle comment above, that reaches only a peer
that has stopped accepting bytes on a sink configured `write_timeout_ms = 0`.
- On shutdown, connected clients receive
`event: disconnect / data: {"reason":"server_shutdown"}`.
- HTTP/2 is negotiated via ALPN when TLS is enabled; plaintext is HTTP/1.1.
+4
View File
@@ -60,6 +60,10 @@ from = "end"
beyond end-of-file, or an inode change. An inode change where the new file is
already larger than the recorded position is treated as an atomic save, not a
rotation, and the position is preserved.
- A rotation that renames in place — what a size-capped writer does — puts the
same inode back under a name `pattern` also matches. Its watcher resumes at
the position the original reached, so `from = "start"` reads the tail an
unfinished read left behind rather than the whole archive a second time.
- A line is parsed as JSON only when it is an object whose top-level keys are
all drawn from `time`, `level`, `msg` and `fields` — the four an entry can
carry. `time` is read as RFC3339Nano. Any other key, and any non-object line,