logwisp/doc/filters.md

Filters

Filters decide which entries continue through a pipeline. They run in the flow, after rate limiting and before formatting.

[[pipelines.flow.filters]]
type     = "include"
logic    = "or"
patterns = ["ERROR", "WARN"]

Options

Option	Type	Default	Description
type	string	include	include (only matches pass) or exclude (matches are dropped)
logic	string	or	or (any pattern matches) or and (every pattern matches)
patterns	[]string	[]	Go RE2 regular expressions

A filter with no patterns passes everything. Invalid patterns fail at startup with the filter index and the offending pattern in the message.

What Gets Matched

Patterns are matched against a single string assembled from the entry:

"<source> <level> <message>"

Empty parts are omitted, so an entry with no detected level matches "<source> <message>". This means a pattern can target the source name or the level as easily as the message body:

Pattern	Matches
"^app\\.log "	Entries whose source is app.log
"ERROR"	Level ERROR, or the word ERROR anywhere in the message

The structured fields payload is not part of the match text.

For entries that arrived over a chain link, the source used here is the bare source — the node/source prefix is applied later, by the formatter — so filtering by originating node requires matching on the message, or filtering on the node that produces the entries.

Filter Types

include

Only matching entries pass. Everything else is dropped.

[[pipelines.flow.filters]]
type     = "include"
patterns = ["ERROR", "WARN", "FATAL"]

exclude

Matching entries are dropped. Everything else passes.

[[pipelines.flow.filters]]
type     = "exclude"
patterns = ["/healthz", "TRACE"]

Logic

or (default)

logic    = "or"
patterns = ["ERROR", "WARN"]
# passes: "ERROR in module"      "WARN: low memory"
# blocks: "INFO: started"

and

logic    = "and"
patterns = ["database", "ERROR"]
# passes: "ERROR: database connection failed"
# blocks: "ERROR: file not found"

With logic = "and" on an exclude filter, an entry is dropped only when it matches every pattern.

Filter Chains

Filters are evaluated in declaration order and an entry must survive all of them. The first filter to reject an entry ends its life; later filters never see it.

# 1. keep only production traffic
[[pipelines.flow.filters]]
type     = "include"
patterns = ["prod-", "production"]

# 2. of that, keep only failures
[[pipelines.flow.filters]]
type     = "include"
patterns = ["ERROR", "EXCEPTION", "FATAL"]

# 3. minus known noise
[[pipelines.flow.filters]]
type     = "exclude"
patterns = ["ECONNRESET", "broken pipe"]

Order matters for cost, not for correctness: put the most selective filter first so later ones evaluate fewer entries.

Pattern Syntax

Go's RE2 syntax. No backreferences and no lookaround — RE2 guarantees linear time, which is exactly what you want in a log hot path.

Need	Pattern
Literal substring	ERROR
Case-insensitive	(?i)error
Whole word	\\berror\\b
Alternation	ERROR|WARN|FATAL
Character class	[0-9]{3}
Anchors	^ERROR, ERROR$
Any characters	.*exception.*

Remember that TOML basic strings process escapes, so a regex backslash needs doubling: "\\berror\\b". TOML literal strings avoid the issue: '\berror\b'.

Anchors apply to the assembled match text, which begins with the source name — so ^ERROR will not match an entry whose source is non-empty. Use \\bERROR\\b instead unless you mean to anchor on the source.

Common Recipes

Severity floor

[[pipelines.flow.filters]]
type     = "include"
patterns = ["ERROR", "FATAL", "CRITICAL"]

Noise reduction

[[pipelines.flow.filters]]
type     = "exclude"
patterns = ["/healthz", "/metrics", "\\bping\\b"]

Secret suppression — see Security; filters are the only redaction mechanism LogWisp currently offers.

[[pipelines.flow.filters]]
type     = "exclude"
patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret", "token"]

Note this drops the whole entry, it does not redact part of it.

Per-application routing — run one pipeline per application, each with its own include filter, rather than trying to route inside one pipeline. Sinks fan out to all sinks in a pipeline; there is no conditional routing.

Statistics

Each filter reports type, logic, pattern_count, total_processed, total_matched, and total_dropped. The chain reports filter_count, total_processed, and total_passed; the pipeline derives total_filtered as the difference.

Performance

Patterns compile once at startup. Every entry that reaches the filter stage is evaluated against every filter until one rejects it, so cost scales with the number of patterns and their complexity. Prefer literal substrings and simple alternations over broad .* wildcards.

Filters log at DEBUG on every entry — pattern text, match results, and the final decision. That is invaluable when a filter is not behaving as expected and very expensive in production; keep logging.level at info or higher on a busy pipeline.