v0.16.1 doc update
This commit is contained in:
+157
-150
@@ -1,185 +1,192 @@
|
||||
# Filters
|
||||
|
||||
LogWisp filters control which log entries pass through the pipeline using pattern matching.
|
||||
|
||||
## Filter Types
|
||||
|
||||
### Include Filter
|
||||
|
||||
Only entries matching patterns pass through.
|
||||
Filters decide which entries continue through a pipeline. They run in the flow,
|
||||
after rate limiting and before formatting.
|
||||
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
logic = "or" # or|and
|
||||
patterns = [
|
||||
"ERROR",
|
||||
"WARN",
|
||||
"CRITICAL"
|
||||
]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
```
|
||||
|
||||
### Exclude Filter
|
||||
|
||||
Entries matching patterns are dropped.
|
||||
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"DEBUG",
|
||||
"TRACE",
|
||||
"health-check"
|
||||
]
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
## Options
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `type` | string | Required | Filter type (include/exclude) |
|
||||
| `logic` | string | "or" | Pattern matching logic (or/and) |
|
||||
| `patterns` | []string | Required | Pattern list |
|
||||
| `type` | string | `include` | `include` (only matches pass) or `exclude` (matches are dropped) |
|
||||
| `logic` | string | `or` | `or` (any pattern matches) or `and` (every pattern matches) |
|
||||
| `patterns` | []string | `[]` | Go RE2 regular expressions |
|
||||
|
||||
A filter with no patterns passes everything. Invalid patterns fail at startup
|
||||
with the filter index and the offending pattern in the message.
|
||||
|
||||
## What Gets Matched
|
||||
|
||||
Patterns are matched against a single string assembled from the entry:
|
||||
|
||||
```
|
||||
"<source> <level> <message>"
|
||||
```
|
||||
|
||||
Empty parts are omitted, so an entry with no detected level matches
|
||||
`"<source> <message>"`. This means a pattern can target the source name or the
|
||||
level as easily as the message body:
|
||||
|
||||
| Pattern | Matches |
|
||||
|---------|---------|
|
||||
| `"^app\\.log "` | Entries whose source is `app.log` |
|
||||
| `"ERROR"` | Level `ERROR`, or the word `ERROR` anywhere in the message |
|
||||
|
||||
The structured `fields` payload is **not** part of the match text.
|
||||
|
||||
For entries that arrived over a chain link, the `source` used here is the bare
|
||||
source — the `node/source` prefix is applied later, by the formatter — so
|
||||
filtering by originating node requires matching on the message, or filtering on
|
||||
the node that produces the entries.
|
||||
|
||||
## Filter Types
|
||||
|
||||
### include
|
||||
|
||||
Only matching entries pass. Everything else is dropped.
|
||||
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN", "FATAL"]
|
||||
```
|
||||
|
||||
### exclude
|
||||
|
||||
Matching entries are dropped. Everything else passes.
|
||||
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["/healthz", "TRACE"]
|
||||
```
|
||||
|
||||
## Logic
|
||||
|
||||
### or (default)
|
||||
|
||||
```toml
|
||||
logic = "or"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
# passes: "ERROR in module" "WARN: low memory"
|
||||
# blocks: "INFO: started"
|
||||
```
|
||||
|
||||
### and
|
||||
|
||||
```toml
|
||||
logic = "and"
|
||||
patterns = ["database", "ERROR"]
|
||||
# passes: "ERROR: database connection failed"
|
||||
# blocks: "ERROR: file not found"
|
||||
```
|
||||
|
||||
With `logic = "and"` on an `exclude` filter, an entry is dropped only when it
|
||||
matches *every* pattern.
|
||||
|
||||
## Filter Chains
|
||||
|
||||
Filters are evaluated in declaration order and an entry must survive all of
|
||||
them. The first filter to reject an entry ends its life; later filters never see
|
||||
it.
|
||||
|
||||
```toml
|
||||
# 1. keep only production traffic
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["prod-", "production"]
|
||||
|
||||
# 2. of that, keep only failures
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "EXCEPTION", "FATAL"]
|
||||
|
||||
# 3. minus known noise
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["ECONNRESET", "broken pipe"]
|
||||
```
|
||||
|
||||
Order matters for cost, not for correctness: put the most selective filter first
|
||||
so later ones evaluate fewer entries.
|
||||
|
||||
## Pattern Syntax
|
||||
|
||||
Patterns support regular expression syntax:
|
||||
Go's RE2 syntax. No backreferences and no lookaround — RE2 guarantees linear
|
||||
time, which is exactly what you want in a log hot path.
|
||||
|
||||
### Basic Patterns
|
||||
- **Literal match**: `"ERROR"` - matches "ERROR" anywhere
|
||||
- **Case-insensitive**: `"(?i)error"` - matches "error", "ERROR", "Error"
|
||||
- **Word boundary**: `"\\berror\\b"` - matches whole word only
|
||||
| Need | Pattern |
|
||||
|------|---------|
|
||||
| Literal substring | `ERROR` |
|
||||
| Case-insensitive | `(?i)error` |
|
||||
| Whole word | `\\berror\\b` |
|
||||
| Alternation | `ERROR\|WARN\|FATAL` |
|
||||
| Character class | `[0-9]{3}` |
|
||||
| Anchors | `^ERROR`, `ERROR$` |
|
||||
| Any characters | `.*exception.*` |
|
||||
|
||||
### Advanced Patterns
|
||||
- **Alternation**: `"ERROR|WARN|FATAL"`
|
||||
- **Character classes**: `"[0-9]{3}"`
|
||||
- **Wildcards**: `".*exception.*"`
|
||||
- **Line anchors**: `"^ERROR"` (start), `"ERROR$"` (end)
|
||||
Remember that TOML basic strings process escapes, so a regex backslash needs
|
||||
doubling: `"\\berror\\b"`. TOML literal strings avoid the issue:
|
||||
`'\berror\b'`.
|
||||
|
||||
### Special Characters
|
||||
Escape special regex characters with backslash:
|
||||
- `.` → `\\.`
|
||||
- `*` → `\\*`
|
||||
- `[` → `\\[`
|
||||
- `(` → `\\(`
|
||||
Anchors apply to the assembled match text, which begins with the source name —
|
||||
so `^ERROR` will not match an entry whose source is non-empty. Use
|
||||
`\\bERROR\\b` instead unless you mean to anchor on the source.
|
||||
|
||||
## Filter Logic
|
||||
## Common Recipes
|
||||
|
||||
### OR Logic (default)
|
||||
Entry passes if ANY pattern matches:
|
||||
```toml
|
||||
logic = "or"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
# Passes: "ERROR in module", "WARN: low memory"
|
||||
# Blocks: "INFO: started"
|
||||
```
|
||||
|
||||
### AND Logic
|
||||
Entry passes only if ALL patterns match:
|
||||
```toml
|
||||
logic = "and"
|
||||
patterns = ["database", "ERROR"]
|
||||
# Passes: "ERROR: database connection failed"
|
||||
# Blocks: "ERROR: file not found"
|
||||
```
|
||||
|
||||
## Filter Chain
|
||||
|
||||
Multiple filters execute sequentially:
|
||||
**Severity floor**
|
||||
|
||||
```toml
|
||||
# First filter: Include errors and warnings
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
|
||||
# Second filter: Exclude test environments
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["test-env", "staging"]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "FATAL", "CRITICAL"]
|
||||
```
|
||||
|
||||
Processing order:
|
||||
1. Entry arrives from source
|
||||
2. Include filter evaluates
|
||||
3. If passed, exclude filter evaluates
|
||||
4. If passed all filters, entry continues to sink
|
||||
**Noise reduction**
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Pattern Compilation
|
||||
- Patterns compile once at startup
|
||||
- Invalid patterns cause startup failure
|
||||
- Complex patterns may impact performance
|
||||
|
||||
### Optimization Tips
|
||||
- Place most selective filters first
|
||||
- Use simple patterns when possible
|
||||
- Combine related patterns with alternation
|
||||
- Avoid excessive wildcards (`.*`)
|
||||
|
||||
## Filter Statistics
|
||||
|
||||
Filters track:
|
||||
- Total entries evaluated
|
||||
- Entries passed
|
||||
- Entries blocked
|
||||
- Processing time per pattern
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### Log Level Filtering
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN", "FATAL", "CRITICAL"]
|
||||
```
|
||||
|
||||
### Application Filtering
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["app1", "app2", "app3"]
|
||||
type = "exclude"
|
||||
patterns = ["/healthz", "/metrics", "\\bping\\b"]
|
||||
```
|
||||
|
||||
### Noise Reduction
|
||||
**Secret suppression** — see [Security](security.md); filters are the only
|
||||
redaction mechanism LogWisp currently offers.
|
||||
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"health-check",
|
||||
"ping",
|
||||
"/metrics",
|
||||
"heartbeat"
|
||||
]
|
||||
type = "exclude"
|
||||
patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret", "token"]
|
||||
```
|
||||
|
||||
### Security Filtering
|
||||
```toml
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"password",
|
||||
"token",
|
||||
"api[_-]key",
|
||||
"secret"
|
||||
]
|
||||
```
|
||||
Note this drops the whole entry, it does not redact part of it.
|
||||
|
||||
### Multi-stage Filtering
|
||||
```toml
|
||||
# Include production logs
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["prod-", "production"]
|
||||
**Per-application routing** — run one pipeline per application, each with its
|
||||
own include filter, rather than trying to route inside one pipeline. Sinks fan
|
||||
out to *all* sinks in a pipeline; there is no conditional routing.
|
||||
|
||||
# Include only errors
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "EXCEPTION", "FATAL"]
|
||||
## Statistics
|
||||
|
||||
# Exclude known issues
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["ECONNRESET", "broken pipe"]
|
||||
```
|
||||
Each filter reports `type`, `logic`, `pattern_count`, `total_processed`,
|
||||
`total_matched`, and `total_dropped`. The chain reports `filter_count`,
|
||||
`total_processed`, and `total_passed`; the pipeline derives
|
||||
`total_filtered` as the difference.
|
||||
|
||||
## Performance
|
||||
|
||||
Patterns compile once at startup. Every entry that reaches the filter stage is
|
||||
evaluated against every filter until one rejects it, so cost scales with the
|
||||
number of patterns and their complexity. Prefer literal substrings and simple
|
||||
alternations over broad `.*` wildcards.
|
||||
|
||||
Filters log at DEBUG on every entry — pattern text, match results, and the
|
||||
final decision. That is invaluable when a filter is not behaving as expected and
|
||||
very expensive in production; keep `logging.level` at `info` or higher on a busy
|
||||
pipeline.
|
||||
|
||||
Reference in New Issue
Block a user