← All field notes
Field note

Clattermark: log alerts that know when to stay quiet

A central syslog collector sees everything, which is exactly the problem: the one line that matters arrives between hundreds that do not. A simple "grep and post" alerter teaches people to mute the channel. Clattermark is a log alert daemon built around filtering, grouping, and suppressing first, and notifying only for what remains.

From lines to incidents

Clattermark follows any number of newline-delimited log files, starting at the end of each one and surviving rotations. Named rules match records with regular expressions, and common RFC 3339 syslog envelopes are parsed into host, process, PID, timestamp, and message fields. When alert processing is saturated it applies backpressure instead of dropping lines.

Matches then pass through exclusions: static patterns, per-process rules, structured-JSON conditions, and dynamic exclusions that can be added at runtime through an authenticated API. Surviving alerts are normalized and deduplicated with cooldowns, and related events are grouped into durable incidents that produce periodic and "it went quiet" notifications rather than one message per line. Deduplication and incident state live in Redis or KeyDB, so several replicas cooperate instead of each reporting the same event.

Optional judgement, safe defaults

Before sending, Clattermark can consult a small HTTP decision API, for example a classifier that knows which recurring messages are harmless. It is an extension point rather than a dependency, and when it is unavailable the default is to fail open: an outage in the filter should never hide an operational event.

Alerts are delivered as native Tintwire cards, with a Mattermost-compatible webhook as fallback. Log content is treated as untrusted input: it can shape an alert's text but is never treated as authentication or identity. The exclusion API requires a token and is rate limited, and health and Prometheus metrics endpoints are meant for loopback or a monitoring network.

Source and documentation: github.com/kilo666mj/clattermark.