Watchdog: Notifications
When Watchdog sends emails: status transitions, recipients, subject and body of alert emails, delivery through the queue and known limits of alerting.
Watchdog: Notifications
Watchdog notifies by email only. There are no other channels such as chat or SMS.
When an email is sent
After every check result Watchdog compares the check's previous status with the new one. The statuses are OK, WARN, FAIL and UNKNOWN. WARN and FAIL count as problematic.
| Previous | New | |
|---|---|---|
OK or UNKNOWN |
WARN or FAIL |
Failure email |
WARN or FAIL |
OK |
Recovery email |
WARN |
FAIL or the other way round |
none |
WARN or FAIL |
same status | none |
WARN or FAIL |
UNKNOWN |
none |
UNKNOWN |
OK |
none |
A newly created check starts with the status UNKNOWN. A first result of OK therefore sends no email, a first result of FAIL or WARN does.
If a check goes from
FAILtoUNKNOWNand then toOK, Watchdog sends no recovery email. If it fails again afterwards, a new failure email follows.
Recipients
The email goes to all contacts assigned to the check's client under Alert contacts. Each contact receives a separate email. If the client has no contacts nothing happens, not even a log entry.
Format
The email is an HTML email without a plain-text alternative. The sender comes from MAILER_FROM.
| Part | Content |
|---|---|
| Subject (failure) | [WATCHDOG] FAIL: <client> - <check type> (<target>), WARN instead of FAIL for warnings |
| Subject (recovery) | [WATCHDOG] RECOVERED: <client> - <check type> (<target>) |
| Header | FAIL: <client> (red) or RECOVERED: <client> (green), the check type below |
| Table | Client, target (depends on the type, see below), Check, Status, HTTP Code (only if present), Message, Time (DD.MM.YYYY HH:MM:SS) |
| Footer | Watchdog Monitoring |
The email contains no link back to the UI. The header reads FAIL on every failure email, even when the status is WARN, and the Status row in the table shows the real value.
The target is one field per check type that names the monitored object:
| Check type | Label | Value |
|---|---|---|
| HTTP, HTTP Content | URL | Label of the selected URL, otherwise the URL |
| SSL Certificate | Hosts | Host names, comma-separated |
| Docker, Docker Exec | Container | Container name |
| Disk Space, File Age, File Size | Path | Path |
| Log File | Log path | Path |
| Process | Process | Process name |
| TCP Port, Redis | Host:Port | host:port |
| Database | DSN | The DSN, without the user name and password from the separate fields |
| DNS | Hostname | Host name |
| Load Average | no target |
Delivery
The email is not sent inside the check itself. The check puts a message into the notifications queue, which the worker container processes and sends through MAILER_DSN. If sending fails, Watchdog retries up to three times (each with a doubled delay), after which the message lands in the failed transport. Nothing consumes that transport, you can inspect it with php bin/console messenger:failed:show.
With several contacts a retry runs for all contacts again. Contacts who already received the email then get it a second time.
Testing locally
In the local stack and on stage Mailpit catches all emails. Open its UI at http://localhost:8128 locally and at 127.0.0.1:8128 on the stage server. MAILER_DSN=smtp://mailpit:1025 stays set for that. Live has no Mailpit container and needs a real SMTP server.
Known limits
| Topic | Behaviour |
|---|---|
| Ongoing outage | There are no reminder emails. Exactly one failure email per outage |
| Escalation | A change from WARN to FAIL sends no second email |
| Agent offline | No alert. The offline indicator is shown in the UI only |
| Check stops running | No alert when results stop arriving, for example with a stalled scheduler |
| Test email | There is no function to send a test email. Use Run now on a check that is guaranteed to fail |