Deutsche Version

Back to overview

Watchdog: Troubleshooting

Show the logs of all containers with make stage-logs or make live-logs. On live, in production mode, the app writes only errors (and deprecations) as JSON to stderr.

Login

Login always fails

• Check: Does the env file still contain APP_ADMIN_PASSWORD_HASH=not-set or an empty value? Did Docker Compose alter the hash?
• Solution: Set a real bcrypt hash. In .env.stage and .env.live every $ must be doubled to $$, in .env.local put the hash in single quotes. Then restart the containers so they re-read the env file.

Checks do not run

A dashboard check stays PENDING

• Check: Are the scheduler and worker containers running? Are client and check active? Is Dashboard (Network) selected as runner?
• Solution: Start the missing containers (docker compose -f compose.live.yml up -d scheduler worker). Activate client and check. In development mode (local, stage) the scheduler writes the line Dispatch due checks tick with evaluated and dispatched to the log file var/log/dev.log every minute. In the local stack it also appears in the container output because the scheduler runs with -vv there.

An agent check stays PENDING

• Check: Is the agent Online under Agents? Is the check's runner Agent (Local) with an agent assigned? Is the client active?
• Solution: See the next section. Run now triggers a run that the agent picks up on its next tick.

The agent shows Offline or Never seen

• Check: The agent container's logs. Typical messages are Dashboard URL is required, Agent token is required and Agent token rejected by dashboard (401).
• Solution: Set WATCHDOG_DASHBOARD_URL and WATCHDOG_AGENT_TOKEN. Replace a rejected token via Regenerate Token. An agent without a token exits immediately and Docker keeps restarting it. Make sure the agent can reach the dashboard URL (firewall, HTTPS).

Agent check results are missing after a dashboard outage

• Check: Did the agent fail to load the check list five times in a row?
• Solution: In that case the agent empties its list and runs checks again only after the next successful fetch (at the latest after the 5-minute configuration interval). No action is needed.

Emails

No alert emails arrive

• Check: Does the client have at least one contact under Alert contacts? Is the worker container running? Are MAILER_DSN and MAILER_FROM correct? Was there a status change at all (see Notifications)?
• Solution: Assign contacts and start the worker. On stage and locally you see the emails in Mailpit. php bin/console messenger:failed:show in the app container lists failed deliveries.

No recovery email after an outage

• Check: Did the check return UNKNOWN between FAIL and OK, for example because of missing configuration or an unreachable Docker socket?
• Solution: Fix the cause of the UNKNOWN. A recovery email exists only for the direct change from WARN or FAIL to OK.

Individual check types

Docker check: Docker socket not available at /var/run/docker.sock

• Check: Is the socket mounted in the container that runs the check?
• Solution: Mount /var/run/docker.sock:/var/run/docker.sock:ro. For the agent the line is commented out in the Compose files.

Docker check: Docker API error: Cannot connect to Docker socket: Permission denied

• Check: Does the container run as a user without access to the socket's group?
• Solution: Add group_add with the socket's group ID (stat -c %g /var/run/docker.sock on the host), as worker and scheduler do.

Disk, file or log check: Path not found or File not found

• Check: Is /:/host/root:ro mounted and HOST_ROOT=/host/root set? Does the path exist on the host?
• Solution: Set the volume and the variable. The docker-compose.agent.yml template does not contain HOST_ROOT.

Process check always reports FAIL

• Check: Does the agent service have pid: host?
• Solution: Add the line and restart the agent.

SSL check reports Connection to <host>:443 failed

• Check: Is the host reachable, is the port right and does the chain come from a publicly trusted CA? Certificates from a private CA fail verification.
• Solution: Set the port in the check settings. For private CAs the form offers the Allow self-signed option, which cannot be saved in the UI though (see SSL Certificate).

DNS check with a custom resolver: exit 127

• Check: Is dig installed in the container?
• Solution: Install dig (package bind-tools on Alpine) in the image or use the system resolver by leaving Custom resolver empty.

HTTP check: URL is rejected on save

• Check: Does the URL have a host name with a dot (top-level domain)?
• Solution: The form does not accept http://localhost or single-label host names. Use a full name or an IP address.

Custom check does not appear in the selection

• Check: Does the class carry #[AutoconfigureTag('watchdog.check')] and implement CheckInterface? Is the image rebuilt?
• Solution: See Writing Custom Checks.

Operations

429 Too many requests for agents

• Check: Do several agents report from the same IP address (NAT, proxy)?
• Solution: The limit is 60 requests per minute and IP. Each agent needs about two to four requests per minute. Make sure TRUSTED_PROXIES is set correctly behind a proxy so that the real IP counts.

Old results disappear

• Check: Does the check have a value under Keep results for (days)?
• Solution: The scheduler deletes older results daily at 03:00. Clear the field if you want to keep all results.

Database file not writable

• Check: Is /app/var/data.db in the volume owned by www-data?
• Solution: make stage-update sets the permissions during an update. Alternatively inside the container: docker compose -f compose.stage.yml exec --user root app chown www-data:www-data /app/var/data.db.