Monitoring and alerts

What Yggdrasil Panel measures, what it keeps, and what it will wake you up about. Read this before you set your first alarm threshold, and again when you want to know why a number looks the way it does.

Live CPU and memory

A running server’s page shows current CPU and memory read straight from Docker (GET /api/servers/{id}/stats): CPU percent, memory used, and the container’s memory limit.

The CPU number needs one sentence of explanation, because it is not the number docker stats prints. Yggdrasil reports a server’s CPU as a share of the whole host, from 0 to 100, and caps it at 100. Docker’s usual formula multiplies by the core count, so a container burning 1.2 of your 8 cores reads 120%. On a per-server gauge that looks alarming and means nothing. Yggdrasil divides the container’s CPU delta by the system-wide delta, which already spans every core, so the same container reads 15% — fifteen percent of the machine. That is the same scale as the host CPU figure on the dashboard, so the two can be compared directly, and a server’s CPU can never exceed 100.

The consequence for thresholds: a CPU alarm at 80 means “this one server is using 80% of the entire host”, not “80% of one core”. Set them accordingly.

Metrics history

A sampler runs every 5 minutes. For each running server it records one row — CPU percent, memory in MB, and player count — into the metrics table. Player count comes from the rune’s query protocol, polled on 127.0.0.1; a game with no query, or one that doesn’t answer, records -1.

Samples are pruned to a rolling 7-day window, swept roughly hourly.

GET /api/servers/{id}/metrics returns the samples for the last N hours — default 24, maximum 168 (7 days). The server page’s History section draws the three series as sparklines with 24h / 3d / 7d range buttons. Every sample also feeds the resource alarms below.

Quiet hours

GET /api/servers/{id}/quiet-hours mines the sampled player counts to answer “when is nobody playing?”. It takes the last 14 days of samples, ignores the -1 no-data rows, buckets the rest by server-local hour (0–23), and returns the average players per hour along with the quietest hour and its average.

The server page shows the result as a hint next to auto-restart — “Quietest around 05:00 (avg 0.4 players, last 14 days)” — with a Start there link that anchors auto-restart to that hour. So the recommendation is actionable, not just informative: you schedule a disruptive job on evidence rather than a guess. See auto-restart.

The bucketing uses the panel host’s local time, and so does the scheduler, so the hour the hint names is the hour a restart anchored there actually fires.

When there is no usable data — a brand-new server, or a game with no query protocol — the response says has_data: false and the UI stays silent rather than recommending an hour it invented.

Auto-restart itself is a per-server “restart every N hours” toggle backed by an ordinary managed schedule row, so the restart runs through the same machinery as any other scheduled job, including the rune’s countdown warnings and an optional backup first. See Backups and schedules.

Resource alarms

Each server has three optional thresholds, all off until you set them (zero or blank disables):

Threshold	Field	Unit	Measured
CPU	`cpu_alarm_pct`	percent of the whole host	every metrics sample (5 min)
Memory	`mem_alarm_mb`	megabytes	every metrics sample (5 min)
Disk	`disk_alarm_mb`	megabytes of the data directory	hourly

All three are edge-triggered. Crossing into breach fires exactly one notification. Nothing repeats while the alarm stays up. When the metric recovers, you get one all-clear. There is no reminder loop and no alert spam, which means an alarm you dismiss is an alarm you will not hear about again until it clears and re-fires.

CPU and memory require a sustained breach: the value has to sit at or above the threshold for 2 consecutive samples before the alarm fires. With a 5-minute sampler, that is roughly 10 minutes — long enough to ignore a one-off spike during world generation, short enough to catch a real leak. A single sample below the threshold resets the streak. The notification text spells the arithmetic out, for example ⚠️ Skyrealm CPU high: 91% (≥ 85% for ~10 min).

Disk is different in two ways. It’s checked on its own hourly timer, because measuring means walking the whole data directory, and the first sweep waits 2 minutes after the panel starts so boot isn’t competing with a tree walk. It also fires on a single measurement at or above the threshold — no sustained-breach requirement, since a directory doesn’t spike.

What clears when a server stops

CPU and memory alarm state is dropped when a server stops. A fresh run starts from zero rather than staying latched on a value from the last one.

Disk alarm state persists. This is deliberate and it is the behaviour that surprises people: a stopped server’s data directory still occupies exactly as much disk as it did while running. Worlds, in-place backups and logs don’t shrink because you pressed Stop. So stopping a server does not silence its disk alarm and does not produce an all-clear — you get the all-clear when the directory actually falls back under the threshold, which means deleting something.

Alarms go out through your configured notification channels — see Notifications.

Watchdog

The watchdog is per-server auto-heal, off by default, and available only on runes that declare a query protocol (there’s nothing to health-check otherwise, so the UI gates the toggle).

The reconciler’s 20-second tick health-checks every watchdog-enabled running server by speaking its query protocol on 127.0.0.1 with a 3-second timeout. This catches a specific and nasty failure: the container is up, Docker is happy, and the game inside is hung. A container-liveness check sees nothing wrong; the watchdog sees no answer.

3 consecutive failed checks trigger a heal. Yggdrasil recreates and restarts the server and notifies you before and after.
After a heal, a 4-minute cooldown suspends checks so the server has room to boot without being restarted again mid-start.
Any successful check resets the failure streak.

If a server needs 5 heals within 30 minutes, it is crash-looping and restarting it is not helping. The watchdog quarantines it: auto-heal pauses, you get one alert saying so, and Yggdrasil leaves it alone. Starting the server manually clears the quarantine and gives it a fresh chance. Turning the toggle off, or stopping or deleting the server, also clears its state.

Start-failure detection

The watchdog only heals servers the panel already believes are running. A container that comes up and whose game process crashes straight back out is a different failure, and it has its own handling.

When a start attempt fails — the container exits before it ever signals readiness — Yggdrasil grabs the container’s log tail immediately, while the crashed container still exists, then retries. There are 3 total attempts (the original plus two automatic retries), each after a 15-second backoff so a slow image pull or a dependency still coming up can settle. Retries are logged, not notified, which keeps a transient blip quiet.

After the third failure Yggdrasil gives up, leaves the server stopped, and sends one actionable alert with the last 40 log lines attached (trimmed to 1500 characters so the notification stays sendable). That is the whole point of the feature: the alert tells you why, not just that.

The failure count resets the moment a start succeeds, or whenever you start or stop the server yourself — a fresh intent gets a fresh retry budget. If you take over during a backoff, the retry stands down.

Two adjacent cases are covered by the same machinery:

Slow start. A server that is still starting after 5 minutes without crashing gets one heads-up with its latest log tail, then Yggdrasil keeps waiting. The readiness window is 10 minutes; a server still up at the end of it is marked running regardless.
Stalled start across a panel restart. If the panel restarts while a server is starting, Yggdrasil re-attaches to it. A container that has been up well past the readiness window and never signalled ready — the game process died but its wrapper stayed alive — is marked stopped, and you get an alert with the log tail.

Kvasir Watchers

A watcher is a log rule: if this pattern matches at least N lines within the last W seconds of a server’s log, act. It reads the container’s own stdout/stderr, so the same mechanism covers a game server’s crash spam, a WordPress failed-login burst, a database’s error storm or an HTTP 5xx spike. Watchers live under Settings → Kvasir Watchers (admin only).

Every running server is scanned about every 30 seconds; a watcher fires at most once per 10 minutes so a sustained condition doesn’t flood you.
A watcher is scoped to one server or to every server.
Action Notify sends the matched lines to your notification channels. Action Notify + Kvasir also hands them to the AI to explain what’s happening and propose a fix — it needs a configured provider and proactive monitoring on (see the Kvasir guide).

You rarely have to start from nothing:

Runes ship defaults. A rune can declare watchers: — the author’s knowledge of what its log looks like when things go wrong. They’re seeded per server at create and (re)install as ordinary rules (marked ᚱ rune in the list) that you can edit, disable or delete. Your changes stick across reinstalls; deleting one and reinstalling restores the default.
Kvasir can suggest rules. Pick a server under Settings → Kvasir Watchers and press Suggest: the AI reads the server’s app type and a sample of its recent log and proposes up to five rules, each with its reasoning. Every proposal is validated (the pattern must be a working regex, bounds are clamped) and nothing is created until you add it.

Patterns are Go/RE2 regular expressions matched per line. Prefer patterns anchored in how your log actually formats trouble — the suggestion flow exists precisely because a generic pattern either misses or matches routine lines.

The alert policy

A watcher tripping is not the same as something being worth your attention. A site on the public internet is scanned continuously — requests for /.env, /.git/config, /wp-admin/setup-config.php arrive all day and are answered with a 404, which is already the correct response. If every one of those pages you, the alerts become noise and you switch detection off, which is worse than never having had it.

So every detection from a watcher or a traffic-spike anomaly passes through a policy before anything is sent. It sorts each situation into one of two classes:

Class	What it means	What happens
`routine`	Vulnerability scanning the server already refused, or a handful of hits spread thinly across many sources	Recorded, summarised in the daily digest, never sent as an alert
`incident`	One source hammering, or a flood large enough to matter whatever its shape	Sent to your notification channels

Four rules do the work, in this order:

Refusal comes first. A scan the server answered with 404/403 is settled — the answer was “no”, and it stays routine however concentrated it is. Nearly every traffic spike on a public site is exactly this shape: one scanner walking a wordlist. Treating each one as an incident is what buries the alerts that matter.
Concentration. If one address accounts for most of the traffic and it is getting through, blocking that address would actually stop it — so it is an incident, and Kvasir is told which address to propose blocking.
Distribution. If the traffic comes from many addresses with no dominant one, blocking them one at a time cannot win; this panel has seen 56 distinct addresses against a single site in a day. Such a situation only alerts once it is genuinely loud, and Kvasir is explicitly told not to propose per-address blocks — the useful answer is a rate limit or a challenge at the edge.
Volume. Below a small threshold nothing pages on its own — two failed logins is a normal day. Above a much larger one it pages whatever the responses say, because a refused flood is still a flood.

Your own address is never an attack source. Any address an administrator has signed in to the panel from in the last 14 days is excluded from the traffic analysis, and refused outright if something tries to block it. An admin clicking through a WordPress dashboard produces one source, many requests, in a burst — by shape alone that is indistinguishable from a scrape, and this panel really did propose blocking its owner’s home connection for it. With blocking set to automatic that proposal would have been carried out.

A situation alerts at most once per hour per (server, rule), so an attack that runs all afternoon is one message rather than one per scan tick. And a rule that trips is announced once, not twice: when Kvasir is set to explain an event, its explanation is the message. If the AI provider can’t be reached, the raw detection is sent instead — a detection is a fact and does not depend on a model.

Nothing is thrown away. Every detection is recorded with the verdict and the reason for it, and the daily ops digest ends with what was handled quietly:

🔕 Handled quietly: 34 routine situations (612 hits) recorded without paging you
• Watcher WordPress login attempts — 22× (410 hits)
• Watcher PHP fatal errors — 12× (202 hits)

That line matters as much as the alerts do: without it, a quiet day and a broken detector look identical.

The classification is deliberately made in code rather than by the AI. Whether you get woken up should be reproducible and testable, not a function of what a model felt like answering. The model explains what happened; it does not decide whether you hear about it.

Player anomalies (a mass disconnect, an unusual influx) bypass this policy — they carry no traffic sources to judge, and are governed by their own cooldown instead.

Incidents that aren’t traffic

Some situations are incidents by their nature rather than by measurement. A crash has no sources and no hit count, so putting it through the traffic classifier would place it under every threshold and file it as routine. These skip the classifier but keep the record and the once-per-window dedupe every other detection gets:

Situation	Raised	Repeats
A container exited unexpectedly	Immediately	Hourly, so a crash-looping server is one message an hour rather than one per exit
A server is still down	Two hours after the exit	Daily
The panel can’t write a server’s files	Within an hour of it happening	Daily

“Still down” is the one that matters most. A crash tells you a server died; nothing used to tell you it was still dead. A DayZ server here exited at 02:04, the panel caught it, sent one message — and it stayed down for two days while the nightly schedules kept recording “update: skipped — server is stopped”. One message at two in the morning is a report that a server went down, not that a server is down.

It only fires when the last thing that happened to the server was an unexpected exit with no operator action after it. Someone who stops a server has said what they want, and paging about those would drown the ones that fell over.

“Can’t write a server’s files” catches the commonest way a server quietly half-breaks. Images that start as root often chown their data directory to PUID:PGID and drop to it, and PUID defaults to 1000 in most of them — while the panel’s account is whatever useradd --system picked. When they differ, the container takes the directory, and backups, restores and the Files tab fail for that server while the app itself carries on looking healthy. The check is a write probe rather than a comparison of user ids: comparing would flag the runes that use a different uid on purpose (mosquitto runs as the broker’s own) and would miss a directory that matches on paper but still can’t be written because of its mode.

The History tab

Every server has a History tab answering “what happened to this, and who did it”:

Server events — started, stopped, restarted, installed, cloned, with the person’s username against each. A scheduled action shows as schedule, one Kvasir took as kvasir, and one issued from Discord names the Discord user. An entry with no actor is a crash: that is how you tell “it fell over” from “someone stopped it”.
Security & health events — the rune’s declared events, rolled up per source.
Player sessions — who was on and when, for runes that record it.

The first section merges three things the panel already recorded separately: the audit log, the crash list and the schedule run log. Having them apart meant “why did this restart at 4am?” took three views to answer — and once genuinely did, when a nightly update schedule was restarting servers that had been deliberately stopped.

Rune versions

The servers list shows each server’s rune and its version — wordpress v18, karakeep v3.

An asterisk after the version means the running container is older than the rune. Runes update themselves from the catalog, but a container keeps the startup script, environment and mounts it was created with until the server is restarted. Without that marker a rune fix that is published but not yet in effect looks exactly like one that didn’t work.

Restarting the server applies the new version — a restart recreates the container, which is why it picks up rune, environment and mount changes while a plain container restart would not.

Host system info

GET /api/system/info backs the dashboard’s host panel (admin only):

Docker reachability
server count, running count, user count, rune count
Go version, architecture, CPU count
host CPU percent, sampled from /proc/stat over a 150 ms window (-1 when unavailable, such as on a non-Linux dev box)
total and used RAM, read from /proc/meminfo (MemTotal minus MemAvailable); zero on a host without /proc, and the dashboard omits the card
free and total bytes on the filesystem holding the Yggdrasil data directory

Low-disk alert

Separately from the per-server disk alarms, Yggdrasil watches free space on the data filesystem every 5 minutes. When free space drops below 10% it sends one notification and stops. It re-arms once free space recovers to 15% or more, so a filesystem hovering at the line doesn’t produce a stream of alerts.

The alert also says what is using the space, when it can. “7% free” is true and not very useful: the obvious next move is to go looking at your servers, and on a panel box that is usually the wrong place — see below.

Docker leftovers, and why the disk fills

Restarting a server doesn’t call docker restart; it removes the container, re-pulls the image and creates a new one. That is what makes Restart a deploy. It also means the image being replaced stays on disk, untagged, and nothing ever removes it.

On a long-lived panel this dominates everything else. Measured on one production box: every server’s data directory together came to about 5 GB, while Docker was holding 157 GB of images, 103 GB of it reclaimable, with 304 of 354 images untagged and unreferenced. Ranking servers by disk usage answers the question you asked and points at the wrong thing.

So there is a second watcher, independent of free space. It checks every 6 hours and warns when reclaimable leftovers pass both 5 GB and 10% of the volume — two conditions, because a share-only rule nags a small VPS about 2 GB and a size-only rule stays quiet on a large disk that is slowly filling. It clears below 8%.

That clear threshold is above zero on purpose. A box that has just been cleaned as thoroughly as possible still holds several gigabytes of reclaimable tagged-but-unused images, orphan volumes and build cache — none of which a dangling-image prune touches. Clearing is the only way the warning can fire a second time, so a threshold set below that floor would latch it on permanently.

The Statistics page

Statistics (admin only) is where this is visible. It shows the machine’s CPU, memory and disk over time, ranks your servers against each other on each, and — separately, because it is a different question — splits the disk itself into server data, Docker images, volumes, build cache and everything else. The three host graphs on the Dashboard link into it.

Per-server disk comes from walking each data directory, which is done hourly. A server whose directory has never been walked reads as “not measured yet” rather than as zero, because zero is a real answer and the absence of a measurement is not.

When there is space to reclaim, the page says so and offers to take it. Reclaim space removes only dangling images: untagged, unreferenced, impossible to start a server from, and the daemon refuses to remove any image a container still uses — running or stopped. The destructive variant (docker image prune -a, which also removes the images of stopped servers and forces a re-pull before they can start again) is deliberately not reachable from the panel. The action is admin-only and audited as system.prune_images.

Updates

GET /api/version reports the running build, the latest GitHub release tag, whether an update is available, and whether the panel can update itself. The GitHub release lookup is cached for 6 hours so the version endpoint — hit on every page load — doesn’t hammer the API. Check now (POST /api/system/check-update) invalidates that cache and refetches, so a just-published release shows up immediately.

Self-update works only for a released vX.Y.Z build with the updater installed by install.sh. The panel runs sandboxed and cannot escalate its own privileges: instead it writes the target tag to a request file in its state directory, and a systemd path unit picks that up and runs a root oneshot updater. If the helper isn’t present, the panel says so and points you at the manual steps rather than failing obscurely. GET /api/system/update-status surfaces the last result the helper wrote, which is where a checksum or download failure shows up after a restart poll times out.

Auto-update is opt-in, with a chosen hour (default 4, server-local). A loop wakes every 20 minutes and acts at most once a day: it runs at the first check at or after your chosen hour that hasn’t run today. The catch-up window matters — gating on the exact hour meant a panel that was rebooting, asleep or offline during that one hour silently skipped the whole day. A self-update restarts only the panel binary; your game and app containers keep running, so the impact is a few seconds of UI downtime.

Security advisories

If something in Yggdrasil turns out to be dangerous, the panel tells you where you already are — at the top of every page, until somebody dismisses it.

There is no push. The project has no register of installs and does not want one, so nothing can be sent to your panel; instead your panel fetches a small public file from the project’s repository, on the same six-hour cadence as the release check. That request says nothing about you: no instance id, no version, no configuration. It is an anonymous GET for a file anyone can read.

It works whether or not the beacon is on. Carrying advisories on the beacon would have been simpler, but the beacon is opt-out — and an admin who switched it off is exactly the person who still needs to hear that their panel has a hole in it.

You only see what concerns you. Each advisory names the release that introduced the problem and the release that fixes it, and your panel shows it only if the version you are running falls in between. Once you are on the fixed release it stops appearing by itself, with nothing to dismiss. A build the panel can’t place — a dev build, something compiled by hand — is treated as affected, because for a security notice the safe mistake is the one that shows it to you anyway.

An advisory is text. It is a headline, a short explanation, and at most one link, all rendered as plain text; a link anywhere other than the project’s own repository or website is dropped and the advisory shows without it. Nothing in an advisory is a command, and nothing the panel reads there can make it install anything: the in-panel updater only ever installs an official release whose checksum matches. That boundary is the point — a channel that could be made to act would be worth attacking.

Dismissing is per advisory and applies to the whole panel, not to your own account: the question is whether somebody dealt with it, not whether everyone read it. It is recorded in the audit log.

Host operating system

Settings → System → Operating system reports what the host has pending: how many updates, how many of those are security, and whether something has asked for a reboot.

The numbers come from the host’s own tooling, so they agree with what the box tells you when you log into it over SSH — apt-check where it exists (Ubuntu’s, the one behind the login banner), falling back to counting apt list --upgradable. That fallback has no security breakdown, and the card says so rather than showing a zero it can’t stand behind: Ubuntu publishes security updates into both -security and -updates, so counting suites would under-report them.

If apt’s package list hasn’t been refreshed in a couple of days the card says that too. A count of zero from a stale list means nothing has been checked, not that nothing is pending.

Yggdrasil does not install them. That’s deliberate, and worth understanding:

Upgrading Docker restarts docker.service, which stops every running server. As a side effect of “update the OS”, with no warning, that’s the most disruptive thing the panel could do.
A kernel update needs a reboot, which is a decision about your players, not a button.
unattended-upgrades already does unattended patching properly, and it ships with the distro.

So the panel tells you, and you choose when:

sudo apt update && sudo apt upgrade

Pick a quiet hour — a server’s Auto-restart dialog names its calmest one, mined from real player counts. See auto-restart.

This exists because the security policy says the fix for the open Docker advisories is keeping Docker updated on the host. Until now nothing in the panel would tell you whether you had.