Help · Operators

Node checks

The Checks tab on a node's page lists what ProofKit tests on the node: its agent, disk, cover site, each transport from outside, host limits, load, links, carrier accounts and price. A failing check says what to run on the server to fix it.

Open in the app All help topics
example
Hysteria2 on UDP 443      failing
A test connection from outside timed out:
nothing answered on UDP 443.
Fix: open UDP 443 in your hosting panel
as well as on the server.

The Checks tab is the first tab of a node's page. It shows where the node is on its way to carrying users and each check behind it, the failing ones first. A check says what ProofKit saw and, when something is wrong, how to fix it, with the commands to run on the server as root. Each command has its own Copy button.

  • Failing: something is wrong, and the fix is under it. Some failing checks hold the node back from review, and the tab says which.
  • Warning: worth fixing. It holds the node back from review only for an entry with no link.
  • Pending: no answer yet. No test connection has come back, or ProofKit's own test connections are not reliable at the moment, so nothing is judged. A pending check can hold the node back from review as a failing one would, and it never sends you a notice.
  • Ok and info: nothing to do, except that one info check, Data plane, offers an upgrade you may want.

Check again asks ProofKit for a new round of test connections to the node. A node can ask once in 5 minutes, and the tab reads the checks again a minute later. It says so at once when nothing can be asked: the node's agent is not connected, the node is suspended, or an entry has no link to an exit.

ProofKit also writes to you when a check of one of your nodes has failed for a while, with its fix, and again when a node starts carrying users. Both notices are on by default, by email and in Telegram if you linked it. You can turn each off under Settings on the Wallet page, as "Node checks" and "Node live".

Agent

Three checks read the node's agent.

  • Agent first connection (agent_never_connected): the agent has not sent a first heartbeat. It is pending for 15 minutes after you register the node, failing after, and it holds the node back from review. Run the install command on the server, as root. The dashboard showed it when you registered the node, and Reinstall makes a new one. If you already ran it, the agent's log says why it cannot connect.
  • Agent connection (agent_offline): the last heartbeat is 30 minutes old or older. A node counts as offline after 30 minutes without a heartbeat: the routes on it are closed, and after 24 hours without one it is suspended. Check that the server is up and the agent runs, and read its log. If it is not running, restart it. If the server was rebuilt, use Reinstall.
  • Agent version (agent_outdated): a Docker or Podman node runs an older agent than the network serves. It is a warning and holds nothing back. ProofKit releases a new agent image once a night, and a Docker install updates itself between 23:00 and 06:00 UTC, through its Watchtower container; if the node is still behind after that, update by hand. In the install directory, pull the new image and start the agent on it. The restart cuts the node's connections once. ProofKit tells you only after the check has held for 48 hours. A systemd agent updates itself and is not judged here. See Updating the Agent.

The tab gives the commands for the way your agent runs. Agent first connection gives the agent's last lines of log (docker logs --tail 50 dvpn-agent, podman logs --tail 50 dvpn-agent or journalctl -u dvpn-agent -n 50 --no-pager). Agent connection gives the same lines of log and the agent's restart (docker restart dvpn-agent, podman restart dvpn-agent or systemctl restart dvpn-agent). Agent version gives the update instead: cd /opt/proofkit-dvpn, then docker compose pull and docker compose up -d, or podman compose pull and podman compose up -d.

Disk

Disk space (disk_low) reads the free disk the agent reports. It warns under 2 GB and fails under 1 GB, and a failing disk holds the node back from review. With no reading yet it is pending and holds nothing back, because the agent may only be quiet. Free some space or grow the disk. On a Docker or Podman node, below 2 GB, pulling a new agent image can fail and leave the node offline, and the commands the tab gives (docker image prune -f or podman image prune -f) remove images that no container uses and no tag names, such as the agent's old image. A systemd node has no images: the tab gives sudo journalctl --vacuum-size=200M, which trims the system journal, where the agent's log goes, to 200 MB.

Cover site

Cover site (cover_site, exits only) tests the site your Reality connections borrow the identity of. Reality needs a large, ordinary site that completes TLS 1.3 with X25519. ProofKit makes that handshake itself, from its own servers, when the node's cover site changes and once a day. The check is pending until the first handshake and failing when the site does not complete it or has moved behind an anti-DDoS service, and either holds the node back from review. Choose another site in the node's Settings; Regenerate picks one for you. Changing it restarts Xray on the node. See SNI (Reality cover site).

Transports from outside

ProofKit's own prober makes real test connections to your node from its servers and fetches a small file through each transport the node offers, the way a user's app would. While the node is not live, in its first week, a line that has not got through is tried again about every 5 minutes; a line that works is tried every 6 hours.

  • One check for each line (transport_unreachable). An exit has a line for each transport it offers: Reality on TCP 443, Hysteria2 on UDP 443 and XHTTP on TCP 8444. An entry has a line for each exit it links to, over each transport the route carries: Reality on TCP 443, Hysteria2 on a UDP relay port in 30000-39999 and XHTTP on a TCP relay port in 40000-49999. A line fails when 3 of its newest 4 test connections were refused, timed out, were cut off or came back altered; with fewer than 4, when the newest 2 were. A test connection that reached the exit but met an error from the test server, or did not run, says nothing either way.
  • Test connections through the exit (exit_probe_failing): 3 or more of ProofKit's last 4 test connections through the exit failed. The exit takes connections, but they do not get through. Restart the agent: it brings Xray up again, and WARP where it is on. If it keeps failing after that, update the agent and open a ticket in Support.

A line that is refused or times out means nothing from outside reaches Xray on its port. A firewall in your hosting panel often drops it: open the port there as well as on the server. Hosting panels often open TCP only, so Hysteria2 needs UDP 443 opened by hand. Another program on port 443, a web server for example, blocks Reality too. The command the tab gives shows what listens: ss -tlpn | grep ':443' for Reality, ss -ulpn | grep ':443' for Hysteria2 and ss -tlpn | grep ':8444' for XHTTP. For an entry's lines, open UDP 30000-39999 and TCP 40000-49999 in the hosting panel's firewall, check that the link to the exit is up, and fix the exit's own checks first if it fails them.

A line that is cut or altered after it opens has a port that answers, and something on the way spoils the connection: a firewall or filter in front of the server that inspects traffic or, on an entry, a link that loses packets. The agent's log shows Xray's errors.

When a large share of the network's test connections fail at once, that says ProofKit's own prober is unwell and not your node. Every line and the exit's test connections then read pending, say so, and send no notice.

Host limits

Host network limits (host_tuning) reads how the host took the network settings the agent sets when it starts: the connection table, the buffers and the backlogs. It warns when the host took only part of them or refused them, and it holds nothing back. The node then runs on the kernel's small defaults, and a full connection table is what that costs.

  • A Docker or Podman node: from agent 0.4.21 the agent leaves the settings it wanted in config/99-proofkit-sysctl.conf, beside the compose file. On the server, copy it to /etc/sysctl.d/, run sudo sysctl --system and restart the agent: it reads the size of the connection table when it starts.
  • A systemd node: sudo sysctl --system prints the setting that failed, for example because a kernel module is not loaded or the kernel lacks the key. Fix that on the host. The agent keeps its settings in /etc/sysctl.d/98-swap.conf, /etc/sysctl.d/99-vpn.conf and /etc/sysctl.d/99-keepalive.conf and writes them again at every start, so editing them does not last. Then restart the agent.

See When Your Node Is Overloaded for what the table's size does.

Load

Load (overloaded) is a warning while an overload is open on the node: the platform sends it no new connections until its load comes down. The node's page says what was passed: memory, load, the connection table, dropped connections or bandwidth. A full connection table is the agent's to size (see Host limits above). Memory and connections come from real use, so a bigger server is the fix. Raising max_users does not help. See When Your Node Is Overloaded.

Entries

Links to exits (entry_unlinked, entries only) warns while an entry has no active link to an exit, and it holds the entry back from review: an entry carries traffic only through a link. In My Nodes, open the entry's manage row and pick Link to origin. See Edge ↔ Origin Links.

Carrier accounts

Carrier accounts (carrier_account_error, exits only) warns while one of your olcRTC carrier accounts reads error for a carrier the exit serves. Rooms already up keep serving, but no new room is made on that account. Open olcRTC carriers on your dashboard and connect the account again. See olcRTC carriers.

Price ceiling

Price ceiling (price_above_ceiling) warns while the node is approved for automatic routing but priced above the network's ceiling for it, which the check names in TON per GB. Users still reach the node when they choose it themselves. Lower the price to the ceiling or under in the node's Settings and the node is routed to automatically. The price can change once in 24 hours. See Price per GB (TON).

Data plane

Data plane (data_plane_layout) is an info check, shown only while ProofKit has switched it on for the node and only for a node on agent 0.5.0 or later that still runs the agent and the VPN processes (Xray, nginx, WARP) in one container. Each agent update recreates that container and cuts every connection through the node. Run the one-line upgrade at a quiet hour and the VPN processes move into a container of their own, which an agent update leaves alone. It cuts the node's connections once, and the same command with --rollback-layout undoes it. It holds nothing back and sends no notice. See Updating the Agent.

Review

Three checks say where the node is. Waiting for review and routing paused have nothing to fix; suspended says how to bring the node back. All three are info.

  • Waiting for review (in_review): ProofKit reviews each new node before it routes users to it automatically. Until then users reach it only when they choose it themselves. The review starts once no check that holds the node back is failing or pending.
  • Routing paused (paused): ProofKit has paused routing to this node. It routes nobody new to it, and routes nobody has used for two hours are closed. Routing comes back once the platform approves the node again.
  • Suspended (suspended): the node is offered to nobody. A node is suspended after 24 hours without a heartbeat or when its health score falls under 30, and an admin can suspend one. A reconnecting agent does not lift it. Register the node again from the same server, with the same type, or Reinstall it. Either keeps its ID and stake (Reinstall keeps its settings and price too) and gives a new install command: run it on the server, as root, and the new agent's first connection makes the node active. A health score still under 30 suspends it again. If you do not know why it was suspended, open a ticket in Support.

The state above the checks is one of: registered (its agent has not connected yet), checking (a check that holds it back from review is failing or pending), waiting for review, live (approved and active, so users are routed to it), approved, not routed automatically (priced above the ceiling), routing paused, offline and suspended. A node that has been live is not sent back to checking by a check that starts to fail: its state follows routing, and its checks still show.

Each check, in the operator docs

The operator docs give each check its own section: what it means, why it matters and the fix, with the commands. Pick the one the Checks tab names.

  • Agent first connection (agent_never_connected): the agent has not sent its first heartbeat.
  • Agent connection (agent_offline): the last heartbeat is 30 minutes old or older.
  • Agent version (agent_outdated): a container node runs an older agent than the network serves.
  • Disk space (disk_low): under 2 GB free warns, and under 1 GB fails.
  • Cover site (cover_site), exits only: the site Reality borrows the identity of does not pass.
  • Each line, from outside (transport_unreachable): a test connection over one transport did not get through.
  • Test connections through the exit (exit_probe_failing): 3 or more of the last 4 failed.
  • Host network limits (host_tuning): the host took only part of the network settings the agent sets.
  • Load (overloaded): an overload is open, and the node gets no new connections until it ends.
  • Links to exits (entry_unlinked), entries only: the entry has no active link to an exit.
  • Carrier accounts (carrier_account_error), exits only: an olcRTC carrier account reads error.
  • Price ceiling (price_above_ceiling): the price is above the network's ceiling for automatic routing.
  • Data plane (data_plane_layout): the agent and the VPN processes share one container, so an agent update cuts the node's connections.
  • Waiting for review (in_review): ProofKit reviews a new public node before it routes users to it; nothing to fix.
  • Routing paused (paused): ProofKit routes nobody new to the node; nothing to fix.
  • Suspended (suspended): the node is offered to nobody.