Help · Operators

When Your Node Is Overloaded

An overloaded node stops taking new connections until its load comes down, and the platform moves load off it: a route in use is closed when its user holds the same route through another node, and on an exit node routes nobody has used for two hours are closed a few at a time. You are notified when it starts.

Open in the app All help topics
example
Memory 91% in 3 of 5 readings:
  new connections  refused
  routes in use    moved, 1 per 5 min
  unused routes    exit: 5% an hour

Your agent reports its load every 30 seconds. When a threshold is passed (memory over 90%, a load average above twice its CPU cores over the last 15 minutes, the kernel's connection table 80% full, connections dropped by that table when full or 10 or more dropped by a full listen queue, or bandwidth over 80% of the link speed) and it stays that way in 3 of 5 readings, the node is overloaded: the platform stops sending it new connections and the marketplace waits before offering it again. The notice you receive says what was passed, in figures: "memory at 92 %", for example, or "the connection table was full: 37 connections dropped". You get at most one an hour for a node; every start and end is in the node's event list. A node whose max_users is filled with live users takes no new connections from that moment, with no notice: that is your cap, not an overload.

What happens to the sessions on it

While the overload lasts, the platform moves load off the node:

  • Routes nobody is using, on an exit node. A route whose user has moved no traffic for two hours is closed, about 5% of the node's sessions an hour. Only users' own routes: the routes of a partner's subscription and of an account's extra links are left, because they would be set up again on the same node. On an entry node nothing unused is closed: a route nobody is using holds nothing on an entry, so closing it would not unload the node.
  • Routes in use. Once the overload has lasted 15 minutes and memory is above 80%, one route in use is closed every 5 minutes — but only a route whose user already holds the same route through another node of the same country, at the same price or less, so that user goes on without connecting again. A user's only route through your node is not closed.

A session keeps the price it locked for as long as it runs, unless you lower yours: a drop reaches it once the lower price has stood for 3 hours. A closed route earns nothing more; what it carried is already in your earnings.

What actually helps

  • A full connection table is the agent's to size: from version 0.4.20 it sets the table, the network buffers and the backlogs to the machine's memory at every start. It never lowers a size your host already holds higher. On a Docker host whose security profile does not let it set them, the server keeps the kernel's small defaults, and from 0.4.21 the agent leaves them as a file, config/99-proofkit-sysctl.conf beside its compose file: copy it to /etc/sysctl.d/, run sudo sysctl --system and restart the agent.
  • Memory and connections come from real use — a bigger box is the fix. Raising max_users does not help here: it is a safety net for the node's config size, not a load control, and lowering it does not unload a node either.
  • If the platform has paused routing to your node (the dashboard says so), the node is being emptied: the platform routes nobody new to it, every route nobody has used for two hours is closed, and above 80% memory routes in use are moved the same way. Routing comes back once the platform approves the node again.
  • A price change does not change the load of the sessions already running: a rise never reaches them, and a drop only lowers what they pay, once the lower price has stood for 3 hours.