Off-box alerting for Rocket Pool nodes: pages you when a client hangs or the whole box is gone (open-source agent, looking for feedback)

Disclosure: I’m the developer. NodeBeat is a hosted alerting service
with a free tier; the agent that runs on your node is open source
(Apache-2.0).

Why another monitoring tool, when Smartnode has alerts?

Smartnode’s built-in alerting is good, and I’d keep it. Its default rules
cover clients not synced, low disk, low ETH balance, proposals, sync
committees and updates. Two gaps remain:

  1. It runs on the node. Alertmanager lives in the same Smartnode
    stack, so if the machine freezes, loses power or drops off the
    network, nothing is left to page you. You find out from beaconcha.in,
    after the misses.
  2. There’s no rule for a client that’s down or hung. “Not synced for
    5 minutes” catches a node that falls behind, but not reliably an EL
    that stops answering or a beacon node that stops following the head.
    That’s the failure that quietly costs attestations.

NodeBeat covers those two gaps and is meant to run alongside Smartnode.

How it works

  • A small Go agent polls your beacon node (Beacon API) and execution
    client (JSON-RPC) every second. Read-only: no keys, no signing
    endpoints, no restarts, and it never touches your node wallet. It
    listens on loopback only and makes outbound connections only.
  • A client that’s dead or hung pages you in seconds. In
    fault-injection runs on a test devnet, fault to page took 5.6–8.6s
    (9 runs). A client that still answers but stops advancing pages in
    about a minute.
  • Alerting runs off-box, so a frozen, unplugged or offline node still
    pages you, via Telegram, Discord, Slack, email, PagerDuty, Opsgenie or
    webhook.
  • Given your validator indices, it also alerts on missed attestations and
    proposals, effectiveness and slashing, plus disk, peers, sync and clock
    drift.
  • Trust: signed releases (cosign), SBOM, a public
    threat model
    and alert rules,
    one-command uninstall.

What I’d like from Rocket Pool operators

  1. Would you run this next to your node, and if not, why? Criticism
    of the trust model is the most useful feedback.
  2. Which alerts would be noise for you, and what’s missing?
  3. Testers: the free tier covers 1 node. For operators with several
    machines, I’m giving a few people 3 months of the paid plan free (up
    to 5 nodes, no card) in exchange for a short weekly feedback note.
    Reply here or email [email protected].

Setup on a Smartnode host: Smartnode keeps the client APIs closed to
the host by default, so set Expose RPC Ports (Execution Client) and
Expose API Port (Consensus Client) to Open to Localhost in
rocketpool service config first. They bind to 127.0.0.1 only. The
install guide covers the rest (Smartnode’s P2P port, optional metrics).

Tested on Smartnode v1.24.3 with Geth + Lighthouse, and outside it with
Geth, Reth, Lighthouse and Teku. Nethermind, Besu, Nimbus, Prysm and
Lodestar are auto-detected but untested on my side, so if you run one of
those, I especially want to hear from you.

Code: GitHub - nodebeat/agent · GitHub
Smartnode setup: NodeBeat — Validator Monitoring That Pages in Under 10 Seconds | Ethereum & Cosmos

Is it safe l? Where can it send alerts? How much memory does it use? Noise would be update alerts and low ETH alerts (until they lower what low is).

Thanks for the questions!

Safe? The agent only reads. It makes GET calls to the beacon node’s API and read-only JSON-RPC calls to the execution client. It never touches keys or the validator client, and it never restarts or changes anything on the box. It listens on localhost only and makes outbound HTTPS connections only. The server sends it a node name and an upload address, never config or commands, so the hosted side can’t change what runs on your machine. Releases are signed with cosign and come with an SBOM. Threat model: agent/docs/THREAT_MODEL.md at main · nodebeat/agent · GitHub . On Smartnode, the only change is setting the client APIs to “Open to Localhost”, so nothing new is reachable from the network.

Where do alerts go? Telegram, Discord, Slack, email, PagerDuty, Opsgenie or a webhook. The free tier covers Telegram, Discord, Slack and email. Alerting runs off the node, so it still pages you when the whole box is down.

Memory? Measured on a Geth + Lighthouse devnet with 128 validators tracked and both clients’ metrics scraped: about 160 MB RSS in total. The agent itself is about 20 MB; the rest is Grafana Alloy, which does the scraping and buffers data to disk so nothing is lost while the network is down (~130–145 MB steady, briefly up to ~240 MB at startup). For comparison, your clients use several GB.

Noise: Good to know. Neither of those alerts exists here: there’s no update-available alert and no ETH-balance alert. Hot-path pages are only for things that cost duties: a client down or hung, out of sync, head stalled, missed attestations or proposals. Disk, peers and RPC quality go to a separate warning channel, after a delay. Is there an alert you’d actually want that Smartnode doesn’t give you?