Smartnode Safe Recovery Planner and Per-Key Diagnosis
What is the work being proposed?
Validator recovery is one of the worst times for a node operator to be unsure about what Smartnode has actually changed.
Today, Smartnode can write validator keys as they are found before it knows whether the rest of the validator set can also be recovered. On a node with mnemonic-derived and imported/custom keys, recovery can write several keys and then fail when a later key cannot be derived or loaded. The command has failed, but the node has already been modified, leaving the operator to work out what was recovered and what still needs attention.
The consequences are not theoretical. Issue #440 records an operator interpreting a test-recovery failure as evidence that their seed was wrong and exiting double-digit pools, only to later confirm that the seed was valid.
We propose to complete and upstream a plan-before-write recovery flow for:
-
wallet recover -
wallet rebuild -
wallet test-recovery
Smartnode will first discover and validate every in-scope validator key without touching the live keystore. Only after the recovery plan is complete will writes begin.
By default, if any required key is unresolved, invalid, or conflicting, no validator-key writes will occur. An operator who intentionally wants a partial recovery can explicitly opt in with --allow-partial-recover, which will write only validated keys and clearly report what happened for each pubkey.
The work also includes:
-
per-key discovery and write results;
-
the inactive-validator filtering fix;
-
legacy minipool and megapool coverage;
-
safe/idempotent reruns;
-
human-readable and JSON/API output;
-
recovery documentation for imported keys and solo migrations.
This does not attempt transactional rollback across validator clients or claim to recover an imported key that has already been lost. The guarantee is narrower: Smartnode determines the recovery state before live keystore mutation begins.
Is there any related work this builds off of?
Yes. This work addresses Smartnode issues #407 and #572. Issue #440 provides operator-impact context.
Before applying, we asked whether #407 and #572 were still wanted and whether anyone was already working on them. Sma@0xforna@0xforna@0xfornaxtnode maintainer @0xfornax confirmed that they are still wanted and that they are not aware of anyone currently working on them:
We have also opened an unfunded feasibility prototype against current Smartnode master:
The prototype demonstrates the recovery-plan model, plan/commit separation, inactive-validator filtering, per-key outcomes, and a writer-spy test showing zero writes when default recovery contains an unresolved key.
It is not the completed grant deliverable. Full integration into the recovery commands, CLI/API behavior, production writer adapters, broader failure testing, documentation, and maintainer-requested revisions remain to be completed.
The prototype work is not being billed retroactively.
Will the results of this project be entirely open source?
Yes. All Smartnode code, tests, fixtures, and documentation produced under the grant will be public and contributed under the repository’s existing license.
Benefit
| Group | Benefit |
|---|---|
| Potential rETH holders | More reliable node-operator recovery reduces avoidable operational risk. |
| rETH holders | Fewer recovery mistakes and avoidable validator downtime. |
| Potential NOs | A clearer and safer disaster-recovery path. |
| NOs | Direct benefit: recovery fails safely by default and reports what happened for each validator key. |
| Community | Fewer ambiguous recovery cases and clearer information when helping operators troubleshoot. |
| RPL holders | Indirect benefit through better operator reliability and confidence. |
Which non-Rocket Pool groups benefit?
The implementation is Rocket Pool-specific. The general plan-before-write pattern may be useful to other validator tooling projects, but no external protocol or commercial product is needed for the work to be useful.
Work
Who is doing the work?
Crackdevs
-
Technical Lead: Olamide Afunsho
-
Project Manager: Ifedemiji Omoniyi
-
Handle: @Crackdevs
-
Contact: info@crackdevs . org
Olamide will lead implementation, testing, and technical revisions. Ifedemiji will manage milestones, documentation, coordination, and delivery.
What is the background of the team?
Crackdevs works on open-source protocol tooling and testing infrastructure.
Our most relevant execution evidence is the Smartnode feasibility PR itself. Before requesting funding, we traced the current recovery path, implemented the recovery-plan model in Go, separated read-only planning from writes, added per-key results, and built failure-focused tests around the zero-write requirement.
The team has also worked on protocol testing and failure-recovery tooling across projects including Zcash and Mina.
Milestones and deadlines
Duration: 5 weeks
| Milestone | Deliverables | Payment |
|---|---|---|
| M1 — Production integration foundation | Incorporate maintainer feedback, confirm validator-state handling, adapt the prototype to current Smartnode abstractions, and integrate the shared planner into the recovery flows. | $2,000 |
| M2 — Recovery behavior and testing | Complete recover/rebuild/test-recovery integration, fail-closed default behavior, partial recovery, CLI/API results, production writer adapters, inactive-filter fix, failure handling, and rerun tests. | $2,800 |
| M3 — Final validation and documentation | Integration testing, documentation, reproduction instructions, in-scope maintainer review fixes, and release notes. | $1,000 |
How will the work be tested?
Testing is included throughout the five weeks.
The main safety test is straightforward: if default recovery cannot resolve or validate every required key, the writer must record zero live keystore writes.
Coverage will include:
-
mnemonic-only recovery;
-
mixed mnemonic/custom keys;
-
missing custom key material;
-
invalid or mismatched key material;
-
derivation-limit failure;
-
already-installed and conflicting keys;
-
inactive/exited validators;
-
explicit partial recovery;
-
injected write failure;
-
safe reruns;
-
deterministic, secret-free CLI/API results.
Also, no operator mnemonic, private key, password, or keystore will be requested for testing.
How will the work be maintained?
We will provide a 90-day defect-fix period for issues within the delivered scope.
The implementation will live directly in Smartnode rather than as a separate service that Rocket Pool needs to operate.
Costs
Acceptance criteria
The grant is complete when:
-
The shared recovery planner is integrated into
recover,rebuild, andtest-recovery. -
An unresolved, invalid, or conflicting required key produces zero live writes by default.
-
Partial recovery requires explicit opt-in and reports every relevant pubkey.
-
Write failures clearly report written, failed, and not-attempted results, with safe reruns.
-
Minipool/megapool state handling and inactive-validator filtering are covered by tests.
-
Human-readable and JSON/API output is deterministic and does not expose secret key material.
-
Tests, documentation, and reproduction instructions are complete.
-
A production-ready upstream PR is submitted and in-scope maintainer review feedback is addressed.
Final merge timing is controlled by Smartnode maintainers and is not part of the five-week acceptance requirement.
Proposed payment schedule
Total request: $5,800 over five weeks.
-
M1: $2,000
-
M2: $2,800
-
M3: $1,000
No upfront payment is requested.
The existing feasibility prototype is not included as paid work.
Who will directly receive payment?
Crackdevs
EVM address:
0xFC4fBB3Be6039635087a96Fe252beAFB13B11406
How will the GMC verify delivery?
Work will remain public through the Smartnode PR and working branches.
Each milestone will include the relevant commits, completed acceptance items, test results, and short reproduction instructions. The zero-write requirement will be verified through automated tests.
What cost-saving alternatives were considered?
The work reuses Smartnode’s existing wallet, beacon-state discovery, custom-key handling, and validator-client infrastructure.
The grant does not include a separate service, dashboard, new cryptography, key custody, recovery of already-lost keys, cross-client transactional rollback, or a full live-node matrix across every validator client.
Have you already been compensated by Rocket Pool for this work?
No.
Conflict of Interest
Any conflicts of interest to disclose?
No.
Will another vested project benefit financially?
Other than Crackdevs receiving the stated compensation for completing the grant, no external protocol or project in which we have a vested interest will directly benefit financially.