# Round 40 - GMC Call for Grant Applications - Deadline is September 7 **URL:** \ **Category:** Grants / Bounties\ **Tags:** gmc\_grant\_apps, gmc\_round\_40\ **Created:** [8 August 2026 00:27 UTC](https://dao.rocketpool.net/t/round-40-gmc-call-for-grant-applications-deadline-is-september-7/4019 "2026-08-08T00:27:41Z")\ **Posts on this page:** 1\ **Showing post:** 9
**Author:** ![Crackdevs](https://dao.rocketpool.net/letter_avatar_proxy/v4/letter/c/e9c0ed/32.png) [@Crackdevs](https://dao.rocketpool.net/u/Crackdevs)\ **Post date:** [7 September 2026 05:45 UTC](https://dao.rocketpool.net/t/round-40-gmc-call-for-grant-applications-deadline-is-september-7/4019/9 "2026-09-07T05:45:33Z")
# Smartnode Safe Recovery Planner and Per-Key Diagnosis ### What is the work being proposed? Validator recovery is one of the worst times for a node operator to be unsure about what Smartnode has actually changed. Today, Smartnode can write validator keys as they are found before it knows whether the rest of the validator set can also be recovered. On a node with mnemonic-derived and imported/custom keys, recovery can write several keys and then fail when a later key cannot be derived or loaded. The command has failed, but the node has already been modified, leaving the operator to work out what was recovered and what still needs attention. The consequences are not theoretical. Issue #440 records an operator interpreting a `test-recovery` failure as evidence that their seed was wrong and exiting double-digit pools, only to later confirm that the seed was valid. We propose to complete and upstream a **plan-before-write recovery flow** for: - `wallet recover` - `wallet rebuild` - `wallet test-recovery` Smartnode will first discover and validate every in-scope validator key without touching the live keystore. Only after the recovery plan is complete will writes begin. By default, if any required key is unresolved, invalid, or conflicting, **no validator-key writes will occur**. An operator who intentionally wants a partial recovery can explicitly opt in with `--allow-partial-recover`, which will write only validated keys and clearly report what happened for each pubkey. The work also includes: - per-key discovery and write results; - the inactive-validator filtering fix; - legacy minipool and megapool coverage; - safe/idempotent reruns; - human-readable and JSON/API output; - recovery documentation for imported keys and solo migrations. This does not attempt transactional rollback across validator clients or claim to recover an imported key that has already been lost. The guarantee is narrower: Smartnode determines the recovery state before live keystore mutation begins. ### Is there any related work this builds off of? Yes. This work addresses Smartnode issues #407 and #572. Issue #440 provides operator-impact context. Before applying, we asked whether #407 and #572 were still wanted and whether anyone was already working on them. Sma@0xforna@0xforna@0xfornaxtnode maintainer @0xfornax confirmed that they are still wanted and that they are not aware of anyone currently working on them: > > > Assume I have 9 minipools, 5 leb8s and then 1 leb8 which was a 0x00 solo validat…or i converted followed by 3 leb 8s from credit. > > If I run \`rocketpool wallet recover\` or \`rocketpool wallet rebuild\` now, I believe the behavior will recover the first 5 keys, and then fail unless I've already imported my solo key. > > Unfortunately, I'll have 5 out of 8 of the keys that I can recover in my wallet, which is a bit of a POLA violation. > > Instead, smartnode should fail fully- if all keys can't be recovered, none are. > > Additionally, it should have a new flag --allow-partial-recover or something that overrides this behavior, and recovers as many keys as it can (critically, after the failure, it should keep trying to recover subsequent keys). > > Edit: > > The response \_must\_ inform the user which pubkeys were successfully recovered, and which were not, in the case of --allow-partial-recover. > > If --allow-partial-recover \_is not passed\_, and recovery fails on iterating to the limit, we should ask the user if they did a solo migration and remind them that they must deal with the solo validator's keys manually, and keep them securely backed up, as they will not be recoverable by the node wallet's mnemonic. > > Additionally, we should update \`rocketpool wallet test-recovery\` to be very verbose when recovery fails- which validators keys were successfully recovered, which failed to be recovered, which (if any) were imported from custom\_keys. \`test-recovery\` should also take the opportunity, should it find any \`custom\_keys\`, to provide a similar admonition about backing up the imported key. We have also opened an **unfunded feasibility prototype** against current Smartnode master: > > > \*\*Status\*\*: This is an unfunded feasibility prototype for a Round 40 grant appli…cation, not a production-complete or merge-ready implementation. It demonstrates the recovery-plan model, plan/commit separation, inactive-validator filtering fix, and zero-write safety test. Remaining work includes integration into \`recover\`, \`rebuild\`, and \`test-recovery\`; CLI/API behavior; production writer adapters; broader failure-path tests; documentation; and maintainer-requested revisions. > > \--- > > \### Description > This PR introduces the \*\*Safe Recovery Planner & Per-Key Diagnosis\*\* architecture for validator key recovery, prototyping a solution for #407 and #572; #440 provides operator-impact context. > > Previously, Smartnode persisted validator keys to disk immediately as each individual key was discovered. In mixed-key setups (mnemonic-derived, custom EIP-2335 keystores, megapools), encountering an error on a later key left the operator in a partially-written, inconsistent state. Additionally, inactive/exited validators on the Beacon chain were filtered into \`filteredPubkeys\`, but the target map was populated from the unfiltered list, causing recovery to fail on exited validators (#407). > > \### Key Architectural Changes > 1. \*\*Two-Stage Plan-Before-Write Execution (\`shared/services/recovery\`)\*\*: > - \*\*Plan Phase (\`PlanValidatorRecovery\`)\*\*: Pure, read-only discovery that assesses minipool/megapool keys, checks Beacon statuses, inspects custom keys, tests mnemonic derivation, and checks installed keystores. \*\*Zero disk writes\*\*. > - \*\*Commit Phase (\`CommitPlan\`)\*\*: Enforces the safety invariant: in default mode, if any in-scope key is unresolved or invalid, recovery halts before touching disk (\`ErrUnresolvedKeysRefusingCommit\`). An explicit \`--allow-partial-recover\` option allows committing only validated keys while recording skipped keys. > 2. \*\*Deterministic Per-Key Diagnosis (\`types.ValidatorPlanEntry\`)\*\*: > - Explicit outcomes: \`mnemonic\_valid\`, \`custom\_valid\`, \`installed\_valid\`, \`unresolved\`, \`invalid\_material\`, and \`excluded\_inactive\`. > 3. \*\*Issue #407 Bug Fix\*\*: > - Fixed \`recoverNodeKeys\` in \`rocketpool/api/wallet/recover.go\` so \`pubkeyMap\` correctly iterates over \`filteredPubkeys\`. > 4. \*\*Spy-Writer Test Suite\*\*: > - Comprehensive tests including \`TestMixedRecoveryWithOneUnresolvedKey\_DefaultFlowZeroWrites\` asserting through a writer-spy test that \`writerSpy.TotalWrites() == 0\` on unresolved key failure under default mode. > > \### Testing Done > \- \`go test -v -count=1 ./shared/services/recovery/...\` (All 6 test suites passing) > \- \`go test -v -count=1 ./rocketpool/api/wallet/...\` (All wallet tests passing) > \- Passed \`golangci-lint/v2@v2.13.1\` with 0 issues. > > \--- > \*Note for maintainers: As this is a contribution from a new fork, the GitHub Actions workflows are awaiting approval to run (\`action\_required\`). Could a maintainer please approve the workflow runs? Thank you!\* The prototype demonstrates the recovery-plan model, plan/commit separation, inactive-validator filtering, per-key outcomes, and a writer-spy test showing zero writes when default recovery contains an unresolved key. It is not the completed grant deliverable. Full integration into the recovery commands, CLI/API behavior, production writer adapters, broader failure testing, documentation, and maintainer-requested revisions remain to be completed. The prototype work is not being billed retroactively. ### Will the results of this project be entirely open source? Yes. All Smartnode code, tests, fixtures, and documentation produced under the grant will be public and contributed under the repository’s existing license. # Benefit | Group | Benefit | | --- | --- | | Potential rETH holders | More reliable node-operator recovery reduces avoidable operational risk. | | rETH holders | Fewer recovery mistakes and avoidable validator downtime. | | Potential NOs | A clearer and safer disaster-recovery path. | | NOs | Direct benefit: recovery fails safely by default and reports what happened for each validator key. | | Community | Fewer ambiguous recovery cases and clearer information when helping operators troubleshoot. | | RPL holders | Indirect benefit through better operator reliability and confidence. | ### Which non-Rocket Pool groups benefit? The implementation is Rocket Pool-specific. The general plan-before-write pattern may be useful to other validator tooling projects, but no external protocol or commercial product is needed for the work to be useful. # Work ### Who is doing the work? **Crackdevs** - **Technical Lead:** Olamide Afunsho - **Project Manager:** Ifedemiji Omoniyi - **Handle:** @Crackdevs - **Contact:** info@crackdevs . org Olamide will lead implementation, testing, and technical revisions. Ifedemiji will manage milestones, documentation, coordination, and delivery. ### What is the background of the team? Crackdevs works on open-source protocol tooling and testing infrastructure. Our most relevant execution evidence is the Smartnode feasibility PR itself. Before requesting funding, we traced the current recovery path, implemented the recovery-plan model in Go, separated read-only planning from writes, added per-key results, and built failure-focused tests around the zero-write requirement. The team has also worked on protocol testing and failure-recovery tooling across projects including Zcash and Mina. ### Milestones and deadlines **Duration: 5 weeks** | Milestone | Deliverables | Payment | | --- | --- | --- | | **M1 — Production integration foundation** | Incorporate maintainer feedback, confirm validator-state handling, adapt the prototype to current Smartnode abstractions, and integrate the shared planner into the recovery flows. | **$2,000** | | **M2 — Recovery behavior and testing** | Complete recover/rebuild/test-recovery integration, fail-closed default behavior, partial recovery, CLI/API results, production writer adapters, inactive-filter fix, failure handling, and rerun tests. | **$2,800** | | **M3 — Final validation and documentation** | Integration testing, documentation, reproduction instructions, in-scope maintainer review fixes, and release notes. | **$1,000** | ### How will the work be tested? Testing is included throughout the five weeks. The main safety test is straightforward: if default recovery cannot resolve or validate every required key, the writer must record **zero live keystore writes**. Coverage will include: - mnemonic-only recovery; - mixed mnemonic/custom keys; - missing custom key material; - invalid or mismatched key material; - derivation-limit failure; - already-installed and conflicting keys; - inactive/exited validators; - explicit partial recovery; - injected write failure; - safe reruns; - deterministic, secret-free CLI/API results. Also, no operator mnemonic, private key, password, or keystore will be requested for testing. ### How will the work be maintained? We will provide a **90-day defect-fix period** for issues within the delivered scope. The implementation will live directly in Smartnode rather than as a separate service that Rocket Pool needs to operate. # Costs ### Acceptance criteria The grant is complete when: 1. The shared recovery planner is integrated into `recover`, `rebuild`, and `test-recovery`. 2. An unresolved, invalid, or conflicting required key produces **zero live writes by default**. 3. Partial recovery requires explicit opt-in and reports every relevant pubkey. 4. Write failures clearly report written, failed, and not-attempted results, with safe reruns. 5. Minipool/megapool state handling and inactive-validator filtering are covered by tests. 6. Human-readable and JSON/API output is deterministic and does not expose secret key material. 7. Tests, documentation, and reproduction instructions are complete. 8. A production-ready upstream PR is submitted and in-scope maintainer review feedback is addressed. Final merge timing is controlled by Smartnode maintainers and is not part of the five-week acceptance requirement. ### Proposed payment schedule **Total request: $5,800 over five weeks.** - M1: **$2,000** - M2: **$2,800** - M3: **$1,000** No upfront payment is requested. The existing feasibility prototype is not included as paid work. ### Who will directly receive payment? **Crackdevs** **EVM address:** `0xFC4fBB3Be6039635087a96Fe252beAFB13B11406` ### How will the GMC verify delivery? Work will remain public through the Smartnode PR and working branches. Each milestone will include the relevant commits, completed acceptance items, test results, and short reproduction instructions. The zero-write requirement will be verified through automated tests. ### What cost-saving alternatives were considered? The work reuses Smartnode’s existing wallet, beacon-state discovery, custom-key handling, and validator-client infrastructure. The grant does not include a separate service, dashboard, new cryptography, key custody, recovery of already-lost keys, cross-client transactional rollback, or a full live-node matrix across every validator client. ### Have you already been compensated by Rocket Pool for this work? No. # Conflict of Interest ### Any conflicts of interest to disclose? No. ### Will another vested project benefit financially? Other than Crackdevs receiving the stated compensation for completing the grant, no external protocol or project in which we have a vested interest will directly benefit financially. --- _[View the full topic](https://dao.rocketpool.net/t/round-40-gmc-call-for-grant-applications-deadline-is-september-7/4019)._