The Black Swan thread

Black swans are sudden and devastating events that are extremely unlikely to occur and thus very difficult to protect against. If they don’t happen, any capital spent to prepare is wasted; however, if they occur it is almost always more severe than was prepared for. Probably the most feared from the standpoint of Rocket Pool is a double attestation or other client bug affecting all validators running that software, although others (smartnode malicious code, AWS hack, dappnode software issue, etc) are also possible.
As we reduce ETH bond through LEB, it increases the potential damage to rETH from black swans. It is also increasingly evident that large amounts of RPL collateral are unlikely to actually help when such an event occurs. As such, I think it makes sense to formulate an action plan for if and how we prepare for black swan events, and what this insurance system would look like.

My argument for the terms collateral and insurance; you definitely don’t have to agree with this to participate in the thread:

So a long time ago @valdorff asked me a question that I’ve been struggling with ever since: “What does the RPL bond represent and why do we need it? If we can’t explain this, it will always create FUD.”
Here’s my answer, which is still not elegant unfortunately: the minimum RPL requirement does indeed represent collateral, but only collateral on the amount of ETH you use to generate rewards. For current minipools, you get rewards on 15% of 16 protocol ETH (2.4 ETH); for this you put 10% (1.6 ETH) at stake- 66% collateralized in RPL, added to your collateralized ETH bond and giving an LTV ratio in the realm of tradfi. If your collateral gave you access to the full 16 ETH, you should be getting rewards for the full 16 protocol ETH (like if you put 16ETH worth of USDC collateral on AAVE and got slightly less than 16 ETH to stake with). Also, if your collateral represented access to the full protocol ETH, your LTV ratio in LEB 4 might only be 24%, inclusive of your ETH bond- this would not be a safe level under any system ever invented.
Based on this, the maximum RPL allowable stake cannot represent collateral. It represents insurance against extremely unlikely events: black swan correlated slashing (has never happened), attestation leak/abandoned validator (very rare and isolated), and repeated MEV theft (has never happened). Like insurance, RPL stakers are paid a monthly premium with the understanding they will pay out if a rare event occurs. In other walks of life, risk is spread amongst many individuals so that only 1%, 3%, 5% of the total assets need to be held in cash reserves. However, currently Rocket Pool has each minipool individually insured with an average of 70% RPL reserve.
This is why (without getting into the specifics of numbers) I believe there does not need to be a link between minimum RPL (collateral) and maximum RPL (insurance); both should be decided separate based on the needs for each.

RPL is not good insurance for black swans:

Correlated slashing damage to rETH increases with by much more than the square of the number of validators slashed. This is because

  1. Correlated slashing per validator in ETH is like (1+ [%validators slashed in a month]x3x31)
  2. The number of rocket pool validators is also proportional to [% validators slashed in a month]
  3. With slashing below the level of ETH collateral, no RPL is touched and rETH not affected

So for example, if 9000 minipools exist and rocket pool validators mirror Ethereum validator market share, a 5% correlated slashing would not affect rETH share
A 15% correlated slashing not effect rETH share
A 25% correlated slashing would destroy 18,562 ETH from rETH
In LEB 8, the damage to rETH for 5/15/25% correlated slashing would be: 0 ETH, 9382 ETH, 36,562 ETH
In LEB 4, the damage to rETH for 5/15/25% correlated slashing would be: 742 ETH, 14782 ETH, 45,562 ETH

These numbers are far above our ability to provide RPL liquidity. In terms terms of RPL, in LEB 8 (15% client, 9382 ETH gives about 600,000 RPL auctioned in a month at current numbers); however, since LEB is expected to double the minipools, we would talk about more than 1.2 million RPL needed unless RPL ratio also rises 2x; and this is still an underestimate as rational actors would front-run a correlated slashing by either selling RPL before the auction or pulling ETH liquidity.

The combined equation for damage to rETH from a correlated slashing event is something like ((1 + [%Ethereum validators slashed]x3x31)-[ETH Bond]) x [%RP validators slashed] x [#RP validators])
This gives a few conclusions:

  1. RPL liquidity is not sufficient of handle a market dump for even a 15% minority client correlated slashing, and or a much smaller slashing with LEB 4.
  2. The ways to reduce the injury to rETH are either to:
    a. increase ETH bond or decrease #minipools (LEB obviously is intended to do the opposite)
    b. find a way to add liquidity RPL so it can act as insurance, or make a different insurance system
    c. decrease the correlation of rocketpool validators to Ethereum validators slashed [eg have a higher number of minority clients, better education on upgrades, etc)

So I was hoping this thread would be an incubator for ideas about how to do (b) and/or (c). I’ll put my ideas on insurance pools as the first comment, but I’d prefer every crazy idea that you think of to get thrown around.

5 Likes

An idea:

Overview: Two insurance pools that work synergistically to reduce loss to rETH, where risk and cost are both spread amongst participants rather than on an individual minipool basis. The first pool (rETH) reduces impact to rETH from correlated slashing and attestation leak. The second pool (RPL) reduces variance risk to NOs and indirectly reduces impact to rETH by reducing the total amount in a correlated slashing. For this, 25% emissions currently targeted at NOs for collateral will be initially used, although these could shift between pools based on supply/demand. I would anticipate this percentage would rise in LEB 8 and LEB 4 as the total impact of correlated slashing increases while the emissions from RPL would not.

Pool #1: rETH insurance secured by rETH

How it works: Anyone can permissionlessly stake to the rETH insurance pool, and get paid a staking yield in RPL to do so. If there is a correlated slashing event (or abandoned validator > NO bond), assets from the insurance pool are burned and used to make the rETH whole up to the total amount lost.
Staked asset: rETH. This has several attributes that make it attractive to back rETH.

  1. It is yield bearing already, so it is more capital efficient than ETH or USDC
  2. It has good liquidity that we are actively incentivizing and has a soft peg that seems very unlikely to fall more than 5% relative to ETH even in times of high stress.
  3. There will be a LOT of ETH ejected from validators into the deposit pool during correlated slashing, so burning rETH will be relatively simple and will actually minimize ETH capital inefficiency.
  4. It provides another way to gain yield on rETH which drives rETH demand and buoys RPL price.

How much backing: I will base this on a being able to fully insure a 20% correlated slashing (reasoning will be under minipool below), or 6480 ETH currently. For LEB 8, I target 15% (28,000 ETH after 3x TVL increase); for LEB 4, I target 10% (40,000 after 7x TVL increase)

Incentive structure: Rewards would be 10% of 660,000 NO RPL emisions initially. 66,000 RPL rewards, at the target pool size of 6480 ETH, this equates to 15% yield per year at current levels in RPL. The stakers lose if correlated slashing occurs; they win if (as is anticipated) no large correlated slashings occur. If a lower number of people are willing to stake rETH, yield will be higher but risk will be concentrated; if a larger number of people stake, yield is lower but risk is also spread out. I expect a higher proportion of NO rewards will be required with LEB.

Nuts and bolts: rETH can be staked at any time, but can only be unstaked at the end of any rewards period. rETH unstaking will be frozen if a correlated slashing event is occurring (ie, during the 2 weeks after) until after payout has been resolved.

Pool #2: NO insurance secured by RPL

Reason: Ethereum slashing was set up for total loss of ETH bond for a ~33% slashing- this is the level that has the potential to harm the protocol. However, because Rocket Pool NOs effectively operate on leveraged ETH, they will have total loss of ETH bond at ~16% correlated slashing. In LEB 8 they will have total loss at ~7% correlated slashing, and in LEB 4 they have total loss at ~3.5% correlated slashing. As such, almost all NOs in LEB 8, and all NOs in LEB 4 will have the potential for complete loss of ETH through no fault of their own if a client bug happens. This also sets up a moral hazard, as if any client over 7% (LEB 8) will cause total loss, NOs have no incentive to choose a 10% client over a 20% or 30% client. We currently have no accurate way to track or incentivize NOs who use minority clients. This pool is intended to drastically reduce risk/volatility for NOs and incentivize use of smaller market share clients, which ultimately protects rETH and lowers the insurance needs of Pool #1.

How it works: Node operators have an option to stake into a RPL NO insurance pool. There is no maximum or minimum amount. In the event of a slashing event, each affected minipool is responsible for the 1 ETH base penalty (ie, the entirety of isolated slashings). After this, most (75%) of the correlated penalty up to pre-defined maximum is reimbursed in the form of RPL by the RPL insurance pool to the node operator. The maximum is based on whatever client Ethereum market share is desired (I will target 15% for now, 10% for LEB 8, 7.5% for LEB 4). After this pre-defined maximum, the minipool is responsible for 100% of additional penalty.
This RPL is not auctioned or disbursed immediately; rather, it stays in the staking pool and a portion is withdrawable every reward period for the next 12 reward periods. This has several benefits. First, it means the staking pool is immediately ready for another correlated slashing, and generating yield for the new owner. Second, it prevents catastrophic RPL price collapse as only ~8% will be released per month. Lastly, because this RPL is going to NOs, they may use it for running minipools/staking and not even sell it at all.
Reimbursement from this pool will need a social layer component to prevent misuse: specifically, slashings that are part of malicious attack would not be reimbursed; slashings that are due to failure to upgrade client in a reasonable time would not be reimbursed; maybe a few other examples. But the expectation is that most correlated slashings are faultless and will be reimbursed.

How much backing: Maximum payout will be with a 15% slashing: 10.5 ETH/minipool * 15% of 9000 minipools = 14,121. With LEB 8 this will be

Incentive structure: 15% of 660,000 RPL rewards annually. Given a target 14,121 ETH equivalent (1,008,000 RPL), this is 9.8% yield/year. In LEB 8, target 10% correlation, 2X number of minipools = 12,555 ETH equivalent. In LEB 4, target 7.5% correlation, 4x number minipools = 10,732 ETH equivalent. Overall, the security payment may fall with LEB

Nuts and bolts: RPL can be staked at any time, but can only be unstaked at the end of any rewards period. RPL unstaking will be frozen if a correlated slashing event is occurring (ie, during the 2 weeks after) until after payout has been resolved.

Benefits

rETH pool #1 provides much more security against correlated slashing than the current system for 10% the capital

Neither pool has min or max caps as the individual contributing is irrelevant; APR will rise and fall based on market forces

Provides another use case for rETH, since we feel TVL will be rETH bound in the medium term

Incentivizes minority clients by providing ‘free’ 75% slashing insurance, even if we can’t determine who has a minority client

Greatly reduces variance for NOs by decreasing the risk of total loss if their client has a fatal bug

Leaves existing individual RPL staking relatively unchanged and still be useful for MEV theft/abandoned validator

RPL insurance pool #2 keeps RPL in the hands of NOs, which is intended to reduce price impact

Fits in well with other ideas regarding collateral/RPL rewards, like knoshua’s https://dao.rocketpool.net/t/redesigning-no-rewards or @nonfungibleyokem’s https://dao.rocketpool.net/t/maximum-collateral-should-be-reduced-from-150-to-100 by allowing unlimited RPL whale staking, while reducing cost so more RPL rewards can be awarded per minipool.

Downsides

10% of current NO RPL rewards go toward rETH holders, who likely will sell them

If the inducements are successful, RPL insurance pool will need higher levels of collateral as Rocketpool minipool client diversity will look very different from Ethereum client diversity.

Back-to-back correlated slashings would bankrupt the rETH pool; a double correlated slashing would cause the initially slashed NOs to lose all their insurance payout. However, I think we have to assume back-to-back correlated slashing events are extremely unlikely, and are not worth preparing against.

RPL Pool #2 has no way to different someone who has an isolated slashing in the middle of a correlated slashing; this could lead to some rare fringe cases (NO has lost withdrawal keys, decides to self slash in the middle of a correlated slashing to get some return on their ETH)



Reply to knoshua

Thank you, those are excellent points. Sadly I know very little solidity to determine the most efficient way to detect correlated slashing. Practically, the isolated slashing penalty is applied immediately and the proportional (correlated) is ~18 days later, so I would assume identifying any validator ejected with that second penalty could trigger rETH burn from the pool; it likely would benefit from some sort of oracle, although since these events are super rare, very public, and since instantaneous recognition (<24 hours) is not necessary, it seems that a contract that could be called by any observer on #trading may suffice.
And I agree, for #1 the most elegant would be to just send rETH to a burn address/reduce supply without plan for reimbursement; I wasn’t sure if the rETH:ETH ratio accurately reflects rETH sent to the burn address without interacting with the burn contract. If you know I’ll update the proposal!
For #2 the social layer is a issue, and it’s definitely not the kind of thing I’d want to send to snapshot (particularly after this week). I was thinking more like oDAO or GMC; if a majority decides the correlated slashing (the entire event, not a per minipool decision) was malicious or through user fault (defined ahead of time), no reimbursement would occur. However, unfortunately I think this pool cannot be automated/permissionless as those decisions have to be made by humans for the time being.
I think encouragement of malicious behavior is essentially negligible as the correlated penalty paid as a percentage of ETH by the minipool remains higher than solo validators, even accounting for reimbursement if the actor somehow went undetected. Also, the pool pays progressively less once correlation is over 15% (or my goal 10% in LEB 8, or 7.5% in LEB 4), so by 30% it pays nothing- and an attack with fewer than 30% validators I think has minimal risk to Ethereum in terms of delayed finality etc.

reply to mao

My little quibble, whether it is relevant or not you can decide, is that RPL was not required for NOs until a relatively recent tokenomics change (see the white paper). So this would be somewhat a return to the prior conception. However, I completely agree that we could use RPL if we could get a good liquidation process. Currently we have liquidity in the realm of 800 ETH (this is on rocket scan, i can’t verify, nor can i tell what it was prior to the most recent RPL downturn); to handle a 15% (moderate) correlated slashing we would need liquidity in the realm of 10,000 ETH where liquidity could not be pulled after a correlated slashing was identified. I can’t think of a way to do this less expensively than my insurance pools, but if you have ideas to inexpensively increase downside liquidity, that’s what this thread is about!

reply to valdorff

You certainly don’t have to take a step back, i’ll bold that part of my original question, cause i tend to write a lot and obscure important parts. Not preparing for black swans is a very reasonable choice- maybe even rational; I am not currently preparing for a nuclear war, although the chance is above 0. However, since RPL collateral is not needed for abandoned validators (they get booted at 16E), and won’t come into play until like the 13th MEV theft for a given minipool (so 2026), saying that we don’t need RPL for black swan events is like (i feel) saying that we don’t need RPL to secure the protocol. And if no one thinks RPL is useful for security, we should consider @knoshua’s proposal to just treat it as a flat access fee/reward to run a minipool. And we should definitely not have it in the Rocket Pool explainer that RPL is “insurance,” or use it as a selling point that RPL provides additional security to rETH, if we know that is false.

Regarding LIDO, the insurance pool is not for black swan events, it would be woefully inadequate. From what i can reconstruct from: https://research.lido.fi/t/redirecting-incoming-revenue-stream-from-insurance-fund-to-dao-treasury/
it is a fund to reimburse if a single large Lido validator gets 30% of nodes slashed, or a small validator gets 100% slashed (essentially both are isolated slashing). Sort of like if Whatwall migrated and got slashed in the process. They don’t have a plan in place if one of their large validators gets 100% slashed, because the correlation penalty starts to kick in, and they certainly don’t have one for a systemwide client bug. In those cases, stETH takes the loss.
We are better than Lido from LSD security and this is one of many reasons i think rETH is far superior to stETH, but we are also paying a LOT more already (ie about 26% of our TVL is RPL ‘insurance’ compared to their 0.1% bond. And because we are operating on an insurance per minipool basis, on average only 8,000 of that 40,000 ETH would be available for any particular correlated slashing. So 2x more collateral for 30 times less LSD (60x improvement), paying 260x as much; still doesn’t seem like a good deal to me. I am hoping to see some more bang for that buck.

reply to yorickdowne

Love your work on ethstaker. small mancrush. carry on.

reply to cyberhorse

This is how I see rETH- a premium product where its security through decentralization and protection of staker value is the gold standard for the industry. I think this consumer protection is extremely attractive to (particularly) institutional investors. Otherwise, rocket pool is just a LSD that charges higher fees and has fewer DeFi interfaces.
However, currently our entire insurance comes from node operator ETH bond, which is going down in LEB 8 and LEB 4.

reply to jasperthegovghost

If I am understanding correctly (which is a rare occurrence), you suggest two separate rETH pools, where the senior tranche gets higher yield but is first up for slashing, and the junior gets lower yield for lower risk, only being used if the senior tranche is liquidated? This seems like an excellent addition so investors can self-select based on risk tolerance, if with some added complexity.
In terms of cost/reward, my calculation is that for 10% of our NO RPL inflation, we can offer ~13% yield (in addition to rETH appreciation) for a pool large enough to sustain a 20% correlated slashing penalty (single tranche); this reward seems fairly commensurate to higher risk defi applications, and I’m not sure the risk is much greater than the counterparty of a perpetuals exchange; plus currently rETH really has no substantial defi integrations besides what we are paying for.
My take is that we are already paying these emissions to NOs (ourselves) for insurance/collateral, but we are not actually getting any benefit; and if we are expected to be rETH bound, then finding ways to drive rETH demand, even at the cost of NO demand (lower emissions) will help the protocol.
Could you explain the statement: “introducing RPL rewards would broaden but not strengthen the insurance.”? Is there another way that rocket pool can incentivize the junior/senior tranche?

Both interesting designs. For the rETH pool I’d like to know how a black swan event is detected and coverage is triggered. Can we do it without oDAO?

For the second pool I’m worried about the social layer part or the implications for attacking ethereum.

Just wanted to point out that liquidation of rETH to cover rETH is not necessary. It suffices to destroy the backing rETH. Reducing rETH supply is equivalent to increasing ETH backing.

1 Like

Rocketpool was built with a clear concept of RPL used as additional collateral. We have seen what even something comparably irrelevant like the recent LEB8 Collateral discussion stirred up. I think the main approach should be to do everything possible/reasonable to strengthen RPL in market cap and liquidity and improve the efficiency of the collateral liquidation process as much as possible to enable it to provide the security it is supposed to. This in my opinion is the healthiest option and does not exclude any of the other meassures.

What shouldnt be done is to errode the RPL tokenomics by building solutions arround it instead of by utilizing and improving it.

Can we take a step back and discuss why we need to improve our black swan coverage?

Even if we used strictly LEB4s and had zero RPL, we would currently have almost 40k ETH protecting our 2% market share vs Lido’s ~5000 ETH covering their 30%. Is there a strong reason that being 100x better than the market leader is insufficient?

6 Likes

Is there a strong reason that being 100x better than the market leader is insufficient?

Like, I realize that’s, like, rhetorical, but, like, can I pretend it’s a real question?

In which case: Yes 100x %. Lido is permission-ed. 29 operators all of which are KYCd. So their insurance is more layer-0: “We will destroy you, your reputation, your business, AND your family, if you cross us. Now, relax. We are family. Kiss the ring.”

RocketPool is permission-less, anyone can be a node operator. Which means we need to work a little harder at protecting the protocol and stakers when an operator screws up.

Source: Am both RocketPool oDAO member and Lido node operator.

1 Like

We aren’t really talking about NOs crossing us here. It’s about something like 10% of all ethereum validators being slashed because of a crazy client bug or something. Probably something like this would have little to do with the NOs choices, so I don’t see why a professional KYCed operator would be less likely to be affected.

1 Like

It was absolutely not rhetorical. I want a solid justification for spending money on something where we trounce our competition.

Fwiw, I don’t think Lido NOs would get destroyed if there was, eg, a client bug that caused a black swan. We’ve twice seen a Lido NO go down for hours; there have been zero repercussions I’m aware of. In case of something like a client bug, there’s even a credible “not my fault”. I don’t think layer zero is getting to play here.

I have seen no evidence that Lido is better at preventing an issue from happening, and it’s clear to me that rETH holders would take less damage than stETH holders.

If you have some insight you can share about how Lido is more black swan resistant for some reason, that would be relevant. It would need to be a strong effect to make up for the 100x more damage taken.

To me the real issue re Black Swan protection is that Rocketpool NO’s are the insurance policy that protect rETH investors - firstly because its their ETH which gets slashed first and secondly because they have RPL at stake.

To my knowledge no other staking provider has such a level of insurance protection for depositors. And yet I do not see anywhere in our marketing any reference to it or any comparison to the level of protection offered by our competitors.

One issue which becomes vitally important when an institution is researching where to invest is the level of protection provided for their investment. Rocketpool would normally fail the normal credit checks as the company is tiny compared to the level of funds potentially being invested. Thus under normal circumstances a big institution would pass us by.

However if it became obvious that by using the Rocketpool structure every retail investor has insurance protection provided by the shareholders of the operator we might be viewed differently. Even more so when it is pointed out that the protection is not just a promise to pay, but ETH locked in the validator as well as RPL locked in the node.

Which of our competitors can offer that?

1 Like

I think there might be a more capital-efficient variant of this. What if we introduced junior and senior tranches for rETH? rETH deposited in the junior tranche would direct some of its rewards to the senior tranche, however, the senior tranche would assume all slashing risk for the junior tranche.

The problem both of our variants face is that we will have to compete with DeFi. In my case, users will have to choose between high yields in DeFi or slashing protection in the vault. Introducing RPL rewards would broaden but not strengthen the insurance. In your case, the protocol has to pay to make the insurance pool yield competitive with DeFi. RPL payments here would deepen but not broaden the coverage.

One path could be to start with the junior and senior tranches and then phase into only the senior tranche as we develop the protocol further.

I’ll hit these points…

  • Abandoned validators won’t autoexit losslessly for LEB8s (until we have forced exits perhaps)
  • MEV: I think this needs to change to repay the “theft”, not just get penalized. Also, RPL should get penalized first so we don’t lose our preferred ETH collateral. This is theory ofc, not implemented atm.
  • RPL can’t currently be effectively liquidated for the minimum if 50% of validators are slashed.

I think only the minimum is used for security (and even then we have work to do per the top 2 bullets). As I wrote in my initial forum post about LEB max collateral, I believe the max is serving roles like privileged speculation, voting, and friction.

I like knoshua’s idea, with my main uncertainty being that it’s a departure from expectations folks may have joined under.

@Valdorff Thanks! I agree with your point #1 in LEB 8. But currently, today, and ongoing for any 16E minipools:

Abandoned validators don’t use RPL, MEV theft doesn’t use RPL. And forget 50% slashing (which is an Ethereum ending type of event), based on my calculations :nerd_face: RPL is not touched in correlated slashings <16.1%, and even if LPs don’t pull assets we won’t have liquidity to handle a correlated slashing of >16.7% without taking a short trip to 0. But maybe the numbers on rocket scan (~800ETH of downside liquidity) or my understanding of the auction process are wildly off.

This is why I think it’s somewhat disingenuous for the Rocket Pool docs/our outreach to refer to RPL as collateral or insurance, as it is really unable to act as either by its own design. I think RPL’s fundamental use cases are the key to its value; speculation, voting, and lowering friction do not seem to add much value at all (although ENS holders may disagree).

Regarding your comment to Yorickdowne, Lido is not better at black swan protection; they have no black swan protection. Zero. If such a thing happens, it will be entirely bourne by stETH holders. But I also think Lido’s relationship with stakers is different from ours; maybe my view is idealistic. stETH holders are customers, and are getting the Walmart brand of decentralization/insurance, and are paying less fees for that. rETH holders are part of our community. So if we can get better protection for our community members for a small cost, to me it is worthwhile.

So is your position:

  1. correlated slashing events are so unlikely it is not worth the tradeoffs to mitigate them
  2. we already provide optimal correlated slashing protection including in LEB without needing RPL
  3. our black swan protection is not entirely sufficient, but we only need to be ahead of our competition; any additional costs are wasted as they do not generate more rETH demand
1 Like

I’d say my position is:

  • We should enable actual RPL collateral for MEV theft and leakage scenarios. There’s no reason not to have these.
  • We should consider using RPL collateral for egregiously bad performance.
  • The costs of mitigating black swan events are so high that the cure is worse than the disease.
    • If competitors are doing better than us, there may be a market reason to do this even if not truly +EV. In our current reality, we can already market our massive advantage, so there’s little marginal benefit here.
1 Like