MEV and Penalty System

When Rocket Pool launched on mainnet, the situation around MEV in a PoS world was uncertain and the likes of Flashbots were still actively developing specifications for their solutions. All we knew for certain was that MEV would be present and we needed a way to ensure that Rocket Pool validators would share MEV rewards with the pool that supplied half the capital for their stake. Rocket Pool is designed to protect Node Operators (NO) from a malicious Oracle DAO, so minipool upgrades were designed to be opt-in. Therefore, we needed to include something at launch in the initial minipool delegate contract that could be used to punish non-compliant NO. Otherwise, adding this at a later date would require NO to opt into a new delegate contract that could penalise them. Which NOs could simply refuse to do and bypass any penalty system we implemented. So we added the mechanic of a penalty percentage which would be able to set per minipool by some yet-to-be-designed system along with a max penalty rate which could be set by the guardian account. The purpose of the max penalty was so once the situation around MEV became clearer, we could set this to a value which is just high enough to make cheating -EV and then we could take that power away from the guardian. Essentially, postponing the decision of what that value needed to be into the future.

With the merge fast approaching, we now have to implement the logic behind the penalty system. Both in terms of what amount of a penalty gets applied (contract functionality) and also how we detect non-compliant behaviour (watchtower functionality).

We have included in the merge readiness contract upgrades a system similar to the oDAO balance and price submission systems. In which oDAO members can come to consensus in favour of applying a penalty to a minipool. Each penalty received increases the penalty percentage for that minipool by some amount. There is also a buffer of 2 strikes before the penalty percentage starts to increase to reduce the likelihood of a misconfiguration resulting in a harsh penalty. The consensus percent and the penalty amount are both configurable values which we have initially set to 51% and 10% respectively. 10% was chosen as a trade-off as 1.6 ETH is greater than the majority of MEV opportunities and is a strong enough disincentive to encourage the desired behaviour but also doesn’t immediately destroy the validator’s capital.

For example, if a NO proposes a block that doesn’t meet the protocol requirements (i.e. incorrect fee_distributor), the oDAO members watchtower will detect this and vote in favour of applying a penalty to the relevant minipool. Once 51% of oDAO votes in favour, the penalty count for that minipool is incremented. Nothing happens at this stage. If another invalid block is proposed, the same thing happens and the penalty count is incremented again. But once again, nothing else happens. On the third invalid block, once the penalty count is incremented, a 10% penalty rate is set on the minipool. This 10% penalty is applied when the minipool exits. After the NOs share is calculated during distribution, 10% is taken off that value and is given to the deposit pool instead. Each subsequent invalid block increases the penalty rate by another 10% up to the max penalty rate (whose value is TBD).

We suspect that in the future this system might see significant changes. For example, with forced exits and the RPL-first slashing ideas coming out of the LEB research. What I’d like to focus on is the other side of this system. How we monitor the MEV and report it on-chain via the oDAO voting mechanism.

mev-boost by Flashbots

We propose adopting mev-boost by Flashbots as the MEV service provider for Rocket Pool. Flashbots is the leading MEV research project in the space and they offer the most prominent MEV product on PoW Ethereum today (80% of PoW hashrate accepts Flashbots bundles). We expect it to remain the dominant MEV solutions provider after the merge. mev-boost allows validators to outsource block production to third parties who specialise in MEV extraction and pay the validator who proposes the block for including it in the chain. When it’s a validator’s turn to propose a block, they query an mev-boost relay for an execution payload which pays the highest and then includes that in their proposed block. Behind the scenes the relay communicates with builders and searchers where an auction is held for the blockspace. Flashbots’ goal is to have a decentralised system of builders and relays in the future with some sort of reputation system keeping them in check. But the initial launch design has them operating the sole builder and relay on the network.

There is discussion on adding the ability to query the relay for which block they proposed for a given slot. This would allow the querying of Flashbots’ relay for each block proposed by a Rocket Pool validator to determine if they proposed a Flashbots block or not. (Recently proposed blocks · Issue #120 · flashbots/mev-boost · GitHub).

The mev-boost system is designed as a side car that falls back to local execution client block production in case a relay is down or inaccessible. However, this poses a problem, as distinguishing between a naive, locally-built block and one which contains MEV extraction is challenging. If we cannot distinguish between locally-built blocks and blocks that extract MEV then we cannot effectively punish and discourage cheating. A large NO could run his own searcher and build his own blocks that have back channel MEV extraction and it would be difficult to impossible to detect and therefore penalise. Or NOs could join some other non-Flashbots MEV network and extract MEV that way without detection.

Alternatively, forcing Rocket Pool NOs to propose only Flashbots built blocks and nothing else is not a viable option. As we cannot put the liveliness of Ethereum in the hands of a single centralised Flashbots relay. In time, as the Flashbots network grows to several relays and builders, this may become a viable option.

With these problems in mind, we propose a plan which consists of a short-term, mid-term and long-term solution to the problem.

Short Term

In the short-term, Rocket Pool validators are required to set their fee_recipient to their distributor contract address (or smoothing pool if opted in). We integrate mev-boost into the smartnode stack so that all Rocket Pool validators propose Flashbots-built blocks and fallback to local block production. The oDAOs will monitor the fee_recipient and penalise non-compliant validators via the penalty system mentioned earlier. We will monitor validators and if we discover new methods for extracting MEV that bypass the fee_recipient payment, we will include additional checks to the oDAO monitoring process to penalise these as well. This can be done retroactively so there is risk to even attempt gaming the system now knowing it could be detected later with more sophisticated methods. We can query Flashbots’ relay for every proposed block and keep record of which blocks are proposed by them and which aren’t. This will help detect malicious actors and allow us to react.

This short-term solution has some flaws:

  1. We are unable to force validators into extracting MEV on behalf of the protocol. It is perfectly legal in the protocol for NOs to only produce blocks locally and if NOs refuse to propose Flashbots blocks it would result in reduced yield for rETH holders. In practice though, the substantial increase in NO yield from MEV is a strong economic incentive to participate so it is unlikely to be widespread.
  2. Without being able to force validators to propose Flashbots blocks, NOs could opt in to the smoothing pool and receive benefits from it while at the same time not contribute to it. We could kick those not contributing but once again we have no way of knowing if it was an innocent situation and Flashbots relay was simply not available to propose a block for them. This is exacerbated by the fact that block proposals are a rare event so any leniency gives way for exploitation.
  3. There are sophisticated MEV-extraction methods that are difficult or impossible to detect and therefore punish. fee_recipient is the most common and simple way to pay validators for blockspace but there are other methods which are indistinguishable from a regular Ethereum transaction.

We don’t anticipate these flaws to pose a significant threat to the protocol in the short term as running a profitable builder requires significant scale that is not achievable by most participants. But an actor with enough time and resources could exploit our lower capital requirement for validators to increase their blockspace for their own MEV extraction operation.

And once again, NOs attempting to exploit this do so at a risk to their capital upon being detected.

Mid Term

In the mid-term, Flashbots’s mission is to build up a decentralised network of builders and relays. At which point, Rocket Pool can decide that the system is decentralised and has sufficient redundancy that we can rely on it entirely for block production and prevent locally-built blocks from being acceptable. At this time, all blocks proposed by Rocket Pool validators must come from one of a set of whitelisted relays. This set of relays would be modifiable by the oDAO so we can add and remove entries from it as needed.

Pros

  1. By removing the ability to locally produce blocks, we can know for certain whether someone is attempting to cheat the protocol and can punish accordingly.
  2. Smoothing pool participants that do not propose Flashbots blocks can be punished and kicked.
  3. Guarantees that minipools (and therefore rETH users) will have higher returns (projected to be almost double) than self-built blocks ensuring Rocket Pool remains competitive with other staking services.

Cons

  1. Although a decentralised network of relays is more robust, it’s still not as decentralised as everyone being able to build their own blocks from a public mempool.
  2. In black swan event, we could still negatively impact the liveliness of the Ethereum network.

Long Term

In the long-term, Ethereum is planned to feature in-protocol PBS. This is still an active area of research but Rocket Pool would need to adapt to support this protocol change.

12 Likes

Thank you, Kane, for the great write-up.

  1. There is a proposed penalty of 1.6eth per offence. Does this number change if the minipool exits with a lower balance than 32eth? Eg 24eth after leaking.

  2. This may be out of scope, but how do you see this penalty change for a 4eth collateral LEB minipool?

1 Like

Thanks for putting this together :slight_smile:

Two things

1.

Nothing happens at this stage. If another invalid block is proposed, the same thing happens and the penalty count is incremented again. But once again, nothing else happens. On the third invalid block, once the penalty count is incremented, a 10% penalty rate is set on the minipool. This 10% penalty is applied when the minipool exits.

Currently MEV rewards are ~0.02 ETH per block on average; If we have LEB minipools without force-exit withdrawal keys, the threshold of stealing just enough MEV (on avg) would be extremely low. Thus, to minimize risk, we should consider launching LEB once force-exit keys are available.

2. RE: Mid/Short Term

If i understood correctly, the trust assumption is that oDAO penalizes malicious MEV-NO.
What speaks against oDAO members becoming block builders / relay operators?

Pro arguments

  1. Potentially more profitable, since there are multiple relays
  2. Not depending on a single entity (FB)
  3. Potentially a more sustainable way to fund oDAO members (e.g. with a small fee for each block instead of paying through newly minted RPL)

Please let me know what you think :smiley:

disclaimer: i am an odao member

3 Likes

thanks Kane,

i concur that should be kicked, 3 proposals is 6/7 months of validating, or the node is down for 6 months (and not kicked by beaconchain?) or is malicious.
after the second strike, maybe every 12/24 months, the penalty in my view should be reduced so after 24/36 months of regular good behaviour penalty strikes will be 0

1 Like

We’ve been discussing Butta’s idea of using the oDAO as a relay/builder network. I like it. It could look like this.

  • oDAO runs relays and builders, but not searchers, since searchers are so very quant/AI/ML heavy
  • oDAO no longer gets paid in RPL, that goes to NOs now
  • oDAO gets a small cut of each block’s fees/MEV
    → Less sell pressure on RPL. oDAO needs to cover gas in ETH; makes sense to get paid in ETH

Technical note:

  • There’s a RocketPool-run relay load balancer (CloudFlare?) that distributes traffic to the various oDAO members, taking region into account. That way oDAO members don’t have to each be globally available, and the RocketPool relay as a whole is globally available
  • Unsure about builder duty rotation … to be discussed
4 Likes

I think this short term downside is further mitigated by the fact that the fee recipient is still enforced for local blocks. So the smoothing pool would miss out on Flashbots MEV, but still receive the benefits from priority fees earned by the a locally built block.

For the medium term, I have doubts that a sufficiently decentralized relay/builder ecosystem will materialize in practice. Others in e.g. the Ethereum R&D discord have also mentioned the possibility of ending up with just 1 builder. As it’s a very specialized job the best builder might be able to outcompete others for searchers’ business.
In my opinion we should plan for this possibility, meaning we might need to rely on the ‘short term’ solution until the long term PBS infrastructure becomes available.

There’s a lot to like here - it’s quite an aligned duty and immediately adds another relay/builder besides Flashbots.
I wonder if the oDAO will have the capacity to carry out these duties though. Block building (combining the most profitable searcher transactions) is still specialized, meaning they’ll need both beefy hardware and good building algorithms - not their core business. If they can’t build blocks competitive with other builders, then validators are still likely not to select them. Unless we want to force RP NOs to always use the oDAO builder (potentially costing them and the protocol some profit.) Perhaps it’s more feasible for the oDAO to only run the relay, since that’s mostly just a trust assumption, which they’re already well positioned for.
One other nag that I’ll keep repeating is that any and all additional oDAO duties further enshrine them in the protocol. And as far I know, the ‘end goal’ is still to phase out the oDAO. As long as this is not definitely off the table, we should expand oDAO duties with great restraint.

3 Likes

If they can’t build blocks competitive with other builders, then validators are still likely not to select them

It only requires one “good” block builder within the oDAO, which can be incentivized.
Meaning NO would get the best possible block and pay oDAO to find the best possible block - worst case is that all oDAO members use Flashbots and just forward the block, but even then, it would be the same trust assumption than directly depending on Flashbots.

Perhaps it’s more feasible for the oDAO to only run the relay, since that’s mostly just a trust assumption, which they’re already well positioned for.

Basically what I said above, they can run/depend on Flashbots.

One other nag that I’ll keep repeating is that any and all additional oDAO duties further enshrine them in the protocol.

RP will always need a trusted block builder; assuming that at some point all current oDAO duties are dealt with differently, then oDAO would just become bDAO (blockbuilder DAO)

  1. The proposed penalty is actually 10% of the NOs capital so only 1.6 ETH for a newly created minipool. So if the exit balance was 24, it would be 0.8 ETH ((24 - 16) * 0.1).
  2. We will need to adjust the penalty system for LEBs but LEBs will require a delegate upgrade so we will have the chance to modify it at that time. Right now we have to make use of the system we put in place at launch.
1 Like

Thanks for the response. I’ll be trying to word my response with support LEBs in mind.

So if the exit balance was 24, it would be 0.8 ETH ((24 - 16) * 0.1).

Shouldn’t we keep it at 10% of the protocol’s capital instead to try to recover as much as possible for the protocol?

As such, the payout during exit will be (eg. 16 ETH minipool, 2x offence, exit at 24 ETH) :

  • mev penalty = # of offences * 10% of protocol ETH = 2 * 1.6 = 3.2 ETH
  • payout = validator balance - protocol ETH - mev penalty = 24 - 16 - 3.2 = 4.8 ETH

We can’t change the mechanic now it’s built into the current minipool delegate which is an opt in upgrade for node operators. Any change requires some benefit to the node operator to incentive upgrading.

One potential issue with there being 2 unpenalized strikes per minipool, is that once withdrawals are enabled, an operator could potentially simply burn their ‘free’ strikes for each minipool/validator, pocketing the tips for each, exit the minipool, and then reenter and repeat.

It might make sense that instead of a financial penalty being imosed on those ‘free’ strikes, the minipool contract could have its exit distribution to the withdrawal address delayed for a period of time. Say, 2 months for the first strike, and 6 months for the second, or if there is a year of good operation on that minipool after the 2nd strike, the delay could be waived.

4 Likes

I don’t really understand this concern. Rocket Pool’s choice of block builder has no significant influence on the liveliness of Ethereum.

What are the concerns with Flashbots relay availability? Wouldn’t the oDAO be susceptible to the same availability issues and then couldn’t we use oDAO consensus to gauge Flashbots downtime and reasonably detect genuine fall back to local execution? It might not be a perfect system, but since there are also two free strikes, it doesn’t have to be perfect.

I’m a bit concerned with all the avenues we are creating for a sophisticated attacker:

  • rare, high MEV blocks can be profitably stolen entirely
  • as @NonFungibleYokem points out, free strike system can be gamed
  • if we only check for fee recipient and not if it’s a flashbots block, side channels can be used for MEV

Most people won’t be able to do any of this, but if we make this too attractive as a package, we will attract the people that can.

1 Like

I don’t really understand this concern. Rocket Pool’s choice of block builder has no significant influence on the liveliness of Ethereum.

I think you might have misunderstood. I am saying that forcing NOs to only propose Flashbots blocks and not allowing fallback to local execution would be an issue. If the Flashbots relay went down then all Rocket Pool NOs would only be able to propose empty execution payloads which would affect Ethereum’s execution liveliness.

Wouldn’t the oDAO be susceptible to the same availability issues and then couldn’t we use oDAO consensus to gauge Flashbots downtime and reasonably detect genuine fall back to local execution?

In the perfect world where availability is a black/white. But in reality, what could appear to be unavailable from one location isn’t necessarily true for others. The NO can’t distinguish between the relay being completely unavailable and just being unavailable to them. If it was just unavailable for the NO, then it would propose a locally-produced block thinking the oDAO will see it’s down too but then the oDAO would penalise it because, in reality, it was a localised issue.

Contrary to my initial reaction to this idea, this does seem like the best solution from what we have available. If the likelihood of the above happening is low enough then it still becomes +EV to propose the locally-produced block and risk the penalty.

Say the priority fees per block are 0.1 ETH (based on last 7 days) and we make the penalty 1.6 ETH for simplicity. Where E(x) is expected value and y is the likelihood of getting reported, we have:

E(x) = 0.1y - 1.6(1-y)
0 = 0.1y - 1.6(1-y)
y = 1.6/1.7
y ≈ 0.9411

So, expected value is 0 when the likelihood is about 94.11%. That means EV is break even when a false report happens every 1 in 17 times. As long as it’s less likely than that then it makes sense to propose locally-produced blocks when the relay appears down.

Most people won’t be able to do any of this, but if we make this too attractive as a package, we will attract the people that can.

I entirely agree and is similar to what I touched on in my post about a well-resourced actor. Which is why I was proposed moving to Flashbots-only blocks in the medium term. But I am now leaning towards the idea of oDAO monitoring the relay for down time.

1 Like

Gotcha, thanks for clarifying.

Yeah, it’s never going to be perfect, but the decentralized nature of oDAO might help here. It might even make sense to lower the threshold for oDAO consensus for this specific issue. If 25% or 33% of oDAO are reporting availability issues, it’s pretty likely that the relay had an issue in at least some locations. This is assuming that oDAO nodes are running in a pretty diversified set of locations.

1 Like

Can the oDAO increment the penalty by a non-integer or apply it multiple times for the same offense?
Under under the currently proposed scheme, a NO who ‘steals’ two 0.05 blocks will be penalized more than a NO who steals a single 14 ETH block, and far out of proportion to to any actual injury to rETH value. Thus there is little reason for the rational NO not to steal every 3.3+ ETH block, and then go back to honest validation until the next lottery block. This could be extremely lucrative for an actor with many minipools. Whereas an unfortunate or incompetent (non-malicious) operator will have disproportional penalties assigned in 99% of cases.
If, as you said, retroactive penalties can be assigned, can the increment percent be 1%, and a node operator who steals a 6.4 ETH block simply be voted 20 increments (ie the ~3.2 ETH taken from protocol rewards)?

The more efficient way to do something like that would be to let the oDAO vote for a variable penalty size. For example, if a NO steals a “0.05 block”, they all could vote for a 0.1 ETH penalty and if someone steals a 5 ETH block, they could vote for a 5 ETH penalty.

With something like that, we would need to be a bit careful with the lower end blocks. MEV-boost value of a block is only a proxy of MEV of a block. But I think we could get around that by having a minimum penalty value and a bit of a buffer for low MEV-boost blocks.

I’m afraid a change like that would have to wait for the next upgrade after Redstone at this point.

Another method would be to do something like “automatic force exit at 5 strikes, but you can be voted to force-exit at 1-4 strikes”. That could be oDAO initially, but no reason that couldn’t be real governance eventually. Pretty much just establishing some “this isn’t plausible” subjective metric.

I agree with knoshua and valdoff- the issue is that I don’t think we have a timeframe for forced exits, and I assume the contract was written so that you couldn’t have variable penalties for each offense (Kane can correct if I’m wrong, I’m which case knoshua’s idea is clearly superior). In fact, many of the issues with rocketpool was that they had to anticipate potential problems like 2 years in advance.
My clumsy attempt was to try to fit into the existing smart contract until we can strong-arm NOs into an upgrade.