Ad hoc oDAO Audit

Governance
1

During the latest reward tree generation we noticed that Fireeyes has not yet upgraded to smartnode v1.7.1 (released December 5), because they submitted a merkle tree root based on rewards tree v1, instead of v2. That’s why I decided to have a quick look at how the rest of the oDAO is doing.

I looked at the RocketNetworkBalances smart contract. oDAO nodes count up all the ETH in the protocol that belongs to rETH and submit the result there every ~21 hours in order to update the rETH exchange rate. During the latest update, Consensys reported the totalETH to be 173797548491634763613513 (Consensys Submit Balances Transaction), while other nodes (outside of Fireeyes) reported 173797548491634763611383 (for example: Nimbus Submit Balance Transaction). This indicates that Consensys is running an outdated version of the smartnode software as well, which calculates the ETH balance differently.

Furthermore, I noticed that the last transaction from the Etherscan oDAO node was 6 days and 8 hours ago. Since then, there have been 15 opportunities for oDAO nodes to participate in reaching consensus: 7 submit balances, 7 submit prices and 1 submit reward snapshot. Given that it takes more than half of the nodes to submit to reach consensus, it is incredibly unlikely for the Etherscan node to miss out on 15 straight opportunities while performing well ( 0.5**15 = 0.00003051757). This seems to indicate that either the Etherscan node has gone offline during the last week or is experiencing significant performance issues.

Summary

It appears that both Fireeyes and Consensys have not yet updated to Smartnode v1.7.1. Etherscan may be offline or experiencing performance issues. I didn’t find any irregularities with the remaining oDAO nodes.

Questions for Fireeyes/Consensys

  • Can you confirm that you were not running Smartnode v1.7.1 at the time of tree generation/the last balance submission?
  • If so, why didn’t you upgrade to it when it was communicated that: “Oracle DAO members must update to this version before Rewards Interval 4, on December 22nd”
  • Can you tell us how you handle smartnode upgrades. Who is tracking new releases and how? Who is responsible for upgrading?
  • Assuming you did not upgrade to v1.7.1 in time, what did you learn from this incident and how are you looking to avoid it in the future?

Questions for Etherscan

  • What is your process for monitoring your node to ensure it is online and performing its duties?
  • Are you aware of any current or recent issues with your node? If so, when did you notice and what have you done to rectify them?
  • If there was an incident, what did you learn and how are you looking to avoid it in the future?
19 Likes
2

Thanks for this work @knoshua!

Additional questions I have for all oDAO members:

  • what criteria (eg performance issues, comms issues, etc) are worth a kick?
  • what criteria are worth a challenge?
    -when a member fails to perform their duties should there be an expectation of reparations? Should this be income in the problem interval, bond forfeiture, something else?

Also… Do all oDAO members have active forum accounts? If not, I worry community questions may not receive appropriate attention.

2 Likes
3

Just answering for myself:

Additional questions I have for all oDAO members:

what criteria (eg performance issues, comms issues, etc) are worth a kick?

I don’t believe it’s my place to establish this, but I’ll support criteria that are generally agreed upon by the community. I do think that two consecutive egregious failures probably warrant a vote. And I think it’s good governance to have multiple votes on the same topic if the same failures continue. i.e. a vote after the second and each additional egregious failure.

what criteria are worth a challenge?

Failure to submit price & balance data after a given threshold seems fair, maybe 10 days?

-when a member fails to perform their duties should there be an expectation of reparations?

Yes. Ideally, the protocol wouldn’t reward inactivity, but until that’s coded into the protocol an odao member who hasn’t performed the agreed upon duty ought to make both rEth and RPL holders whole.

Should this be income in the problem interval, bond forfeiture, something else?

I feel like it should be limited to the problem interval or time period. Bond forfeiture is pretty extreme and I’m not likely to support it unless an actor is actively malicious against the platform. In any normal case that we exit an odao member I’m going to plan to fully return their bond.

Also… Do all oDAO members have active forum accounts? If not, I worry community questions may not receive appropriate attention.

I’d like to suggest a final consideration: ODAO members are chosen for more than their ability to submit data, they’re also chosen as aligned stewards of the Ethereum ecosystem. It’s ideal that we don’t vilify people who are working toward the same goals we are just because they’re not focused on RP like we are. I think the ultimate goal should to be to maintain strong positive relationships with the wide Ethereum community as we harden the ODAO to perform necessary functions. Lets not create a name for ourselves as an aggressive or militant community, but rather a community who strives for best practice in staking.

5 Likes
4

I believe it is a mistake to mix oracle duties with relationship building. I think it’s very reasonable to expect oracle operators to be great node operators, given that we have plenty of those around here. We shouldn’t have to compromise on that and destabilize the protocol in order to maintain good relationships with the wider community. I’m in favor of clearly separating oracle operators and contract-upgrade controlling multisig members.

7 Likes
5

Absolutely - I should be clear that I don’t view “you are failing to perform this set of duties” as being terribly well correlated with “you are bad” (in general, for Ethereum, even for RP). But I do think “performing oDAO duties well” should be expected of oDAO members. To @knoshua’s point, it may be better to explictly separate out the 3 things we try to achieve at once: oracle duties, contract upgrades, public goods funding.

5 Likes
6

quick update:

3 Likes
7

Is there actually an ODAO management committee? If simple things like the latest smartnode version are not being updated promptly this suggests that there is either no effective notification process to ODAO’s of necessary tasks or more likely no checking process to see that they have been done.

If there are tools available to check that the ODAO’s are all up to date maybe they should be provided in such a way that ODAO’s can quickly check their dashboard in a manner similar to what node operators can do via Grafana. Perhaps NO’s could also have access to a Rocketpool wide ODAO state of health.

This may raise vilification issues such as that mentioned by Superphiz but on the other hand if I was running an ODAO I would be mortified if it showed my node with any kind of operating deficiency.

Going forward if we want to have investors trusting Rocketpool with their ETH I believe that such an ODAO performance indicator including some of the best known names in Ethereum can only give confidence to those who may think we are just a bunch of hobbyists. But only if the indicator shows that the ODAO is performing well.

1 Like
8

It’s good to recognize these three separate purposes for the oDAO and the issues that arise from having them combined. In an ideal world I agree it’d be better to have them split to begin with. But we already have an existing oDAO. Any resolution needs to take into account how we get from A to B (and if the cost of doing so makes sense, given A.)

Oracle duties and contract upgrades both require trust (in the current design.) Having these combined into a single DAO is efficient, even if it opens additional attack vectors (i.e. manipulating oracle duties to then profit from a malicious contract upgrade.) I think the reputation at stake for oDAO members is an effective enough safeguard though. I’d say we need a trustless oracle design to materially improve on things here. As far as I know, this is not yet possible but may be in the future with further Ethereum upgrades.

The whole ‘public good’ angle is definitely vague and problematic. It raises the question on whether the oDAO is/should be a charity or payment for duties rendered, and to what extent. Not all oDAO members are equal in this respect. Perhaps we could devise a framework where we make this explicit and split the oDAO payments between ‘payment for duties’ and ‘public good.’ All oDAO members would receive the same base payment for duties, and the pDAO could vote on how the ‘public good’ portion should be distributed between the differnent oDAO members.

@cyberhorse There are private communication channels that oDAO members use to coordinate. The community has repeatedly expressed the desire for more openness / insight in oDAO communication channels. Recently, oDAO discord announcements have moved towards a public channel, but no furher steps have been taken yet.
@langers What is the status on further opening up the oDAO communication channels? Is this still on the agenda?

There are no oDAO-specific monitoring tools I’m aware of (either private or public.) Could be an area of interest for a GMC grant or bounty?

1 Like
9

I think oDAO metrics are lacking, but they do exist.

Turn this on:

oDAO
oDAO1081×678 61 KB

And import this: [ODAO Member Dashboard | Grafana Labs]
(ODAO Member Dashboard | Grafana Labs)

4 Likes
10

As of 12 hours ago, Consensys oDAO node still appears to be on the outdated smartnode version. It is still submitting different balances and not contributing to consensus.

cc @taskiner

1 Like
11

oDao members should be the most engaged members of our community, especially for on-going developments given their share of RPL rewards. If they fail to monitor and perform basic duties, including failure to promptly respond here, we should work towards immediate replacement.

We all know there are many wanting to become oDao members who are significantly more involved. Leeway to members that are on the oDao likely due to brand recognition reflects poorly on RP, and even worse if we allow members to continue who failed their basic oDao duties. We cannot rely on reputation alone since it has proven to provide little to no benefit to the protocol.

oDao should fund their notification system and work alongside the team to ensure their daily/weekly/monthly duties are noted in advance if the Grafana dashboard/oDao channels are insufficient.

1 Like
12
  • Can you confirm that you were not running Smartnode v1.7.1 at the time of tree generation/the last balance submission?

No, we weren’t.

To be honest, I haven’t been tracking the “releases” channel. I just keep an eye on the “odao” and I didn’t see it there.

  • Can you tell us how you handle smartnode upgrades. Who is tracking new releases and how? Who is responsible for upgrading?

To date, it has been relatively casual eyeballing of the odao channel - especially when referenced. It is true this could be done better. We have over the last month added more resources to the team and this should enable us to engage more proactively with Rocketpool oDAO and be a more responsive member.

  • Assuming you did not upgrade to v1.7.1 in time, what did you learn from this incident and how are you looking to avoid it in the future?

I think we need to keep an eye on the “releases” channel and also be more engaged more broadly which is what we intend when we return in January.

3 Likes
13

Assuming that @kuhan is from Consensys their lack of attention to detail could be considered lamentable. Especially so as Consensys Ventures owns a 10% preference share in Rocketpool Pty Ltd, meaning that they have more control over Rocketpool’s development resource than most other oDAO members.

It would seem that the “oDAO management committee” is the oDAO Discord channel which appears to manage with a very light touch in terms of the accountability of oDAO members. Perhaps there are delicate egos involved. Maybe there is a difference when an oDAO member is a corporation represented by a salaried staff member rather then an individual with their own reputation on the line.

Whatever the oDAO management process is, it needs to be tightened up a bit in my opinion especially if we intend to appoint more oDAO members.

14

So, to be clear “ConsenSys Ventures” != “Consensys Software Inc”. Two completely separate entities.

15

Update: With the latest round of submit balances, Consensys is agreeing with the rest of the oDAO again.

4 Likes
16

Cryptomanufaktur is voting to scrub minipools:

scrub
scrub845×501 31.5 KB

Etherscan

A bit over 300 days ago, Consensys oDAO node also voted to scrub a bunch of minipools

It could just be a coincidence, but both incidents involve minipools from the node wallet whatwall.eth (aka thomasg). Also, Consensys was correctly submitting balances in between scrub votes, which seems to indicate that the clients were correctly synced.

Can we track down why this is happening and rule out any bugs in the vote scrub logic?

17

Yep we’re looking at this. It may be related to us using an Erigon archive node. Discussing tests with Joe to see how we’ll run an A/B against Erigon and Alchemy and supply the Erigon team with some actionable logs.

Edit: Erigon team found an eth_getLogs regression and released hotfix 2.35.1.

2 Likes