Rewards Tree Spec v9

I’d like to kick off discussion on Rewards Tree Spec v9. I’d like to see the following changes:

Changes to Specifications

  1. Rewards merkle tree file format must be fully specified
  2. It should be a binary format
  3. It should no longer contain the CID of the minipool performance file

The motivation is:

  1. Enable the oDAO to run alternative treegen implementations
  2. Remove the need for compression, which currently ties the resulting CIDs (which are part of the consensus vote) to specific versions of the specific compression library used in smartnode
  3. Make performance file generation optional (simply reduces specification work)

In more detail, the reason the oDAO cannot currently run independent implementations of treegen is that after building the tree, they must:

  1. serialize an unspecified performance json file with the exact same bytes as every other oDAO member
  2. name the file exactly the same as the other oDAO members
  3. compress that file with the exact same version of this library as other oDAO members
  4. compute a CID in a manner exactly the same as the other oDAO members
  5. serialize an unspecified merkle tree json file with the exact same bytes as every other oDAO member, containing the CID from the first file
  6. repeat steps 3 and 4 with the merkle tree json file

The zstd library used by watchtower specifically warns against this:

[…] the encoding should never be assumed to remain the same, and you should not use hashes of compressed output for similarity checks.

So, instead of using this compression library, we should just remove the need for compression in the first place. In earlier experiments I’ve done, binary formats had roughly the same file size as compressed json, or even smaller, which make them a good option. In keeping with ethereum practices, ssz seems like the obvious format choice.

The process, then, would become:

  1. Serialize the merkle tree ssz file with the same name as the rest of the oDAO
  2. Compute the CID

Minipool performance files are not mission-critical and will still be made available, but not necessarily on IPFS.

Additional tooling will be published to convert ssz trees back into json, and json trees will still be usable to claim rewards.

Process Changes

From v9 and onward, tree specifications, both for the generation of the tree, and the format, name, and ipfs tree of the resulting artifacts will be written as RPIPs.

Process note: Past rewards tree versions have required a sentiment vote and (somewhat inconsistently) a full pDAO vote to ratify them. Since part of this proposal is to begin codifying the rewards tree specs with RPIPs, I think we should just do a single poll for the RPIP itself, when ready.

I’m writing this thread now to kick off discussion. A poll will follow when the RPIP is written.


I fully support this. I would just add that I also want to see the rules around attestation inclusion made explicit (or at least explicitly defer to the Ethereum consensus specs) in v9, as discussed in discord.

Note that we have RPIP-51: Current Rewards Tree Specification calling for the RPIP you are proposing.

I’ve created a proof-of-concept.

Smartnode changes to support SSZ: Link
And a demo repo that tests and benchmarks it: Link

The demo parses a canonical, odao-generated rewards tree json file, serializes it as ssz, parses that ssz, and then serializes the result as json again. It makes sure that the json is identical.

Benchmarks show ssz to be approximately the same size as the compressed json files, but when testing parsing, a mainnet json file takes ~0.175 seconds, and a mainnet ssz file takes ~0.0003 seconds.

Generating proofs (which are saved in json, but not ssz), takes an additional 0.04 seconds.

I’m happy with this, so I’ll move on to writing a formal SSZ specification and subsequently a draft RPIP.


A draft of the ssz specification is available here: Link

Please provide feedback, if any, in this thread.

I reviewed this offline with Patches, we had some back and forth about the fact that the network is changing from a string (current JSON) to a uint64 (proposed spec) but we’ve worked it out. Since the JSON is what ultimately lands in the GitHub repo, and this isn’t a spec for the JSON, the JSON is going to keep the string around despite disagreeing with this spec a bit. Tools for converting back and forth will need to be able to handle that. Otherwise it seems good to me, I say ship it.

The RPIP draft is prepared and ready for feedback: Link

For the hands-on among you, a sample smart node implementation is available here: Link
A demo tool that can convert between ssz and json formats, as well as calculate IPFS CIDs is available here: Link

There are unit tests in both repos that validate different parts of the process.

1 Like

Sentiment Poll Time!

  • I support this change
  • I do not support this change and will comment below
0 voters