[Meta] Updated Guides - What Does a Node Operator Need to Know?

So one of the most prominent pieces of feedback I’ve seen with respect to node operation is that it’s intimidating, scary, and fraught with unknowns so people don’t want to try it. We also have a lot of success stories from people who felt this way and tried it anyway, and are now successfully running a node. This makes me think a lot of it comes down to education and explanation.

Below is my first pass at a list of things that typical users will need to know/do in order to be successful node operators, so we can try to establish why people are intimidated and what we can do about it. The intent is to comprehensively document everything involved and provide useful guidance, tutorials, and direction so more people make the trip from scared to successful. I’ll kickstart the process below, but a lot of it will be TBD to be filled by the community. At some point when this is complete, it may end up manifesting as part of the docs or an interactive decision tree website or something.

For Full Custody Node Ops (Home Stakers)

Things I will need:

  • A “good” ISP
  • A computer dedicated to staking
  • Reliable electricity

Things I will do:

  • Build a computer from scratch, or buy a premade one
  • Set up the BIOS with my customizations like auto-power-on
  • Install Linux on it, or use the preinstalled macOS if going that route (or use the turn-key setup provided by Avado / DAppNode)
  • Connect to the machine after preparing the OS (e.g., connect locally or find its IP address and SSH in)
  • Port forward in my router for P2P ports

For Offsite Node Ops (VPS / Cloud)

Things I will need:

  • A VPS / cloud provider
  • $ for monthly expenses

Things I will do:

  • Buy a remote machine instance, which comes with a premade OS
  • Configure the machine using the VPS / cloud provider’s settings UI

Common Ground for New Node Ops

  • Connect via SSH
  • Set up my OS appropriately
    • Deal with LVM / partitioning
    • Set up swap space / swappiness / cache pressure
    • Set up user accounts and permissions (if going Native Mode)
    • Set up WiFi (if using that)
  • Performance test to ensure it’s good enough
    • Overclock if necessary (Pi users unite)
  • Secure the OS (Mentor’s guide)

Common Ground for All Node Ops (e.g. including Solo Stakers)

  • All of the above plus…
  • Set up a secure data enclave (e.g. @ken’s Aegis)
  • Install the Smartnode stack
  • Configure the stack
  • Start it
  • Transfer ETH to my node wallet
  • Buy RPL and transfer that to my node wallet
  • Invoke Smartnode commands to stake RPL + node deposit
  • Monitor things and get notified of issues
  • Diagnose and resolve said issues when they occur

Let’s take these one at a time, Q&A style.

Questions a User Will Have

Prerequisites

What is Ethereum staking?

(Documentation here, the guides cover this)

What are all the parts? I see stuff like “EC”, “CC”, “BN”, “VC”, “Smartnode”, and “minipools” thrown around a lot?

(Better documentation and glossary of this)

What does a node operator do?

(Docs)

That sounds complicated. I see that I can just stake for rETH with one click. Why would I want to be a node operator for Rocket Pool instead of doing that?

(Detailed writeup that covers the benefits of node operation and the Smartnode stack)

I’ve heard about this “slashing” thing and this “leaking” thing that means I get penalized for doing a bad job. I don’t want either of those and I’m scared that they’ll happen to me.

(Explanation of terms, description of leaking vs. slashing and offline penalties, assurance of slashing protection)

I’m willing to learn a little more. What’s involved in being a node operator? How would I even get started?

(Lead-in to the next part)

Style Selection

Ok, so I should run a node at home if possible. How do I know if I can do this?

You need to answer these two questions:

  • Is your ISP / data plan good enough?
  • Is your electricity reliable enough?

If you can answer yes to both of them, then you can run a node at home.
If not, you should look at the VPS / cloud option.

Is my electricity reliable enough?

If your electricity is mainly on 24/7 and only goes off for a few hours during a severe storm once or twice a year, it’s reliable enough.
If it regularly goes off for hours at a time (like several times per month), it’s probably not reliable enough.

Is my ISP reliable enough?

To answer this, you need to know the following:

  • Your uptime (time connected to the Internet without losing service) should be as close to 100% as possible. Going offline for half an hour once a month for maintenance is acceptable. Going offline for half an hour every other night is not.
  • If you have a data cap, it will need to be at least 1.5 TB per month. Ethereum nodes use a lot of data, and you’re going to want to use some for your own home usage too. Ideally you should get a plan that doesn’t have a data cap.
  • You will need upload and download bandwidth of at least 10 Mbps (which is 1.25 MB/s). Note that this includes upload too, which a lot of ISPs try to hide. Any lower than that, and you could run into performance issues.
  • You will need to have access to Port Forwarding. If you have your own router, you likely have this capability already. If you use a router that your ISP gave you, you’ll have to look through their own documentation or contact their support to ask if they let you do this.
  • You need to make sure that blockchain activity is not prohibited by your ISP’s Terms of Service. Every ISP is different; the vast majority don’t prohibit this, but you should check with them all the same.

Local Node Construction

Ok, I have stable electricity and a good ISP! I can run a node at home. Do I need to build a computer? I don’t know how to do that but I’m willing to learn.

(Insert documentation about options like NUCs and building a computer here)

That sounds scary and I might break something. Can I just buy a computer that doesn’t require assembly, but still make sure it’s secure myself?

(Insert documentation about pre-built NUCs and mini-PCs here. We should probably pick a few “well-known” models and provide detailed steps on how to set them up as examples.)

That still involves construction! I literally want something I can take out of the box, turn on, and get going. I don’t want to screw around with a BIOS or installing an OS or anything like that. I just want to pay money to a company that I trust and get a working product. Whaddya got?

(Insert Avado and DAppNode links here? Maybe it’s the Rocket Pi’s time to shine?)

How do I set up the BIOS so it automatically powers on after a power failure (or other things)?

I don’t really know how to handle the gamut of BIOSes out there… suggestions encouraged here.

How do I install/prepare the OS?

(Step-by-step guide for one or two OS’s, probably Ubuntu and maybe macOS)

I want to use Windows because it’s all I know how to do and I don’t want to mess with Linux.

While we strongly encourage the use of Linux for security, performance, and cost reasons, we can show you how to use Windows if you really want to.

(Insert WSL2 tutorial here)

What’s SSH and how do I set it up? How do I use it? Do I need to use it?

(Description of what it is, tutorials of how to use it)

If you’re not comfortable logging in remotely and your setup allows you to have easy access to a monitor and keyboard plugged into the node, you’re welcome to ignore SSH entirely and just use the local terminal.

What’s Port Forwarding? How do I do it?

(We really need some examples of various router interfaces here, not just a little “figure it out yourself” thing)

VPS / Cloud Providers

Well, my electricity or ISP don’t cut it. I want to run a node, but it’ll have to be offsite. How do I pick an option?

(Docs about providers, pros/cons, upfront and monthly costs, testimonials from other node operators that use said providers, legal or country-based restrictions)

Ok, I picked a provider. What do I do next?

(Step-by-step walkthrough of how to set up a Smartnode-capable instance with each provider we recommend)

Common Ground Setup

Most of this is already covered by the guides because it’s less about Q&A and more like “OK, I have my setup, let’s do this… give me some instructions”.

Funding Questions

How do I get ETH?

(Tutorials for how to use a CEX / an on-ramp)

How do I get RPL?

(Tutorials for how to use a hot wallet and a DEX)

How do I send my ETH and RPL to my node?

(Tutorials for using CEX / hot wallet interfaces, with lots of disclaimers in caps about ensuring proper addresses and what not)

Existing Node Op Questions

I’m solo staking already and literally didn’t read any of the previous documentation. I just want to attach Rocket Pool to my existing stack. Help me out here.

(Dedicated guide for Hybrid Mode users here)

Monitoring and Troubleshooting

How do I back up my node?

If you have the 24-word recovery mnemonic for your wallet safely stored, you’re all set. You can reproduce everything in the event of a catastrophic failure by building a new machine and setting it up using these guides, then running rocketpool wallet recover to regenerate your wallet and minipool validator keys. You’ll lose a little bit of ETH on the Beacon Chain during the process due to the aforementioned leaking, but it’s not worth concerning yourself over.

Ok, but I want to take an actual snapshot / backup of my computer that I can just restore when I get the new one. How do I do that?

(Need some good docs and suggestions for processes here)

How do I monitor my node’s performance?

(Grafana, Beaconcha.in)

I don’t have time to watch stats all day, I just want to be notified when something goes wrong and I need to look at it.

(Beaconcha.in notification services, other stuff Mentor has enumerated in his new PR)

Help! My validators are offline and I’m losing ETH! What do I do?!

(Troubleshooting guide, ends with links to the Discord #support channel for live support)

Discussion

A lot of this is obviously placeholder. In some cases, I’m looking for suggestions and/or content people can provide for each of them. In others, I’m just trying to figure out if the questions even make sense and are the kinds of things people will ask, and if the workflow through them is the way to go.

Another meta-question is “how much of this can we automate?” Like, we could certainly ship a pre-canned OS image that the user flashes which takes care of a lot of the manual steps but it’s a lot of work to build and maintain. There are other companies already that are doing this kind of thing.

Perhaps we end up making dedicated guides for lots of different users: PC builders, NUC users, Pi users, Avado / DAppNode users, VPS and cloud users, hybrid mode users, and so on… some of which are not mutually exclusive. It’s a lot of content, but clearly there’s demand for it.

Whatever the case, I think having a “place” someone can go, which actively addresses these common questions in a step-by-step fashion, would go a long way towards assuaging the audience that’s interested but too intimidated. The guides in the docs website right now seem like a logical place to put this, though they’d likely need some restructuring to handle all of these forks. It’s entirely possible that there’s a better way to capture all of this and guide someone through the process.

Let me know what you think.

4 Likes

Here are some guides I’ve thrown together:
Patches’ Rocket Pool Guides

and here is a very popular guide for configuring tailscale

I would love to see these migrated to a dev-or-community-run site.

Additionally, a community run project (rocketpool.support) is very close to launching which should cover a lot of troubleshooting topics, such as Help! My validators are offline and I’m losing ETH! What do I do?!

Also, for the ssh section-

I’ve had to help a lot of NOs get through this. I think the existing guides do a good job of explaining what needs to be done, but a poor job of explaining the why.

Whatever new content we produce should be explicit about what files need to exist on the server, what their contents need to be, and what files need to exist on the client. This distinction is not understood by many new users.

Further the use of bashrc aliases is a bit crippling for some NOs. I’ve seen plenty of macbooks that refused to source .bashrc when sessions are open (perhaps because they’re zsh). I’d recommend examples of client ~/.ssh/config files, with a minimal viable setting:

Host mynode
	User myuser
	IdentityFile ~/.ssh/rocketpool
	HostName 192.168.1.254

Mac users may also want

   UseKeychain no
   AddKeysToAgent no

as they occasionally ask why they aren’t prompted for the key password after the first login, and want tighter security.

On top of this, most people go on to set up tailscale and need help handling the wireguard IP as well as the local IP.

Major pain has also arisen from subtle differences between openssh, the windows ssh command, and whatever unholy things apple did to ssh.

Honestly the nuances of the client are just as important as the server, and we don’t do a great job documenting them. This new thing, whatever it is, should let the user choose which client OS and server OS they’re using and it should present them with the proper steps for that combo.

Too much information at once could scare newbies.
As a former newbie , i would prefer an nice discord support team than a giant and complete Bible to read.
I can tell you if it wasn’t with the help of jcpl, I would probably give up on this amazing and pleasurable activity that is supporting Ethereum decentralization.

Another worry for potential new node operators is the possible cost of hardware. It should be pointed out that you absolutely do not need to buy the latest and greatest hardware to run a node. As we all know here a Raspberry Pie will do the job just as well as the latest Nuc i7. An 8th gen I3 Nuc runs fantastic with Nimbus.

Until participating in a Rocket Pool beta a couple of years ago, I had never run a Linux server and overall had little experience with Linux. It may be well understood by us, but I wonder if people first exploring the potential of being a node operator know they can do it on a testnet for almost no cost. The cheapest VPS on Contabo is like $7 / month and is more than sufficient for testing. This is how I started. The first step is ensuring prospective node operators know that they can try it out with very little commitment.

I wrote a guide documenting what I learned and published it to Medium. It is outdated and I’ve learned a lot since then. For mainnet I decided to go with a bare metal server hosted with Hetzner even though I saw more than adequate performance with Contabo. The guide could use a lot of updates, but it does cover the process from end to end. From selecting the VPS, initial setup, downloading and configuring your SSH client, etc. This level of detail may encourage more people to check out running a node.