Blockchain Tales: How to do NodeOps without DevOps, with Parity Substrate and Monk.io

Photo by Christina @ wocintechchat.com on Unsplash

The NodeOps and DevOps challenge

Infrastructure management is hard. But blockchain infrastructure is harder. Much, much harder. With blockchain infra, you have to ensure that your nodes are highly available and that their keys are kept safe and secure. The current go-to tools to achieve that are Terraform and Kubernetes (+ Helm). These tools are famously difficult to use and you often need an expert (a DevOps engineer) to use them correctly. DevOps talents are not easy to find and are expensive to hire (~$100k/year). Even if money is not an issue for you, implementation of a highly availabiliable setup will cost you time. This is a huge barrier for many Blockchain Companies, and for the blockchain community in general. Especially when running a robust and secure infrastructure is essential.

At Nodle, we are building an IoT-focused chain on top of Parity Substrate, in our opinion, one of the best frameworks on the market to achieve this. Up until recently our infrastructure was mostly composed of individual servers running nodes, packaged in a docker container. This made infrastructure maintenance inefficient and time consuming (imagine SSHing into 20 different servers to upgrade or change all your nodes one by one). Until then, we had been focused on building the chain and its modules. Now that we have to scale our node infrastructure to respond to the increase in traffic they receive, infrastructure management became an issue. We started looking for a DevOps but that’s when we came across Monk.io.

Our Solution

When we heard about Monk, we were immediately interested. Monk is a completely novel approach to cloud deployments. On one hand you can think of it as a deployment tool which replaces Docker Compose, Terraform and Kubernetes. On the other hand, it is like a platform by itself because it comes with a dedicated hub (they call it the “Monkhub”) — a hub full of ready to use components and stacks that are prepared and configured by experts. It isn’t a wrapper around Kubernetes nor a managed service but rather a new generation orchestration platform which helped us combine ease of deployment and management without sacrificing our freedom to architect our own node management stack as we wanted to.

With Monk, everything starts with a simple YAML “template” which we deploy on a “cluster” (eventually multi cloud). We ended up developing a simple set of “templates” to deploy and manage Parity Substrate-based nodes. Today, we are happy to make them available to the community and demo our usage of them.

Deploying the Nodes with the Monk Template

To demonstrate the capabilities of the node along with the template, we will show you how to create the following setup:

1. Deploy two load balanced nodes exposed behind an HTTP load balancer (later, we will show you how to turn it into an HTTPs one).
2. We then want to monitor the nodes, which means having a prometheus and grafana built in, with a preconfigured dashboard.
3. Make sure the node’s data is persistent in order to speed up future synchronizations.
4. We don’t want to deploy our servers ourselves; this needs to be simple for us.

This is what we propose you to setup

In the rest of this post we will assume you have a valid installation of monk, if that’s not the case please consider joining their public beta.
Once this is the case, make sure the monk daemon is running in the background by running “monkd” in your terminal.

The Template

We have created a template for you for this exact setup. It’s part of the substrate-nodeops repository, which also includes some docker image builders. Go ahead and clone it on your machine:

git clone https://github.com/ETeissonniere/substrate-nodeops
cd substrate-nodeops/monk

Once you are inside the folder you need to do one more step to install the templates on your machine:

monk load core/*.yml examples/lb_nodes.yml

Let’s explore a bit our example template. First we define two nodes ran by their container image, in our case we use a polkadot node syncing with the main network:

lbdemo-node-a:
  defines: runnable
  inherits: ./substrate
  containers:
    node:
      image: docker.io/eteissonniere/polkadot
      image-tag: latest
  variables:
    node-name: "lbdemo-gateway-a"
    chain: "polkadot"
lbdemo-node-b:
  defines: runnable
  inherits: ./substrate
  containers:
    node:
      image: docker.io/eteissonniere/polkadot
      image-tag: latest
  variables:
    node-name: "lbdemo-gateway-b"
    chain: "polkadot"

If you wanted to tune the node parameters or run another node you could simply modify the variables or the “image” value, you can check the docker images built from the same repo on DockerHub.

We then define another container for prometheus:

lbdemo-monitoring-prometheus:
  defines: runnable
  inherits: ./prometheus
  variables:
    target-nodes: <- get-hostname("eteissonniere/lbdemo-node-a", "node") `:${node-port}` " " get-hostname("eteissonniere/lbdemo-node-b", "node") `:${node-port}` concat-all

Note the line starting with “<- get-hostname”, this is how we indicate to prometheus which nodes to get metrics from, if we wanted to connect to a third node we’d simply add it there.

Then we do the same for grafana:

lbdemo-monitoring-grafana:
  defines: runnable
  inherits: ./grafana
  variables:
    prometheus-endpoint: <- "http://" get-hostname("eteissonniere/lbdemo-monitoring-prometheus", "prometheus") ":9090" concat-all

Again, we some of monk’s feature to tell grafana to which prometheus instance it needs to connect to.

Finally, we define a “process-group” which is a way to specify a group of containers to start so that we don’t have to start them one by one:

lbdemo:
  defines: process-group
  balancers:
    defines: balancers
    app-balancer:
      port: 9944
      type: http
      health-check:
        kind: tcp
        request: ""
        interval: 5
        response: ""
      instances:
        - eteissonniere/lbdemo-node-a
        - eteissonniere/lbdemo-node-b
  runnable-list:
    - eteissonniere/lbdemo-node-a
    - eteissonniere/lbdemo-node-b
    - eteissonniere/lbdemo-monitoring-prometheus
    - eteissonniere/lbdemo-monitoring-grafana

Note how it also let’s specify a load balancer easily, it would end up being more verbose and complicated with another tool. You can also modify it very easily to include some certificates and setup an HTTPs load balancer.

Creating the Cluster

Creating a cluster is very easy, we simply use the following three commands and follow the instructions:

monk cluster new
monk cluster provider add
monk grow

When it comes to the “grow” command we selected 4 machines of type “e2-standard-2” (on GCP) to create the cluster so that the nodes have some room and enough resources to consume. Don’t forget to remember the tag you choose as you will need it later, we used “demo” for or example.

If you wanted to, you could grow your cluster multiple time and even make it multi cloud by creating nodes on both AWS and GCP.

In order to check that the cluster was correctly created you can simply run “mncc cluster peers”. This is what we have so that you can compare:

Deploying the Nodes

Deploying the complete stack of nodes and their monitoring infrastructure is as easy as running the following (replace “demo” with the tag you chose previously):

monk run -t demo eteissonniere/lbdemo

Note how this will display a summary of the node’s IP addresses along with their load balancer, this will be useful later. You can also view what’s running on your cluster by using the command “mncc ps”, here is a sample output for your reference:

Testing

If you check the output of the “run” command, one of the lines will indicate you the public IP of the grafana instance (the one with port “80”). You can simply open it in your browser (default username and password are “admin:password”) and check the dashboard “Substrate Node Metrics” to get some details on your nodes!

Example Grafana dashboard for one of the nodes

You should also be able to query the load balancer’s IP to connect to the node’s websocket port.

This concludes today’s post! Don’t hesitate to join the Monk Open Beta or check the NodeOps repository on GitHub.