How to make a cluster

You can install multiple hosts, if they reach the same redis instance and elastic (data plane), they behave like a cluster. If hosts are in different regions, they can reach the redis and elastic over wireguard There is no need directly communication between hosts.

Create Cluster

3 Master/Worker

We will install on 3 nodes that are both master and worker role

At least 3 hosts we need to perform a cluster.

	Host A	Host B	Host C
public IP	192.168.88.51	192.168.88.52	192.168.88.53

Prepare env

First select Host A as master. And copy some keys from A to others

On Host A

    ferrumgate --show-config-all

copy last line which starts as ferrumgate --set-config-all "ABCD"

On Host B and on Host C

on bash paste it

    ferrumgate --set-config-all "ABDEF"

    ferrumgate --restart

on each machine /etc/ferrumgate/env file, REDIS_PASS, REDIS_INTEL_PASS must be same, REDIS_LOCAL_PASS can be different. ES_PASS, ES_INTEL_PASS must be same

Creating

Prepare hosts for creating cluster, we will create secure link with Wireguard, first of let's set hosts public ip and port that each host can reach each other, below are local network, if you create hosts on different regions (USA,EUROPE,EMEA) use public ip and give permission over firewall

On Host A

    export CLUSTER_PUBLIC_IP=192.168.88.51 # your machine public reachable ip for master nodes
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.51 # your machine public reachable ip for worker nodes
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs1

On Host B

    export CLUSTER_PUBLIC_IP=192.168.88.52 # this ip is Host B public ip for master nodes
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.52 # this ip is Host B public ip for worker nodes
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs2

On Host C

    export CLUSTER_PUBLIC_IP=192.168.88.53 # this ip is Host C public ip for master nodes
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.53 # this ip is Host C public ip for worker nodes
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs3

Then on each machine

    ferrumgate --show-cluster-config

copy last line that starts with PEER= to a file like below

PEER=fs1/192.168.88.51:54321/169.254.254.203/c6acb8b6c89765fb33bf88f06b80a511ff073e2cfcc8191f7c1a223d0221fa1d PEER=fs2/192.168.88.52:54321/169.254.254.204/c6002ac8172af117ee32aa8d0d600130c38a12fe3c70dfeeb24a191c69b1706e PEER=fs3/192.168.88.53:54321/169.254.254.205/b01cef104a0971436328c2f271f5c9bc7f5df25f8e56fdc96c24a4fca0cda72b

All ips that starts with 169.254.. must be unique, please check them if they conflict you can use ferrumgate --regenerate-cluster-ip

then on each machine

    ferrumgate --create-cluster

and paste the collected lines and press ctrl+d, it will show some errors(you can not add this host to cluster) then on current machine (this is important)

    ferrrumgate --restart

Testing

Host ips are different on your configuration. Please get host internal ip on each host with

    ferrumgate --show-config CLUSTER_NODE_IP

Host A -> 169.254.254.203 (this is different on your deploy)

Host B -> 169.254.254.204 (this is different on your deploy)

Host C -> 169.254.254.205 (this is different on your deploy)

On Host A

    ping 169.254.254.204
    ping 169.254.254.205

On Host B

    ping 169.254.254.203
    ping 169.254.254.205

Adding a Master/Worker Node

Let's call this machine Host D with 192.168.88.54

On Host D

    export CLUSTER_PUBLIC_IP=192.168.88.54 # your machine public reachable ip for master nodes
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.54 # your machine public reachable ip for worker nodes
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs4

ferrumgate --restart

On Host A

ferrumgate --show-config-all

copy last line that starts with ferrumgate --set-config-all, and paste it to Host D

On Host D

ferrumgate --restart

On all nodes

ferrumgate --show-cluster-config

Copy last lines that starts with PEER= to a file

and paste it to all nodes

ferrumgate --update-cluster

ferrumgate --restart

Removing a Master/Worker Node

On all hosts that will be in cluster, get config with below command

ferrumgate --show-config-all

copy last line that starts with ferrumgate --set-config-all to a file

then on all nodes

ferrumgate -update-cluster

1 Master 2 Worker

At least 3 hosts we need to perform a cluster.

	Host A	Host B	Host C
public IP	192.168.88.51	192.168.88.52	192.168.88.53

Prepare env

Make hosts master or worker

On Host A

    ferrumgate --upgrade-to-master
    ferrumgate --restart

On Host B and Host C

    ferrumgate --upgrade-to-worker
    ferrumgate --restart

First select Host A as master. And copy some keys from A to others

On Host A

    ferrumgate --show-config-all

copy last line which starts as ferrumgate --set-config-all "ABCD"

On Host B and on Host C

on bash paste it

    ferrumgate --set-config-all "ABDEF"

    ferrumgate --restart

On Host A

    export CLUSTER_PUBLIC_IP=192.168.88.51 # your machine public reachable ip for master nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IP=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.51 # your machine public reachable ip for worker nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IPW=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs1

On Host B

    export CLUSTER_PUBLIC_IP=192.168.88.52 # this ip is Host B public ip for master nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IP=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.52 # this ip is Host B public ip for worker nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IPW=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs2

On Host C

    export CLUSTER_PUBLIC_IP=192.168.88.53 # this ip is Host C public ip for master nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IP=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORT=54321
    export CLUSTER_PUBLIC_IPW=192.168.88.53 # this ip is Host C public ip for worker nodes
    # if you are creating between regions
    #export CLUSTER_PUBLIC_IPW=$(curl --silent ifconfig.me)
    export CLUSTER_PUBLIC_PORTW=54320
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IP=$CLUSTER_PUBLIC_IP
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORT=$CLUSTER_PUBLIC_PORT
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_IPW=$CLUSTER_PUBLIC_IPW
    ferrumgate --set-config CLUSTER_NODE_PUBLIC_PORTW=$CLUSTER_PUBLIC_PORTW
    ferrumgate --set-config CLUSTER_NODE_HOST=fs3

Creating

On Host A

    ferrumgate --show-cluster-config

copy last line that starts with PEERW,

On Host B and Host C paste and press ctrl+d

    ferrumgate --cluster-join

    ferrumgate --restart

    ferrumgate --show-cluster-config

and copy last line that starts with PEERW

On Host A

    ferrumgate --cluster-add-worker

paste and press ctrl+d

Testing

On Host A

    ferrumgate --show-config CLUSTER_NODE_IPW

On Host B and Host C

    ping $HOST_A_CLUSTER_NODE_IPW

Removing a Worker

On master machine Host A

    ferrumgate --cluster-remove-worker

enter hostname or ip then press ctrl+d or just close worker machine

Firewall

Don't forget to give permission from your firewall,

master role hosts
1. 80 (tcp) (UI http)
2. 443(tcp)(UI https)
3. 54000-55000 (wireguard udp)
worker role hosts
1. tcp and udp that gateway uses (and default created ones 9999 tcp,udp)
2. 54000-55000 (wireguard udp)