Dynamic Cluster Reconfiguration

Overview

The Blueworx Platform can be configured to allow BRM servers to be added and removed dynamically without shutting down and reconfiguring the Blueworx Platform.

​

Enabling Dynamic Cluster Reconfiguration

To enable dynamic cluster reconfiguration the following entries must be added to zoo.cfg on each BRM.

reconfigEnabled=true
standaloneEnabled=false
skipACL=true
After restarting BRMthe zoo.cfg will be automatically migrated to the new ZooKeeper format.
$BRM_CONF/zoo.cfg
This file contains the standard zoo.cfg without the server entries. The file can still be manually edited however servers can no longer be directly modified.
$BRM_CONF/zoo.cfg.bak
This file contains a back up of the original zoo.cfg from before the migration. The file can be used to restore the original configuration. This is helpful when dynamically adding a server if a mistake is made.
$BRM_CONF/zoo.cfg.dynamic.*
This file contains the servers and is managed directly by ZooKeeper. Everytime a server is added or removed the zoo.cfg.dynamic is recreated. This must NOT be manually edited.

​

Querying Cluster Members

To query configured cluster members the brm --server command is used. This will list all configured BRM servers rather than all operational BRM servers. For example:

brm --server

 Running commands: --server
-----------------------------

1=1.2.3.4:2888:3888:participant;0.0.0.0:9696
2=1.2.3.5:2888:3888:participant;0.0.0.0:9696

Each line represents a single BRM server and has the following format.

<myid>=<address>:<port1>:<port2>[:role];[<client port address>:]<client port>
myid
The myid of the server, this MUST match the $BRM_HOME/zookeeper/data_dir/myid on the server.
address
The address of the BRM server.
port1
Port for following the quorum leader. This is used internally by ZooKeeper.
port2
Port for quorum leader election. This is used internally by ZooKeeper.
role
Optional role of the server. If specified this should be participant.
client port address
Optional address for clients to connect to. This defaults to 0.0.0.0.
client port
Port for clients to connect to. This must match the port configuration option in the brm section.

​

Adding a BRM Server

A BRM server can be added to an existing cluster. This is achieved with the following steps.

  1. Create $BRM_CONF/zoo.cfg on the new server. The configuration file must contain both the new server definition and at least one existing cluster member.

    For example to add a server 1.2.3.6 (as server 3) to the above example the $BRM_CONF/zoo.cfg should be:

    standaloneEnabled=false
reconfigEnabled=true
clientPort=9696
dataDir=/opt/blueworx/rm/zookeeper/data_dir
syncLimit=2
initLimit=5
tickTime=2000
skipACL=yes
server.1=1.2.3.4:2888:3888
server.2=1.2.3.5:2888:3888
server.3=1.2.3.6:2888:3888

  2. Create $BRM_HOME/zookeeper/data_dir/myid with the correct server id.

    Following the example above on 1.2.3.6 the following command is executed:

    echo 3 > /opt/blueworx/rm/zookeeper/data_dir/myid

  3. Start up the new BRM server.

    This will cause the $BRM_CONF/zoo.cfg to become migrated. The server will temporarily join the cluster and receive update server details. The new server details will NOT include this server (as its temporary). So if the new server is accidently shutdown the $BRM_CONF/zoo.cfg must be restored from $BRM_CONF/zoo.cfg.bak

  4. Add the new BRM server permenantly to the cluster. This is achieved using brm --server "add=X=Y" where X is the server myid and Y is the server details. The server details can be found by executing brm --server and finding the new server entry.

    Based on the example above the 1.2.3.6 server can be added with the following command:

    brm --server "add=3=1.2.3.6:2888:3888:participant;0.0.0.0:9696"

    Note: The server must be added otherwise when the server is shutdown it will not reconnect to the cluster.
    Note: The double quotes are required otherwise the shell will incorrectly intercept the semicolon.
    Note: Only one server should be added at a time. If multiple new servers are added this can accidently create a split cluster. The new servers will create a new sub cluster alongside the original cluster.
    Note: If a mistake is made the new BRM server should be shutdown and the $BRM_CONF/zoo.cfg restored from $BRM_CONF/zoo.cfg.bak

​

Removing a BRM Server

To remove a BRM server execute brm --server remove=X where X is the myid of the server.

In the example above to remove the 1.2.3.6 server the following command is executed:

brm --server remove=3

Note: This will remove the server from the configuration of each server. The server will still be connected until shutdown.
Note: The brm --server remove=X must be executed on a running and connected server.
Note: It is important to remove the server before shutting the server down. This is especially important when going from two servers to one server. If the server is shutdown before removing it the single remaining BRM will not start. This is because the remaining server is part of a two server cluster and requires the other server to be running.
Parent topic: Redundancy