Add and activate data nodes

Learn how to add data nodes to a cluster in an on-premises installation.

You can add one or more nodes to a deployment by cloning the configuration of one of the active nodes in the deployment. The new nodes are automatically started up as inactive (database not running) and can then be added to the cluster as data nodes.

This article does not describe how to prepare the physical or virtual hosts for the new nodes. For more information about how to prepare the hosts, see System requirements.

This procedure is carried out using Exasol Deployment Tool (c4) and ConfD.

Prerequisites

Set the config path

When you make changes to an existing deployment you must use the same configuration that was used when creating the deployment. If the configuration file that was used is not named config and/or is not located in the current working directory (on the host where you run c4), you must prepend the c4 commands with the path to the configuration file using CCC_CONFIG=<path-to-config>.

Example:
Copy 
CCC_CONFIG=./path_to_config/my_config c4 <command>

In the following examples, the default configuration file is used.

Download Exasol to the cloned node

If the deployment was created with the configuration parameter CCC_HOST_CLEANUP unset (empty) or explicitly set to true, you must download the software package of the currently installed Exasol version into the home directory of the node that you will clone.

If the deployment was created with the configuration parameter CCC_HOST_CLEANUP=false, the Exasol installation package should already be present on the cloned node. In this case you can skip this step.

Example:

  1. Check the value of CCC_HOST_CLEANUP

    Copy 
    CCC_CONFIG=config ./c4 config -K ccc_host_cleanup
    Found 1 results for query `ccc_host_cleanup`:

        CCC_HOST_CLEANUP (bool)

          Default value : true
          Current value : [empty]
    ...      

  2. Check the Exasol version

    Copy 
    CCC_CONFIG=config ./c4 ps
          N  PLAY_ID   NODE  MEDIUM  INSTANCE  DB_VERSION  EXTERNAL_IP     INTERNAL_IP  STAGE  STATE      UPTIME    TTL
      ┌─  1  c3275f84  11    host    -         8.34.0      203.0.113.11    10.0.0.11    d      running    04:35:16  +∞
    ...

  3. Download Exasol to the node that you will clone (node 11 in this case)

    Copy 
    CCC_CONFIG=config ./c4 connect -t 1.11/host
    VERSION=8.34.0
    curl https://x-up.s3.amazonaws.com/releases/exasol/linux/x86_64/$VERSION/exasol-$VERSION.tar.gz -O

When the Exasol package has been downloaded, use Control+D to disconnect from the host.

Procedure

Step 1: Prepare the hosts

Prepare the new physical or virtual host as needed depending on your existing deployment environment. For information about system requirements, see System requirements.

All nodes in the cluster should be assigned consecutive and evenly spaced static IPv4 addresses in the same subnet.

Step 2: Reserve IP addresses

On a jump host that has SSH access to the cluster nodes, run the c4 command CCC_CONFIG=config c4 host reserve <PLAY_ID> with the following command-line parameters:

Parameter Description

--ccc-host-reserved-addrs

Private IP address of a host that you want to add to the cluster.

You must repeat the parameter for each additional host, separated by a space: --ccc-host-reserved-addrs <host_1> --ccc-host-reserved-addrs <host_2> and so on.
--ccc-host-reserved-external-addrs

Public IP address of a host that you want to add to the cluster.

You must repeat the parameter for each additional host, separated by a a space: --ccc-host-reserved-external-addrs <host_1> --ccc-host-reserved-external-addrs <host_2> and so on.

This parameter is only required if the existing data nodes were deployed with public IP addresses (CCC_HOST_EXTERNAL_ADDRS was set at deploy time).

These parameters must be defined on the command line as part of the c4 host reserve command, they cannot be defined in the configuration file or as environment variables before the command.

For more details about parameter formats, see Parameters in c4.

If the CCC_HOST_RESERVED_ADDRS and CCC_HOST_RESERVED_EXTERNAL_ADDRS parameters are already defined in the configuration file, the values in the configuration file will be overwritten by the c4 host reserve command.

To find the PLAY_ID and other details of of the deployment, use c4 ps.

Example:

Determine the play ID of the deployment

Copy 
CCC_CONFIG=config ./c4 ps
      N  PLAY_ID   NODE  MEDIUM  INSTANCE     EXTERNAL_IP     INTERNAL_IP  STAGE  STATE      UPTIME    TTL
  ┌─  1  3a4a7d8d  11    host    c5d.2xlarge  203.0.113.11    10.0.0.11    d      running    04:35:16  +∞
  │   1  3a4a7d8d  12    host    c5d.2xlarge  203.0.113.12    10.0.0.11    d      running    04:35:16  +∞
  └─  1  3a4a7d8d  13    host    c5d.2xlarge  203.0.113.13    10.0.0.13    d      running    04:35:15  +∞

Reserve private and public IP addresses for the new nodes

Copy 
CCC_CONFIG=config ./c4 host reserve --ccc-host-reserved-addrs 10.0.0.14 --ccc-host-reserved-addrs 10.0.0.15 --ccc-host-reserved-external-addrs 203.0.113.14 --ccc-host-reserved-external-addrs 203.0.113.15 3a4a7d8d 
INFO[2024-06-18 08:53:46] Reserving new nodes for deployment: ****************************
INFO[2024-06-18 08:53:49] Done

Step 3: Connect to COS

Connect to the cluster operating system (COS) using c4 connect -i PLAY_ID -s cos.

Example:
Copy 
CCC_CONFIG=config ./c4 connect -i 3a4a7d8d -s cos

For more information about how to use c4 connect, see How to use c4.

This procedure uses the command-line tool confd_client, which is available on all database nodes. For more information, see ConfD.

Placeholder values are indicated with UPPERCASE characters. Replace the placeholders with your own values.

Step 4: Add the nodes to the deployment

To add the nodes to the deployment, use the ConfD job infra_instances_add with the following parameters:

Parameter name Data type Description
nid integer The ID of an active node that should be cloned to create new nodes
num_nodes integer The number of nodes that you want to create

The new nodes will be created using the configuration of the node specified in nid and consecutively numbered following the highest node ID of the existing nodes.

Example:
Copy 
confd_client infra_instances_add nid: 11 num_nodes: 2

The added nodes will automatically start up and reach deployment stage c (COS service running, database not running). At this point the nodes are not yet part of the cluster.

Step 5: Add the nodes as reserve nodes

To add the new nodes to the cluster as reserve nodes, use the ConfD job db_add_reserve_nodes with the following parameters:

Parameter Name Data type Description
db_name string The name of the database
node_list list List of node IDs (integers) to add as reserve nodes
Example:
Copy 
confd_client db_add_reserve_nodes db_name: MY_DATABASE node_list: '[14, 15]'

Step 6: Append the nodes to the data volume

The blocks on the new nodes must be appended to the current address space of the existing data volume. To append the nodes to the volume, use the ConfD job st_volume_append_node with the following parameters:

Parameter name Data type Description
vname integer name of the data volume
node_num integer Number of nodes
node_ids list List of node IDs (integers)

To get the volume name, use the ConfD job st_volume_list. For example:

Copy 
confd_client st_volume_list | grep name
...
name: DataVolume1
...
Example:
Copy 
confd_client st_volume_append_node vname: DataVolume1 node_num: 2 node_ids: '[14, 15]'

Step 7: Stop the database

If the database is running, stop it now using the ConfD job db_stop. For example:

Copy 
confd_client db_stop db_name: MY_DATABASE

Step 8: Activate the nodes as data nodes

To activate the nodes as data nodes, use the ConfD job db_enlarge with the following parameters:

Parameter name Data type Description
db_name string The name of the database
num_new_nodes string, integer The number of nodes to be added as active nodes
Example:
Copy 
confd_client db_enlarge db_name: MY_DATABASE num_new_nodes: 2

Step 9: Start the database

Start the database using the ConfD job db_start. For example:

Copy 
confd_client db_start db_name: MY_DATABASE

Step 10: Reorganize the database

After restarting the database, you must run the REORGANIZE statement to reorganize the database internally. This statement redistributes the data across the nodes and reconstitutes the distribution and partitioning status.

Copy 
REORGANIZE DATABASE;

You can run this command in the built-in SQL client in c4 using c4 connect -t <DEPLOYMENT>/db. For example:

Copy 
CCC_CONFIG=config ./c4 connect -t 1/db
> reorganize database;

Verification

To verify that the nodes have been added, use the ConfD job node_list.

Copy 
confd_client node_list