ConfD
This section describes the ConfD API in Exasol.
ConfD is a low-level API that can be used to perform all administrative actions in an Exasol system. You can interact with ConfD directly by connecting to a node in the cluster and using the confd_client command-line tool, or from an external tool using XML-RPC.
ConfD abstracts the administration and configuration tasks into jobs that can take various parameters. You submit a job by connecting to a node directly on the command line or using XML-RPC. The jobs are then orchestrated into a scheduler that executes them asynchronously.
In the ConfD scheduler, each job has its own pipeline in which several stages are defined. The scheduler saves the result for the job at each stage. The client that submitted the job can do additional actions while the pipeline is being executed.
The final stage depends on whether the job modifies the database or if it is a read-only job. The job is marked as finished
when the final stage has completed. The client can then query the result using its job ID.
Some settings have no dedicated ConfD jobs and are instead managed with the general_settings job. To know more about these settings, use the settings_list job.
The stages in a job pipeline
-
submitted: The job ID has been received and the job is waiting in the queue to be scheduled.
-
scheduled: The scheduler has examined the job and accepted it, assigned the configuration update (if necessary), and started execution in a dedicated worker thread.
-
executed: The commands are being executed. A job may consist of multiple commands which are documented in their own sections within the job file, including the result and output.
-
finished: All commands of the job have been executed and stored within the job file. This is the final stage for a read-only job.
-
committed: All nodes have received the changes made by this job, and the updated configuration has been synchronized. This is the final stage for a job that modifies the database.
Each database node contains a configuration file /exa/etc/EXAConf where most of the configuration settings that are managed by the ConfD jobs are stored. Manual changes to this file are normally not necessary and may affect the functionality of the database.
Do not change the EXAConf file unless you have been requested to do so by Exasol Support.
Authentication
ConfD supports basic authentication with a user and password. When you deploy a new Exasol instance, the deployment process creates the user admin with the password specified in the CCC_PLAY_ADMIN_PASSWORD
parameter in the c4 configuration. You can change this password using the ConfD job user_passwd.
You can also create a new user to interact with ConfD. For more information about creating users, see Add Users.
The new user must be assigned to the exaadm group.
If you created a new user, we recommend using confd_client instead of XML-RPC to avoid having to authenticate with a new user.
How to use ConfD
ConfD is available on all nodes. You can interact with ConfD directly on a node using the command line tool confd_client, or indirectly using XML-RPC to connect to the access node. The names of the ConfD jobs and parameters are the same regardless of if you use confd_client or XML-RPC.
The code examples in the Administration section of this documentation usually show how to submit jobs using confd_client. If you want to use XML-RPC to carry out an administration task, refer to the respective job descriptions in the ConfD reference (this section) for guidance.
confd_client
The command-line tool confd_client is available on all nodes and allows you to interact directly with ConfD in a terminal after connecting to a node.
ConfD jobs are executed with the user that is logged in to the node, which in most cases is root
. There is no extra authentication step when using confd_client in this way.
To use confd_client, connect to the deployment using c4 connect
, then run confd_client
in the cos environment.
To get more details about how to use confd_client, use confd_client --help
when you have connected to the node.
Syntax of confd_client commands
To run a job using confd_client you can use different syntax formats. In most cases, you can use any of the syntax formats for a command. For more information about the correct syntax for a specific ConfD job, refer to the job description in this reference.
In the following examples, the job db_start
is executed with the parameter db_name
using three different formats:
confd_client -c db_start -a 'db_name: Exasol'
# this format can only be used for jobs that only have a single parameter
XML-RPC using Python
You can interact with ConfD using an XML-RPC interface in Python. By default, the XML-RPC interface is available on port 20003 on the access node.
XML-RPC can only be used on the access node, not on a database node.
When using XML-RPC, you must authenticate with a user. For more details, see Authentication.
Example:
import xmlrpc.client
import requests, urllib3, ssl
import sys
import json
import pprint
# Disable SSL warnings
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
# Enter the IP address of the database access node and the COS port (default port: 20003)
database_host = '203.0.113.78:20003'
# Enter the username and password you are connecting with.
user = 'admin'
pw = 'exasol'
connection_string = f'https://{user}:{pw}@{database_host}'
sslcontext = ssl._create_unverified_context()
# Establish connection
conn = xmlrpc.client.ServerProxy(connection_string, context = sslcontext, allow_none=True)
Once you are connected, you can execute a job:
# Choose the job you want to run
job_name = 'db_start'
# Set the parameters for the chosen job
params = {'db_name': 'Exasol'}
# Execute the job and wait for it to finish and return the result
conn.job_exec(job_name, {'params': params})
For more examples on how you can interact with ConfD using XML-RPC, see the confd_xmlrpc script on GitHub.
ConfD jobs
BucketFS jobs
Job Name | Description |
---|---|
bucket_add | This job adds one or more buckets to a BucketFS service. |
bucket_delete | This job deletes a bucket. |
bucket_modify | This job modifies a bucket. |
bucketfs_add | This job adds a BucketFS service. |
bucketfs_delete | This job deletes a BucketFS service. |
bucketfs_info | This job provides information about a BucketFS service. |
bucketfs_list | This job lists BucketFS services. |
bucketfs_modify | This job modifies a BucketFS service. |
Database jobs
Job Name | Description |
---|---|
db_add_reserve_nodes | Adds the specified nodes as reserve nodes for the given database. The nodes must already be added to the cluster and configured accordingly. |
db_backup_abort | This job aborts a running database backup. |
db_backup_add_schedule | This job adds a backup schedule to the given storage volume with the given level and expiration time. |
db_backup_change_expiration | This job changes the expiration time of the given backup. |
db_backup_delete_unusable | This job deletes all unusable backups for the given database. An unusable backup is a backup which cannot be read or used anymore, for example a level 1 backup without a corresponding level 0 backup. |
db_backup_list | This job returns a list of available backups for the given database. |
db_backup_modify_schedule | This job modifies the given backup schedule with new parameters. The database must be running to run this job. |
db_backup_progress | This job returns the progress of the currently-running backup for a given database. |
db_backup_remove_schedule | This job removes the given backup schedule. |
db_backup_start | This job starts a backup of the given database to the given archive volume with given level and expiration time. The database must be running in order to write a backup. |
db_backups_delete | This job deletes the specified backup(s). |
db_configure | This job changes the configuration of a database. The database must be offline to change its configuration. |
db_configure_kerberos | Sets parameters for DB Kerberos authentication. |
db_create | Creates a new database. A database with this name or UUID must not already exist. |
db_delete | This job deletes a database. The database being deleted must already exist. |
db_enlarge | This job increases the number of active database nodes. The database must be stopped to run this job. The nodes to be added must be added already as reserve nodes. Afterwards, the database is started with the new nodes automatically. |
db_info | Returns a dictionary with the main database and volume parameters. |
db_list | This job returns the name of all created databases. |
db_remove_reserve_nodes | This job removes the specified nodes as reserve nodes from the given database. The specified nodes must already be added as reserve nodes. |
db_reset | This job resets the database. After performing this operation, the database will be empty and all data (including metadata) is deleted. The database must be offline to perform a reset. |
db_restore | This job restores the given database from the given backup. The backup must meet certain prerequisites defined in documentation, such as originating from a database with the same number of nodes. The database must be offline to perform a restore. |
db_resume_nodes | This job resumes the suspended nodes in the given database. |
db_snapshot_backup_add_schedule | This job adds a database snapshot schedule for the given database. The database must be configured with a snapshot_sync_volume, which is used for the schedule. |
db_snapshot_backup_list | This job returns a list of available snapshots for the given database. |
db_snapshot_backup_modify_schedule | This job modifies the given snapshot schedule with new parameters. The database must be running to run this job. |
db_snapshot_backup_remove_schedule | This job removes the given snapshot schedule. |
db_snapshot_backup_start | This job starts a database snapshot with the given expiration time. The database must be running to run this job. |
db_snapshot_backups_delete | This job deletes the given database snapshots. |
db_start | Starts the database. The database must already exist and must not be running. |
db_state | This job returns the current state of the given database. Some common states are 'Running' (meaning the database is online), 'Setup' (meaning the database is offline), 'Startup' (meaning the database is starting up), and 'Shutdown' (meaning the database is shutting down). |
db_stop | Stops a running database. The database nodes will remain online and available. |
db_suspend_nodes | This job suspends the given nodes in the given database. Suspending a node means that the node is marked as unavailable, even though it continues to run. This can be used to perform maintenance on a node. If this is done to an active node while the database is running, the database will shutdown and restart with a reserve node instead (if configured). |
Group jobs
Job Name | Description |
---|---|
group_create | This job creates a new group with a given name and syncs the change on all nodes. |
group_delete | This job deletes a group (identified by groupname) and syncs the change on all nodes. |
group_info | This job returns details about a given group (identified by groupname). |
group_list | This job returns a list of all groups in the configuration. The job has no parameters. |
Infrastructure jobs
Job Name | Description |
---|---|
infra_db_scale | This job updates a node(s) of the cluster and updates the database configuration. |
infra_db_start | Start an existing database and all of its nodes. |
infra_db_stop | Stops an existing database in the cluster and shutdown the instances. |
infra_deployment_metadata_get | This job provides deployment metadata for the cluster. |
infra_firewall_update | This job updates the firewall rules of the deployment. |
infra_instances_add | This job adds an additional node(s) to a cluster. You can create the new node(s) using an existing node configuration or the cluster configuration. |
infra_instances_remove | This job removes a node(s) from the cluster. |
infra_instances_start | This job starts an existing node(s) of the cluster. |
infra_instances_stop | This job stops an existing node(s) of the cluster. |
infra_instances_update | This job updates nodes based on the cluster configuration. |
infra_overview_get | This job provides and overview of the cluster state (e.g., running, stopped). |
infra_resources_update | This job updates all other resources of the respective platform. |
infra_worker_db_add | This job creates a worker cluster and adds necessary additional nodes. |
infra_worker_db_remove | This job deletes a worker cluster and removes its nodes. |
License jobs
Job Name | Description |
---|---|
license_download | This job returns the contents of the license file as a string. |
license_info | This job returns the fields of the license. |
license_run_check | This job checks the running databases against the license limits. |
license_upload | This job uploads a new license to the cluster. It may take a few minutes for the new license to take effect. |
Node jobs
Job Name | Description |
---|---|
node_add | This job adds a new node to the database. |
node_add_batch | This job adds new nodes that are defined in a list. |
node_clone_batch | This job adds new nodes by cloning an existing node that is used as a template. |
node_info | This job shows extended node information for a single node. |
node_ip_batch | This job changes the IP addresses in a list of nodes. |
node_list | This job lists all the nodes in the current configuration. Options with empty values are omitted. |
node_remove | This job removes one or more nodes from the cluster. The node cannot be part of an EXAStorage volume or a database unless the "force" flag is set. If the node is in a suspended state, the job will fail. |
node_resume | This job will resume one or more suspended nodes. |
node_state | This job shows the current state of all nodes. |
node_suspend | This job suspends one or more nodes. |
node_wait_offline | This job waits for the given nodes to finish shutdown. It returns a list of offline nodes (may be empty in case of errors). |
node_wait_ready | This job waits for the given nodes to finish booting and be ready to accept ConfD job requests. It returns a list of ready nodes (may be empty in case of errors). |
nodes_terminate | This job stops all Exasol processes on the given nodes. The nodes remain up and running. |
Object Volume Jobs
Job Name | Description |
---|---|
object_volume_add | Adds a new volume to EXAStorage pointing to a cloud storage. |
object_volume_info | This job provides information about the data volume. |
object_volume_list | This job lists all data volumes. |
object_volume_remove | This job removes a data volume and its associated running storage service. The data volume cannot be in use when you use this command. |
Other jobs
Job Name | Description |
---|---|
affinity_reset | This job resets affinities of nodes after the exainit script has run, and selects a new master node if necessary |
cert_update | This job uploads a new TLS certificate which is only used for the given database. This certificate is used after the next database restart. |
conn_get_remote_nid | Return the physical node ID This job determines the physical node ID (root node ID) of the calling process. |
debug_collect | This job retuns log events about debug information that fits the search parameter values. |
general_settings | This job changes EXAConf file settings that do not need a parent job. |
log_collect | This job retuns log events that fit the search paramater values. |
master_change | This job selects a new master node from a list of nodes. Do not execute other jobs until this operation is done. |
settings_list | The job lists current settings in the EXAConf file, a specific section in the file, or a specific subsection. |
update_system | Handle update operations for the COS cluster. |
Plugin jobs
Job Name | Description |
---|---|
plugin_add | This job adds a new plugin to a bucket. |
plugin_info | This job provides information about a plugin. |
plugin_list | This job lists all plugins. |
plugin_remove | This job removes a plugin. |
Remote volume jobs
Job Name | Description |
---|---|
remote_volume_add | This job creates a remote volume. |
remote_volume_info | This job provides information about the remote volume. |
remote_volume_list | This job lists all remote volumes. |
remote_volume_list_details | This job lists all remote volumes. |
remote_volume_remove | This job deletes a remote volume. |
remote_volume_state | This job retuns the state of the remote volume. For example, it is running. |
Storage jobs
Job Name | Description |
---|---|
st_device_add | Add a new disk (device) to an existing node |
st_device_clear_errors | Clear device errors on a specified node and device |
st_device_disable | Disable a disk (device) on a specified node |
st_device_enable | Enable a disk (device) on a specified node |
st_device_enlarge | Enlarge a disk (device) on a specified node |
st_device_info | Return information about a disk identified by node ID and device name. An exception is thrown if the disk is not found. |
st_device_modify | Modify a disk (device) for a specified node |
st_device_remove | Remove a disk (device) on a given node This operation is only allowed if the disk is currently not in use. |
st_get_partition_id | Return the EXAStorage service partition ID |
st_node_force_bg_recovery_limit_calibration | Force background recovery throughput limit (re)calibration |
st_node_get_link_speed | Return the network link speed in MBit/s |
st_node_list | Return information about all storage nodes |
st_node_restore | Restore a node specified by node ID |
st_node_resume | Resume a suspended node specified by node ID |
st_node_set_bg_recovery | Enable/disable background recovery (data restoration) |
st_node_set_bg_recovery_limit | Set background recovery throughput limit |
st_node_set_space_warn_threshold | Set a new space warning threshold |
st_node_stop_restore | Stop restoration of a node specified by node ID |
st_node_suspend | Suspend a node specified by node ID |
st_volume_add_label | Add a label to a volume |
st_volume_append_node | Append nodes to a volume |
st_volume_change_owner | Change the owner (user/group) of a specified volume |
st_volume_change_permissions | Change permissions of an existing volume |
st_volume_clear_data | Clear data on one or more specified nodes |
st_volume_create | This job creates a new local volume and returns the volume ID. The type of volume (data or archive) depends on the 'type' parameter value. |
st_volume_decrease_redundancy | Decrease the redundancy of a volume |
st_volume_delete | Delete a volume and release the allocated space on every node and disk |
st_volume_enlarge | Enlarge an existing volume |
st_volume_increase_redundancy | Increase the redundancy of a volume |
st_volume_info | Return information about a volume given the ID or volume name. |
st_volume_list | Return a list of all volumes |
st_volume_lock | Lock a specified volume |
st_volume_move_node | Move nodes on a specified volume |
st_volume_remove_label | Remove a label from a specified volume |
st_volume_set_forced_red_level | Force operations to be directly read from/written to a specific redundancy level |
st_volume_set_io_status | Enable or disable application and internal I/O operations on a volume |
st_volume_set_ports | Change IP port numbers for a specified archive volume |
st_volume_set_priority | Change the priority for I/O operations on a volume |
st_volume_set_shared | Set/unset the 'shared' flag of a specified volume The 'shared' flag cannot be changed if the volume is currently opened by more than one user. |
st_volume_unlock | Unlock a specified volume |
User jobs
Job Name | Description |
---|---|
user_create | This job creates a new user and syncs the change on all nodes. |
user_delete | This job deletes a given user (identified by username) and syncs the changes on all nodes. |
user_info | This job returns information about a given user (identified by username). |
user_list | This job returns a list of all users including all user details. |
user_modify | This job modifies a given user (identified by username) and syncs the changes on all nodes. |
user_passwd | This job changes the password for a given user (identified by username) and syncs the changes on all nodes. |