Create Remote Archive Volume

This section explains how to create a remote archive volume for database backups.

Backups are stored on archive volumes in a compressed format. With on-premises installations of Exasol you can create archive volumes either locally in the cluster or on a remote location. For cloud deployments, only remote archive volumes are supported.

The following examples describe how to create a remote archive volume in an Amazon S3 bucket. Exasol recommends using Amazon S3 buckets to store backups for cloud deployments on AWS and when installing Exasol as an application on AWS instances.

This procedure can be carried out using the Administration API or ConfD.

Prerequisites

General prerequisites

  • All nodes must be able to reach the remote target.
  • The user must have read-write access on the remote host

Amazon S3 specific prerequisites

  • An existing S3 bucket. If you do not have a bucket, see How Do I Create an S3 Bucket?.

    The bucket URL must be in the following format:

    http[s]://<bucketname>.s3[-<region>].amazonaws.com/[<optional-directory>/]

    An Amazon S3 bucket URL using the legacy global endpoint format (<bucket-name>.s3.amazonaws.com) may need up to 24 hours after bucket creation before it becomes available.

    A fully qualified Amazon S3 bucket URL that includes the AWS region (<bucket-name>.s3.<region-code>.amazonaws.com) will become available immediately.

  • Read-write access to the S3 bucket. If the nodes are on a private network, make sure that an S3 endpoint is configured for your VPC and that the route table for your subnet is updated accordingly to store backups in the bucket. For more information, see Endpoints for Amazon S3.
  • A secret access key for the S3 bucket. If you do not have a key, see Managing Access Keys for IAM Users.

Azure specific prerequisites

  • A container in your Azure Blob. If you do not have a container, see Create a storage account and Create a container.
  • The URL of the container (in the container properties) that you created, in the following format:
    • http://storage_container_name.blob.corewindows.net/container_name
    • https://storage_container_name.blob.corewindows.net/container_name
  • Read-write access to the Blob storage. If the nodes are on a private network, make sure that a service endpoint for Microsoft Storage is configured in your VNet to store backups in the container. For more information, see Virtual Network service endpoints.
  • An access key for the Blob storage account. You can find the key under Settings option of your storage account.

GCP specific prerequisites

  • You should have the Google Cloud Storage buckets ready. If you have not created the buckets, see Creating storage buckets.
  • You should have the URL of the bucket in one of the following formats:
    • http://<bucketname>.storage.googleapis.com
    • https://<bucketname>.storage.googleapis.com
  • In the GCS web console page for your bucket, under Settings > Interoperability, create an Access key for your service account and save the generated Secret.
  • In case the nodes are on a private network, you need to assure that Private Google Access is enabled in your subnet to store backups in the storage. For more information, see Configuring Private Google Access.

Procedure - Administration API

The following examples use curl in a Linux terminal to send REST calls to endpoints in the Administration API. You can also use other interfaces and languages to interact with the API. For more information, see Administration API.

Placeholder values are styled as Bash variables, for example: $EXASOL_IP. Replace the placeholders with your own values.

The option --insecure or -k tells curl to bypass the TLS certificate check. This option allows you to connect to a HTTPS server that uses a self-signed certificate or a certificate that is not valid. Only use this option if certificate verification is not possible and you trust the server.

To create a remote archive volume, send a PUT request to the /api/v1/databases/Exasol/volumes endpoint.

Specify the url to the bucket (url), the volume name (volumeName), and the volume type (volumeType) as a part of the request.

curl --insecure -X 'POST' \
  'https://$EXASOL_IP/api/v1/databases/Exasol/volumes' \
  -H 'accept: application/json' \
  -H 'Authorization: Basic $TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "$S3_BUCKET",
  "volumeName": "$VOLUME_NAME",
  "volumeType": "s3"
}'

To verify that the new remote archive volume was created, send a GET request to the /api/v1/databases/Exasol/volumes endpoint.

curl --insecure -X 'GET' \
  'https://$EXASOL_IP/api/v1/databases/Exasol/volumes' \
  -H 'accept: application/json' \
  -H 'Authorization: Basic $TOKEN' 

The output should be similar to the following:

[
  {
    "id": "10003",
    "name": "$VOLUME_NAME",
    "type": "s3",
    "owner": [
      500,
      500
    ],
    "url": "$S3_BUCKET/$FOLDER"
  },
  {
    "id": "10002",
    "name": "default_backup_volume",
    "type": "s3",
    "owner": [
      500,
      500
    ],
    "url": "$S3_BUCKET/backup"
  },
  {
    "id": "10001",
    "name": "default_logrotation_volume",
    "type": "s3",
    "owner": [
      0,
      0
    ],
    "url": "$S3_BUCKET/logs"
  }
]

Procedure - ConfD

The following examples use ConfD through the command-line tool confd_client, which is accessible on all database nodes. You can also access ConfD through XML-RPC in your own Python programs. For more information, see ConfD.

Placeholder values are indicated with UPPERCASE characters. Replace the placeholders with your own values.

  1. To find the name of the database, use the ConfD job db_list. For example:

    confd_client db_list
    - Exasol
  2. The remote archive volume must be created with the same owner as the database that will write backups to the volume. To find the owner, use the ConfD job db_info. For example:

    confd_client -c db_info -a 'db_name: Exasol' | grep owner -A 2
    owner:
    - 500
    - 500
  3. To create a remote archive volume, use the ConfD job remote_volume_add.

    confd_client -c remote_volume_add -a '{url: ARCHIVE_VOLUME_URL, vol_type: TYPE, username: VOLUME_USERNAME, password: VOLUME_PW, owner: OWNER_TUPLE}'

    For example, to create a remote archive volume on an S3 bucket:

    confd_client -c remote_volume_add -a '{url: https://my_bucket.s3.eu-west-1.amazonaws.com, vol_type: s3, username: backup_user, password: 123456789, owner: [500,500]}'
  4. To verify that the volume was created, use the ConfD job remote_volume_list. For example:

    confd_client -c remote_volume_list
    - r0001
  5. To check the properties of the volume, use the ConfD job remote_volume_info and insert the name of the volume. For example:

    confd_client -c remote_volume_info -a 'remote_volume_name: r0001'