Create Remote Archive Volume with Amazon S3

You need a remote archive volume to store your Exasol backups. If you did not create a remote archive volume during installation, you can create one using EXAoperation. Exasol recommends using S3 bucket to store your backups for Exasol on AWS.

Prerequisites

  • An existing S3 bucket. If you do not have a bucket, see How Do I Create an S3 Bucket?.

  • The URL to the S3 bucket in one of the following formats:

    • http://bucketname.s3.amazonaws.com/optional-directory/

    • https://bucketname.s3.amazonaws.com/optional-directory/

  • The user must have read and write access to the S3 bucket.

  • If the nodes are on a private network, make sure that an S3 endpoint is configured for your VPC and that the route table for your subnet is updated accordingly. For more information, see Endpoints for Amazon S3.

  • A secret access key for the S3 bucket. If you do not have a key, see Managing Access Keys for IAM Users.

  • If DNS hostnames are used, ensure that the cluster has DNS servers configured.

  • All nodes must be able to reach the remote storage.

A fully qualified S3 URL in the format <bucket-name>.s3.<region-code>.amazonaws.com will become available immediately when you have created the bucket. A URL in the legacy global endpoint format <bucket-name>.s3.amazonaws.com may need up to 24 hours to become available.

Configure remote archive volume in EXAoperation

Do the following to configure the remote archive volume:

  1. Log in to EXAoperation.
  2. Go to Services > EXAStorage and click on Add Remote Volume.

  3. Enter the following details:
    • Archive URL: Use the URL of the remote archive destination. For information about the supported protocols and example URLs, see Other protocols with URL examples.
    • User: If you have configured EC2 instance role, leave this field empty. If not, enter the aws_access_key_id here.
    • Password: If you have configured EC2 instance role, leave this field empty. If not, enter the aws_secret_access_key here
    • Allowed Users: Select the EXAoperation users that can use the remote archive volume.
    • Read-only Users: Select the EXAoperation users that can read from the remote archive volume.

      Any user that you add as to the Read-only Users list will be limited to read-only access, even if the user is added to the Allowed Users list. If you have only one user in the Allowed Users list, do not add this user to the Read-only Users list.

    • Labels: Add a label to the remote archive volume for easy identification.
    • Options: You can add one or more options (separated by commas) depending on the protocol used. For more information, see Remote archive volume options.
  4. Click on Add to create the volume.

Remote archive volume options

The following options can be set in the Options field when creating a remote archive volume (depending on the protocol). You can enter multiple options separated by commas.

Option Description
cleanvolume Remote backups will be deleted on expiry. See also Backup Essentials.
noverifypeer Do not check the server certificate.
nocompression Write raw (uncompressed) data to the archive volume.
forcessl Use STARTTLS in the FTP connection.
webdav Set this option when using WebDAV for http (unencrypted) URLs.
webhdfs Set this option when connecting using the WebHDFS protocol (Apache Hadoop).
s3 Volume uses the S3 object format. This option is not required if the remote archive is an Amazon S3 bucket.
timeout=seconds Override the default client/server response time. For example: timeout=300
verbose Write more information to the logs. This will be required when troubleshooting problems with backups.

Other protocols with URL examples

The port numbers in this table are examples. Most protocols allow you to set the port.

Protocol URL example
FTP

ftp://<ftpserver>:2021/optional-directory/

FTPS ftps://<ftpserver>:2021/optional-directory/
SMB smb:////<smbserver>:2139/optional-directory/
WebHDFS

http://<hadoop-server>:2080/optional-directory/

https://<hadoop-server>:20443/optional-directory/

Azure Blob Storage http://storage_container_name.blob.corewindows.net/container_name

https://storage_container_name.blob.corewindows.net/container_name

For more information, see Create Remote Archive Volume with Azure Blob Storage.
Google Cloud Storage

http://<bucketname>.storage.googleapis.com

https://<bucketname>.storage.googleapis.com

For more information, see Create Remote Archive Volume with Google Cloud Storage.