S3 Backups: Support for backups larger than 5 TiB
Details
| Detail name | Value |
|---|---|
| Changelog Number | 15464 |
| Type | Improvement |
| Status | Resolved |
| Fix Versions | Exasol 7.1.23 |
| Resolution Date | 2023-09-11 |
Background
AWS S3 has a file size limitation of 5 TiB. Backup files are single objects. If the size of the backup per node reaches the 5 TiB limit, then the backup fails.
Improvement
Backups are no longer restricted to 5 TiB/node. The backups can be configured to split automatically once a certain size is reached. To activate this behavior, use the following new parameters when configuring a remote volume:
| Parameter Name | Description | Example |
|---|---|---|
| auto-split-length | The maximum size of each backup file defined with a unit (e.g gb, gib, TB, TiB). | 5 TiB |
| auto-split-suffix | A string appended to the end of the name of the backup file. | part |
| auto-split-prefix | A string prepended to the name of the backup file. | split |
Changed behavior
This improvement doesn't change the default behavior. Meaning that the volume has to be configured with specific parameters for it to behave as previously described. If the new behavior is configured, the newly created backups will be incompatible with all previous backends. Older backups can be read by this new backend.
