Sizing Considerations

This section explains how to determine the disk space and RAM that will be required for your Exasol database.

Factors that impact sizing

There are several factors that determine the storage disk space and RAM that will be required for an Exasol system. The major factors to consider are:

The expected volume of your raw data

The volume of your uncompressed (raw) data has the largest impact on the estimation of the required storage disk space. The larger the data volume, the more storage space is required. Table data in an Exasol database is automatically compressed to optimize disk space usage.

Performance

The amount of RAM that is allocated for the database (DB RAM) will have a direct impact on performance. When estimating how much DB RAM you need, you can use the rule-of-thumb of 10% of the volume of your uncompressed data. For a more precise value you can calculate the amount of active data, or use the actual values in Exasol system tables.

Cluster redundancy

The planned redundancy for the cluster affects the required storage space. With redundancy 2 – meaning that the same data is stored in two segments – the required storage space is effectively doubled.

Number of reserve nodes

Reserve nodes are used in case of a node failure. A reserve node must have the same hardware configuration as the active nodes in a cluster. For more information, see Fail Safety (Cloud)

Backup strategy

Certain backup strategies require more storage space. If you plan on storing backups in the cluster (local backups) with redundancy 2, then this must be taken into account when estimating disk size. For more information, see Backup and Restore.

Operating system reserved RAM

We recommend that you reserve 10% RAM for the operating system on each node. This must be considered when calculating the amount of RAM to allocate for the database (DB RAM).

Database disk space

The total required disk space for an Exasol system is the sum of the required database disk space and the required backup disk space. The factors that you should consider when calculating the database disk space are:

Compressed data volume

Table data in an Exasol database is automatically compressed. A typical compression rate is ~2.5, which means that if the raw data volume is 2500 GiB, the estimated compressed data volume is ~1000 GiB.

Index volume

Indexes are automatically created and maintained by the database and require disk space. The index size depends on the chosen data model and queries, and can range from 2% to over 100% of the compressed data. A typical Exasol system will have an index volume of about 15% of compressed data.

Statistical and auditing data volume

Statistical data volume is small. However, if you switch auditing on in the system, the required disk space increases because each login and each query is stored in the corresponding auditing tables.

Temporary data and fragmentation

When intermediate results do not fit into the database RAM they are swapped out to a temporary volume. We recommend that you reserve extra headroom for temporary DB RAM.

The persistent volume can become fragmented to some degree, which increases the amount of disk space taken up by the data. We recommend that you reserve additional disk space to avoid running out of space due to fragmentation.

As a rule of thumb, add 60% of the compressed data volume (without redundancy) as combined additional headroom for temporary data and fragmentation.

Database disk space calculation

To calculate the required disk space for the database, use the following equation:

((compressed data volume + index volume + statistical and auditing data volume) * redundancy) 
+ headroom for temporary data and fragmentation
= Required database disk space

Example

In this example we have the following input parameters:

• Raw data volume: 2500 GiB
• Redundancy: 2

Using the above equation and rule-of-thumb guidelines, the calculation will be as follows:

MAX( COMPRESSED DATA VOLUME * ESTIMATED DB RAM PCT, INDEX VOLUME * INDEX SCALE FACTOR ) 
+ COMPRESSED DATA VOLUME * TEMP DB RAM HEADROOM PCT 
+ MAX UDF RAM * NUMBER OF CORES * NUMBER OF NODES 
= Required DB RAM

MAX(1000 GiB * 20%, 150 GiB * 1.3) + 1000 GiB * 5% = 250 GiB

Choose EC2 Instance Type

Amazon EC2 instance types are named according to the following convention:

When choosing EC2 instance types for your Exasol deployments on AWS, use the following guidelines:

Instance family: Depending on your requirements, we recommend:

• c for workloads that require high CPU performance
• r for workloads that are data-heavy but compute-light
• m for a balance between CPU and memory optimization

Instance generation: Use the latest available generation for the selected instance type.

Processor family: Use only a (AMD) or i (Intel) processors. AWS Graviton (ARM) processors are not supported.

• Which processor architectures are available may differ depending on the instance generation.
• A certain processor family be implicit for some instance types.

Additional capability: Always specify d (instance store volumes).

Exasol will not work if the “d” option is omitted.

Instance size: Choose instance sizes based on Sizing Considerations and the total RAM that will be used by all your instances.