Architecture overview

Database layer

Exasol is a massively parallel processing (MPP) database designed on a shared-nothing architecture. Data is distributed across all nodes in the cluster and constantly synchronized between the nodes. All nodes operate on an equal basis, there is no central master node.

This processing paradigm is also known as SPMD (single program, multiple data).

Each query sent to an Exasol database is first accepted by the node that the client is connected to, then distributed to all nodes in the cluster over the private network. Algorithms optimize the query, determine the best plan of action, and generate indexes as needed. Exasol processes partial results based on local data sets, then delivers the global results back to the client.

Data is stored in columns and primarily held in RAM to deliver high-speed query performance. If the data exceeds the available memory, an intelligent storage management tier handles disk-to-memory swapping automatically.

Performance optimization mechanisms

Exasol is optimized for high performance through several automated mechanisms that do not require user intervention:

Automatic index creation, maintenance, and deletion

Indexes that are required for JOIN operations are automatically created and subsequently maintained upon DML that modifies indexed tables. Indexes that have not been used for more than five weeks are automatically dropped.

Automatic table reorganization

A DELETE operation will initially only mark rows as deleted within a table. When more than 25% of the rows are marked as deleted, the table is automatically reorganized and the indexes on that table are rebuilt.

Optimizer statistics gathering

The Exasol optimizer works cost based, using statistics about object attributes such as the size and number of table rows. These statistics are automatically gathered after each DML statement without any administrative effort.

Query cache

The query cache stores a SELECT query together with its result if certain conditions are fulfilled. If the same query is sent again, the database reads the result directly out of the cache instead of executing the query. This enhances performance when you frequently have many identical queries (low write/high read scenarios).

Zone maps (partition pruning)

Zone maps, also known as partition pruning, allows a database to decrease the I/O and computation cost of certain queries by skipping segments that are not necessary when processing the data. For more information, see Zone maps.

Virtual schemas

Virtual schemas is a federation feature that makes external data sources accessible as if the data lived inside the Exasol database. The contents of the external data sources are mapped to virtual tables, which can be queried as any regular Exasol table.

By using virtual schemas, you can implement a consistent access layer for your reporting tools. You can also use it for agile and flexible ETL processing, since there is no need to change anything in Exasol if you change or extend the objects in the underlying system.

Network layer

An Exasol cluster is normally configured with two networks using separate physical interfaces on the nodes: a private network for internal communication between the nodes in the cluster, and a public network that allows connections to the database from outside of the cluster.

The physical networks must be configured using network management tools on the Linux hosts. You can then specify the private and public IP addresses for the nodes in the Exasol deployment configuration. By design, you cannot manage the network infrastructure directly from Exasol.

Cluster architecture

A node in an Exasol cluster is a virtual or physical host with its own CPUs and memory. The total number of CPUs and the total amount of memory in a cluster determines the compute power.

Each node hosts the Exasol database engine, the cluster operating system (COS), the storage engine (EXAStorage), a BucketFS file service for storing additional files such as drivers and scripts, and the software needed for system administration.

The nodes are connected to each other over a private network (intra-cluster network). You can configure a public network to allow access from outside of the cluster.

System administration

The administration stack in Exasol provides several interfaces that enable you to deploy and manage your databases and clusters using both built-in and external tools. These interfaces are available on every node in a cluster.

• ConfD is a low-level API that is used to perform all system administrative actions in Exasol. You can interact with ConfD directly using the command-line tool confd_client, or indirectly through XML-RPC in your own Python programs.

• Exasol Admin is a web interface that allows you to carry out common administration tasks from your browser, such as monitoring your database, generating and analyzing logs, creating backup schedules, updating Exasol, and managing files in BucketFS.

• Exasol Deployment Tool (c4) is a multi-purpose command line tool for creating, configuring, and managing Exasol deployments. You can also use c4 to access other administration interfaces in Exasol.

Exasol supports encrypted key-based and basic authentication for system administration users. Which administrative tasks a user is allowed perform in Exasol depends on the group the user belongs to.

Monitoring and auditing

Exasol includes an auditing mechanism that allows you to view detailed information about all sessions such as driver, client, host, and operating system user, as well as information about all executed SQL statements, including CPU usage, network usage, and disk read/write operations. The auditing information is stored in system tables in the database, and does not require any transaction log or database schema modification.

Exasol does not include an internal syslog service – messages generated by the components in Exasol are instead forwarded to the host syslog service. You can use a third-party syslog management/monitoring tool to subscribe to the relevant syslog messages.