Optimized DBRAM locking in case of heavy atomic commit pressure

Detail name	Value
Changelog Number	10569
Type	Improvement
Status	Resolved
Fix Versions	Exasol 7.0.4, Exasol 6.2.14
Resolution Date	2020-11-26

Background

Database memory (DBRAM) is the core of Exasol's in-memory data processing. It is a shared RAM for reading that also provides space for session-exclusive modified data (TEMP, new persistent data). To ensure data consistency and stability the access to DBRAM has to be protected by locking mechanisms.

In Exasol each commit has two phases: The non-atomic part for modified user data and the atomic part for metadata updates. While the non-atomic part can be done by multiple commits simultaneously, the atomic part is exclusive for one commit at the same time. Atomic commits not only impede other atomic commits but also restrict certain internal structures from being updated (device locking).

In most cases atomic commits are fast. If the commit frequency is extremely high, then the duration of atomic commits can increase into the seconds. Other factors like database size per node, node count, and concurrent disk reading can also play a role in the scenario. In such scenarios commits can become a bottleneck. 

Certain operations on persistent data need both of the above locks: they lock the DBRAM and then the persistent device. The atomic commit also locks the persistent device (exclusively).

Problem

Extensively running many parallel ETL processes can cause the above locking situation to occur more frequently. In such cases, all operations can be affected. Even very simple SELECT queries can experience variable durations due to a series of atomic commits locking the DBRAM.

Improvement

The locking of DBRAM and persistent device has been optimized to avoid any sessions holding both locks. This will lead to more predictable query durations in high-commit scenarios.