High memory usage when inserting into distributed table with many varchar columns
Details
| Detail name | Value |
|---|---|
| Changelog Number | 8211 |
| Type | Bug |
| Status | Resolved |
| Affected Versions | Exasol 6.1.0, Exasol 6.2.beta1, Exasol 6.2.rc1 |
| Fix Versions | Exasol 6.2.0, Exasol 7.0.0, Exasol 6.1.5 |
| Resolution Date | 2019-07-05 |
Situation
Data parallelism is a key to performance. Therefore, Exasol executes multiple hundreds or even thousands of rows together. For many worker threads (cores), large data types (varchar) and lots of processed columns the heap memory requirements can grow into megabytes or even gigabytes.
When a session reaches its heap memory limit (4 GB is default), it will be terminated and hence the desired query cannot be executed.
Problem
When inserting into a distributed table with several hundred varchar columns, heap memory allocation may lead to session out of memory. The scenario is not considered a corner case as the database supports tables with several thousand columns.
Fix
The high memory usage will be fixed.
Workaround
Increasing the session heap memory limit e.g. from 4 to 8 GB may avoid the session out of memory. This can be achieved by the following database parameter:
-maxProcessHeapMemory=8192
