Resolved OOM Issue in Parquet Loader with Configurable Batch Size and Optimized Defaults
Details
| Detail name | Value |
|---|---|
| Changelog Number | 26411 |
| Type | Bug |
| Status | Resolved |
| Affected Versions | |
| Fix Versions | Exasol 2025.1.3, Exasol 2025.2.0 |
| Resolution Date | 2025-09-17 |
Description
The Parquet Loader encounters Out of Memory (OOM) issues when the MaxConnections parameter is set to values greater than 1 in environments with limited memory availability. This issue is triggered during the import of multiple Parquet files when the system fetches batches of data faster than they can be inserted into the target table. Over time, this causes memory buildup, leading to memory exhaustion and a crash in the process.
Cause
The Parquet Reader fetches multiple batches of data in parallel, but the insertion into the target table cannot keep up with the batch-fetching speed. This mismatch results in excessive memory usage and accumulation, especially in scenarios with limited resources.
Workaround
To address the issue before applying the fix, one of the following approaches can be used:
- Set MaxConnections=1
- Use a Smaller Batch Size
- Split the Query into Smaller Chunks
- Reduce parallelism to serialize batch fetching.
- Although this slows down the data import process, it prevents memory buildup entirely.
- Configure a smaller batch size using the following settings:
- Lower batch sizes reduce the memory footprint during data import operations.
- It is available in Exasol 2025.1.2 CHANGELOG: Enhanced Parquet Reader Batch Size Configuration.
- Connection String Parameter: MaxRows
- DB Parameter: etlParquetMaxRows
- Manually divide the data load into smaller subsets or partitions.
- Sequentially load smaller chunks of data to minimize memory usage.
Fix
The issue has been resolved in the latest versions by introducing configuration options and modifying default behavior to reduce memory consumption during Parquet data imports.
Changes Made
- Configurable Batch Size
- Default Batch Size Change
- The batch size is now configurable using:
- Users can dynamically adjust the batch size based on memory availability and workload requirements.
- Connection String Parameter: MaxRows
- DB Parameter: etlParquetMaxRows
- The default batch size has been reduced from 65536 (previous versions) to 4096 in the latest version.
- This adjustment ensures lower memory usage by default, making the system more resilient in environments with limited resources.
These changes prevent memory buildup caused by the mismatch between batch-fetching and insertion speed, ensuring smoother execution and better support for systems with limited resources.
