Enhanced Parquet Reader Batch Size Configuration

Details

Detail name Value
Changelog Number 26410
Type Improvement
Status Resolved
Fix Versions Exasol 2025.1.3, Exasol 2025.2.0
Resolution Date 2025-09-17

Background

Previously, the batch size for the Parquet reader was hardcoded to a default value of 65536, which could not be modified by users. This limitation reduced flexibility when dealing with a variety of workloads, especially in scenarios involving large datasets or constrained memory environments. The inability to adjust the batch size could lead to suboptimal memory utilization, reduced system performance, and difficulties in fine-tuning behavior for specific use cases.

In response to these limitations, a new connection string parameter, MaxRows, was introduced. This parameter (mapped to the DB parameter etlParquetMaxRows) enables users to configure the batch size dynamically based on their specific performance and memory requirements. However, it was necessary to validate this parameter and implement proper fallback mechanisms to ensure a robust and user-friendly implementation.

Improvement

The Parquet reader now supports dynamic batch size configuration through the MaxRows parameter, allowing users to customize the batch size for processing. This feature provides greater flexibility and resolves memory-related issues by enabling workload-specific tuning.

Functionality

  • Connection String Parameter: MaxRows
  • DB Parameter: etlParquetMaxRows
  • Default Value: 4096
  • Validation: To ensure system stability, the MaxRows parameter must have a value greater than zero. If an invalid value is specified in the database parameter, the system will revert to the default batch size as a fallback mechanism. However, if an invalid value is provided in the connection string, an exception will be thrown. 
ETL-2224: Connection String parameter (MaxRows) contains an invalid value (**). Please provide a value greater than 0.

Example

ETL-2224: Connection String parameter (MaxRows) contains an invalid value (-10). Please provide a value greater than 0.

Parameter Details

  • DB Parameter: etlParquetMaxRows
  • Connection String Parameter: MaxRows
  • When set, this batch size will apply globally to all queries within the database.
  • If provided, this overwrites the global DB parameter for a specific connection.

Behavior

  1. Valid Value: If the user specifies a MaxRows value greater than 0, the Parquet reader adopts this value as the batch size.
  2. Invalid or Missing Value: If the MaxRows parameter is invalid or missing:
  • When specified via the database parameter, the system will default to the pre-configured batch size as a fallback mechanism.
  • When specified via the connection string, the system will throw an exception to ensure proper error handling.

Key Benefits

  • Memory Optimization: Users can adjust the batch size in memory-constrained environments to avoid out-of-memory issues.
  • Performance Tuning: Allows for fine-grained control over system performance based on workload size.
  • Graceful Handling: Fallback to a default batch size ensures the system remains stable even when misconfigurations occur.

Usage Example

  • Setting a custom batch size: 
--Connection String: 
CREATE OR REPLACE CONNECTION aws_conn 
TO 'https://xxxxx.s3.amazonaws.com/;MaxRows=2048' 
USER 'your_user' IDENTIFIED BY 'your_password';

By allowing dynamic configuration of batch size, this feature enhances system flexibility, robustness, and adaptability across various operating environments.