Enhanced AWS Logging for Improved ETL (Parquet Reader) Observability and Debugging

Details

Detail name Value
Changelog Number 26843
Type Improvement
Status Resolved
Fix Versions Exasol 2025.1.3, Exasol 2025.2.0
Resolution Date 2025-09-17

Description

To enhance the observability and debugging capabilities of ETL - Parquet Reader processes involving AWS services, structured and detailed logging for all AWS-related operations has been implemented. This feature ensures that key details (e.g., API calls, responses, execution times, and errors) are captured in log files, enabling easier troubleshooting and monitoring.

Logs will now include AWS interactions, provided that mandatory configuration parameters are set correctly.

Improvement

  1. Activation:
    Logging for AWS-related operations is disabled by default. To activate it:
  2. Parameter Levels:
    The DebugMode parameter accepts integer values ranging from 0 to 6 with the following behavior:
  3. Additional Requirements:
    Even if DebugMode is activated, the following configurations must also be set:
  4. Without these additional settings, AWS logs will not appear in the ETL logs.
  • Configure the DB parameter etlParquetDebugMode. The default value is 0.
  • Alternatively, use the Connection String parameter DebugMode, which overrides the value of the DB parameter.
  • 0: Off (default)
  • 1: Fatal only
  • 2: Error + Fatal
  • 3: Warn + Error + Fatal
  • 4: Info + Warn + Error + Fatal
  • 5: Debug + Info + Warn + Error + Fatal
  • 6 and above: Trace + Debug + Info + Warn + Error + Fatal
  • etlLogLevel must be 2 or higher.
  • traceLevel must be 2 or higher.

By adjusting these parameters appropriately, you can gain granular control over the level of logging detail captured in your system for efficient troubleshooting and performance monitoring.