Importing Parquet files

Supported data types

Source data tracking

Column mapping

This feature is available in Exasol 2025.1.5 and later.

You can specify source column names by using WITH SOURCE COLUMN NAMES=() in the IMPORT statement. This allows you to explicitly map Parquet columns to columns in the target table.

SkipCols cannot be used together with the SOURCE COLUMN NAMES option. If both are specified in the same query, the import operation fails with the exception ETL-2230.

For an example of how to use column mapping, see Example 3 - Use explicit column mapping.

Column skipping

The SkipCols configuration parameter enables selective exclusion of columns from the imported Parquet data based on their index positions (0-based). This provides precise control over which columns to include or exclude while importing data. Skipped columns are defined as part of the connection string using ranges or discrete column indices.

SkipCols cannot be used together with the SOURCE COLUMN NAMES option. If both are specified in the same query, the import operation fails with the exception ETL-2230.

For more information and examples of use, see Configuration parameters and Examples.

Load files using pattern matching (S3)

This feature is available in Exasol 2026.1 and later.

Pattern matching is only supported when importing Parquet files from S3 storage.

The FILE parameter in IMPORT FROM PARQUET supports glob patterns when accessing data from S3-compatible storage systems. This allows importing multiple files without specifying each file explicitly. Glob patterns are interpreted in the file path and enable flexible selection of files across directories and subdirectories.

Supported patterns

The characters *, ?, [, ] are interpreted as glob pattern operators if they are used in the file path. To match them literally, escape them using a backslash /

Examples

Import all Parquet files from the bucket root:

IMPORT INTO my_table
FROM PARQUET
AT 's3://bucket/'
USER '...'
IDENTIFIED BY '...'
FILE '*.parquet';

Import recursively from a directory and its subdirectories:

...
FILE 'data/**/*.parquet';

Import files from a directory (non-recursive):

...
FILE 'data/*';

Match single character:

...
FILE 'data/file?.parquet';

Match a character set:

...
FILE 'data/file[abc].parquet';

Exclude characters:

...
FILE 'data/file[!abc].parquet';

Usage notes

• Glob patterns are supported only in the FILE clause.

• The bucket name in the AT clause must be specified explicitly and does not support glob patterns.

• Glob patterns are supported only for S3-compatible storage systems.

• Recursive matching is only enabled when using the exact syntax /**/. For example:

  • data/**/*.parquet → recursive

  • data**/*.parquet → not recursive

Error handling

• If no files match the specified pattern, the query fails.

• If any matched file is not a valid Parquet file or has an incompatible schema, the entire import fails.

• If multiple patterns are specified, each pattern must match at least one file.

• Glob patterns are not allowed in the connection definition (AT clause).

• Using glob patterns with unsupported storage systems results in an error.

Permissions

The user must have permission to use the s3:ListBucket action in AWS. Without this permission, the import fails. For more information, refer to the Amazon AWS documentation.

Configuration parameters

Configuration parameters can be set to be query-specific using the connection string, and as global defaults using database parameters. If a parameter is set in the connection string, it will override the default database parameter.

Usage notes

• Connection string parameters are case-insensitive.

• Connection string parameters override database parameter values when both are provided.

• Default values are sourced from the database parameters if not expressly set.

Log levels (AWS)

The log level determines the granularity and amount of information logged by the system. Select a log level to tailor the logging output based on the use case, such as error debugging, performance monitoring, or high-level event tracking.

To enable structured logging for AWS, the database parameters -etlLogLevel and -traceLevel must both be set to at least 2.

Examples

Exaple 1: Create a connection object

-- Amazon S3 connection
CREATE OR REPLACE CONNECTION my_conn
    TO 'https://my_s3_bucket.s3.eu-west-1.amazonaws.com'
    USER 'my_access_key' IDENTIFIED BY 'my_secret_key'
;

-- S3-compatible storage
CREATE OR REPLACE CONNECTION my_conn
    TO 's3://my_bucket;EndPoint=https://example.com:9000'
    USER 'my_user' IDENTIFIED BY 'my_secret'
;

-- Azure Blob Storage
CREATE CONNECTION my_conn
    TO 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'
    USER 'my_account_name' IDENTIFIED BY 'my_account_key'
;

Example 2 - Create schema and table with file tracking

In this example we connect using a stored connection (see examples above), create a new schema and table, then load data from two Parquet files, using SQL options to store source metadata for row-origin tracking.

CREATE SCHEMA IF NOT EXISTS MY_SCHEMA;

CREATE TABLE MY_SCHEMA.MY_TABLE (sourcefile HASHTYPE (256 BIT), c1 INT, rownumber INT);

IMPORT INTO MY_SCHEMA.MY_TABLE
    FROM PARQUET AT my_conn
    FILE 'nested/path/to/test_1.parquet' FILE 'nested/path/to/test_2.parquet' 
    WITH 
      SOURCE FILE HASH_SHA256 = sourcefile
      SOURCE ROW NUMBER = rownumber 
;

Example 3 - Use explicit column mapping

CREATE SCHEMA IF NOT EXISTS MY_SCHEMA;

CREATE TABLE MY_SCHEMA.MY_TABLE (
  c1 INT,
  name VARCHAR(256),
  mapped_column VARCHAR(256),
  sourcefile HASHTYPE (256 BIT),
  rownumber INT
);

IMPORT INTO MY_SCHEMA.MY_TABLE
  FROM PARQUET AT my_s3_bucket
  FILE 'file_1.parquet'
  FILE 'file_2.parquet'
  WITH
    SOURCE COLUMN NAMES = ('col1', 'col2', 'renamed_col')
    SOURCE FILE HASH_SHA256 = sourcefile
    SOURCE ROW NUMBER = rownumber
;

Example 4 - Use configuration parameters in connection string

The following examples show a connection to AWS. The procedure is identical for all storage types.

-- Limit the number of files to import concurrently to 2
-- Increase the number of buffers to fetch during the import to 4
-- Exclude columns 1, 3, 4, 5, 6, 7, 8, and 11 from the imported dataset 
CREATE OR REPLACE CONNECTION my_conn
    TO 'https://my_s3_bucket.s3.eu-west-1.amazonaws.com;MaxConnections=2;MaxBatchFetchSize=4;SkipCols=1,3..8,11'
    USER 'my_access_key' IDENTIFIED BY 'my_secret_key'
;

-- Set a dynamic batch size
CREATE OR REPLACE CONNECTION my_conn 
    TO 'https://my_s3_bucket.s3.eu-west-1.amazonaws.com/;MaxRows=2048' 
    USER 'my_access_key' IDENTIFIED BY 'my_secret_key'
;