Glob Pattern Support for PARQUET Import (S3-Compatible Storage)

Details

Detail name Value
Changelog Number 28706
Type New Feature
Status Resolved
Fix Versions Exasol 2026.1.0
Resolution Date 2026-05-15

Summary

Enhanced the IMPORT FROM PARQUET functionality to support glob patterns in the FILE parameter for S3-compatible storage systems (Example AWS S3 and MinIO). This enables scalable and flexible loading of multiple files across directories and nested subdirectories.

Background

Previously, the system supported importing Parquet files only via explicitly specified absolute file paths. This limitation restricted scalability when working with multiple files stored across folders and subfolders.

With this enhancement, users can now leverage glob patterns to dynamically match and import multiple files efficiently.

Scope of Enhancement

  • Applies only to PARQUET imports
  • Supported only for S3-compatible storage systems like:
  • Glob pattern support is available only in the FILE clause
  • Bucket name remains mandatory and does not support glob patterns
  • AWS S3
  • MinIO

New Capabilities

Glob Pattern Support in FILE Clause

The following glob patterns are now supported:

Pattern Description
* Matches zero or more characters
? Matches exactly one character
[abc] Matches one character from the set
[!abc] / [^abc] Matches one character not in the set
/**/ Enables recursive directory traversal

If any glob characters appear in the filename, they are interpreted as glob expressions.

If users need literal matching of glob characters, they must escape them using {{}}.

Supported Usage Examples

1. Import All .parquet Files in Bucket Root

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE '*.parquet';

Behavior: Fetches all .parquet files in the root directory.

2. Recursive Import from Folder and Subfolders

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder/**/*.parquet';

Behavior: Recursively fetches all .parquet files from testfolder/ and all nested subdirectories.

3. Non-Recursive Import from Folder

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder/*';

Behavior: Loads all files directly under testfolder/. Subdirectories are excluded.

3. Single-Character Match (?)

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'data/test?.parquet';

Behavior:
Matches files like test1.parquet, testA.parquet but not test10.parquet.

4. Character Set Match ([abc])

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'data/Test[abc].parquet';

Behavior:
Matches:

  • data/Testa.parquet
  • data/Testb.parquet
  • data/Testc.parquet

5. Negation ([!abc])

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'data/Test[!abc].parquet';

Behavior:
Matches files other than:

  • Testa.parquet
  • Testb.parquet
  • Testc.parquet

6. Invalid Recursive Pattern (Not Recursive)

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder**/*.parquet';

Behavior

This pattern will NOT perform recursive traversal.

Explanation

Recursive traversal is supported only when the pattern explicitly uses the directory separator with double asterisk:

/**/

Correct recursive syntax example:

testfolder/**/*.parquet

In the invalid example:

testfolder**/*.parquet

The ** is appended directly to testfolder without a separating /. Therefore:

  • It is interpreted as part of the folder name pattern.
  • It matches directories whose names start with testfolder followed by any characters.
  • It does not trigger recursive traversal logic.

Effectively, the pattern behaves like:

  • Match directories such as:
  • Then match .parquet files directly inside those matched directories.
  • Subdirectories under those directories will not be traversed recursively.
  • testfolder
  • testfolder1
  • testfolder_backup

Important Rule

Recursive matching requires the exact syntax:

/**/

If the directory separator / is missing, recursion will not occur.

Error Handling & Validation Rules

1. Folder Without Glob Pattern (No Trailing Slash)

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder';

Behavior:
Treated as a filename.

Exception: 

Parquet file (s3://bucket-s3/testfolder) not found. IOError: Path does not exist 'bucket-s3/testfolder'. Detail: [errno 2] No such file or directory

2. Directory With Trailing Slash But No Glob

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder/';

Behavior:
Rejected because no glob pattern is provided.

Exception: 

Parquet file (s3://bucket-s3/testfolder/) not found. IOError: Not a regular file: 'bucket-s3/testfolder/'

3. No Files Matched by Glob Pattern

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'test/*.parquet';

Exception: 

Failed to list files for (s3://bucket-s3/test/*.parquet). Error: No files found matching the specified path and glob pattern.

4. Multiple Glob Patterns

If multiple file patterns are provided:

  • At least one file must match per pattern
  • If any pattern matches zero files → query fails

5. Glob Pattern in Connection Definition (Not Allowed)

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 's3://bucket-s3/Test*/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder/*';

Behavior:
Rejected — glob patterns are not allowed in the connection definition.

Exception: 

Glob pattern is not supported in connection definition for host (s3://bucket-s3/Test*/). Please remove glob pattern from connection definition and use it only in file path.

6. Glob Pattern in Bucket Name (Not Allowed)

Bucket name must be explicit and does not support pattern matching.

7. Unsupported Host

IMPORT INTO (c1 VARCHAR(256)) 
FROM PARQUET 
AT 'https://www.testhost.com/'  
USER '******'
IDENTIFIED BY '******'  
FILE 'testfolder/*';

Behavior:

If glob pattern is used for non-S3-compatible storage:

Exception: 

Glob pattern is not supported for host (https://www.testhost.com/testfolder/*). Please remove glob pattern or use a supported host.

8. Invalid or Malformed Glob Pattern

Malformed patterns will result in validation failure with a descriptive error message.

Limitations

  • Feature available only for:
  • Not supported for:
  • Glob pattern cannot be used in:
  • IMPORT FROM PARQUET
  • S3-compatible storage (like AWS S3, MinIO)
  • Other remote storage systems
  • Other file formats (CSV, etc.)
  • Bucket name
  • Connection definition (AT clause)

Benefits

  • Improved scalability for large datasets
  • Simplified multi-file imports
  • Recursive data loading support
  • Enhanced S3 compatible storage integration
  • Strong validation and predictable error handling

Runtime Behavior, Permissions & Validation Considerations

Runtime Overhead

When a glob pattern is used in the FILE clause, the system must first resolve the pattern by listing and matching all candidate objects in the specified S3-compatible storage location. This resolution step introduces additional overhead to the query runtime compared to importing a single explicitly specified file. The overhead depends on the number of objects present in the bucket or directory structure.

File Type Validation

During glob resolution:

  • All files matching the glob pattern are collected, irrespective of file type.
  • The system does not pre-filter by extension at the listing stage.

If any matched file is not a valid Parquet file, the import operation will fail with an exception.

After successful glob resolution:

  • The import process begins for the matched files.
  • During import, if a Parquet file contains a schema that is incompatible with the target table definition, the query will be aborted immediately.
  • No partial import will be committed.

This ensures strict schema validation and predictable behavior.

Permission Requirements (AWS S3 / S3-Compatible Storage)

For glob pattern resolution to function correctly, the user must have appropriate permissions to list objects in the bucket.

Specifically for AWS S3, the user (IAM user or role) must have permission for:

s3:ListBucket

Without the required ListBucket permission, glob pattern resolution will fail before import begins.

Example error:

Failed to list files for (s3://bucket-s3/testfolder/*.parquet). 
Error: IOError: When listing objects under key 'testfolder/' in bucket 'bucket-s3': 
AWS Error ACCESS_DENIED during ListObjectsV2 operation: 
User: arn:aws:iam::<Account ID>:user/<IAM User> is not authorized to perform: s3:ListBucket 
on resource: "arn:aws:s3:::bucket-s3" because no resource-based policy allows the s3:ListBucket action

This requirement also applies to other S3-compatible storage systems (such as MinIO), where equivalent listing permissions must be granted.

Changed behavior

Previously, special characters such as '*', '?', '[ ]', and similar symbols were treated as literal characters when specified in the `FILE` clause. With this enhancement, these characters are now interpreted as glob pattern operators and are resolved accordingly during file matching. If users intend to match filenames that contain these characters literally, they must explicitly escape them using the backslash (`\`) character. This behavioral change may impact existing import statements where filenames include glob-related characters, as those characters will no longer be treated as plain text by default.