Remote Server Support for Parquet Imports (HTTP/HTTPS, FTP/FTPS, SFTP)

Details

Detail name	Value
Changelog Number	28781
Type	New Feature
Status	Resolved
Fix Versions	Exasol 2025.1.9, Exasol 2026.1.0, Exasol 2025.2.1
Resolution Date	2026-02-18

Description

Overview

Previously, Exasol ETL Parquet imports supported only AWS S3 buckets. Remote services such as HTTP(S), FTP(S), and SFTP were not supported for Parquet file imports.

This update introduces comprehensive remote server support for Parquet file imports, enabling users to seamlessly import Parquet files from:

AWS S3 buckets (existing functionality, now enhanced)
HTTP and HTTPS servers
FTP and FTPS servers
SFTP servers

Users can now directly query Parquet files hosted on any accessible remote server, significantly simplifying data integration workflows across diverse storage platforms.

How Remote File Reading Works

Range-Based Fetching (Recommended Mode)

When a remote server supports range requests, Exasol reads Parquet files intelligently by downloading only the specific portions needed for query execution:

Metadata Retrieval: First, the Parquet file's metadata (schema, row group information) is downloaded using a range request
Selective Data Reading: Only the row groups required to satisfy your query are downloaded
Efficient Processing: Data is processed as it streams in, minimizing memory usage

Benefits:

Minimal Network Traffic: Only necessary data is transferred
Faster Query Execution: No waiting for complete file downloads
Lower Memory Usage: Streaming processing reduces memory footprint
Cost Savings: Reduced cloud egress charges for cloud-hosted files

Example Scenario:

You have a 10 GB Parquet file containing 12 months of sales data
Your query filters for data from January 2024 only
With range-based fetching: Only ~850 MB (January's row groups) is downloaded
Without range-based fetching: The entire 10 GB would need to be downloaded

Important: Selective data reading (downloading only required row groups) is only supported for servers that support range-based fetching. Without range support, the entire file must be downloaded.

Full File Download Mode (Fallback)

When a remote server does not support range requests or does not support HEAD requests (file size retrieval), Exasol automatically falls back to full file download mode:

Complete Download: The entire Parquet file is downloaded into memory
Buffer-Based Processing: The file must fit within the configured buffer size
Single-Threaded Processing: Parallel reads are disabled (set to 1 thread)
Standard Processing: Once downloaded, the file is processed normally. If the file size exceeds the buffer size, the file will be downloaded twice, leading to increased query time.

Important Considerations:

Memory Buffer Size:

Files are downloaded into a memory buffer controlled by etlParquetMaxBufferSize (DB parameter) or MaxBufferSize (connection string parameter)
Default buffer size: 314,572,800 bytes (300 MB)

File Size Limitations:

The Parquet file's metadata alone must fit within the buffer size
If metadata exceeds buffer size, the following error occurs:

ETL-2221: Parquet file (https://www.test-host.com/test.parquet) reading failed. 
Invalid: Insufficient Metadata. Increase the Parquet buffer size.

Solution: Increase the buffer size using the connection string parameter:

-- Example: Set buffer to 500 MB for this query
IMPORT INTO my_table
FROM PARQUET AT 'https://your-server.com/;MaxBufferSize=524288000'
FILE 'test.parquet';

Server Capability Requirements

For Optimal Performance (Range-Based Fetching):

Your remote server must support:

HTTP (S) / FTP (S) / SFTP Range Requests
HEAD Requests (file size retrieval)

Allows partial file downloads
Required for selective data reading
Enables parallel row group fetching

Allows Exasol to determine file size before downloading
Enables calculation of optimal range requests
Standard HTTP method supported by most servers
Required for range-based fetching strategy

Fallback Behavior When Capabilities Are Missing:

Server Support	Range Requests	HEAD Request	Behavior	Selective Reading	Performance Impact
Full Support	✅ Yes	✅ Yes	Range-based fetching	✅ Supported	Optimal - fastest, lowest bandwidth
No Range Support	❌ No	✅ Yes	Download the full file once or twice.	❌ Not supported	Lower - complete download
No HEAD Support	✅ Yes	❌ No	Download the full file once or twice.	❌ Not supported	Lower - complete download
No Support	❌ No	❌ No	Download the full file once or twice.	❌ Not supported	Lower - complete download

Note: When full file download is activated, the maximum parallel reads per file is automatically set to 1, regardless of database or user configuration.

Disabling Full File Download Mode

If you want to prevent full file downloads and only allow range-based fetching (recommended for bandwidth control and cost management):

Set the buffer size parameter to a negative value (e.g., -1):

Via connection string:

IMPORT INTO my_table
FROM PARQUET AT 'https://remote-server.com/;MaxBufferSize=-1'
FILE 'data.parquet';

Effect: Queries will fail if the remote server does not support range requests, ensuring you only work with optimized scenarios and preventing unexpected large downloads.
Error

ETL-2235: File (https://remote-server.com/data.parquet) cannot be read because the host server is not supported.

Feature Compatibility

All existing Parquet functionalities are fully supported:

Column projection (reading only selected columns)
Multiple Compression formats are supported
Source tracking columns (row number, filename)

No changes to your existing Parquet queries are required when migrating from S3 to other remote servers.

For information regarding parquet features, please refer the ETL Parquet import from AWS S3 documentation (see Load data from Apache Parquet files in Amazon S3 on AWS).

New Features

1. HTTP(S) Server Support

Customer Benefit: Access Parquet files from any web server or HTTP-based storage system.

IMPORT INTO sales_data
FROM PARQUET AT 'https://data-warehouse.company.com'
FILE 'exports/sales_2024_q1.parquet';

Security Features:

SSL/TLS encryption (HTTPS)
Certificate validation (configurable)

For detailed configuration of HTTP(S) connections, including certificate validation, SSL configuration, proxy settings, and authentication options, please refer to the CSV import documentation:

All HTTP(S) functionalities described for CSV imports are also available for Parquet imports.

2. FTP/FTPS Server Support

Customer Benefit: Import from traditional FTP servers commonly used in enterprise environments.

Example:

IMPORT INTO inventory_data
FROM PARQUET AT 'ftps://ftp.vendor.com'
USER 'ftpuser'
IDENTIFIED BY 'password'
FILE 'inventory_feed.parquet';

Supported FTP Features:

Standard FTP (port 21)
FTPS (FTP over SSL/TLS)
Active and passive mode
Explicit and implicit SSL
Username/password authentication

For detailed configuration of FTP/FTPS connections, including SSL modes, authentication methods, connection settings, and security options, please refer to the CSV import documentation. All FTP/FTPS functionalities described for CSV imports are also available for Parquet imports.

3. SFTP Server Support

Customer Benefit: Secure file access via SSH protocol for sensitive data transfers.

Example:

IMPORT INTO confidential_data
FROM PARQUET AT 'sftp://secure-server.company.com/data'
USER 'analyst'
IDENTIFIED BY 'ssh_password'
FILE 'customer_pii.parquet';

Security Features:

Public key authentication support
Password authentication

For detailed configuration of SFTP connections, including authentication methods, and advanced security options, please refer to the CSV import documentation. All SFTP functionalities described for CSV imports are also available for Parquet imports.

Configurable Buffer Management

Customer Benefit: Fine-tune memory usage and performance based on your workload.

Configuration Parameters:

Parameter	Location	Default	Purpose
etlParquetMaxBufferSize	DB parameter	314572800 (in bytes) (300 MB)	Global buffer size for all imports
MaxBufferSize	Connection string (per import)	Inherits DB param	Override buffer size for a single import

etlParquetMaxBufferSize = 0 or MaxBufferSize = 0: Unlimited buffer size.
etlParquetMaxBufferSize > 0 or MaxBufferSize > 0: Limited buffer size. For example, MaxBufferSize=1024 limits the buffer to 1024 bytes.
etlParquetMaxBufferSize < 0 or MaxBufferSize < 0: Enforces range-based fetching. For example, MaxBufferSize=-1 restricts data fetching to range-based methods only.

Configuration Examples

Per-query buffer increase for large files:

IMPORT INTO large_dataset
FROM PARQUET AT 'https://data-server.com/;MaxBufferSize=1073741824'  -- 1 GB for this query
FILE 'huge_file.parquet';

Disable full file downloads (force range-based only):

IMPORT INTO my_table
FROM PARQUET AT 'https://server.com/;MaxBufferSize=-1'
FILE 'data.parquet';

Recommendations:

For servers with range support: Keep default (300 MB) or small buffer
For servers without range support: Set buffer larger than your largest file via connection string
For strict bandwidth control: Set to -1 to prevent fallback mode
Note: Buffer size adjustments can be made via connection string parameter and DB Parameter

Intelligent Fallback Mechanism

Customer Benefit: Automatic adaptation to server capabilities ensures maximum compatibility.

How It Works:

Initial Detection: Exasol determine file size and range support
Mode Selection:
Automatic Adjustment: Parallel reads adjusted based on mode
Transparent Operation: No user intervention required

If both supported → Range-based fetching (optimal, selective reading enabled)
If either missing → Perform a full file download once or twice as a fallback (no selective reading)

User Visibility:

Error messages clearly state when buffer increases are needed

Performance Guidelines

Recommended Server Configuration

For Best Performance and Selective Data Reading:

Use servers that support HTTP (S) / FTP (S) / SFTP range requests (mandatory for selective reading)
Ensure HEAD request support for file size retrieval

Compatibility

Backward Compatible: All existing S3 Parquet queries continue working without modification
No SQL Changes: Same syntax for S3, HTTP, FTP, SFTP sources
Feature Parity: All Parquet features work with remote files (subject to server capabilities)
Mixed Sources: Can query S3, HTTP, FTP, SFTP files in same database
CSV Configuration Compatible: All HTTP/FTP/SFTP settings from CSV imports apply to Parquet

Limitations and Best Practices

Mandatory Requirements:

⚠️ Selective data reading requires range-based fetching - server must support HTTP (S) / FTP (S) / SFTP range requests
⚠️ HEAD request support is required for optimal performance
⚠️ Full download mode requires file metadata to fit in configured buffer size

Limitations:

Selective data reading only supported with range-capable servers
Full file download requires entire file to fit in memory buffer. This may trigger Out of Memory due to the file is loaded in memory.
Servers without range support limited to single-threaded processing
Network latency affects performance more than S3
Very large files (>buffer size) require range-capable servers
Parallel reads disabled in full download mode

Best Practices:

Always use servers with range request support for optimal performance and selective reading
Verify server capabilities (range + HEAD support) before deployment
Use MaxBufferSize=-1 in production to enforce range-only mode
Configure buffer via connection string when full download is unavoidable
Use HTTPS/FTPS/SFTP for all sensitive data transfers