Load data from Google BigQuery

Learn how to connect and load data from Google BigQuery using Simba BigQuery JDBC driver.

For more information about data type mappings and how to migrate data, see the BigQuery to Exasol migration scripts in our GitHub repository.

Prerequisites

Procedure

Step 1: Create service account

  1. Log in to Google Cloud Console and create a new project or select an existing project.

  2. From the navigation menu, select IAM & Admin > Service accounts.

  3. Click Create Service Account for your project.

  4. Enter the service account details and grant the corresponding BigQuery roles from the Role dropdown list.

  5. Select the checkbox next to Furnish a new private key and ensure JSON is selected as Key type.

  6. Save the JSON file to your computer.

  7. Click Create.

For more information, refer to the Google Cloud documentation.

Step 2: Add the JDBC driver

  1. Create a configuration file settings.cfg with the following settings:

    Copy 
    DRIVERNAME=BIGQUERY
    PREFIX=jdbc:bigquery:
    FETCHSIZE=100000
    INSERTSIZE=-1
    NOSECURITY=YES

    The file must end with an empty line (line break followed by zero characters).

  2. Upload the settings.cfg file and the driver .jar file to BucketFS in the Exasol cluster.

    In Exasol 2025.1 and later you can upload files using Exasol Admin. You can also use any compatible file transfer tool, including curl on the command line. See also Manage files in BucketFS.

    In Exasol SaaS you must upload the driver through the web console.

    If the driver was downloaded as a .tar.gz or .zip archive, make sure that you extract and upload only the .jar file along with the settings.cfg file.

    Example (using curl in Linux):
    Copy 
    export WRITE_PW=<your_bucketfs_write_password>
    export DATABASE_NODE_IP=<ip_address_of_cluster_node>
    export PORT=<bucketfs_port> # default: 2581
    export DRIVER=<driver_filename> # for example: exajdbc.jar

    curl -v --insecure -X PUT -T settings.cfg https://w:$WRITE_PW@$DATABASE_NODE_IP:$PORT/default/drivers/jdbc/exasol/settings.cfg
    curl -v --insecure -X PUT -T $DRIVER https://w:$WRITE_PW@$DATABASE_NODE_IP:$PORT/default/drivers/jdbc/exasol/$DRIVER

    The option --insecure or -k tells curl to bypass the TLS certificate check. This option allows you to connect to a HTTPS server that does not have a valid certificate. Only use this option if certificate verification is not possible and you trust the server.

Step 3: Upload JSON key file to BucketFS

Upload the saved JSON key file to the same bucket you uploaded the JDBC driver to. For more information, see Manage files in BucketFS.

Step 4: Create database connection

Create the database connection using the following pattern:

Copy 
CREATE CONNECTION BQ_CON TO 'jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<your-project-id>;OAuthType=0;Timeout=10000;OAuthServiceAcctEmail=<your-service-account>;OAuthPvtKeyPath=/exa/data/bucketfs/<bucketfs-service>/.dest/<bucket-name>/<your-account-keyfile>;Location=<location>[;EnableSession=1]';

The Location parameter defines where data can be stored and processed. The location can be either a region (a specific geographic place, such as Frankfurt) or a multi-region (EU or US). For more information, see BigQuery locationsin the Google Cloud documentation.

The EnableSession=1 property is only required when using Simba BigQuery JDBC Driver version 1.3.0.1001 or later. For more information, see the following article in our Knowledge Base:

BigQuery: Transaction control statements are supported only in scripts or sessions.

Examples

Connection details:

  • Project ID: exa-migration
  • Service account: migration-test@exa-migration.iam.gserviceaccount.com
  • BucketFS service: bfsdefault
  • Bucket name: default
  • JSON key-file: /drivers/jdbc/bigquery/my-key.json
  • Location: EU (multi-region)

Resulting connection statement:

Copy 
CREATE CONNECTION BQ_CON TO 'jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=exa-migration;OAuthType=0;Timeout=10000;OAuthServiceAcctEmail=migration-test@exa-migration.iam.gserviceaccount.com;OAuthPvtKeyPath=/exa/data/bucketfs/bfsdefault/.dest/default/drivers/jdbc/bigquery/my-key.json;Location=EU;EnableSession=1';

Test your connection using this statement:

Copy 
IMPORT FROM JDBC AT BQ_CON STATEMENT 'SELECT 1';

Load data

Use IMPORT to load data from a table or SQL statement using the connection that you created.