Explicit auth and emulating Google Cloud Storage as object storage for Iceberg

[szisiu]

Our development and test environments use MinIO as an S3-compatible object store for Iceberg. Since our production environment uses GCS, this requires us to install heavy Amazon S3 libraries for the Hive Metastore (HMS) in our development setup. This adds unnecessary dependencies, increases build size, and creates a divergence between our development and production environments.

We could replace MinIO with a lightweight, GCS-compatible emulator to better align our development environment with production. The recommended solution is fsouza/fake-gcs-server, which can be easily integrated into our existing containerized setup.

The new Trino native GCS file system (fs.native-gcs.enabled=true) does not provide a mechanism to disable authentication, making it impossible to use with local, unauthenticated GCS emulators like fake-gcs-server.

Even when gcs.endpoint is configured to point to a local emulator, the underlying Google Cloud client library still attempts to perform a real OAuth2 token exchange with the public Google endpoint (https://oauth2.googleapis.com/token). This fails because a dummy credential file must be provided (to avoid an "ADC not found" error), but this dummy credential is invalid for the real authentication service.

There should be a way to use the recommended native client for local testing without it attempting to contact external authentication services.

Environment

• Trino Version: latest
• Connector: Iceberg (iceberg.catalog.type=hive_metastore)
• Storage Emulator: fsouza/fake-gcs-server
• Metastore: Hive Metastore 4.1
• Setup: Docker Compose

Configuration

1. docker-compose.yml template:

   services:
     gcs-emulator:
       image: fsouza/fake-gcs-server
       container_name: gcs-emulator
       ports:
         - "4443:4443"
       command: -scheme http -public-host gcs-emulator:4443
   
     mysql-db:
       image: mysql:8.0
       container_name: mysql-db
       environment:
         - MYSQL_ROOT_PASSWORD=secret
         - MYSQL_DATABASE=metastore
         - MYSQL_USER=hive
         - MYSQL_PASSWORD=hive
   
     hive-metastore:
       image: apache/hive:4.0.0
       container_name: hive-metastore
       depends_on: [mysql-db]
       ports: ["9083:9083"]
       volumes:
         - ./hive/conf/metastore-site.xml:/opt/hive/conf/metastore-site.xml
   
     trino:
       image: trinodb/trino:latest
       container_name: trino
       ports: ["8080:8080"]
       volumes:
         - ./trino/catalog:/etc/trino/catalog

2. metastore-site.xml:

   <configuration>
       <property>
           <name>javax.jdo.option.ConnectionURL</name>
           <value>jdbc:mysql://mysql-db:3306/metastore</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionDriverName</name>
           <value>com.mysql.cj.jdbc.Driver</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionUserName</name>
           <value>hive</value>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionPassword</name>
           <value>hive</value>
       </property>
       <property>
           <name>fs.gs.project.id</name>
           <value>dummy</value> 
       </property>
       <property>
           <name>fs.gs.auth.type</name>
           <value>NONE</value> <!-- or UNATHENTICATED, see https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.8/gcs/CONFIGURATION.md#user-credentials -->
       </property>
       <property>
           <name>google.cloud.storage.api.endpoint</name>
           <value>http://gcs-emulator:4443</value>
       </property>
   </configuration>

3. iceberg.properties (the problematic configuration):

   connector.name=iceberg
   iceberg.catalog.type=hive_metastore
   hive.metastore.uri=thrift://hive-metastore:9083
   
   # Configuration for Trino's Native GCS Client, see: https://trino.io/docs/current/object-storage/file-system-gcs.html#general-configuration
   fs.native-gcs.enabled=true
   gcs.endpoint=http://gcs-emulator:4443
   
   # A dummy key file is required to prevent "ADC not found" errors,
   # but this key triggers a real auth attempt.
   # The content of the file is a syntactically valid but fake key.
   gcs.json-key-file-path=/etc/trino/dummy-credentials.json 

   (Note: The user will need to create and mount a dummy credentials file for this step, but the key is that even with it, the process fails).

   # Generate the key
   openssl genrsa -out private_key.pem 2048
   openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in private_key.pem
   
   # Then copy the output and format for JSON
   # Replace each newline with \n in the JSON file

4. Start the services and prepare the emulator:

   # Start all containers
   docker-compose up -d
   
   # Create a bucket in the emulator
   curl -X POST -H "Content-Type: application/json" \
     --data '{"name": "test-bucket"}' \
     http://localhost:4443/storage/v1/b

5. Run the failing SQL command via the Trino CLI or any client:

   CREATE SCHEMA iceberg.test_schema
   WITH ( location = 'gs://test-bucket/test_schema' );

Actual Result

The query fails with an error indicating an authentication failure with the real Google OAuth2 endpoint.

Query failed: GCS service error listing files: gs://test-bucket/test_schema.db/

...
Caused by: com.google.cloud.storage.StorageException: Error getting access token for service account: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid grant: account not found"}
...

Analysis and Suggested Solution

The root cause is that the native GCS client's authentication logic is tightly coupled with the official Google authentication libraries, which do not recognize the gcs.endpoint property for authentication calls. The client sees a service account key and immediately attempts to exchange it for a real token.

A new configuration property is needed to explicitly disable this behavior for local testing. I suggest adding a property like gcs.authentication-type with a possible value of NONE.

Refs:

• https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.8/gcs/CONFIGURATION.md#authentication
• https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.1.8/gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystemConfiguration.java
• https://github.com/trinodb/trino/blob/master/lib/trino-filesystem-gcs/src/main/java/io/trino/filesystem/gcs/GcsFileSystemModule.java

[wendigo]

This is not the use case we want to support. If there is broader interest in that - we can revisit.

[szisiu]

The core of the changes revolves around enhancing the Google Cloud Storage (GCS) authentication mechanism, making it explicit, clear and more flexible.

Key Changes:

• GCS Authentication Refactoring:
  • The previous authentication logic, which was mostly conditional, has been replaced with a strategy pattern based on the new GcsAuthType enum. This enum includes types like SERVICE_ACCOUNT_KEY, APPLICATION_DEFAULT, ACCESS_TOKEN, and UNAUTHENTICATED.
  • The GcsDefaultAuth class has been renamed to GcsServiceAccountKeyAuth to more accurately reflect its purpose.
  • New authentication handler classes have been introduced:
    • GcsAdcAuth: For handling authentication via Google Application Default Credentials.
    • GcsNoAuth: For scenarios requiring no authentication.
  • The configuration in GcsFileSystemConfig has been updated to use the gcs.auth-type property, replacing the boolean gcs.use-access-token.
  • The GcsFileSystemModule now dynamically binds the appropriate authentication implementation based on the selected GcsAuthType.

Code Diff:

Here are some of the key code modifications:

// ... existing code ...
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;

import static com.google.common.collect.ImmutableList.toImmutableList;
import static java.nio.charset.StandardCharsets.UTF_8;

public class GcsServiceAccountKeyAuth
        implements GcsAuth
{
    private final Optional<GoogleCredentials> jsonGoogleCredential;

    @Inject
    public GcsServiceAccountKeyAuth(GcsFileSystemConfig config)
            throws IOException
    {
        String jsonKey = config.getJsonKey();
        String jsonKeyFilePath = config.getJsonKeyFilePath();

        if (jsonKey != null) {
            try (InputStream inputStream = new ByteArrayInputStream(jsonKey.getBytes(UTF_8))) {
                jsonGoogleCredential = Optional.of(GoogleCredentials.fromStream(inputStream).createScoped(DEFAULT_SCOPES));
            }
        }
        else if (jsonKeyFilePath != null) {
            try (FileInputStream inputStream = new FileInputStream(jsonKeyFilePath)) {
                jsonGoogleCredential = Optional.of(GoogleCredentials.fromStream(inputStream).createScoped(DEFAULT_SCOPES));
            }
        }
        else {
            jsonGoogleCredential = Optional.empty();
        }
    }

    @Override
    public void setAuth(StorageOptions.Builder builder, ConnectorIdentity identity)
    {
        GoogleCredentials credentials = jsonGoogleCredential.orElseGet(() -> {
            try {
                return GoogleCredentials.getApplicationDefault();
            }
            catch (IOException e) {
                // This is consistent with the GCP SDK when no credentials are available in the environment
                return null;
            }
        });

        if (credentials != null) {
            builder.setCredentials(credentials);
        }
    }
}

package io.trino.filesystem.gcs;

import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.storage.StorageOptions;
import io.trino.spi.security.ConnectorIdentity;

import java.io.IOException;

public class GcsAdcAuth
        implements GcsAuth
{
    @Override
    public void setAuth(StorageOptions.Builder builder, ConnectorIdentity identity)
    {
        GoogleCredentials credentials;
        try {
            credentials = GoogleCredentials.getApplicationDefault();
        }
        catch (IOException e) {
            // This is consistent with the GCP SDK when no credentials are available in the environment
            credentials = null;
        }
        if (credentials != null) {
            builder.setCredentials(credentials);
        }
    }
}

public class GcsFileSystemConfig
{
// ... existing code ...
    private String projectId;
    private Optional<String> endpoint = Optional.empty();

    private GcsAuthType authType = GcsAuthType.SERVICE_ACCOUNT_KEY;
    private String jsonKey;
    private String jsonKeyFilePath;
    private int maxRetries = 20;
// ... existing code ...
    public GcsFileSystemConfig setBatchSize(int batchSize)
    {
        this.batchSize = batchSize;
        return this;
    }

    @NotNull
    public GcsAuthType getAuthType()
    {
        return authType;
    }

    @Config("gcs.auth-type")
    @ConfigDescription("Authentication type for GCS: APPLICATION_DEFAULT, ACCESS_TOKEN, SERVICE_ACCOUNT_KEY, UNAUTHENTICATED")
    public GcsFileSystemConfig setAuthType(GcsAuthType authType)
    {
        this.authType = authType;
        return this;
    }

    @Nullable
    public String getProjectId()
// ... existing code ...
    public GcsFileSystemConfig setEndpoint(String endpoint)
    {
        this.endpoint = Optional.ofNullable(endpoint);
        return this;
    }

    @Nullable
    public String getJsonKey()
    {
// ... existing code ...
    @AssertTrue(message = "Either gcs.use-access-token or gcs.json-key or gcs.json-key-file-path must be set")
    public boolean isAuthMethodValid()
    {
        return switch (authType) {
            case APPLICATION_DEFAULT, ACCESS_TOKEN, UNAUTHENTICATED -> jsonKey == null && jsonKeyFilePath == null;
            case SERVICE_ACCOUNT_KEY -> jsonKey != null || jsonKeyFilePath != null;
        };
    }
}

package io.trino.filesystem.gcs;

import com.google.inject.Binder;
import com.google.inject.Scopes;
import io.airlift.configuration.AbstractConfigurationAwareModule;

import static io.airlift.configuration.ConditionalModule.conditionalModule;
import static io.airlift.configuration.ConfigBinder.configBinder;

public class GcsFileSystemModule
        extends AbstractConfigurationAwareModule
{
    @Override
    protected void setup(Binder binder)
    {
        configBinder(binder).bindConfig(GcsFileSystemConfig.class);
        binder.bind(GcsStorageFactory.class).in(Scopes.SINGLETON);
        binder.bind(GcsFileSystemFactory.class).in(Scopes.SINGLETON);

        switch (buildConfigObject(GcsFileSystemConfig.class).getAuthType()) {
            case UNAUTHENTICATED:
                binder.bind(GcsAuth.class).to(GcsNoAuth.class).in(Scopes.SINGLETON);
                break;
            case ACCESS_TOKEN:
                binder.bind(GcsAuth.class).to(GcsAccessTokenAuth.class).in(Scopes.SINGLETON);
                break;
            case APPLICATION_DEFAULT:
                binder.bind(GcsAuth.class).to(GcsAdcAuth.class).in(Scopes.SINGLETON);
                break;
            case SERVICE_ACCOUNT_KEY:
            default:
                binder.bind(GcsAuth.class).to(GcsServiceAccountKeyAuth.class).in(Scopes.SINGLETON);
                break;
        }
    }
}

[szisiu]

directly related issues an PRs:

• Should be possible to use GCS without authentication #25810
• Allow using GCS without providing credentials #25811 (v477)
• Add gcs.auth-type configuration #26681 (v478)
• Remove deprecated gcs.use-access-token from gcs filesystem #26941 (v479)