nessie cannot obtain the S3 access key and secret #11759
Description
Issue description
I deployed Nessie using Helm with the following configuration
catalog:
enabled: true
iceberg:
defaultWarehouse: big-data
objectStoresHealthCheckEnabled: true
warehouses:
- location: s3://big-data/iceberg/
name: big-data
storage:
s3:
buckets:
- accessKeySecret:
awsAccessKeyId: username
awsSecretAccessKey: password
name: test-big-data-minio-cluster-big-data-admin-kv
name: big-data
pathPrefix: iceberg/The relevant configuration for the generated pod is as follows
containers:
- env:
- name: quarkus.datasource.postgresql.username
valueFrom:
secretKeyRef:
key: username
name: postgresql-nessie-user
- name: quarkus.datasource.postgresql.password
valueFrom:
secretKeyRef:
key: password
name: postgresql-nessie-user
- name: nessie.catalog.service.s3.buckets.bucket1.access-key
value: >-
urn:nessie-secret:quarkus:nessie-catalog-secrets.s3.buckets.bucket1.access-key
- name: nessie-catalog-secrets.s3.buckets.bucket1.access-key.name
valueFrom:
secretKeyRef:
key: username
name: test-big-data-minio-cluster-big-data-admin-kv
- name: nessie-catalog-secrets.s3.buckets.bucket1.access-key.secret
valueFrom:
secretKeyRef:
key: password
name: test-big-data-minio-cluster-big-data-admin-kvThe generated ConfigMap nessie is as follows
application.properties
nessie.catalog.default-warehouse=big-data
nessie.catalog.object-stores.health-check.enabled=true
nessie.catalog.service.s3.buckets.bucket1.name=big-data
nessie.catalog.service.s3.buckets.bucket1.path-prefix=iceberg/
nessie.catalog.service.s3.default-options.endpoint=http://test-big-data-minio-cluster-hp.big-data
nessie.catalog.service.s3.default-options.path-style-access=true
nessie.catalog.service.s3.default-options.region=us-east-1
nessie.catalog.warehouses."big-data".location=s3://big-data/iceberg/
nessie.version.store.persist.cache-invalidations.service-names=nessie-mgmt
nessie.version.store.persist.cache-invalidations.valid-tokens=934ec3b3da3a66a938b35bd3280d3841c4ee6f47
nessie.version.store.persist.jdbc.datasource=postgresql
nessie.version.store.type=JDBC2
quarkus.datasource.postgresql.jdbc.url=jdbc:postgresql://postgresql-nessie-rw:5432/nessie?currentSchema=nessie
quarkus.http.port=19120
quarkus.log.category."org.projectnessie".level=INFO
quarkus.log.console.enable=true
quarkus.log.console.format=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n
quarkus.log.console.level=ALL
quarkus.log.file.enable=false
quarkus.log.level=INFO
quarkus.log.sentry=false
quarkus.management.port=9000
quarkus.oidc.tenant-enabled=false
quarkus.otel.sdk.disabled=true
But nessie cannot obtain the S3 access key and secret
INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/quarkus-run.jar
INFO running in /deployments
_ _ _ ____
| \ | | (_) / __ \
| \| | ___ ___ ___ _ ___ / /__\/ ___ _ ____ _____ _ __
| . ` |/ _ \/ __/ __| |/ _ \ \___. \/ _ \ '__\ \ / / _ \ '__|
| |\ | __/\__ \__ \ | __/ /\__/ / __/ | \ V / __/ |
\_| \_/\___||___/___/_|\___| \____/ \___|_| \_/ \___|_|
https://projectnessie.org/
Powered by Quarkus 3.30.2
2025-12-17 10:22:40,278 WARN [org.hib.val.int.met.agg.CascadingMetaDataBuilder] (main) HV000271: Using `@Valid` on a container (java.util.List) is deprecated. You should apply the annotation on the type argument(s). Affected element: ContentService#getMultipleContents(String, String, List, boolean, RequestMeta)
2025-12-17 10:22:40,311 WARN [org.hib.val.int.met.agg.CascadingMetaDataBuilder] (main) HV000271: Using `@Valid` on a container (java.util.List) is deprecated. You should apply the annotation on the type argument(s). Affected element: ContentService#getMultipleContents(String, String, List, boolean, RequestMeta)
2025-12-17 10:22:40,605 WARN [io.qua.config] (main) Unrecognized configuration key "quarkus.log.sentry" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2025-12-17 10:22:40,610 WARN [io.qua.config] (main) The "quarkus.log.file.enable" config property is deprecated and should not be used anymore.
2025-12-17 10:22:40,610 WARN [io.qua.config] (main) The "quarkus.log.console.enable" config property is deprecated and should not be used anymore.
2025-12-17 10:22:40,874 INFO [org.pro.ser.cat.sec.SecretsProducers] (main) No external secrets manager has been configured, secrets are retrieved only from the Quarkus configuration.
2025-12-17 10:22:40,909 INFO [org.pro.nes.net.AddressResolver] (main) Using nameserver kube-dns.kube-system.svc.cluster.local/10.43.0.10 with search list [big-data.svc.cluster.local, svc.cluster.local, cluster.local]
2025-12-17 10:22:40,936 INFO [org.pro.ser.dis.CacheInvalidationSender] (main) Sending remote cache invalidations to service name(s) [nessie-mgmt]
2025-12-17 10:22:40,956 INFO [org.pro.ser.dis.CacheInvalidationSender] (vert.x-eventloop-thread-1) Service names for remote cache invalidations [nessie-mgmt] now resolve to [10.42.3.165]
2025-12-17 10:22:40,957 WARN [org.pro.ser.con.ConfigChecks] (main) Both authentication (AuthN) and authorization (AuthZ) are disabled, all requests to Nessie will be permitted. This means: everybody with access to Nessie can read, write and change everything. Recommended action: Enable AuthN & AuthZ, see https://projectnessie.org/nessie-latest/configuration/
2025-12-17 10:22:40,967 INFO [org.pro.qua.pro.sto.PersistProvider] (main) Using objects cache with 54591 MB, with soft-references disabled, enabling distributed cache invalidations.
2025-12-17 10:22:40,968 INFO [org.pro.qua.pro.sto.Jdbc2BackendBuilder] (main) Selected datasource: postgresql
2025-12-17 10:22:41,123 INFO [org.pro.qua.pro.sto.PersistProvider] (main) Creating/opening version store JDBC2 ...
2025-12-17 10:22:41,125 INFO [org.pro.qua.pro.sto.PersistProvider] (main) Using JDBC2 version store (catalog: nessie, schema: nessie, table 'refs2' looks compatible, table 'objs2' looks compatible)
2025-12-17 10:22:41,290 INFO [io.quarkus] (main) Nessie 0.106.0 on JVM (powered by Quarkus 3.30.2) started in 2.172s. Listening on: http://0.0.0.0:19120. Management interface listening on http://0.0.0.0:9000.
2025-12-17 10:22:41,291 INFO [io.quarkus] (main) Profile prod activated.
2025-12-17 10:22:41,291 INFO [io.quarkus] (main) Installed features: [agroal, amazon-sdk-dynamodb, amazon-sdk-secretsmanager, azure-keyvault-secret, cassandra-client, cdi, google-cloud-bigtable, google-cloud-secret-manager, hibernate-validator, jdbc-h2, jdbc-mariadb, jdbc-postgresql, logging-sentry, micrometer, mongodb-client, narayana-jta, oidc, opentelemetry, reactive-routes, rest, rest-jackson, security, security-properties-file, smallrye-context-propagation, smallrye-health, smallrye-openapi, vault, vertx]
2025-12-17 10:22:50,223 ERROR [org.pro.ser.cat.ObjectStoresHealthCheck] (executor-thread-3) Failed to ping warehouse 'big-data', error ID 5aff2f99-4676-4a56-b62e-5b17d84ab0f5: java.lang.IllegalArgumentException: Missing access key and secret for STATIC authentication mode
at org.projectnessie.catalog.files.s3.S3Utils.lambda$newCredentialsProvider$0(S3Utils.java:170)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at org.projectnessie.catalog.files.s3.S3Utils.newCredentialsProvider(S3Utils.java:168)
at org.projectnessie.catalog.files.s3.S3Clients.serverCredentialsProvider(S3Clients.java:130)
at org.projectnessie.catalog.files.s3.S3ClientSupplier.getClient(S3ClientSupplier.java:89)
at org.projectnessie.catalog.files.s3.S3ClientSupplier.getClient(S3ClientSupplier.java:81)
at org.projectnessie.catalog.files.s3.CatalogProducers_ProducerMethod_s3ClientSupplier_EyTd7wIsN-D4oy5TC3dx1Me4V9c_ClientProxy.getClient(Unknown Source)
at org.projectnessie.catalog.files.s3.S3ObjectIO.ping(S3ObjectIO.java:71)
at org.projectnessie.catalog.files.DelegatingObjectIO.ping(DelegatingObjectIO.java:35)
at org.projectnessie.catalog.files.api.CatalogProducers_ProducerMethod_objectIO_Ed95GkFIhtTZERiabosXvHelB7Q_ClientProxy.ping(Unknown Source)
at org.projectnessie.server.catalog.ObjectStoresHealthCheck.call(ObjectStoresHealthCheck.java:61)
at org.projectnessie.server.catalog.ObjectStoresHealthCheck_ClientProxy.call(Unknown Source)
at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:35)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap.subscribe(UniOnFailureFlatMap.java:34)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:35)
at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:35)
at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
at io.quarkus.smallrye.health.runtime.QuarkusAsyncHealthCheckFactory$1$1.call(QuarkusAsyncHealthCheckFactory.java:42)
at io.quarkus.smallrye.health.runtime.QuarkusAsyncHealthCheckFactory$1$1.call(QuarkusAsyncHealthCheckFactory.java:39)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$4(ContextImpl.java:192)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221)
at io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1586)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)