feat:Add hadoop catalog mode(include s3,s3a(minio) and hdfs )

[awol2005ex]

feat:Add hadoop catalog mode(include s3,s3a(minio) and hdfs )

[liurenjie1024]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

[awol2005ex]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

I have many Iceberg tables on Hadoop HDFS catalog, and I don't want to migrate them. StaticTable is very difficult to use and lacks practical utility.

[liurenjie1024]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

I have many Iceberg tables on Hadoop HDFS catalog, and I don't want to migrate them. StaticTable is very difficult to use and lacks practical utility.

Hi, @awol2005ex The Catalog trait if an open trait, so you can maintain customized implementation outside of community if you need one. But the community will deprecate file system base catalog.

[awol2005ex]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

I have many Iceberg tables on Hadoop HDFS catalog, and I don't want to migrate them. StaticTable is very difficult to use and lacks practical utility.

Hi, @awol2005ex The Catalog trait if an open trait, so you can maintain customized implementation outside of community if you need one. But the community will deprecate file system base catalog.

My main goal is to query Iceberg tables using iceberg-python, but this library depends on iceberg-rust. However, since iceberg-rust doesn't support HDFS or Hadoop catalog, it's essentially unusable for my needs.

[liurenjie1024]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

I have many Iceberg tables on Hadoop HDFS catalog, and I don't want to migrate them. StaticTable is very difficult to use and lacks practical utility.

Hi, @awol2005ex The Catalog trait if an open trait, so you can maintain customized implementation outside of community if you need one. But the community will deprecate file system base catalog.

My main goal is to query Iceberg tables using iceberg-python, but this library depends on iceberg-rust. However, since iceberg-rust doesn't support HDFS or Hadoop catalog, it's essentially unusable for my needs.

I don't remember iceberg-python has migrated all its catalog implementation to iceberg-rust? cc @Fokko @kevinjqliu

[awol2005ex]

Hi, @awol2005ex thanks for the contribution. The community direction is to deprecate pure file system catalog(like hadoop catalog). If you need to query iceberg table on file systems without relying on a catalog, you could use StaticTable.

I have many Iceberg tables on Hadoop HDFS catalog, and I don't want to migrate them. StaticTable is very difficult to use and lacks practical utility.

Hi, @awol2005ex The Catalog trait if an open trait, so you can maintain customized implementation outside of community if you need one. But the community will deprecate file system base catalog.

My main goal is to query Iceberg tables using iceberg-python, but this library depends on iceberg-rust. However, since iceberg-rust doesn't support HDFS or Hadoop catalog, it's essentially unusable for my needs.

I don't remember iceberg-python has migrated all its catalog implementation to iceberg-rust? cc @Fokko @kevinjqliu

iceberg-python is just a wrapper for pyiceberg-core, and pyiceberg-core is located within the iceberg-rust repository

[Xuanwo]

iceberg-python is just a wrapper for pyiceberg-core, and pyiceberg-core is located within the iceberg-rust repository

Hi, this understanding is incorrect.

While pyiceberg did attempt to reuse some iceberg-rust code, everything is entirely optional. As of now, pyiceberg's catalog implementations are written purely in Python.