Skip to content

Extend Wayang with spatial operators#699

Open
MaxSpeer wants to merge 1 commit intoapache:mainfrom
Spatial-Data-MP:main
Open

Extend Wayang with spatial operators#699
MaxSpeer wants to merge 1 commit intoapache:mainfrom
Spatial-Data-MP:main

Conversation

@MaxSpeer
Copy link

Hi everyone,
together with @zkaoudi we worked on extending wayang with spatial operations.
Please consider our PR. We are looking forward to your feedback.

Summary

This PR introduces spatial processing capabilities in Wayang by adding two new logical operators: SpatialFilter and SpatialJoin, with platform-specific implementations for Java, Spark, and PostgreSQL/JDBC.
The goal is to enable spatial filter and join operations end-to-end across Wayang’s Java, Spark, and Postgres execution paths and remain extendable for other spatial operations.

What Changed

  • Added new logical operators in the common layer: SpatialFilterOperator and SpatialJoinOperator.
  • Added spatial abstractions in core: SpatialPredicate and SpatialGeometry.
  • Added a dedicated spatial plugin module: wayang-plugins/wayang-spatial.
  • Added mappings and operators for: Java, Spark, and PostgreSQL/JDBC.
  • Extended Java API with: spatialFilter(...) and spatialJoin(...).
  • Added GeoJsonFileSource support in the spatial plugin (logical + Java implementation).
  • Extended JDBC SQL planning/execution path to include spatial filter/join operators in SQL query construction.

Design Choices

  1. Core remains only defines spatial contracts (SpatialGeometry, SpatialPredicate) and has no direct JTS/Sedona/PostGIS dependency.
  2. Backend-specific dependencies are isolated in the spatial plugin JTS/Sedona/PostGIS specifics are implemented in wayang-spatial, keeping the API/core clean and modular.

Platform Implementation Notes

  • Java: Uses JTS-based predicate evaluation and spatial index support for joins.
  • Spark: Uses Apache Sedona for distributed spatial filtering and joining.
  • PostgreSQL/JDBC: Generates spatial SQL clauses, enabling DB-side execution of filter/join. (Limitation: JDBCExecutor does not support multiple sources at the moment)

Testing

  • Unit tests for: WayangGeometry, Java operators, Spark operators, JDBC SQL generation.
  • Java API tests for: spatial filter/join usage and chained operations.
  • Postgres integration tests are present but disabled by default due to external DB dependency.

---------
Co-authored-by: treyfel <58570309+treyfel@users.noreply.github.com>
Co-authored-by: maxspeer <82819900+MaxSpeer@users.noreply.github.com>
@mspruc
Copy link
Contributor

mspruc commented Feb 26, 2026

Hi thanks for your contribution,
I will have a deeper look at your PR later but in the meantime, can you clean up the various System.out.println(...) statements?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants