From 3750f8eca7aabe1fccbd44a79ab435f84bfc4527 Mon Sep 17 00:00:00 2001 From: HideBa Date: Wed, 17 Dec 2025 13:36:40 +0100 Subject: [PATCH 1/5] docs: organise docs for LLM --- .agent/rules/general.md | 5 + .cursor/rules/general.mdc | 74 +-- .cursor/rules/memory/productContext.md | 131 ------ .cursor/rules/rust.mdc | 10 +- .llm/docs/productContext.md | 28 ++ .../memory => .llm/docs}/specification.md | 56 +-- AGENTS.md | 422 +----------------- CLAUDE.md | 38 +- 8 files changed, 114 insertions(+), 650 deletions(-) create mode 100644 .agent/rules/general.md delete mode 100644 .cursor/rules/memory/productContext.md create mode 100644 .llm/docs/productContext.md rename {.cursor/rules/memory => .llm/docs}/specification.md (87%) mode change 120000 => 100644 CLAUDE.md diff --git a/.agent/rules/general.md b/.agent/rules/general.md new file mode 100644 index 0000000..c11fa2a --- /dev/null +++ b/.agent/rules/general.md @@ -0,0 +1,5 @@ +--- +trigger: always_on +--- + +This file is equivalent to what is written in .cursor/rules/general.mdc. See the file. diff --git a/.cursor/rules/general.mdc b/.cursor/rules/general.mdc index ec9a671..d1dc3c5 100644 --- a/.cursor/rules/general.mdc +++ b/.cursor/rules/general.mdc @@ -7,36 +7,24 @@ alwaysApply: true ## General Guidelines -- The user is more proficient in programming than Cline and Cusor, but requests Cline's and Cursor assistance for coding to save time. - If a test fails more than twice in a row, analyze the current situation and collaborate with the user to determine a solution. Avoid trial-and-error testing without a hypothesis. -- The user has extensive knowledge gained from GitHub and can implement individual algorithms and libraries faster than Cline and Cursor. Code should be written while explaining to the user, using test cases to verify correctness. -- However, Cline and Cursor is not good at handling processing logic based on the current context. If the context is unclear, confirm with the user. -- Once I ask you to do "memory it", you should update [general.mdc](mdc:.cursor/rules/general.mdc) or other corresponding document files such as [productContext.md](mdc:.cursor/rules/memory/productContext.md), [specification.md](mdc:.cursor/rules/memory/specification.md), [progress.md](mdc:.cursor/rules/memory/progress.md) so they are kept update to date. Even I don't ask, you should ask me to save memory or not when you think it's needed. - ---- - -# Cline and Cursor Memory Bank - -I am a specialized software engineer with a distinct characteristic: my memory resets completely between sessions. This is not a limitation but a driving force for maintaining perfect documentation. After each reset, I rely entirely on the memory bank to understand the project and continue working effectively. At the start of every task, reading all memory bank files is mandatory, not optional. - -## Memory Bank Structure - -All files are stored under [productContext.md](mdc:.cursor/rules/memory/productContext.md), [progress.md](mdc:.cursor/rules/memory/progress.md), [specification.md](mdc:.cursor/rules/memory/specification.md) and `.cursor/memory/*`. The memory bank consists of mandatory core files and optional context files, all formatted in Markdown. +- The user has extensive knowledge gained from GitHub and can implement individual algorithms and libraries faster than you. Code should be written while explaining to the user, using test cases to verify correctness. +- When you work on the bigger tasks, you should plan first and document list of tasks to be done and update progress.md ### Core Files (Mandatory) -1. **`productContext.md`** [productContext.md](mdc:.cursor/rules/memory/productContext.md) +1. **`productContext.md`** [productContext.md](mdc:.llm/docs/productContext.md) - Explains the purpose of the project. - Identifies the problem it solves. - Describes expected functionality and user experience goals. -2. **`specification.md`** [specification.md](mdc:.cursor/rules/memory/specification.md) +2. **`specification.md`** [specification.md](mdc:.llm/docs/specification.md) - Explains the detail of FlatCityBuf specification - Describes its encoding strategy and decisions made ### Additional Context Files -Additional files and folders can be created inside `memory/rules/memory*` if they aid in organization: +Additional files and folders can be created inside `.llm/docs/*` if they aid in organization: - Documentation for complex features. - Integration specifications. @@ -46,55 +34,7 @@ Additional files and folders can be created inside `memory/rules/memory*` if the --- -# Git Workflow - -## Commit Best Practices - -1. **Review Changes** - - ```bash - git status - git diff - git log - ``` - -2. **Analyze Changes** - - Identify modified or added files. - - Understand the nature of changes (new feature, bug fix, refactoring, etc.). - - Evaluate the impact on the project. - - Ensure no sensitive information is exposed. -3. **Create Meaningful Commit Messages** - - ```bash - git commit -m "fix: Resolve issue with authentication timeout" - ``` - -## Pull Request Best Practices - -1. **Review Branch Status** - - ```bash - git status - git diff main...HEAD - git log - ``` - -2. **Analyze Changes** - - Review all commits made since branching off `main`. - - Assess change scope and impact. - - Ensure no sensitive data is committed. -3. **Create a Pull Request** - - ```bash - gh pr create --title "feat: Improve Rust error handling" --body "Improved error handling with Result." - ``` - ---- - -# Local MCP - -## `serena` - -- `serena` is a tool that can be used to get overview of the code base. +## Integration with GitHub +you can expect that runtime has gh CLI, rather than using GitHub MCP, simply use gh CLI to create pull requests, etc. --- diff --git a/.cursor/rules/memory/productContext.md b/.cursor/rules/memory/productContext.md deleted file mode 100644 index 6d1c794..0000000 --- a/.cursor/rules/memory/productContext.md +++ /dev/null @@ -1,131 +0,0 @@ -# **Cloud-Optimized CityJSON** - -## **1. Introduction** - -- **Motivation & Project Context**: - - - Standardizing **3D city model data formats** is crucial for long-term semantic storage of urban environments. - - **CityJSON**, a widely adopted **OGC standard**, provides a structured JSON-based format for 3D city models. - - **CityJSONSeq** improved streaming but lacks **cloud-native optimizations** for handling large-scale datasets. - -- **Problem Statement**: - - - Existing 3D model formats like **CityJSON and CityJSONSeq** are **not optimized** for large-scale **cloud processing**. - - **Scalability challenges** arise from high **storage costs, slow queries, and inefficient downloading** of large datasets. - - **Limited support for binary serialization** and **spatial indexing** prevents efficient cloud-based data retrieval. - - **Research Gaps**: - - Few studies have evaluated **FlatBuffers in geospatial applications**. - - Limited focus on **efficient cloud-native processing** of 3D city models. - - **Preserving CityJSON's semantic richness** while optimizing for **fast cloud retrieval** remains a challenge. - -- **Goal of This Specification**: - - Develop an **optimized CityJSON format** based on **FlatBuffers**, improving: - - **Data retrieval speed** via **spatial indexing (Hilbert R-tree)**. - - **Query performance** through **efficient attribute-based and spatial searches**. - - **Cloud efficiency** with **HTTP Range Requests for partial fetching**. - - Ensure **backward compatibility** with **CityJSON 2.0**. - ---- - -## **2. Design Goals and Requirements** - -- **Performance & Efficiency**: - - - Reduce **processing overhead** using **FlatBuffers' zero-copy access**. - - **Optimize storage** via **binary encoding**, reducing file sizes. - -- **Cloud & Web Compatibility**: - - - **Enable partial data retrieval** via **HTTP Range Requests**. - - **Support spatial sorting and indexing** for scalable cloud processing. - -- **Scalability & Integration**: - - - Ensure **interoperability** with **existing GIS tools** (QGIS, Cesium, Mapbox). - - **Reduce cloud storage & computation costs**. - -- **End-User Goals**: - - **Faster downloads** of arbitrary **3D city model subsets**. - - **Web applications** that **load city models instantly**. - ---- - -## **3. Data Model and Encoding Structure** - -### **3.1 Enhancements to CityJSON** - -- **CityJSON 2.0**: - - - **JSON-based format** for 3D city models. - - Uses **shared vertex lists** to improve storage efficiency. - -- **CityJSONSeq (Streaming Format)**: - - Breaks datasets into **individual objects** for **incremental processing**. - - Still **text-based**, leading to **higher memory usage**. - -### **3.2 FlatBuffers-Based Encoding** - -- **Schema Definition**: - - - Stores **CityObjects as FlatBuffers tables**. - - Implements **hierarchical storage** with **efficient geometry encoding**. - -- **Memory Optimization**: - - Uses **separate arrays for geometric primitives** (solids, shells, surfaces, rings). - - **Avoids nested JSON objects**, leading to **faster parsing**. - -### **3.3 File Structure** - -| **Component** | **Description** | -| ------------------- | --------------------------------------------------- | -| **Magic Bytes** | File identifier for format validation. | -| **Header** | Stores **metadata, CRS, transformations**. | -| **Spatial Index** | **Byte offsets** for fast random access. | -| **Attribute Index** | **Byte offsets** for fast random access. | -| **Features** | Encodes **CityJSON objects as FlatBuffers tables**. | - ---- - -## **4. Data Organization and Storage Mechanism** - -### **4.1 Spatial Indexing** - -- Implements a **Packed Hilbert R-tree** to: - - Maximally fill the available space in the node. - - Enable **selective data retrieval** within a bounding box. - - Support **three types of spatial queries**: - - **Bounding Box (bbox)**: Find all features that intersect with a given bounding box. - - **Point Intersection**: Find all features whose bounding box contains a given point. - - **Nearest Neighbor**: Find the feature whose bounding box centroid is nearest to a given point. - -### **4.2 Attribute Indexing** - -- Implements a **Static(Implicit) B+tree** to: - - Enable **efficient attribute-based querying**. - - Support **range queries** and **Exact Match queries**. - - Maximally fill the available space in the node except for the rightmost leaf node. - -### **4.3 HTTP Range Requests** - -- Enables **partial fetching**: - - Download **only required city features**, reducing data transfer. - - Spatial index and attribute index are used to determine the range of features to download. - ---- - -### **5 Rust-Based Implementation** - -- Developed as a **Rust library** for: - - **Encoding and decoding FlatBuffers-based CityJSON**. - - **Integrating with GIS workflows**. -- **WebAssembly support** for in-browser processing. - ---- - -## **6. Use Cases and Applications** - -### **6.1 Urban Planning & Smart Cities** - -- **Faster, interactive 3D city analysis** in smart city applications. -- **Real-time urban simulations**. -- **Massive data processing** diff --git a/.cursor/rules/rust.mdc b/.cursor/rules/rust.mdc index 459e7cc..2b3383e 100644 --- a/.cursor/rules/rust.mdc +++ b/.cursor/rules/rust.mdc @@ -52,6 +52,8 @@ alwaysApply: true - Follow **Rust’s API guidelines** for public interfaces. - Use **builder patterns** for complex configurations. +- Try to proper trait definition and implementation to invert dependencies and testability. +- reexport public types and functions from the root crate. --- @@ -78,11 +80,3 @@ mentation - Use `tracing` for structured logging. - Enable debug assertions wit_assert!()`. - ---- - -## Final Notes - -- Follow **Rust's idiomatic coding practices**. -- Endut e, safety, and maintainability**. -- Maintain a **black-and-white, pixelated/nerdy ltao will remain robust, efficient, and maintainable across its m crates and modules. πŸš€ diff --git a/.llm/docs/productContext.md b/.llm/docs/productContext.md new file mode 100644 index 0000000..06bf4c3 --- /dev/null +++ b/.llm/docs/productContext.md @@ -0,0 +1,28 @@ +# **Cloud-Optimized CityJSON** + +## **1. Introduction** + +- **Motivation & Project Context**: + + - Standardizing **3D city model data formats** is crucial for long-term semantic storage of urban environments. + - **CityJSON**, a widely adopted **OGC standard**, provides a structured JSON-based format for 3D city models. + - **CityJSONSeq** improved streaming but lacks **cloud-native optimizations** for handling large-scale datasets. + +- **Problem Statement**: + + - Existing 3D model formats like **CityJSON and CityJSONSeq** are **not optimized** for large-scale **cloud processing**. + - **Scalability challenges** arise from high **storage costs, slow queries, and inefficient downloading** of large datasets. + - **Limited support for binary serialization** and **spatial indexing** prevents efficient cloud-based data retrieval. + - **Research Gaps**: + - Few studies have evaluated **FlatBuffers in geospatial applications**. + - Limited focus on **efficient cloud-native processing** of 3D city models. + - **Preserving CityJSON's semantic richness** while optimizing for **fast cloud retrieval** remains a challenge. + +- **Goal of This Specification**: + - Develop an **optimized CityJSON format** based on **FlatBuffers**, improving: + - **Data retrieval speed** via **spatial indexing (Hilbert R-tree)**. + - **Query performance** through **efficient attribute-based and spatial searches**. + - **Cloud efficiency** with **HTTP Range Requests for partial fetching**. + - Ensure **backward compatibility** with **CityJSON 2.0**. + +--- diff --git a/.cursor/rules/memory/specification.md b/.llm/docs/specification.md similarity index 87% rename from .cursor/rules/memory/specification.md rename to .llm/docs/specification.md index b607f55..48a4d6e 100644 --- a/.cursor/rules/memory/specification.md +++ b/.llm/docs/specification.md @@ -57,43 +57,24 @@ FlatCityBuf supports CityJSON's Geometry Templates for efficient representation This separation allows defining complex shapes once in the header and instantiating them multiple times within features using only an index, a reference point index, and a transformation matrix. -### design rationale - -the schema design follows several key principles: - -1. **flatbuffers efficiency**: uses flatbuffers' zero-copy access for fast data retrieval -2. **hierarchical structure**: maintains cityjson's hierarchical object model -3. **shared vertices**: uses indexed vertices to reduce redundancy -4. **semantic preservation**: maintains rich semantic information from cityjson - ## file storage overview -a flatcitybuf file consists of the following sections: - ```mermaid -graph TD - A[FlatCityBuf File] --> B[Magic Bytes] - A --> C[Header Size] - A --> D[Header] - A --> E[R-tree Index] - A --> F[Attribute Index] - A --> G[Features] - - B --> B1[8 bytes identifier] - C --> C1[4 bytes uint32] - D --> D1[FlatBuffers Header] - E --> E1[Packed R-tree] - F --> F1[Static B+tree Index] - G --> G1[FlatBuffers Features] +block-beta + columns 1 + file["FlatCityBuf File Structure"] + + block:sections + columns 6 + magic["Magic Bytes
(8 bytes)"] + hsize["Header Size
(4 bytes)"] + header["Header
(FlatBuffers)"] + rtree["R-tree Index
(Spatial)"] + attr["Attribute Index
(B+tree)"] + features["Features
(FlatBuffers)"] + end ``` -1. **magic bytes**: 8 bytes identifier for the file format ('fcb\0\1\0\0\0\0\0') -2. **header size**: 4 bytes uint32 indicating the size of the header in bytes -3. **header**: flatbuffers-encoded header containing metadata, schema, and index information -4. **r-tree index**: packed r-tree for spatial indexing -5. **attribute index**: static b+tree indices for attribute queries -6. **features**: the actual city objects encoded as flatbuffers - each section is aligned to facilitate efficient http range requests, allowing clients to fetch only the parts they need. ## rtree indexing @@ -301,14 +282,3 @@ the encoding follows these principles: 2. **enum with extension marker**: special enum values combined with string fields handle extended types 3. **unified attribute storage**: extension attributes are treated the same as core attributes 4. **root properties**: extension properties are stored in the header's attributes field - -### benefits - -this implementation balances: - -- **performance**: maintains fast access to core data structures -- **flexibility**: supports any cityjson extension -- **self-containment**: makes files usable without external references -- **simplicity**: uses consistent patterns for extension handling - -the schema is agnostic about the validity of extension data, focusing on accurately representing extended cityjson files in an efficient binary format. diff --git a/AGENTS.md b/AGENTS.md index 9590d0e..2a02fcc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,415 +1,37 @@ -# AGENTS.md +# Important Rules for LLM -Agent guidance for working with the FlatCityBuf codebase. This file is shared across multiple AI coding assistants (Claude Code, Cursor, Cline, etc.). +## General Guidelines -## Project Overview +- If a test fails more than twice in a row, analyze the current situation and collaborate with the user to determine a solution. Avoid trial-and-error testing without a hypothesis. +- The user has extensive knowledge gained from GitHub and can implement individual algorithms and libraries faster than you. Code should be written while explaining to the user, using test cases to verify correctness. +- When you work on the bigger tasks, you should plan first and document list of tasks to be done and update progress.md -FlatCityBuf is a cloud-optimized binary format for storing and retrieving 3D city models. It combines the semantic richness of CityJSON with the performance benefits of FlatBuffers binary serialization and advanced spatial indexing techniques. The project enables efficient partial data retrieval from cloud storage using HTTP range requests. +### Core Files (Mandatory) -### Key Features +1. **`productContext.md`** [productContext.md](mdc:.llm/docs/productContext.md) -- **Zero-Copy Access**: FlatBuffers enables direct memory access without parsing -- **HTTP Range Optimization**: File structure aligned for efficient partial downloads -- **Hilbert Ordering**: Features sorted by Hilbert curve for spatial locality -- **Hierarchical Geometry**: Efficient encoding of 3D boundaries using dimensional arrays -- **Extension Support**: Full compatibility with CityJSON extension mechanism + - Explains the purpose of the project. + - Identifies the problem it solves. + - Describes expected functionality and user experience goals. -### Design Goals +2. **`specification.md`** [specification.md](mdc:.llm/docs/specification.md) + - Explains the detail of FlatCityBuf specification + - Describes its encoding strategy and decisions made -1. **Performance**: Reduce processing overhead using FlatBuffers' zero-copy access and optimize storage via binary encoding -2. **Cloud Compatibility**: Enable partial data retrieval via HTTP Range Requests with spatial sorting and indexing -3. **Scalability**: Ensure interoperability with existing GIS tools (QGIS, Cesium, Mapbox) and reduce cloud storage costs -4. **User Experience**: Faster downloads of arbitrary 3D city model subsets and instant web application loading +### Additional Context Files ---- - -## Working with AI Assistants - -### General Guidelines for AI Agents - -- The user is proficient in programming and requests AI assistance to save time, not for basic explanations -- If a test fails more than twice in a row, analyze the situation and collaborate with the user rather than trial-and-error testing -- Write code with explanations and use test cases to verify correctness -- If context is unclear, confirm with the user before proceeding -- When asked to "memory it", update relevant documentation files in `.cursor/rules/memory/` (productContext.md, specification.md, etc.) - -### Memory Bank System - -AI agents operate with memory resets between sessions. The memory bank under `.cursor/rules/memory/` is mandatory reading at the start of every task: - -**Core Files:** -- `productContext.md` - Project purpose, problem statement, and user experience goals -- `specification.md` - Detailed FlatCityBuf specification, encoding strategy, and design decisions +Additional files and folders can be created inside `.llm/docs/*` if they aid in organization: -**Additional Context:** -- Store complex feature documentation, integration specs, API docs, testing strategies, and deployment procedures under `.cursor/rules/memory/` +- Documentation for complex features. +- Integration specifications. +- API documentation. +- Testing strategies. +- Deployment procedures. --- -## Architecture Overview - -### Core Components - -1. **FlatBuffers Schemas** (`/src/fbs/`) - - `header.fbs` - File metadata, transformations, and index information - - `geometry.fbs` - 3D geometry structures and semantic surfaces - - `feature.fbs` - City objects and their attributes - - `extension.fbs` - Support for CityJSON extensions - -2. **Rust Workspace** (`/src/rust/`) - - `fcb_core` - Core library for reading/writing FlatCityBuf format - - `cli` - Command-line interface for file conversion and analysis - - `fcb_wasm` - WebAssembly bindings for browser usage - - `fcb_py` - Python bindings using PyO3 and maturin - - `fcb_api` - HTTP API server for FlatCityBuf operations - -3. **Indexing Structures** - - **Packed R-tree**: 2D spatial indexing using Hilbert space-filling curves for bbox/point/nearest-neighbor queries - - **Static B+Tree**: Attribute-based indexing with efficient range queries and exact match queries - - Both indices store byte offsets for direct feature access - -### File Format Structure - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Magic Bytes β”‚ 8 bytes - File identifier (fcb\0\1\0\0\0\0\0) -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Header Size β”‚ 4 bytes - Size of header -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Header β”‚ Variable - Metadata & schema -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ R-tree Index β”‚ Variable - Spatial index -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Attribute Index β”‚ Variable - B+Tree indices -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Features β”‚ Variable - City objects -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` +## Integration with GitHub -### Query Execution Flow - -**Spatial Queries:** -- Traverse R-tree using HTTP range requests -- Retrieve feature offsets from leaf nodes -- Batch nearby features to minimize requests - -**Attribute Queries:** -- Traverse B+Tree index to find matching keys -- Handle duplicates via payload section with prefetching and batch resolution -- Use prefetching and batch resolution for efficiency - -**Feature Retrieval:** -- Use offsets to fetch specific byte ranges -- Decode features using FlatBuffers zero-copy access -- Process geometry and attributes on demand +you can expect that runtime has gh CLI, rather than using GitHub MCP, simply use gh CLI to create pull requests, etc. --- - -## Common Development Commands - -### Python Development - -```bash -# Build and install Python bindings in development mode -cd src/rust && make py-develop - -# Build Python wheel for distribution -cd src/rust && make py-build - -# Run Python tests -cd src/rust && make py-test - -# Clean Python build artifacts -cd src/rust && make py-clean -``` - -### Rust Building and Testing - -```bash -# Build all Rust crates (excluding WASM) -cd src/rust -cargo build --workspace --all-features --exclude fcb_wasm --release - -# Run pre-commit checks (formatting, linting, tests) -cd src/rust && make pre-commit - -# Run tests with nextest (faster test runner) -cd src/rust -cargo nextest run --all-features --workspace --exclude fcb_wasm --exclude fcb_py - -# Run specific integration test -cd src/rust && cargo test -p fcb_core --test e2e - -# Run benchmarks -cd src/rust && cargo bench -p fcb_core --bench read -- --release -``` - -### Code Generation - -```bash -# Generate Rust code from FlatBuffers schemas -make gen-all # Runs all generation scripts -# OR specifically: -./scripts/gen_rust.sh # Generates Rust bindings from .fbs files -``` - -### WebAssembly Build - -```bash -# Build WASM module for release -cd src/rust/wasm && wasm-pack build --target web --release --out-dir ../../ts - -# Build WASM module for debug -cd src/rust && make wasm-build -``` - -### Linting and Formatting - -```bash -cd src/rust - -# Format code -cargo fmt --all - -# Run clippy with auto-fix -cargo clippy --fix --allow-dirty --workspace --all-targets --all-features --exclude fcb_wasm - -# Run clippy for WASM target -cargo clippy --fix --allow-dirty -p fcb_wasm --target wasm32-unknown-unknown - -# Check for security vulnerabilities -cargo audit -``` - ---- - -## Language-Specific Guidelines - -### Rust Development - -**General Principles:** -- Write idiomatic Rust code: clear, efficient, and maintainable -- Prioritize safety, performance, and modularity -- Follow Rust naming conventions: `snake_case` for variables/functions, `PascalCase` for types, `SCREAMING_SNAKE_CASE` for constants -- Keep code DRY using functions, modules, and generics -- Use explicit, descriptive names for variables, functions, and types -- Avoid `unwrap()` except in test cases; ensure proper error handling -- Fix grammar mistakes in comments when found - -**Error Handling:** -- Use `thiserror` for custom error types in library code (NOT `anyhow` unless explicitly approved) -- Avoid panics in library code; return errors instead -- Handle errors and edge cases early - -**Performance:** -- Use iterators instead of loops for better performance and readability -- Minimize memory allocations by using borrowed references (`&str`, `&[u8]`) -- Optimize for human readability while maintaining machine efficiency -- Use `criterion` for benchmarking - -**Async Programming:** -- Use `tokio` as the async runtime -- Prefer channels over mutexes where applicable -- Implement structured concurrency using `tokio::select!` -- Use `tokio::sync::mpsc` for multi-producer, single-consumer communication - -**API Design:** -- Follow Rust's API guidelines for public interfaces -- Use builder patterns for complex configurations - -**Testing:** -- Write unit tests with `#[cfg(test)]` -- Use integration tests for public APIs in `tests/` directory -- Mock external dependencies where necessary -- Use `tokio::test` for async tests - -**Documentation:** -- Write Rustdoc comments for public functions and structs -- Include examples in documentation - -**Dependency Management:** -- Use `cargo-audit` to check for vulnerabilities -- Keep dependencies minimal and up-to-date -- Add crates to workspace's `Cargo.toml` file, not individual crates - -**Logging:** -- Use `tracing` for structured logging -- Enable debug assertions with `debug_assert!()` - -### TypeScript/JavaScript Development - -**Development Philosophy:** -- Write clean, maintainable, and scalable code -- Follow SOLID principles -- Prefer functional and declarative programming patterns -- Emphasize type safety and static analysis -- Practice component-driven development - -**Code Style:** -- Write concise, modular TypeScript code -- Prefer functional components over class components -- Avoid code duplication; prioritize iteration and modularization -- Use descriptive variable names with auxiliary verbs (e.g., `isLoading`, `hasError`) -- File structure: exported component, subcomponents, helpers, static content, types - -**Naming Conventions:** -- Directories & Files: kebab-case (e.g., `components/auth-wizard`) -- Components: PascalCase (e.g., `UserProfile`) -- Variables, Functions, Methods, Props: camelCase (e.g., `fetchData`) -- Boolean Variables: Prefix with verbs (e.g., `isLoading`, `hasError`) -- Event Handlers: Prefix with `handle` (e.g., `handleClick`) -- Custom Hooks: Prefix with `use` (e.g., `useAuth`) -- Constants & Environment Variables: UPPER_CASE (e.g., `API_URL`) - -**TypeScript Usage:** -- Use TypeScript for all code -- Prefer interfaces over types -- Avoid enums; use maps instead -- Enable strict mode for better type safety -- Use TypeScript utility types (`Partial`, `Pick`, `Omit`) -- Apply generics where necessary - -**Syntax and Formatting:** -- Use single quotes (`'`) for strings -- Use early returns to improve readability -- Use `const` whenever possible -- Omit semicolons unless required for disambiguation -- Always use strict equality (`===`) -- Use Prettier for consistent formatting -- Keep line length under 80 characters - -**UI and Styling:** -- Use Tailwind CSS for utility-based styling -- Use Shadcn UI for accessible components -- Use Radix UI primitives where necessary -- Follow mobile-first responsive design -- Implement dark mode using CSS variables or Tailwind -- Ensure high accessibility (a11y) standards using ARIA roles and semantic HTML - -**State Management:** -- Use `useState` for component-level state -- Use `useReducer` for complex state logic -- Use `useContext` for shared state -- Use Redux Toolkit for global state management - -**Performance:** -- Minimize the use of `useState` and `useEffect` -- Use `useCallback` to memoize functions -- Use `useMemo` for expensive calculations -- Implement code splitting using dynamic imports - -**Error Handling:** -- Use Zod for schema validation -- Implement error boundaries for UI stability -- Handle errors at the start of functions -- Use early returns instead of deeply nested `if` statements - -**Testing:** -- Use Jest and React Testing Library for unit tests -- Follow Arrange-Act-Assert pattern -- Mock external dependencies -- Ensure full keyboard navigation support in accessibility tests - -**Security:** -- Sanitize user input to prevent XSS attacks -- Use DOMPurify for HTML sanitization -- Implement secure authentication methods -- Ensure API communication is encrypted over HTTPS - -**Documentation:** -- Use JSDoc for documenting functions and interfaces -- Document public APIs and components -- Include examples where applicable - ---- - -## Git Workflow - -### Commit Best Practices - -1. **Review Changes** - ```bash - git status - git diff - git log - ``` - -2. **Analyze Changes** - - Identify modified or added files - - Understand the nature of changes (new feature, bug fix, refactoring, etc.) - - Evaluate the impact on the project - - Ensure no sensitive information is exposed - -3. **Create Meaningful Commit Messages** - ```bash - git commit -m "fix: Resolve issue with authentication timeout" - ``` - -### Pull Request Best Practices - -1. **Review Branch Status** - ```bash - git status - git diff main...HEAD - git log - ``` - -2. **Analyze Changes** - - Review all commits made since branching off `main` - - Assess change scope and impact - - Ensure no sensitive data is committed - -3. **Create a Pull Request** - ```bash - gh pr create --title "feat: Improve Rust error handling" --body "Improved error handling with Result." - ``` - ---- - -## Important Notes - -- The project uses a Rust workspace structure - always build from `/src/rust/` -- WASM builds require `wasm-pack` and target `wasm32-unknown-unknown` -- Python bindings require `uv` package manager and use maturin for building -- FlatBuffers schemas must be regenerated after changes using `make gen-all` -- HTTP optimization is critical - always consider range request efficiency -- Geometry templates are supported for efficient repeated geometry encoding -- Use `thiserror` for custom error types in library code, avoid `anyhow` except when explicitly approved - ---- - -## Development Tools - -### Local MCP Tool: `serena` - -`serena` is a tool that can be used to get an overview of the code base. - ---- - -## Testing Strategy - -- **Unit tests**: Use `#[cfg(test)]` modules for Rust -- **Integration tests**: Place in `/tests/` directories -- **Use cargo nextest**: For faster test execution -- **Mock HTTP clients**: For remote data tests -- **Python tests**: Use `uv` and pytest -- **Web tests**: Use Jest and React Testing Library - ---- - -## Performance Considerations - -- Minimize HTTP requests through batching -- Use buffered readers with caching -- Prefer iterators over collecting into vectors -- Profile with `criterion` benchmarks -- Implement payload prefetching and batch resolution for attribute queries -- Consider spatial locality when ordering features - ---- - -## Additional Resources - -For detailed specification information, see: -- [specification.md](.cursor/rules/memory/specification.md) - Complete FlatCityBuf format specification -- [productContext.md](.cursor/rules/memory/productContext.md) - Project context and goals diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 120000 index 47dc3e3..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -AGENTS.md \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2a02fcc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,37 @@ +# Important Rules for LLM + +## General Guidelines + +- If a test fails more than twice in a row, analyze the current situation and collaborate with the user to determine a solution. Avoid trial-and-error testing without a hypothesis. +- The user has extensive knowledge gained from GitHub and can implement individual algorithms and libraries faster than you. Code should be written while explaining to the user, using test cases to verify correctness. +- When you work on the bigger tasks, you should plan first and document list of tasks to be done and update progress.md + +### Core Files (Mandatory) + +1. **`productContext.md`** [productContext.md](mdc:.llm/docs/productContext.md) + + - Explains the purpose of the project. + - Identifies the problem it solves. + - Describes expected functionality and user experience goals. + +2. **`specification.md`** [specification.md](mdc:.llm/docs/specification.md) + - Explains the detail of FlatCityBuf specification + - Describes its encoding strategy and decisions made + +### Additional Context Files + +Additional files and folders can be created inside `.llm/docs/*` if they aid in organization: + +- Documentation for complex features. +- Integration specifications. +- API documentation. +- Testing strategies. +- Deployment procedures. + +--- + +## Integration with GitHub + +you can expect that runtime has gh CLI, rather than using GitHub MCP, simply use gh CLI to create pull requests, etc. + +--- From 77468df55a5d6087940d0a32738b858dfaa34d55 Mon Sep 17 00:00:00 2001 From: HideBa Date: Fri, 19 Dec 2025 14:05:02 +0100 Subject: [PATCH 2/5] docs: Add flatcitybuf specification, introduce new LLM and agent documentation, and refactor cursor rules. --- README.md | 50 ++++++++++++++++++++++++------------------ src/rust/cli/README.md | 26 +++++++++++++++++++--- 2 files changed, 52 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 42b148d..00d0988 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ [![Rust](https://img.shields.io/badge/rust-%23000000.svg?style=flat&logo=rust&logoColor=white)](https://www.rust-lang.org/) [![WebAssembly](https://img.shields.io/badge/WebAssembly-654FF0?style=flat&logo=webassembly&logoColor=white)](https://webassembly.org/) -*Bringing the semantic richness of CityJSON with the performance of FlatBuffers* +_Bringing the semantic richness of CityJSON with the performance of FlatBuffers_ [πŸš€ Getting Started](#-getting-started) β€’ [πŸ“Š Benchmarks](#-performance--benchmarks) β€’ [πŸ“– Documentation](#-documentation) β€’ [🀝 Contributing](#-contributing) @@ -46,13 +46,13 @@ Traditional CityJSON formats face significant challenges in large-scale urban ap ### πŸš€ Key Features -| Feature | Benefit | -|---------|---------| -| **⚑ Zero-copy Access** | Access specific city objects without parsing entire files | -| **☁️ Cloud Optimized** | HTTP range requests for partial data retrieval | -| **πŸ—ΊοΈ Spatial Indexing** | Packed R-tree for lightning-fast spatial queries | -| **πŸ” Attribute Indexing** | Static B+Tree for instant attribute-based filtering | -| **🌐 Multi-platform** | Rust core with WASM bindings for web applications | +| Feature | Benefit | +| ------------------------- | --------------------------------------------------------- | +| **⚑ Zero-copy Access** | Access specific city objects without parsing entire files | +| **☁️ Cloud Optimized** | HTTP range requests for partial data retrieval | +| **πŸ—ΊοΈ Spatial Indexing** | Packed R-tree for lightning-fast spatial queries | +| **πŸ” Attribute Indexing** | Static B+Tree for instant attribute-based filtering | +| **🌐 Multi-platform** | Rust core with WASM bindings for web applications | --- @@ -62,12 +62,12 @@ FlatCityBuf delivers **10-20Γ— faster** data retrieval compared to CityJSONTextS ### Speed Comparison Results -| Dataset | CityJSON | FlatCityBuf | **Speed Improvement** | Memory Reduction | -|---------|---------------|------------------|---------------------|------------------| -| 3DBAG | 56 ms | 6 ms | **8.6Γ—** | 4.7Γ— less memory | -| 3DBV | 3.8 s | 122ms | **32.6Γ—** | 4.5Γ— less memory | -| Helsinki | 4.0 s | 132ms | **30.6Γ—** | 2.9Γ— less memory | -| NYC | 887 ms | 43 ms | **20.7Γ—** | 4.1Γ— less memory | +| Dataset | CityJSON | FlatCityBuf | **Speed Improvement** | Memory Reduction | +| -------- | -------- | ----------- | --------------------- | ---------------- | +| 3DBAG | 56 ms | 6 ms | **8.6Γ—** | 4.7Γ— less memory | +| 3DBV | 3.8 s | 122ms | **32.6Γ—** | 4.5Γ— less memory | +| Helsinki | 4.0 s | 132ms | **30.6Γ—** | 2.9Γ— less memory | +| NYC | 887 ms | 43 ms | **20.7Γ—** | 4.1Γ— less memory | > πŸ“ˆ **Performance**: 8.6-256Γ— faster queries with 2.1-6.4Γ— less memory usage @@ -104,7 +104,15 @@ flatcitybuf/ ### πŸ“¦ Installation -#### Package Manager Installation +#### Package Manager Installation (Recommended) + +**Rust CLI**: Install from crates.io + +```bash +cargo install fcb_cli --locked +``` + +This installs the `fcb` binary to your Cargo bin directory (usually `~/.cargo/bin/`). **Python**: Install from PyPI @@ -144,16 +152,16 @@ replace `cargo run -p fcb_cli` with `fcb` in the following commands if you want ```bash # Basic conversion -cargo run -p fcb_cli ser -i input.city.jsonl -o output.fcb +fcb fcb_cli ser -i input.city.jsonl -o output.fcb # With compression and indexing options -cargo run -p fcb_cli ser -i data.city.jsonl -o data.fcb +fcb fcb_cli ser -i data.city.jsonl -o data.fcb # With spatial index and attribute index -cargo run -p fcb_cli ser -i data.city.jsonl -o data.fcb --attr-index attribute_name,attribute_name2 --attr-branching-factor 256 +fcb fcb_cli ser -i data.city.jsonl -o data.fcb --attr-index attribute_name,attribute_name2 --attr-branching-factor 256 # Show information about the file -cargo run -p fcb_cli info -i data.fcb +fcb fcb_cli info -i data.fcb ``` ### πŸ§ͺ Run Benchmarks @@ -192,10 +200,10 @@ This project builds upon the excellent work of the geospatial and 3D GIS communi ### Technical Foundations - **[FlatGeobuf](https://github.com/flatgeobuf/flatgeobuf)** - FlatGeobuf team - *Licensed under BSD 2-Clause License. Provided the foundational spatial indexing algorithms and FlatBuffers integration patterns.* + _Licensed under BSD 2-Clause License. Provided the foundational spatial indexing algorithms and FlatBuffers integration patterns._ - **[CityBuf](https://github.com/3DBAG/CityBuf)** - 3DBAG organisation - *Original FlatBuffers schema for CityJSON features, authored by Ravi Peters (3DGI) and BalΓ‘zs Dukai (3DGI).* + _Original FlatBuffers schema for CityJSON features, authored by Ravi Peters (3DGI) and BalΓ‘zs Dukai (3DGI)._ ### Standards & Specifications diff --git a/src/rust/cli/README.md b/src/rust/cli/README.md index d81a43c..9a26b6d 100644 --- a/src/rust/cli/README.md +++ b/src/rust/cli/README.md @@ -4,14 +4,34 @@ A command-line interface for converting between CityJSON and FlatCityBuf (FCB) f ## Installation +### Option 1: Install from crates.io (Recommended) + ```bash -cargo install fcb_cli +cargo install fcb_cli --locked ``` -Or build from source: +This installs the `fcb` binary to your Cargo bin directory (usually `~/.cargo/bin/`). + +### Option 2: Build from Source ```bash -cargo build --release +# Clone the repository +git clone https://github.com/cityjson/flatcitybuf.git +cd flatcitybuf/src/rust + +# Build in release mode +cargo build --release -p fcb_cli + +``` + +### Option 3: Run with Cargo (Development) + +```bash +cd flatcitybuf/src/rust +cargo run -p fcb_cli -- [args] + +# Example: convert CityJSONSeq to FCB +cargo run -p fcb_cli -- ser -i input.city.jsonl -o output.fcb ``` ## Usage From 53e3316f5012911bbde08c1b56f0ccfdd89720fa Mon Sep 17 00:00:00 2001 From: HideBa Date: Fri, 19 Dec 2025 15:34:23 +0100 Subject: [PATCH 3/5] feat: Enable multi-file input with glob support and transform-aware merging for CityJSON serialization. --- src/rust/cli/Cargo.toml | 1 + src/rust/cli/src/main.rs | 117 +++++++++++++++++++-------- src/rust/cli/src/merger.rs | 160 +++++++++++++++++++++++++++++++++++++ src/rust/cli/src/reader.rs | 146 +++++++++++++++++++++++++++++++++ 4 files changed, 392 insertions(+), 32 deletions(-) create mode 100644 src/rust/cli/src/merger.rs create mode 100644 src/rust/cli/src/reader.rs diff --git a/src/rust/cli/Cargo.toml b/src/rust/cli/Cargo.toml index 4495c7a..cf0bd30 100644 --- a/src/rust/cli/Cargo.toml +++ b/src/rust/cli/Cargo.toml @@ -23,3 +23,4 @@ serde_cbor = { workspace = true } thiserror = { workspace = true } indicatif = { workspace = true } console = { workspace = true } +glob = "0.3" diff --git a/src/rust/cli/src/main.rs b/src/rust/cli/src/main.rs index 8264b58..00956a5 100644 --- a/src/rust/cli/src/main.rs +++ b/src/rust/cli/src/main.rs @@ -1,3 +1,6 @@ +mod merger; +mod reader; + use cjseq::{CityJSON, CityJSONFeature, Transform as CjTransform}; use clap::{ArgAction, Parser, Subcommand}; use console::{style, Term}; @@ -6,14 +9,44 @@ use fcb_core::{ attribute::{AttributeSchema, AttributeSchemaMethods}, deserializer, header_writer::HeaderWriterOptions, - read_cityjson_from_reader, CJType, CJTypeKind, CityJSONSeq, FcbReader, FcbWriter, + FcbReader, FcbWriter, }; +use glob::glob; use indicatif::{ProgressBar, ProgressStyle}; use std::{ fs::File, io::{self, BufReader, BufWriter, Read, Write}, path::PathBuf, }; +use thiserror::Error; + +/// CLI-specific error type +#[derive(Error, Debug)] +pub enum CliError { + #[error("IO error: {0}")] + Io(#[from] std::io::Error), + + #[error("JSON error: {0}")] + Json(#[from] serde_json::Error), + + #[error("Glob pattern error: {0}")] + GlobPattern(#[from] glob::PatternError), + + #[error("Glob error: {0}")] + Glob(#[from] glob::GlobError), + + #[error("Unsupported file format for '{0}': {1}")] + UnsupportedFormat(String, String), + + #[error("Empty file: {0}")] + EmptyFile(String), + + #[error("No input files specified or matched")] + NoInputFiles, + + #[error("FCB core error: {0}")] + FcbCore(#[from] Error), +} #[derive(Parser)] #[command( name = "fcb", @@ -30,9 +63,9 @@ struct Cli { enum Commands { /// Convert CityJSON to FCB Ser { - /// Input file (use '-' for stdin) - #[arg(short = 'i', long)] - input: String, + /// Input files (glob patterns supported, e.g., "cities/*/*.jsonl") + #[arg(short = 'i', long, required = true, num_args = 1..)] + input: Vec, /// Output file (use '-' for stdout) #[arg(short = 'o', long)] @@ -125,7 +158,7 @@ struct SerializeOptions { ge: bool, } -fn serialize(input: &str, output: &str, options: SerializeOptions) -> Result<(), Error> { +fn serialize(inputs: &[String], output: &str, options: SerializeOptions) -> Result<(), CliError> { let term = Term::stderr(); let is_stdout = output == "-"; @@ -145,16 +178,29 @@ fn serialize(input: &str, output: &str, options: SerializeOptions) -> Result<(), .ok(); } - let reader = get_reader(input)?; - let writer = get_writer(output)?; + // Expand glob patterns and collect all input files + let mut input_paths: Vec = Vec::new(); + for pattern in inputs { + let paths: Vec = glob(pattern)?.filter_map(|entry| entry.ok()).collect(); + if paths.is_empty() { + // If no glob match, treat as literal path + input_paths.push(PathBuf::from(pattern)); + } else { + input_paths.extend(paths); + } + } + + if input_paths.is_empty() { + return Err(CliError::NoInputFiles); + } - let reader = BufReader::new(reader); + let writer = get_writer(output)?; let writer = BufWriter::new(writer); // Parse the bbox if provided let bbox_parsed = if let Some(bbox_str) = &options.bbox { Some(parse_bbox(bbox_str).map_err(|e| { - Error::IoError(std::io::Error::new( + CliError::Io(std::io::Error::new( std::io::ErrorKind::InvalidInput, format!("failed to parse bbox: {e}"), )) @@ -169,15 +215,27 @@ fn serialize(input: &str, output: &str, options: SerializeOptions) -> Result<(), term.write_line(&format!("{} Configuration", style("β–Ά").bold().green())) .ok(); term.write_line(&format!( - " {} {}", + " {} {} file(s)", style("Input:").dim(), - if input == "-" { - style("stdin").yellow() - } else { - style(input).yellow() - } + style(input_paths.len()).yellow() )) .ok(); + for (i, path) in input_paths.iter().enumerate().take(5) { + term.write_line(&format!( + " {}. {}", + style(i + 1).dim(), + style(path.display()).yellow() + )) + .ok(); + } + if input_paths.len() > 5 { + term.write_line(&format!( + " {} {} more files...", + style("...").dim(), + style(input_paths.len() - 5).dim() + )) + .ok(); + } term.write_line(&format!( " {} {}", style("Output:").dim(), @@ -245,7 +303,7 @@ fn serialize(input: &str, output: &str, options: SerializeOptions) -> Result<(), term.write_line("").ok(); } - // Create a CityJSONSeq reader + // Read and merge input files if !is_stdout { term.write_line(&format!( "{} Reading CityJSON...", @@ -254,16 +312,9 @@ fn serialize(input: &str, output: &str, options: SerializeOptions) -> Result<(), .ok(); } - let cj_seq = read_cityjson_from_reader(reader, CJTypeKind::Seq)?; - - let CityJSONSeq { cj, features } = match cj_seq { - CJType::Seq(cj_seq) => cj_seq, - _ => { - return Err(Error::IoError(std::io::Error::other( - "failed to read CityJSON Feature", - ))) - } - }; + let merge_result = merger::merge_files(input_paths)?; + let cj = merge_result.metadata; + let features = merge_result.features; if !is_stdout { term.write_line(&format!( @@ -954,7 +1005,7 @@ fn show_info(input: PathBuf) -> Result<(), Error> { Ok(()) } -fn main() -> Result<(), Error> { +fn main() -> Result<(), Box> { let cli = Cli::parse(); match cli.command { @@ -978,12 +1029,14 @@ fn main() -> Result<(), Error> { bbox, ge, }, - ), - Commands::Deser { input, output } => deserialize(&input, &output), - Commands::Cbor { input, output } => encode_cbor(&input, &output), - Commands::Bson { input, output } => encode_bson(&input, &output), - Commands::Info { input } => show_info(input), + )?, + Commands::Deser { input, output } => deserialize(&input, &output)?, + Commands::Cbor { input, output } => encode_cbor(&input, &output)?, + Commands::Bson { input, output } => encode_bson(&input, &output)?, + Commands::Info { input } => show_info(input)?, } + + Ok(()) } #[cfg(test)] diff --git a/src/rust/cli/src/merger.rs b/src/rust/cli/src/merger.rs new file mode 100644 index 0000000..136e875 --- /dev/null +++ b/src/rust/cli/src/merger.rs @@ -0,0 +1,160 @@ +//! Merger module for combining multiple CityJSON/CityJSONSeq files +//! +//! This module handles merging multiple input files with transform alignment. +//! When files have different transforms (scale/translate), vertices from +//! subsequent files are converted to match the first file's transform. + +use cjseq::{CityJSON, CityJSONFeature, Transform}; +use std::path::PathBuf; + +use crate::reader::read_input_file; +use crate::CliError; + +/// Result of merging multiple input files +pub struct MergeResult { + /// Merged CityJSON metadata (from first file) + pub metadata: CityJSON, + /// All features from all files (with aligned transforms) + pub features: Vec, +} + +/// Merge multiple CityJSON/CityJSONSeq files into a single result +/// +/// The first file's transform becomes the reference. Features from subsequent +/// files have their vertices converted to use the reference transform. +pub fn merge_files(paths: Vec) -> Result { + if paths.is_empty() { + return Err(CliError::NoInputFiles); + } + + let mut paths_iter = paths.into_iter(); + + // Read the first file - its transform becomes the reference + let first_path = paths_iter.next().ok_or(CliError::NoInputFiles)?; + let first_data = read_input_file(&first_path)?; + let reference_transform = first_data.metadata.transform.clone(); + + let mut result = MergeResult { + metadata: first_data.metadata, + features: first_data.features, + }; + + // Process remaining files + for path in paths_iter { + let data = read_input_file(&path)?; + + // Check if transforms are the same + if transforms_equal(&data.metadata.transform, &reference_transform) { + // Same transform - just append features + result.features.extend(data.features); + } else { + // Different transform - need to convert vertices + for feature in data.features { + let converted = convert_feature_transform( + feature, + &data.metadata.transform, + &reference_transform, + ); + result.features.push(converted); + } + } + } + + Ok(result) +} + +/// Check if two transforms are equal +fn transforms_equal(a: &Transform, b: &Transform) -> bool { + a.scale == b.scale && a.translate == b.translate +} + +/// Convert a feature's vertices from one transform to another +/// +/// This converts: +/// 1. Integer vertices β†’ real coordinates (using source transform) +/// 2. Real coordinates β†’ integer vertices (using target transform) +fn convert_feature_transform( + mut feature: CityJSONFeature, + source: &Transform, + target: &Transform, +) -> CityJSONFeature { + for vertex in &mut feature.vertices { + if vertex.len() >= 3 { + // Convert to real coordinates using source transform + let real_x = (vertex[0] as f64 * source.scale[0]) + source.translate[0]; + let real_y = (vertex[1] as f64 * source.scale[1]) + source.translate[1]; + let real_z = (vertex[2] as f64 * source.scale[2]) + source.translate[2]; + + // Convert back to integers using target transform + vertex[0] = ((real_x - target.translate[0]) / target.scale[0]).round() as i64; + vertex[1] = ((real_y - target.translate[1]) / target.scale[1]).round() as i64; + vertex[2] = ((real_z - target.translate[2]) / target.scale[2]).round() as i64; + } + } + + feature +} + +#[cfg(test)] +mod tests { + use super::*; + + fn make_transform(scale: [f64; 3], translate: [f64; 3]) -> Transform { + Transform { + scale: scale.to_vec(), + translate: translate.to_vec(), + } + } + + #[test] + fn test_transforms_equal() { + let t1 = make_transform([0.001, 0.001, 0.001], [0.0, 0.0, 0.0]); + let t2 = make_transform([0.001, 0.001, 0.001], [0.0, 0.0, 0.0]); + let t3 = make_transform([0.001, 0.001, 0.001], [1.0, 0.0, 0.0]); + + assert!(transforms_equal(&t1, &t2)); + assert!(!transforms_equal(&t1, &t3)); + } + + #[test] + fn test_convert_feature_transform_identity() { + // Same transform should produce same vertices + let source = make_transform([0.001, 0.001, 0.001], [0.0, 0.0, 0.0]); + let target = source.clone(); + + let mut feature = CityJSONFeature::new(); + feature.vertices = vec![vec![1000, 2000, 3000]]; + + let converted = convert_feature_transform(feature.clone(), &source, &target); + assert_eq!(converted.vertices[0], vec![1000, 2000, 3000]); + } + + #[test] + fn test_convert_feature_transform_different() { + // Source: vertex 1000 means 1.0 real coordinate + let source = make_transform([0.001, 0.001, 0.001], [0.0, 0.0, 0.0]); + // Target: 1.0 real coordinate should become 500 + let target = make_transform([0.002, 0.002, 0.002], [0.0, 0.0, 0.0]); + + let mut feature = CityJSONFeature::new(); + feature.vertices = vec![vec![1000, 2000, 3000]]; + + let converted = convert_feature_transform(feature, &source, &target); + // 1000 * 0.001 = 1.0 -> 1.0 / 0.002 = 500 + assert_eq!(converted.vertices[0], vec![500, 1000, 1500]); + } + + #[test] + fn test_convert_feature_transform_with_translate() { + let source = make_transform([0.001, 0.001, 0.001], [100.0, 200.0, 0.0]); + let target = make_transform([0.001, 0.001, 0.001], [0.0, 0.0, 0.0]); + + let mut feature = CityJSONFeature::new(); + feature.vertices = vec![vec![0, 0, 0]]; // Real: (100, 200, 0) + + let converted = convert_feature_transform(feature, &source, &target); + // Real (100, 200, 0) with target translate (0, 0, 0) and scale 0.001 + // -> (100 / 0.001, 200 / 0.001, 0) = (100000, 200000, 0) + assert_eq!(converted.vertices[0], vec![100000, 200000, 0]); + } +} diff --git a/src/rust/cli/src/reader.rs b/src/rust/cli/src/reader.rs new file mode 100644 index 0000000..4762e3a --- /dev/null +++ b/src/rust/cli/src/reader.rs @@ -0,0 +1,146 @@ +//! Reader module for CityJSON and CityJSONSeq file reading +//! +//! This module provides utilities to read CityJSON (`.json`) and +//! CityJSONTextSequence (`.jsonl`) files and convert them to a unified +//! in-memory representation of CityJSON metadata and features. + +use cjseq::{CityJSON, CityJSONFeature}; +use std::fs::File; +use std::io::{BufRead, BufReader}; +use std::path::Path; + +use crate::CliError; + +/// Detected input file format +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InputFormat { + /// CityJSON file (`.json`) + CityJSON, + /// CityJSONTextSequence file (`.jsonl`) + CityJSONSeq, +} + +impl InputFormat { + /// Detect the format from file extension + pub fn from_path(path: &Path) -> Result { + match path.extension().and_then(|e| e.to_str()) { + Some("json") => Ok(InputFormat::CityJSON), + Some("jsonl") => Ok(InputFormat::CityJSONSeq), + _ => Err(CliError::UnsupportedFormat( + path.display().to_string(), + "expected .json or .jsonl extension".to_string(), + )), + } + } +} + +/// Result of reading an input file +pub struct InputData { + /// CityJSON metadata (first line of CityJSONSeq) + pub metadata: CityJSON, + /// CityJSON features + pub features: Vec, +} + +/// Read a CityJSON or CityJSONSeq file and return unified data +/// +/// - `.json` files are parsed as CityJSON and converted to features +/// - `.jsonl` files are parsed as CityJSONTextSequence directly +pub fn read_input_file(path: &Path) -> Result { + let format = InputFormat::from_path(path)?; + + match format { + InputFormat::CityJSON => read_cityjson_file(path), + InputFormat::CityJSONSeq => read_cityjsonseq_file(path), + } +} + +/// Read a CityJSON file and convert to features +fn read_cityjson_file(path: &Path) -> Result { + let file = File::open(path)?; + let reader = BufReader::new(file); + + // Parse as full CityJSON + let cj: CityJSON = serde_json::from_reader(reader)?; + + // Extract features using cjseq library pattern + let mut features = Vec::new(); + let mut i = 0; + while let Some(feature) = cj.get_cjfeature(i) { + features.push(feature); + i += 1; + } + + // Get metadata (CityJSON without city_objects) + let metadata = cj.get_metadata(); + + Ok(InputData { metadata, features }) +} + +/// Read a CityJSONSeq file (first line is metadata, rest are features) +fn read_cityjsonseq_file(path: &Path) -> Result { + let file = File::open(path)?; + let reader = BufReader::new(file); + let mut lines = reader.lines(); + + // First line is the CityJSON metadata + let first_line = lines + .next() + .ok_or_else(|| CliError::EmptyFile(path.display().to_string()))??; + let metadata: CityJSON = serde_json::from_str(&first_line)?; + + // Remaining lines are CityJSONFeatures + let mut features = Vec::new(); + for line in lines { + let line = line?; + if !line.trim().is_empty() { + let feature: CityJSONFeature = serde_json::from_str(&line)?; + features.push(feature); + } + } + + Ok(InputData { metadata, features }) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + #[test] + fn test_detect_format_jsonl() { + let path = PathBuf::from("test.city.jsonl"); + assert_eq!( + InputFormat::from_path(&path).unwrap(), + InputFormat::CityJSONSeq + ); + } + + #[test] + fn test_detect_format_json() { + let path = PathBuf::from("test.city.json"); + assert_eq!( + InputFormat::from_path(&path).unwrap(), + InputFormat::CityJSON + ); + } + + #[test] + fn test_detect_format_invalid() { + let path = PathBuf::from("test.txt"); + assert!(InputFormat::from_path(&path).is_err()); + } + + #[test] + fn test_read_cityjsonseq_file() { + let test_file = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .join("fcb_core/tests/data/small.city.jsonl"); + + if test_file.exists() { + let result = read_input_file(&test_file).unwrap(); + assert!(!result.features.is_empty()); + } + } +} From e616a0a2e491926a9d933417ce8d820a79ba6c54 Mon Sep 17 00:00:00 2001 From: HideBa Date: Fri, 19 Dec 2025 16:17:21 +0100 Subject: [PATCH 4/5] feat: refactor CLI into a library, add e2e tests, and clean up benchmark artifact --- .gitignore | 1 + src/rust/cli/Cargo.toml | 7 + src/rust/cli/src/lib.rs | 38 +++ src/rust/cli/src/main.rs | 35 +-- src/rust/cli/src/reader.rs | 4 +- src/rust/cli/tests/e2e.rs | 227 ++++++++++++++++++ src/rust/fcb_api/src/openapi/src/apis/mod.rs | 4 +- .../src/openapi/src/models/extent_spatial.rs | 11 +- .../openapi/src/models/feature_collection.rs | 11 +- src/rust/fcb_core/benches/read.rs | 2 +- src/rust/fcb_core/benches/read_http.rs | 2 +- src/rust/fcb_core/benchmark_results.csv | 61 ----- .../benchmark_results_20250531_150434.txt | 129 ---------- .../benchmark_summary_20250531_150434.csv | 16 -- .../comparison_bson_20250531_150434.csv | 16 -- .../comparison_cbor_20250531_150434.csv | 16 -- ...n_cityjsontextsequence_20250531_150434.csv | 16 -- src/rust/fcb_core/http_benchmark_results.csv | 13 - src/rust/fcb_core/src/bin/read_cj.rs | 2 +- src/rust/fcb_core/src/bin/stats.rs | 8 +- src/rust/fcb_core/src/fb/feature_generated.rs | 8 +- src/rust/fcb_core/src/fb/header_generated.rs | 10 +- src/rust/fcb_core/src/reader/attr_query.rs | 9 +- src/rust/fcb_core/src/reader/city_buffer.rs | 4 +- src/rust/fcb_core/src/reader/mod.rs | 16 +- src/rust/fcb_py/examples/basic_usage.py | 12 +- src/rust/fcb_py/uv.lock | 4 +- 27 files changed, 328 insertions(+), 354 deletions(-) create mode 100644 src/rust/cli/src/lib.rs create mode 100644 src/rust/cli/tests/e2e.rs delete mode 100644 src/rust/fcb_core/benchmark_results.csv delete mode 100644 src/rust/fcb_core/benchmark_results_20250531_150434.txt delete mode 100644 src/rust/fcb_core/benchmark_summary_20250531_150434.csv delete mode 100644 src/rust/fcb_core/comparison_bson_20250531_150434.csv delete mode 100644 src/rust/fcb_core/comparison_cbor_20250531_150434.csv delete mode 100644 src/rust/fcb_core/comparison_cityjsontextsequence_20250531_150434.csv delete mode 100644 src/rust/fcb_core/http_benchmark_results.csv diff --git a/.gitignore b/.gitignore index cf08021..254abf6 100644 --- a/.gitignore +++ b/.gitignore @@ -55,6 +55,7 @@ __pycache__/ *.pytest_cache/ temp/ +data/ benchmark_data/ src/rust/fcb_core/tests/data/*.fcb diff --git a/src/rust/cli/Cargo.toml b/src/rust/cli/Cargo.toml index cf0bd30..e9e8d31 100644 --- a/src/rust/cli/Cargo.toml +++ b/src/rust/cli/Cargo.toml @@ -11,6 +11,10 @@ description = "FlatCityBuf is a library for reading and writing CityJSON with Fl name = "fcb" path = "src/main.rs" +[lib] +name = "fcb_cli" +path = "src/lib.rs" + [dependencies] fcb_core = { workspace = true, features = ["http"] } cjseq = { workspace = true } @@ -24,3 +28,6 @@ thiserror = { workspace = true } indicatif = { workspace = true } console = { workspace = true } glob = "0.3" + +[dev-dependencies] +tempfile = { workspace = true } diff --git a/src/rust/cli/src/lib.rs b/src/rust/cli/src/lib.rs new file mode 100644 index 0000000..951ea38 --- /dev/null +++ b/src/rust/cli/src/lib.rs @@ -0,0 +1,38 @@ +//! FCB CLI Library +//! +//! This library exposes the merger and reader modules for integration testing. +//! The main CLI binary is in main.rs. + +pub mod merger; +pub mod reader; + +use fcb_core::error::Error; +use thiserror::Error; + +/// CLI-specific error type +#[derive(Error, Debug)] +pub enum CliError { + #[error("IO error: {0}")] + Io(#[from] std::io::Error), + + #[error("JSON error: {0}")] + Json(#[from] serde_json::Error), + + #[error("Glob pattern error: {0}")] + GlobPattern(#[from] glob::PatternError), + + #[error("Glob error: {0}")] + Glob(#[from] glob::GlobError), + + #[error("Unsupported file format for '{0}': {1}")] + UnsupportedFormat(String, String), + + #[error("Empty file: {0}")] + EmptyFile(String), + + #[error("No input files specified or matched")] + NoInputFiles, + + #[error("FCB core error: {0}")] + FcbCore(#[from] Error), +} diff --git a/src/rust/cli/src/main.rs b/src/rust/cli/src/main.rs index 00956a5..2469353 100644 --- a/src/rust/cli/src/main.rs +++ b/src/rust/cli/src/main.rs @@ -1,9 +1,7 @@ -mod merger; -mod reader; - use cjseq::{CityJSON, CityJSONFeature, Transform as CjTransform}; use clap::{ArgAction, Parser, Subcommand}; use console::{style, Term}; +use fcb_cli::CliError; use fcb_core::error::Error; use fcb_core::{ attribute::{AttributeSchema, AttributeSchemaMethods}, @@ -18,35 +16,6 @@ use std::{ io::{self, BufReader, BufWriter, Read, Write}, path::PathBuf, }; -use thiserror::Error; - -/// CLI-specific error type -#[derive(Error, Debug)] -pub enum CliError { - #[error("IO error: {0}")] - Io(#[from] std::io::Error), - - #[error("JSON error: {0}")] - Json(#[from] serde_json::Error), - - #[error("Glob pattern error: {0}")] - GlobPattern(#[from] glob::PatternError), - - #[error("Glob error: {0}")] - Glob(#[from] glob::GlobError), - - #[error("Unsupported file format for '{0}': {1}")] - UnsupportedFormat(String, String), - - #[error("Empty file: {0}")] - EmptyFile(String), - - #[error("No input files specified or matched")] - NoInputFiles, - - #[error("FCB core error: {0}")] - FcbCore(#[from] Error), -} #[derive(Parser)] #[command( name = "fcb", @@ -312,7 +281,7 @@ fn serialize(inputs: &[String], output: &str, options: SerializeOptions) -> Resu .ok(); } - let merge_result = merger::merge_files(input_paths)?; + let merge_result = fcb_cli::merger::merge_files(input_paths)?; let cj = merge_result.metadata; let features = merge_result.features; diff --git a/src/rust/cli/src/reader.rs b/src/rust/cli/src/reader.rs index 4762e3a..5c250e1 100644 --- a/src/rust/cli/src/reader.rs +++ b/src/rust/cli/src/reader.rs @@ -61,8 +61,8 @@ fn read_cityjson_file(path: &Path) -> Result { let reader = BufReader::new(file); // Parse as full CityJSON - let cj: CityJSON = serde_json::from_reader(reader)?; - + let mut cj: CityJSON = serde_json::from_reader(reader)?; + cj.sort_cjfeatures(cjseq::SortingStrategy::Random); // Extract features using cjseq library pattern let mut features = Vec::new(); let mut i = 0; diff --git a/src/rust/cli/tests/e2e.rs b/src/rust/cli/tests/e2e.rs new file mode 100644 index 0000000..9471c24 --- /dev/null +++ b/src/rust/cli/tests/e2e.rs @@ -0,0 +1,227 @@ +//! End-to-end tests for the FCB CLI multi-file support +//! +//! These tests verify the merger and serialization functionality +//! by copying test files to temp directories and running full pipelines. + +use std::fs::{self, File}; +use std::io::BufReader; +use std::path::PathBuf; +use tempfile::TempDir; + +use fcb_cli::merger::merge_files; +use fcb_cli::reader::{read_input_file, InputFormat}; +use fcb_core::{ + attribute::{AttributeSchema, AttributeSchemaMethods}, + header_writer::HeaderWriterOptions, + FcbReader, FcbWriter, +}; +use std::io::BufWriter; + +/// Get the path to the fcb_core test data directory +fn get_test_data_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .join("fcb_core/tests/data") +} + +/// Copy a test file to a temp directory +fn copy_test_file(temp_dir: &TempDir, filename: &str) -> PathBuf { + let src = get_test_data_dir().join(filename); + let dest = temp_dir.path().join(filename); + fs::copy(&src, &dest).expect("Failed to copy test file"); + dest +} + +/// Helper to read FCB file and get feature count +fn get_fcb_feature_count(path: &PathBuf) -> u64 { + let file = File::open(path).expect("Failed to open FCB file"); + let reader = BufReader::new(file); + let fcb_reader = FcbReader::open(reader) + .expect("Failed to read FCB header") + .select_all() + .expect("Failed to select all"); + fcb_reader.header().features_count() +} + +mod merger_tests { + use super::*; + + #[test] + fn test_merge_single_file() { + let temp_dir = TempDir::new().expect("Failed to create temp dir"); + let file_path = copy_test_file(&temp_dir, "small.city.jsonl"); + + let result = merge_files(vec![file_path]).expect("Merge failed"); + + // small.city.jsonl has 3 features + assert_eq!(result.features.len(), 3); + assert!(result.metadata.transform.scale.len() >= 3); + } + + #[test] + fn test_merge_multiple_files() { + let temp_dir = TempDir::new().expect("Failed to create temp dir"); + let file1 = copy_test_file(&temp_dir, "small.city.jsonl"); + let file2 = copy_test_file(&temp_dir, "noise_extension.city.jsonl"); + + let result = merge_files(vec![file1, file2]).expect("Merge failed"); + + // small has 3 features, noise_extension has 3 features = 6 total + assert_eq!(result.features.len(), 6); + } + + #[test] + fn test_merge_empty_path_list() { + let result = merge_files(vec![]); + assert!(result.is_err()); + } +} + +mod reader_tests { + use super::*; + + #[test] + fn test_read_cityjsonseq_file() { + let temp_dir = TempDir::new().expect("Failed to create temp dir"); + let file_path = copy_test_file(&temp_dir, "small.city.jsonl"); + + let data = read_input_file(&file_path).expect("Failed to read file"); + + assert_eq!(data.features.len(), 3); + assert!(!data.metadata.transform.scale.is_empty()); + } + + #[test] + fn test_format_detection() { + let jsonl_path = PathBuf::from("test.city.jsonl"); + let json_path = PathBuf::from("test.city.json"); + + assert_eq!( + InputFormat::from_path(&jsonl_path).unwrap(), + InputFormat::CityJSONSeq + ); + assert_eq!( + InputFormat::from_path(&json_path).unwrap(), + InputFormat::CityJSON + ); + } +} + +mod serialization_tests { + use super::*; + + #[test] + fn test_full_serialization_pipeline() { + let temp_dir = TempDir::new().expect("Failed to create temp dir"); + + // Copy test files + let file1 = copy_test_file(&temp_dir, "small.city.jsonl"); + let output_path = temp_dir.path().join("output.fcb"); + + // Merge files + let merge_result = merge_files(vec![file1]).expect("Merge failed"); + + // Build schema + let attr_schema = { + let mut schema = AttributeSchema::new(); + for feature in merge_result.features.iter().take(100) { + for (_, co) in feature.city_objects.iter() { + if let Some(attributes) = &co.attributes { + schema.add_attributes(attributes); + } + } + } + if schema.is_empty() { + None + } else { + Some(schema) + } + }; + + // Create FCB writer + let header_options = HeaderWriterOptions { + write_index: true, + feature_count: merge_result.features.len() as u64, + index_node_size: 16, + attribute_indices: None, + geographical_extent: None, + }; + + let mut fcb = FcbWriter::new( + merge_result.metadata, + Some(header_options), + attr_schema, + None, // semantic schema + ) + .expect("Failed to create FCB writer"); + + // Add features + for feature in merge_result.features.iter() { + fcb.add_feature(feature).expect("Failed to add feature"); + } + + // Write to file + let output_file = File::create(&output_path).expect("Failed to create output file"); + let writer = BufWriter::new(output_file); + fcb.write(writer).expect("Failed to write FCB"); + + // Verify output + let feature_count = get_fcb_feature_count(&output_path); + assert_eq!(feature_count, 3); + } + + #[test] + fn test_multi_file_serialization() { + let temp_dir = TempDir::new().expect("Failed to create temp dir"); + + // Copy test files to subdirectories to test merging + let subdir1 = temp_dir.path().join("dir1"); + let subdir2 = temp_dir.path().join("dir2"); + fs::create_dir(&subdir1).expect("Failed to create subdir1"); + fs::create_dir(&subdir2).expect("Failed to create subdir2"); + + let src = get_test_data_dir().join("small.city.jsonl"); + let file1 = subdir1.join("small.city.jsonl"); + let file2 = subdir2.join("small.city.jsonl"); + fs::copy(&src, &file1).expect("Failed to copy to subdir1"); + fs::copy(&src, &file2).expect("Failed to copy to subdir2"); + + let output_path = temp_dir.path().join("merged.fcb"); + + // Merge both files + let merge_result = merge_files(vec![file1, file2]).expect("Merge failed"); + + // Verify we have double the features + assert_eq!(merge_result.features.len(), 6); // 3 + 3 = 6 + + // Create and write FCB + let header_options = HeaderWriterOptions { + write_index: true, + feature_count: merge_result.features.len() as u64, + index_node_size: 16, + attribute_indices: None, + geographical_extent: None, + }; + + let mut fcb = FcbWriter::new( + merge_result.metadata, + Some(header_options), + None, // attr schema + None, // semantic schema + ) + .expect("Failed to create FCB writer"); + + for feature in merge_result.features.iter() { + fcb.add_feature(feature).expect("Failed to add feature"); + } + + let output_file = File::create(&output_path).expect("Failed to create output file"); + let writer = BufWriter::new(output_file); + fcb.write(writer).expect("Failed to write FCB"); + + // Verify output has 6 features + let feature_count = get_fcb_feature_count(&output_path); + assert_eq!(feature_count, 6); + } +} diff --git a/src/rust/fcb_api/src/openapi/src/apis/mod.rs b/src/rust/fcb_api/src/openapi/src/apis/mod.rs index 55bb1e6..b984509 100644 --- a/src/rust/fcb_api/src/openapi/src/apis/mod.rs +++ b/src/rust/fcb_api/src/openapi/src/apis/mod.rs @@ -105,9 +105,9 @@ impl From<&str> for ContentType { if content_type.starts_with("application") && content_type.contains("json") { Self::Json } else if content_type.starts_with("text/plain") { - return Self::Text; + Self::Text } else { - return Self::Unsupported(content_type.to_string()); + Self::Unsupported(content_type.to_string()) } } } diff --git a/src/rust/fcb_api/src/openapi/src/models/extent_spatial.rs b/src/rust/fcb_api/src/openapi/src/models/extent_spatial.rs index a078cad..330544a 100644 --- a/src/rust/fcb_api/src/openapi/src/models/extent_spatial.rs +++ b/src/rust/fcb_api/src/openapi/src/models/extent_spatial.rs @@ -32,14 +32,11 @@ impl ExtentSpatial { } } /// Coordinate reference system of the returned coordinates. Currently only `https://www.opengis.net/def/crs/EPSG/0/7415` is supported. -#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)] +#[derive( + Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize, Default, +)] pub enum Crs { #[serde(rename = "https://www.opengis.net/def/crs/EPSG/0/7415")] + #[default] HttpsColonSlashSlashWwwPeriodOpengisPeriodNetSlashDefSlashCrsSlashEpsgSlash0Slash7415, } - -impl Default for Crs { - fn default() -> Crs { - Self::HttpsColonSlashSlashWwwPeriodOpengisPeriodNetSlashDefSlashCrsSlashEpsgSlash0Slash7415 - } -} diff --git a/src/rust/fcb_api/src/openapi/src/models/feature_collection.rs b/src/rust/fcb_api/src/openapi/src/models/feature_collection.rs index 01512ea..38aee8e 100644 --- a/src/rust/fcb_api/src/openapi/src/models/feature_collection.rs +++ b/src/rust/fcb_api/src/openapi/src/models/feature_collection.rs @@ -40,14 +40,11 @@ impl FeatureCollection { } } /// -#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)] +#[derive( + Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize, Default, +)] pub enum Type { #[serde(rename = "FeatureCollection")] + #[default] FeatureCollection, } - -impl Default for Type { - fn default() -> Type { - Self::FeatureCollection - } -} diff --git a/src/rust/fcb_core/benches/read.rs b/src/rust/fcb_core/benches/read.rs index 7dd0595..6530770 100644 --- a/src/rust/fcb_core/benches/read.rs +++ b/src/rust/fcb_core/benches/read.rs @@ -1108,7 +1108,7 @@ fn calculate_mean(durations: &[Duration]) -> Duration { fn calculate_median(durations: &mut [Duration]) -> Duration { durations.sort(); let len = durations.len(); - if len % 2 == 0 { + if len.is_multiple_of(2) { let mid1 = durations[len / 2 - 1].as_nanos(); let mid2 = durations[len / 2].as_nanos(); Duration::from_nanos(((mid1 + mid2) / 2) as u64) diff --git a/src/rust/fcb_core/benches/read_http.rs b/src/rust/fcb_core/benches/read_http.rs index 86949d5..92f138d 100644 --- a/src/rust/fcb_core/benches/read_http.rs +++ b/src/rust/fcb_core/benches/read_http.rs @@ -251,7 +251,7 @@ fn calculate_statistics(durations: &[Duration]) -> (f64, f64, f64, f64, f64) { .sum::() / durations.len() as f64; - let median_ms = if sorted_durations.len() % 2 == 0 { + let median_ms = if sorted_durations.len().is_multiple_of(2) { let mid1 = sorted_durations[sorted_durations.len() / 2 - 1].as_secs_f64() * 1000.0; let mid2 = sorted_durations[sorted_durations.len() / 2].as_secs_f64() * 1000.0; (mid1 + mid2) / 2.0 diff --git a/src/rust/fcb_core/benchmark_results.csv b/src/rust/fcb_core/benchmark_results.csv deleted file mode 100644 index 554203a..0000000 --- a/src/rust/fcb_core/benchmark_results.csv +++ /dev/null @@ -1,61 +0,0 @@ -Dataset,Format,Iterations,MeanTimeMs,MedianTimeMs,StdDevTimeMs,MemoryBytes,CpuPercent -3DBAG,FlatCityBuf,100,6,6,0,5685248,99.65917088194234 -3DBAG,CityJSONTextSequence,100,54,53,5,26918912,98.69658745517964 -3DBAG,CBOR,100,73,72,6,276119552,98.4202655470156 -3DBAG,BSON,100,113,111,10,338886656,98.51007641856422 -3DBV,FlatCityBuf,100,120,119,3,61882368,99.42229994529738 -3DBV,CityJSONTextSequence,100,3787,3776,42,369442816,99.60727400653408 -3DBV,CBOR,100,5050,5009,226,7212744704,93.75235300819696 -3DBV,BSON,100,8824,8786,288,7987085312,95.91335073897486 -Helsinki,FlatCityBuf,100,127,127,1,4915200,99.85329996279596 -Helsinki,CityJSONTextSequence,100,3485,3473,41,16891904,99.52358228601229 -Helsinki,CBOR,100,6305,6329,232,7478312960,97.55962883602244 -Helsinki,BSON,100,9693,9607,389,7967064064,95.9609870377163 -Ingolstadt,FlatCityBuf,100,0,0,0,9420800,97.73512970575536 -Ingolstadt,CityJSONTextSequence,100,37,36,1,37699584,99.0179637628638 -Ingolstadt,CBOR,100,46,45,2,230326272,98.399472694799 -Ingolstadt,BSON,100,81,79,9,339099648,96.72413180649 -Montreal,FlatCityBuf,100,0,0,0,10928128,99.40305247354932 -Montreal,CityJSONTextSequence,100,50,50,1,43368448,99.08574539447326 -Montreal,CBOR,100,59,58,5,319373312,98.66464430640566 -Montreal,BSON,100,152,150,7,552452096,99.06070233197762 -NYC,FlatCityBuf,100,42,41,1,6897664,99.17255958578058 -NYC,CityJSONTextSequence,100,949,914,90,23789568,95.21999143292643 -NYC,CBOR,100,1395,1377,56,1808646144,98.56324055848587 -NYC,BSON,100,1782,1762,68,2659188736,99.147835025486 -Rotterdam,FlatCityBuf,100,1,1,0,4521984,99.31911218634522 -Rotterdam,CityJSONTextSequence,100,21,20,5,11337728,97.81625002139126 -Rotterdam,CBOR,100,31,30,2,215973888,99.06668173785624 -Rotterdam,BSON,100,66,66,2,356450304,99.36110411041496 -Vienna,FlatCityBuf,100,1,1,0,6012928,99.66218216110155 -Vienna,CityJSONTextSequence,100,46,45,5,19562496,98.2029093708528 -Vienna,CBOR,100,59,58,3,232144896,98.65077285771078 -Vienna,BSON,100,81,81,2,301219840,99.37116258271212 -Zurich,FlatCityBuf,100,149,148,3,6193152,99.29371613622536 -Zurich,CityJSONTextSequence,100,1878,1860,57,32718848,98.57515130737164 -Zurich,CBOR,100,3444,3418,100,4908105728,98.87637777009878 -Zurich,BSON,100,5252,5134,372,7715422208,97.81742179232548 -Subset of Tokyo (PLATEAU),FlatCityBuf,100,94,93,2,15613952,99.38998847372972 -Subset of Tokyo (PLATEAU),CityJSONTextSequence,100,1953,1942,48,75431936,98.92246045025897 -Subset of Tokyo (PLATEAU),CBOR,100,3121,2770,1551,4439490560,89.47518246543649 -Subset of Tokyo (PLATEAU),BSON,100,9739,9057,3969,7798669312,80.36606354599917 -Takeshiba (PLATEAU) Brid,FlatCityBuf,100,0,0,0,20086784,92.0388804522132 -Takeshiba (PLATEAU) Brid,CityJSONTextSequence,100,89,86,10,84180992,93.99670542036908 -Takeshiba (PLATEAU) Brid,CBOR,100,71,69,12,301203456,95.39011371231496 -Takeshiba (PLATEAU) Brid,BSON,100,179,177,6,527892480,97.95965586339418 -Takeshiba (PLATEAU) Rail way,FlatCityBuf,100,2,2,0,4882432,98.4006462076536 -Takeshiba (PLATEAU) Rail way,CityJSONTextSequence,100,38,38,1,17629184,98.05773530645432 -Takeshiba (PLATEAU) Rail way,CBOR,100,47,47,0,117587968,98.68700018503849 -Takeshiba (PLATEAU) Rail way,BSON,100,85,84,4,297861120,97.36989467400537 -Takeshiba (PLATEAU) Transport,FlatCityBuf,100,14,13,1,24150016,97.26916879782397 -Takeshiba (PLATEAU) Transport,CityJSONTextSequence,100,259,254,15,80445440,96.62585143418548 -Takeshiba (PLATEAU) Transport,CBOR,100,344,332,28,589692928,95.11083955615382 -Takeshiba (PLATEAU) Transport,BSON,100,603,584,60,1111769088,93.92702447383952 -Takeshiba (PLATEAU) Tunnel,FlatCityBuf,100,2,2,0,14909440,96.90171344875516 -Takeshiba (PLATEAU) Tunnel,CityJSONTextSequence,100,50,49,7,85753856,96.17469265806488 -Takeshiba (PLATEAU) Tunnel,CBOR,100,156,153,10,408453120,96.38782490919522 -Takeshiba (PLATEAU) Tunnel,BSON,100,251,249,7,647987200,97.85259408345532 -Takeshiba (PLATEAU) Vegetation,FlatCityBuf,100,33,33,3,73842688,97.3924990518894 -Takeshiba (PLATEAU) Vegetation,CityJSONTextSequence,100,900,875,95,207060992,95.69025579016372 -Takeshiba (PLATEAU) Vegetation,CBOR,100,1055,1038,66,1857798144,96.66896798043034 -Takeshiba (PLATEAU) Vegetation,BSON,100,2154,2134,69,4125687808,97.09431786411452 diff --git a/src/rust/fcb_core/benchmark_results_20250531_150434.txt b/src/rust/fcb_core/benchmark_results_20250531_150434.txt deleted file mode 100644 index 4ffab47..0000000 --- a/src/rust/fcb_core/benchmark_results_20250531_150434.txt +++ /dev/null @@ -1,129 +0,0 @@ -FLATCITYBUF BENCHMARK RESULTS WITH STATISTICS -Generated: 2025-05-31 15:04:34 -Iterations per test: 100 -======================================================================================================================== - -ALL RESULTS: ------------------------------------------------------------------------------------------------------------------------- -Dataset Format Mean Time Median Time Std Dev Memory CPU % ------------------------------------------------------------------------------------------------------------------------- -3DBAG FlatCityBuf 6.00ms 6.00ms 0.00ms 5.42 MB 99.66% -3DBAG CityJSONTextSequence 54.00ms 53.00ms 5.00ms 25.67 MB 98.70% -3DBAG CBOR 73.00ms 72.00ms 6.00ms 263.33 MB 98.42% -3DBAG BSON 113.00ms 111.00ms 10.00ms 323.19 MB 98.51% -3DBV FlatCityBuf 120.00ms 119.00ms 3.00ms 59.02 MB 99.42% -3DBV CityJSONTextSequence 3.79s 3.78s 42.00ms 352.33 MB 99.61% -3DBV CBOR 5.05s 5.01s 226.00ms 6.72 GB 93.75% -3DBV BSON 8.82s 8.79s 288.00ms 7.44 GB 95.91% -Helsinki FlatCityBuf 127.00ms 127.00ms 1.00ms 4.69 MB 99.85% -Helsinki CityJSONTextSequence 3.48s 3.47s 41.00ms 16.11 MB 99.52% -Helsinki CBOR 6.30s 6.33s 232.00ms 6.96 GB 97.56% -Helsinki BSON 9.69s 9.61s 389.00ms 7.42 GB 95.96% -Ingolstadt FlatCityBuf 0.00ms 0.00ms 0.00ms 8.98 MB 97.74% -Ingolstadt CityJSONTextSequence 37.00ms 36.00ms 1.00ms 35.95 MB 99.02% -Ingolstadt CBOR 46.00ms 45.00ms 2.00ms 219.66 MB 98.40% -Ingolstadt BSON 81.00ms 79.00ms 9.00ms 323.39 MB 96.72% -Montreal FlatCityBuf 0.00ms 0.00ms 0.00ms 10.42 MB 99.40% -Montreal CityJSONTextSequence 50.00ms 50.00ms 1.00ms 41.36 MB 99.09% -Montreal CBOR 59.00ms 58.00ms 5.00ms 304.58 MB 98.66% -Montreal BSON 152.00ms 150.00ms 7.00ms 526.86 MB 99.06% -NYC FlatCityBuf 42.00ms 41.00ms 1.00ms 6.58 MB 99.17% -NYC CityJSONTextSequence 949.00ms 914.00ms 90.00ms 22.69 MB 95.22% -NYC CBOR 1.40s 1.38s 56.00ms 1.68 GB 98.56% -NYC BSON 1.78s 1.76s 68.00ms 2.48 GB 99.15% -Rotterdam FlatCityBuf 1.00ms 1.00ms 0.00ms 4.31 MB 99.32% -Rotterdam CityJSONTextSequence 21.00ms 20.00ms 5.00ms 10.81 MB 97.82% -Rotterdam CBOR 31.00ms 30.00ms 2.00ms 205.97 MB 99.07% -Rotterdam BSON 66.00ms 66.00ms 2.00ms 339.94 MB 99.36% -Vienna FlatCityBuf 1.00ms 1.00ms 0.00ms 5.73 MB 99.66% -Vienna CityJSONTextSequence 46.00ms 45.00ms 5.00ms 18.66 MB 98.20% -Vienna CBOR 59.00ms 58.00ms 3.00ms 221.39 MB 98.65% -Vienna BSON 81.00ms 81.00ms 2.00ms 287.27 MB 99.37% -Zurich FlatCityBuf 149.00ms 148.00ms 3.00ms 5.91 MB 99.29% -Zurich CityJSONTextSequence 1.88s 1.86s 57.00ms 31.20 MB 98.58% -Zurich CBOR 3.44s 3.42s 100.00ms 4.57 GB 98.88% -Zurich BSON 5.25s 5.13s 372.00ms 7.19 GB 97.82% -Subset of Tokyo (PLATEAU) FlatCityBuf 94.00ms 93.00ms 2.00ms 14.89 MB 99.39% -Subset of Tokyo (PLATEAU) CityJSONTextSequence 1.95s 1.94s 48.00ms 71.94 MB 98.92% -Subset of Tokyo (PLATEAU) CBOR 3.12s 2.77s 1.55s 4.13 GB 89.48% -Subset of Tokyo (PLATEAU) BSON 9.74s 9.06s 3.97s 7.26 GB 80.37% -Takeshiba (PLATEAU) Brid FlatCityBuf 0.00ms 0.00ms 0.00ms 19.16 MB 92.04% -Takeshiba (PLATEAU) Brid CityJSONTextSequence 89.00ms 86.00ms 10.00ms 80.28 MB 94.00% -Takeshiba (PLATEAU) Brid CBOR 71.00ms 69.00ms 12.00ms 287.25 MB 95.39% -Takeshiba (PLATEAU) Brid BSON 179.00ms 177.00ms 6.00ms 503.44 MB 97.96% -Takeshiba (PLATEAU) Rail way FlatCityBuf 2.00ms 2.00ms 0.00ms 4.66 MB 98.40% -Takeshiba (PLATEAU) Rail way CityJSONTextSequence 38.00ms 38.00ms 1.00ms 16.81 MB 98.06% -Takeshiba (PLATEAU) Rail way CBOR 47.00ms 47.00ms 0.00ms 112.14 MB 98.69% -Takeshiba (PLATEAU) Rail way BSON 85.00ms 84.00ms 4.00ms 284.06 MB 97.37% -Takeshiba (PLATEAU) Transport FlatCityBuf 14.00ms 13.00ms 1.00ms 23.03 MB 97.27% -Takeshiba (PLATEAU) Transport CityJSONTextSequence 259.00ms 254.00ms 15.00ms 76.72 MB 96.63% -Takeshiba (PLATEAU) Transport CBOR 344.00ms 332.00ms 28.00ms 562.38 MB 95.11% -Takeshiba (PLATEAU) Transport BSON 603.00ms 584.00ms 60.00ms 1.04 GB 93.93% -Takeshiba (PLATEAU) Tunnel FlatCityBuf 2.00ms 2.00ms 0.00ms 14.22 MB 96.90% -Takeshiba (PLATEAU) Tunnel CityJSONTextSequence 50.00ms 49.00ms 7.00ms 81.78 MB 96.17% -Takeshiba (PLATEAU) Tunnel CBOR 156.00ms 153.00ms 10.00ms 389.53 MB 96.39% -Takeshiba (PLATEAU) Tunnel BSON 251.00ms 249.00ms 7.00ms 617.97 MB 97.85% -Takeshiba (PLATEAU) Vegetation FlatCityBuf 33.00ms 33.00ms 3.00ms 70.42 MB 97.39% -Takeshiba (PLATEAU) Vegetation CityJSONTextSequence 900.00ms 875.00ms 95.00ms 197.47 MB 95.69% -Takeshiba (PLATEAU) Vegetation CBOR 1.05s 1.04s 66.00ms 1.73 GB 96.67% -Takeshiba (PLATEAU) Vegetation BSON 2.15s 2.13s 69.00ms 3.84 GB 97.09% - -CityJSONTextSequence vs FlatCityBuf Comparison (Mean Times): --------------------------------------------------------------------------------------------------------------------------------------------- -Dataset CityJSONTextSequence Mean FCB Mean Time Ratio CityJSONTextSequence StdDev FCB StdDev CityJSONTextSequence Mem FCB Memory Mem Ratio --------------------------------------------------------------------------------------------------------------------------------------------- -3DBAG 54.00ms 6.00ms 9.00x 5.00ms 0.00ms 25.67 MB 5.42 MB 4.73x -3DBV 3.79s 120.00ms 31.56x 42.00ms 3.00ms 352.33 MB 59.02 MB 5.97x -Helsinki 3.48s 127.00ms 27.44x 41.00ms 1.00ms 16.11 MB 4.69 MB 3.44x -Ingolstadt 37.00ms 0.00ms infx 1.00ms 0.00ms 35.95 MB 8.98 MB 4.00x -Montreal 50.00ms 0.00ms infx 1.00ms 0.00ms 41.36 MB 10.42 MB 3.97x -NYC 949.00ms 42.00ms 22.60x 90.00ms 1.00ms 22.69 MB 6.58 MB 3.45x -Rotterdam 21.00ms 1.00ms 21.00x 5.00ms 0.00ms 10.81 MB 4.31 MB 2.51x -Vienna 46.00ms 1.00ms 46.00x 5.00ms 0.00ms 18.66 MB 5.73 MB 3.25x -Zurich 1.88s 149.00ms 12.60x 57.00ms 3.00ms 31.20 MB 5.91 MB 5.28x -Subset of Tokyo (PLATEAU) 1.95s 94.00ms 20.78x 48.00ms 2.00ms 71.94 MB 14.89 MB 4.83x -Takeshiba (PLATEAU) Brid 89.00ms 0.00ms infx 10.00ms 0.00ms 80.28 MB 19.16 MB 4.19x -Takeshiba (PLATEAU) Rail way 38.00ms 2.00ms 19.00x 1.00ms 0.00ms 16.81 MB 4.66 MB 3.61x -Takeshiba (PLATEAU) Transport 259.00ms 14.00ms 18.50x 15.00ms 1.00ms 76.72 MB 23.03 MB 3.33x -Takeshiba (PLATEAU) Tunnel 50.00ms 2.00ms 25.00x 7.00ms 0.00ms 81.78 MB 14.22 MB 5.75x -Takeshiba (PLATEAU) Vegetation 900.00ms 33.00ms 27.27x 95.00ms 3.00ms 197.47 MB 70.42 MB 2.80x - -CBOR vs FlatCityBuf Comparison (Mean Times): --------------------------------------------------------------------------------------------------------------------------------------------- -Dataset CBOR Mean FCB Mean Time Ratio CBOR StdDev FCB StdDev CBOR Mem FCB Memory Mem Ratio --------------------------------------------------------------------------------------------------------------------------------------------- -3DBAG 73.00ms 6.00ms 12.17x 6.00ms 0.00ms 263.33 MB 5.42 MB 48.57x -3DBV 5.05s 120.00ms 42.08x 226.00ms 3.00ms 6.72 GB 59.02 MB 116.56x -Helsinki 6.30s 127.00ms 49.65x 232.00ms 1.00ms 6.96 GB 4.69 MB 1521.47x -Ingolstadt 46.00ms 0.00ms infx 2.00ms 0.00ms 219.66 MB 8.98 MB 24.45x -Montreal 59.00ms 0.00ms infx 5.00ms 0.00ms 304.58 MB 10.42 MB 29.22x -NYC 1.40s 42.00ms 33.21x 56.00ms 1.00ms 1.68 GB 6.58 MB 262.21x -Rotterdam 31.00ms 1.00ms 31.00x 2.00ms 0.00ms 205.97 MB 4.31 MB 47.76x -Vienna 59.00ms 1.00ms 59.00x 3.00ms 0.00ms 221.39 MB 5.73 MB 38.61x -Zurich 3.44s 149.00ms 23.11x 100.00ms 3.00ms 4.57 GB 5.91 MB 792.51x -Subset of Tokyo (PLATEAU) 3.12s 94.00ms 33.20x 1.55s 2.00ms 4.13 GB 14.89 MB 284.33x -Takeshiba (PLATEAU) Brid 71.00ms 0.00ms infx 12.00ms 0.00ms 287.25 MB 19.16 MB 15.00x -Takeshiba (PLATEAU) Rail way 47.00ms 2.00ms 23.50x 0.00ms 0.00ms 112.14 MB 4.66 MB 24.08x -Takeshiba (PLATEAU) Transport 344.00ms 14.00ms 24.57x 28.00ms 1.00ms 562.38 MB 23.03 MB 24.42x -Takeshiba (PLATEAU) Tunnel 156.00ms 2.00ms 78.00x 10.00ms 0.00ms 389.53 MB 14.22 MB 27.40x -Takeshiba (PLATEAU) Vegetation 1.05s 33.00ms 31.97x 66.00ms 3.00ms 1.73 GB 70.42 MB 25.16x - -BSON vs FlatCityBuf Comparison (Mean Times): --------------------------------------------------------------------------------------------------------------------------------------------- -Dataset BSON Mean FCB Mean Time Ratio BSON StdDev FCB StdDev BSON Mem FCB Memory Mem Ratio --------------------------------------------------------------------------------------------------------------------------------------------- -3DBAG 113.00ms 6.00ms 18.83x 10.00ms 0.00ms 323.19 MB 5.42 MB 59.61x -3DBV 8.82s 120.00ms 73.53x 288.00ms 3.00ms 7.44 GB 59.02 MB 129.07x -Helsinki 9.69s 127.00ms 76.32x 389.00ms 1.00ms 7.42 GB 4.69 MB 1620.90x -Ingolstadt 81.00ms 0.00ms infx 9.00ms 0.00ms 323.39 MB 8.98 MB 35.99x -Montreal 152.00ms 0.00ms infx 7.00ms 0.00ms 526.86 MB 10.42 MB 50.55x -NYC 1.78s 42.00ms 42.43x 68.00ms 1.00ms 2.48 GB 6.58 MB 385.52x -Rotterdam 66.00ms 1.00ms 66.00x 2.00ms 0.00ms 339.94 MB 4.31 MB 78.83x -Vienna 81.00ms 1.00ms 81.00x 2.00ms 0.00ms 287.27 MB 5.73 MB 50.10x -Zurich 5.25s 149.00ms 35.25x 372.00ms 3.00ms 7.19 GB 5.91 MB 1245.80x -Subset of Tokyo (PLATEAU) 9.74s 94.00ms 103.61x 3.97s 2.00ms 7.26 GB 14.89 MB 499.47x -Takeshiba (PLATEAU) Brid 179.00ms 0.00ms infx 6.00ms 0.00ms 503.44 MB 19.16 MB 26.28x -Takeshiba (PLATEAU) Rail way 85.00ms 2.00ms 42.50x 4.00ms 0.00ms 284.06 MB 4.66 MB 61.01x -Takeshiba (PLATEAU) Transport 603.00ms 14.00ms 43.07x 60.00ms 1.00ms 1.04 GB 23.03 MB 46.04x -Takeshiba (PLATEAU) Tunnel 251.00ms 2.00ms 125.50x 7.00ms 0.00ms 617.97 MB 14.22 MB 43.46x -Takeshiba (PLATEAU) Vegetation 2.15s 33.00ms 65.27x 69.00ms 3.00ms 3.84 GB 70.42 MB 55.87x diff --git a/src/rust/fcb_core/benchmark_summary_20250531_150434.csv b/src/rust/fcb_core/benchmark_summary_20250531_150434.csv deleted file mode 100644 index 282618b..0000000 --- a/src/rust/fcb_core/benchmark_summary_20250531_150434.csv +++ /dev/null @@ -1,16 +0,0 @@ -Dataset,Fastest_Format,Fastest_Mean_Ms,Most_Consistent_Format,Lowest_StdDev_Ms,Lowest_Memory_Format,Lowest_Memory_Bytes,Lowest_CPU_Format,Lowest_CPU_Percent -3DBAG,FlatCityBuf,6,FlatCityBuf,0,FlatCityBuf,5685248,CB,98.42 -3DBV,FlatCityBuf,120,FlatCityBuf,3,FlatCityBuf,61882368,CB,93.75 -Helsinki,FlatCityBuf,127,FlatCityBuf,1,FlatCityBuf,4915200,BS,95.96 -Ingolstadt,FlatCityBuf,0,FlatCityBuf,0,FlatCityBuf,9420800,BS,96.72 -Montreal,FlatCityBuf,0,FlatCityBuf,0,FlatCityBuf,10928128,CB,98.66 -NYC,FlatCityBuf,42,FlatCityBuf,1,FlatCityBuf,6897664,Ci,95.22 -Rotterdam,FlatCityBuf,1,FlatCityBuf,0,FlatCityBuf,4521984,Ci,97.82 -Vienna,FlatCityBuf,1,FlatCityBuf,0,FlatCityBuf,6012928,Ci,98.20 -Zurich,FlatCityBuf,149,FlatCityBuf,3,FlatCityBuf,6193152,BS,97.82 -Subset of Tokyo (PLATEAU),FlatCityBuf,94,FlatCityBuf,2,FlatCityBuf,15613952,BS,80.37 -Takeshiba (PLATEAU) Brid,FlatCityBuf,0,FlatCityBuf,0,FlatCityBuf,20086784,Fl,92.04 -Takeshiba (PLATEAU) Rail way,FlatCityBuf,2,FlatCityBuf,0,FlatCityBuf,4882432,BS,97.37 -Takeshiba (PLATEAU) Transport,FlatCityBuf,14,FlatCityBuf,1,FlatCityBuf,24150016,BS,93.93 -Takeshiba (PLATEAU) Tunnel,FlatCityBuf,2,FlatCityBuf,0,FlatCityBuf,14909440,Ci,96.17 -Takeshiba (PLATEAU) Vegetation,FlatCityBuf,33,FlatCityBuf,3,FlatCityBuf,73842688,Ci,95.69 diff --git a/src/rust/fcb_core/comparison_bson_20250531_150434.csv b/src/rust/fcb_core/comparison_bson_20250531_150434.csv deleted file mode 100644 index d2b812d..0000000 --- a/src/rust/fcb_core/comparison_bson_20250531_150434.csv +++ /dev/null @@ -1,16 +0,0 @@ -Dataset,BSON_Mean_Ms,FCB_Mean_Ms,Time_Ratio,BSON_Median_Ms,FCB_Median_Ms,BSON_StdDev_Ms,FCB_StdDev_Ms,BSON_Memory_Bytes,FCB_Memory_Bytes,Memory_Ratio,BSON_CPU_Percent,FCB_CPU_Percent -3DBAG,113,6,18.833,111,6,10,0,338886656,5685248,59.608,98.51,99.66 -3DBV,8824,120,73.533,8786,119,288,3,7987085312,61882368,129.069,95.91,99.42 -Helsinki,9693,127,76.323,9607,127,389,1,7967064064,4915200,1620.903,95.96,99.85 -Ingolstadt,81,0,inf,79,0,9,0,339099648,9420800,35.995,96.72,97.74 -Montreal,152,0,inf,150,0,7,0,552452096,10928128,50.553,99.06,99.40 -NYC,1782,42,42.429,1762,41,68,1,2659188736,6897664,385.520,99.15,99.17 -Rotterdam,66,1,66.000,66,1,2,0,356450304,4521984,78.826,99.36,99.32 -Vienna,81,1,81.000,81,1,2,0,301219840,6012928,50.095,99.37,99.66 -Zurich,5252,149,35.248,5134,148,372,3,7715422208,6193152,1245.799,97.82,99.29 -Subset of Tokyo (PLATEAU),9739,94,103.606,9057,93,3969,2,7798669312,15613952,499.468,80.37,99.39 -Takeshiba (PLATEAU) Brid,179,0,inf,177,0,6,0,527892480,20086784,26.281,97.96,92.04 -Takeshiba (PLATEAU) Rail way,85,2,42.500,84,2,4,0,297861120,4882432,61.007,97.37,98.40 -Takeshiba (PLATEAU) Transport,603,14,43.071,584,13,60,1,1111769088,24150016,46.036,93.93,97.27 -Takeshiba (PLATEAU) Tunnel,251,2,125.500,249,2,7,0,647987200,14909440,43.462,97.85,96.90 -Takeshiba (PLATEAU) Vegetation,2154,33,65.273,2134,33,69,3,4125687808,73842688,55.871,97.09,97.39 diff --git a/src/rust/fcb_core/comparison_cbor_20250531_150434.csv b/src/rust/fcb_core/comparison_cbor_20250531_150434.csv deleted file mode 100644 index fffd90e..0000000 --- a/src/rust/fcb_core/comparison_cbor_20250531_150434.csv +++ /dev/null @@ -1,16 +0,0 @@ -Dataset,CBOR_Mean_Ms,FCB_Mean_Ms,Time_Ratio,CBOR_Median_Ms,FCB_Median_Ms,CBOR_StdDev_Ms,FCB_StdDev_Ms,CBOR_Memory_Bytes,FCB_Memory_Bytes,Memory_Ratio,CBOR_CPU_Percent,FCB_CPU_Percent -3DBAG,73,6,12.167,72,6,6,0,276119552,5685248,48.568,98.42,99.66 -3DBV,5050,120,42.083,5009,119,226,3,7212744704,61882368,116.556,93.75,99.42 -Helsinki,6305,127,49.646,6329,127,232,1,7478312960,4915200,1521.467,97.56,99.85 -Ingolstadt,46,0,inf,45,0,2,0,230326272,9420800,24.449,98.40,97.74 -Montreal,59,0,inf,58,0,5,0,319373312,10928128,29.225,98.66,99.40 -NYC,1395,42,33.214,1377,41,56,1,1808646144,6897664,262.211,98.56,99.17 -Rotterdam,31,1,31.000,30,1,2,0,215973888,4521984,47.761,99.07,99.32 -Vienna,59,1,59.000,58,1,3,0,232144896,6012928,38.608,98.65,99.66 -Zurich,3444,149,23.114,3418,148,100,3,4908105728,6193152,792.505,98.88,99.29 -Subset of Tokyo (PLATEAU),3121,94,33.202,2770,93,1551,2,4439490560,15613952,284.328,89.48,99.39 -Takeshiba (PLATEAU) Brid,71,0,inf,69,0,12,0,301203456,20086784,14.995,95.39,92.04 -Takeshiba (PLATEAU) Rail way,47,2,23.500,47,2,0,0,117587968,4882432,24.084,98.69,98.40 -Takeshiba (PLATEAU) Transport,344,14,24.571,332,13,28,1,589692928,24150016,24.418,95.11,97.27 -Takeshiba (PLATEAU) Tunnel,156,2,78.000,153,2,10,0,408453120,14909440,27.396,96.39,96.90 -Takeshiba (PLATEAU) Vegetation,1055,33,31.970,1038,33,66,3,1857798144,73842688,25.159,96.67,97.39 diff --git a/src/rust/fcb_core/comparison_cityjsontextsequence_20250531_150434.csv b/src/rust/fcb_core/comparison_cityjsontextsequence_20250531_150434.csv deleted file mode 100644 index 9b2e9d5..0000000 --- a/src/rust/fcb_core/comparison_cityjsontextsequence_20250531_150434.csv +++ /dev/null @@ -1,16 +0,0 @@ -Dataset,CityJSONTextSequence_Mean_Ms,FCB_Mean_Ms,Time_Ratio,CityJSONTextSequence_Median_Ms,FCB_Median_Ms,CityJSONTextSequence_StdDev_Ms,FCB_StdDev_Ms,CityJSONTextSequence_Memory_Bytes,FCB_Memory_Bytes,Memory_Ratio,CityJSONTextSequence_CPU_Percent,FCB_CPU_Percent -3DBAG,54,6,9.000,53,6,5,0,26918912,5685248,4.735,98.70,99.66 -3DBV,3787,120,31.558,3776,119,42,3,369442816,61882368,5.970,99.61,99.42 -Helsinki,3485,127,27.441,3473,127,41,1,16891904,4915200,3.437,99.52,99.85 -Ingolstadt,37,0,inf,36,0,1,0,37699584,9420800,4.002,99.02,97.74 -Montreal,50,0,inf,50,0,1,0,43368448,10928128,3.969,99.09,99.40 -NYC,949,42,22.595,914,41,90,1,23789568,6897664,3.449,95.22,99.17 -Rotterdam,21,1,21.000,20,1,5,0,11337728,4521984,2.507,97.82,99.32 -Vienna,46,1,46.000,45,1,5,0,19562496,6012928,3.253,98.20,99.66 -Zurich,1878,149,12.604,1860,148,57,3,32718848,6193152,5.283,98.58,99.29 -Subset of Tokyo (PLATEAU),1953,94,20.777,1942,93,48,2,75431936,15613952,4.831,98.92,99.39 -Takeshiba (PLATEAU) Brid,89,0,inf,86,0,10,0,84180992,20086784,4.191,94.00,92.04 -Takeshiba (PLATEAU) Rail way,38,2,19.000,38,2,1,0,17629184,4882432,3.611,98.06,98.40 -Takeshiba (PLATEAU) Transport,259,14,18.500,254,13,15,1,80445440,24150016,3.331,96.63,97.27 -Takeshiba (PLATEAU) Tunnel,50,2,25.000,49,2,7,0,85753856,14909440,5.752,96.17,96.90 -Takeshiba (PLATEAU) Vegetation,900,33,27.273,875,33,95,3,207060992,73842688,2.804,95.69,97.39 diff --git a/src/rust/fcb_core/http_benchmark_results.csv b/src/rust/fcb_core/http_benchmark_results.csv deleted file mode 100644 index 56cfbb6..0000000 --- a/src/rust/fcb_core/http_benchmark_results.csv +++ /dev/null @@ -1,13 +0,0 @@ -Method,FeatureID,Iterations,MeanDurationMs,MedianDurationMs,StdDevDurationMs,MinDurationMs,MaxDurationMs,SuccessRate,TotalBytesTransferred -FlatCityBuf,NL.IMBAG.Pand.0503100000032914,50,935.774,904.973,131.816,754.100,1362.453,100.00,8433800 -3DBAG_API,NL.IMBAG.Pand.0503100000032914,50,2412.465,2416.386,130.141,2120.933,2708.883,100.00,11388550 -FlatCityBuf,NL.IMBAG.Pand.0363100012185598,50,858.038,822.580,128.192,653.616,1261.902,100.00,1891600 -3DBAG_API,NL.IMBAG.Pand.0363100012185598,50,2106.726,2097.295,118.498,1847.265,2460.586,100.00,1953650 -FlatCityBuf,NL.IMBAG.Pand.0014100010938997,50,821.744,792.390,112.283,665.544,1081.178,100.00,6337800 -3DBAG_API,NL.IMBAG.Pand.0014100010938997,50,2254.798,2244.973,92.891,1998.909,2437.335,100.00,7816800 -FlatCityBuf,NL.IMBAG.Pand.0772100000295227,50,1378.656,1384.988,263.945,794.379,1897.999,100.00,873400 -3DBAG_API,NL.IMBAG.Pand.0772100000295227,50,2013.363,2038.606,93.383,1851.911,2239.590,100.00,1124500 -FlatCityBuf,NL.IMBAG.Pand.0153100000261851,50,1070.415,1034.361,166.033,743.244,1535.202,100.00,556400 -3DBAG_API,NL.IMBAG.Pand.0153100000261851,50,2058.810,2082.235,90.661,1848.690,2283.745,100.00,662350 -FlatCityBuf_BBox,bbox_query,50,492.618,494.725,79.383,345.282,670.920,100.00,2948400 -3DBAG_API_BBox,bbox_query,50,7420.307,7415.561,226.060,6963.676,8072.223,100.00,3671300 diff --git a/src/rust/fcb_core/src/bin/read_cj.rs b/src/rust/fcb_core/src/bin/read_cj.rs index 68f63ca..a1ed5f6 100644 --- a/src/rust/fcb_core/src/bin/read_cj.rs +++ b/src/rust/fcb_core/src/bin/read_cj.rs @@ -20,7 +20,7 @@ fn read_cj(input: &str) -> Result<()> { let mut document_nummers = Vec::new(); let mut identifications = Vec::new(); for (i, feature) in features.iter().enumerate() { - if i as u64 % (feature_count / 10) == 0 { + if (i as u64).is_multiple_of(feature_count / 10) { feature.city_objects.iter().for_each(|(_, co)| { if let Some(attributes) = &co.attributes { if let Some(document_number) = diff --git a/src/rust/fcb_core/src/bin/stats.rs b/src/rust/fcb_core/src/bin/stats.rs index 71eaa32..4cd5891 100644 --- a/src/rust/fcb_core/src/bin/stats.rs +++ b/src/rust/fcb_core/src/bin/stats.rs @@ -167,11 +167,9 @@ fn analyze_jsonl(path: &Path) -> Result<(u64, f64)> { let mut feature_count = 0; let mut total_feature_size = 0; - for line in lines { - if let Ok(line_content) = line { - total_feature_size += line_content.len(); - feature_count += 1; - } + for line_content in lines.flatten() { + total_feature_size += line_content.len(); + feature_count += 1; } let avg_feature_size = if feature_count > 0 { diff --git a/src/rust/fcb_core/src/fb/feature_generated.rs b/src/rust/fcb_core/src/fb/feature_generated.rs index 8901992..a54de3d 100644 --- a/src/rust/fcb_core/src/fb/feature_generated.rs +++ b/src/rust/fcb_core/src/fb/feature_generated.rs @@ -1049,7 +1049,7 @@ impl core::fmt::Debug for CityObject<'_> { /// catch every error, or be maximally performant. For the /// previous, unchecked, behavior use /// `root_as_city_feature_unchecked`. -pub fn root_as_city_feature(buf: &[u8]) -> Result { +pub fn root_as_city_feature(buf: &[u8]) -> Result, flatbuffers::InvalidFlatbuffer> { flatbuffers::root::(buf) } #[inline] @@ -1061,7 +1061,7 @@ pub fn root_as_city_feature(buf: &[u8]) -> Result Result { +) -> Result, flatbuffers::InvalidFlatbuffer> { flatbuffers::size_prefixed_root::(buf) } #[inline] @@ -1094,14 +1094,14 @@ pub fn size_prefixed_root_as_city_feature_with_opts<'b, 'o>( /// Assumes, without verification, that a buffer of bytes contains a CityFeature and returns it. /// # Safety /// Callers must trust the given bytes do indeed contain a valid `CityFeature`. -pub unsafe fn root_as_city_feature_unchecked(buf: &[u8]) -> CityFeature { +pub unsafe fn root_as_city_feature_unchecked(buf: &[u8]) -> CityFeature<'_> { flatbuffers::root_unchecked::(buf) } #[inline] /// Assumes, without verification, that a buffer of bytes contains a size prefixed CityFeature and returns it. /// # Safety /// Callers must trust the given bytes do indeed contain a valid size prefixed `CityFeature`. -pub unsafe fn size_prefixed_root_as_city_feature_unchecked(buf: &[u8]) -> CityFeature { +pub unsafe fn size_prefixed_root_as_city_feature_unchecked(buf: &[u8]) -> CityFeature<'_> { flatbuffers::size_prefixed_root_unchecked::(buf) } #[inline] diff --git a/src/rust/fcb_core/src/fb/header_generated.rs b/src/rust/fcb_core/src/fb/header_generated.rs index 61f4e93..d46e1bc 100644 --- a/src/rust/fcb_core/src/fb/header_generated.rs +++ b/src/rust/fcb_core/src/fb/header_generated.rs @@ -3402,7 +3402,7 @@ impl core::fmt::Debug for Header<'_> { /// catch every error, or be maximally performant. For the /// previous, unchecked, behavior use /// `root_as_header_unchecked`. -pub fn root_as_header(buf: &[u8]) -> Result { +pub fn root_as_header(buf: &[u8]) -> Result, flatbuffers::InvalidFlatbuffer> { flatbuffers::root::
(buf) } #[inline] @@ -3412,7 +3412,9 @@ pub fn root_as_header(buf: &[u8]) -> Result Result { +pub fn size_prefixed_root_as_header( + buf: &[u8], +) -> Result, flatbuffers::InvalidFlatbuffer> { flatbuffers::size_prefixed_root::
(buf) } #[inline] @@ -3445,14 +3447,14 @@ pub fn size_prefixed_root_as_header_with_opts<'b, 'o>( /// Assumes, without verification, that a buffer of bytes contains a Header and returns it. /// # Safety /// Callers must trust the given bytes do indeed contain a valid `Header`. -pub unsafe fn root_as_header_unchecked(buf: &[u8]) -> Header { +pub unsafe fn root_as_header_unchecked(buf: &[u8]) -> Header<'_> { flatbuffers::root_unchecked::
(buf) } #[inline] /// Assumes, without verification, that a buffer of bytes contains a size prefixed Header and returns it. /// # Safety /// Callers must trust the given bytes do indeed contain a valid size prefixed `Header`. -pub unsafe fn size_prefixed_root_as_header_unchecked(buf: &[u8]) -> Header { +pub unsafe fn size_prefixed_root_as_header_unchecked(buf: &[u8]) -> Header<'_> { flatbuffers::size_prefixed_root_unchecked::
(buf) } #[inline] diff --git a/src/rust/fcb_core/src/reader/attr_query.rs b/src/rust/fcb_core/src/reader/attr_query.rs index 7dac432..140f3d4 100644 --- a/src/rust/fcb_core/src/reader/attr_query.rs +++ b/src/rust/fcb_core/src/reader/attr_query.rs @@ -393,15 +393,14 @@ impl FcbReader { magic_bytes: 8, header: header_size as u64, rtree_index: self.rtree_index_size(), - attributes: self.attr_index_size() as u64, + attributes: self.attr_index_size(), }; let total_feat_count = result_vec.len() as u64; let attr_index_size = self.attr_index_size(); - self.reader.seek(SeekFrom::Start( - attr_index_start_pos + attr_index_size as u64, - ))?; + self.reader + .seek(SeekFrom::Start(attr_index_start_pos + attr_index_size))?; Ok(FeatureIter::::new( self.reader, @@ -475,7 +474,7 @@ impl FcbReader { magic_bytes: 8, header: header_size as u64, rtree_index: self.rtree_index_size(), - attributes: self.attr_index_size() as u64, + attributes: self.attr_index_size(), }; let total_feat_count = result.len() as u64; diff --git a/src/rust/fcb_core/src/reader/city_buffer.rs b/src/rust/fcb_core/src/reader/city_buffer.rs index 7eb2cc1..603ecf0 100644 --- a/src/rust/fcb_core/src/reader/city_buffer.rs +++ b/src/rust/fcb_core/src/reader/city_buffer.rs @@ -12,11 +12,11 @@ pub struct FcbBuffer { } impl FcbBuffer { - pub fn header(&self) -> Header { + pub fn header(&self) -> Header<'_> { unsafe { size_prefixed_root_as_header_unchecked(&self.header_buf) } } - pub fn feature(&self) -> CityFeature { + pub fn feature(&self) -> CityFeature<'_> { unsafe { size_prefixed_root_as_city_feature_unchecked(&self.features_buf) } } diff --git a/src/rust/fcb_core/src/reader/mod.rs b/src/rust/fcb_core/src/reader/mod.rs index 5a4b920..8918acb 100644 --- a/src/rust/fcb_core/src/reader/mod.rs +++ b/src/rust/fcb_core/src/reader/mod.rs @@ -253,13 +253,13 @@ impl FcbReader { } impl FcbReader { - pub fn header(&self) -> Header { + pub fn header(&self) -> Header<'_> { self.buffer.header() } pub fn root_attr_schema( &self, - ) -> Option>> { + ) -> Option>>> { self.buffer.header().columns() } @@ -392,7 +392,7 @@ impl FallibleStreamingIterator for FeatureIter { } impl FeatureIter { - pub fn cur_feature(&self) -> CityFeature { + pub fn cur_feature(&self) -> CityFeature<'_> { self.buffer.feature() } @@ -404,7 +404,7 @@ impl FeatureIter { to_cj_feature(fcb_feature, root_attr_schema, semantic_attr_schema) } - pub fn get_features(&mut self) -> Result, Error> { + pub fn get_features(&mut self) -> Result>, Error> { // Ok(features) todo!("implement") } @@ -421,7 +421,7 @@ impl FeatureIter { } impl FeatureIter { - pub fn cur_feature(&self) -> CityFeature { + pub fn cur_feature(&self) -> CityFeature<'_> { self.buffer.feature() } pub fn cur_feature_len(&self) -> usize { @@ -439,7 +439,7 @@ impl FeatureIter { todo!("implement") } - pub fn get_current_feature(&self) -> CityFeature { + pub fn get_current_feature(&self) -> CityFeature<'_> { self.buffer.feature() } @@ -502,13 +502,13 @@ impl FeatureIter { iter } - pub fn header(&self) -> Header { + pub fn header(&self) -> Header<'_> { self.buffer.header() } pub fn root_attr_schema( &self, - ) -> Option>> { + ) -> Option>>> { self.buffer.header().columns() } diff --git a/src/rust/fcb_py/examples/basic_usage.py b/src/rust/fcb_py/examples/basic_usage.py index 2b1ce09..0a62873 100644 --- a/src/rust/fcb_py/examples/basic_usage.py +++ b/src/rust/fcb_py/examples/basic_usage.py @@ -67,11 +67,15 @@ def demonstrate_sync_reader(): if city_obj.geometry: for geom in city_obj.geometry: if geom is not None: - print(f" Geometry type: {geom.geometry_type}") + print( + f" Geometry type: {geom.geometry_type}" + ) print(f" Vertices index: {geom.vertices}") print(f" Boundaries: {geom.boundaries}") if geom.semantics: - print(f" Has semantics: {geom.semantics}") + print( + f" Has semantics: {geom.semantics}" + ) else: print(" Geometry is None") @@ -281,7 +285,9 @@ def demonstrate_api_features(): async def main(): """Main example function""" print("=== FlatCityBuf Python Bindings Example ===\n") - print("Demonstrates both local file access and HTTP access with CityJSON support\n") + print( + "Demonstrates both local file access and HTTP access with CityJSON support\n" + ) # Demonstrate synchronous reader demonstrate_sync_reader() diff --git a/src/rust/fcb_py/uv.lock b/src/rust/fcb_py/uv.lock index 0061ed1..8ac2649 100644 --- a/src/rust/fcb_py/uv.lock +++ b/src/rust/fcb_py/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.8.1" resolution-markers = [ "python_full_version >= '3.10'", @@ -40,7 +40,7 @@ wheels = [ [[package]] name = "flatcitybuf" -version = "0.1.2" +version = "0.2.0" source = { editable = "." } dependencies = [ { name = "pytest", version = "8.3.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, From 9dd0222367952d8a2c927bfd44421029b2aa54c8 Mon Sep 17 00:00:00 2001 From: HideBa Date: Fri, 19 Dec 2025 16:21:25 +0100 Subject: [PATCH 5/5] docs: update readme --- README.md | 23 +++++++++++++++-------- src/rust/cli/README.md | 17 +++++++++++++++-- 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 00d0988..b44aa2d 100644 --- a/README.md +++ b/README.md @@ -146,22 +146,29 @@ cd wasm && wasm-pack build --target web --release --out-dir ../../ts ### πŸ› οΈ CLI Usage -#### Convert CityJSONSeq to FlatCityBuf +#### Convert CityJSON/CityJSONSeq to FlatCityBuf -replace `cargo run -p fcb_cli` with `fcb` in the following commands if you want to use the binary directly. +Replace `cargo run -p fcb_cli --` with `fcb` in the following commands if you want to use the installed binary directly. ```bash -# Basic conversion -fcb fcb_cli ser -i input.city.jsonl -o output.fcb +# Basic conversion from CityJSONSeq +fcb ser -i input.city.jsonl -o output.fcb -# With compression and indexing options -fcb fcb_cli ser -i data.city.jsonl -o data.fcb +# Convert standard CityJSON file +fcb ser -i city.city.json -o output.fcb + +# Multiple input files +fcb ser -i file1.city.jsonl file2.city.jsonl -o merged.fcb + +# Glob patterns to process all matching files +fcb ser -i 'data/*.city.jsonl' -o output.fcb +fcb ser -i 'cities/**/*.city.json' -o all_cities.fcb # With spatial index and attribute index -fcb fcb_cli ser -i data.city.jsonl -o data.fcb --attr-index attribute_name,attribute_name2 --attr-branching-factor 256 +fcb ser -i data.city.jsonl -o data.fcb --attr-index attribute_name,attribute_name2 --attr-branching-factor 256 # Show information about the file -fcb fcb_cli info -i data.fcb +fcb info -i data.fcb ``` ### πŸ§ͺ Run Benchmarks diff --git a/src/rust/cli/README.md b/src/rust/cli/README.md index 9a26b6d..b74db28 100644 --- a/src/rust/cli/README.md +++ b/src/rust/cli/README.md @@ -52,7 +52,7 @@ fcb ser -i INPUT -o OUTPUT [OPTIONS] **Options:** -- `-i, --input INPUT` - Input file (use '-' for stdin) +- `-i, --input INPUT...` - Input file(s) or glob patterns (supports multiple files, use '-' for stdin) - `-o, --output OUTPUT` - Output file (use '-' for stdout) - `-a, --attr-index ATTRIBUTES` - Comma-separated list of attributes to create index for - `-A, --index-all-attributes` - Index all attributes found in the dataset @@ -64,9 +64,19 @@ fcb ser -i INPUT -o OUTPUT [OPTIONS] **Examples:** ```bash -# basic conversion +# basic conversion from CityJSONSeq fcb ser -i input.city.jsonl -o output.fcb +# convert CityJSON file (standard .json format) +fcb ser -i city.city.json -o output.fcb + +# multiple input files +fcb ser -i file1.city.jsonl file2.city.jsonl -o merged.fcb + +# glob patterns to process all matching files +fcb ser -i 'data/*.city.jsonl' -o output.fcb +fcb ser -i 'cities/**/*.city.json' -o all_cities.fcb + # with attribute indexing fcb ser -i delft.city.jsonl -o delft_attr.fcb \ --attr-index identificatie,tijdstipregistratie,b3_is_glas_dak,b3_h_dak_50p \ @@ -150,9 +160,12 @@ fcb bson -i INPUT -o OUTPUT ### Input Formats +- **CityJSON** (`.city.json`) - Standard CityJSON files - **CityJSON Text Sequences** (`.city.jsonl`) - Line-delimited CityJSON features - **FCB** (`.fcb`) - FlatCityBuf binary format +> **Multi-file Support:** The `ser` command accepts multiple input files and glob patterns. When merging files with different coordinate transforms, vertices are automatically aligned to the first file's transform. + ### Output Formats - **FCB** (`.fcb`) - FlatCityBuf binary format with optional indexing