Add null engine table implementation #3029

slgeliberty · 2025-11-22T03:18:38Z

Note

Adds end-to-end ClickHouse Null engine support (parsing, DDL/serialization, codegen, SDK validation), skips MV truncation for Null targets, and updates tests/templates.

ClickHouse engine support (Rust/CLI):
- Add Null to ClickhouseEngine with string round‑trip, proto serialization, hashing, and DDL emission in create_table_query.
- Extend engine parsing (TryFrom<&str>, regular engine parser) and SQL parser tests to recognize ENGINE = Null (case-insensitive).
- Partial infra map: accept engine: "Null" via EngineConfig and map to ClickhouseEngine::Null.
- Diff strategy: when auto-populating new materialized views in dev, skip truncation if target table uses Null; add helper to match MV target table.
Code generation:
- Python generator: emit NullEngine(); TypeScript generator: emit engine: ClickHouseEngines.Null.
Python SDK (py-moose-lib):
- Add ClickHouseEngines.Null and NullEngine config class.
- Enforce that NullEngine rejects ORDER BY, PARTITION BY, and SAMPLE BY in OlapConfig.
- Internal serialization: include Null in discriminated union and conversions.
TypeScript SDK (ts-moose-lib):
- Add ClickHouseEngines.Null, NullEngineConfig, and NullConfig<T>; validate that Null forbids ORDER BY/PARTITION BY/SAMPLE BY.
- Infra map builder outputs engine: "Null" for tables using Null.
Tests/templates:
- Add NullEngineTest tables to TS/Python templates and schema expectations.
- New tests for Null engine parsing/round‑trip and config handling in Rust/Python/TS.

^{Written by Cursor Bugbot for commit 88ef238. This will update automatically on new commits. Configure here.}

vercel · 2025-11-22T03:18:42Z

Someone is attempting to deploy a commit to the Fiveonefour Team on Vercel.

A member of the Team first needs to authorize it.

packages/py-moose-lib/moose_lib/dmv2/olap_table.py

packages/py-moose-lib/moose_lib/internal.py

… Tkinter import

apps/framework-cli/src/framework/python/generate.rs

…s_to_python in generate.rs

cursor

Bug: Null engine incorrectly allows PARTITION BY clause

The supports_partition_by matching at lines 2522-2532 doesn't include ClickhouseEngine::Null, which means Null engine tables will incorrectly permit PARTITION BY clauses. This is inconsistent with the Python validation in olap_table.py (lines 207-215) which correctly rejects PARTITION BY for NullEngine. ClickHouse's Null engine doesn't support PARTITION BY, so attempting to create such tables will fail at runtime despite passing Rust-side validation.

apps/framework-cli/src/infrastructure/olap/clickhouse/queries.rs#L2521-L2532

moosestack/apps/framework-cli/src/infrastructure/olap/clickhouse/queries.rs

Lines 2521 to 2532 in 5d180ef

    
           let supports_sample_by = table.engine.is_merge_tree_family(); 
        
           let supports_partition_by = matches!( 
        
               table.engine, 
        
               ClickhouseEngine::MergeTree 
        
                   | ClickhouseEngine::ReplacingMergeTree { .. } 
        
                   | ClickhouseEngine::AggregatingMergeTree 
        
                   | ClickhouseEngine::SummingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedReplacingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedAggregatingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedSummingMergeTree { .. } 
        
                   | ClickhouseEngine::S3 { .. }

packages/py-moose-lib/moose_lib/internal.py

callicles

Thanks a lot for the contribution!

Looks like this is directionality correct - you are missing the TS implementation though 🙂

examples/null-engine-example/null-engine-example-py/requirements.txt

callicles · 2025-11-22T21:59:30Z

examples/null-engine-example/null-engine-example-py/template.config.toml

@@ -0,0 +1,26 @@
+language = "python" # Must be typescript or python


Template definitions are in a different repo - looks like your AI mixed examples and templates.

We run templates/typescript-tests and templates/python-tests in the CI.

You could remove this example altogether and add it there for automated testing.

The prompt could be something along the lines of:

"The end-to-end test is being run by apps/framework-cli-e2e please add the null engine to the templates that are used in the tests: templates/typescript-tests and templates/python-tests"

@callicles:

TS implementation

Moved null-engine-example to template using framwork-cli-e2e

Avoid truncate behavior on null engine table link to olap change on null engine target table of materialized view

slgeliberty · 2025-11-23T09:01:51Z

Error occurs while setting null engine table as target table of materialized view because the webserver is trying to apply TRUNCATE TABLE statement which is not possible on null engine tables.

cursor

Bug: Null engine missing from partition support check

The supports_partition_by check doesn't include ClickhouseEngine::Null in the match statement, which means it defaults to false correctly by the match else clause. However, the explicit enumeration pattern used for other engines should include Null for clarity and consistency with validation in the Python SDK which explicitly lists NullEngine in engines_without_partition_by. The Null engine doesn't support PARTITION BY, ORDER BY, or SAMPLE BY clauses.

apps/framework-cli/src/infrastructure/olap/clickhouse/queries.rs#L2521-L2531

moosestack/apps/framework-cli/src/infrastructure/olap/clickhouse/queries.rs

Lines 2521 to 2531 in 3dcbf7c

    
           let supports_sample_by = table.engine.is_merge_tree_family(); 
        
           let supports_partition_by = matches!( 
        
               table.engine, 
        
               ClickhouseEngine::MergeTree 
        
                   | ClickhouseEngine::ReplacingMergeTree { .. } 
        
                   | ClickhouseEngine::AggregatingMergeTree 
        
                   | ClickhouseEngine::SummingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedReplacingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedAggregatingMergeTree { .. } 
        
                   | ClickhouseEngine::ReplicatedSummingMergeTree { .. }

callicles · 2025-11-24T23:55:54Z

@slgeliberty Nice Progress! This is going in the right direction 😄

…ialized view target table

cursor · 2025-11-25T08:46:29Z

apps/framework-cli/src/infrastructure/olap/clickhouse/diff_strategy.rs

+    ) -> bool {
+        if table.name == target_table {
+            return true;
+        }


Bug: Table matching ignores database when names match

The table_matches_mv_target function returns true when table names match without checking if databases also match. The early return at line 437-439 (if table.name == target_table { return true; }) bypasses all database checks. When a Null engine table exists with the same name as the MV target but in a different database, this incorrectly identifies it as the target. This causes target_is_null_engine to be true and should_truncate to be set to false when it should be true, potentially leading to incorrect materialized view population behavior.

callicles · 2025-12-01T21:29:55Z

@slgeliberty Do you want to push this accross the finish line or do you want us to help?

Add null engine table implementation

1f2b7ca

cursor bot reviewed Nov 22, 2025

View reviewed changes

packages/py-moose-lib/moose_lib/dmv2/olap_table.py Show resolved Hide resolved

packages/py-moose-lib/moose_lib/internal.py Outdated Show resolved Hide resolved

Add null engine in engines_without_partition_by and remove accidendal…

ef633d4

… Tkinter import

cursor bot reviewed Nov 22, 2025

View reviewed changes

apps/framework-cli/src/framework/python/generate.rs Show resolved Hide resolved

Remove import module in moose_lib and implemente null engine in table…

5d180ef

…s_to_python in generate.rs

cursor bot reviewed Nov 22, 2025

View reviewed changes

packages/py-moose-lib/moose_lib/internal.py Outdated Show resolved Hide resolved

fix null key in engine_map dict

175ac2e

slgeliberty marked this pull request as draft November 22, 2025 10:36

slgeliberty marked this pull request as ready for review November 22, 2025 11:24

admin added 2 commits November 22, 2025 14:05

Reset import of import_module in internal.py

11517f5

fix null key in engine_map dict

01c19eb

callicles reviewed Nov 22, 2025

View reviewed changes

Add null engine TS implementation and python/TS automated e2e tests

3dcbf7c

cursor bot reviewed Nov 24, 2025

View reviewed changes

avoid truncate on olap change of null engines tables defined as mater…

88ef238

…ialized view target table

cursor bot reviewed Nov 25, 2025

View reviewed changes

	let supports_sample_by = table.engine.is_merge_tree_family();
	let supports_partition_by = matches!(
	table.engine,
	ClickhouseEngine::MergeTree
	\| ClickhouseEngine::ReplacingMergeTree { .. }
	\| ClickhouseEngine::AggregatingMergeTree
	\| ClickhouseEngine::SummingMergeTree { .. }
	\| ClickhouseEngine::ReplicatedMergeTree { .. }
	\| ClickhouseEngine::ReplicatedReplacingMergeTree { .. }
	\| ClickhouseEngine::ReplicatedAggregatingMergeTree { .. }
	\| ClickhouseEngine::ReplicatedSummingMergeTree { .. }
	\| ClickhouseEngine::S3 { .. }

		@@ -0,0 +1,26 @@
		language = "python" # Must be typescript or python

Add null engine table implementation #3029

Are you sure you want to change the base?

Add null engine table implementation #3029

Uh oh!

Conversation

slgeliberty commented Nov 22, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Nov 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Null engine incorrectly allows PARTITION BY clause

Uh oh!

Uh oh!

callicles left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

callicles Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

slgeliberty Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

slgeliberty commented Nov 23, 2025

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Null engine missing from partition support check

Uh oh!

callicles commented Nov 24, 2025

Uh oh!

cursor bot Nov 25, 2025

Choose a reason for hiding this comment

Bug: Table matching ignores database when names match

Uh oh!

callicles commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

slgeliberty commented Nov 22, 2025 •

edited by cursor bot

Loading

callicles left a comment •

edited

Loading