DuckDB, Postgres, SQLite: NOT NULL and NOTNULL expressions #1927

ryanschneider · 2025-07-05T22:31:55Z

Fixes #1920 by adding NOT NULL generally as an alias for IS NOT NULL, (only in ParserState::Normal) and adds the non-standard NOTNULL keyword supported by DuckDB, SQLite, and Postgres. It also adds a new ParserState::ColumnDefinition to avoid incorrectly parsing NOT NULL as IS NOT NULL in column definitions.

Since this is my first non-trivial PR please provide any feedback! I confirmed all tests pass w/ cargo test and cargo test --all-features but if there's anything else I need to do please let me know, thanks!

src/dialect/duckdb.rs

src/parser/mod.rs

src/dialect/mod.rs

tests/sqlparser_common.rs

ryanschneider · 2025-07-11T00:22:06Z

Rebased against main to resolve conflicts.

… Expr::IsNotNull.

ryanschneider · 2025-07-11T19:05:32Z

@iffyio I went with the new ParserState::ColumnDefinition idea mentioned here: #1927 (comment) I think it leads to the cleanest code changes, let me know what you think! Also merged in main and resolved conflicts.

iffyio

Thanks @ryanschneider! The changes look reasonable to me overall, left some comments

tests/sqlparser_clickhouse.rs

src/parser/mod.rs

iffyio · 2025-07-16T14:16:53Z

src/parser/mod.rs

+        /// In those cases we use the following macro to parse instead of calling [parse_expr] directly.
+        macro_rules! parse_expr_normal {
+            ($option:expr) => {
+                if matches!(self.peek_token().token, Token::LParen) {


Suggested change

if matches!(self.peek_token().token, Token::LParen) {

if self.peek_token_ref().token == Token::LParen {

src/parser/mod.rs

src/dialect/mod.rs

ryanschneider · 2025-07-17T19:51:57Z

Thanks for the latest round of feedback @iffyio, I've been busy with some unrelated work so haven't had time to look at it all yet or start on the changes but hopefully will have some space cycles the next couple days, I'll re-request review when it's ready!

Co-authored-by: Ifeanyi Ubah <ify1992@yahoo.com>

- use `concat!` for longer SQL strings - replace macro w/ `parse_column_option_expr()` - add SQL examples to docstrings - use `peek_token_ref()` instead of `matches!` - `in_normal_state` -> `pub(crate) in_column_definition_state` - revert fmt change that moved use statements - add `GENERATED` test and use [ParserState::Normal] for parsing inner expr.

ryanschneider · 2025-07-17T22:53:17Z

Ok @iffyio I believe I addressed all your latest feedback in 1466e2a, let me know! Thanks!

iffyio

Thanks @ryanschneider! Could we do a pass through the options mentioned here and here to extend them with similar support for the NOT NULL clause? Beyond that I think the changes look good! (Also headsup that there's currently a docs failure in ci on the branch)

iffyio · 2025-07-18T13:56:43Z

tests/sqlparser_common.rs

+#[test]
+fn parse_notnull_unsupported() {
+    // Only Postgres, DuckDB, and SQLite support `x NOTNULL` as an expression
+    // All other dialects consider `x NOTNULL` like `x AS NOTNULL` and thus
+    // consider `NOTNULL` an alias for x.
+    let dialects = all_dialects_except(|d| d.supports_notnull_operator());
+    let _ = dialects
+        .verified_only_select_with_canonical("SELECT NULL NOTNULL", "SELECT NULL AS NOTNULL");
+}
+
+#[test]
+fn parse_notnull_supported() {
+    // Postgres, DuckDB and SQLite support `x NOTNULL` as an alias for `x IS NOT NULL`
+    let dialects = all_dialects_where(|d| d.supports_notnull_operator());
+    let _ = dialects.expr_parses_to("x NOTNULL", "x IS NOT NULL");
+}
+
+#[test]
+fn test_notnull_precedence() {
+    // For dialects which support it, `NOT NULL NOTNULL` should
+    // parse as `(NOT (NULL IS NOT NULL))`
+    let supported_dialects = all_dialects_where(|d| d.supports_notnull_operator());
+    let unsupported_dialects = all_dialects_except(|d| d.supports_notnull_operator());
+
+    assert_matches!(
+        supported_dialects.expr_parses_to("NOT NULL NOTNULL", "NOT NULL IS NOT NULL"),
+        Expr::UnaryOp {
+            op: UnaryOperator::Not,
+            ..
+        }
+    );
+
+    // for unsupported dialects, parsing should stop at `NOT NULL`
+    unsupported_dialects.expr_parses_to("NOT NULL NOTNULL", "NOT NULL");
+}


Since they're the same feature, can we merge these scenarios into the same fn parse_notnull(){} test function?.

Also for the comments, ideally we skip mentioning the individual dialects (Postgres, duckdb etc), so that the comment doesnt' become stale as other/new dialects are extended with support for this feature.

iffyio · 2025-07-18T13:57:29Z

tests/sqlparser_common.rs

+#[test]
+fn parse_not_null_supported() {
+    let _ = all_dialects().expr_parses_to("x NOT NULL", "x IS NOT NULL");
+    let _ = all_dialects().expr_parses_to("NULL NOT NULL", "NULL IS NOT NULL");
+}
+
+#[test]
+fn test_not_null_precedence() {
+    assert_matches!(
+        all_dialects().expr_parses_to("NOT NULL NOT NULL", "NOT NULL IS NOT NULL"),
+        Expr::UnaryOp {
+            op: UnaryOperator::Not,
+            ..
+        }
+    );
+    assert_matches!(
+        all_dialects().expr_parses_to("NOT x NOT NULL", "NOT x IS NOT NULL"),
+        Expr::UnaryOp {
+            op: UnaryOperator::Not,
+            ..
+        }
+    );
+}


Similar to the notnull scenarios, can we merge these into the same function?

ryanschneider · 2025-07-18T15:27:46Z

Sorry I should've mentioned that I did check the other ColumnOptions and confirmed that they don't need to use parse_column_option_expr but after thinking about it more I'll go ahead and use it anyways for clarity. Also fixing the doc failure and consolidating the tests.

ryanschneider · 2025-07-18T15:50:03Z

Ok, I consolidated the tests, fixed the cargo doc error and use parse_column_option_expr in all spots for consistency.

ryanschneider commented Jul 6, 2025

View reviewed changes

src/dialect/duckdb.rs Outdated Show resolved Hide resolved

ryanschneider commented Jul 6, 2025

View reviewed changes

src/parser/mod.rs Outdated Show resolved Hide resolved

iffyio reviewed Jul 8, 2025

View reviewed changes

src/dialect/mod.rs Outdated Show resolved Hide resolved

src/dialect/mod.rs Outdated Show resolved Hide resolved

tests/sqlparser_common.rs Outdated Show resolved Hide resolved

ryanschneider requested a review from iffyio July 8, 2025 20:33

ryanschneider added 6 commits July 10, 2025 17:18

DuckDB, Postgres, SQLite: NOT NULL and NOTNULL expressions

3b5cdce

fixup: change to supports_is_not_null_alias(IsNotNullAlias)

fe51680

fixup: make NOTNULL/NOT NULL an alias for Expr::IsNotNull

ab5c571

fixup: simplify tests

75ba821

fixup: add precedence tests as well

4009fa7

fixup: handle expr NOT NULL directly in parse_subexpr.

ab6607b

ryanschneider force-pushed the not-null-and-notnull-support branch from 307c465 to ab6607b Compare July 11, 2025 00:21

ryanschneider added 3 commits July 11, 2025 11:56

fixup: Add ParserState::ColumnDefinition to avoid parsing NOT NULL as…

375fe8c

… Expr::IsNotNull.

Merge branch 'main' into not-null-and-notnull-support

40abd90

fixup: fmt

45fff12

ryanschneider added 3 commits July 14, 2025 12:27

Return to ParserState::Normal when parsing DEFAULT sub-expression.

bf25f42

Handle NOT NULL in MATERIALIZED and CHECK options as well.

bc454c6

Merge branch 'main' into not-null-and-notnull-support

b7d4e1e

iffyio reviewed Jul 16, 2025

View reviewed changes

ryanschneider and others added 3 commits July 17, 2025 14:44

Update tests/sqlparser_clickhouse.rs

7371bff

Co-authored-by: Ifeanyi Ubah <ify1992@yahoo.com>

Merge branch 'main' into not-null-and-notnull-support

0c56a51

ryanschneider requested a review from iffyio July 17, 2025 22:53

iffyio reviewed Jul 18, 2025

View reviewed changes

fixup: address next round of feedback

8b376b8

ryanschneider requested a review from iffyio July 18, 2025 15:50

	if matches!(self.peek_token().token, Token::LParen) {
	if self.peek_token_ref().token == Token::LParen {

DuckDB, Postgres, SQLite: NOT NULL and NOTNULL expressions #1927

Are you sure you want to change the base?

DuckDB, Postgres, SQLite: NOT NULL and NOTNULL expressions #1927

Conversation

ryanschneider commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanschneider commented Jul 11, 2025

Uh oh!

ryanschneider commented Jul 11, 2025

Uh oh!

iffyio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iffyio Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanschneider commented Jul 17, 2025

Uh oh!

ryanschneider commented Jul 17, 2025

Uh oh!

iffyio left a comment

Choose a reason for hiding this comment

Uh oh!

iffyio Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

iffyio Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

ryanschneider commented Jul 18, 2025

Uh oh!

ryanschneider commented Jul 18, 2025

Uh oh!

Uh oh!

ryanschneider commented Jul 5, 2025 •

edited

Loading