[PostgreSQL] Extend the allowed Debezium properties #1247
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,6 +13,7 @@ | |
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| import ballerinax/cdc; | ||
|
|
||
| # Represents the PostgreSQL logical decoding plugins. | ||
|
|
@@ -24,18 +25,72 @@ public enum PostgreSQLLogicalDecodingPlugin { | |
| DECODERBUFS = "decoderbufs" | ||
| } | ||
|
|
||
| # Represents publication autocreate modes. | ||
| public enum PublicationAutocreateMode { | ||
| ALL_TABLES = "all_tables", | ||
| DISABLED = "disabled", | ||
| FILTERED = "filtered" | ||
| } | ||
|
|
||
| # Represents LSN flush modes. | ||
| public enum LsnFlushMode { | ||
| MANUAL = "manual", | ||
| CONNECTOR = "connector", | ||
| CONNECTOR_AND_DRIVER = "connector_and_driver" | ||
| } | ||
|
Comment on lines
+35
to
+40
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== Candidate manifest files =="
fd '^(Ballerina\.toml|Dependencies\.toml|Package\.md|README\.md)$' -x sh -c '
printf "\n--- %s ---\n" "$1"
sed -n "1,220p" "$1"
' sh {}
echo
echo "== Debezium / CDC driver references =="
rg -n 'postgresql\.cdc\.driver|debezium-connector-postgres|Debezium Postgres Driver|lsn\.flush\.mode|connector_and_driver' .Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 29179 🏁 Script executed: # Check the full postgresql.cdc.driver version in Dependencies.toml
cat ballerina/Dependencies.toml | grep -A 10 'name = "postgresql.cdc.driver"'Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 583 🏁 Script executed: # Check build.gradle for CDC driver version
grep -n 'stdlibPostgresCdcDriverVersion\|postgresql.*cdc.*driver' build.gradle | head -20Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 204 🏁 Script executed: # Check if there's a gradle.properties or similar with version definitions
fd '(gradle\.properties|version\.txt|versions\.gradle)$' -x cat {}Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 1257 🏁 Script executed: # Search for any Debezium version references in the native build or pom files
fd '(pom\.xml|build\.gradle)' -x grep -l 'debezium\|cdc.*driver' {} 2>/dev/null | head -5Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 105 🏁 Script executed: # Find and examine native build files for Debezium/CDC driver dependencies
find ./native -name 'pom.xml' -o -name 'build.gradle' 2>/dev/null | head -5Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 112 🏁 Script executed: # Check for any native module files that might reference Debezium version
fd 'pom\.xml' ./native -x grep -l 'debezium' {} 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 73 🏁 Script executed: # Search for any version or release documentation
fd '(CHANGELOG|RELEASE|NEWS|VERSION)' -i | grep -i 'cdc\|debezium' | head -10Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 73 🏁 Script executed: # Check native build gradle files for CDC driver or Debezium references
find ./native -name '*.gradle' -x grep -n 'debezium\|cdc.*driver\|postgres.*connector' {} 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 73 🏁 Script executed: # Examine native/build.gradle for CDC driver dependencies
cat ./native/build.gradle | grep -A 5 -B 5 'cdc\|debezium\|postgres' -iRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 934 🏁 Script executed: # Check if there's a separate CDC driver repository or if it's defined in build
rg 'postgresql\.cdc\.driver|Debezium' ballerina/Dependencies.toml -A 5 -B 2Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 705 🏁 Script executed: # Search for any gradle property or variable that tracks CDC driver bundled Debezium version
rg 'debezium.*version|cdc.*version' -i gradle.properties build.gradleRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 400 🏁 Script executed: # Check if this PR changes any CDC driver version by looking for gradle version assignments
grep -r 'stdlibPostgresCdcDriverVersion\|postgresql.cdc.driver' gradle.properties build.gradle native/build.gradle 2>/dev/nullRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 268 🏁 Script executed: # Search for any Debezium or version configuration in tests
rg 'debezium|3\.0\.|3\.4\.' -i ballerina/tests listener_tests.bal | head -20Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 149 🏁 Script executed: # Check README or documentation for Debezium version requirements
cat ballerina/README.md | grep -A 5 -B 5 'debezium\|cdc.*driver\|3\.0\|3\.4' -iRepository: ballerina-platform/module-ballerinax-postgresql Length of output: 999 🏁 Script executed: # Search for any version constraints or compatibility notes
rg 'version.*3\.' ballerina/ -A 2 -B 2Repository: ballerina-platform/module-ballerinax-postgresql Length of output: 336 🌐 Web query:
💡 Result: ballerinax/postgresql.cdc.driver version 1.0.0 is a Ballerina library that provides the Debezium PostgreSQL connector dependencies for Change Data Capture (CDC) functionality in Ballerina applications. It enables seamless listening to changes (inserts, updates, deletes) in PostgreSQL databases. Key details: - Released on 2025-05-29 (latest and only release as of 2026-03-19). - Compatible with Ballerina Language version 2201.12.0. - Bundles Debezium Postgres Driver version 3.0.8.Final. - Usage: Import alongside ballerinax/cdc and ballerinax/postgresql modules: import ballerinax/cdc; import ballerinax/postgresql; import ballerinax/postgresql.cdc.driver as _; Then configure a postgresql:CdcListener to capture events with remote functions like onCreate, onUpdate, onDelete. It is a support module for Ballerina's CDC framework, which is built on Debezium. No additional driver configuration needed as it's pre-bundled. Available on Ballerina Central: https://central.ballerina.io/ballerinax/postgresql.cdc.driver/latest. Source: https://github.com/ballerina-platform/module-ballerinax-postgresql.cdc.driver. Citations:
Remove The enum value Also applies to: lines 69-72 🤖 Prompt for AI Agents |
||
|
|
||
| # PostgreSQL replication configuration (logical decoding). | ||
| # | ||
| # + pluginName - Logical decoding plugin to use (pgoutput, decoderbufs) | ||
| # + slotName - Name of the PostgreSQL logical replication slot | ||
| # + slotDropOnStop - Drop replication slot when connector stops | ||
| # + slotStreamParams - Custom replication slot parameters | ||
| public type ReplicationConfiguration record {| | ||
| PostgreSQLLogicalDecodingPlugin pluginName = PGOUTPUT; | ||
| string slotName = "debezium"; | ||
| boolean slotDropOnStop = false; | ||
| string slotStreamParams?; | ||
| |}; | ||
|
|
||
| # PostgreSQL publication configuration (pgoutput plugin). | ||
| # | ||
| # + publicationName - Name of PostgreSQL publication | ||
| # + publicationAutocreateMode - Mode for auto-creating publications | ||
| public type PublicationConfiguration record {| | ||
| string publicationName = "dbz_publication"; | ||
| PublicationAutocreateMode publicationAutocreateMode = ALL_TABLES; | ||
| |}; | ||
|
Comment on lines
+48
to
+62
There was a problem hiding this comment. Reject mixed deprecated and nested configs, or merge them field-by-field.
Also applies to: 108-119 🤖 Prompt for AI Agents |
||
|
|
||
| # PostgreSQL streaming and status configuration. | ||
| # | ||
| # + statusUpdateInterval - Interval for sending status updates to PostgreSQL in seconds | ||
| # + xminFetchInterval - Interval for fetching current xmin position in seconds | ||
| # + lsnFlushMode - LSN flushing strategy | ||
| public type StreamingConfiguration record {| | ||
| decimal statusUpdateInterval = 10; | ||
| decimal xminFetchInterval = 0; | ||
| LsnFlushMode lsnFlushMode?; | ||
| |}; | ||
|
|
||
| # Represents the configuration for the Postgres CDC database connection. | ||
| # | ||
| # + connectorClass - The class name of the PostgreSQL connector implementation to use | ||
| # + hostname - The hostname of the PostgreSQL server | ||
| # + port - The port number of the PostgreSQL server | ||
| # + databaseName - The name of the PostgreSQL database from which to stream the changes. | ||
| # + includedSchemas - A list of regular expressions matching fully-qualified schema identifiers to capture changes from | ||
| # + excludedSchemas - A list of regular expressions matching fully-qualified schema identifiers to exclude from change capture | ||
| # + includedTables - Regex patterns for tables to capture (mutually exclusive with `excludedTables`) | ||
| # + excludedTables - Regex patterns for tables to exclude (mutually exclusive with `includedTables`) | ||
| # + includedColumns - Regex patterns for columns to capture (mutually exclusive with `excludedColumns`) | ||
| # + excludedColumns - Regex patterns for columns to exclude (mutually exclusive with `includedColumns`) | ||
| # + messageKeyColumns - Composite message key columns for change events | ||
| # + tasksMax - The PostgreSQL connector always uses a single task and therefore does not use this value, so the default is always acceptable | ||
| # + pluginName - Deprecated: Use `replicationConfig.pluginName` instead | ||
| # + slotName - Deprecated: Use `replicationConfig.slotName` instead | ||
| # + publicationName - Deprecated: Use `publicationConfig.publicationName` instead | ||
| # + replicationConfig - Replication configuration (logical decoding plugin, slot name and parameters). Takes priority over deprecated top-level fields | ||
| # + publicationConfig - Publication configuration (publication name and autocreate mode). Takes priority over deprecated top-level fields | ||
| public type PostgresDatabaseConnection record {| | ||
|
Comment on lines
+92
to
94
There was a problem hiding this comment. Document Line 119 adds a new public field, but the Also applies to: 117-119 🤖 Prompt for AI Agents |
||
| *cdc:DatabaseConnection; | ||
| string connectorClass = "io.debezium.connector.postgresql.PostgresConnector"; | ||
|
|
@@ -44,16 +99,54 @@ public type PostgresDatabaseConnection record {| | |
| string databaseName; | ||
| string|string[] includedSchemas?; | ||
| string|string[] excludedSchemas?; | ||
| string|string[] includedTables?; | ||
| string|string[] excludedTables?; | ||
| string|string[] includedColumns?; | ||
| string|string[] excludedColumns?; | ||
| cdc:MessageKeyColumns[] messageKeyColumns?; | ||
| int tasksMax = 1; | ||
| # Deprecated: Use `replicationConfig.pluginName` instead. | ||
| @deprecated | ||
| PostgreSQLLogicalDecodingPlugin pluginName = PGOUTPUT; | ||
| # Deprecated: Use `replicationConfig.slotName` instead. | ||
| @deprecated | ||
| string slotName = "debezium"; | ||
| # Deprecated: Use `publicationConfig.publicationName` instead. | ||
| @deprecated | ||
| string publicationName = "dbz_publication"; | ||
| ReplicationConfiguration replicationConfig?; | ||
| PublicationConfiguration publicationConfig?; | ||
| StreamingConfiguration streamingConfig?; | ||
| |}; | ||
|
|
||
| # PostgreSQL CDC listener configuration including database connection, storage, and CDC options. | ||
| # | ||
| # + database - PostgreSQL database connection, logical decoding, and capture settings | ||
| # + options - PostgreSQL-specific CDC options including snapshot, heartbeat, signals, and data type handling | ||
| public type PostgresListenerConfiguration record {| | ||
| PostgresDatabaseConnection database; | ||
| *cdc:ListenerConfiguration; | ||
| PostgreSqlOptions options = {}; | ||
| |}; | ||
|
|
||
| # PostgreSQL-specific CDC options for configuring snapshot behavior and data type handling. | ||
| # | ||
| # + extendedSnapshot - Extended snapshot configuration with PostgreSQL-specific lock timeout and query settings | ||
| # + dataTypeConfig - Data type handling configuration including schema change tracking | ||
| # + heartbeatConfig - Heartbeat configuration for keeping the PostgreSQL replication slot active | ||
| public type PostgreSqlOptions record {| | ||
| *cdc:Options; | ||
| ExtendedSnapshotConfiguration extendedSnapshot?; | ||
| cdc:DataTypeConfiguration dataTypeConfig?; | ||
| cdc:RelationalHeartbeatConfiguration heartbeatConfig?; | ||
| |}; | ||
|
|
||
| # Represents the extended snapshot configuration for the PostgreSQL CDC listener. | ||
| # | ||
| # + lockTimeout - Lock acquisition timeout in seconds | ||
| # + isolationMode - Transaction isolation level during snapshot | ||
| public type ExtendedSnapshotConfiguration record {| | ||
| *cdc:RelationalExtendedSnapshotConfiguration; | ||
| decimal lockTimeout = 10; | ||
| cdc:SnapshotIsolationMode isolationMode?; | ||
| |}; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 1719
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 261
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 1680
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 261
🌐 Web query:
Debezium PostgreSQL 3.0.8 publication.autocreate.mode no_tables💡 Result:
In Debezium PostgreSQL connector version 3.0.8, the publication.autocreate.mode configuration property controls how the connector handles creation of the PostgreSQL publication used for logical replication with the pgoutput plugin. The no_tables mode creates (or uses an existing) publication with no tables assigned to it initially. The user must then manually add tables to the publication using SQL commands like ALTER PUBLICATION mypub ADD TABLE myschema.mytable;. This is useful when the connector lacks privileges to auto-add tables or when fine-grained control over publication tables is needed. This option was added via DBZ-8156 and documented in a merged pull request in August 2024, present in Debezium 3.0.8 (released March 2025). Other common modes include: - disabled: No auto-creation; publication must pre-exist. - all_tables: Creates publication for all tables (default in older versions). - filtered: Creates publication based on table.include.list/table.exclude.list filters. Configure as: "publication.autocreate.mode": "no_tables"
Citations:
no_tablesconfig option for `publication.auto… debezium/debezium#5780🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 688
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 2881
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 2803
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 3615
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 186
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 830
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
🌐 Web query:
ballerinax postgresql.cdc.driver 1.0.0 Debezium version💡 Result:
The ballerinax/postgresql.cdc.driver version 1.0.0 bundles Debezium Postgres Driver version 3.0.8.Final.
Citations:
🏁 Script executed:
Repository: ballerina-platform/module-ballerinax-postgresql
Length of output: 73
Add the missing
NO_TABLESenum member toPublicationAutocreateMode.The currently published
ballerinax/postgresql.cdc.driver1.0.0 bundles Debezium Postgres Driver 3.0.8.Final, which supportspublication.autocreate.mode=no_tables. Omitting it from the enum blocks users from configuring this valid mode through the public API.Fix
public enum PublicationAutocreateMode { ALL_TABLES = "all_tables", DISABLED = "disabled", - FILTERED = "filtered" + FILTERED = "filtered", + NO_TABLES = "no_tables" }📝 Committable suggestion
🤖 Prompt for AI Agents