Skip to content

fix: pcc sync - multiple segments have the same sourceId causing errors#4046

Merged
themarolt merged 13 commits intomainfrom
fix/pcc
Apr 22, 2026
Merged

fix: pcc sync - multiple segments have the same sourceId causing errors#4046
themarolt merged 13 commits intomainfrom
fix/pcc

Conversation

@themarolt
Copy link
Copy Markdown
Contributor

@themarolt themarolt commented Apr 22, 2026

Note

Medium Risk
Changes matching and write-path behavior in the PCC sync worker and alters how job failures are persisted, which could affect what gets skipped vs upserted and how failures are diagnosed.

Overview
Prevents PCC sync crashes/incorrect writes when multiple CDP subproject segments share the same sourceId: findSegmentBySourceId now detects ambiguity (returns up to 2 candidates), the consumer logs/records an AMBIGUOUS_SEGMENT_MATCH sync error, and the row is skipped; schema-mismatch diagnostics also capture ambiguity details.

Hardens insightsProjects upserts against global unique-name collisions by splitting UPDATE vs INSERT checks, adding broader conflict guards (including NULL segmentId rows), and treating Postgres duplicate-key errors as a non-fatal “name conflict” outcome.

Improves Snowflake job failure reporting by passing the original thrown value into MetadataStore.markFailed, serializing rich error details (message/name/stack/cause/extra fields), and (for PCC) persisting transformed/skipped metrics on failure.

Reviewed by Cursor Bugbot for commit 229486c. Bugbot is set up for automated code reviews on this repo. Configure here.

…be detected later in the code

Signed-off-by: Uroš Marolt <uros@marolt.me>
…be detected later in the code

Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Uroš Marolt <uros@marolt.me>
Copilot AI review requested due to automatic review settings April 22, 2026 06:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts PCC sync behavior in pcc_sync_worker to avoid Postgres unique-name violations when multiple CDP segments share the same PCC sourceId, by narrowing which insightsProjects rows get updated and by proactively detecting name collisions.

Changes:

  • Reworks the insightsProjects name-conflict pre-check to align with the global partial unique index on (name) WHERE deletedAt IS NULL.
  • Scopes the insightsProjects UPDATE to the matched subproject segmentId instead of bulk-updating all segments sharing sourceId.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts Outdated
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Signed-off-by: Uroš Marolt <uros@marolt.me>
Copilot AI review requested due to automatic review settings April 22, 2026 07:39
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts Outdated
Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts Outdated
Signed-off-by: Uroš Marolt <uros@marolt.me>
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copilot AI review requested due to automatic review settings April 22, 2026 09:33
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts Outdated
Comment thread services/libs/snowflake/src/metadataStore.ts
Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts Outdated
Comment thread services/apps/pcc_sync_worker/src/consumer/pccProjectConsumer.ts
Signed-off-by: Uroš Marolt <uros@marolt.me>
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@themarolt themarolt requested a review from mbani01 April 22, 2026 10:10
@themarolt themarolt merged commit f7f9693 into main Apr 22, 2026
15 checks passed
@themarolt themarolt deleted the fix/pcc branch April 22, 2026 12:27
skwowet pushed a commit that referenced this pull request Apr 28, 2026
…rs (#4046)

Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants