Pre-hardening IOC identity writes before Event Injection (follow-up to Discussion #1236)

[tanmayjoddar]

Following the discussion in #1236, I am opening this issue as an implementation tracker.

Context

• Current deployment is mostly single-worker:
  • greedybear/settings.py#L213 (Q_CLUSTER["workers"] = 1)
  • docker/default.yml#L69 (single qcluster service)
• So current practical severity is relatively low.
• However, this is a good point to harden IOC identity handling before Event Injection / multi-source ingestion.

Scope of this issue

1. Add DB-level uniqueness for IOC identity (preferred key: ("name", "type")).
2. Add migration/merge path for pre-existing duplicate IOC rows (if any).
3. Make IOC write path concurrency-safe (e.g., atomic upsert or transaction-safe handling).
4. Add a regression test covering concurrent writes for same IOC identity.

Why now

This keeps ingestion behavior stable as concurrency grows, without over-rotating severity for current single-worker topology.

Acceptance criteria

• IOC identity uniqueness is enforced at DB level.
• Duplicate IOC rows are merged safely during migration.
• Concurrent writes do not create duplicate IOC identity rows.
• Regression test added for concurrent same-key writes.

Related references

• Discussion: #1236
• Baseline refactor: PR #624
• Nearby correctness fixes: #967, #974, #1007, #1010
• Nearby performance fixes: #1012, #1019, #1024

[tanmayjoddar]

Hey @regulartim , following up on the discussion (#1236), I’ve opened this issue to track the proposed hardening steps.
Let me know if the scope looks good or if you’d like any adjustments — happy to start working on this.

[regulartim]

Hey @tanmayjoddar ! :) I am wondering if it would be a good idea to solve this in 2 separate PRs: uniqueness and data migration first, then update write path. Tests go along with the functionality they test. What do you think?

[tanmayjoddar]

Thanks @regulartim — I like this split, it keeps the changes focused and easier to review.

It also maps well to what I had in mind (originally 4 phases), just grouped into 2 PRs:

PR 1: identity + migration

• enforce DB-level uniqueness on IOC identity (name, type)
• add a migration to detect and merge any existing duplicates (handling counters, timestamps, and M2M relations)
• include tests for constraint enforcement and migration correctness

PR 2: write-path hardening

• update the IOC write path to be concurrency-safe (transaction-safe handling / upsert pattern)
• add regression tests for concurrent same-key writes

This way we first ensure the data state is canonical, and then harden the runtime write behavior on top of it.

If this looks good to you, I’ll start with PR 1.

[regulartim]

Yes, looks good! :)