Add `data_source` field to IOC model for multi-source tracking

[manik3160]

Description

While reviewing the database schema in greedybear/models.py, I noticed that the IOC model currently lacks a generic way to track the provenance of the intelligence it stores.

Right now, an IOC's origins are mapped using the sensors and general_honeypot ManyToMany fields:

# greedybear/models.py
class IOC(models.Model):
    # ...
    # FEEDS - list of honeypots from general list, from which the IOC was detected
    general_honeypot = models.ManyToManyField(GeneralHoneypot, blank=True)

    # SENSORS - list of T-Pot sensors that detected this IOC
    sensors = models.ManyToManyField(Sensor, blank=True)

However, because GreedyBear currently pulls almost exclusively from T-Pot Elasticsearch, there is no field to specify the high-level data source (e.g., T-Pot, an external API, a manual submission, etc.).

As we expand GreedyBear to accept data from external applications via the planned Event Collector API, we will need a reliable way to differentiate these sources at the database level.

Impact

Without a generic data source tracking field:

• We cannot easily filter or serve feeds based on specific upstream providers.
• It becomes difficult to audit data quality, reliability, or noise levels per source over time.
• The Event Collector API will struggle to label incoming externally-injected events accurately without polluting existing T-Pot-specific fields like sensors.

Proposed Solution

Add a data_source field to the IOC model to track where the data originated.

This could be implemented either as a simple CharField with choices, or ideally as a ForeignKey to a new DataSource model to allow dynamic addition of new integrations:

# greedybear/models.py
class DataSource(models.Model):
    name = models.CharField(max_length=64, unique=True)
    description = models.TextField(blank=True)
    active = models.BooleanField(default=True)

    def __str__(self):
        return self.name

class IOC(models.Model):
    # ... existing fields
    data_source = models.ForeignKey(
        DataSource,
        on_delete=models.SET_NULL,
        null=True,
        blank=True
    )
    # ...

This would immediately lay the groundwork for the multi-source ingestion required by the Event Collector API.

Happy to work on a PR for this if you think the approach makes sense!

[manik3160]

Hey @regulartim, if you think this addition would be useful at this stage, I’d be happy to work on the implementation and open a PR.

[regulartim]

Hey @manik3160 ! I think this is definitely something that will be necessary for the Event Collector API, but I would keep this as a part of the GSoC project itself. I would like leave the decision how to implement this requirement to the future GSoC participant.

[manik3160]

Makes total sense @regulartim! I agree it’s best kept within the GSoC scope. I'll make sure to detail the implementation approach in my proposal :)

[MohidNaghman1]

Hi @regulartim @mlodic,

I'm preparing my GSoC proposal for the Event Collector API (#6) and came across this issue. I noticed your recent comment that this should be part of the GSoC project scope—which makes total sense for establishing proper data provenance from day one.

I've reviewed the current IOC model and general_honeypot/sensors fields. Before I finalize my proposal, I'd like to discuss implementation approaches:

Option A: ForeignKey to DataSource Model (Recommended)

class DataSource(models.Model):
    name = models.CharField(max_length=64, unique=True)
    description = models.TextField(blank=True)
    active = models.BooleanField(default=True)
    created_at = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return self.name

class IOC(models.Model):
    # ... existing fields
    data_source = models.ForeignKey(
        DataSource,
        on_delete=models.SET_NULL,
        null=True,
        blank=True,
        related_name='iocs'
    )

Pros: Extensible for future integrations, audit trail, allows metadata per source
Cons: Extra DB query (mitigated with select_related)

Option B: CharField with Choices

class IOC(models.Model):
    DATA_SOURCE_CHOICES = [
        ('tpot', 'T-POT Instance'),
        ('external_api', 'External API'),
        ('manual', 'Manual Submission'),
    ]
    data_source = models.CharField(
        max_length=50,
        choices=DATA_SOURCE_CHOICES,
        null=True,
        blank=True
    )

Pros: Lightweight, no extra queries
Cons: Hardcoded choices, requires code changes to add new sources

My Preference:

Option A (DataSource ForeignKey) because:

• GreedyBear will likely accept data from multiple external sources over time
• Needs audit trail (which source is most reliable?)
• Aligns with your multi-honeypot architecture philosophy
• Event Collector API can register sources dynamically

Proposed Implementation Scope (for GSoC):

• Create DataSource model
• Add data_source ForeignKey to IOC
• Migration to populate existing T-POT IOCs with data_source="T-POT"
• Update IOCSerializer to include data_source
• Add API filtering: GET /api/iocs/?data_source=external_api
• Comprehensive pytest coverage (90%+)
• Documentation for Event Collector API integration

Background:

I have production experience building event-driven systems (AETERNA) with PostgreSQL, async processing, and API design. While my background is primarily FastAPI + SQLAlchemy, I'm comfortable transferring to Django ORM for this project. Database design patterns and relationships (one-to-many, filtering, eager loading) are consistent across ORMs—my proposal will detail the implementation timeline and testing strategy.

Which approach resonates with your vision for the Event Collector API?

Thanks!

[vishwakarmameenal105-a11y]

Hi, i would like to work on this? Can i get this assigned to me ?

[regulartim]

Hey @vishwakarmameenal105-a11y ! As I already said:

I think this is definitely something that will be necessary for the Event Collector API, but I would keep this as a part of the GSoC project itself. I would like leave the decision how to implement this requirement to the future GSoC participant.