Add Audit Trail Export for Compliance Documentation (SAR Filing, Regulatory Exams)

[matheusybsen]

Problem:
When financial institutions use yente for sanctions/PEP screening, they need to document screening decisions for regulatory compliance, but yente currently doesn't provide audit trail export functionality for screening operations.

Compliance teams need to prove to regulators:

• Which version of sanctions data was used for each screening
• What entities were screened and when
• What matches were found (including score and match logic)
• Who performed the screening and what decision was made

Without audit trails, institutions cannot:

• Complete SAR filings with required screening documentation
• Pass regulatory exams (OCC, FDIC, FinCEN auditors request screening logs)
• Respond to legal discovery requests
• Demonstrate compliance program effectiveness

Solution:
Add audit trail export functionality to yente:

1. Screening Log Capture:

   • Log all /match and /search API calls
   • Store: timestamp, query parameters, matched entities, scores, dataset versions
   • Include user/system identifier (if provided in request headers)

2. Export Endpoint:

   • New endpoint: GET /audit/export
   • Query parameters: date range, entity type, dataset filter
   • Output formats: CSV (for Excel analysis), JSON (for downstream systems), PDF (for regulatory submission)

3. Compliance Report Fields:

   • Screening timestamp
   • Query details (name, DOB, country, etc.)
   • Dataset version used (e.g., "OpenSanctions 2026-02-14")
   • Match results: entity ID, name, sanctions program, match score, match explanation (from logic-v2)
   • Decision outcome (if provided by caller via custom field)

Regulatory Drivers:

• FinCEN SAR Requirements: Banks must document "description of suspicious activity" including screening results
• OCC Heightened Standards (31 CFR 1010.610): Requires "recordkeeping and reporting" of sanctions screening
• OFAC 50% Rule Compliance: Must document screening logic for ownership structures
• EU 6AMLD: Requires audit trails of AML controls

Use Cases:

• SAR Filing: Export screening evidence showing entity matched OFAC SDN list
• Regulatory Exam: Provide auditors with 90-day screening log showing coverage
• Internal Audit: Quarterly review of screening decisions and false positive rates
• Legal Discovery: Respond to subpoenas requesting screening records

Implementation Notes:

• Could leverage existing ElasticSearch infrastructure (yente already has audit logs for index operations per v5.1.0)
• Privacy consideration: Allow filtering to exclude PII in exported logs (GDPR compliance)
• Performance: Async export for large date ranges (return job ID, poll for completion)

Alternatives Considered:

• Application-level logging: Capture API responses in calling application - but doesn't include dataset version metadata
• Database query logs: Use ElasticSearch query logs - but lacks business context and user decisions
• Manual documentation: Screenshot each match - not scalable, error-prone

I can help with:

• Defining regulatory documentation requirements from FinCEN/OFAC/OCC guidance
• Examples of compliant audit trail formats for SAR filing
• Testing export functionality against real compliance workflows

[pudo]

Hey @matheusybsen! Thanks for sharing that idea! As you may have seen, yente 5.0 and newer already keeps an audit log ES index in which some of the things you're describing are being recorded - we just don't expose it as an API endpoint yet.

To my mind, there's two parts to this: a) documenting the state of the system at the time at which some screening occurred (eg. which data version was being used), and b) documenting the screening query and it's result. So far, we're only planning to use the audit log for the first part: make a record of when which version of the data was live, and if any errors had occurred related to data updates.

The nasty bit about b) (recording the screening payload and results) is that it fundamentally alters what yente is/does from a data privacy point of view. Rather than being a transient processor of counterparty data, it would make the tool into a store of customer PII - which is highly regulated information in the case of financial companies. So: we most likely want to be able to turn this off in our hosted API because otherwise there will be European lawyers trying to kill me on my way home.

Doesn't mean that on-prem yente cannot do this. It also would be interesting from the point of view of doing watchlist monitoring: when you store the previously screened entities, you can then cross-reference them with incoming deltas to provide alerts when a non-matching candidate suddenly picks up a risk.

But then I guess we at least need to document backup strategies, time-based deletion, and a few other data handling patterns like that.

[matheusybsen]

Thanks for the detailed response!

Re: Audit log for data versioning
That makes perfect sense. Documenting which data version was live during screening (part a) solves the core regulatory requirement. Auditors need to know "we screened against OFAC SDN list version X on date Y" and your existing audit log handles that.

Re: Privacy concerns with query/result logging
Completely understand the GDPR/privacy issue. You're right that storing screening payloads transforms yente from a transient processor to a data controller, which changes the compliance posture entirely.

Suggested approach:
Given the privacy concerns, would it make sense to:

1. Document the existing audit log for regulatory use cases (showing how to prove data version/uptime for compliance)
2. Make query/result logging opt-in with clear warnings about data controller responsibilities
3. Add retention policies (auto-delete after 90 days, configurable) to minimize PII storage
4. Provide anonymization options (hash entity names, redact DOB, keep only match metadata)

For on-prem deployments where institutions manage their own PII governance, this would be valuable. The watchlist monitoring use case you mentioned (alerting when previously screened entities gain new sanctions) is exactly what banks need for ongoing customer screening.

Implementation thought:
Could the audit log capture just:

• Timestamp
• Dataset version
• Match occurred: yes/no
• Entity ID matched (OpenSanctions ID, not customer PII)
• Match score

This would document "we screened someone and found a match to entity XYZ123 in dataset version 2026-02-14" without storing customer PII. Institutions could correlate this with their own systems.

Re: Hosted vs on-prem
Totally makes sense to keep this off in your hosted API. For on-prem users who accept data controller obligations, documenting backup/retention strategies would be valuable.

Happy to help document compliance use cases for the existing audit log if that would be useful.