mastr: duplicate key violation for EinheitMastrNummer in changed_dso_assignment table

[maurerle]

Description of the issue

I got a lot of errors in todays export: Gesamtdatenexport_20260225.zip

2026-02-25 08:00:36,412 [ERROR] Error processing file 'EinheitenAenderungNetzbetreiberzuordnungen_1.xml': 'Execution failed on sql 'INSERT INTO changed_dso_assignment ("EinheitMastrNummer
", "LokationMastrNummer", "NetzbetreiberMastrNummerAlt", "NetzbetreiberMastrNummerNeu", "ArtDerAenderung", "RegistrierungsdatumNetzbetreiberzuordnungsaenderung", "Netzbetreiberzuordnungsa
enderungsdatum", "NetzanschlusspunktMastrNummer", "DatenQuelle", "DatumDownload") VALUES (:EinheitMastrNummer, :LokationMastrNummer, :NetzbetreiberMastrNummerAlt, :NetzbetreiberMastrNumme
rNeu, :ArtDerAenderung, :RegistrierungsdatumNetzbetreiberzuordnungsaenderung, :Netzbetreiberzuordnungsaenderungsdatum, :NetzanschlusspunktMastrNummer, :DatenQuelle, :DatumDownload)': (psy
copg2.errors.UniqueViolation) duplicate key value violates unique constraint "changed_dso_assignment_pkey"
DETAIL:  Key ("EinheitMastrNummer")=(SEE974690188358) already exists.
2026-02-25 08:03:09,722 [ERROR] Error processing file 'Marktrollen.xml': 'Execution failed on sql 'INSERT INTO market_roles ("Id", "Wert", "DatenQuelle", "DatumDownload") VALUES (:Id, :We
rt, :DatenQuelle, :DatumDownload)': (psycopg2.errors.NotNullViolation) null value in column "MastrNummer" of relation "market_roles" violates not-null constraint
DETAIL:  Failing row contains (null, null, null, null, null, null, null, null, null, bulk, 2026-02-25, 1, Übertragungsnetzbetreiber).
2026-02-25 08:03:09,728 [INFO] Creating table 'grid_connections'...
2026-02-25 08:03:14,140 [INFO] From the downloaded xml files following new attribute was introduced: grid_connections.NetzbetreiberMaStRNummer
2026-02-25 08:03:22,942 [INFO] Processing file 'Netze.xml'...
2026-02-25 08:03:22,942 [INFO] Creating table 'grids'...
2026-02-25 08:03:23,135 [ERROR] Error processing file 'Netze.xml': 'could not convert string to float: '334, 335''
2026-02-25 08:03:59,736 [ERROR] Error processing file 'EinheitenAenderungNetzbetreiberzuordnungen_2.xml': 'Execution failed on sql 'INSERT INTO changed_dso_assignment ("EinheitMastrNummer", "LokationMastrNummer", "NetzanschlusspunktMastrNummer", "NetzbetreiberMastrNummerAlt", "NetzbetreiberMastrNummerNeu", "ArtDerAenderung", "RegistrierungsdatumNetzbetreiberzuordnungsaenderung", "Netzbetreiberzuordnungsaenderungsdatum", "DatenQuelle", "DatumDownload") VALUES (:EinheitMastrNummer, :LokationMastrNummer, :NetzanschlusspunktMastrNummer, :NetzbetreiberMastrNummerAlt, :NetzbetreiberMastrNummerNeu, :ArtDerAenderung, :RegistrierungsdatumNetzbetreiberzuordnungsaenderung, :Netzbetreiberzuordnungsaenderungsdatum, :DatenQuelle, :DatumDownload)': (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "changed_dso_assignment_pkey"
DETAIL:  Key ("EinheitMastrNummer")=(SEE971165771602) already exists.
2026-02-25 08:06:33,876 [ERROR] Error processing file 'EinheitenAenderungNetzbetreiberzuordnungen_3.xml': 'Execution failed on sql 'INSERT INTO changed_dso_assignment ("EinheitMastrNummer
", "LokationMastrNummer", "NetzanschlusspunktMastrNummer", "NetzbetreiberMastrNummerAlt", "NetzbetreiberMastrNummerNeu", "ArtDerAenderung", "RegistrierungsdatumNetzbetreiberzuordnungsaend
erung", "Netzbetreiberzuordnungsaenderungsdatum", "DatenQuelle", "DatumDownload") VALUES (:EinheitMastrNummer, :LokationMastrNummer, :NetzanschlusspunktMastrNummer, :NetzbetreiberMastrNum
merAlt, :NetzbetreiberMastrNummerNeu, :ArtDerAenderung, :RegistrierungsdatumNetzbetreiberzuordnungsaenderung, :Netzbetreiberzuordnungsaenderungsdatum, :DatenQuelle, :DatumDownload)': (psy
copg2.errors.UniqueViolation) duplicate key value violates unique constraint "changed_dso_assignment_pkey"
DETAIL:  Key ("EinheitMastrNummer")=(SEE973817225167) already exists.

Steps to Reproduce

mastr_downloader = Mastr(engine=self.engine)
mastr_downloader.download()

Ideas of solution

I guess that this does not happen every day, but still.
I thought that Bundesnetzagentur fixed this on their end.
Is there some way to communicate this to the BNetzA?

Context and Environment

• Version used: v0.16.1
• Operating system: Linux
• Environment setup and (python) version: Venv, Python 3.13

Workflow checklist

• I have checked the documentation and confirmed this issue is not already addressed there.
• I am aware of the workflow in CONTRIBUTING.md

[Simon-Will]

Hi Florian,

there are multiple issues here:

• duplicate keys in changed_dso_assignment: this is a known problem on open-mastr side (see Apparent duplicates in einheitenaenderungnetzbetreiberzuordnungen are dropped #577). The SQLALchemy model for this table is just wrong in open-mastr since it incorrectly assumes that each MaStR unit occurs only once. This is going to be fixed by 516 dynamic tables and columns #718
• Complete failure to import Marktrollen.xml: This is because the file Marktrollen.xml as it exists in current MaStR exports contains only a simple value list of available roles, whereas it used to contain what is now MarktakteureUndRollen.xml.
  • This is also fixed by 516 dynamic tables and columns #718.
  • Still, we could think about a fix also in the 0.* version. Any thoughts?
• Importing Netze.xml fails with could not convert string to float: '334, 335': I don't understand this one right now because I don't see why it would even try to convert "334, 335" to a float. Need to take a closer look at that.

I'm running an import of today's export with the state of #718 to check if everything goes well there. The first problem there is that some new XSD file is apparently invalid XML. That's something that we should definitely report to BNetzA. I'm going to do that.

Edit: Sent the following request for correction to BNetzA:

Sehr geehrtes MaStR-Team der Bundesnetzagentur,

im jüngsten Update der MaStR-Dokumentation wurden die XSD-Dateien geändert, v.a. minOccurs hinzugefügt/gelöscht. In der Datei EinheitenKernkraft.xsd fehlt bei der Gemeinde ein Slash (/). Dort steht:

<xs:element type="xs:string" name="Gemeinde" minOccurs="0">

Am Ende fehlt ein Slash. Es muss also heißen:

<xs:element type="xs:string" name="Gemeinde" minOccurs="0"/>

Leider ist die Datei dadurch ungültig und kann nicht automatisch genutzt werden. Ich würde mich freuen, wenn Sie sie bald korrigieren könnten.

Freundliche Grüße und vielen Dank
Simon Will

[FlorianK13]

Importing Netze.xml fails with could not convert string to float: '334, 335': I don't understand this one right now because I don't see why it would even try to convert "334, 335" to a float. Need to take a closer look at that.

@Simon-Will I assume this is some "Replace ID with actual meaning" thing, where 334 and 335 actually represent some string (Like "Braunkohle" or whatever).

So I assume that open-mastr does "315" -> lookup[number("315")] -> "Braunkohle" but number("314, 315") then fails.

But this is all guessing, I did not look into the error yet.

[Simon-Will]

@FlorianK13, you're right. I took a quick look at Netze.xml and it's the Bundesland field. But there are dozens of occurrences with more than one number here. And that's also not new in the newest version of the file. Not sure why it occurs now.