Problem with fr_aube (and other French NETEX files)

[ue71603]

In
./tool_scripts/script_regression_v1.1.txt log1.log fr_aube

there is a problem here
recursive_attributes.py

def embedding_obj_iter(
    serializer: Serializer, deserialized: Tid, interesting_classes: Optional[set[type[Tid]]], ignore: Optional[set[type[Tid]]]
) -> Generator[tuple[Optional[bytes], Tid, list[int]], None, None]:
    assert deserialized.id is not None, "deserialised.id must not be none"

    for obj, path in recursive_attributes(deserialized, []):
        if hasattr(obj, "id") and obj.id is not None:
            if (ignore is None or obj.__class__ not in ignore) and (interesting_classes is None or obj.__class__ in interesting_classes):
                yield serializer.encode_key(obj.id, obj.version if hasattr(obj, "version") else None, obj.__class__, include_clazz=True), obj, path

The problem is that __class__ is somewhat not valid and then the byte serializer gets a problem.
I don't know why this happens.

D:\development\github\badger\.venv\Scripts\python.exe -X pycache_prefix=C:\Users\Gm\AppData\Local\JetBrains\PyCharmCE2025.1\cpython-cache "C:/Program Files/JetBrains/PyCharm Community Edition 2025.1.1/plugins/python-ce/helpers/pydev/pydevd.py" --multiprocess --qt-support=auto --client 127.0.0.1 --port 51644 --file D:\development\github\badger\tools\script_runner.py ./tool_scripts/script_regression_v1.1.txt log1.log fr_aube --begin_step=4 
Connected to pydev debugger (build 251.25410.122)
2026-01-13 21:39:40,138 - script_runner - INFO - fr_aube - step: 4: conv.epip_db_to_db d:/aux_testing_processing/fr_aube/03-import.mdbx d:/aux_testing_processing/fr_aube/04-netex-database.mdbx
the arguments ['d:/aux_testing_processing/fr_aube/03-import.mdbx', 'd:/aux_testing_processing/fr_aube/04-netex-database.mdbx']
<class 'domain.netex.model.simple_point_version_structure.SimplePointVersionStructure'>
2026-01-13 21:46:00,147 - script_runner - ERROR - <class 'domain.netex.model.simple_point_version_structure.SimplePointVersionStructure'>
Traceback (most recent call last):
  File "D:\development\github\badger\tools\script_runner.py", line 436, in main
    result = load_and_run(script_name, script_args)
  File "D:\development\github\badger\tools\script_runner.py", line 116, in load_and_run
    result = main_function(*args1)
  File "D:\development\github\badger\conv\epip_db_to_db.py", line 310, in main
    epip_db_to_db(source_path, Path(target))
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\conv\epip_db_to_db.py", line 152, in epip_db_to_db
    source_db.insert_any_object_on_queue(txn_write1, all_psas())
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\storage\mdbx\core\implementation.py", line 242, in insert_any_object_on_queue
    for obj in objects:
               ^^^^^^^
  File "D:\development\github\badger\conv\epip_db_to_db.py", line 138, in all_psas
    embedding[1].id: getRef(obj, StopPlaceRef) for key, obj, embedding in resolve_embeddings_iterable(source_db, txn_write1, StopPlace)
                                                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\storage\mdbx\core\references.py", line 46, in resolve_embeddings_iterable
    for candidate in embedding_obj_iter(storage.serializer, obj, interesting_classes, ignore):
                     ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\domain\netex\services\recursive_attributes.py", line 186, in embedding_obj_iter
    yield serializer.encode_key(obj.id, obj.version if hasattr(obj, "version") else None, obj.__class__, include_clazz=True), obj, path
          ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\development\github\badger\storage\mdbx\serialization\byteserializer.py", line 53, in encode_key
    encoded_bytes.extend(self.class_idx[clazz])
                         ~~~~~~~~~~~~~~^^^^^^^
KeyError: <class 'domain.netex.model.simple_point_version_structure.SimplePointVersionStructure'>
2026-01-13 21:46:00,158 - script_runner - INFO - Execution time: 380.0 seconds for fr_aube - step: 4: conv.epip_db_to_db d:/aux_testing_processing/fr_aube/03-import.mdbx d:/aux_testing_processing/fr_aube/04-netex-database.mdbx

2026-01-13 21:46:00,158 - script_runner - ERROR - Script conv.epip_db_to_db returned an error. Terminating the block of scripts: fr_aube

Process finished with exit code 1

[skinkie]

The issue is in the source data, but we have an issue in processing it. Lets define all the issues:

1. The are using ids on objects that are not first class objects and never will (Centroid, Location) our software currently expect that once an id is defined, it might be a first class object and must be processed as such. We must change this. Then again, why would Centroid and Location have a unique id, it is bloat in the files.
2. You can see below that both a longitude/latitude and a gml:pos is defined. I'll leave it up to the reader if such ambiguities are a good idea™. But you will also notice that the Lambert zone II (unit meter) gml:pos is likely transformed from WGS84, and is defined in such accuracy that it absolutely does not make sense.

  <StopPlace id="FR:10387:StopPlace:gen1040953:AUBE" version="1">
     <Name>TROYES - GARE SNCF</Name>
     <Centroid id="AUBE:Centroid:3a2cf186a2ce42849f7421cba847eeb8:LOC" version="any">
        <Location id="AUBE:Location:0a1ee66d1f2f4dc2bf0f81e853ca5295:LOC">
           <Longitude>4.0656439</Longitude>
           <Latitude>48.2960202</Latitude>
           <gml:pos srsName="EPSG:27572">728315.866706993 2367745.38162884</gml:pos>
        </Location>
     </Centroid>

I'll resolve the processing issue I hope that @TuThoThai and @thbar would do some documentation updates.

[skinkie]

In addition:

1. For some reason the people of France use versionRef for the relationship between a reference and the object it is referenced to. While I would completely support this breaking change in NeTEx v3 @TuThoThai it has not been the interpretation of NeTEx. Hence this is likely the only production file that is using versionRef while it should be using version, because the content is in the same file.
2. The relationship between JourneyPatternRef towards a ServiceJourneyPattern has been made implicit, I think we can say that versionRef is invalid because the reference is in the same file, nameOfRefClass is missing the reference is not pointing to a JourneyPattern, but a ServiceJourneyPattern.

  <ServiceJourneyPattern id="AUBE:ServiceJourneyPattern:11:LOC" version="1">
     <Name>SAINTE-SAVINE - LYC.HERRIOT/MALAD - Sainte-Savine</Name>
     <PrivateCode>15768708</PrivateCode>
     <RouteRef ref="AUBE:Route:1007126-A:LOC" versionRef="1"/>
     <DestinationDisplayRef ref="AUBE:DestinationDisplay:11:LOC" versionRef="1"/>
     <pointsInSequence>
        <StopPointInJourneyPattern id="AUBE:StopPointInJourneyPattern:11x1:LOC" version="any" order="6">
           <ScheduledStopPointRef ref="AUBE:ScheduledStopPoint:24:LOC" versionRef="1"/>
           <ForAlighting>false</ForAlighting>
           <ForBoarding>true</ForBoarding>
           <RequestStop>false</RequestStop>
        </StopPointInJourneyPattern>
        <StopPointInJourneyPattern id="AUBE:StopPointInJourneyPattern:11x2:LOC" version="any" order="7">
           <ScheduledStopPointRef ref="AUBE:ScheduledStopPoint:25:LOC" versionRef="1"/>
           <ForAlighting>true</ForAlighting>
           <ForBoarding>false</ForBoarding>
           <RequestStop>false</RequestStop>
        </StopPointInJourneyPattern>
     </pointsInSequence>
     <ServiceJourneyPatternType>passenger</ServiceJourneyPatternType>
  </ServiceJourneyPattern>

  <ServiceJourney id="AUBE:ServiceJourney:1020985-1026819:LOC" version="1">
     <PrivateCode>NAVETTE04  GARE/E. HERRIOT</PrivateCode>
     <TransportMode>coach</TransportMode>
     <dayTypes>
        <DayTypeRef ref="AUBE:Timetable:DayType:8127:LOC" versionRef="1"/>
     </dayTypes>
     <JourneyPatternRef ref="AUBE:ServiceJourneyPattern:11:LOC" versionRef="1"/>
     <passingTimes>
        <TimetabledPassingTime version="any">
           <ArrivalTime>07:35:00</ArrivalTime>
           <ArrivalDayOffset>0</ArrivalDayOffset>
           <DepartureTime>07:35:00</DepartureTime>
           <DepartureDayOffset>0</DepartureDayOffset>
        </TimetabledPassingTime>
        <TimetabledPassingTime version="any">
           <ArrivalTime>07:50:00</ArrivalTime>
           <ArrivalDayOffset>0</ArrivalDayOffset>
           <DepartureTime>07:50:00</DepartureTime>
           <DepartureDayOffset>0</DepartureDayOffset>
        </TimetabledPassingTime>
     </passingTimes>
  </ServiceJourney>

With the migration to the NeTEx schema version 2, we now always have nameOfRefClass filled in, but it seems that are not updating any missing or invalid references.

[ue71603]

• "id"-ing stupid stuff: We remove those in a fix step (based on a list of elements)? When you do a template on how to open/update an mdbx for one of the fixes I can give it a go as well. We need to have comments in the files when and it which sequence they are to be used. Also, in the best cases they do no harm if applied unnecessarily.
• Copy refversion to version migt also be a fix step. Do we really want to support refversion anywhere? If not, we should address this is NeTEx.
• The JourneyPatternRef: should this also be a fix step?

[skinkie]

JourneyPatternRef shouldn't have been an issue in the first place, that should already have been fixed after the importing step. But I am not first targetting the (embedded) DayTypeRef thing.

[ue71603]

only ideas... :-)

[skinkie]

@ue71603
52bbf72

[ue71603]

fr_aube tested with a lot of fixes. same for fr_ardennes. Now working on fr_sncf_ter.