Checking for equality on Mmif instances fails

[marcverhagen]

Bug Description

When comparing two instances of Mmif I get an error.

Reproduction steps

Here is some minimal code to replicate this:

from mmif import Mmif

minimal_mmif = '''
{
  "metadata": {
    "mmif": "http://mmif.clams.ai/1.0.0"
  },
  "documents": [
    {
      "@type": "http://mmif.clams.ai/vocabulary/VideoDocument/v1",
      "properties": {
        "mime": "video",
        "id": "d1",
        "location": "file:///mnt/llc/llc_data/clams/chyron-batch-running/cpb-aacip-507-154dn40c26.mp4"
      }
    }
  ],
  "views": []
}'''

m1 = Mmif(minimal_mmif)
m2 = Mmif(minimal_mmif)
m1 == m2

Expected behavior

No response

Log output

$ python test_mmif_equality.py
Traceback (most recent call last):
  File "/Users/marc/Documents/git/clams/aapb-brandeis-datahousing/test.py", line 27, in <module>
    m1 == m2
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/mmif/serialize/model.py", line 270, in __eq__
    len(DeepDiff(self, other, report_repetition=True, exclude_types=[datetime],
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 361, in __init__
    self._diff(root, parents_ids=frozenset({id(t1)}), _original_type=_original_type)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 494, in _diff_obj
    self._diff_dict(
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 706, in _diff_dict
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 1741, in _diff
    self._diff_iterable(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 753, in _diff_iterable
    self._diff_iterable_in_order(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 884, in _diff_iterable_in_order
    self._diff_by_forming_pairs_and_comparing_one_by_one(
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 976, in _diff_by_forming_pairs_and_comparing_one_by_one
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/diff.py", line 480, in _diff_obj
    t1 = detailed__dict__(level.t1, ignore_private_variables=self.ignore_private_variables)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/deepdiff/helper.py", line 742, in detailed__dict__
    value = getattr(obj, key)
  File "/Applications/ADDED/venv/clams/datahouse/lib/python3.10/site-packages/mmif/serialize/annotation.py", line 374, in text_language
    raise ValueError("Only TextDocument can have `text` field.")
ValueError: Only TextDocument can have `text` field.

Screenshots

No response

Additional context

No response

[ledibr]

Using similar minimal code (from another directory since running inside the mmif-python directory makes it unhappy):

from mmif import Mmif

minimal_mmif = '''
{
  "metadata": {
    "mmif": "http://mmif.clams.ai/1.0.4"
  },
  "documents": [
    {
      "@type": "http://mmif.clams.ai/vocabulary/VideoDocument/v1",
      "properties": {
        "mime": "video",
        "id": "m1",
        "location": "file:///llc_data/clams/hichy-evaluation-dataset/videos/250717/hi-chy-hi/cpb-aacip-058a21352e3.mp4"
      }
    }
  ],
  "views": []
}'''

if __name__ == '__main__':
    m1 = Mmif(minimal_mmif)
    m2 = Mmif(minimal_mmif)
    print(m1 == m2)

First error encountered:

Traceback (most recent call last):
  File "C:\Users\Lindsay\PycharmProjects\CLAMS\app-heuristic-chyron-understanding\mmif_test_equality.py", line 24, in <module>
    print(m1 == m2)
          ^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\mmif\serialize\model.py", line 270, in __eq__
    len(DeepDiff(self, other, report_repetition=True, exclude_types=[datetime],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 361, in __init__
    self._diff(root, parents_ids=frozenset({id(t1)}), _original_type=_original_type)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 494, in _diff_obj
    self._diff_dict(
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 706, in _diff_dict
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1741, in _diff
    self._diff_iterable(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 753, in _diff_iterable
    self._diff_iterable_in_order(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 884, in _diff_iterable_in_order
    self._diff_by_forming_pairs_and_comparing_one_by_one(
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 976, in _diff_by_forming_pairs_and_comparing_one_by_one
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 480, in _diff_obj
    t1 = detailed__dict__(level.t1, ignore_private_variables=self.ignore_private_variables)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\helper.py", line 742, in detailed__dict__
    value = getattr(obj, key)
            ^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\mmif\serialize\annotation.py", line 158, in parent
    raise ValueError(f'Annotation {self.id} does not have a parent view, or its ID does not follow the expected format. ')
ValueError: Annotation m1 does not have a parent view, or its ID does not follow the expected format.

Temporary fix: Appending something to the start of the doc ID, so now we have "np:m1" in the id field.

Problem: This is happening for essentially the same reason as below, but for the parent key.

Solution: Obviously, this can be easily circumvented by giving originating documents a dummy parent id ("np" used here to stand for "no parent"), but doesn't fix broader issue of keys we don't want being compared.

Second error encountered (replicating Marc's bug):

Traceback (most recent call last):
  File "C:\Users\Lindsay\PycharmProjects\CLAMS\app-heuristic-chyron-understanding\mmif_test_equality.py", line 24, in <module>
    print(m1 == m2)
          ^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\mmif\serialize\model.py", line 270, in __eq__
    len(DeepDiff(self, other, report_repetition=True, exclude_types=[datetime],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 361, in __init__
    self._diff(root, parents_ids=frozenset({id(t1)}), _original_type=_original_type)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 494, in _diff_obj
    self._diff_dict(
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 706, in _diff_dict
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1741, in _diff
    self._diff_iterable(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 753, in _diff_iterable
    self._diff_iterable_in_order(level, parents_ids, _original_type=_original_type, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 884, in _diff_iterable_in_order
    self._diff_by_forming_pairs_and_comparing_one_by_one(
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 976, in _diff_by_forming_pairs_and_comparing_one_by_one
    self._diff(next_level, parents_ids_added, local_tree=local_tree)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 1747, in _diff
    self._diff_obj(level, parents_ids)
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\diff.py", line 480, in _diff_obj
    t1 = detailed__dict__(level.t1, ignore_private_variables=self.ignore_private_variables)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\deepdiff\helper.py", line 742, in detailed__dict__
    value = getattr(obj, key)
            ^^^^^^^^^^^^^^^^^
  File "C:\Users\Lindsay\miniconda3\Lib\site-packages\mmif\serialize\annotation.py", line 411, in text_language
    raise ValueError("Only TextDocument can have `text` field.")
ValueError: Only TextDocument can have `text` field.

Problem: In DeepDiff's detailed__dict__ method, the loop beginning on line 736 with for key in dir(obj) is iterating through literally every key it is possible for the object to have as returned by dir(). When the error occurs, the code is comparing the two MMIFs at the Document level, and is attempting to get the value of the text_language key; however, because the document is a VideoDocument, annotation.py raises a ValueError.

Solution: I don't know enough about DeepDiff to know if there's a reasonably easy way to resolve this. I suspect the eventual solution might have to involve serious restructuring of MMIFs or simply writing our own equality method, since DeepDiff is displaying unwanted behavior when it comes to parsing MMIF structure. Maybe text_language and text_value need to be handled differently and not treated as default properties? Or have them return something like "no text field" instead of raising a ValueError?

Upon commenting out the text field-related parts of annotation.py, the equality method works as intended and returns True for the example case given. Other tests evaluated:

• Using a separate but identical MMIF object for comparison: True
• Different MMIF versions: False
• Different doc id: False
• Different doc type: False
• Different mime: False
• Different location: False

All of the above behave as expected. I have opted not to fiddle around with more complicated testing yet, given the structural nature of the existing issues, but will do so once those are resolved.

[keighrim]

The equality check for Mmif objects was failing due to the DeepDiff library's aggressive attribute access, which triggered ValueError exceptions in certain properties. I'm proposing replacing DeepDiff with a simpler serialization-based comparison. This new approach distinguishes between reproducibility runtime metadata from contextual runtime metadata, such as timestamps and stack traces, which are specific to the runtime environment, not contributing the the reproducibilty of the pipeline.

The implementation in my PR introduced a _contextual_attributes set in the base MmifObject class to mark these varying attributes. The serialize method was updated to accept an include_context parameter, defaulting to True, which filters out these attributes when set to False. The __eq__ method now performs a string comparison of the two objects' serialized JSON representations, with contextual attributes excluded. This change was propagated recursively to nested objects, ensuring that attributes like ViewMetadata.timestamp and ErrorDict.stackTrace are ignored during comparison, thus providing a more robust and semantically correct equality check. This also removed the deepdiff dependency entirely.