Inference: onnx2torch crashes with models using same padding

[trivoldus28]

Severity is high due to same padding being used in many models. However, it's possible to side step the problem by using other checkpointing mechanisms.

Error is something like this

"/home/tri/zetta_local/venv/python312/lib/python3.12/site-packages/onnx2torch/converter.py",

line 110, in convert

torch_module, onnx_mapping = converter(onnx_node, onnx_graph)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File

"/home/tri/zetta_local/venv/python312/lib/python3.12/site-packages/onnx2torch/node_converters/c

onv.py", line 48, in _

padding, input_padding_module = onnx_auto_pad_to_torch_padding(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File

"/home/tri/zetta_local/venv/python312/lib/python3.12/site-packages/onnx2torch/utils/padding.py"

, line 33, in onnx_auto_pad_to_torch_padding

raise NotImplementedError(f'"{auto_pad}" auto_pad is not implemented')

NotImplementedError: "SAME_UPPER" auto_pad is not implemented

[trivoldus28]

A solution that seems to work is to monkey patch

onnx_auto_pad_to_torch_padding_original = onnx2torch.utils.padding.onnx_auto_pad_to_torch_padding  # save
def onnx_auto_pad_to_torch_padding_patched(
    auto_pad: str,
    onnx_padding: Tuple[int, ...],
) -> Tuple[Union[int, Tuple[int, ...]], Optional[nn.Module]]:
    if auto_pad in ("SAME_UPPER"):
        return "same", None
    return onnx_auto_pad_to_torch_padding_original(auto_pad=auto_pad, onnx_padding=onnx_padding)
onnx2torch.utils.padding.onnx_auto_pad_to_torch_padding = onnx_auto_pad_to_torch_padding_patched  # monkey patch
onnx2torch.node_converters.conv.onnx_auto_pad_to_torch_padding = onnx_auto_pad_to_torch_padding_patched

edit: more concise code

[nkemnitz]

Yeah, looks like it's missing from onnx2torch... I think Kisuk explicitly set the pad values to avoid this issue in the past.
Your fix is correct, though. torch.onnx.export(F.Conv2d(..., padding='same')) gets stored as SAME_UPPER. We should try to submit it to onnx2torch. Although I am a bit concerned about the lack of activity on that project...

[trivoldus28]

Another option is to wrap onnx model, something like this. Should give best compatibility as long as onnxruntime is maintained. But then you can't use torch.compile()

import torch
import onnxruntime as ort
import numpy as np

class ONNXWrapper(torch.nn.Module):
    def __init__(self, onnx_path):
        super().__init__()
        self.session = ort.InferenceSession(onnx_path)

    def forward(self, x: torch.Tensor):
        inputs = {self.session.get_inputs()[0].name: x.detach().cpu().numpy()}
        outputs = self.session.run(None, inputs)
        return torch.tensor(outputs[0])

[nkemnitz]

Still no sign of life on the onnx2torch repo. We might need to think about an alternative there.

[trivoldus28]

I think just patch it like I posted, has been working fine the past month, or fork the repo and patch there (and maybe submit a PR)

[nkemnitz]

PR is already submitted, but got no reeaction (Link).

We can merge your patch to zutils if you would like to create a PR, but long-term I would like to avoid any dependencies on unmaintained packages, nor add another repo we have to maintain ourselves.

[trivoldus28]

Then what about the other solution of using onnxruntime?

[nkemnitz]

Oh, missed the notification here, sorry!

I originally removed onnxruntime only because it adds duplicate CUDA libraries which were bloating the docker image size. But more importantly it can cause differences in inference results that the user needs to be aware of - e.g. mismatches between mixed precision and TensorCore settings between onnxruntime and pytorch.