Support huggingface transformers models and datasets

[aguschin]

We need to add support for the huggingface transformers models and datasets.

[judahrand]

Are there any predictions for timelines on when this might be implemented?

[shcheklein]

@judahrand could you please describe the use case a bit more? so that's easier for us to prioritize this. thanks!

[judahrand]

@judahrand could you please describe the use case a bit more? so that's easier for us to prioritize this. thanks!

We use huggingface transformer models and would prefer to save them in their native form so that they can be picked up by BentoML for deployment.

[judahrand]

I looks like work was started on this: #553

[aguschin]

@judahrand, unfortunately, I'm at capacity now. I'd love to see this implemented, but don't have time 😞

Why you need this implemented specifically in MLEM? Looks like simply saving it with HF + using Bento would work.

[judahrand]

@judahrand, unfortunately, I'm at capacity now. I'd love to see this implemented, but don't have time 😞

Why you need this implemented specifically in MLEM? Looks like simply saving it with HF + using Bento would work.

We would like to use MLEM to power our 'model registry' and then deploy those models using BentoML. I suppose we can save the model to the model registry using the standard HF implementation and manually track it using DVC but we'd like to use MLEM if possible.

[aguschin]

Thanks @judahrand! To better understand your use case: so you are looking not for model deployment only, but for "model registry" kind of things as well (like managing multiple models, registering versions, etc). Did you take a look at our docs for model registry already? https://dvc.org/doc/use-cases/model-registry