Mistral Models ct2 support

[manishiitg]

Feature request

ctranslate2 doesn't support mistral model conversion out of the box
OpenNMT/CTranslate2#1501

currently when using ct2 it automatically converts the model on the fly.

can we allow support to convert model to ct2 offline? that way i can convert mistral model offline and then use it with TGI.

Motivation

mistral support

Your contribution

[michaelfeil]

This issue should be solved upstream in ctranslate2. Once ctranslate2 supports mistral, tgi can bump the version.

Allowing a pre-converted model is not really on the roadmap, tgi really wants to download .safetensors in pytorch format. in that way, ct2 quantization, just quick-fixes this issue.