Fix: Added check the type of tokenizer.json's model_merges

[N-E-W-T-O-N]

Modern Tokenizer store merge's value as list of lists of strings. But models like GPT-2 or gemma-2 store as a list of strings
Ex Mistral

Added a check for this fix

[N-E-W-T-O-N]

Mistral

merges": [
      [
        "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
        "\n"
      ],
      [
        "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
        "▁"
      ],
      [

[N-E-W-T-O-N]

Gpt-2

"merges":["Ġ t","Ġ a","h " ...

[N-E-W-T-O-N]

@shaaga @sayanshaw24

[sayanshaw24]

Modern Tokenizer store merge's value as list of lists of strings. But models like GPT-2 or gemma-2 store as a list of strings Ex Mistral

Added a check for this fix

awesome, thanks for this! could you add a unit test here as well to ensure expected functionality: https://github.com/microsoft/onnxruntime-extensions/tree/main/test?

[sayanshaw24]

/azp run onnxruntime-extensions.CI

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[N-E-W-T-O-N]

Using phi-4 as an example

[N-E-W-T-O-N]

The following Functions is only called when use_fast=True used in passed to tokenizer

[sayanshaw24]

please use test/data/phi-4-base or test/data/phi-4-mini-reasoning, we'd like to minimize tokenizer files checked into the repo; (also, FYI onnxruntime-extensions only needs the tokenizer.json and tokenizer_config.json files to load tokenizers).

[azure-pipelines]

Commenter does not have sufficient privileges for PR 987 in repo microsoft/onnxruntime-extensions

[N-E-W-T-O-N]

Hi @sayanshaw24
Revert back the change as requested and updated the branch .
Please merge my pr

[sayanshaw24]

/azp run onnxruntime-extensions.CI

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[N-E-W-T-O-N]

Merge the latest code .Please rerun the pipeline

[snnn]

/azp where

[azure-pipelines]

Azure DevOps orgs getting events for this repository:

• aiinfra

[N-E-W-T-O-N]

Hi @sayanshaw24 pr please merge my PR