Special token ids aren't validated when loading a model

[KerfuffleV2]

When loading the GGUF model, special token ids aren't validated to be in range, this can lead to index errors later on when they're used to looked up tokens, etc.

Here's an example: https://huggingface.co/Undi95/Mistral-11B-OmniMix/blob/main/config.json

We have

  "eos_token_id": 32000,

However the model's vocab size is 32,000 so that is out of bounds. Currently trying to load that model just crashes.

[TheBloke]

Regarding that model, I'll tell Undi he's missing the token in added tokens, or that the eos token is wrong.

He's had a few model config issues recently.

[KerfuffleV2]

He's had a few model config issues recently.

Coming from you, words to strike fear in the heart of anyone that makes models and is smaller than Meta. :)

[staviq]

I have some validation in #3538 in llm_load_vocab, it shouldn't be hard to extend it

[KerfuffleV2]

@staviq

I have some validation in #3538 i

From looking at that, seems like it's dealing with a different kind of validation. I don't think my fix (if I actually got it right finally) should conflict with that.