Error while using llama.cpp with model bge-small-en-v1.5 (converted to .gguf with convert_hf_to_gguf.py)

[michael5511b]

Hello,

I am a new user of llama.cpp. I converted the bge-small-en-v1.5 model into .gguf format with the convert_hf_to_gguf.py script that came with llama.cpp. The conversion seemed successful, but when I ran:
llama-cli -m ~/<gguf-directory>/bge-small-en.gguf -p "Tell me a story."
I get the error message:
(on llama.cpp version 3368)

llama_new_context_with_model:        CPU  output buffer size =     0.12 MiB
llama_new_context_with_model:        CPU compute buffer size =    16.00 MiB
llama_new_context_with_model: graph nodes  = 428
llama_new_context_with_model: graph splits = 1
GGML_ASSERT: /tmp/llama.cpp-20240710-10175-gh93hn/src/llama.cpp:14560: strcmp(res->name, "result_output") == 0 && "missing result_output tensor"

(on llama.cpp version 2950)

generate: n_ctx = 512, n_batch = 2048, n_predict = 128, n_keep = 0
[CLS] Tell me a story.[SEP] [end of text]

llama_print_timings:        load time =      13.01 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =       1.11 ms /     1 tokens
Log end

both versions I tried does not work,

• version 3368: GGML_ASSERT: /tmp/llama.cpp-20240710-10175-gh93hn/src/llama.cpp:14560: strcmp(res->name, "result_output") == 0 && "missing result_output tensor"
• version 2950: Does not generate any text.

I also tried all different quantization levels for the .gguf conversion, but all of them have the same error messages. I was wondering if this particular model is just not compatible with llama.cpp or if I missed anything here....

[ggerganov]

This is an embedding model - use llama-embedding or llama-server --embeddings. See #7712 for more info

[michael5511b]

Hello, thank you for your work, and thank you for the reply!

When I use:
llama-embedding -m ~/Downloads/bge-small-en-v1.5-f32.gguf -e -p "Tell me a story" command, it only returned an embedding vector.
I also used the llama-server --embeddings version of this command and accessed web UI on http://localhost:8080/, when I typed Tell me a story in the text box and hit send, it showed:

llama_get_logits_ith: invalid logits id 49, reason: no logits
Segmentation fault

in the terminal, and nothing was generated in the UI.

I saw in both #7712 and #2180 have discussion of using the embedding vector to perform inference on the prompt, but neither of these post have a solution. Is inference with embedding model not supported at the moment?

[ggerganov]

Embedding models cannot generate text. They can only compute vector embeddings of text