Really struggling, latest version of llama-server, Mistral Nemo Instruct 2407, adding tool capability #16652
Replies: 1 comment
-
|
Okay, a lot more investigation, and for some reason (note I am using a custom mistral ninja template) but I stil find that when the slot reached the STOP in the model result, it is still in Chat format: Content-only, which is why the JSON for the tool call gets missed, as a tool call and returned to the user query.. Nothing wrong with the template, according to logs.. it just doesn't get set to COMMON_CHAT_FORMAT_OAI_COMPAT or COMMON_CHAT_FORMAT_MISTRAL_NEMO during the |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Okay, after three days.. now probably a .jinja file expert, but still struggling. Start the server with a custom .jinja file, modeled on the on mistralai-Mistral-Nemo-Instruct-2407.jinja, tools successfully load, the LLM can run the tool, but it won't do it in inline, instead I keep getting the one turn problem, it outputs the TOOL_CALL JSON only, instead of represssing that and responding based on TOOL_RESULTS. Whatever input I add, then it replies with the user friendly response of the previous query. Anyone get this working?
Beta Was this translation helpful? Give feedback.
All reactions