webui: add OAI-Compat Harmony tool-call streaming visualization and persistence in chat UI

[ServeurpersoCom]

- Purely visual and diagnostic change, no effect on model context, prompt
  construction, or inference behavior

- Captured assistant tool call payloads during streaming and non-streaming
  completions, and persisted them in chat state and storage for downstream use

- Exposed parsed tool call labels beneath the assistant's model info line
  with graceful fallback when parsing fails

- Added tool call badges beneath assistant responses that expose JSON tooltips
  and copy their payloads when clicked, matching the existing model badge styling

- Added a user-facing setting to toggle tool call visibility to the Developer
  settings section directly under the model selector option

Close #16597

[ServeurpersoCom]

I have to do a little cleaning, the patch was not merged properly on my side. -> draft

[ServeurpersoCom]

This PR is now clean, but it was developed after this one: #16562

[allozaur]

Alright, @ServeurpersoCom, let's move forward with this one after merging #16562 ;) Let me know when you've addressed the merge conflicts and I'll gladly review the code

[ServeurpersoCom]

For the tool call inspector, do you prefer having one spoiler block per tool call, or a single aggregated spoiler wrapping all tool calls in the message?

It's rebased/reworked now. I push --force :)

[ServeurpersoCom]

Feel free to dissect the architecture as deep as you want! Component boundaries, store coupling, service layering, anything that smells non-idiomatic.
Also, if we end up polishing this feature further, I’m thinking it could live in a dedicated module for cleaner boundaries ?

lib/
 └─ toolcalls/
     ├─ toolcall-service.ts
     ├─ toolcall-store.ts
     ├─ ToolCallBlock.svelte
     └─ ToolCallItem.svelte

[ServeurpersoCom]

And we could even imagine the architecture being reusable later : like having a small JavaScript execution module decoupled from the UI, so the model could actually interact with a JS thread it coded itself.
That would also cover, in a more generic way, the proposal from PR #13501 by @samolego but in this case, the model would generate and run its own JS tools. Done properly, it’s no more of a security risk than the HTML/JS preview you get in Hugging Face Chat or Claude!

[ServeurpersoCom]

Includes a very small optimization from the previous PR (scroll listener removal). It landed here intentionally :D

[ServeurpersoCom]

Testing :

Add this

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "simple_addition_tool",
        "description": "A dummy calculator tool used for testing multi-argument tool call streaming.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "type": "number",
              "description": "The first number to add."
            },
            "b": {
              "type": "number",
              "description": "The second number to add."
            }
          },
          "required": ["a", "b"]
        }
      }
    }
  ]
}

Here :

And ask model :

[allozaur]

Just few cosmetics

[ServeurpersoCom]

Rebase / Format / Build

[allozaur]

@ServeurpersoCom please re-base & rebuild

[ServeurpersoCom]

@ServeurpersoCom please re-base & rebuild

rebased and rebuilt

[SadaleNet]

Hello @ServeurpersoCom. Thank you for implementation tool-calling. I'm highly interested in it. I was trying to reproduce your 2+2 calculation example. Unfortunately, the tool calling part wasn't working as shown below.

I wonder how I could attach a script to perform the actual computation. Do I need to use a browser script to detect the presence of the tool calling clipboard copy button and send the tool calling response to an API. Or if there's an API endpoint that my script/program need to listen for tool calling events so that my script can perform the computation?

[ServeurpersoCom]

Hello @ServeurpersoCom. Thank you for implementation tool-calling. I'm highly interested in it. I was trying to reproduce your 2+2 calculation example. Unfortunately, the tool calling part wasn't working as shown below.

I wonder how I could attach a script to perform the actual computation. Do I need to use a browser script to detect the presence of the tool calling clipboard copy button and send the tool calling response to an API. Or if there's an API endpoint that my script/program need to listen for tool calling events so that my script can perform the computation?

It's a tools_calls debugger-only, and it works. On your screenshot you get a "simple_addition_tool" tag under the (empty) assistant message. Hover or click to read the function written by your model!

[ServeurpersoCom]

https://github.com/user-attachments/assets/ac188e22-9bbf-48a0-8e12-f655ec5a4ecd
We’re working hard with Alek on the MCP client. Here’s what it does in dev

[SadaleNet]

Hello @ServeurpersoCom. Thank you for implementation tool-calling. I'm highly interested in it. I was trying to reproduce your 2+2 calculation example. Unfortunately, the tool calling part wasn't working as shown below.
I wonder how I could attach a script to perform the actual computation. Do I need to use a browser script to detect the presence of the tool calling clipboard copy button and send the tool calling response to an API. Or if there's an API endpoint that my script/program need to listen for tool calling events so that my script can perform the computation?

It's a tools_calls debugger-only, and it works. On your screenshot you get a "simple_addition_tool" tag under the (empty) assistant message. Hover or click to read the function written by your model!

oh. ok. So, at this point, is there any way that I could actually execute the calculation function with the UI provided by llama-server? I've got a CLI program working that can do the calculation work using the OpenAi-compatible API of llama-server. I don't want to reinvent the UI if it already exist.

[ServeurpersoCom]

Hello @ServeurpersoCom. Thank you for implementation tool-calling. I'm highly interested in it. I was trying to reproduce your 2+2 calculation example. Unfortunately, the tool calling part wasn't working as shown below.
I wonder how I could attach a script to perform the actual computation. Do I need to use a browser script to detect the presence of the tool calling clipboard copy button and send the tool calling response to an API. Or if there's an API endpoint that my script/program need to listen for tool calling events so that my script can perform the computation?

It's a tools_calls debugger-only, and it works. On your screenshot you get a "simple_addition_tool" tag under the (empty) assistant message. Hover or click to read the function written by your model!

oh. ok. So, at this point, is there any way that I could actually execute the calculation function with the UI provided by llama-server? I've got a CLI program working that can do the calculation work using the OpenAi-compatible API of llama-server. I don't want to reinvent the UI if it already exist.

Yes you don’t need to reinvent the wheel. But the UI is still in development, and it’s a heavy piece of work. That’s exactly why MCP exists: the model emits tool calls, you wrap them, and you send them to an MCP server that returns the result into the context. If you’re comfortable with sysadmin work, I can give you what you need.

[SadaleNet]

Hello @ServeurpersoCom. Thank you for implementation tool-calling. I'm highly interested in it. I was trying to reproduce your 2+2 calculation example. Unfortunately, the tool calling part wasn't working as shown below.
I wonder how I could attach a script to perform the actual computation. Do I need to use a browser script to detect the presence of the tool calling clipboard copy button and send the tool calling response to an API. Or if there's an API endpoint that my script/program need to listen for tool calling events so that my script can perform the computation?

It's a tools_calls debugger-only, and it works. On your screenshot you get a "simple_addition_tool" tag under the (empty) assistant message. Hover or click to read the function written by your model!

oh. ok. So, at this point, is there any way that I could actually execute the calculation function with the UI provided by llama-server? I've got a CLI program working that can do the calculation work using the OpenAi-compatible API of llama-server. I don't want to reinvent the UI if it already exist.

Yes you don’t need to reinvent the wheel. But the UI is still in development, and it’s a heavy piece of work. That’s exactly why MCP exists: the model emits tool calls, you wrap them, and you send them to an MCP server that returns the result into the context. If you’re comfortable with sysadmin work, I can give you what you need.

Oh sure. Tell me more. I just need a starting point. Particularly on how to intercept the model's tool call signal and return the appropriate result to the model while using the UI of llama-server. Just to confirm, would this mechanism work in the current master branch of llama.cpp?

As for MCP, if it's not absolutely required, I guess I can explore that on my own later. :P

Again, thanks a lot for working on this feature. People like me are highly thankful of your work.

EDIT: Oh wait. Did you actually mean that the UI isn't ready and I have to use other methods to get the tool-calling mechanism working for now?

[ServeurpersoCom]

EDIT: Oh wait. Did you actually mean that the UI isn't ready and I have to use other methods to get the tool-calling mechanism working for now?

Absolutely

[SadaleNet]

I've figure out how to get this feature to work and created a repo for that: https://github.com/SadaleNet/llamacpp-tool-calling-python

Again, thanks for your groundwork. My script wouldn't be working without your greater prior work.

[ServeurpersoCom]

I've figure out how to get this feature to work and created a repo for that: https://github.com/SadaleNet/llamacpp-tool-calling-python

Again, thanks for your groundwork. My script wouldn't be working without your greater prior work.

Cool !

You can try this one; it's integrated in TypeScript. Just edit the config.js file to configure your MCP server. It's still under development; for the time being, I've put my MCP server's URL directly into it (easy to find with git grep). :) It supports WebSocket and Streamable-HTTP, and configurable agentic loop (chain of toolcall)

https://github.com/ServeurpersoCom/llama.cpp/tree/mcp-client-alpha I add full UI settings soon.

I have also a full backend Node.js OAI-Compat reverse proxy version supporting stdio/ws/streamable-http transport (testing-branch18 and nexts). It's like llama-swap, but for adding tools to any OAI basic client (bot, "jarvis" style voice assistant, etc.)

Both handle the context well, they allow for long chains of complex development if you put an MCP server like the one I put as an example, it performs almost as well as Claude in computer use depending on the model.