Llama.cpp Tool Calling with Python

Purpose: Exposes existing tool calling capability that's integrated into llama-server UI via Python scripts.

Main feature: Very simple codebase. Easily understood and highly hackable. Can easily add your own functionalities with Python scripting.

I made it for my own use. I've only tested that with Qwen3-VL-30B-A3B Q4M.

This program might also work with other OpenAI API compatible service providers.

Usage

1. Modify tools/__init__.py to enable/disable the tools available for use
2. Run llama-server provided by llama.cpp
3. Run python3 reverse_proxy.py --remote http://127.0.0.1:8080. Replace the --remote with the llama.cpp's server URL of your own
4. Copy the JSON content shown on the terminal
5. Open up the browser at http://127.0.0.1:8081 (Warning: Use 8081 instead of 8080. We have to go thru the reverse proxy server for the tools to work)
6. On the web server's UI, paste the content of llama-server > Settings > Developer > Custom JSON and save it

Developing new tools

1. Create a .py file in the tools folder
2. Define METADATA and toolcall_handler(). You can see how other tools are implemented as reference
3. Don't forget to add your tools to tools/__init__.py

Known issues

• The tool call's response doesn't get added to the message log. For now, only the message that performs the tool call can read the response from the tool. I think it isn't ready on the llama.cpp side yet.
  • It can significantly degrade the output quality because the any message after the immediate tool-calling message won't have the context on the tool call's output!
• Python eval tool: For security, you need to manually review the expression to be evaluated and press enter on terminal to run it. Do not bliding press enter or it could destroy your computer!
• Image generation: For the model I used, its performance is meh. I might as well just ask it to generate the prompt
• Known bug: Apparently the way that I handle multi-tool calling might be incorrect because if multiple tools got called, only one of them would get shown on the UI of the llama.cpp server.

License

GPL. Reason: I took code from ComfyUI for a tool and it's realeased under GPL.