External controlled tool calling with `ToolCallingManager` cannot really work.

[chickenlj]

Bug description

As defined below, executeToolCalls(Prompt prompt, ChatResponse chatResponse) in Spring AI requires prompt to get Options configuration as the context to get tool definition, prepare ToolContext, etc.

But if I use ToolCall outside the scope of ChatClient, there's no way for me to get the prompt that was used in ChatClient that inited the tool calling request.

public interface ToolCallingManager {

	/**
	 * Resolve the tool definitions from the model's tool calling options.
	 */
	List<ToolDefinition> resolveToolDefinitions(ToolCallingChatOptions chatOptions);

	/**
	 * Execute the tool calls requested by the model.
	 */
	ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse chatResponse);

}

Simply put, where I can get the prompt instance to call toolCallingManager.executeToolCalls(prompt, chatResponse);

Environment
1.0.0-M6

[chickenlj]

One way to solve that is to declare Prompt and FunctionCallBack explicitly instead of letting ChatClient create it

ChatModel chatModel = ...
ToolCallingManager toolCallingManager = ToolCallingManager.builder().build();

ChatOptions chatOptions = ToolCallingChatOptions.builder()
    .toolCallbacks(new CustomerTools())
    .internalToolExecutionEnabled(false)
    .build();
Prompt prompt = new Prompt("Tell me more about the customer with ID 42", chatOptions);

ChatResponse chatResponse = chatModel.call(prompt);

while (chatResponse.hasToolCalls()) {
    ToolExecutionResult toolExecutionResult = toolCallingManager.executeToolCalls(prompt, chatResponse);

    prompt = new Prompt(toolExecutionResult.conversationHistory(), chatOptions);

    chatResponse = chatModel.call(prompt);
}

But that's sometimes very inconvenient. More importantly, I worry it will lose some updates on the Prompt inside call() process because I see copy and merge operations here and there inside call().

[rwankar]

This is how I'm doing it. So far I've not run into an issue with copy/merge.

[iAMSagar44]

How are you guys handling ChatMemory in this flow if you are not using ChatClient?

[ThomasVitale]

We have added an example of user-controlled tool execution with chat memory to the documentation: https://docs.spring.io/spring-ai/reference/api/tools.html#_user_controlled_tool_execution.

I hope that helps until memory becomes a first-class citizen in ChatClient and will be possible to get the tool execution raised at that level transparently as well.