Feature Discussion: Code Generation + Execution

[taylorn-ai]

I'd like to suggest an enhancement to this project: supporting code generation and execution using Azure OpenAI and Azure Container Apps dynamic sessions.

Use Case
In many scenarios, especially technical or data-heavy contexts, it's useful to go beyond just generating code. Allowing users to execute the generated code (e.g. Python) inline and return results would open up more interactive and practical use cases - such as data analysis, file processing, or visualization. I have a basic working example, but it needs some work.

Potential Approach

1. Use an agent-based loop (LangChain or similar) to generate code.
2. Offload execution to a secure, ephemeral environment using Azure Container Apps session pools.
3. Return the result, including stdout, stderr, and any generated files or images, back to the UI.

Benefits

• Adds practical utility to LLM responses.
• Keeps code execution isolated and secure.
• Enables building intelligent agents that can reason and act on data.

Would this be in scope for the project? If so, I’d be interested in discussing implementation options or contributing a prototype.

[pamelafox]

@taylorn-ai Thanks for raising this discussion. I think it's really powerful when you can use a Code Interpreter in a sandboxed environment (like Container Apps Dynamic Sessions).

I think we would need to use the agent-based loop after the retrieval step, to allow the LLM to either directly answer the question or to use a tool for executing code first. It would be tricky to mix the code tool in with the retrieval step, since that has two implementation approaches already (cloud-based agentic retrieval or the in-code query rewriting step).

What do you think about that sequencing of the calls?

We do not yet use an agent framework in this repo, due to it being unclear which framework to use. There will soon be another option for a Microsoft agent framework to use which I'm interested in using this repo - should be available in a few weeks. I think the first step would be to port to that framework. Then I'm hoping it'll be easier for us to optionally add on a tool like execute_code. Let me also find the Bicep for the Container Apps pools to make sure we can easily provision that.

It'd be helpful to hash out a design now. We'd want it to be an optional feature, enabled via an azd environment variable, that doesn't over-complicate the default flow. That's always the hard part of adding features, I find - figuring out how to avoid over complicating the default flow.

Let me know what you think.

[taylorn-ai]

@pamelafox
My version of the app has an ungrounded mode (i.e. no RAG), so it was easier for me to implement. I actually created a new approach, with a custom run_with_streaming method to handle the tool calls back and forth, and it seems to work well. However, I think using LangChain/LangGraph might have been more robust, but I was hesitant to add more dependencies.