Adding Excel Copilot example

[yanisoln]

Hey!
I know this idea wasn't listed in the original issue, but after doing a bit of research and building a prototype, I found it both fun and useful, so I thought it was worth sharing here.

This PR introduces a basic implementation of the “Excel Copilot” example I mentioned here, using Terminator + Tauri.

It can read a PDF, extracts structured data, and fills in the Excel UI live through the SDK toolcalls. I focused on making the interaction smooth and visible to the user to highlight the SDK’s potential in "Copilot-style" automation.

It’s probably not perfect, and I’m sure there are things that could be improved, feel free to let me know what you'd like to see changed or adjusted.
Happy to iterate or adapt it more closely to your vision!

Here is a video showing it in action :

terminator_excel.mp4

/claim #24

[louis030195]

thats pretty cool

something i learned with excel / google sheet also is that you can paste tables formatted in TSV format so you can create a tool for the LLM to paste TSV in the spreadsheet to fill stuff quite fast

another idea i had with excel / sheets is to create a tool for the terminator LLM to actually use Microsoft Copilot or Sheets Gemini lol, maybe for some use cases

also did not read the full code but i recommend maintaining each app/window full tree in the prompt so the LLM has full context and can use IDs as selector to interact with specific element

wdyt, any thoughts on this?

also when merging would be cool to be able to provide a .exe / .dmg for ppl to try out this easily (i can help on this)

[yanisoln]

something i learned with excel / google sheet also is that you can paste tables formatted in TSV format so you can create a tool for the LLM to paste TSV in the spreadsheet to fill stuff quite fast

I think that's actually a good idea to let the LLM create data faster and save on tokens, will do some research and try to implement it.
As for Google Sheets, I actually tried making it work too, but unfortunately the cells aren’t exposed via UIA, so it’s really tricky to target or interact with them directly in the same way as the Excel desktop app.

another idea i had with excel / sheets is to create a tool for the terminator LLM to actually use Microsoft Copilot or Sheets Gemini lol, maybe for some use cases

This could probably be useful for trickier things, like asking directly to copilot to use style or conditional style on the sheet, or maybe add graphs. I think it's worth exploring for some cases.

also did not read the full code but i recommend maintaining each app/window full tree in the prompt so the LLM has full context and can use IDs as selector to interact with specific element

The good thing with Excel is that the cells have their name as automation id (A1, C5, D8, etc...) so it's really easy to target them, so unless we have to interact with the ribbon (which is not needed since Excel have shortcuts for barely anything), I think including the whole tree would mostly just burn through tokens for not much gain.

also when merging would be cool to be able to provide a .exe / .dmg for ppl to try out this easily (i can help on this)

Not a problem, I can provide that

[louis030195]

i guess for google sheets its easy to paste data using TSV but to read you can use gemini that tool call the actual gemini of sheets lol

[yanisoln]

Okay, so I added interaction with MS Copilot (for styling, conditional formatting, etc.), and google sheet support by… asking Gemini to talk to Gemini (lol). That said, Gemini in Google Sheets is quirky, it struggles with follow-ups and sometimes does random stuff or write result in the chat instead of the sheet (as seen in the second video).

Also I added TSV pasting from the LLM to speed things up, and packaged a .exe for Windows that should (normally) work anywhere, as long as webview2 is installed, let me know if that's not the case.

Finally, tried building a .dmg for macOS, but it doesn’t work yet, the macOS app will crash as soon as it try to interact with the Excel window, I will probably open another PR to add proper macOS support and push the prototype further when i'll find some time.

2025-06-03.19-40-13.mp4

2025-06-04.03-54-22.mp4

I'll also add some UI improvement and fix some UI related bugs.

Let me know what you think!

[louis030195]

some issues:

• api key keep being lost (i do a few message then it's lost)

• cannot scroll up
• sometimes i could not see the input chat when i was in split screen (some responsive issue)

could not test anything else, did not work :/

it looks really promising beside that

[yanisoln]

• api key keep being lost (i do a few message then it's lost)

Okay that's weird, i'm going to look into it, same for the scrolling up issue, also are you testing from the code directly or the binary in excel-copilot/bin ?

• sometimes i could not see the input chat when i was in split screen (some responsive issue)

I think it's because the app consider you did not configure Gemini yet bc you got an error on your last message, when it's the case the input text bar does not appear, so that is linked with the previous bug
edit: actually you are right, there is some responsiveness issues.

Will look into it and be back with fixes.

[yanisoln]

Update :
I improved the responsiveness of the app (especially in split-screen mode), so it should now behave better.

The bug with scrolling should be gone.

Regarding the issue with the API key being lost, I wasn’t able to reproduce it on my side. I managed to send several messages with Gemini without any problem. I’m not exactly sure what caused the issue, it might be related to how the API key was obtained or configured in your environment.

That said, I added a second provider as a fallback: you can now use OpenAI’s API (GPT-4.1 mini) if needed. It should offer a more stable alternative in case Gemini runs into issues.

Let me know how it goes if you test it again! 🙌

[Anshgrover23]

@yanisoln can u add a new video, by using the highlight method as it is becoming difficult to identify the difference b/w your hand drawn changes and the terminator changes(automatic cursor changes or clicks) lol, happy to review more after that.

[yanisoln]

@yanisoln can u add a new video, by using the highlight method as it is becoming difficult to identify the difference b/w your hand drawn changes and the terminator changes(automatic cursor changes or clicks) lol, happy to review more after that.

Hey! Thanks for the feedback. But i'm not 100% sure I understand what you're asking. Just to clarify, I didn't touch my mouse or keyboard at all during the videos while the app was working (except between prompts, of course). Everything was triggered by the LLM with toolcalls powered by the SDK.

[Anshgrover23]

@yanisoln try running this example, to know how highlight works. https://github.com/mediar-ai/terminator/blob/main/examples/gmail_automation.py

[yanisoln]

@yanisoln try running this example, to know how highlight works. https://github.com/mediar-ai/terminator/blob/main/examples/gmail_automation.py

Here you go 👍 I also added extra delays between interactions, to make it even more easier to see what is happening.

2025-06-07.21-47-32.mp4