Model Context Protocol

[gaborvar]

Has the team thought about leveraging MCP? https://www.anthropic.com/news/model-context-protocol
Support for MCP is growing, including in Visual Studio. My scenario required adding tool calls to the sample app code base but as the integration parties may include other LLMs, MCP seems to be a more manageable path forward rather than calling them with custom tool_calls. (some may call it a step towards agentic)

[pamelafox]

I made a local MCP server for the repo as a demo, but I don't think that's what you're describing. What sort of MCP integration are you imagining?

[gaborvar]

The repo demonstrates brilliantly many of the RAG approaches in the pre-agentic era. If you ask the question what is the next level beyond the existing RAG approaches the answer might be:

1. Intelligently choose from among the existing approaches based on the task by an orchestrator agent.
2. Apply LLM based preprocessing in the document retrieval step in the RAG search. This may seem contradicting the purpose of RAG (controlled fact database), however if preprocessing of the search results is set up carefully you may retain factuality while adding more intelligence and versatility by applying LLMs in this step. This is a generalisation of the existing "STEP1" where you apply a model to make the search step more targeted by formulating the search string.
3. Invoke additional models to execute certain tasks. For example quality of the output can benefit from an LLM further processing the output of the chat. Example scenario: if the chatbot decides that it reached its own limits it may pass on the conversation to a human agent. Context and status of the conversation can be summarised by an LLM for the human agent.

Any of these evolution paths can be thought about as an evolution of the existing architecture towards an agentic future.
MCP may be helpful to identify and call the specific LLM.
It is an interesting proposition if the existing class based implementation (which is a great idea) of the different RAG approaches could evolve to host further approaches beyond static RAG: MCP based mediation of available LLMs, each implemented in an approach class.

The repo can be a great frontend and UI for an agentic architecture.

[pamelafox]

We are certainly thinking of an agentic future, with more tool use, planning, and reflection.
@mattgotteiner just posted a blog post with some reflection experiments: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/bonus-rag-time-journey-agentic-rag/4404652
We want to put a simplified reflection step in this repo.

We're also separately looking into a more agentic approach to the query planning as well, we hope to have some drafts of that in the next few months.

Right now, since we're only using the openai SDK, we will have to implement flows at a fairly low-level. We haven't determined yet whether we need to move to an SDK like openai-agents or semantic-kernel to make it easier to do tool looping, sub-agent delegation, MCP server integration, etc.
I had a stream yesterday about various agent SDKs available for Python:
https://www.youtube.com/watch?v=LflBM3ntUSY

Great topic for discussion!

Also curious if there are any actions you'd want the app to do on behalf of a user, like emailing or storing in drive. I'm experimenting with that as well.