How handle long conversation history

[cahuja1992]

1. As history grows summarization of conversation would be useful.
2. If a question is just to format the previous response or anything that doesn't need further retrieval at that time shouldn't we be detecting if retrieval is needed?

[placerda]

Thanks @cahuja1992

1. As history grows summarization of conversation would be useful.

This makes sense, but would include one more call to the LLM, how long do you think it would make sense to start summarizing?

2. If a question is just to format the previous response or anything that doesn't need further retrieval at that time shouldn't we be detecting if retrieval is needed?

Now we have a triage function that does exactly this.

[cahuja1992]

@placerda As chat history is passed in the triage, there we will reach the token limits. So one idea is to as we reach 90% of the token limits, spawn a thread of chat summarization so that for the next question we can use the summary.

[cahuja1992]

@placerda As chat history is passed in the triage, there we will reach the token limits. So one idea is to as we reach 90% of the token limits, spawn a thread of chat summarisation so that for the next question we can use the summary.

@placerda Any thoughts on this approach? If we proactively keep on summarizing the conversation, then we do not even see significant differences in latency.

[placerda]

Hi @cahuja1992 this thread has been open for some time, but the topic is still valid. Since we are now relying on the AI Foundry Agent Service, conversation history is managed at the thread level, and the service builds context directly from the thread. It would be useful to run some tests with very long threads to see how this impacts response time and whether we occasionally hit the context window limit.

If that turns out to be the case, summarization could be a good option — likely combined with creating a new thread when reaching the limit and injecting a summary plus the most recent turns.

I’ll leave this issue open so we can analyze it further. In the meantime, would you be open to testing this scenario with the latest GPT-RAG release? The new version includes significant changes in orchestration that may affect how this behaves.

Thanks again for keeping this discussion going, your input is very valuable 🙂