Usability issues with the inline completer

[jtpio]

When trying #15 locally with MistralAI, the inline completer seems to be "off" most of the time, making it difficult to use:

I believe this was already noticed in #8 and may be the same issue.

It's not clear yet whether it's because the Mistral API is slow to respond, because of its 1 req/s limit or how the inline completer is currently set up.

[brichet]

I think it's a mix of all of these.

• the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).
• currently we trigger the API as soon as a change occurs, and the following changes during the next second are not taken into account.

Maybe we could make it more reliable with a better use of the inline completer and the throttler.
I'm thinking about:

• not triggering the API on each change. We could wait a small timeout (200ms) to expect the user is not typing anymore.
• after having trigger the API, keeping track of the following changes would allow us to cancel the next suggestion (which is probably no longer valid) and trigger another as soon as possible.

[jtpio]

the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).

2s sounds like a lot. Wondering if we can already see some improvements with other providers like in #17 for example. If that's the case maybe there isn't much we can do about the Mistral provider.

[brichet]

the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).

2s sounds like a lot. Wondering if we can already see some improvements with other providers like in #17 for example. If that's the case maybe there isn't much we can do about the Mistral provider.

As far as I know, Groq does not provide a completion model out of the box (not with langchain at least).

[jtpio]

OK, we could try with OpenAI then.