Script allows sending 'infinite' amounts of text, but without context/memory

[CrazySwede78]

So looking through the script, basically, your code takes a text document, breaks it into pieces, matching tokens (And I am still trying to figure out how you get accurate token count for the openai API without using tiktoken or similar library) and sending each piece separately to the API )And with the ability to set a system message and/or prompt with each chunk) So yeah, in a way you are able to send near infinite texts to the API

But that does not change the underlying issue that the openai API models lack the conversationalbuffermemory that ChatGPT has, so each of those chunks would get treated and responded to without any inherent context or identification with any of the other chunks you send, so even if you would get replies, none of them would make any sense for the whole, as none of those would actually ever understand the full text sent, only the individual chunk sent in that call to it.

I mean, I guess i see the benefit of this script if you want to hit the rate limits of your gpt-4 API access, but other than that? Could you provide a sensible use-case for this script please? Also, for the sake of transparency and openness, I think you should mention the detail that the API can't connect the chunks and will only respond on an individual basis, as there seems to be some confusion regarding the true capabilities of your script.

[jwestonmoss]

if you are using the chatgpt endpoint and set a conversation token, it should be capable of continuing the same conversation within its own memory context. i have tested this in previous integrations and it works. haven't tried this project yet though.

but you are right. i glanced at the script and was trying to understand the value-add. perhaps some examples of a supported example usage would be good to put in the readme.

[microprediction]

I had the same thought. I believe the endpoint is stateless. Happy to be corrected.

[CrazySwede78]

if you are using the chatgpt endpoint and set a conversation token, it should be capable of continuing the same conversation within its own memory context. i have tested this in previous integrations and it works. haven't tried this project yet though.

but you are right. i glanced at the script and was trying to understand the value-add. perhaps some examples of a supported example usage would be good to put in the readme.

Yes, if you use something like conversationbuffermemory in the script, you can create a form of memory for the API, but it is done by sending the entire conversation back through the API after each completion, and is limited by the max_tokens of the engine, so 4K/8K/32K(comingsoon-tm-). You can never extend this memory beyond the API token limit for prompt/completion combined. So anything of any significant length will very quickly run up that memory, not to mention that each call would cost incrementally more for each call.

Having written a couple of scripts, trying different forms of memory and context methods, I found it to be reliable, but costly, and that is not counting the 32K model which would push the costs even higher.

I believe that open source llm models will very shortly be both readily available, have a lower 'bar' required to effectively train and use them, and as already shown in many open source llm's, very capable on a fraction of the computing power. That's when we will be able to start using near unlimited size of data,, limited by local compute power and effective memory storage solutions. For now, I have a tough time thinking of a use case for this method, as it is not very effective, highly limited and costly.

I still appreciate the authors idea and work though, and hope it will help inspire others!

[microprediction]

In my experience it helps to try to be explicit if you wish to carry state forward (e.g. this kind of thing)

One idea would be to ask for a summary, or compression, of previous steps. Though that is a bit orthogonal to the functionality provided here.