Add completion provider customization

[s-kostyaev]

Added ellama-completion-provider custom variable to specify the LLM provider for completions. Updated ellama-stream call in ellama-complete-impl to use this new provider and modified the filter function to handle prefix trimming correctly.

Fix #275

[tusharhero]

Regression?

[tusharhero]

Why did you remove all of the keybindings? Some of which were useful
IMO. I think you just asked to remove system prompts from blueprints
keymap and add them to the transient menu, right?

Details

diff --git a/ellama-blueprint.el b/ellama-blueprint.el
index 7210061..896e6f8 100644
--- a/ellama-blueprint.el
+++ b/ellama-blueprint.el
@@ -44,14 +44,19 @@
   (ellama-transient-set-system-from-buffer)
   (kill-buffer (current-buffer)))
 
+;;;###autoload
+(defun ellama-blueprint-chat-with-system-kill-buffer ()
+  "Chat with the system message from the current blueprint and kill the buffer."
+  (interactive)
+  (let ((buf (current-buffer)))
+    (ellama-chat-with-system-from-buffer)
+    (kill-buffer buf)))
+
 (defvar-keymap ellama-blueprint-mode-map
   :doc "Local keymap for Ellama blueprint mode buffers."
   :parent global-map
-  "C-c C-c" #'ellama-send-buffer-to-new-chat-then-kill
-  "C-c C-k" #'ellama-kill-current-buffer
-  "C-c y" #'ellama-blueprint-set-system-kill-buffer
-  "C-c c" #'ellama-blueprint-create
-  "C-c v" #'ellama-blueprint-fill-variables)
+  "C-c C-c" #'ellama-transient-blueprint-mode-menu
+  "C-c C-k" #'ellama-kill-current-buffer)
 
 (defvar ellama-blueprint-font-lock-keywords
   '(("{\\([^}]+\\)}" 1 'font-lock-keyword-face))
@@ -66,8 +71,28 @@
   :group 'ellama
   (setq font-lock-defaults '((("{\\([^}]+\\)}" 1 font-lock-keyword-face t))))
   (setq header-line-format
-	(substitute-command-keys
-	 "`\\[ellama-send-buffer-to-new-chat-then-kill]' to send `\\[ellama-kill-current-buffer]' to cancel `\\[ellama-blueprint-set-system-kill-buffer]' to set system `\\[ellama-blueprint-create]' to create new blueprint `\\[ellama-blueprint-fill-variables]' to fill variables")))
+	(concat
+	 (propertize
+	  (substitute-command-keys
+	   "`\\[ellama-transient-blueprint-mode-menu]' to continue")
+	  'help-echo "mouse-1: show menu"
+	  'mouse-face 'header-line-format
+	  'face 'ellama-context-line-face
+	  'keymap (let ((m (make-sparse-keymap)))
+		    (define-key m [header-line mouse-1] #'ellama-transient-blueprint-mode-menu)
+		    (define-key m [mode-line mouse-1] #'ellama-transient-blueprint-mode-menu)
+		    m))
+	 " "
+	 (propertize
+	  (substitute-command-keys
+	   "`\\[ellama-kill-current-buffer]' to cancel")
+	  'help-echo "mouse-1: kill buffer"
+	  'mouse-face 'header-line-format
+	  'face 'ellama-context-line-face
+	  'keymap (let ((m (make-sparse-keymap)))
+		    (define-key m [header-line mouse-1] #'ellama-kill-current-buffer)
+		    (define-key m [mode-line mouse-1] #'ellama-kill-current-buffer)
+		    m)))))
 
 (defvar ellama-blueprint-buffer "*ellama-blueprint-buffer*"
   "Buffer for prompt blueprint.")
diff --git a/ellama-transient.el b/ellama-transient.el
index 7ba878d..6213244 100644
--- a/ellama-transient.el
+++ b/ellama-transient.el
@@ -301,6 +301,22 @@ Otherwise, prompt the user to enter a system message."
     ("n" "New blueprint" ellama-blueprint-new)]
    ["Quit" ("q" "Quit" transient-quit-one)]])
 
+;;;###autoload
+(transient-define-prefix ellama-transient-blueprint-mode-menu ()
+  ["Blueprint Commands"
+   ["Chat"
+    ("c" "Send to chat" ellama-send-buffer-to-new-chat-then-kill)]
+   ["System Message"
+    ("s" "Set system and chat" ellama-blueprint-chat-with-system-kill-buffer)
+    ("S" "Set system and quit" ellama-blueprint-set-system-kill-buffer)]
+   ["Create"
+    ("C" "Create new blueprint from buffer" ellama-blueprint-create)]
+   ["Variables"
+    ("v" "Fill variables" ellama-blueprint-fill-variables)]
+   ["Quit"
+    ("k" "Kill" ellama-kill-current-buffer)
+    ("q" "Quit" transient-quit-one)]])
+
 ;;;###autoload
 (transient-define-prefix ellama-transient-main-menu ()
   "Main Menu."
diff --git a/ellama.el b/ellama.el
index d4b70a4..27b70ef 100644
--- a/ellama.el
+++ b/ellama.el
@@ -1636,6 +1636,20 @@ the full response text when the request completes (with BUFFER current)."
 			 :filter (when (derived-mode-p 'org-mode)
 				   #'ellama--translate-markdown-to-org-filter)))))))
 
+;;;###autoload
+(defun ellama-chat-with-system-from-buffer ()
+  "Start a new chat session with a system message created from the current buffer."
+  (interactive)
+  (let* ((prompt (read-string "Prompt: "))
+	 (content (buffer-substring-no-properties (point-min) (point-max)))
+	 (system (if (derived-mode-p 'org-mode)
+		     (ellama-convert-org-to-md content)
+		   content)))
+    (ellama-chat
+     prompt
+     t
+     :system system)))
+
 (defvar ellama-context-global)
 
 ;;;###autoload

[s-kostyaev]

Regression?

Nope. Try it.

[s-kostyaev]

Too many keys to show it in header line. You can call all of it from blueprint mode transient menu

[tusharhero]

I see. You have moved them to the transient menu, I guess? Please add
an keybinding or transient menu option to create new variation of an
existing blueprint. Also, I think there needs to be an infix transient
to send the blueprint as a system prompt, as I suggested before.

**** Blueprints Support Enhancements
- Added ~ellama-blueprint-run~ function to run a chat with an LLM using a
  specified blueprint and optional pre-filled variables.
- Added documentation for blueprints in the README file.
- Added a new transient prefix command ~ellama-transient-blueprint-menu~ for
  managing blueprint-related commands, including creating blueprints from buffer
  or as new ones, and chatting with selected blueprints.
- Added a main menu option for chatting with blueprints.
- Added custom variable ~ellama-blueprints~ to store user-defined blueprints.
- Created commands ~ellama-create-blueprint~ and ~ellama-new-blueprint~ for
  creating new blueprints from an existing one and from scratch.

[tusharhero]

Regression?

Nope. Try it.

The coloring is off too

Org mode capture buffer for example.

[s-kostyaev]

Regression?

Nope. Try it.

The coloring is off too Org mode capture buffer for example.

I will fix it

[s-kostyaev]

Please add
an keybinding or transient menu option to create new variation of an
existing blueprint.

C in transient menu

[s-kostyaev]

Also, I think there needs to be an infix transient
to send the blueprint as a system prompt, as I suggested before.

s in transient menu

[tusharhero]

Please add
an keybinding or transient menu option to create new variation of an
existing blueprint.

C in transient menu

Community blueprint?

[s-kostyaev]

Please add
an keybinding or transient menu option to create new variation of an
existing blueprint.

C in transient menu

Community blueprint?

No, in blueprint mode transient menu, C-c C-c C

[tusharhero]

Wow amazing! 🤩 👍

Please add
an keybinding or transient menu option to create new variation of an
existing blueprint.

C in transient menu

Community blueprint?

No, in blueprint mode transient menu, C-c C-c C

[Hi-Angel]

Figured I'd CC @dgutov, in case they might want to review since it's about company completion. The discussion above is unrelated to company, it's instead here.

[tusharhero]

I doubt its specifically related to company, but they could still provide some guidance generally.

Figured I'd CC @dgutov, in case they might want to review since it's about company completion. The discussion above is unrelated to company, it's instead here.

[s-kostyaev]

I doubt its specifically related to company, but they could still provide some guidance generally.

Figured I'd CC @dgutov, in case they might want to review since it's about company completion. The discussion above is unrelated to company, it's instead here.

It works for me in corfu. But sometimes nothing of generated completions matches prefix.

[dgutov]

ellama-generate-completion-list seems like a "heavy" function.

Since it does some non-trivial work, it would be better to return a reference to a function that conforms to a "completion table" interface, instead of calling it right away.

[dgutov]

while-no-input returns t when interrupted, so this seems not ideal. UIs would try to ignore errors, of course, but you might still see them in Messages.

For company specifically, you can try the :company-use-while-no-input t property instead. Corfu just aborts all completion tables on input, though, so it would be compatible with it anyway.

[Hi-Angel]

Thank you, works great! For posterity, I'm evaluating this small configuration, using qwen2.5:3b:

Details

(progn
  (require 'ellama)
  (require 'llm-ollama)
  (setopt ellama-completion-provider
	      (make-llm-ollama
	       :chat-model "qwen2.5:3b"
	       :default-chat-non-standard-params
	       '(("num_ctx" . 32768))))

  (setq-local completion-at-point-functions '(ellama-complete-at-point)))

I noticed two odd usability inconveniences though:

1. When I get completion and I press Backspace to delete text, I get visible lags.

2. After completion appears and you continue typing, completion disappears.

   The usual case for completion is so you type, and you get a pop up suggestion, and you're like "Oh yeah! That's what I wanted to write!" and you chose it. IOW, you don't wait for completion, you're just typing text, and at some point you get completions that happen to be on point and you chose them.

   I understand LLMs are slow, but we don't have to ask it completion on every letter. Instead when a model replies it may be useful not to drop its message, but instead check if the prefix that already was written matches what's suggested by model, and only send another request if no completions match.

   So, like, how it works right now: every time I type a letter and company-idle-delay times out, Ellama sends another request without waiting for reply to previous one.

   How it could work:

   1. Once request sent, save start-pos, end-pos points in the buffer from which text to send was extracted
   2. Avoid new requests till reply was processed
   3. Upon getting a reply:

      1. If

         (or (> start-pos end-pos)
             (> (- end-pos start-pos) some-large-number))

         then drop completion. That just checks if a user moved a point elsewhere, to avoid unnecessary processing.

      2. Extract text between start-pos and (point) and try as a prefix to the completions. Filter-in ones that match. If no matches, drop the completion completely and send another request.

What do you think?

[s-kostyaev]

What do you think?

If we want to avoid lags we need to rewrite completions generation to async instead and manage state. Completions match checks will works on done only, so it will match rarely. Not sure if it will be useful. And we need to carefully handle all edge cases during state management. Sounds scary to me 🙂

[Hi-Angel]

Alright, let me use the current implementation for a day tomorrow for some text buffers, just to see how it feels/goes.

[Hi-Angel]

Well, yeah, in the current state I confirm it's not useful, because Ellama sends many requests and generating a single reply takes about 10 seconds (presumably due to the amount of requests). But please don't close, I want to take a look on a weekend at pre-existing state of art in other editors.

[tusharhero]

But please don't close, I want to take a look on a weekend at
pre-existing state of art in other editors.

I believe that completing this PR will require considerable time.
Therefore, we can leave it open until its completion.

@s-kostyaev Could you please create a new branch to merge the commit that resolves
issue #275? This may necessitate some advanced Git operations.

[tusharhero]

@s-kostyaev see #284

[Hi-Angel]

Thank you! So, I'm trying it out, but this time I set up ollama on a remote server in LAN (because I figured my laptop is too weak). So, when I'm trying to initialize it with this code:

(progn
  (setq-default ellama-transient-host "8.8.8.8") ; arbitrary ip for testing purposes
  (require 'ellama)
  (require 'llm-ollama)
  (setopt ellama-completion-provider
	      (make-llm-ollama
	       :chat-model "qwen2.5:3b"
	       :default-chat-non-standard-params
	       '(("num_ctx" . 32768))))

  (setq-local completion-at-point-functions '(ellama-complete-at-point)
              company-backends '(company-capf)))

I get curl error code 7: Failed to connect to host.. I realize the 8.8.8.8 here is kinda fake, but it behaves same way with an actual IP with Ollama running. I'm not sure what curl is trying to connect to — bpftrace shows the process being launched is /usr/bin/curl --no-buffer --config -.

P.S.: also, there seems to be a bug with error reporting, because the error isn't actually being shown, instead Ellama complains about invalid arguments. The error instead can be seen in stacktrace. This one is unrelated to this branch, I see same behavior on master while I've been testing this.

[tusharhero]

A good tool to use is SSH tunneling in this case.

[Hi-Angel]

But what curl is trying to do?

[tusharhero]

But what curl is trying to do?

I don't understand what's happening here, am I missing some context?

curl seems to be complaining about not being able to connect to
8.8.8.8.

[Hi-Angel]

But what curl is trying to do?

I don't understand what's happening here, am I missing some context?

curl seems to be complaining about not being able to connect to 8.8.8.8.

As mentioned here, it behaves same way with an actual IP with Ollama running. I am unclear what curl is trying to do here.

[tusharhero]

But what curl is trying to do?

I don't understand what's happening here, am I missing some context?
curl seems to be complaining about not being able to connect to 8.8.8.8.

As mentioned here, it behaves same way with an actual IP with Ollama running. I am unclear what curl is trying to do here.

Are you able to curl <actual ollama instance>?

[Hi-Angel]

That was a good question… the answer is "no" and I'm confused. For the context: the reason I didn't consider curl is attempting to connect to ollama is because while experimenting here I saw a POST being sent to it. Seeing "connection refused" from curl right now makes no sense to me.

Okay, sorry for the noise, I'll have to dig into it.

[tusharhero]

That was a good question… the answer is "no" and I'm confused. For
the context: the reason I didn't consider curl is attempting to
connect to ollama is because while experimenting
here I saw a POST
being sent to it. Seeing "connection refused" from curl right now
makes no sense to me.

Okay, sorry for the noise, I'll have to dig into it.

Did you also put the correct port in the test command, like curl serverip:11434?

[Hi-Angel]

Yeah, it seems ollama by default just blocks requests not coming from localhost. My fault here, I was confused due to seeing a POST /api/generate in the logs during the experiments. No idea where could've it come from, can't reproduce that anymore.

[tusharhero]

Yeah, it seems ollama by default just blocks requests not coming
from localhost.

That is nice, and probably because of this.

[s-kostyaev]

Thank you! Soo… okay, it seems to change completions when I enter new sentences. Looks good, I guess I'll test it out with an actual text 😊

What's your experience? Is it useful?

[Hi-Angel]

Thank you! Soo… okay, it seems to change completions when I enter new sentences. Looks good, I guess I'll test it out with an actual text 😊

What's your experience? Is it useful?

Sorry, I was trying to get it to work at different times during the week, but something always went wrong, and I had very little time to debug it because it is the end of the quarter, so I had lots of stuff to do. I found that Ubuntu Ollama snap is buggy because it doesn't pass env. variables through, which I reported via an email to the maintainer today (apparently snap has no packages bugtracker 🤷‍♂️), got no reply; found a workaround via launching a snap shell and then inside it the ollama serve; and now my remote ollama instance works fine and I tried to connect Emacs to it, but Ellama fails with some vague error about invalid arguments, and I didn't have time to dig further just yet. Still on it 😅

[Hi-Angel]

Sorry for silly question, but where is curl or plz called from? I'm trying to figure out why curl is returning an error despite Ellama being accessible at ellama-transient-host, I get this stack:

 […stack skipped…]
  plz--respond(#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n")
  apply(plz--respond (#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n"))
  timer-event-handler([t 26605 33859 376261 nil plz--respond (#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n") nil 439000 nil])

…and I try to grep for plz or curl over the branch but I see no relevant results at all.

[s-kostyaev]

Sorry for silly question, but where is curl or plz called from? I'm trying to figure out why curl is returning an error despite Ellama being accessible at ellama-transient-host, I get this stack:

 […stack skipped…]
  plz--respond(#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n")
  apply(plz--respond (#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n"))
  timer-event-handler([t 26605 33859 376261 nil plz--respond (#<process plz-request-curl> #<buffer  *plz-request-curl*> "exited abnormally with code 7\n") nil 439000 nil])

…and I try to grep for plz or curl over the branch but I see no relevant results at all.

It is called by llm library

[Hi-Angel]

So, I made it to work. A few problems I noticed offhand:

1. If ollama returns an error, it seems Ellama won't attempt to contact it anymore. Easy to reproduce by e.g. not launching ollama serve, then typing text so that capf triggers a curl error, and then launching ollama serve. The only way to Ellama try to contact it again it seems is by restarting Emacs.

2. Design situation: the problem with curl I was fighting above came down to provider needing to have param (… :host "the-ip" …). Given that Ellama provides ellama-transient-host configuration variable for a host, it would seem logical if :host was using this variable and not just defaulting to "localhost". A user would never guess they have to change the host in multiple places.

   Idk how fixable it is, but if it isn't, I think it's at least worth mentioning in ellama-transient-host documentation.

I'll try to test it IRL tomorrow, but from what I see the completion it seems almost never appears.

[tusharhero]

So, I made it to work. A few problems I noticed offhand:

1. If ollama returns an error, it seems Ellama won't attempt to
   contact it anymore. Easy to reproduce by e.g. not launching
   `ollama serve`, then typing text so that capf triggers a curl
   error, and then launching `ollama serve`. The only way to
   Ellama try to contact it again it seems is by restarting
   Emacs.

Can you try it on the master branch? I just had this situation and it
just works™ without restarting Emacs.

[Hi-Angel]

Does completion sends whole buffer? I'm judging by this warning:

level=WARN source=runner.go:129 msg="truncating input prompt" limit=32768 prompt=59100 keep=4 new=32768

and a long time for Ollama to reply. I think it would be useful to perhaps limit the context to current line or to line above… Or maybe to the number of symbols passed as num_ctx, although I presume the latter may be complicated.

[s-kostyaev]

Does completion sends whole buffer? I'm judging by this warning:

level=WARN source=runner.go:129 msg="truncating input prompt" limit=32768 prompt=59100 keep=4 new=32768

and a long time for Ollama to reply. I think it would be useful to perhaps limit the context to current line or to line above… Or maybe to the number of symbols passed as num_ctx, although I presume the latter may be complicated.

It sends active region or current buffer up to the point.

[s-kostyaev]

Does completion sends whole buffer? I'm judging by this warning:

level=WARN source=runner.go:129 msg="truncating input prompt" limit=32768 prompt=59100 keep=4 new=32768

and a long time for Ollama to reply. I think it would be useful to perhaps limit the context to current line or to line above… Or maybe to the number of symbols passed as num_ctx, although I presume the latter may be complicated.

We can't rely on num_ctx in general. It can be set outside of Emacs. It works for ollama only.

[Hi-Angel]

Does completion sends whole buffer? I'm judging by this warning:

level=WARN source=runner.go:129 msg="truncating input prompt" limit=32768 prompt=59100 keep=4 new=32768

and a long time for Ollama to reply. I think it would be useful to perhaps limit the context to current line or to line above… Or maybe to the number of symbols passed as num_ctx, although I presume the latter may be complicated.

It sends active region or current buffer up to the point.

I see. Well, there couldn't be an active region while you're typing, so it comes down to always sending the full buffer. I'm not sure it's okay though, because text (and code for that matter) buffers can be very large. E.g. ellama.el file alone is 88k. The the text buffer where I'm testing completion is 293k. .

[s-kostyaev]

Does completion sends whole buffer? I'm judging by this warning:

level=WARN source=runner.go:129 msg="truncating input prompt" limit=32768 prompt=59100 keep=4 new=32768

and a long time for Ollama to reply. I think it would be useful to perhaps limit the context to current line or to line above… Or maybe to the number of symbols passed as num_ctx, although I presume the latter may be complicated.

It sends active region or current buffer up to the point.

I see. Well, there couldn't be an active region while you're typing, so it comes down to always sending the full buffer. I'm not sure it's okay though, because text (and code for that matter) buffers can be very large. E.g. ellama.el file alone is 88k. The the text buffer where I'm testing completion is 293k. .

We can introduce an option to control how large part should we send for completion.