Integrate functionality from the atom-elixir alchemist-server version

[tonini]

@msaraiva it would be great if we could integrate the code/parser and the code/metabuilder (and the rest which is needed too) into the alchemist-server. Then I could start to refactor a lot of the server itself and also simplify the API a lot for example the definition and completion lookup for example. 🎉

[msaraiva]

@tonini I can start working on the integration in a couple of days. It would be great if we could eventually refactor the completion lookup or even reimplement it from scratch. During the whole process of creating atom-elixir, autocompletion was the hardest part to extend (and still is). I guess the reason is that it was originally designed only for iex. Right?

[tonini]

Sounds good to me :)

About the completion functionality. We used the autocompletion functionality served by elixir IEx.Helpers itself which we also improved with the help of José.

It would be great if you could write down some of the issues which you had, I'm really curious about these.

[msaraiva]

@tonini I'm currently adding a couple of features to the parser and also refactoring some parts. I expect to finish this by the end of the week. As soon I get everything working, I'll merge into my alchemist-server fork and create a PR.

Regarding the completion implementation, don't worry. As I wrote "we could eventually refactor the completion lookup". It's not a priority right now. I'll write down some of the issues I had when I have a chance ;)

Cheers.

[tonini]

@msaraiva These are great news :) Thanks a lot for all your effort.

[msaraiva]

@tonini Quick update:

There are two features I'm currently working on:

1. Introspecting behaviours information (callbacks, docs, ...), including those injected by use clauses)
2. Storing variables and attributes definition lines in the metadata

I'm resuming my work on these features today, so I hope I'll get something stable in the next few days. March was a really hard month for me to do any OSS work. Sorry about that.

Cheers.

[tonini]

Hey @msaraiva

This sounds great! 👍

Don't worry, just take your time.

Enjoy the weekend. 😄

[msaraiva]

@tonini Ok, let's get started ;)

msaraiva@9dd2331

[tonini]

@msaraiva this is amazing :D ❤️ 💛 💙

Btw, currently the alchemist-server also uses the complete autocompleter module because of the aliases usage. But since Elixir 1.2 we don't need that anymore, instead we could simple add the aliases to the application env when needed. -> https://github.com/elixir-lang/elixir/blob/master/lib/iex/lib/iex/autocomplete.ex#L175

[msaraiva]

@tonini I just took a look at the new autocomplete code and I afraid I'll still have to copy most of the code and then make my own changes. Here is the main issue:

• The only information the expand function returns is the hint and a list of suggestions as strings. As you can see in this screenshot, I introspect much more information, namely specs, doc summary, type of suggestion and the module where the function was originally defined.

The current implementation doesn't have any extension point where I can add what I need and, since I don't see iex ever using those extra info, I'll probably never be able to use this function directly.

Thoughts?

[tonini]

Hi @msaraiva

Sorry for my late response.

I think you're right about this. What do you think of implementing your autocomplete module for the alchemist-server? I guess people would really like it, I saw some people already talking about it in the slack and irc channel. :-) They love how atom-elixir does it. ;-)

[msaraiva]

Hi @tonini
/cc @mat-mcloughli

Months ago I started to remove all logic from atom-elixir and moved to a new pure Elixir lib (temporally called ElixirSense) that could be used not only by alchemist and atom but also by vscode or any other editor/tool. The main goal was to create a unified way to introspect information from code/beans and provide that information through a well defined API. All functions available are editor agnostic and exchange not only strings but also compound data types like lists and maps. Any editor specific implementation, e.g. formatting, can be done by the plugin/package's author easily. Here's an example of how Alchemist.API.Comp would look like:

defmodule Alchemist.API.Comp do

  @moduledoc false

  @spec request(String.t) :: no_return
  def request(args) do
    {{hint, buffer_file, line}, _} =  Code.eval_string(args)
    buffer = File.read!(buffer_file)

    ElixirSense.suggestions(hint, buffer, line)
    |> Enum.map(&format_suggestion/1)
    |> Enum.each(&IO.puts/1)
    IO.puts "END-OF-COMP"
  end

  defp format_suggestion(%{type: :variable, name: name}) do
    "#{name};var"
  end
  defp format_suggestion(%{type: :attribute, name: name}) do
    "#{name};attribute"
  end
  defp format_suggestion(%{type: :hint, value: value}) do
    "#{value};hint"
  end
  defp format_suggestion(%{type: :module, name: name, subtype: subtype, summary: summary}) do
    "#{name};module;#{subtype};#{summary}"
  end
  defp format_suggestion(%{type: :callback, name: name, arity: arity, args: args, origin: mod_name, summary: desc, spec: spec}) do
    "#{name}/#{arity};callback;#{args};#{mod_name};#{desc};#{spec}"
  end
  defp format_suggestion(%{type: :return, description: description, spec: spec, snippet: snippet}) do
    "#{description};return;#{spec};#{snippet}"
  end
  defp format_suggestion(%{type: type, name: func, arity: arity, args: args, origin: mod_name, summary: summary, spec: spec}) do
    "#{func}/#{arity};#{type};#{args};#{mod_name};#{summary};#{spec}"
  end
end

Note: Vscode and Atom will use a different formatter (JSON).

The code above actually works and I was using it in atom-elixir for my tests. As you can see, there's no need to do any parsing in the client and you don't need to pass any use/alias/import/behaviour information since the library will handle all that using the new parser. Here's the implementation of ElixirSense.suggestions/3:

  @spec suggestions(String.t, String.t, non_neg_integer) :: [Suggestion.suggestion]
  def suggestions(hint, code, line) do
    buffer_file_metadata = Parser.parse_string(code, true, true, line)
    %State.Env{
      imports: imports,
      aliases: aliases,
      vars: vars,
      attributes: attributes,
      behaviours: behaviours,
      module: module,
      scope: scope
    } = Metadata.get_env(buffer_file_metadata, line)

    Suggestion.find(hint, [module|imports], aliases, vars, attributes, behaviours, scope)
  end

So, basically the only thing we have to do is formatting the result of the available functions. Currently they are: docs/3, suggestions/3, definition/3 and a new function called signature/3 that can provide live information about the functions/params as you write (just like vscode's cmd+shift+space):

iex> code = ~S'''
...> defmodule MyModule do
...>   alias List, as: MyList
...>
...> end
...> '''
iex> ElixirSense.signature("MyList.flatten(par0, ", code, 3) 
%{active_param: 1,
  signatures: [
    %{name: "flatten", params: ["list"]},
    %{name: "flatten", params: ["list", "tail"]}]}

Other existing functions in atom-elixir like expand, quote and match can be easily moved to the new API in no time.

There are also other things halfway done like function find/refactor using xref and a new custom parser that will enable us to retrieve line+column information - last time I've checked, the default one could only retrieve the line, which is terrible and makes some of the functionality not as precise as it should be.

The bad news is that my last commit was about 3 months ago. Since then, I've had practically no time for OSS projects. The good news is that I will try to take a couple of days next week to finish the most important part so people can start using/testing and, depending on the feedback, we can finally release the first version.

So, let's see how it goes ;)

Cheers.

[mat-mcloughlin]

you missed an n its @mat-mcloughlin