Expose internal packages

[darksnow]

I've been creating a CLI tool for a client which will be used to automate certain workflows. Part of this work involves working with Lagoon. As such I had hoped to use machinery and quickly found that it doesn't expose a lot of the work done by the lagoon CLI tool. Looking here I was delighted to see some packages exported that did a lot of the work.

However, they are unusable in the current form. Some code in the /pkg/ directory references code from the /internal/ directory, so any project trying to use the exported packages from here, can't use the data types as they are protected.

The solution is quite simple but has far reaching consequences. I've forked this project and moved everything from internal to pkg, updated all code references and recompiled. The tool works exactly as it did before but now I can use the packages from my other project.

When I started my project I was copying chunks of code from lagoon-cli unto my own project. Where it called machinery or other code I imported the relevant packages as needed. There are obvious maintainability issue with that but it did work for trivial use cases.

I'm sure there were good reasons to protect the code in question inside the internal directory but reading it, it looks like it's structured as reusable packages. This worked well for me as it could be easily moved. An expansion of this idea could be to expose a few internal function so they can be used, but that's a different discussion.

[shreddedbacon]

The reason for it being in internal was for exactly this sort of thing, where an unstable API was not meant for external consumption as it was being developed. The CLI is not the best place to extract client libraries from either, this is why machinery exists. It is what we are going to be supporting long term, and eventually the CLI will just consume machinery as the client.

Can you give a better breakdown of what it is that machinery can't do for you though?

[darksnow]

Thanks for the quick response.

Just quickly, machinery lack the ability to login, unless I've missed something.

I've developing a CLI tool so my starting point was to look at this repo to see how you do it. My intent was to use machinery but the first thing my tool needs to do is refresh any access token and machinery does not do login.

I was then copying and pasting some code from lagoon-cli, which turns into a maintenance headache so I thought I'd import code from the pkg directory. As part of that work I called graphql.VerifyTokenExpiry() which takes a lagoon.Config.

While the graphql package looks like it could be reusable, it needs a type from internal so despite being exposed in pkg, it can't actually be used.

There's maybe a more nuanced discussion around moving that Config type out to the /pkg/lagoon to allow graphql to be used but when I looked at the code base I felt that things were structured neatly and the internal packages, especially lagoon seemed to fit into the exposed /pkg/lagoon/ so I took the easy option and merged them.

[shreddedbacon]

A large majority of what is in pkg and internal is going to go away, we don't plan to support it.

Machinery can do logins (or at least token retrievals https://github.com/uselagoon/machinery/blob/main/utils/sshtoken/sshtoken.go. But once you have a token, you may need to write your own wrapper depending on how your custom CLI works.

Machinery is pretty simple to use though once you have a token: https://github.com/uselagoon/lagoon-cli/blob/main/cmd/project.go#L283-L291 for example, retrieving a minimal project. Create a client lc from the machinery client, pass it the endpoint and token, then execute the query or mutation passing in the expected values and the client and the output is whatever the output of the query or mutation client function spits out.

What you're doing is not really something we are actually ready to support properly though. Machinery is in progress to be actually released (notice it is still a v0), and the CLI is in a transition phase (also it is a v0) to move away from the builtins to using machinery.

The sshtoken call could be executed whenever your CLI is called, there is no need to store the token really, but you can write your own storage method too, its just a token at the end of the day. The only reason to store it is to reduce the calls to the SSH endpoint to generate a token (ie multiple successive calls to the API in a short period of time).

We may include a configuration utility in machinery in the future, as we plan to eventually have lagoon-cli and lagoon-sync and any other tools we may build use the one set of configurations, and machinery is where that unified config would live.

[shreddedbacon]

Additionally, we are working on implementing redirect based authentication rather than using SSH retrieved tokens #309 as this is a first run, it lives in the CLI, but my goal would be to move the auth functionality into machinery in a similar manner to the sshtoken utility. Combine these two things with a config management from machinery (when it exists) and you've got the components to build whatever you want using an actually supported library.

[darksnow]

Thanks, that does clarify things.

I'm already using machinery to create a project and set environment variables but it seems I misunderstood how the login system works in the CLI. The SSH token is not as long lived as I thought, so perhaps I've over engineered this and I should get the token as required and just keep it in memory for the requests I need.

Thanks for the clarification, I'll move everything over to just using machinery.

[shreddedbacon]

it seems I misunderstood how the login system works in the CLI.

Easy to do, it isn't really the best method, but as the PR linked shows, we are looking at exposing the CLI to be able to login via the same authentication provider that the UI uses, which makes handling refreshing tokens much much easier in the long run, you'd just need somewhere to store and retrieve it from.

The SSH token is not as long lived as I thought, so perhaps I've over engineered this and I should get the token as required and just keep it in memory for the requests I need.

Yes, this depends on the provider that hosts Lagoon, if you're using amazee.io hosted Lagoon, our tokens are short lived but I'm not really sure of the backstory on why, but we can probably review this setting to increase the lifetime.

Everything you've mentioned is super valuable feedback though, and it is all stuff we've definitely thought about as we move more and more into machinery, you just got ahead of us 🚀

[darksnow]

Thanks for that, always good to be useful.

I'm currently building against my forked version of lagoon-cli but I've changed the code that creates projects and env vars to just use machinery, working on tokens now.

Thanks for the clarification, I'll keep an eye on machinery as you make changes.