Snow chibi git repo siblings

[Retropikzel]

This pull request adds similar (git (url ...)) field functionality to repository siblings as #1042 added to packages. It allows using git repositories as snow-fort repositories. Example of one here https://codeberg.org/retropikzel/snow-chibi-testing and main repository file using it here https://retropikzel.neocities.org/s/repo.scm.

To test these repositories you can run snow-chibi --use-curl --host=https://retropikzel.neocities.org search foo and it should find the (foo bar) library that is in the git repository. To install the test package run snow-chibi --use-curl --host=https://retropikzel.neocities.org --impls=chibi install foo.bar. The package exports one procedure called baz, which should output "Bazinga". Here is test program for copypasting:

(import (scheme base)
        (foo bar))

(baz)

The pull request also adds new command called git-index. It works little bit like regular index command, except it should be run inside a git repository. It outputs the packages into file called "snow-fort-repo.scm" and adds the current git hash, and tag if present, to the given package or packages. The git repository can then be added as a sibling to snow-fort repository and packages are automatically searched from "snow-fort-repo.scm" file.

Example usage, (foo bar) git repositorys makefile:

VERSION=1.0.0

package:
        snow-chibi package --version=${VERSION} --authors=Retropikzel foo/bar.sld

index:
        snow-chibi git-index foo-bar-${VERSION}.tgz

And the snow-fort-repo.scm now looks like:

(repository
  (package
    (git
      (hash "e4cd6ad91c31e1efdbe02cce3980214160821556")
      (url "https://git@codeberg.org/retropikzel/snow-chibi-testing.git"))
    (authors "Retropikzel")
    (version "1.0.0")
    (library
      (name
        (foo bar))
      (path "foo/bar.sld")
      (foreign-depends)
      (depends
        (scheme base)
        (scheme write)))))

This pull request does not add support for the main repository to be a git repository. It also should not disturb any existing type of use of snow-chibi.

[ashinn]

Thank you very much for working on this!

[Retropikzel]

Simplified the git url fixing based on fixes you made for issue #1065.

[Retropikzel]

I'm adding support for git siblings in snow-fort and during that I noticed that the updated field is added to package when uploaded, so I'm adding that to the git-index command.

[ashinn]

This becomes a standard protocol - people are expected to include this file in their git repos - so we should consider the name carefully. Also it's used elsewhere so it should be defined as a top-level variable and reused.

Maybe snow-repo.scm is better since it's shorter. Another option is .snow-repo.scm to be less obtrusive to non-snow uses.

Another option is to not define a whole repository but just commit individual .pkg files to git. We could even reference git URL's directly in the downstream dependencies without any packaging, but would have no way to resolve unknown chained dependencies.

[Retropikzel]

I agree it should be thought trough carefully.

Guix uses .guix-channel, so .snow-repo.scm I think would work nicely.

I think the siblings idea is too good to be thrown out. So when thinking about individual package files perhaps a combined approach could work?
If there is a snow/repo.scm for repository metadata. Name, url, siblings and in the future anything that is needed can go there. Packages could go into snow/pkgs/. Then using using my monorepo as example https://codeberg.org/retropikzel/scheme-libraries which has libraries like (retropikzel cgi), (retropikzel csv) and so on. Lets say I package them with snow-chibi git-package --version=1.0.0 retropikzel/cgi.sld and same for csv. Would directory snow/pkgs then have files retropikzel-cgi-1.0.0.scm and retropikzel-csv-1.0.0.scm? Each containing single package definition. Or would it contain files retropikzel-cgi.scm and retropikzel-csv.scm each containing list of package definitions for each version?

"We could even reference git URL's directly in the downstream dependencies without any packaging, but would have no way to resolve unknown chained dependencies."

The combined way could solve the problem of not finding dependencies. I could have all the repositories that have my packages dependencies in my snow/repo.scm as siblings. Perhaps you could show me some examples of what you mean? How would I reference a package file?

[ashinn]

OK, let's go with .snow-repo.scm. This specifies the fully qualified imports and exports of our packages, within which plain library imports such as (import (foo bar)) are resolved.

Within the .snow-repo.scm, currently dependencies are similar to the libraries themselves: (depends (foo bar)). The location and version of this are taken from context, with the precedence:

1. libraries defined in the same repository
2. libraries defined in sibling repositories
3. other locations

(a precedence which is not currently enforced but should be). However even with a sibling repository there's no checksum or version control. So we can extend the dependencies declaration to allow:

(depends
  (foo bar (>= "1.2.0"))
  (baz qux
    (uri "https://git@codeberg.org/retropikzel/snow-chibi-testing.git")
    (path "foo/bar.sld")
    (sha-3 "abc..."))
  )

in the spirit of the R6RS versions, or if the tail encoding looks confusing maybe:

(depends
  (library (foo bar) (>= "1.2.0"))
  (library (baz qux)
    (uri "https://git@codeberg.org/retropikzel/snow-chibi-testing.git")
    (path "foo/bar.sld")
    (sha-3 "abc..."))
  )

[Retropikzel]

I will switch it to .snow-repo.scm.

Perhaps the location and version of dependency should be it's own issue and solved in another pull request? For example one thing that comes to mind is that if I have (foo a) and (foo b), a depends on (foo c) version 2, b on version 3. How is it solved when packages are installed globally? I agree that defining dependency versions is important, but propably remain too hard for me to solve at least for now.

I will add, because the version problem is also interesting, that I have thought about project specific lockfiles. Say you could install package with snow-chibi install --project (retropikzel hello) and the library would then be under ./snow, and the dependency would be in .snow-lock.scm with the version. Then to get the same ./snow directory you could do snow-chibi install --project .snow-lock.scm or similar. For example Akku and npm use lockfiles like this.

I think the

(depends
  (library (foo bar) (>= "1.2.0"))
  (library (baz qux)
    (uri "https://git@codeberg.org/retropikzel/snow-chibi-testing.git")
    (path "foo/bar.sld")
    (sha-3 "abc..."))
  )

Form is clearer and leaves up room for additional data too.

[ashinn]

Yes, this is already a large PR, location and versioning should be handled separately.

Regarding conflicting transitive dependencies, that fundamentally can't be solved cleanly, but we can try. In JVM-land it's common to use shadow dependencies (one version of a whole library is copied under a separate package), which usually works, but if you have two definitions of the same class and instances of one are passed to methods that expect instances of the other, it breaks.

For the non-conflicting case we can consider either separate versioned installs, or something similar to pyenv.

[ashinn]

Can you make this a top-level definition?

[Retropikzel]

Hopefully it is now top level enough?

[ashinn]

Thanks!

[Retropikzel]

Thank you!