yarn2nix

[moretea]

Facebook recently published yarn, a package manager that behaves like Bundler and Cargo by having a lock file that describes the exact versions of all transitive dependencies.

I investigated the yarn.lock format, and was pleasantly surprised that it actually contains both direct download links to (and SHA1 hashes of) the .tgz files that make up packages.

I'm very much open to merge this with @svanderburg's node2nix, but I'm not sure that that's appropriate.

Progress

• Get yarn into nixpkgs (yarn: init at 0.17.8 #20635)
• Finish implementing yarn2nix

[offlinehacker]

Nice :)

[offlinehacker]

What happens if package.json refrences git/github package?

[phanimahesh]

Would you mind if yarn2nix was a go/rust binary instead of a ruby script? Or a nodejs script, since we are anyway working with the node ecosystem. Having to pull in ruby to work on node just doesn't feel good. I can volunteer for part of the effort. Nice excuse to try out something new too, IMO.

[zimbatm]

I think it would be more maintainable to use yarn's parser directly. Here I made a small program that turns the yarn.lock into JSON: https://github.com/numtide/yarnlock2json . The URL still has to be split to extract the sha from the fragment.

[wmertens]

@moretea do you need help with this? This finally seems like a long-term viable approach… Use Yarn to get a fixed set of deps, create the deps, and let yarn do the node_modules build.

I agree with @zimbatm that we should use yarn's own parser. I had a look at the yarn2nix and it seems very simple, so perhaps it can be built directly on top off zimbatm's script?

Also, perhaps each yarn package should be a separate nix expression, so that they get reused between projects? I think it would mean walking the dep tree an building each dep in turn, creating packages as you go, and linking them into the .cache on next invocations.

[zimbatm]

Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there. Another solution would be to work with yarn to see if we can pass them the list of source derivations.

Regarding the node_modules folder, it's not practical to use symlinks in there unfortunately. If you search for it you'll see that others are trying to tackle the issue but in short it's due to two things: (1) some modules actually rely on the node_modules structure by doing things like require("../other_module"). (2) node resolves module paths to their absolute path at import. So now you have a module at /nix/store/...-my-module trying to load ../other_module and it breaks. What might work though is have the whole node_modules stored in a derivation so it only gets re-built when the package.json or yarn.lock file changes.

[wmertens]

Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there.

I was thinking that we can reproduce the cache dir layout. After all, we have the full source available.

Another solution would be to work with yarn to see if we can pass them the list of source derivations.

Actually that might work too, maybe with some sort of plugin system if we can't come up with something that upstream would want to support. What would that look like, ideally?

Regarding the node_modules folder, it's not practical to use symlinks in there unfortunately.

I was not referring to that; AFAICT, Yarn uses its cache dir to store tarballs and build products. These build products are then copied into node_modules, so most likely the build product folders can be symlinks. So the node_modules folder is a bunch of copies, but when that built application is stored in the nix store, it can be deduped if the optimize option is turned on. So that's not a problem.

[zimbatm]

Alright that makes sense. I have a notion that this same problem is quite general to all package managers. The difficulty for nix is that we don't want to deal with dependency resolution, only work over a resolved set. We want to fetch the dependencies but not deal with installing them. But provide library dependencies for C extensions. And finally link the result so the package can use them. So that means nix and $lang package manager are quite intertwined. If we can figure out the right interfaces for each of these steps, it will be much easier to communicate with $lang and ask them to insert the proper hooks where we need them. For now my best idea is that $lang provides a lockfile that contains the dependencies, where to fetch then including the SHA. $lang developer shouldn't have to commit the generated nix code. Nix knows how to read that lockfile and prefetch the dependencies from that. After that it's a bit fuzzy and probably varies. Ideally we can build each dependency in isolation but it's not always possible. If there is a package repository like rubygems or npn we should push for having the SHA included to make things faster. The SHA must be calculated the same way as nix though. Hmm that's it for now. Not really helping this issue in particular.

…

On Fri, 16 Dec 2016, 20:36 Wout Mertens, ***@***.***> wrote: > > Ideally we want the source to be fetched by nix but it's not clear to me > how to plug that into yarn yet. Maybe we can reproduce yarn's file layout > and ignore yarn install but I suspect there are some hidden complexities > there. > I was thinking that we can reproduce the cache dir layout. After all, we have the full source available. > Another solution would be to work with yarn to see if we can pass them the > list of source derivations. > Actually that might work too, maybe with some sort of plugin system if we can't come up with something that upstream would want to support. What would that look like, ideally? > Regarding the node_modules folder, it's not practical to use symlinks in > there unfortunately. > I was not referring to that; AFAICT, Yarn uses its cache dir to store tarballs and build products. These build products are then copied into node_modules, so most likely the build product folders can be symlinks. So the node_modules folder is a bunch of copies, but when that built application is stored in the nix store, it can be deduped if the optimize option is turned on. So that's not a problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20637 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMsNQzJh7dXxWHjTU2aEtlAIQiRCYkks5rIvZXgaJpZM4K6DHj> .

[moretea]

@phanimahesh Would you mind if yarn2nix was a go/rust binary instead of a ruby script?

Not at all. It's just a prototype at the moment and I just like to use Ruby to prototype these kinds of things.

@zimbatm I think it would be more maintainable to use yarn's parser directly.

Awesome. I actually know little about javascript; I just want to package some stuff.

@zimbatm Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there.

@wmertens I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.

I tried to do exactly that, but yarn still insists on writing random files in that cache directory. When I tried to fill it with the correct packages, it still failed because of missing packages.

@wmertens So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

It's a bit more subtile than that. The directory structure of node_modules is a bit weird; npm/yarn support having multiple versions of different packages in there. This in itself is not a weird thing to do; Cargo supports the same features.
The problem is that the directory names do not have a version number. Having multiple dependency versions is handled by having another node_modules directory in a $ROOT_DIR/node_modules/some_package/ directory. Unfortunately, I could not find any documentation on the exact logic of how these decisions are made.

@zimbatm For now my best idea is that $lang provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.

Fully agreed. Luckily a yarn.lock file contains all that!

---

I would very much appreciate any help with this project ;)

[zimbatm]

It's good to see that we are all pretty much on the same page even if the details are still fuzzy. I might have some time allocated over the holiday period, I will know next Tuesday.

…

On Fri, 16 Dec 2016, 22:23 Maarten Hoogendoorn, ***@***.***> wrote: @phanimahesh <https://github.com/phanimahesh> Would you mind if yarn2nix was a go/rust binary instead of a ruby script? Not at all. It's just a prototype at the moment and I just like to use Ruby to prototype these kinds of things. @zimbatm <https://github.com/zimbatm> I think it would be more maintainable to use yarn's parser directly. Awesome. I actually know little about javascript; I just want to package some stuff. @zimbatm <https://github.com/zimbatm> Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there. @wmertens <https://github.com/wmertens> I was thinking that we can reproduce the cache dir layout. After all, we have the full source available. I tried to do exactly that, but yarn still insists on writing random files in that cache directory. When I tried to fill it with the correct packages, it still failed because of missing packages. @wmertens <https://github.com/wmertens> So the node_modules folder is a bunch of copies, but when that built application is stored in the nix store, it can be deduped if the optimize option is turned on. So that's not a problem. It's a bit more subtile than that. The directory structure of node_modules is a bit weird; npm/yarn support having multiple versions of different packages in there. This in itself is not a weird thing to do; Cargo supports the same features. The problem is that the directory names do not have a version number. Having multiple dependency versions is handled by having another node_modules directory in a $ROOT_DIR/node_modules/some_package/ directory. Unfortunately, I could not find any documentation on the exact logic of how these decisions are made. @zimbatm <https://github.com/zimbatm> For now my best idea is that $lang provides a lockfile that contains the dependencies, where to fetch then including the SHA. $lang developer shouldn't have to commit the generated nix code. Nix knows how to read that lockfile and prefetch the dependencies from that. After that it's a bit fuzzy and probably varies. Ideally we can build each dependency in isolation but it's not always possible. Fully agreed. Luckily a yarn.lock file contains all that! ------------------------------ I would very much appreciate any help with this project ;) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20637 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMsEHQWABoIju_cUb-C0BOMxQmidxUks5rIw9hgaJpZM4K6DHj> .

[moretea]

I started to patch node2nix to support importing a yarn.lock file in svanderburg/node2nix#28.

It was actually a lot less work than I anticipated.

I'm closing this issue and proposing that we move the discussion to that PR.