Replies: 7 comments 36 replies
-
|
Thanks for creating this. Let's use one thread per topic for easy to discuss. |
Beta Was this translation helpful? Give feedback.
-
|
@segiddins It seems that there are no more topics to be discussed for now. Could you share your opinions for existing 3 topics? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the feedback everyone! The overarching goal of this project is to improve the general state of things surrounding gems with native extensions. That means security, ergonomics, developer experience, performance, etc., with the understanding that a system that we all like working with is better for everyone and provides us more leverage to have "secure by default" options where possible. A big part of this work, in addition to supporting the extended identifiers for platform selection, will be improving the tooling around native extensions. By having a workflow that "just works" that folks can adopt, we can make it easy for gem maintainers to ship gems that are easy to install for users, have source provenance, come from trusted (aka SLSA compliant) workflows, and support the most popular platforms that gem consumers are using. Additionally, we want to lower the barrier to shipping precompiled gems, as they have proven wildly popular with users overall (a huge thanks to @flavorjones for everything he's done in this space). Finally, this will be an opportunity for the rubygems team to help build out parts of this toolchain, and continue maintaining it as a critical part of the ruby packaging ecosystem moving forwards. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
@kou I appreciate that you brought this feedback to us. Since I've stepped into the director role, I've gone deeper into security than previously. One aspect is that I'm participating in several working groups across language ecosystems. A common theme is learning what has worked for another ecosystem and thinking about how that could apply to the Ruby ecosystem. In particular, advancing supply chain security is one of our top priorities. We're leveraging the experiences from these other ecosystems and we're receiving funding to implement them. In this particular case, the precompiled binary project is our big focus this year. We appreciate the work Samuel is doing here to move our community forward. We are grateful for the feedback and want this to proceed with involvement from our community. Many of you are deeper in one aspect or another in how you develop gems so it's a gift when you bring our awareness to concerns there. It's easy to miss that we haven't shared a specific detail. Heck, I've thought that we should refine our 'how to work in the open' approach with significant changes like what we're exploring here. Kudos to Sam for putting his work out there so early on. This will be a big step forward for RubyGems. Much gratitude to all of you for being willing to bring your concerns to us. I’d like to know how else I can assist with these efforts. Please feel free to contact me directly if you’d like us to adjust how we roll projects like this out. |
Beta Was this translation helpful? Give feedback.
-
|
@mghaught Thanks for sharing your thoughts. Currently, I think that existing threads in this discussion are stopped by waiting for Samuel's comments. How to proceed from this? |
Beta Was this translation helpful? Give feedback.
-
|
Security about binaries built by developers. https://github.com/orgs/rubygems/discussions/8575
|
Beta Was this translation helpful? Give feedback.
-
|
The developers might not be evil, they system might be just compromised. I have bigger trust in RubyGems infra, OTOH if there is some issue, it might have even bigger impact. But we all have some level of trust to RubyGems infra anyway, so this would not be new concern per-se |
Beta Was this translation helpful? Give feedback.
-
|
I've discussed the possibility of using RubyGems-owned infrastructure to build arbitrary gems, but this is not something that will be feasible for us to implement for the time being. That being said, the status quo is that gems with precompiled binaries are already built on developer machines, so this is no worse, and we plan to release tooling that helps developers to build & publish their gem matrix with trusted provenance, ideally with an SLSA level 3 compliant builder workflow |
Beta Was this translation helpful? Give feedback.
-
|
@voxik do you have trust in for example GitHub Actions environment (or any other CD) with shared build provenance? IMHO that's best and independent solution. |
Beta Was this translation helpful? Give feedback.
-
|
@segiddins Can we use GitHub Actions for the "RubyGems-owned infrastructure"? (It seems that the "builder workflow" is a GitHub Actions workflow.) If we can, I can help for preparing it. BTW, can we clarify the security part of this proposal?
It means that precompiled binaries aren't safer than normal gems, right?
This is the important part for security in this proposal, right? |
Beta Was this translation helpful? Give feedback.
-
|
If there was some standard GH action to build binary gems, that could work 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Binary gem maintainers need to synchronize with CRuby releases. NOTE: This is not a new problem with the proposal. The current binary gem mechanism also has this problem. CRuby will be released every Christmas. If binary gem maintainers don't release binary gems that support the new CRuby release soon, users can't use the new CRuby release. It's (high?) maintenance cost for binary gem maintainer. It's inconvenient for CRuby users. |
Beta Was this translation helpful? Give feedback.
-
As a gem developer perspective, this proposal doesn't improve the situation... If precompiled gem mechanism is improved (as a user perspective) without solving existing painpoints, users request precompiled gem to gem developers more. Gem developers will need more maintenance costs. I'm afraid of decreasing gem developers by more maintenance costs. Decreasing gem developers isn't a good situation in Ruby ecosystem. So this proposal without solving existing painpoint doesn't improve the situation.
As a gem user perspective, I agree with this. But I already mentioned in https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12917093 , building the picked source distribution will be failed because a user environment doesn't have enough tools to build the source distribution.
Can we clarify this? Pre-conditions:
If Python 3.13 is released, A releases 1.0.2 that includes Python 3.9-3.13 wheels and Python 3.13 wheels for 1.0.0 and 1.0.1 aren't released, right? Or A releases only Python 3.13 wheel for 1.0.1 because 1.0.1 is the latest release and 1.0.0 isn't the latest release? (I think that the former is correct.) I think that we can do better than wheels in Python :-) This is exactly a painpoint focused in this thread. It's inconvenient for gem developers. If RubyGems builds and publishes precompiled gems for new Ruby from normal gems, gem developers don't need to release a precompiled gem for new Ruby by themselves. It's my proposal for this painpoint and I think that this proposal (adding a Ruby version to tag) matches my proposal. |
Beta Was this translation helpful? Give feedback.
-
Interesting. Is the CI build action executed in repositories of each gem or RubyGems-owned repository? I think that we can implement notification on new Ruby release. We can collaborate with Ruby core team. I can help it as a member of Ruby core team. |
Beta Was this translation helpful? Give feedback.
-
|
@kou currently it could be done by using API access to repo and |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I understand that the CI build action executed in a repository of each gem. |
Beta Was this translation helpful? Give feedback.
-
FYI: There is a pandas 2.3.0 example: pandas-dev/pandas#61564 pandas 2.3.0 didn't provide Python 3.9 wheels (FYI: they are provided now) and |
Beta Was this translation helpful? Give feedback.
-
|
Binary bindings gem maintainers need to update bundled external libraries when any vulnerabilities are found in external libraries. (e.g. libxml2 in Nokogiri case.) NOTE: This is not a new problem with the proposal. The current binary gem mechanism also has this problem. Because, in general, binary bindings gem includes depended libraries. It's high maintenance cost for binary bindings gem maintainers because nobody knows when a vulnerability is found. If updated binary bindings gems aren't released immediately, users will be at risk. |
Beta Was this translation helpful? Give feedback.
-
|
This idea will not provide benefits without packaging system integration idea... Or... if another libxml2 bindings including bindings for other languages such as lxml for Python have similar patches, this idea may provide benefits without packaging system integration idea. |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate this concern, but I don't think it's exactly what we're trying to address with this work. C libraries bundled into a gem in reality are no different from the source of the gem, in terms of the risks & responsibilities they entail. Part of depending on |
Beta Was this translation helpful? Give feedback.
-
|
Ah, sorry. I should have mentioned that my base concern explicitly. As I mentioned in https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13243151 , my concern is that implement this proposal without solving existing painpoints may decrease gem developers by increased gem maintenance cost. Can we implement this proposal after we solve existing precompiled gem painpoints? |
Beta Was this translation helpful? Give feedback.
-
|
Can you please start a new thread in this discussion listing the pain points you'd like to see solved, and explain why they need to be solved first before this proposal moves forward? |
Beta Was this translation helpful? Give feedback.
-
|
OK. I've started a new thread: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13293536 |
Beta Was this translation helpful? Give feedback.
-
|
Developer experience may not be improved by this proposal. Let's use a separated topic for developer experience. Because this proposal focus on developer experience too. https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13238676
|
Beta Was this translation helpful? Give feedback.
-
|
This proposal improves developer experience by providing a workflow to build precompiled gems, right? https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13238676
Do you have a plan to use existing rake-compiler-dock approach? As a rake-compiler maintainer, the cross-compiling approach used by rake-compiler is another painpoint of building precompiled gems. If we depend on it, I think that developer experience will be worse. Because cross-compiling is more difficult than normal build. |
Beta Was this translation helpful? Give feedback.
-
|
@kou I agree that cross-compilation is difficult and it would be better to provide tools that compile natively on a variety of hardware and operating systems in cloud environments. In the python community there are a few tools that provide a framework for native compilation (not cross), linting, and testing. Two that are mentioned on https://pythonwheels.com/ are: both of which are built using lower-level projects:
In particular, I like https://github.com/pypa/cibuildwheel and I have a weak opinion that we should use it as a basis for solving this problem in Ruby. There is also existing code that we can start with when piecing together some of this functionality. For example, the code in Nokogiri that lints the dynamic libraries across Linux, Windows, and MacOS (similar in purpose to And to be clear, I would be very interested in helping to build any tools that improve the developer experience in this area. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing tools used for wheels. I agree that https://github.com/pypa/cibuildwheel like tool is useful for us even if we build binary gems by developers or by RubyGems.org. Based on https://cibuildwheel.pypa.io/en/stable/#usage and https://cibuildwheel.pypa.io/en/stable/#how-it-works , https://github.com/pypa/cibuildwheel uses native (not cross) compilation as much as possible. I also agree that https://github.com/pypa/auditwheel / https://github.com/matthew-brett/delocate like linters are useful for us. Nokogiri's code will help us for this area.
Me too. |
Beta Was this translation helpful? Give feedback.
-
|
Which existing pain points of binary gems should we solved before this proposal moves forward? This proposal may improve usability for the following case:
Because this proposal adds Ruby version to binary gem version. (FYI: I doubt that this improves usability. See https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12917093 for details.) If this proposal improves usability, users want to use more binary gems and users will request binary gems to gem developers. If we don't solve existing pain points of binary gems before this proposal moves forward, gem developers' maintenance costs will be increased. It may reduce gem developers and it's worse for Ruby ecosystem. Here are existing pain points (including proposals that solve them):
|
Beta Was this translation helpful? Give feedback.
-
|
FYI: RFC for this proposal: rubygems/rfcs#60 Security and developer experience aren't included. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I published a post outlining some goals for improved precompiled gem support that has received some feedback.
The high-level goal is to introduce support for something like python's "wheel", where gems are able to target a very specific "platform" such that they can contain a single precompiled binary that runs on that platform (as opposed to the current state, where e.g. nokogiri ships a arm64-darwin gem version that contains separate binaries for ruby 3.1, 3.2, 3.3, 3.4).
I wanted to create this discussion as a place where folks can leave their feedback, with an emphasis on proposing alternatives to the different parts.
Beta Was this translation helpful? Give feedback.
All reactions