Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/guide/remoteassetify.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
id: remoteassetify.py
title: Using remoteassetify.py
description: This document describes the usage of remoteassetify.py locally and in automation
slug: /guide/remoteassetify-usage-guide
---

# Using `remoteassetify.py`

## Requirements

The `remoteassetify.py` script requires these programs to function:

- `enosys`: Blocking command execution
- `rpmspec`: Processing specs
- `curl`: Downloading files

By default, the `enosys` program from `util-linux` is required to block command execution while processing specs.
If you are not on Linux, or you use a Linux distribution that does not provide it, use the `--unsafe-optional-enosys` option to disable this feature.

## Security note

Even though `remoteassetify.py` makes an attempt to filter out arbitrary code execution and uses `enosys` to block command execution, it is not a sandbox.
Do not run `remoteassetify.py` on untrusted specs without review or adequate protection.

## Local usage

By default, `remoteassetify.py` accepts the path to a spec file and performs these actions:

- Parse the spec to extract remote assets
- Download the remote assets from it into the `_assets` directory
- If any `#!RemoteAsset` lines have missing or differing checksums, generate a patch to update them.

Differing checksums at this point should be verified:

- If you updated a package, the URL and checksum to source code packages should change.
This is expected.
- If the URL did not change at all, but the checksum changed, then the old and the new download are different.
This requires manual verification.

This on its own implements a trust on first use (TOFU) model for the downloaded files.
If upstream provides methods to further verify the downloaded files, you can also do that after downloading.

You can pipe the output patch into `git apply` to fix the spec file:

```
$ scripts/remoteassetify.py SPECS/hello/hello.spec | git apply
```

For details on options, see:

```
$ scripts/remoteassetify.py --help
```

## Automation usage

Pull requests to openRuyi that contains changes to remote assets will trigger a automatic workflow to check all specs with changed remote assets using `remoteassetify.py`.
To avoid excessive downloads, if the URLs and checksums of remote assets are all changed for a spec, it is not checked.

For checked specs, any missing or differing checksum on a remote asset or a remote asset that failed to download causes the workflow to fail.
Please follow the annotations and workflow summary to see what needs updating.
If your pull request changes many files, note that on GitHub, only the first 10 warnings will be shown.
Therefore, check the workflow summary for full details.

On failure, a patch is provided in the workflow summary and as an artifact for convenience.
These can be accessed using the "Summary" link in the sidebar in workflow details.
However, any unexpected differing checksums still requires verification.
1 change: 1 addition & 0 deletions docs/packaging-guidelines/rpmspecification.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ VCS: git:https://git.example.org/project.git
3. For any network-fetched `Source`, a `#!RemoteAsset` comment MUST immediately precede the `Source` declaration. If multiple external sources exist, each MUST be individually annotated.

4. For any `Source` fetched using the HTTP or HTTPS protocol, the SHA-256 checksum of the source archive MUST be documented on the line following the `#!RemoteAsset` comment.
For conveience, it can be generated automatically with [remoteassetify](/docs/guide/remoteassetify-usage-guide).

5. If the tarball filename is obscured or cannot be algorithmically inferred from the URL, `Source` SHOULD explicitly dictate the desired tarball name via a URL fragment (e.g., `#/name.tar.gz`) to guarantee predictable local file naming.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
id: remoteassetify.py
title: 使用 remoteassetify.py
description: 这篇文章描述 remoteassetify.py 如何本地和自动化使用
slug: /guide/remoteassetify-usage-guide
---

# 使用 `remoteassetify.py`

## 系统依赖

`remoteassetify.py` 脚本依赖以下几个程序来工作

- `enosys`:阻拦任意命令执行
- `rpmspec`:解析 spec 文件
- `curl`:下载文件

在默认情况下 `util-linux` 的 `enosys` 会被用来在解析 spec 文件的时候阻拦任意命令执行。
如果你不使用 Linux,或者你的 Linux 发行版没有提供 `enosys`,可以使用 `--unsafe-optional-enosys` 选项禁用这个功能。

## 安全提示

虽然 `remoteassetify.py` 会尝试过滤掉 spec 文件中的任意代码执行,并使用 `enosys` 阻拦任意命令执行,它并不构成一个安全沙箱。
未经审阅或使用充分安全措施情况下,不要使用 `remoteassetify.py` 处理不可信的 spec 文件。

## 本地使用

默认情况下,`remoteassetify.py` 接收一个 spec 文件的路径作为参数,然后执行如下操作:

- 处理 spec 文件,提取其中的 remote asset
- 将 remote asset 下载到 `_assets` 目录
- 如果 spec 文件有 `#!RemoteAsset` 行缺少 checksum 或 checksum 不同,生成一个修改的补丁

如果这时发现有 remote asset 的 checksum 不同,需要检查:

- 如果有一个包被更新,则它的源码 URL 和 checksum 都应该不同。
这属于正常情况。
- 如果 URL 没有变化,而 checksum 变了,则说明本次下载的文件与上次不同。
这种情况下,应当检查变化的原因。

这样的做法对下载的文件实现了首次信任 (trust on first use, TOFU) 模型。
如果上游提供了其它方式验证下载的文件,在此也可以使用。

如需修改 spec 文件,可以将输出的补丁用管道送入 `git apply`:

```
$ scripts/remoteassetify.py SPECS/hello/hello.spec | git apply
```

更多参数信息,请详见

```
$ scripts/remoteassetify.py --help
```

## 自动化使用

向 openRuyi 提交的 PR (Pull Request) 如果包含对 remote asset 的更改,则会触发自动工作流对有 remote asset 变更的 spec 文件进行 `remoteassetify.py` 检查。
为了避免下载量过多,如果 spec 的 remote asset 的 URL 和 checksum 都没有变化,则不会自动重新检查。

对于被检查的 spec,如果缺少 checksum 或 checksum 不同,或文件下载失败,则会导致工作流失败。
在此情况下应根据标注信息和工作流 summary 页面,检查哪些文件需要更改。
如果你的 PR 修改很多文件,请注意 GitHub 上只会显示前 10 个警告。
完整的修改信息需要在工作流 summary 页面查看。

检查失败的情况下,为了更新方便,在工作流 summary 和 artifacts 中会有一个修改补丁。
在工作流详情页点击“Summary”链接可以查看。
但是请注意,所有未预期的 checksum 不同仍需要手动检查。
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ VCS: git:https://git.example.org/project.git
3. 对于网络来源的 `Source`,其行前必须添加 `#!RemoteAsset` 注释;存在多条网络来源 `Source` 时,每条均必须标识。

4. 对于 HTTP 和 HTTPS 协议来源的 `Source`,在 `#!RemoteAsset` 注释后,必须添加来源文件的 sha256 值。
为了方便,可以使用 [remoteassetify](/docs/guide/remoteassetify-usage-guide) 自动生成。

5. 对于无法从 URL 解析出 tarball 文件名的情形,`Source` 应当使用 URL 片段显式给出 tarball 名称,以保证源文件命名可预测。

Expand Down
Loading