From 29465b21597469d962a4bb1642eac1e685649b6c Mon Sep 17 00:00:00 2001 From: Jeaeun Kim Date: Tue, 23 Sep 2025 21:27:26 +0900 Subject: [PATCH 1/3] SPICE-0022: Custom HTTP Headers --- spices/SPICE-0022-http-headers.adoc | 92 +++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 spices/SPICE-0022-http-headers.adoc diff --git a/spices/SPICE-0022-http-headers.adoc b/spices/SPICE-0022-http-headers.adoc new file mode 100644 index 0000000..eca46ad --- /dev/null +++ b/spices/SPICE-0022-http-headers.adoc @@ -0,0 +1,92 @@ += Custom HTTP Headers + +* Proposal: link:./SPICE-0022-http-headers.adoc[SPICE-0022] +* Author: link:https://github.com/kyokuping[kyokuping] +* Status: Proposed +* Category: Language, Standard Library, Tooling + +== Introduction + +This SPICE proposes a new evaluator option to allow adding custom HTTP headers to outbound requests, with the ability to target specific URL prefixes. + +== Motivation + +When Pkl interacts with HTTP resources, it may need to provide authentication tokens or other custom headers to access them. For example, a Pkl module might be hosted in a private repository that requires an `Authorization` header for access. + +Currently, there is no way to add custom headers to Pkl's HTTP requests. This limits Pkl's ability to interact with a wide range of HTTP-based resources. + +== Proposed Solution + +A new HTTP setting will be added, called `headers`. This setting allows users to specify a list of headers to be added to outbound HTTP requests. To avoid leaking credentials, headers can be configured on a per-URL-prefix basis. + +For example, to provide an authentication token for a specific host, the configuration would look like this: + +.~/.pkl/settings.pkl +[source,pkl] +---- +amends "pkl:settings" + +http { + headers { + ["https://my.private.repo/"] { + Pair("Authorization", "Bearer my-secret-token") + } + } +} +---- + +This will add the `Authorization` header to all requests sent to `https://my.private.repo/` and its sub-paths. + +== Detailed design + +The `pkl.EvaluatorSettings.Http` class will be extended with a new `headers` property. + +.pkl:EvaluatorSettings +[source,pkl] +---- +class Http { + // ... existing properties + + /// HTTP headers to add to outbound requests targeting specified URLs. + headers: Mapping>>? +} +---- + +The `HttpRewrite` typealias from `SPICE-0016-http-rewrites` will be reused to define the URL prefixes. + +When multiple prefixes match a request URL, the longest prefix will be chosen, ensuring that the most specific rule is applied. + +A new CLI flag, `--http-header`, will be introduced to allow specifying headers from the command line. The flag will accept key-value pairs in the format `=:[,:...]`. + +Example: +[source,shell] +---- +pkl eval \ + --http-header "https://my.private.repo/:Authorization=Bearer my-secret-token" \ + myModule.pkl +---- + +The HTTP client will be modified to check for matching header configurations for each outgoing request and add the corresponding headers. + +== Compatibility + +This change is strictly backward-compatible. Existing Pkl programs will continue to work as-is. + +== Future directions + +Future enhancements could include support for more complex header manipulation, such as removing default headers or adding headers based on regular expression matching on the URL. + +== Alternatives considered + +=== Global HTTP headers + +The initial proposal involved a simpler approach of having a single set of custom headers that would be applied to all outbound HTTP requests. This was rejected due to security concerns. Sending the same set of headers, which might include authentication tokens, to all hosts is a significant security risk. The per-URL-prefix approach provides a more secure way to manage custom headers. + +=== Auth-specific support + +Someone could argue it'd make more sense to specifically support auth handling rather than exposing the full ability to add custom HTTP headers. While this can be considered more future-proof in terms of exposing minimal API surface to maintain compatibility, this would limit the potential of supporting many different use cases outside of auth. That being said, adding separate auth support can also be considered later, mostly for supporting complex auth flows like OAuth. + +== Acknowledgements + +Thanks to the Pkl team for their feedback on the initial proposal, which helped shape the more secure and flexible design presented in this SPICE. + From b5cdcad4b19eb065129de0e3c55ed533cdb651d9 Mon Sep 17 00:00:00 2001 From: kyokuping Date: Mon, 20 Oct 2025 02:40:55 +0900 Subject: [PATCH 2/3] Refine with URL patterns and header name validation --- spices/SPICE-0022-http-headers.adoc | 101 ++++++++++++++++++++++++---- 1 file changed, 88 insertions(+), 13 deletions(-) diff --git a/spices/SPICE-0022-http-headers.adoc b/spices/SPICE-0022-http-headers.adoc index eca46ad..049bead 100644 --- a/spices/SPICE-0022-http-headers.adoc +++ b/spices/SPICE-0022-http-headers.adoc @@ -17,9 +17,9 @@ Currently, there is no way to add custom headers to Pkl's HTTP requests. This li == Proposed Solution -A new HTTP setting will be added, called `headers`. This setting allows users to specify a list of headers to be added to outbound HTTP requests. To avoid leaking credentials, headers can be configured on a per-URL-prefix basis. +A new HTTP setting will be added, called `headers`. This setting allows users to specify a list of headers to be added to outbound HTTP requests. To avoid leaking credentials, headers can be configured on a per-URL-pattern basis. -For example, to provide an authentication token for a specific host, the configuration would look like this: +For example, to provide an authentication token for specific subdomains, the configuration would look like this: .~/.pkl/settings.pkl [source,pkl] @@ -28,14 +28,14 @@ amends "pkl:settings" http { headers { - ["https://my.private.repo/"] { - Pair("Authorization", "Bearer my-secret-token") + ["https://*.my.private.repo/*"] { + ["authorization"] = "Bearer my-secret-token" } } } ---- -This will add the `Authorization` header to all requests sent to `https://my.private.repo/` and its sub-paths. +This will add the `authorization` header to all requests sent to `https://*.my.private.repo/` (subdomains of `my.private.repo`) and its subpaths. == Detailed design @@ -48,33 +48,103 @@ class Http { // ... existing properties /// HTTP headers to add to outbound requests targeting specified URLs. - headers: Mapping>>? + headers: Mapping|String>>? } ---- -The `HttpRewrite` typealias from `SPICE-0016-http-rewrites` will be reused to define the URL prefixes. +The details for `URLPattern` and `HttpHeaderName` will be provided in the following sections. -When multiple prefixes match a request URL, the longest prefix will be chosen, ensuring that the most specific rule is applied. - -A new CLI flag, `--http-header`, will be introduced to allow specifying headers from the command line. The flag will accept key-value pairs in the format `=:[,:...]`. +A new CLI flag, `--http-header`, will be introduced to allow specifying headers from the command line. The flag will accept key-value pairs in the format `=:[,:...]`. Example: [source,shell] ---- pkl eval \ - --http-header "https://my.private.repo/:Authorization=Bearer my-secret-token" \ + --http-header "https://*.my.private.repo/*:authorization=Bearer my-secret-token" \ myModule.pkl ---- The HTTP client will be modified to check for matching header configurations for each outgoing request and add the corresponding headers. +=== The `URLPattern` typealias + +The `URLPattern` typealias defines glob patterns that are used to match the URLs of HTTP requests and determine which HTTP headers to apply. + +To prevent accidental matches, the pattern should end with a slash (`/`) or a wildcard (`*`). + +If several patterns match a given request URL, the one with the highest specificity is selected. The specificity of a pattern is evaluated using the following process: + +1. Decompose the pattern into its components (scheme, host, path). +2. Compute a score for the pattern by combining the host score and the path score: + - First, calculate a "base score" by adding points for each character and glob feature in the string: + - 10 points for every literal character (including escaped characters) + - 5 points for each character set (`[]`) + - 4 points for every single character wildcard (`?`) + - 2 points for each wildcard (`*`) + - 1 point for every globstar (`**`) + - For alternations (`{}`), use the average score of the included patterns. + - Then, multiply the host score by a weight of 100. +3. When two patterns have identical scores, the longer pattern is considered more specific. + +=== The `HttpHeaderName` typealias + +The `HttpHeaderName` typealias is used to represent the name of an HTTP header, defined as follows: + +.pkl:EvaluatorSettings +[source,pkl] +---- +// Reserved HTTP header names that are inappropriate to be set by the user +typealias ReservedHttpHeaderName = + "accept-charset" + | "accept-encoding" + | "access-control-request-headers" + | "access-control-request-method" + | "connection" + | "content-length" + | "cookie" + | "date" + | "dnt" + | "expect" + | "host" + | "keep-alive" + | "origin" + | "permissions-policy" + | "referer" + | "te" + | "trailer" + | "transfer-encoding" + | "upgrade" + | "via" + +// Reserved HTTP header prefixes that are inappropriate to be set by the user +const local ReservedHttpHeaderPrefix = new Listing { + "proxy-" + "sec-" + "access-control-" +} +const local hasReservedHttpHeaderPrefix = (header: String) -> + ReservedHttpHeaderPrefix.any((it) -> header.startsWith(it)) + +// Regex for validating HTTP header names based on RFC 7230 +const local httpHeaderNameRegex = Regex("^[a-zA-Z0-9!#\\$%&'*+-.^_`|~]+$") +const local hasValidHttpHeaderName = (header: String) -> + httpHeaderNameRegex.findMatchesIn(header) + +typealias HttpHeaderName = String( + this == toLowerCase(), // Only accept lowercase header names + !(this is ReservedHttpHeaderName), + !hasReservedHttpHeaderPrefix.apply(this), + hasValidHttpHeaderName +) +---- + == Compatibility This change is strictly backward-compatible. Existing Pkl programs will continue to work as-is. == Future directions -Future enhancements could include support for more complex header manipulation, such as removing default headers or adding headers based on regular expression matching on the URL. +Future enhancements could include support for more complex header manipulation, such as removing default headers. == Alternatives considered @@ -86,7 +156,12 @@ The initial proposal involved a simpler approach of having a single set of custo Someone could argue it'd make more sense to specifically support auth handling rather than exposing the full ability to add custom HTTP headers. While this can be considered more future-proof in terms of exposing minimal API surface to maintain compatibility, this would limit the potential of supporting many different use cases outside of auth. That being said, adding separate auth support can also be considered later, mostly for supporting complex auth flows like OAuth. +=== Plain Text Patterns + +The initial proposal used plain text prefixes for matching URLs. After considering use cases that require more flexible matching, the design was updated to use glob patterns. + +Glob patterns allow for more powerful and concise rules. For example, a single pattern like `*.service.internal` can apply a common authentication header to all internal services. Similarly, a pattern like `*.my-company.com` can apply a rule to all subdomains of a company. + == Acknowledgements Thanks to the Pkl team for their feedback on the initial proposal, which helped shape the more secure and flexible design presented in this SPICE. - From fef2604a87a9f1e56ae24359e2280c7e7083d23c Mon Sep 17 00:00:00 2001 From: kyokuping Date: Tue, 3 Mar 2026 17:52:45 +0900 Subject: [PATCH 3/3] Move to applying all matching header configurations --- spices/SPICE-0022-http-headers.adoc | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/spices/SPICE-0022-http-headers.adoc b/spices/SPICE-0022-http-headers.adoc index 049bead..8202a67 100644 --- a/spices/SPICE-0022-http-headers.adoc +++ b/spices/SPICE-0022-http-headers.adoc @@ -72,19 +72,7 @@ The `URLPattern` typealias defines glob patterns that are used to match the URLs To prevent accidental matches, the pattern should end with a slash (`/`) or a wildcard (`*`). -If several patterns match a given request URL, the one with the highest specificity is selected. The specificity of a pattern is evaluated using the following process: - -1. Decompose the pattern into its components (scheme, host, path). -2. Compute a score for the pattern by combining the host score and the path score: - - First, calculate a "base score" by adding points for each character and glob feature in the string: - - 10 points for every literal character (including escaped characters) - - 5 points for each character set (`[]`) - - 4 points for every single character wildcard (`?`) - - 2 points for each wildcard (`*`) - - 1 point for every globstar (`**`) - - For alternations (`{}`), use the average score of the included patterns. - - Then, multiply the host score by a weight of 100. -3. When two patterns have identical scores, the longer pattern is considered more specific. +If several patterns match a given request URL, all matching patterns will be considered and their headers will be applied. === The `HttpHeaderName` typealias