Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,16 @@ name: Run unit tests
on:
push

env:
GO_VERSION: 1.22

jobs:
test:
strategy:
matrix:
module: [., v2]
include:
- module: .
go-version: "1.22"
- module: v2
go-version: "1.26"
runs-on: ubuntu-latest
steps:
- name: Checkout code
Expand All @@ -16,9 +21,10 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
go-version: ${{ matrix.go-version }}

- name: Run tests
working-directory: ${{ matrix.module }}
# Run up to 3 times in case there is a flaky test
run: |
retry() {
Expand Down
89 changes: 43 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
- [Quick start](#quick-start)
- [Error handling](#error-handling)
- [Situational](#situational)
- [Global](#global)
- [Per-instance classifier](#per-instance-classifier)
- [`deixis/faults`](#deixisfaults)
- [Fallback](#fallback)
- [Priority](#priority)
Expand All @@ -15,7 +15,7 @@
- [Throttle ratio](#throttle-ratio)
- [Throttle minimum rate](#throttle-minimum-rate)
- [Throttle window](#throttle-window)
- [Accepted errors](#accepted-errors)
- [Rejected error classifier](#rejected-error-classifier)
- [Under the hood](#under-the-hood)
- [Inspirations](#inspirations)
- [Further reading](#further-reading)
Expand All @@ -28,16 +28,19 @@ Distributed services are particularly susceptible to cascading failures when par

In normal conditions, when resources meet demand, Bulwark operates passively, allowing all traffic to flow without interference. Unlike traditional throttling mechanisms, Bulwark does not queue requests, ensuring no additional latency is introduced to request handling.

**Requires Go 1.26+** (`github.com/deixis/bulwark/v2`). The v1 module (`github.com/deixis/bulwark`) requires Go 1.22+.

## Quick start

```go
package main

import (
"context"
"errors"
"fmt"

"github.com/deixis/bulwark"
"github.com/deixis/bulwark/v2"
)

func main() {
Expand Down Expand Up @@ -68,28 +71,20 @@ func main() {
return nil
})
if err != nil {
if err == bulwark.ClientSideRejectionError {
if errors.Is(err, bulwark.ErrClientSideRejection) {
// Call dropped
}

// Handle error
}

// When the throttled function needs to return a value, this function can be used.
// When the throttled function needs to return a value, use the generic Throttle function.
msg, err := bulwark.Throttle(ctx, throttle, bulwark.Medium, func(ctx context.Context) (string, error) {
// Call external service here...
var err error
if err != nil {
// Wrap error when it should be considered for throttling.
// By default, errors are ignored unless they are from the `faults` package.
// See the Error handling section for more info.
return bulwark.RejectedError(err)
}

return "Hello", nil
})
if err != nil {
if err == bulwark.ClientSideRejectionError {
if errors.Is(err, bulwark.ErrClientSideRejection) {
// Call dropped
}

Expand Down Expand Up @@ -119,39 +114,38 @@ if err != nil {

> 💡 Wrapping errors with `bulwark.RejectedError` is suitable for initial implementations and simple use cases. However, avoid adding excessive error-handling logic within the throttled function, because it is not easily reusable and can lead to inconsistencies.

### Global
### Per-instance classifier

Bulwark provides a global function, `bulwark.IsRejectedError`, to classify errors for throttling. This is especially useful for handling well-known error types across the codebase, reducing logic duplication in throttled functions.
`WithRejectedErrorFunc` sets the per-instance function that classifies errors as capacity rejections. This is especially useful for handling well-known error types across the codebase, reducing logic duplication in throttled functions.

Errors wrapped with `bulwark.RejectedError(err)` are always treated as capacity issues, so you don't need to include them in your `bulwark.IsRejectedError` implementation.
Errors wrapped with `bulwark.RejectedError(err)` are always treated as capacity issues, so you don't need to include them in your classifier.

```go
bulwark.IsRejectedError = func(err error) bool {
// For example, all timeouts could be considered as a capacity problem.
tempErr, ok := err.(interface {
Timeout() bool
})
if ok && tempErr.Timeout() {
return true
}
// a "Connection Reset by Peer" could also show symptoms of a capacity problem.
if errors.Is(err, syscall.ECONNRESET) {
return true
}
// Include the default logic
if bulwark.DefaultRejectedError(err) {
return true
}

return false // Use true or false by default to have a white/black list approach.
}
throttle := bulwark.NewAdaptiveThrottle(
bulwark.StandardPriorities,
bulwark.WithRejectedErrorFunc(func(err error) bool {
// For example, all timeouts could be considered as a capacity problem.
tempErr, ok := err.(interface {
Timeout() bool
})
if ok && tempErr.Timeout() {
return true
}
// a "Connection Reset by Peer" could also show symptoms of a capacity problem.
if errors.Is(err, syscall.ECONNRESET) {
return true
}
// Include the default logic
return bulwark.DefaultRejectedErrorFunc(err)
}),
)
```

> 💡 This approach works well in codebases with consistent error definitions for capacity-related issues. For instance, an [Echo](https://echo.labstack.com) server might override `bulwark.IsRejectedError` to include `echo.*HTTPError`.
> 💡 This approach works well in codebases with consistent error definitions for capacity-related issues. For instance, an [Echo](https://echo.labstack.com) server might use `WithRejectedErrorFunc` to include `echo.*HTTPError`.

### `deixis/faults`

Bulwark integrates with the [`deixis/faults`](https://github.com/deixis/faults) library through `bulwark.DefaultRejectedError`. This integration provides a structured and consistent way to categorise errors using well-defined primitives, offering significant benefits beyond load shedding.
Bulwark integrates with the [`deixis/faults`](https://github.com/deixis/faults) library through `bulwark.DefaultRejectedErrorFunc`. This integration provides a structured and consistent way to categorise errors using well-defined primitives, offering significant benefits beyond load shedding.

```go
err := throttle.Throttle(ctx, bulwark.Medium, func(ctx context.Context) error {
Expand Down Expand Up @@ -288,7 +282,7 @@ Higher values of `k` mean that the throttle will react more slowly when a backen
```go
throttle := bulwark.NewAdaptiveThrottle(
bulwark.StandardPriorities,
bulwark.WithAdaptivethrottleatio(1.1),
bulwark.WithAdaptiveThrottleRatio(1.1),
)
```

Expand Down Expand Up @@ -318,21 +312,24 @@ throttle := bulwark.NewAdaptiveThrottle(
)
```

### Accepted errors
### Rejected error classifier

Set the function that determines whether an error should be considered for the throttling. When the call to `fn` returns true, the error is NOT counted towards the throttling.
Set the per-instance function that determines whether an error returned by the throttled function should be counted as a capacity rejection. Defaults to `DefaultRejectedErrorFunc`.

```go
isAcceptedErrors := func(err error) bool {
return errors.Is(err, context.Canceled) // || other conditions
}
throttle := bulwark.NewAdaptiveThrottle(
bulwark.StandardPriorities,
bulwark.WithAcceptedErrors(isAcceptedErrors),
bulwark.WithRejectedErrorFunc(func(err error) bool {
// context.Canceled is not a capacity issue — don't count it.
if errors.Is(err, context.Canceled) {
return false
}
return bulwark.DefaultRejectedErrorFunc(err)
}),
)
```

> Errors unrelated to resource constraints or a service's inability to handle traffic should be allowed. For instance, errors caused by invalid user requests or authentication failures should be accepted.
> Only errors that indicate the backend is under resource pressure should return true. Errors caused by invalid requests, authentication failures, or client cancellations should return false.

## Under the hood

Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ module github.com/deixis/bulwark
go 1.22

require (
github.com/deixis/faults v0.0.0-20240817153531-c0ec10db827f
github.com/deixis/faults v1.0.1
golang.org/x/time v0.6.0
)
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
github.com/deixis/faults v0.0.0-20240817153531-c0ec10db827f h1:n8+Ze8qDZh8DzSdknFqzXpvU3xjVrhqShgyx1xwC8ek=
github.com/deixis/faults v0.0.0-20240817153531-c0ec10db827f/go.mod h1:TmAFyR/M6swaIznYCjZBqZMVJg5MYOJFOsTYOawLZK4=
github.com/deixis/faults v1.0.1 h1:4KbZaJvqOfc2cWh3CjWU2ynGWRY/OpDr2DOgp2j6zeQ=
github.com/deixis/faults v1.0.1/go.mod h1:TmAFyR/M6swaIznYCjZBqZMVJg5MYOJFOsTYOawLZK4=
golang.org/x/time v0.6.0 h1:eTDhh4ZXt5Qf0augr54TN6suAUudPcawVZeIAPU7D4U=
golang.org/x/time v0.6.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
Loading
Loading