Introduce user scanner to Ruler/Alertmanager #6999

SungJin1212 · 2025-08-29T10:13:53Z

This PR introduces a user scanner to reduce the number of list calls to object storage.

Which issue(s) this PR fixes:
Fixes #6914

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24

I wonder if we should extract user scanner into a dedicated package? pkg/storage/tsdb might not be the best place.

SungJin1212 · 2025-08-30T03:31:59Z

@yeya24
Yeah, agree. What about /pkg/usersscanner?

yeya24 · 2025-08-30T04:34:17Z

Yeah, agree. What about /pkg/usersscanner?

I am thinking about /pkg/util/users which includes everything user related, not just user scanner.

SungJin1212 · 2025-08-30T05:10:27Z

I am thinking about /pkg/util/users which includes everything user related, not just user scanner.

ok, it looks good.
I moved user-related files to /pkg/util/users, and relocated some test files to prevent cycle imports.
I renamed some files:
users_scanner_config.go -> scanner_config.go
updater.go -> index_updater.go

danielblando · 2025-09-02T18:11:42Z

What is the difference for us between user and tenant? It is a bit weird to have /pkg/util/users/tenant/.

SungJin1212 · 2025-09-03T00:37:02Z

@danielblando
I think tenant should be integrated into users, but I've left it like this for now due to a cyclic import. I'll look into a way to integrate it.

SungJin1212 · 2025-09-03T11:27:49Z

@danielblando @yeya24
I've moved the tenant package to the users package. I also deprecated util.StringsContain and replaced it with slices.Contains to resolve a cyclic import.

yeya24 · 2025-09-14T06:33:04Z

pkg/util/errors/errors.go

 	}
 }
+
+func IsOneOfTheExpectedErrors(f ...objstore.IsOpFailureExpectedFunc) objstore.IsOpFailureExpectedFunc {


Why this function is not part of the bucket package but in a generate pkg/util/errors? This seems specific to object store failures so I am not sure if it should be in the generic error file

I moved IsOneOfTheExpectedErrors to the bucket package. It causes a cyclic import if it is in the tsdb package.

What do you think of using only the list strategy and only exposing cache_ttl as config?

yeya24 · 2025-09-14T06:41:30Z

pkg/alertmanager/alertstore/bucketclient/bucket_client.go

+func NewBucketAlertStore(bkt objstore.InstrumentedBucket, userScannerCfg users.UsersScannerConfig, cfgProvider bucket.TenantConfigProvider, logger log.Logger, reg prometheus.Registerer) (*BucketAlertStore, error) {
+	alertBucket := bucket.NewPrefixedBucketClient(bkt, alertsPrefix)
+
+	usersScanner, err := users.NewScanner(userScannerCfg, alertBucket, logger, extprom.WrapRegistererWith(prometheus.Labels{"component": "alertmanager"}, reg))


Do we support user index based user scanner? Who will work as the user index updater for AM and Ruler bucket?

Oh, the user index updater would also be necessary for Alertmanager and Ruler.

Yeah, for compactor, we use Ring to choose one compactor pod to periodically update the user index in the storage bucket. I think we need the same thing for Ruler/AM as they are using different S3 buckets

How about adding clean_up_interval for the user index updater? Currently, it uses clean_up_interval in the Compactor.

yeya24 · 2025-09-14T06:42:35Z

pkg/storage/tsdb/bucketindex/loader_test.go

-	cortex_testutil "github.com/cortexproject/cortex/pkg/storage/tsdb/testutil"
 	"github.com/cortexproject/cortex/pkg/util/services"
 	"github.com/cortexproject/cortex/pkg/util/test"
+	cotex_testutil "github.com/cortexproject/cortex/pkg/util/testutil"


This seems a typo.

SungJin1212 · 2025-09-16T01:34:13Z

@yeya24
I introduced a flag clean_up_interval for the user index updater and added updaters to Alertmanager and Ruler.

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

pull-request-size bot added the size/L label Aug 29, 2025

dosubot bot added component/alertmanager component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. labels Aug 29, 2025

yeya24 reviewed Aug 29, 2025

View reviewed changes

pull-request-size bot added size/XXL and removed size/L labels Aug 30, 2025

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 3 times, most recently from e8c616c to 04acdd5 Compare September 1, 2025 06:31

SungJin1212 marked this pull request as draft September 1, 2025 06:38

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 2 times, most recently from ee1808c to 15f2ccd Compare September 1, 2025 06:51

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from b0e778d to 22768c9 Compare September 3, 2025 11:17

SungJin1212 marked this pull request as ready for review September 4, 2025 01:25

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 2 times, most recently from fbfa1dd to 2953348 Compare September 10, 2025 01:59

SungJin1212 requested review from yeya24 and danielblando September 10, 2025 08:52

yeya24 reviewed Sep 14, 2025

View reviewed changes

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 3 times, most recently from f262ab1 to fd54015 Compare September 15, 2025 10:59

SungJin1212 requested a review from yeya24 September 17, 2025 03:55

SungJin1212 added 13 commits September 30, 2025 14:13

Introduce user scanner to Ruler/Alertmanager

232442a

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix tests

df4823b

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix lint

2b5f05b

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

Move user related files to pkg/util/users

d885788

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix lint

66590ff

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

move tenant pkg

92b57e2

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

move tenant to users pkg

40d8b73

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix lint

543eb6a

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix typo

919e0d1

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

Move IsOneOfTheExpectedErrors to bucket package

4d28f31

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

Add index updater to alertmanager/ruler

c065ccf

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix lint

99eb945

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

fix test

271165b

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from fc2f24c to 271165b Compare September 30, 2025 05:13

update json config

2be7635

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>

Introduce user scanner to Ruler/Alertmanager #6999

Are you sure you want to change the base?

Introduce user scanner to Ruler/Alertmanager #6999

Conversation

SungJin1212 commented Aug 29, 2025

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yeya24 commented Aug 30, 2025

Uh oh!

SungJin1212 commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielblando commented Sep 2, 2025

Uh oh!

SungJin1212 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SungJin1212 commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented Sep 16, 2025

Uh oh!

Uh oh!

SungJin1212 commented Aug 30, 2025 •

edited

Loading

SungJin1212 commented Aug 30, 2025 •

edited

Loading

SungJin1212 commented Sep 3, 2025 •

edited

Loading