Skip to content

Conversation

@SungJin1212
Copy link
Member

@SungJin1212 SungJin1212 commented Nov 16, 2025

I've found a panic would be occur in below scenario, and added test case to show it.
This PR adds channel close check logic to prevent the panic.

Scenario

  1. Ingester health check fail in pool
  2. closableHealthAndIngesterClient.Close() called
  3. streamPushChan closed
  4. job is nil in case job := <-c.streamPushChan
  5. get panic when err = stream.Send(job.req)

Related logs:

ts=2025-11-14T17:01:37.161148853Z caller=pool.go:184 level=warn msg="removing ingester failing healthcheck" addr=10.240.7.140:9095 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x25f81cc]

goroutine 292091 [running]:
github.com/cortexproject/cortex/pkg/ingester/client.(*closableHealthAndIngesterClient).worker.func1()
	/__w/cortex/cortex/pkg/ingester/client/client.go:235 +0xec
created by github.com/cortexproject/cortex/pkg/ingester/client.(*closableHealthAndIngesterClient).worker in goroutine 291921
	/__w/cortex/cortex/pkg/ingester/client/client.go:229 +0x117

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212 SungJin1212 force-pushed the fix-panic-in-pushStream branch from 44c7a06 to 381352f Compare November 16, 2025 22:46
@yeya24
Copy link
Contributor

yeya24 commented Nov 17, 2025

@alexqyle @danielblando Can you help take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants