-
Notifications
You must be signed in to change notification settings - Fork 7
feat(csharp): implement CircuitBreaker (WI-3.1) #164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
fd6548e to
03f7027
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/161/files) to review incremental changes. - [**stack/wi-1.2-tag-definition-system**](#161) [[Files changed](https://github.com/adbc-drivers/databricks/pull/161/files)] - [stack/wi-2.1-telemetry-data-models](#162) [[Files changed](https://github.com/adbc-drivers/databricks/pull/162/files/ab7fa964ff62f3fc9884034e17a7e57630fa8037..a566292aec78d19717c92e28f135535b09f25c80)] - [stack/wi-2.1-exception-classifier](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files/a566292aec78d19717c92e28f135535b09f25c80..baa7a2ae32662fddc65272e0264e8bb7d1644716)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/baa7a2ae32662fddc65272e0264e8bb7d1644716..03f7027e6731efe032c15555afe517ba49de3651)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/03f7027e6731efe032c15555afe517ba49de3651..1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6..eb382cb291c120a5f3cc3a1c38e0975b99c1369f)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/eb382cb291c120a5f3cc3a1c38e0975b99c1369f..67723fabe6f62d7ed16591c3e88e96aa269daddd)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/67723fabe6f62d7ed16591c3e88e96aa269daddd..6b66d37e9d97ca621d88c48a58ac60b2487425ea)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/6b66d37e9d97ca621d88c48a58ac60b2487425ea..2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2..39f6aed55278a533390e9aadf655f80dc11159c2)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/39f6aed55278a533390e9aadf655f80dc11159c2..4473de5ca3cfca8579818e6d58f8a2b12e869a47)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/4473de5ca3cfca8579818e6d58f8a2b12e869a47..94b678636d76a6d41a6612f76d00b4caccdab48a)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/94b678636d76a6d41a6612f76d00b4caccdab48a..ce00998cbd0372d94303ad1d69e9711e4489fe96)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/ce00998cbd0372d94303ad1d69e9711e4489fe96..2646e86223ff1e7706b20d5970e556ec2f17867b)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/2646e86223ff1e7706b20d5970e556ec2f17867b..0b9ebd3867250d92d0d8007cb17d6ce471d5560a)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/0b9ebd3867250d92d0d8007cb17d6ce471d5560a..4f553284c30eb7efcf67369c58dddd56675cd0be)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/4f553284c30eb7efcf67369c58dddd56675cd0be..1f7cde0c5642072b06588665b16ee3a30a90d256)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/1f7cde0c5642072b06588665b16ee3a30a90d256..c65e9fea7c65fa456f0114e95c867ee15f21bd87)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
03f7027 to
59b0221
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/162/files) to review incremental changes. - [**stack/wi-2.1-telemetry-data-models**](#162) [[Files changed](https://github.com/adbc-drivers/databricks/pull/162/files)] - [stack/wi-2.1-exception-classifier](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files/1e58d3c3785fa7ec1b83da01f80ddea1f6167851..0dac01831e7d9d313c67dc31e4aacceb17e74298)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/0dac01831e7d9d313c67dc31e4aacceb17e74298..59b0221cb4c9262d80a35041a2f1098376f6e19e)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/59b0221cb4c9262d80a35041a2f1098376f6e19e..8c30fc0649b09bc38e09cfd4d6875d66963ff6c0)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/8c30fc0649b09bc38e09cfd4d6875d66963ff6c0..a6e926c8017e9a3b3b6de31bbbafb367adaba884)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/a6e926c8017e9a3b3b6de31bbbafb367adaba884..c53df5d3c0124c490b920e1e1a611dd9c24e02a4)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/c53df5d3c0124c490b920e1e1a611dd9c24e02a4..de8757a697dd023628011d1aff9961896560bc95)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/de8757a697dd023628011d1aff9961896560bc95..0b77f8373958342da429c20f7e30c02105402331)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/0b77f8373958342da429c20f7e30c02105402331..9090bdefba63d6c7fbff45bf60c2c63668f3884e)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/9090bdefba63d6c7fbff45bf60c2c63668f3884e..0a0159524a429726078bd7340057672d6927d1cd)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/0a0159524a429726078bd7340057672d6927d1cd..75039c6574c2dc437f5d670e71b938b98719c06f)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/75039c6574c2dc437f5d670e71b938b98719c06f..254cdc75487f3e9344d3df6fb9b9cbf49fd03228)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/254cdc75487f3e9344d3df6fb9b9cbf49fd03228..7371da59309d109e8d457f4c27edd13adfa38a2c)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/7371da59309d109e8d457f4c27edd13adfa38a2c..5ff7e96827faa69e8bae1d5b5da06a9f95b91a8c)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/5ff7e96827faa69e8bae1d5b5da06a9f95b91a8c..7757345889dbfd0b1dcb22556e2e6c746d7fa0f0)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/7757345889dbfd0b1dcb22556e2e6c746d7fa0f0..2364122ad5402c9205008f39acaec6a400a4db98)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/2364122ad5402c9205008f39acaec6a400a4db98..698f3ea13f65a17b62385be8e8e4032497f88993)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
59b0221 to
2d85d86
Compare
1438311 to
2146f50
Compare
| /// <summary> | ||
| /// Circuit has been manually isolated and will not allow requests. | ||
| /// </summary> | ||
| Isolated = 3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
where is this used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch - removed. It wasn't being used anywhere.
samikshya-db
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that circuit breaker integration will bring in the per-host isolation, this PR looks good to me at the moment.
2146f50 to
696c1a9
Compare
696c1a9 to
86e186f
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/163/files) to review incremental changes. - [**stack/wi-2.1-exception-classifier**](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/dadc94197833c6a9face109ecaf7a50f17c08ea5..86e186f72b19a571cd3fc494fda621f4898d275c)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/86e186f72b19a571cd3fc494fda621f4898d275c..21469f9f7118348b828246e05dc0d9d1bab55bbc)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/21469f9f7118348b828246e05dc0d9d1bab55bbc..ae6ea52ba0af87cd4d5ca53f27b753ebaed5df1f)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/ae6ea52ba0af87cd4d5ca53f27b753ebaed5df1f..b0f161bc4041e64a225105942addea4b7d3d1f2e)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/b0f161bc4041e64a225105942addea4b7d3d1f2e..0073207f11c99f80b033c3c4c38ea371f6674909)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/0073207f11c99f80b033c3c4c38ea371f6674909..aab39e2182791ff3aba93564e9afd503ee094276)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/aab39e2182791ff3aba93564e9afd503ee094276..96701feff8e255edea8d24e7ed430d92234be8f6)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/96701feff8e255edea8d24e7ed430d92234be8f6..31b84f716bc3c3b7c14bc2d4b11d733b039e5865)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/31b84f716bc3c3b7c14bc2d4b11d733b039e5865..ac0a15e5fa0263aac2e80ed14125e94b7f2daf06)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/ac0a15e5fa0263aac2e80ed14125e94b7f2daf06..c9f7325b2ad1709d837cc66e27c2c80996d3cfac)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/c9f7325b2ad1709d837cc66e27c2c80996d3cfac..816bcdd6648e411e1de928e012a51e824d5b821e)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/816bcdd6648e411e1de928e012a51e824d5b821e..713a0637551df22544bb2d3abeb32874dcc084ab)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/713a0637551df22544bb2d3abeb32874dcc084ab..982306d1389e52e0764264dd1acb65b112d3ae23)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/982306d1389e52e0764264dd1acb65b112d3ae23..f62a13a707256bfc67499d94d1de49dc05dbc81b)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/f62a13a707256bfc67499d94d1de49dc05dbc81b..56c505931cec06ee74b894b132c2b498589a2722)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
Implement circuit breaker pattern for telemetry with three states: - Closed: Normal operation, tracks consecutive failures - Open: Rejects all requests after failure threshold (throws CircuitBreakerOpenException) - HalfOpen: Tests recovery after timeout, closes after success threshold Features: - Thread-safe state management using Interlocked operations - Configurable failure threshold (default: 5) - Configurable timeout (default: 1 minute) - Configurable success threshold for recovery (default: 2) - CircuitBreakerConfig.FromTelemetryConfiguration() for integration - Comprehensive test coverage (29 unit tests) Files: - src/Telemetry/CircuitBreaker.cs - Main circuit breaker implementation - src/Telemetry/CircuitBreakerConfig.cs - Configuration class - src/Telemetry/CircuitBreakerOpenException.cs - Exception for open state - test/Unit/Telemetry/CircuitBreakerTests.cs - Unit tests Co-Authored-By: Claude <noreply@anthropic.com>
- Replace custom CircuitBreaker implementation with Polly's ResiliencePipeline - Add Polly 8.5.0 as a dependency - Remove CircuitBreakerOpenException (use Polly's BrokenCircuitException instead) - Remove CircuitBreakerConfig (accept parameters directly in constructor) - Update tests to work with Polly's constraints (MinimumThroughput >= 2, BreakDuration >= 500ms) Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>
86e186f to
48f9307
Compare
🥞 Stacked PR
Use this link to review incremental changes.