feat(csharp): implement MetricsAggregator (WI-3.5) by jadewang-db · Pull Request #167 · adbc-drivers/databricks

jadewang-db · 2026-01-22T18:42:14Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/wi-3.1-feature-flag-cache [Files changed]
- stack/wi-3.4-databricks-telemetry-exporter [Files changed]
  - stack/wi-3.5-metrics-aggregator [Files changed]
    - stack/wi-3.5-circuit-breaker-manager [Files changed]
      - stack/e2e-feature-flag-cache-tests [Files changed]
        
        stack/databricks-activity-listener [Files changed]
        
        stack/circuit-breaker-telemetry-exporter [Files changed]
        
        stack/telemetry-client-manager-wi-3.2 [Files changed]
        
        stack/telemetry-client-wi-5.5 [Files changed]
        
        stack/telemetry-client-manager-e2e-wi-7 [Files changed]
        stack/telemetry-client-e2e-tests-wi-7-standalone [Files changed]
        stack/wi-6.1-databricks-connection-telemetry-integration [Files changed]
        stack/wi-6.2-telemetry-tags-driver-activities [Files changed]
        stack/wi-9-full-integration-e2e-tests [Files changed]

Implements per-host feature flag caching with reference counting to avoid repeated API calls and rate limiting. Key features: - FeatureFlagContext: Holds cached telemetry enabled state, last fetched timestamp, reference count, and configurable cache duration (default 15 min) - FeatureFlagCache: Singleton managing per-host contexts with thread-safe ConcurrentDictionary storage API: - GetInstance(): Returns the singleton instance - GetOrCreateContext(host): Creates/returns context and increments RefCount - ReleaseContext(host): Decrements RefCount, removes context when zero - IsTelemetryEnabledAsync(): Returns cached value if valid, otherwise fetches Thread safety ensured via ConcurrentDictionary and Interlocked operations. Includes 46 comprehensive unit tests covering all exit criteria. Co-Authored-By: Claude <noreply@anthropic.com>

…(WI-3.1) Refactored FeatureFlagCache based on updated design doc requirements: - Moved from Telemetry namespace to root namespace (AdbcDrivers.Databricks) to make it a generic, reusable component - Added HTTP API integration to fetch flags from /api/2.0/connector-service/feature-flags/OSS_JDBC/{version} - Implemented background refresh scheduler with server-provided TTL - Added FeatureFlagsResponse model for API response parsing - Updated FeatureFlagContext interface: - GetFlagValue(string) - get individual flag value - GetAllFlags() - get all cached flags as dictionary - IsFeatureEnabled(string) - check if flag is "true" - Shutdown() - stop background refresh scheduler - IDisposable for proper cleanup - Updated FeatureFlagCache.GetOrCreateContext() to accept HttpClient and driver version parameters - Updated all unit tests for new interface Co-Authored-By: Claude <noreply@anthropic.com>

…I-3.1) Integrated feature flag cache into the connection lifecycle: - Fetch feature flags from server during connection initialization - Merge flags into Properties dictionary with proper priority: User Properties > Feature Flags > Driver Defaults - Track host for proper context cleanup on Dispose - Release feature flag context when connection is disposed - All feature flag operations are fail-safe (errors logged, not thrown) The feature flag endpoint used is: GET /api/2.0/connector-service/feature-flags/OSS_JDBC/{driver_version} Co-Authored-By: Claude <noreply@anthropic.com>

…-3.1) - Add EnsureSuccessStatusCode pattern for HTTP response handling - Extract common HTTP fetch code into single FetchFeatureFlags method - Make feature flag endpoint configurable via optional parameter - Replace Debug.WriteLine with Activity trace pattern - Add E2E tests for FeatureFlagCache using real Databricks instance Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…WI-3.1) - Move MergePropertiesWithFeatureFlags, TryGetHost, CreateFeatureFlagHttpClient, and MergeProperties helper methods from DatabricksConnection to FeatureFlagCache - Replace Debug.WriteLine with ActivitySource tracing for structured events - DatabricksConnection now delegates to FeatureFlagCache.GetInstance().MergePropertiesWithFeatureFlags() Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…ructor (WI-3.1) Replace hardcoded "1.0.0" with ApacheUtility.GetAssemblyVersion() to use the actual driver version in the test constructor. Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…-3.1) - Add proxy support using HiveServer2ProxyConfigurator - Handle protocol prefix in host (e.g., "https://myhost.databricks.com") - Add configurable timeout via FeatureFlagTimeoutSeconds parameter - Use consistent User-Agent format: DatabricksJDBCDriverOSS/{version} (ADBC) - Rename variables to localProperties/remoteProperties for clarity - Remove IsFeatureEnabled method from FeatureFlagContext - Use EnsureSuccessOrThrow extension method for HTTP error handling - Enhance E2E tests to verify flags fetched and cache cleanup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add support for OAuth client_credentials (M2M) authentication in addition to token-based (PAT) auth for feature flag API calls. This ensures feature flags work with all supported authentication methods. - Add AuthHelper class with shared token extraction methods - Update FeatureFlagCache to use AuthHelper.GetAccessToken - Update HttpHandlerFactory to use AuthHelper.GetTokenFromProperties Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move the test factory method from production to test code: - Make FeatureFlagContext constructor internal instead of private - Make Ttl setter internal to allow tests to configure TTL - Remove CreateForTesting from FeatureFlagContext.cs - Add CreateTestContext helper method in FeatureFlagCacheTests.cs This addresses the PR review feedback that test-only code should not be in production source files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Implement the HTTP exporter that sends telemetry events to Databricks service. Key features: - ITelemetryExporter interface with ExportAsync method - Creates TelemetryRequest wrapper with uploadTime and protoLogs - Uses /telemetry-ext for authenticated requests - Uses /telemetry-unauth for unauthenticated requests - Implements retry logic for transient failures - Uses ExceptionClassifier for terminal vs retryable errors - Never throws exceptions (all swallowed and logged at TRACE level) - Cancellation is propagated (not swallowed) Files added: - src/Telemetry/ITelemetryExporter.cs - src/Telemetry/DatabricksTelemetryExporter.cs - test/Unit/Telemetry/DatabricksTelemetryExporterTests.cs Co-Authored-By: Claude <noreply@anthropic.com>

Implement MetricsAggregator that aggregates Activity data by statement_id and handles exception buffering with terminal vs retryable classification. Key features: - ProcessActivity extracts tags and aggregates by statement_id using ConcurrentDictionary<string, StatementTelemetryContext> - CompleteStatement emits aggregated TelemetryEvent - RecordException flushes terminal exceptions immediately - RecordException buffers retryable exceptions until CompleteStatement - FlushAsync exports when batch size or time interval reached - Uses TelemetryTagRegistry to filter tags - Creates TelemetryFrontendLog wrapper with workspace_id - All exceptions swallowed and logged at TRACE level Implementation details: - Connection events emit immediately (no aggregation needed) - Statement events aggregate until CompleteStatement is called - Timer-based periodic flush using System.Threading.Timer - Thread-safe aggregation using ConcurrentDictionary - Nested StatementTelemetryContext holds aggregated metrics and buffered exceptions per statement Test coverage: - 29 unit tests covering all exit criteria - Tests for exception handling, tag filtering, frontend log wrapping - End-to-end statement lifecycle tests Co-Authored-By: Claude <noreply@anthropic.com>

jadewang-db had a problem deploying to azure-prod January 22, 2026 18:42 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from a53a7bb to 33801c7 Compare January 22, 2026 22:24

jadewang-db had a problem deploying to azure-prod January 22, 2026 22:24 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 33801c7 to 67723fa Compare January 23, 2026 21:14

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:15 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 67723fa to c53df5d Compare January 23, 2026 21:43

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:43 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from c53df5d to fb56e1b Compare January 26, 2026 22:43

jadewang-db had a problem deploying to azure-prod January 26, 2026 22:43 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from fb56e1b to 70426b8 Compare January 26, 2026 23:01

jadewang-db had a problem deploying to azure-prod January 26, 2026 23:01 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from b0f161b to 6a82f08 Compare January 28, 2026 00:11

jadewang-db had a problem deploying to azure-prod January 28, 2026 00:11 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 6a82f08 to dc02969 Compare January 28, 2026 01:08

jadewang-db had a problem deploying to azure-prod January 28, 2026 01:08 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from dc02969 to 694bc3a Compare January 28, 2026 04:37

jadewang-db had a problem deploying to azure-prod January 28, 2026 04:37 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 694bc3a to c038ac2 Compare February 2, 2026 22:11

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:11 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from c038ac2 to 6513e06 Compare February 2, 2026 22:19

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:19 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 6513e06 to f27766c Compare February 3, 2026 00:50

jadewang-db had a problem deploying to azure-prod February 3, 2026 00:51 — with GitHub Actions Error

Jade Wang and others added 9 commits February 3, 2026 18:10

Address comments and refactoring

80bed7a

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from f27766c to 9647b2a Compare February 3, 2026 18:24

jadewang-db had a problem deploying to azure-prod February 3, 2026 18:24 — with GitHub Actions Error

Jade Wang and others added 3 commits February 3, 2026 18:31

jadewang-db force-pushed the stack/wi-3.5-metrics-aggregator branch from 9647b2a to d220707 Compare February 3, 2026 18:32

jadewang-db requested a deployment to azure-prod February 3, 2026 18:32 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(csharp): implement MetricsAggregator (WI-3.5)#167

feat(csharp): implement MetricsAggregator (WI-3.5)#167
jadewang-db wants to merge 12 commits intomainfrom
stack/wi-3.5-metrics-aggregator

jadewang-db commented Jan 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jadewang-db commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jadewang-db commented Jan 22, 2026 •

edited

Loading