feat(csharp): implement DatabricksActivityListener (WI-3.4) #170

jadewang-db · 2026-01-22T18:42:18Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/wi-3.1-feature-flag-cache [Files changed]
- stack/wi-3.4-databricks-telemetry-exporter [Files changed]
  - stack/wi-3.5-metrics-aggregator [Files changed]
    - stack/wi-3.5-circuit-breaker-manager [Files changed]
      - stack/e2e-feature-flag-cache-tests [Files changed]
        
        stack/databricks-activity-listener [Files changed]
        
        stack/circuit-breaker-telemetry-exporter [Files changed]
        
        stack/telemetry-client-manager-wi-3.2 [Files changed]
        
        stack/telemetry-client-wi-5.5 [Files changed]
        
        stack/telemetry-client-manager-e2e-wi-7 [Files changed]
        stack/telemetry-client-e2e-tests-wi-7-standalone [Files changed]
        stack/wi-6.1-databricks-connection-telemetry-integration [Files changed]
        stack/wi-6.2-telemetry-tags-driver-activities [Files changed]
        stack/wi-9-full-integration-e2e-tests [Files changed]

Implements per-host feature flag caching with reference counting to avoid repeated API calls and rate limiting. Key features: - FeatureFlagContext: Holds cached telemetry enabled state, last fetched timestamp, reference count, and configurable cache duration (default 15 min) - FeatureFlagCache: Singleton managing per-host contexts with thread-safe ConcurrentDictionary storage API: - GetInstance(): Returns the singleton instance - GetOrCreateContext(host): Creates/returns context and increments RefCount - ReleaseContext(host): Decrements RefCount, removes context when zero - IsTelemetryEnabledAsync(): Returns cached value if valid, otherwise fetches Thread safety ensured via ConcurrentDictionary and Interlocked operations. Includes 46 comprehensive unit tests covering all exit criteria. Co-Authored-By: Claude <noreply@anthropic.com>

…(WI-3.1) Refactored FeatureFlagCache based on updated design doc requirements: - Moved from Telemetry namespace to root namespace (AdbcDrivers.Databricks) to make it a generic, reusable component - Added HTTP API integration to fetch flags from /api/2.0/connector-service/feature-flags/OSS_JDBC/{version} - Implemented background refresh scheduler with server-provided TTL - Added FeatureFlagsResponse model for API response parsing - Updated FeatureFlagContext interface: - GetFlagValue(string) - get individual flag value - GetAllFlags() - get all cached flags as dictionary - IsFeatureEnabled(string) - check if flag is "true" - Shutdown() - stop background refresh scheduler - IDisposable for proper cleanup - Updated FeatureFlagCache.GetOrCreateContext() to accept HttpClient and driver version parameters - Updated all unit tests for new interface Co-Authored-By: Claude <noreply@anthropic.com>

…I-3.1) Integrated feature flag cache into the connection lifecycle: - Fetch feature flags from server during connection initialization - Merge flags into Properties dictionary with proper priority: User Properties > Feature Flags > Driver Defaults - Track host for proper context cleanup on Dispose - Release feature flag context when connection is disposed - All feature flag operations are fail-safe (errors logged, not thrown) The feature flag endpoint used is: GET /api/2.0/connector-service/feature-flags/OSS_JDBC/{driver_version} Co-Authored-By: Claude <noreply@anthropic.com>

…-3.1) - Add EnsureSuccessStatusCode pattern for HTTP response handling - Extract common HTTP fetch code into single FetchFeatureFlags method - Make feature flag endpoint configurable via optional parameter - Replace Debug.WriteLine with Activity trace pattern - Add E2E tests for FeatureFlagCache using real Databricks instance Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…WI-3.1) - Move MergePropertiesWithFeatureFlags, TryGetHost, CreateFeatureFlagHttpClient, and MergeProperties helper methods from DatabricksConnection to FeatureFlagCache - Replace Debug.WriteLine with ActivitySource tracing for structured events - DatabricksConnection now delegates to FeatureFlagCache.GetInstance().MergePropertiesWithFeatureFlags() Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…ructor (WI-3.1) Replace hardcoded "1.0.0" with ApacheUtility.GetAssemblyVersion() to use the actual driver version in the test constructor. Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…-3.1) - Add proxy support using HiveServer2ProxyConfigurator - Handle protocol prefix in host (e.g., "https://myhost.databricks.com") - Add configurable timeout via FeatureFlagTimeoutSeconds parameter - Use consistent User-Agent format: DatabricksJDBCDriverOSS/{version} (ADBC) - Rename variables to localProperties/remoteProperties for clarity - Remove IsFeatureEnabled method from FeatureFlagContext - Use EnsureSuccessOrThrow extension method for HTTP error handling - Enhance E2E tests to verify flags fetched and cache cleanup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add support for OAuth client_credentials (M2M) authentication in addition to token-based (PAT) auth for feature flag API calls. This ensures feature flags work with all supported authentication methods. - Add AuthHelper class with shared token extraction methods - Update FeatureFlagCache to use AuthHelper.GetAccessToken - Update HttpHandlerFactory to use AuthHelper.GetTokenFromProperties Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move the test factory method from production to test code: - Make FeatureFlagContext constructor internal instead of private - Make Ttl setter internal to allow tests to configure TTL - Remove CreateForTesting from FeatureFlagContext.cs - Add CreateTestContext helper method in FeatureFlagCacheTests.cs This addresses the PR review feedback that test-only code should not be in production source files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Implement the HTTP exporter that sends telemetry events to Databricks service. Key features: - ITelemetryExporter interface with ExportAsync method - Creates TelemetryRequest wrapper with uploadTime and protoLogs - Uses /telemetry-ext for authenticated requests - Uses /telemetry-unauth for unauthenticated requests - Implements retry logic for transient failures - Uses ExceptionClassifier for terminal vs retryable errors - Never throws exceptions (all swallowed and logged at TRACE level) - Cancellation is propagated (not swallowed) Files added: - src/Telemetry/ITelemetryExporter.cs - src/Telemetry/DatabricksTelemetryExporter.cs - test/Unit/Telemetry/DatabricksTelemetryExporterTests.cs Co-Authored-By: Claude <noreply@anthropic.com>

Implement MetricsAggregator that aggregates Activity data by statement_id and handles exception buffering with terminal vs retryable classification. Key features: - ProcessActivity extracts tags and aggregates by statement_id using ConcurrentDictionary<string, StatementTelemetryContext> - CompleteStatement emits aggregated TelemetryEvent - RecordException flushes terminal exceptions immediately - RecordException buffers retryable exceptions until CompleteStatement - FlushAsync exports when batch size or time interval reached - Uses TelemetryTagRegistry to filter tags - Creates TelemetryFrontendLog wrapper with workspace_id - All exceptions swallowed and logged at TRACE level Implementation details: - Connection events emit immediately (no aggregation needed) - Statement events aggregate until CompleteStatement is called - Timer-based periodic flush using System.Threading.Timer - Thread-safe aggregation using ConcurrentDictionary - Nested StatementTelemetryContext holds aggregated metrics and buffered exceptions per statement Test coverage: - 29 unit tests covering all exit criteria - Tests for exception handling, tag filtering, frontend log wrapping - End-to-end statement lifecycle tests Co-Authored-By: Claude <noreply@anthropic.com>

Implement CircuitBreakerManager as a singleton that manages circuit breakers per host. Each host gets its own circuit breaker instance for isolation, preventing one failing endpoint from affecting others. Key features: - Singleton pattern with GetInstance() method - Per-host circuit breaker isolation using ConcurrentDictionary - Thread-safe concurrent access - Case-insensitive host matching - Support for both default and custom configurations This follows the JDBC driver pattern in CircuitBreakerManager.java. Co-Authored-By: Claude <noreply@anthropic.com>

Add comprehensive E2E tests for feature flag fetching from real Databricks endpoints and validate caching and reference counting behavior: - FeatureFlagCache_FetchFromRealEndpoint_ReturnsBoolean: Tests real endpoint - FeatureFlagCache_CachesValue_DoesNotRefetchWithinTTL: Validates caching - FeatureFlagCache_InvalidHost_ReturnsDefaultFalse: Tests error handling - FeatureFlagCache_RefCountingWorks_CleanupAfterRelease: Tests ref counting Additional tests cover: - Cache expiry and refetch behavior - Null/empty host handling - Unknown host behavior - Multiple hosts with independent ref counts - Concurrent reference counting thread safety - False value caching - Cancellation propagation Co-Authored-By: Claude <noreply@anthropic.com>

Add DatabricksActivityListener that listens to 'Databricks.Adbc.Driver' ActivitySource, extracts metrics from activities, and delegates to MetricsAggregator. This implements Phase 5 of the telemetry design. Key features: - ShouldListenTo returns true for 'Databricks.Adbc.Driver' source - Sample callback respects feature flag (AllDataAndRecorded when enabled, None when disabled) - ActivityStopped callback delegates to MetricsAggregator.ProcessActivity - All callbacks wrapped in try-catch with TRACE logging - StopAsync flushes pending metrics via MetricsAggregator.FlushAsync - Supports dynamic feature flag checking via optional Func<bool> Co-Authored-By: Claude <noreply@anthropic.com>

jadewang-db had a problem deploying to azure-prod January 22, 2026 18:42 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 3721021 to 2e123ec Compare January 22, 2026 22:24

jadewang-db had a problem deploying to azure-prod January 22, 2026 22:24 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 2e123ec to 39f6aed Compare January 23, 2026 21:14

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:15 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 39f6aed to 9090bde Compare January 23, 2026 21:43

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:43 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 9090bde to 045b1e5 Compare January 26, 2026 22:43

jadewang-db had a problem deploying to azure-prod January 26, 2026 22:43 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 045b1e5 to 30bed36 Compare January 26, 2026 23:01

jadewang-db had a problem deploying to azure-prod January 26, 2026 23:01 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 95c74d1 to 70df609 Compare January 28, 2026 01:08

jadewang-db had a problem deploying to azure-prod January 28, 2026 01:08 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 70df609 to f2bf2bf Compare January 28, 2026 04:37

jadewang-db had a problem deploying to azure-prod January 28, 2026 04:37 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from f2bf2bf to 87647cf Compare February 2, 2026 22:11

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:11 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 87647cf to 070f2c8 Compare February 2, 2026 22:19

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:19 — with GitHub Actions Error

jadewang-db force-pushed the stack/databricks-activity-listener branch from 070f2c8 to db230ba Compare February 3, 2026 00:50

jadewang-db had a problem deploying to azure-prod February 3, 2026 00:50 — with GitHub Actions Error

Jade Wang and others added 9 commits February 3, 2026 18:10

Address comments and refactoring

80bed7a

jadewang-db force-pushed the stack/databricks-activity-listener branch from db230ba to 3d67065 Compare February 3, 2026 18:24

jadewang-db had a problem deploying to azure-prod February 3, 2026 18:24 — with GitHub Actions Error

Jade Wang and others added 6 commits February 3, 2026 18:31

jadewang-db force-pushed the stack/databricks-activity-listener branch from 3d67065 to abc323e Compare February 3, 2026 18:32

jadewang-db requested a deployment to azure-prod February 3, 2026 18:32 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(csharp): implement DatabricksActivityListener (WI-3.4) #170

feat(csharp): implement DatabricksActivityListener (WI-3.4) #170

Uh oh!

jadewang-db commented Jan 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(csharp): implement DatabricksActivityListener (WI-3.4) #170

Are you sure you want to change the base?

feat(csharp): implement DatabricksActivityListener (WI-3.4) #170

Uh oh!

Conversation

jadewang-db commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jadewang-db commented Jan 22, 2026 •

edited

Loading