feat(csharp): implement DatabricksTelemetryExporter (WI-3.4) #166

jadewang-db · 2026-01-22T18:42:13Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/wi-3.1-feature-flag-cache [Files changed]
- stack/wi-3.4-databricks-telemetry-exporter [Files changed]
  - stack/wi-3.5-metrics-aggregator [Files changed]
    - stack/wi-3.5-circuit-breaker-manager [Files changed]
      - stack/e2e-feature-flag-cache-tests [Files changed]
        
        stack/databricks-activity-listener [Files changed]
        
        stack/circuit-breaker-telemetry-exporter [Files changed]
        
        stack/telemetry-client-manager-wi-3.2 [Files changed]
        
        stack/telemetry-client-wi-5.5 [Files changed]
        
        stack/telemetry-client-manager-e2e-wi-7 [Files changed]
        stack/telemetry-client-e2e-tests-wi-7-standalone [Files changed]
        stack/wi-6.1-databricks-connection-telemetry-integration [Files changed]
        stack/wi-6.2-telemetry-tags-driver-activities [Files changed]
        stack/wi-9-full-integration-e2e-tests [Files changed]

jadewang-db · 2026-01-23T18:16:40Z

csharp/src/Telemetry/DatabricksTelemetryExporter.cs

+
+                    await SendRequestAsync(endpointUrl, json, ct).ConfigureAwait(false);
+
+                    Debug.WriteLine($"[TRACE] DatabricksTelemetryExporter: Successfully exported telemetry to {endpointUrl}");


please follow other part of the driver using activity tracer for tracing

Done - Replaced Debug.WriteLine with ActivitySource-based tracing following the pattern used in FeatureFlagCache.cs and other parts of the driver. Added activity events for: success, retry, terminal error, and exhausted retries.

jadewang-db · 2026-01-23T18:17:07Z

csharp/src/Telemetry/DatabricksTelemetryExporter.cs

+        /// <summary>
+        /// Sends the telemetry request with retry logic.
+        /// </summary>
+        private async Task SendWithRetryAsync(string json, CancellationToken ct)


consider using polly for retry.

Polly is not currently used in this project, and adding it would introduce a new dependency. The existing retry pattern in RetryHttpHandler.cs also uses manual loops. For telemetry which has simple requirements (fixed delay, limited retries, never block driver operations), the current implementation is sufficient and consistent with the rest of the codebase. If we decide to adopt Polly project-wide in the future, we can revisit this.

jadewang-db · 2026-02-02T22:26:25Z

csharp/test/E2E/Telemetry/ClientTelemetryE2ETests.cs

+            OutputHelper?.WriteLine($"Created {logs.Count} telemetry logs for batch send");
+
+            // Send telemetry - should not throw
+            var exception = await Record.ExceptionAsync(() => exporter.ExportAsync(logs));


even we are not throwing the exeception, but we should verify that the exportAsync succeed, eg we got 200 response from server.

maybe return a succeed flag incidating success or not.

This was already addressed in a previous commit - ExportAsync now returns Task<bool> where true indicates a successful HTTP 2xx response and false indicates failure. The E2E tests have also been updated to verify success using Assert.True(success, ...).

Implements per-host feature flag caching with reference counting to avoid repeated API calls and rate limiting. Key features: - FeatureFlagContext: Holds cached telemetry enabled state, last fetched timestamp, reference count, and configurable cache duration (default 15 min) - FeatureFlagCache: Singleton managing per-host contexts with thread-safe ConcurrentDictionary storage API: - GetInstance(): Returns the singleton instance - GetOrCreateContext(host): Creates/returns context and increments RefCount - ReleaseContext(host): Decrements RefCount, removes context when zero - IsTelemetryEnabledAsync(): Returns cached value if valid, otherwise fetches Thread safety ensured via ConcurrentDictionary and Interlocked operations. Includes 46 comprehensive unit tests covering all exit criteria. Co-Authored-By: Claude <noreply@anthropic.com>

…(WI-3.1) Refactored FeatureFlagCache based on updated design doc requirements: - Moved from Telemetry namespace to root namespace (AdbcDrivers.Databricks) to make it a generic, reusable component - Added HTTP API integration to fetch flags from /api/2.0/connector-service/feature-flags/OSS_JDBC/{version} - Implemented background refresh scheduler with server-provided TTL - Added FeatureFlagsResponse model for API response parsing - Updated FeatureFlagContext interface: - GetFlagValue(string) - get individual flag value - GetAllFlags() - get all cached flags as dictionary - IsFeatureEnabled(string) - check if flag is "true" - Shutdown() - stop background refresh scheduler - IDisposable for proper cleanup - Updated FeatureFlagCache.GetOrCreateContext() to accept HttpClient and driver version parameters - Updated all unit tests for new interface Co-Authored-By: Claude <noreply@anthropic.com>

…I-3.1) Integrated feature flag cache into the connection lifecycle: - Fetch feature flags from server during connection initialization - Merge flags into Properties dictionary with proper priority: User Properties > Feature Flags > Driver Defaults - Track host for proper context cleanup on Dispose - Release feature flag context when connection is disposed - All feature flag operations are fail-safe (errors logged, not thrown) The feature flag endpoint used is: GET /api/2.0/connector-service/feature-flags/OSS_JDBC/{driver_version} Co-Authored-By: Claude <noreply@anthropic.com>

…-3.1) - Add EnsureSuccessStatusCode pattern for HTTP response handling - Extract common HTTP fetch code into single FetchFeatureFlags method - Make feature flag endpoint configurable via optional parameter - Replace Debug.WriteLine with Activity trace pattern - Add E2E tests for FeatureFlagCache using real Databricks instance Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…WI-3.1) - Move MergePropertiesWithFeatureFlags, TryGetHost, CreateFeatureFlagHttpClient, and MergeProperties helper methods from DatabricksConnection to FeatureFlagCache - Replace Debug.WriteLine with ActivitySource tracing for structured events - DatabricksConnection now delegates to FeatureFlagCache.GetInstance().MergePropertiesWithFeatureFlags() Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…ructor (WI-3.1) Replace hardcoded "1.0.0" with ApacheUtility.GetAssemblyVersion() to use the actual driver version in the test constructor. Co-Authored-By: Claude (databricks-claude-opus-4-5) <noreply@anthropic.com>

…-3.1) - Add proxy support using HiveServer2ProxyConfigurator - Handle protocol prefix in host (e.g., "https://myhost.databricks.com") - Add configurable timeout via FeatureFlagTimeoutSeconds parameter - Use consistent User-Agent format: DatabricksJDBCDriverOSS/{version} (ADBC) - Rename variables to localProperties/remoteProperties for clarity - Remove IsFeatureEnabled method from FeatureFlagContext - Use EnsureSuccessOrThrow extension method for HTTP error handling - Enhance E2E tests to verify flags fetched and cache cleanup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add support for OAuth client_credentials (M2M) authentication in addition to token-based (PAT) auth for feature flag API calls. This ensures feature flags work with all supported authentication methods. - Add AuthHelper class with shared token extraction methods - Update FeatureFlagCache to use AuthHelper.GetAccessToken - Update HttpHandlerFactory to use AuthHelper.GetTokenFromProperties Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move the test factory method from production to test code: - Make FeatureFlagContext constructor internal instead of private - Make Ttl setter internal to allow tests to configure TTL - Remove CreateForTesting from FeatureFlagContext.cs - Add CreateTestContext helper method in FeatureFlagCacheTests.cs This addresses the PR review feedback that test-only code should not be in production source files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Implement the HTTP exporter that sends telemetry events to Databricks service. Key features: - ITelemetryExporter interface with ExportAsync method - Creates TelemetryRequest wrapper with uploadTime and protoLogs - Uses /telemetry-ext for authenticated requests - Uses /telemetry-unauth for unauthenticated requests - Implements retry logic for transient failures - Uses ExceptionClassifier for terminal vs retryable errors - Never throws exceptions (all swallowed and logged at TRACE level) - Cancellation is propagated (not swallowed) Files added: - src/Telemetry/ITelemetryExporter.cs - src/Telemetry/DatabricksTelemetryExporter.cs - test/Unit/Telemetry/DatabricksTelemetryExporterTests.cs Co-Authored-By: Claude <noreply@anthropic.com>

jadewang-db had a problem deploying to azure-prod January 22, 2026 18:42 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 2ec6d67 to d92ee73 Compare January 22, 2026 19:27

jadewang-db had a problem deploying to azure-prod January 22, 2026 19:27 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from d92ee73 to 254a4aa Compare January 22, 2026 22:24

jadewang-db had a problem deploying to azure-prod January 22, 2026 22:24 — with GitHub Actions Error

jadewang-db marked this pull request as ready for review January 22, 2026 22:30

jadewang-db commented Jan 23, 2026

View reviewed changes

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 254a4aa to eb382cb Compare January 23, 2026 21:14

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:15 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from eb382cb to a6e926c Compare January 23, 2026 21:43

jadewang-db had a problem deploying to azure-prod January 23, 2026 21:43 — with GitHub Actions Error

jadewang-db had a problem deploying to azure-prod January 28, 2026 00:11 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 5ff7011 to 7dc3993 Compare January 28, 2026 01:08

jadewang-db had a problem deploying to azure-prod January 28, 2026 01:08 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 7dc3993 to c4383b5 Compare January 28, 2026 04:37

jadewang-db had a problem deploying to azure-prod January 28, 2026 04:37 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from c4383b5 to 9fc10b1 Compare February 2, 2026 22:11

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:11 — with GitHub Actions Error

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 9fc10b1 to 2481ccf Compare February 2, 2026 22:19

jadewang-db had a problem deploying to azure-prod February 2, 2026 22:19 — with GitHub Actions Error

jadewang-db requested review from eric-wang-1990 and samikshya-db February 2, 2026 22:23

jadewang-db commented Feb 2, 2026

View reviewed changes

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 2481ccf to 2147cce Compare February 3, 2026 00:50

jadewang-db had a problem deploying to azure-prod February 3, 2026 00:50 — with GitHub Actions Error

Jade Wang and others added 9 commits February 3, 2026 18:10

Address comments and refactoring

80bed7a

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from 2147cce to ad67802 Compare February 3, 2026 18:24

jadewang-db had a problem deploying to azure-prod February 3, 2026 18:24 — with GitHub Actions Error

Jade Wang and others added 2 commits February 3, 2026 18:31

jadewang-db force-pushed the stack/wi-3.4-databricks-telemetry-exporter branch from ad67802 to 55938fe Compare February 3, 2026 18:32

jadewang-db requested a deployment to azure-prod February 3, 2026 18:32 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(csharp): implement DatabricksTelemetryExporter (WI-3.4) #166

feat(csharp): implement DatabricksTelemetryExporter (WI-3.4) #166

Uh oh!

jadewang-db commented Jan 22, 2026 •

edited

Loading

Uh oh!

jadewang-db Jan 23, 2026

Uh oh!

jadewang-db Feb 3, 2026

Uh oh!

jadewang-db Jan 23, 2026

Uh oh!

jadewang-db Feb 3, 2026

Uh oh!

jadewang-db Feb 2, 2026

Uh oh!

jadewang-db Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		await SendRequestAsync(endpointUrl, json, ct).ConfigureAwait(false);

		Debug.WriteLine($"[TRACE] DatabricksTelemetryExporter: Successfully exported telemetry to {endpointUrl}");

feat(csharp): implement DatabricksTelemetryExporter (WI-3.4) #166

Are you sure you want to change the base?

feat(csharp): implement DatabricksTelemetryExporter (WI-3.4) #166

Uh oh!

Conversation

jadewang-db commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Uh oh!

jadewang-db Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

jadewang-db Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

jadewang-db Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

jadewang-db Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

jadewang-db Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

jadewang-db Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jadewang-db commented Jan 22, 2026 •

edited

Loading