Transports: DRY HTTP Transport & Client classes #5099

p-datadog · 2025-11-27T14:33:11Z

What does this PR do?

Removes duplicated code from Transport classes and deletes most of the HTTP::Client classes.

This PR continues on the path started in #5095.

Most of the HTTP::Client classes were functionally identical, except for the Tracing::Traces one which had additional logic for API version downgrades. The difference between the Client classes was in send_XXX_payload methods which were largely identical. They have been replaced by a single send_request method with the XXX as the argument. Then, all of the HTTP::Client classes have been removed except for the Tracing::Traces one which derives from the core HTTP::Client.

This PR moves also moves the downgrading methods to the core class because they will be used by DI in the next PR. Accordingly, some unit tests have been moved to core. Not all because I think there needs to be additional work to have mock APIs, since existing unit tests under Tracing::Traces utilize the APIs in Traces.

Finally, a base Transport class in core has been created with the API management code that was identical across all of the Transport classes in the various components. The per-component Transport classes remain since they still have unique methods but these now all derive from the core base class.

Motivation:
Making transport layer more legible to implement API downgrades for DI

Change log entry
None

Additional Notes:

How to test the change?

Existing tests

github-actions · 2025-11-27T14:33:25Z

Thank you for updating Change log entry section 👏

^{Visited at: 2025-12-02 22:11:17 UTC}

github-actions · 2025-11-27T14:33:39Z

Typing analysis

Note: Ignored files are excluded from the next sections.

`steep:ignore` comments

This PR clears 2 steep:ignore comments.

steep:ignore comments (+0-2)

✅ Cleared:

lib/datadog/core/telemetry/transport/http/telemetry.rb:19
lib/datadog/data_streams/transport/http/client.rb:14

Untyped methods

This PR introduces 7 untyped methods and 2 partially typed methods, and clears 7 untyped methods and 2 partially typed methods.

Untyped methods (+7-7)

❌ Introduced:

sig/datadog/core/transport/transport.rbs:9
└── def initialize: (untyped version) -> void
sig/datadog/core/transport/transport.rbs:18
└── def initialize: (untyped version) -> void
sig/datadog/core/transport/transport.rbs:44
└── def initialize: (untyped apis, untyped default_api, logger: untyped) -> void
sig/datadog/core/transport/transport.rbs:46
└── def current_api: () -> untyped
sig/datadog/core/transport/transport.rbs:50
└── def set_api!: (untyped api_id) -> untyped
sig/datadog/core/transport/transport.rbs:54
└── def downgrade!: () -> untyped
sig/datadog/tracing/transport/traces.rbs:53
└── def stats: () -> untyped

✅ Cleared:

sig/datadog/tracing/transport/traces.rbs:71
└── def initialize: (untyped apis, untyped default_api, ?logger: untyped) -> void
sig/datadog/tracing/transport/traces.rbs:75
└── def stats: () -> untyped
sig/datadog/tracing/transport/traces.rbs:77
└── def current_api: () -> untyped
sig/datadog/tracing/transport/traces.rbs:83
└── def downgrade!: () -> untyped
sig/datadog/tracing/transport/traces.rbs:85
└── def change_api!: (untyped api_id) -> untyped
sig/datadog/tracing/transport/traces.rbs:92
└── def initialize: (untyped version) -> void
sig/datadog/tracing/transport/traces.rbs:101
└── def initialize: (untyped version) -> void

Partially typed methods (+2-2)

❌ Introduced:

sig/datadog/core/transport/transport.rbs:52
└── def downgrade?: (untyped response) -> (false | untyped)
sig/datadog/tracing/transport/traces.rbs:51
└── def send_traces: (Array[Tracing::TraceOperation] traces) -> untyped

✅ Cleared:

sig/datadog/tracing/transport/traces.rbs:73
└── def send_traces: (Array[Tracing::TraceOperation] traces) -> untyped
sig/datadog/tracing/transport/traces.rbs:81
└── def downgrade?: (untyped response) -> (false | untyped)

Untyped other declarations

This PR introduces 15 untyped other declarations, and clears 14 untyped other declarations. It decreases the percentage of typed other declarations from 68.87% to 68.81% (-0.06%).

Untyped other declarations (+15-14)

❌ Introduced:

sig/datadog/core/transport/transport.rbs:5
└── @version: untyped
sig/datadog/core/transport/transport.rbs:7
└── attr_reader version: untyped
sig/datadog/core/transport/transport.rbs:14
└── @version: untyped
sig/datadog/core/transport/transport.rbs:16
└── attr_reader version: untyped
sig/datadog/core/transport/transport.rbs:23
└── @apis: untyped
sig/datadog/core/transport/transport.rbs:25
└── @default_api: untyped
sig/datadog/core/transport/transport.rbs:27
└── @logger: untyped
sig/datadog/core/transport/transport.rbs:29
└── @current_api_id: untyped
sig/datadog/core/transport/transport.rbs:31
└── @client: untyped
sig/datadog/core/transport/transport.rbs:33
└── attr_reader client: untyped
sig/datadog/core/transport/transport.rbs:35
└── attr_reader apis: untyped
sig/datadog/core/transport/transport.rbs:37
└── attr_reader default_api: untyped
sig/datadog/core/transport/transport.rbs:39
└── attr_reader current_api_id: untyped
sig/datadog/core/transport/transport.rbs:41
└── attr_reader logger: untyped
sig/datadog/core/transport/transport.rbs:42
└── attr_accessor self.http_client_class: untyped

✅ Cleared:

sig/datadog/tracing/transport/traces.rbs:49
└── @apis: untyped
sig/datadog/tracing/transport/traces.rbs:51
└── @default_api: untyped
sig/datadog/tracing/transport/traces.rbs:53
└── @logger: untyped
sig/datadog/tracing/transport/traces.rbs:55
└── @current_api_id: untyped
sig/datadog/tracing/transport/traces.rbs:57
└── @client: untyped
sig/datadog/tracing/transport/traces.rbs:61
└── attr_reader client: untyped
sig/datadog/tracing/transport/traces.rbs:63
└── attr_reader apis: untyped
sig/datadog/tracing/transport/traces.rbs:65
└── attr_reader default_api: untyped
sig/datadog/tracing/transport/traces.rbs:67
└── attr_reader current_api_id: untyped
sig/datadog/tracing/transport/traces.rbs:69
└── attr_reader logger: untyped
sig/datadog/tracing/transport/traces.rbs:88
└── @version: untyped
sig/datadog/tracing/transport/traces.rbs:90
└── attr_reader version: untyped
sig/datadog/tracing/transport/traces.rbs:97
└── @version: untyped
sig/datadog/tracing/transport/traces.rbs:99
└── attr_reader version: untyped

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept to the end of the line to remove it from the stats.

datadog-datadog-prod-us1 · 2025-11-27T14:42:35Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c87e32b | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

pr-commenter · 2025-11-27T15:00:38Z

Benchmarks

Benchmark execution time: 2025-12-03 20:38:20

Comparing candidate commit c87e32b in PR branch dry-transport with baseline commit 48fa3f2 in branch master.

Found 2 performance improvements and 1 performance regressions! Performance is the same for 41 metrics, 2 unstable metrics.

scenario:profiling - Allocations (profiling disabled)

🟩 throughput [+275753.174op/s; +282543.999op/s] or [+5.702%; +5.843%]

scenario:profiling - Allocations (profiling enabled)

🟩 throughput [+269961.246op/s; +278054.255op/s] or [+5.625%; +5.794%]

scenario:profiling - intern mixed existing and new

🟥 throughput [-2.850op/s; -1.722op/s] or [-9.131%; -5.517%]

p-datadog · 2025-12-02T22:13:15Z

lib/datadog/core/transport/http/client.rb

          private

-          def send_request(request, &block)
+          def send_request_impl(request, &block)


I would love a better name for this method but can't think of one at the moment.

I think this PR changed every send_request to not pass in a block. Being that the case, why not merge these two and make send_request directly trigger the public_send?

I did this change and tests majorly broke because they do a lot of mocking. I do like the change, I suppose need to spend more time rewriting the tests.

Yeah... somehow not surprised. This is why I mentioned

TBH I suspect in the long run it may be easier to start from scratch and rebuilding the support for the required feature-set VS slowly untangling the current mess back into a good design step by step.

I believe this design is at this point so weird that "chipping" at it is incredibly hard.

ivoanjo

Thanks for picking this up! Left a few notes.

TBH I suspect in the long run it may be easier to start from scratch and rebuilding the support for the required feature-set VS slowly untangling the current mess back into a good design step by step.

But having said that, I'll take any improvement I can get.

ivoanjo · 2025-12-03T12:02:31Z

lib/datadog/core/transport/transport.rb

+      # Raised when configured with an unknown API version
+      class UnknownApiVersionError < StandardError
+        attr_reader :version
+
+        def initialize(version)
+          super
+
+          @version = version
+        end
+
+        def message
+          "No matching transport API for version #{version}!"
+        end
+      end
+
+      # Raised when the API verson to downgrade to does not map to a
+      # defined API.
+      class NoDowngradeAvailableError < StandardError
+        attr_reader :version
+
+        def initialize(version)
+          super
+
+          @version = version
+        end
+
+        def message
+          "No downgrade from transport API version #{version} is available!"
+        end
+      end


So I realize this already partially existed before.

Having said that -- these two situations only happen when we have a bug in our code that can't be recovered from, and we don't particularly have a need to access the version or to recover from the individual errors.

So my suggestion is -- let's get rid of these and replace them with raise ArgumentError, "Unknown API version: #{api_id}" (or similar) below. I think that saves us some code and we can always introduce the more specific classes if we ever need to do anything with them.

I can do this, I don't think I'm strongly attached to the existing exceptions.

ivoanjo · 2025-12-03T12:05:48Z

lib/datadog/core/transport/transport.rb

+        class << self
+          # The HTTP client class to use for requests, derived from
+          # Core::Transport::HTTP::Client.
+          #
+          # Important: this attribute is NOT inherited by derived classes -
+          # it must be set by every Transport class that wants to have a
+          # non-default HTTP::Client instance.
+          attr_accessor :http_client_class
+        end


This... is a kinda weird pattern. May I suggest making it an argument that gets passed to the constructor, or a regular method that needs to be overwritten by subclasses?

My thinking in general was to make the transports more declarative but I can try a version with an instance method.

So I think declarative sometimes works really well...

In our case, I think it's made it really hard to reason and fix things with the current transport design. My thinking is that at this point we should strip out as much as possible of it until we get to a better state. Then, once we're happier with the transport design, we can reintroduce the declarative features as a nicer DSL for underpinnings we're happy with, and it won't affect with the overall design we're trying to fix.

ivoanjo · 2025-12-03T12:08:47Z

lib/datadog/core/transport/transport.rb

+
+      # Base class for transports.
+      class Transport
+        attr_reader :client, :apis, :default_api, :current_api_id, :logger


Minor: I think some (all?) of these can be made private

ivoanjo · 2025-12-03T12:15:19Z

lib/datadog/core/transport/http/client.rb

          private

-          def send_request(request, &block)
+          def send_request_impl(request, &block)


I think this PR changed every send_request to not pass in a block. Being that the case, why not merge these two and make send_request directly trigger the public_send?

ivoanjo · 2025-12-03T12:26:58Z

spec/datadog/core/transport/transport_spec.rb

+  shared_context 'APIs with fallbacks' do
+    let(:current_api_id) { :v2 }
+    let(:apis) do
+      Datadog::Core::Transport::HTTP::API::Map[
+        v2: api_v2,
+        v1: api_v1
+      ].with_fallbacks(v2: :v1)
+    end
+
+    let(:api_v1) { instance_double(Datadog::Core::Transport::HTTP::API::Instance, 'v1', encoder: encoder_v1) }
+    let(:api_v2) { instance_double(Datadog::Core::Transport::HTTP::API::Instance, 'v2', encoder: encoder_v2) }
+    let(:encoder_v1) { instance_double(Datadog::Core::Encoding::Encoder, 'v1', content_type: 'text/plain') }
+    let(:encoder_v2) { instance_double(Datadog::Core::Encoding::Encoder, 'v2', content_type: 'text/csv') }
+  end


Minor: Is it me or does every entry in this spec include this shared_context? E.g. I think we can simplify it

Probably due to copy-pasting from the tracing test.

(I think we overuse shared_contexts sometimes, so I'm always on the look out for simplifying them)

ivoanjo · 2025-12-03T12:28:21Z

spec/datadog/core/transport/transport_spec.rb

+  describe '#initialize' do
+    include_context 'APIs with fallbacks'
+
+    it { is_expected.to have_attributes(apis: apis, current_api_id: current_api_id) }
+  end


Minor: Maybe remove this? This test is trivial and this behavior ends up being exercised separately by the other tests

github-actions bot added core Involves Datadog core libraries tracing labels Nov 27, 2025

p-datadog force-pushed the dry-transport branch 4 times, most recently from ac5f229 to 3108b35 Compare December 2, 2025 15:32

DRY transports by extracting base class with API functionality

f3dc3d2

p-datadog force-pushed the dry-transport branch from 8ba9902 to f3dc3d2 Compare December 2, 2025 22:03

p-datadog changed the title ~~Telemetry: DRY transport~~ Transports: DRY HTTP Client classes Dec 2, 2025

p-datadog commented Dec 2, 2025

View reviewed changes

p-datadog changed the title ~~Transports: DRY HTTP Client classes~~ Transports: DRY HTTP Transport & Client classes Dec 2, 2025

p-datadog marked this pull request as ready for review December 2, 2025 22:15

p-datadog requested review from a team as code owners December 2, 2025 22:15

p-datadog requested a review from vpellan December 2, 2025 22:16

ivoanjo reviewed Dec 3, 2025

View reviewed changes

remove send_request_impl

c87e32b

Transports: DRY HTTP Transport & Client classes #5099

Are you sure you want to change the base?

Transports: DRY HTTP Transport & Client classes #5099

Conversation

p-datadog commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Typing analysis

steep:ignore comments

Untyped methods

Untyped other declarations

Uh oh!

datadog-datadog-prod-us1 bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:profiling - Allocations (profiling disabled)

scenario:profiling - Allocations (profiling enabled)

scenario:profiling - intern mixed existing and new

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

p-datadog commented Nov 27, 2025 •

edited

Loading

github-actions bot commented Nov 27, 2025 •

edited

Loading

github-actions bot commented Nov 27, 2025 •

edited

Loading

`steep:ignore` comments

datadog-datadog-prod-us1 bot commented Nov 27, 2025 •

edited

Loading

pr-commenter bot commented Nov 27, 2025 •

edited

Loading