Databricks support changes #9877

pawel-big-lebowski · 2025-10-29T09:20:27Z

What Does This Do

Spark listeners on databricks environments are instantiated in slightly different way than on regular vanilla Spark setups. This PR assures the datadog listener instruments Openlineage listener correctly on the databricks environment.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-10-29T09:59:05Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/databricks-openlineage-support
git_commit_date	1761927667	1761931096
git_commit_sha	`33bbc70`	`85ed98d`
release_version	1.55.0-SNAPSHOT~33bbc70b29	1.55.0-SNAPSHOT~85ed98d2e0

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1761932939	1761932939
ci_job_id	1209897075	1209897075
ci_pipeline_id	80920587	80920587
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-yfkjd1mi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-yfkjd1mi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 11 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.039 s) : 0, 1039010
Total [baseline] (8.693 s) : 0, 8693285
Agent [candidate] (1.03 s) : 0, 1029911
Total [candidate] (8.66 s) : 0, 8659535
section iast
Agent [baseline] (1.171 s) : 0, 1170616
Total [baseline] (9.358 s) : 0, 9357936
Agent [candidate] (1.167 s) : 0, 1166661
Total [candidate] (9.374 s) : 0, 9374090

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.039 s	-
Agent	iast	1.171 s	131.606 ms (12.7%)
Total	tracing	8.693 s	-
Total	iast	9.358 s	664.651 ms (7.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.03 s	-
Agent	iast	1.167 s	136.75 ms (13.3%)
Total	tracing	8.66 s	-
Total	iast	9.374 s	714.554 ms (8.3%)

gantt
    title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.479 ms) : 0, 1479
crashtracking [candidate] (1.468 ms) : 0, 1468
BytebuddyAgent [baseline] (708.335 ms) : 0, 708335
BytebuddyAgent [candidate] (702.5 ms) : 0, 702500
GlobalTracer [baseline] (246.745 ms) : 0, 246745
GlobalTracer [candidate] (244.768 ms) : 0, 244768
AppSec [baseline] (32.898 ms) : 0, 32898
AppSec [candidate] (32.109 ms) : 0, 32109
Debugger [baseline] (6.55 ms) : 0, 6550
Debugger [candidate] (6.341 ms) : 0, 6341
Remote Config [baseline] (693.012 µs) : 0, 693
Remote Config [candidate] (678.354 µs) : 0, 678
Telemetry [baseline] (14.444 ms) : 0, 14444
Telemetry [candidate] (14.54 ms) : 0, 14540
Flare Poller [baseline] (6.624 ms) : 0, 6624
Flare Poller [candidate] (6.37 ms) : 0, 6370
section iast
crashtracking [baseline] (1.487 ms) : 0, 1487
crashtracking [candidate] (1.477 ms) : 0, 1477
BytebuddyAgent [baseline] (831.394 ms) : 0, 831394
BytebuddyAgent [candidate] (828.231 ms) : 0, 828231
GlobalTracer [baseline] (235.104 ms) : 0, 235104
GlobalTracer [candidate] (234.766 ms) : 0, 234766
AppSec [baseline] (28.189 ms) : 0, 28189
AppSec [candidate] (27.247 ms) : 0, 27247
Debugger [baseline] (6.123 ms) : 0, 6123
Debugger [candidate] (6.112 ms) : 0, 6112
Remote Config [baseline] (613.256 µs) : 0, 613
Remote Config [candidate] (600.655 µs) : 0, 601
Telemetry [baseline] (8.53 ms) : 0, 8530
Telemetry [candidate] (8.432 ms) : 0, 8432
Flare Poller [baseline] (4.118 ms) : 0, 4118
Flare Poller [candidate] (4.088 ms) : 0, 4088
IAST [baseline] (33.571 ms) : 0, 33571
IAST [candidate] (34.295 ms) : 0, 34295

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.032 s) : 0, 1031782
Total [baseline] (10.911 s) : 0, 10911151
Agent [candidate] (1.031 s) : 0, 1031096
Total [candidate] (10.906 s) : 0, 10906194
section appsec
Agent [baseline] (1.203 s) : 0, 1203439
Total [baseline] (10.965 s) : 0, 10965118
Agent [candidate] (1.219 s) : 0, 1218501
Total [candidate] (11.11 s) : 0, 11109558
section iast
Agent [baseline] (1.159 s) : 0, 1159446
Total [baseline] (11.077 s) : 0, 11076868
Agent [candidate] (1.17 s) : 0, 1169505
Total [candidate] (11.141 s) : 0, 11140861
section profiling
Agent [baseline] (1.178 s) : 0, 1178199
Total [baseline] (10.919 s) : 0, 10918775
Agent [candidate] (1.176 s) : 0, 1176096
Total [candidate] (10.912 s) : 0, 10911614

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.032 s	-
Agent	appsec	1.203 s	171.657 ms (16.6%)
Agent	iast	1.159 s	127.664 ms (12.4%)
Agent	profiling	1.178 s	146.417 ms (14.2%)
Total	tracing	10.911 s	-
Total	appsec	10.965 s	53.967 ms (0.5%)
Total	iast	11.077 s	165.717 ms (1.5%)
Total	profiling	10.919 s	7.624 ms (0.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	appsec	1.219 s	187.405 ms (18.2%)
Agent	iast	1.17 s	138.409 ms (13.4%)
Agent	profiling	1.176 s	145.0 ms (14.1%)
Total	tracing	10.906 s	-
Total	appsec	11.11 s	203.363 ms (1.9%)
Total	iast	11.141 s	234.667 ms (2.2%)
Total	profiling	10.912 s	5.42 ms (0.0%)

gantt
    title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.483 ms) : 0, 1483
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (703.622 ms) : 0, 703622
BytebuddyAgent [candidate] (703.043 ms) : 0, 703043
GlobalTracer [baseline] (245.062 ms) : 0, 245062
GlobalTracer [candidate] (245.191 ms) : 0, 245191
AppSec [baseline] (32.25 ms) : 0, 32250
AppSec [candidate] (32.431 ms) : 0, 32431
Debugger [baseline] (6.381 ms) : 0, 6381
Debugger [candidate] (6.374 ms) : 0, 6374
Remote Config [baseline] (678.926 µs) : 0, 679
Remote Config [candidate] (698.283 µs) : 0, 698
Telemetry [baseline] (15.288 ms) : 0, 15288
Telemetry [candidate] (14.308 ms) : 0, 14308
Flare Poller [baseline] (5.738 ms) : 0, 5738
Flare Poller [candidate] (6.438 ms) : 0, 6438
section appsec
crashtracking [baseline] (1.478 ms) : 0, 1478
crashtracking [candidate] (1.489 ms) : 0, 1489
BytebuddyAgent [baseline] (726.639 ms) : 0, 726639
BytebuddyAgent [candidate] (737.206 ms) : 0, 737206
GlobalTracer [baseline] (235.82 ms) : 0, 235820
GlobalTracer [candidate] (239.385 ms) : 0, 239385
AppSec [baseline] (174.481 ms) : 0, 174481
AppSec [candidate] (174.784 ms) : 0, 174784
Debugger [baseline] (5.872 ms) : 0, 5872
Debugger [candidate] (5.937 ms) : 0, 5937
Remote Config [baseline] (627.689 µs) : 0, 628
Remote Config [candidate] (633.25 µs) : 0, 633
Telemetry [baseline] (8.385 ms) : 0, 8385
Telemetry [candidate] (8.463 ms) : 0, 8463
Flare Poller [baseline] (3.877 ms) : 0, 3877
Flare Poller [candidate] (3.959 ms) : 0, 3959
IAST [baseline] (25.05 ms) : 0, 25050
IAST [candidate] (25.284 ms) : 0, 25284
section iast
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.492 ms) : 0, 1492
BytebuddyAgent [baseline] (822.456 ms) : 0, 822456
BytebuddyAgent [candidate] (831.542 ms) : 0, 831542
GlobalTracer [baseline] (233.528 ms) : 0, 233528
GlobalTracer [candidate] (234.854 ms) : 0, 234854
AppSec [baseline] (29.786 ms) : 0, 29786
AppSec [candidate] (27.654 ms) : 0, 27654
Debugger [baseline] (6.045 ms) : 0, 6045
Debugger [candidate] (6.132 ms) : 0, 6132
Remote Config [baseline] (602.689 µs) : 0, 603
Remote Config [candidate] (600.727 µs) : 0, 601
Telemetry [baseline] (8.437 ms) : 0, 8437
Telemetry [candidate] (8.452 ms) : 0, 8452
Flare Poller [baseline] (4.168 ms) : 0, 4168
Flare Poller [candidate] (4.079 ms) : 0, 4079
IAST [baseline] (31.65 ms) : 0, 31650
IAST [candidate] (33.243 ms) : 0, 33243
section profiling
crashtracking [baseline] (1.448 ms) : 0, 1448
crashtracking [candidate] (1.44 ms) : 0, 1440
BytebuddyAgent [baseline] (729.647 ms) : 0, 729647
BytebuddyAgent [candidate] (728.111 ms) : 0, 728111
GlobalTracer [baseline] (221.178 ms) : 0, 221178
GlobalTracer [candidate] (221.172 ms) : 0, 221172
AppSec [baseline] (32.316 ms) : 0, 32316
AppSec [candidate] (32.277 ms) : 0, 32277
Debugger [baseline] (9.944 ms) : 0, 9944
Debugger [candidate] (12.142 ms) : 0, 12142
Remote Config [baseline] (1.523 ms) : 0, 1523
Remote Config [candidate] (1.584 ms) : 0, 1584
Telemetry [baseline] (12.345 ms) : 0, 12345
Telemetry [candidate] (9.894 ms) : 0, 9894
Flare Poller [baseline] (4.124 ms) : 0, 4124
Flare Poller [candidate] (4.068 ms) : 0, 4068
ProfilingAgent [baseline] (109.636 ms) : 0, 109636
ProfilingAgent [candidate] (109.791 ms) : 0, 109791
Profiling [baseline] (110.314 ms) : 0, 110314
Profiling [candidate] (110.413 ms) : 0, 110413

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/databricks-openlineage-support
git_commit_date	1761927667	1761931096
git_commit_sha	`33bbc70`	`85ed98d`
release_version	1.55.0-SNAPSHOT~33bbc70b29	1.55.0-SNAPSHOT~85ed98d2e0

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1761933434	1761933434
ci_job_id	1209897076	1209897076
ci_pipeline_id	80920587	80920587
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-84nrxqqd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-84nrxqqd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 3 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-131.199µs; -94.484µs] or [-6.382%; -4.596%]	unstable [-126.172op/s; +371.609op/s] or [-5.743%; +16.915%]	1.943ms	2319.656op/s	2.056ms	2196.938op/s
scenario:load:insecure-bank:tracing:high_load	worse [+62.795µs; +98.680µs] or [+3.473%; +5.457%]	unstable [-361.833op/s; +161.395op/s] or [-14.566%; +6.497%]	1.889ms	2383.875op/s	1.808ms	2484.094op/s
scenario:load:insecure-bank:iast_FULL:high_load	worse [+331.534µs; +456.504µs] or [+5.934%; +8.171%]	unstable [-131.247op/s; +23.622op/s] or [-15.869%; +2.856%]	5.981ms	773.250op/s	5.587ms	827.062op/s
scenario:load:petclinic:tracing:high_load	worse [+0.913ms; +1.323ms] or [+4.959%; +7.184%]	unstable [-39.967op/s; +11.029op/s] or [-15.768%; +4.351%]	19.530ms	239.000op/s	18.412ms	253.469op/s
scenario:load:petclinic:profiling:high_load	better [-1090.851µs; -665.358µs] or [-5.433%; -3.314%]	unstable [-14.215op/s; +35.403op/s] or [-6.113%; +15.225%]	19.200ms	243.125op/s	20.078ms	232.531op/s
scenario:load:petclinic:no_agent:high_load	better [-1.272ms; -0.862ms] or [-6.712%; -4.552%]	unstable [-13.090op/s; +42.402op/s] or [-5.313%; +17.210%]	17.879ms	261.031op/s	18.946ms	246.375op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.199 ms) : 1187, 1211
.   : milestone, 1199,
iast (3.255 ms) : 3208, 3302
.   : milestone, 3255,
iast_FULL (5.587 ms) : 5532, 5642
.   : milestone, 5587,
iast_GLOBAL (3.58 ms) : 3528, 3633
.   : milestone, 3580,
profiling (2.056 ms) : 2038, 2073
.   : milestone, 2056,
tracing (1.808 ms) : 1793, 1823
.   : milestone, 1808,
section candidate
no_agent (1.203 ms) : 1191, 1215
.   : milestone, 1203,
iast (3.241 ms) : 3198, 3285
.   : milestone, 3241,
iast_FULL (5.981 ms) : 5920, 6042
.   : milestone, 5981,
iast_GLOBAL (3.66 ms) : 3610, 3711
.   : milestone, 3660,
profiling (1.943 ms) : 1926, 1959
.   : milestone, 1943,
tracing (1.889 ms) : 1871, 1907
.   : milestone, 1889,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.199 ms [1.187 ms, 1.211 ms]	-
iast	3.255 ms [3.208 ms, 3.302 ms]	2.056 ms (171.4%)
iast_FULL	5.587 ms [5.532 ms, 5.642 ms]	4.388 ms (365.9%)
iast_GLOBAL	3.58 ms [3.528 ms, 3.633 ms]	2.381 ms (198.5%)
profiling	2.056 ms [2.038 ms, 2.073 ms]	856.333 µs (71.4%)
tracing	1.808 ms [1.793 ms, 1.823 ms]	608.984 µs (50.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.203 ms [1.191 ms, 1.215 ms]	-
iast	3.241 ms [3.198 ms, 3.285 ms]	2.038 ms (169.4%)
iast_FULL	5.981 ms [5.92 ms, 6.042 ms]	4.778 ms (397.2%)
iast_GLOBAL	3.66 ms [3.61 ms, 3.711 ms]	2.457 ms (204.3%)
profiling	1.943 ms [1.926 ms, 1.959 ms]	739.785 µs (61.5%)
tracing	1.889 ms [1.871 ms, 1.907 ms]	686.014 µs (57.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.946 ms) : 18752, 19140
.   : milestone, 18946,
appsec (19.447 ms) : 19242, 19652
.   : milestone, 19447,
code_origins (19.518 ms) : 19321, 19716
.   : milestone, 19518,
iast (18.331 ms) : 18150, 18512
.   : milestone, 18331,
profiling (20.078 ms) : 19879, 20277
.   : milestone, 20078,
tracing (18.412 ms) : 18230, 18594
.   : milestone, 18412,
section candidate
no_agent (17.879 ms) : 17693, 18065
.   : milestone, 17879,
appsec (19.437 ms) : 19236, 19638
.   : milestone, 19437,
code_origins (19.208 ms) : 19017, 19399
.   : milestone, 19208,
iast (18.385 ms) : 18199, 18572
.   : milestone, 18385,
profiling (19.2 ms) : 19004, 19396
.   : milestone, 19200,
tracing (19.53 ms) : 19332, 19729
.   : milestone, 19530,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.946 ms [18.752 ms, 19.14 ms]	-
appsec	19.447 ms [19.242 ms, 19.652 ms]	501.114 µs (2.6%)
code_origins	19.518 ms [19.321 ms, 19.716 ms]	572.413 µs (3.0%)
iast	18.331 ms [18.15 ms, 18.512 ms]	-614.955 µs (-3.2%)
profiling	20.078 ms [19.879 ms, 20.277 ms]	1.132 ms (6.0%)
tracing	18.412 ms [18.23 ms, 18.594 ms]	-533.715 µs (-2.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.879 ms [17.693 ms, 18.065 ms]	-
appsec	19.437 ms [19.236 ms, 19.638 ms]	1.558 ms (8.7%)
code_origins	19.208 ms [19.017 ms, 19.399 ms]	1.329 ms (7.4%)
iast	18.385 ms [18.199 ms, 18.572 ms]	506.328 µs (2.8%)
profiling	19.2 ms [19.004 ms, 19.396 ms]	1.321 ms (7.4%)
tracing	19.53 ms [19.332 ms, 19.729 ms]	1.651 ms (9.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	pawel.leszczynski/databricks-openlineage-support
git_commit_date	1761927667	1761931096
git_commit_sha	`33bbc70`	`85ed98d`
release_version	1.55.0-SNAPSHOT~33bbc70b29	1.55.0-SNAPSHOT~85ed98d2e0

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1761933073	1761933073
ci_job_id	1209897078	1209897078
ci_pipeline_id	80920587	80920587
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-pnjimc6m 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-pnjimc6m 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (2.467 ms) : 2416, 2518
.   : milestone, 2467,
iast (2.203 ms) : 2139, 2266
.   : milestone, 2203,
iast_GLOBAL (2.254 ms) : 2190, 2318
.   : milestone, 2254,
profiling (2.501 ms) : 2328, 2674
.   : milestone, 2501,
tracing (2.033 ms) : 1984, 2083
.   : milestone, 2033,
section candidate
no_agent (1.477 ms) : 1465, 1488
.   : milestone, 1477,
appsec (3.641 ms) : 3428, 3853
.   : milestone, 3641,
iast (2.21 ms) : 2146, 2274
.   : milestone, 2210,
iast_GLOBAL (2.265 ms) : 2201, 2329
.   : milestone, 2265,
profiling (2.064 ms) : 2012, 2115
.   : milestone, 2064,
tracing (2.025 ms) : 1975, 2075
.   : milestone, 2025,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.468 ms, 1.491 ms]	-
appsec	2.467 ms [2.416 ms, 2.518 ms]	987.593 µs (66.8%)
iast	2.203 ms [2.139 ms, 2.266 ms]	723.139 µs (48.9%)
iast_GLOBAL	2.254 ms [2.19 ms, 2.318 ms]	774.542 µs (52.4%)
profiling	2.501 ms [2.328 ms, 2.674 ms]	1.022 ms (69.1%)
tracing	2.033 ms [1.984 ms, 2.083 ms]	553.82 µs (37.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.465 ms, 1.488 ms]	-
appsec	3.641 ms [3.428 ms, 3.853 ms]	2.164 ms (146.6%)
iast	2.21 ms [2.146 ms, 2.274 ms]	733.399 µs (49.7%)
iast_GLOBAL	2.265 ms [2.201 ms, 2.329 ms]	788.359 µs (53.4%)
profiling	2.064 ms [2.012 ms, 2.115 ms]	587.046 µs (39.8%)
tracing	2.025 ms [1.975 ms, 2.075 ms]	548.444 µs (37.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~85ed98d2e0, baseline=1.55.0-SNAPSHOT~33bbc70b29
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.975 s) : 14975000, 14975000
.   : milestone, 14975000,
appsec (14.732 s) : 14732000, 14732000
.   : milestone, 14732000,
iast (18.544 s) : 18544000, 18544000
.   : milestone, 18544000,
iast_GLOBAL (17.95 s) : 17950000, 17950000
.   : milestone, 17950000,
profiling (15.769 s) : 15769000, 15769000
.   : milestone, 15769000,
tracing (15.149 s) : 15149000, 15149000
.   : milestone, 15149000,
section candidate
no_agent (14.917 s) : 14917000, 14917000
.   : milestone, 14917000,
appsec (14.779 s) : 14779000, 14779000
.   : milestone, 14779000,
iast (18.736 s) : 18736000, 18736000
.   : milestone, 18736000,
iast_GLOBAL (18.038 s) : 18038000, 18038000
.   : milestone, 18038000,
profiling (15.275 s) : 15275000, 15275000
.   : milestone, 15275000,
tracing (15.229 s) : 15229000, 15229000
.   : milestone, 15229000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.975 s [14.975 s, 14.975 s]	-
appsec	14.732 s [14.732 s, 14.732 s]	-243.0 ms (-1.6%)
iast	18.544 s [18.544 s, 18.544 s]	3.569 s (23.8%)
iast_GLOBAL	17.95 s [17.95 s, 17.95 s]	2.975 s (19.9%)
profiling	15.769 s [15.769 s, 15.769 s]	794.0 ms (5.3%)
tracing	15.149 s [15.149 s, 15.149 s]	174.0 ms (1.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.917 s [14.917 s, 14.917 s]	-
appsec	14.779 s [14.779 s, 14.779 s]	-138.0 ms (-0.9%)
iast	18.736 s [18.736 s, 18.736 s]	3.819 s (25.6%)
iast_GLOBAL	18.038 s [18.038 s, 18.038 s]	3.121 s (20.9%)
profiling	15.275 s [15.275 s, 15.275 s]	358.0 ms (2.4%)
tracing	15.229 s [15.229 s, 15.229 s]	312.0 ms (2.1%)

github-actions · 2025-10-31T14:39:08Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

mobuchowski · 2025-10-31T14:57:38Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java

+        try {
+          log.debug("Getting OpenLineage conf from the listener");
+          Object openLineageConf = listener.getClass().getMethod("getConf").invoke(listener);
+          if (openLineageConf != null) {
+            InstanceStore.of(SparkConf.class)
+                .put("openLineageSparkConf", (SparkConf) openLineageConf);
+          }
+        } catch (IllegalAccessException | NoSuchMethodException | InvocationTargetException e) {
+          log.warn(
+              "Issue when obtaining OpenLineage conf (possibly unsupported OpenLineage version): {}",
+              e.getMessage());
+        }


Maybe instead of reflection, we should find exactly the databricks forked class, and write an instrumentation similar to what we do for SparkConf itself?

I can try this @mobuchowski.
Reflection is a workaround for the LiveListenerBusAdvice not working on databricks.

This comment has been minimized.

Sign in to view

pawel-big-lebowski force-pushed the pawel.leszczynski/databricks-openlineage-support branch 3 times, most recently from 4a46baf to f8f26bf Compare October 31, 2025 11:55

databricks support changes

cef2b46

pawel-big-lebowski force-pushed the pawel.leszczynski/databricks-openlineage-support branch from f8f26bf to cef2b46 Compare October 31, 2025 13:56

pawel-big-lebowski marked this pull request as ready for review October 31, 2025 14:38

pawel-big-lebowski requested a review from a team as a code owner October 31, 2025 14:38

pawel-big-lebowski added inst: apache spark Apache Spark instrumentation type: enhancement Enhancements and improvements labels Oct 31, 2025

mobuchowski reviewed Oct 31, 2025

View reviewed changes

update log

85ed98d

aboitreaud force-pushed the pawel.leszczynski/databricks-openlineage-support branch from d6565eb to 85ed98d Compare October 31, 2025 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Databricks support changes #9877

Databricks support changes #9877

Uh oh!

pawel-big-lebowski commented Oct 29, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

pr-commenter bot commented Oct 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 31, 2025 •

edited

Loading

Uh oh!

mobuchowski Oct 31, 2025

Uh oh!

pawel-big-lebowski Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Databricks support changes #9877

Are you sure you want to change the base?

Databricks support changes #9877

Uh oh!

Conversation

pawel-big-lebowski commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

This comment has been minimized.

pr-commenter bot commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

github-actions bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mobuchowski Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

pawel-big-lebowski Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pawel-big-lebowski commented Oct 29, 2025 •

edited

Loading

pr-commenter bot commented Oct 29, 2025 •

edited

Loading

github-actions bot commented Oct 31, 2025 •

edited

Loading