Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

charlesmyu · 2025-10-30T17:07:30Z

What Does This Do

The primary addition in this PR is support for the Databricks-specific SparkPlanInfo constructor. The available public methods and constructors on Databricks 17.3 LTS (based on Spark 4.0.0), for example, look like this:

def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String, 
	explainId: Option[Int]
): org.apache.spark.sql.execution.SparkPlanInfo;
def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String
): org.apache.spark.sql.execution.SparkPlanInfo;

val nodeName: <?>;
val simpleString: <?>;
val children: <?>;
val metadata: <?>;
val metrics: <?>;
val estRowCount: <?>;
val rddScopeId: <?>;
val explainId: <?>;

The last three methods (estRowCount, rddScopeId, explainId) are non-standard, and thus must be accounted for in these specific constructors.

Note that we attempt the Databricks constructor first since it has more info, and then the standard constructor. Both are unlikely to exist simultaneously, but this is just to be defensive. In addition, we try to take advantage of the MethodHandles class where possible for reflection, but can't use it to load any of the getters in SparkPlanInfoUtils as all the methods are inherited (MethodHandles uses getDeclaredMethod which excludes inherited methods).

Additional, smaller changes worth noting:

Moves the SparkPlanInfo constructor logic into a dedicated class, SparkPlanInfoUtils.
Makes a small change in AbstractSparkPlanSerializer to improve how we reflect the simpleString method, specifically preferring the .invoke method provided by MethodHandlers as a nice wrapper for exception handling logic.
- This shouldn't change any behaviour of how we get simpleString.
Swaps us from matching on parameter count to matching on explicit parameter types to avoid false matches, especially as we handle a more diverse set of constructors.
We also explicitly reflect all getters (i.e. nodeName(), simpleString(), children(), and metrics()) to avoid version incompatibilities causing muzzle to reject the entire Spark tracer.
- e.g. in Scala 2.13.0 , scala.Seq (used by children and metrics) was migrated from scala.collections.Seq to scala.collections.immutable.Seq.

Motivation

We would like to support Spark jobs for column-level metadata extraction, but they use a different signature for their SparkPlanInfo constructors. This PR fixes that.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: DJM-974

datadog-datadog-prod-us1 · 2025-10-30T17:19:39Z

🎯 Code Coverage
• Patch Coverage: 0.00%
• Total Coverage: 59.54% (-0.17%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 61f7bdb | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

pr-commenter · 2025-10-30T17:49:56Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1762297540	1762369678
git_commit_sha	`8db72c0`	`61f7bdb`
release_version	1.56.0-SNAPSHOT~8db72c0988	1.56.0-SNAPSHOT~61f7bdb5e9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1762371514	1762371514
ci_job_id	1218061635	1218061635
ci_pipeline_id	81372953	81372953
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-3hf23iru 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-3hf23iru 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 12 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048734
Total [baseline] (8.653 s) : 0, 8652631
Agent [candidate] (1.047 s) : 0, 1047061
Total [candidate] (8.648 s) : 0, 8648209
section iast
Agent [baseline] (1.179 s) : 0, 1179086
Total [baseline] (9.252 s) : 0, 9252033
Agent [candidate] (1.177 s) : 0, 1176875
Total [candidate] (9.263 s) : 0, 9263388

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.179 s	130.351 ms (12.4%)
Total	tracing	8.653 s	-
Total	iast	9.252 s	599.403 ms (6.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.177 s	129.815 ms (12.4%)
Total	tracing	8.648 s	-
Total	iast	9.263 s	615.179 ms (7.1%)

gantt
    title insecure-bank - break down per module: candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (705.37 ms) : 0, 705370
BytebuddyAgent [candidate] (704.34 ms) : 0, 704340
GlobalTracer [baseline] (246.414 ms) : 0, 246414
GlobalTracer [candidate] (246.259 ms) : 0, 246259
AppSec [baseline] (32.498 ms) : 0, 32498
AppSec [candidate] (32.376 ms) : 0, 32376
Debugger [baseline] (6.367 ms) : 0, 6367
Debugger [candidate] (6.41 ms) : 0, 6410
Remote Config [baseline] (724.85 µs) : 0, 725
Remote Config [candidate] (704.774 µs) : 0, 705
Telemetry [baseline] (12.306 ms) : 0, 12306
Telemetry [candidate] (15.727 ms) : 0, 15727
Flare Poller [baseline] (8.819 ms) : 0, 8819
Flare Poller [candidate] (5.059 ms) : 0, 5059
section iast
crashtracking [baseline] (1.472 ms) : 0, 1472
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (828.319 ms) : 0, 828319
BytebuddyAgent [candidate] (826.555 ms) : 0, 826555
GlobalTracer [baseline] (234.062 ms) : 0, 234062
GlobalTracer [candidate] (234.148 ms) : 0, 234148
AppSec [baseline] (26.977 ms) : 0, 26977
AppSec [candidate] (28.633 ms) : 0, 28633
Debugger [baseline] (5.994 ms) : 0, 5994
Debugger [candidate] (6.036 ms) : 0, 6036
Remote Config [baseline] (599.365 µs) : 0, 599
Remote Config [candidate] (588.788 µs) : 0, 589
Telemetry [baseline] (8.358 ms) : 0, 8358
Telemetry [candidate] (8.295 ms) : 0, 8295
Flare Poller [baseline] (4.161 ms) : 0, 4161
Flare Poller [candidate] (4.213 ms) : 0, 4213
IAST [baseline] (34.398 ms) : 0, 34398
IAST [candidate] (32.296 ms) : 0, 32296

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056889
Total [baseline] (10.82 s) : 0, 10819933
Agent [candidate] (1.055 s) : 0, 1054827
Total [candidate] (10.735 s) : 0, 10735019
section appsec
Agent [baseline] (1.227 s) : 0, 1227216
Total [baseline] (10.89 s) : 0, 10889524
Agent [candidate] (1.228 s) : 0, 1228081
Total [candidate] (10.908 s) : 0, 10908033
section iast
Agent [baseline] (1.189 s) : 0, 1188788
Total [baseline] (11.253 s) : 0, 11253256
Agent [candidate] (1.184 s) : 0, 1183959
Total [candidate] (11.194 s) : 0, 11194173
section profiling
Agent [baseline] (1.191 s) : 0, 1190712
Total [baseline] (10.95 s) : 0, 10949972
Agent [candidate] (1.203 s) : 0, 1202755
Total [candidate] (10.938 s) : 0, 10937529

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	appsec	1.227 s	170.326 ms (16.1%)
Agent	iast	1.189 s	131.898 ms (12.5%)
Agent	profiling	1.191 s	133.823 ms (12.7%)
Total	tracing	10.82 s	-
Total	appsec	10.89 s	69.59 ms (0.6%)
Total	iast	11.253 s	433.323 ms (4.0%)
Total	profiling	10.95 s	130.039 ms (1.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.228 s	173.254 ms (16.4%)
Agent	iast	1.184 s	129.132 ms (12.2%)
Agent	profiling	1.203 s	147.928 ms (14.0%)
Total	tracing	10.735 s	-
Total	appsec	10.908 s	173.014 ms (1.6%)
Total	iast	11.194 s	459.154 ms (4.3%)
Total	profiling	10.938 s	202.51 ms (1.9%)

gantt
    title petclinic - break down per module: candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.483 ms) : 0, 1483
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (710.828 ms) : 0, 710828
BytebuddyAgent [candidate] (709.762 ms) : 0, 709762
GlobalTracer [baseline] (248.493 ms) : 0, 248493
GlobalTracer [candidate] (247.887 ms) : 0, 247887
AppSec [baseline] (32.801 ms) : 0, 32801
AppSec [candidate] (32.634 ms) : 0, 32634
Debugger [baseline] (6.535 ms) : 0, 6535
Debugger [candidate] (6.517 ms) : 0, 6517
Remote Config [baseline] (712.595 µs) : 0, 713
Remote Config [candidate] (730.29 µs) : 0, 730
Telemetry [baseline] (13.681 ms) : 0, 13681
Telemetry [candidate] (12.867 ms) : 0, 12867
Flare Poller [baseline] (7.275 ms) : 0, 7275
Flare Poller [candidate] (8.096 ms) : 0, 8096
section appsec
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (731.824 ms) : 0, 731824
BytebuddyAgent [candidate] (732.48 ms) : 0, 732480
GlobalTracer [baseline] (238.609 ms) : 0, 238609
GlobalTracer [candidate] (238.853 ms) : 0, 238853
AppSec [baseline] (175.914 ms) : 0, 175914
AppSec [candidate] (175.654 ms) : 0, 175654
Debugger [baseline] (6.036 ms) : 0, 6036
Debugger [candidate] (6.084 ms) : 0, 6084
Remote Config [baseline] (653.125 µs) : 0, 653
Remote Config [candidate] (658.078 µs) : 0, 658
Telemetry [baseline] (8.649 ms) : 0, 8649
Telemetry [candidate] (8.727 ms) : 0, 8727
Flare Poller [baseline] (3.996 ms) : 0, 3996
Flare Poller [candidate] (4.001 ms) : 0, 4001
IAST [baseline] (25.069 ms) : 0, 25069
IAST [candidate] (25.114 ms) : 0, 25114
section iast
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.471 ms) : 0, 1471
BytebuddyAgent [baseline] (835.689 ms) : 0, 835689
BytebuddyAgent [candidate] (832.1 ms) : 0, 832100
GlobalTracer [baseline] (235.981 ms) : 0, 235981
GlobalTracer [candidate] (235.254 ms) : 0, 235254
AppSec [baseline] (29.823 ms) : 0, 29823
AppSec [candidate] (30.302 ms) : 0, 30302
Debugger [baseline] (6.087 ms) : 0, 6087
Debugger [candidate] (6.012 ms) : 0, 6012
Remote Config [baseline] (609.772 µs) : 0, 610
Remote Config [candidate] (606.031 µs) : 0, 606
Telemetry [baseline] (8.475 ms) : 0, 8475
Telemetry [candidate] (8.424 ms) : 0, 8424
Flare Poller [baseline] (4.127 ms) : 0, 4127
Flare Poller [candidate] (4.147 ms) : 0, 4147
IAST [baseline] (31.715 ms) : 0, 31715
IAST [candidate] (30.825 ms) : 0, 30825
section profiling
ProfilingAgent [baseline] (110.057 ms) : 0, 110057
ProfilingAgent [candidate] (111.507 ms) : 0, 111507
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (728.265 ms) : 0, 728265
BytebuddyAgent [candidate] (735.93 ms) : 0, 735930
GlobalTracer [baseline] (221.929 ms) : 0, 221929
GlobalTracer [candidate] (223.554 ms) : 0, 223554
AppSec [baseline] (31.89 ms) : 0, 31890
AppSec [candidate] (32.444 ms) : 0, 32444
Debugger [baseline] (9.278 ms) : 0, 9278
Debugger [candidate] (9.137 ms) : 0, 9137
Remote Config [baseline] (689.643 µs) : 0, 690
Remote Config [candidate] (728.057 µs) : 0, 728
Telemetry [baseline] (13.702 ms) : 0, 13702
Telemetry [candidate] (14.01 ms) : 0, 14010
Flare Poller [baseline] (4.098 ms) : 0, 4098
Flare Poller [candidate] (4.147 ms) : 0, 4147
Profiling [baseline] (110.725 ms) : 0, 110725
Profiling [candidate] (112.176 ms) : 0, 112176

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1762297540	1762369678
git_commit_sha	`8db72c0`	`61f7bdb`
release_version	1.56.0-SNAPSHOT~8db72c0988	1.56.0-SNAPSHOT~61f7bdb5e9

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1762372002	1762372002
ci_job_id	1218061638	1218061638
ci_pipeline_id	81372953	81372953
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-zrqhfri5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-zrqhfri5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 3 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:appsec:high_load	better [-1046.275µs; -621.156µs] or [-5.340%; -3.170%]	unstable [-16.396op/s; +37.458op/s] or [-6.882%; +15.722%]	18.758ms	248.781op/s	19.592ms	238.250op/s
scenario:load:petclinic:tracing:high_load	worse [+383.734µs; +779.749µs] or [+2.133%; +4.334%]	unstable [-36.192op/s; +19.942op/s] or [-13.953%; +7.688%]	18.575ms	251.250op/s	17.993ms	259.375op/s
scenario:load:petclinic:no_agent:high_load	worse [+1.745ms; +2.134ms] or [+10.196%; +12.465%]	unstable [-56.635op/s; +1.573op/s] or [-20.781%; +0.577%]	19.056ms	245.000op/s	17.117ms	272.531op/s
scenario:load:petclinic:iast:high_load	worse [+704.472µs; +1099.060µs] or [+3.977%; +6.205%]	unstable [-41.019op/s; +15.581op/s] or [-15.574%; +5.916%]	18.616ms	250.656op/s	17.714ms	263.375op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.117 ms) : 16949, 17285
.   : milestone, 17117,
appsec (19.592 ms) : 19391, 19793
.   : milestone, 19592,
code_origins (17.985 ms) : 17805, 18166
.   : milestone, 17985,
iast (17.714 ms) : 17536, 17892
.   : milestone, 17714,
profiling (18.465 ms) : 18282, 18649
.   : milestone, 18465,
tracing (17.993 ms) : 17811, 18175
.   : milestone, 17993,
section candidate
no_agent (19.056 ms) : 18864, 19248
.   : milestone, 19056,
appsec (18.758 ms) : 18564, 18953
.   : milestone, 18758,
code_origins (17.904 ms) : 17724, 18084
.   : milestone, 17904,
iast (18.616 ms) : 18427, 18804
.   : milestone, 18616,
profiling (18.452 ms) : 18272, 18632
.   : milestone, 18452,
tracing (18.575 ms) : 18388, 18761
.   : milestone, 18575,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.117 ms [16.949 ms, 17.285 ms]	-
appsec	19.592 ms [19.391 ms, 19.793 ms]	2.475 ms (14.5%)
code_origins	17.985 ms [17.805 ms, 18.166 ms]	868.767 µs (5.1%)
iast	17.714 ms [17.536 ms, 17.892 ms]	597.096 µs (3.5%)
profiling	18.465 ms [18.282 ms, 18.649 ms]	1.348 ms (7.9%)
tracing	17.993 ms [17.811 ms, 18.175 ms]	876.086 µs (5.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.056 ms [18.864 ms, 19.248 ms]	-
appsec	18.758 ms [18.564 ms, 18.953 ms]	-297.704 µs (-1.6%)
code_origins	17.904 ms [17.724 ms, 18.084 ms]	-1.152 ms (-6.0%)
iast	18.616 ms [18.427 ms, 18.804 ms]	-440.544 µs (-2.3%)
profiling	18.452 ms [18.272 ms, 18.632 ms]	-604.184 µs (-3.2%)
tracing	18.575 ms [18.388 ms, 18.761 ms]	-481.578 µs (-2.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.201 ms) : 1189, 1212
.   : milestone, 1201,
iast (3.189 ms) : 3153, 3225
.   : milestone, 3189,
iast_FULL (5.771 ms) : 5713, 5829
.   : milestone, 5771,
iast_GLOBAL (3.66 ms) : 3591, 3728
.   : milestone, 3660,
profiling (1.912 ms) : 1896, 1928
.   : milestone, 1912,
tracing (1.824 ms) : 1807, 1841
.   : milestone, 1824,
section candidate
no_agent (1.204 ms) : 1192, 1215
.   : milestone, 1204,
iast (3.224 ms) : 3186, 3262
.   : milestone, 3224,
iast_FULL (5.603 ms) : 5549, 5658
.   : milestone, 5603,
iast_GLOBAL (3.628 ms) : 3575, 3680
.   : milestone, 3628,
profiling (1.9 ms) : 1884, 1916
.   : milestone, 1900,
tracing (1.797 ms) : 1783, 1812
.   : milestone, 1797,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.201 ms [1.189 ms, 1.212 ms]	-
iast	3.189 ms [3.153 ms, 3.225 ms]	1.988 ms (165.6%)
iast_FULL	5.771 ms [5.713 ms, 5.829 ms]	4.57 ms (380.6%)
iast_GLOBAL	3.66 ms [3.591 ms, 3.728 ms]	2.459 ms (204.8%)
profiling	1.912 ms [1.896 ms, 1.928 ms]	711.382 µs (59.2%)
tracing	1.824 ms [1.807 ms, 1.841 ms]	623.281 µs (51.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.204 ms [1.192 ms, 1.215 ms]	-
iast	3.224 ms [3.186 ms, 3.262 ms]	2.021 ms (167.9%)
iast_FULL	5.603 ms [5.549 ms, 5.658 ms]	4.4 ms (365.6%)
iast_GLOBAL	3.628 ms [3.575 ms, 3.68 ms]	2.424 ms (201.4%)
profiling	1.9 ms [1.884 ms, 1.916 ms]	696.638 µs (57.9%)
tracing	1.797 ms [1.783 ms, 1.812 ms]	593.825 µs (49.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1762297540	1762369678
git_commit_sha	`8db72c0`	`61f7bdb`
release_version	1.56.0-SNAPSHOT~8db72c0988	1.56.0-SNAPSHOT~61f7bdb5e9

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1762371660	1762371660
ci_job_id	1218061640	1218061640
ci_pipeline_id	81372953	81372953
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-rnm0sqj4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-rnm0sqj4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1488
.   : milestone, 1476,
appsec (3.689 ms) : 3471, 3907
.   : milestone, 3689,
iast (2.207 ms) : 2144, 2271
.   : milestone, 2207,
iast_GLOBAL (2.25 ms) : 2186, 2314
.   : milestone, 2250,
profiling (2.066 ms) : 2014, 2119
.   : milestone, 2066,
tracing (2.032 ms) : 1982, 2082
.   : milestone, 2032,
section candidate
no_agent (1.475 ms) : 1464, 1487
.   : milestone, 1475,
appsec (3.598 ms) : 3389, 3807
.   : milestone, 3598,
iast (2.201 ms) : 2137, 2264
.   : milestone, 2201,
iast_GLOBAL (2.244 ms) : 2180, 2308
.   : milestone, 2244,
profiling (2.066 ms) : 2013, 2118
.   : milestone, 2066,
tracing (2.022 ms) : 1973, 2071
.   : milestone, 2022,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.464 ms, 1.488 ms]	-
appsec	3.689 ms [3.471 ms, 3.907 ms]	2.213 ms (149.9%)
iast	2.207 ms [2.144 ms, 2.271 ms]	731.293 µs (49.5%)
iast_GLOBAL	2.25 ms [2.186 ms, 2.314 ms]	773.719 µs (52.4%)
profiling	2.066 ms [2.014 ms, 2.119 ms]	590.174 µs (40.0%)
tracing	2.032 ms [1.982 ms, 2.082 ms]	555.691 µs (37.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.464 ms, 1.487 ms]	-
appsec	3.598 ms [3.389 ms, 3.807 ms]	2.123 ms (143.9%)
iast	2.201 ms [2.137 ms, 2.264 ms]	725.13 µs (49.1%)
iast_GLOBAL	2.244 ms [2.18 ms, 2.308 ms]	768.673 µs (52.1%)
profiling	2.066 ms [2.013 ms, 2.118 ms]	590.331 µs (40.0%)
tracing	2.022 ms [1.973 ms, 2.071 ms]	546.528 µs (37.0%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~61f7bdb5e9, baseline=1.56.0-SNAPSHOT~8db72c0988
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.55 s) : 15550000, 15550000
.   : milestone, 15550000,
appsec (14.96 s) : 14960000, 14960000
.   : milestone, 14960000,
iast (18.232 s) : 18232000, 18232000
.   : milestone, 18232000,
iast_GLOBAL (17.952 s) : 17952000, 17952000
.   : milestone, 17952000,
profiling (14.981 s) : 14981000, 14981000
.   : milestone, 14981000,
tracing (14.788 s) : 14788000, 14788000
.   : milestone, 14788000,
section candidate
no_agent (15.229 s) : 15229000, 15229000
.   : milestone, 15229000,
appsec (15.145 s) : 15145000, 15145000
.   : milestone, 15145000,
iast (18.456 s) : 18456000, 18456000
.   : milestone, 18456000,
iast_GLOBAL (17.69 s) : 17690000, 17690000
.   : milestone, 17690000,
profiling (14.981 s) : 14981000, 14981000
.   : milestone, 14981000,
tracing (14.846 s) : 14846000, 14846000
.   : milestone, 14846000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.55 s [15.55 s, 15.55 s]	-
appsec	14.96 s [14.96 s, 14.96 s]	-590.0 ms (-3.8%)
iast	18.232 s [18.232 s, 18.232 s]	2.682 s (17.2%)
iast_GLOBAL	17.952 s [17.952 s, 17.952 s]	2.402 s (15.4%)
profiling	14.981 s [14.981 s, 14.981 s]	-569.0 ms (-3.7%)
tracing	14.788 s [14.788 s, 14.788 s]	-762.0 ms (-4.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.229 s [15.229 s, 15.229 s]	-
appsec	15.145 s [15.145 s, 15.145 s]	-84.0 ms (-0.6%)
iast	18.456 s [18.456 s, 18.456 s]	3.227 s (21.2%)
iast_GLOBAL	17.69 s [17.69 s, 17.69 s]	2.461 s (16.2%)
profiling	14.981 s [14.981 s, 14.981 s]	-248.0 ms (-1.6%)
tracing	14.846 s [14.846 s, 14.846 s]	-383.0 ms (-2.5%)

pawel-big-lebowski · 2025-10-31T09:34:01Z

...trumentation/spark/src/main/java/datadog/trace/instrumentation/spark/SparkPlanInfoUtils.java

+            SparkPlanInfo.class,
+            String.class,
+            String.class,
+            scala.collection.immutable.Seq.class,


One of the most painful discrepancies between Scala 2.12 and 2.13 is in scala.Seq. And these bugs are hard to tackle.

I think it makes sense having two separate Spark212PlanInfoUtils2 and Spark213PlanInfoUtils classes, at least for the constructors and databricksConstructor part. This assures code is compiled with the same Scala version as it's run.

Yeah, this makes sense - I was worried initially it was Spark version incompatibilities but if it's Scala there isn't much reason to not have two separate files. I ended up putting them in completely separate files with no common abstraction since I couldn't find a clean way of separating them for now, but I imagine down the line when we need to add more constructors that may become more appealing. Let me know what you think!

261e9b1 (#9888)

…rk fork

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch 2 times, most recently from 06f8fee to f649787 Compare October 30, 2025 22:09

charlesmyu added the inst: apache spark Apache Spark instrumentation label Oct 30, 2025

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from f649787 to 03b891a Compare October 30, 2025 22:33

Base automatically changed from charles.yu/djm-974/reflect-spi-constructor to adrien.boitreaud/fix-muzzle-mismatch October 31, 2025 08:53

pawel-big-lebowski reviewed Oct 31, 2025

View reviewed changes

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from 03b891a to 078c8db Compare October 31, 2025 13:49

Base automatically changed from adrien.boitreaud/fix-muzzle-mismatch to master October 31, 2025 14:35

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch 4 times, most recently from 3c7c187 to ad8c7bb Compare November 5, 2025 03:14

charlesmyu added 3 commits November 5, 2025 10:17

Use SparkPlanInfo constructor that is compatible with Databricks' Spa…

a21283c

…rk fork

Move Spark Plan utils to distinct files for Scala 2.12 and 2.13

e849337

Log error when Databricks args fail to parse

80b72c5

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from ad8c7bb to ac0f2c7 Compare November 5, 2025 18:25

Databricks log debugging

61f7bdb

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from ac0f2c7 to 61f7bdb Compare November 5, 2025 19:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

charlesmyu commented Oct 30, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

pawel-big-lebowski Oct 31, 2025

Uh oh!

charlesmyu Oct 31, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Are you sure you want to change the base?

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Conversation

charlesmyu commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1 bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

pawel-big-lebowski Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

charlesmyu Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charlesmyu commented Oct 30, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Oct 30, 2025 •

edited

Loading

pr-commenter bot commented Oct 30, 2025 •

edited

Loading

charlesmyu Oct 31, 2025 •

edited

Loading