[WIP][SPARK-54314][PYTHON][CONNECT] Improve Server-Side debuggability in Spark Connect by capturing client application's file name and line numbers #53076
+548
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t application's file name and line numbers in PySpark
holdenk
reviewed
Nov 17, 2025
Contributor
holdenk
left a comment
holdenk
left a comment
There was a problem hiding this comment.
This looks neat. If you can rebase it and run the linter that would be awesome. Thanks for working to improve the debugging experience for PySpark connect users :)
Comment on lines
+630
to
+631
| if _is_pyspark_source(filename): | ||
| break |
Contributor
There was a problem hiding this comment.
Maybe add a quick comment on why we're stopping as soon as we encounter a pyspark source file path, presumably this is because we don't need to bubble up to the user the specific client call we've got?
| List[any_pb2.Any]: A list of Any objects, each representing a stack frame in the call stack trace in the user code. | ||
| """ | ||
| call_stack_trace = [] | ||
| if os.getenv("SPARK_CONNECT_DEBUG_CLIENT_CALL_STACK", "false").lower() in ("true", "1"): |
Contributor
There was a problem hiding this comment.
Why system env variable instead of Spark configuration flag? Also if we're adding a new configuration option we should probably document it somewhere (if it's a spark conf flag we have the doc in-line sort of already)
HyukjinKwon
changed the title
HyukjinKwon
changed the title
Member
|
cc @zhengruifeng and @ueshin FYI |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Optionally transmitting client-side code location details (function name, file name and line number) along with actions.
Why are the changes needed?
Right now there is no information sent to Spark Connect server that will aid in pointing the location of the call (i.e. Spark data frame action) in the client application code. By making this change, client application call stack details are sent to the server as a list of (function name, file name, line number) tuples where they can be logged in the server logs, included in corresponding open telemetry spans as attributes etc. This will help users looking from server side UI or Console to quickly pinpoint call locations of erring or slow calls in their own (client application) code without server needing to have access to the actual code.
Does this PR introduce any user-facing change?
It includes a new ENV variable
SPARK_CONNECT_DEBUG_CLIENT_CALL_STACKwhich user can set to true / 1 to opt into transmitting client application code locations to server. If opted into, the client app call stack trace details are included in theuser_context.extensionsfield of the Spark Connect protobufsHow was this patch tested?
By adding new unit test
test_client_call_stack_trace.pyWas this patch authored or co-authored using generative AI tooling?
Yes.
Some of the unit tests were Generated-by: Cursor