Draft
Conversation
Move heartbeat mechanism from asyncio task to background thread for both async and sync activities. This prevents event loop starvation when async activities call blocking/sync code (e.g. DaprClient, subprocess, requests). Also fix silent heartbeat death: both send_periodic_heartbeat (async) and send_periodic_heartbeat_sync now catch exceptions and retry on the next interval instead of dying permanently on the first transient error. Signed-off-by: Anuj Agrawal <anuj.agrawal@atlan.com>
On consecutive heartbeat failures, apply exponential backoff (2^n, capped at 5x base delay) to avoid hammering an unhealthy Temporal server. Reset to normal interval after a successful heartbeat. Logs recovery with failure count for observability. Signed-off-by: Anuj Agrawal <anuj.agrawal@atlan.com>
…nd activity If the heartbeat thread crashes (BaseException escaping retry logic), the activity now detects it and fails fast instead of running to completion without heartbeats. For async activities, a monitor task races against the activity via asyncio.wait. For sync activities, thread_failed is checked after the function returns. Adds tests proving: - heartbeats continue when the event loop is blocked by sync code - async activity is cancelled when the heartbeat thread crashes - sync activity raises when the heartbeat thread crashed during execution
…t timeout - Remove send_periodic_heartbeat (async version) which is no longer called after the switch to thread-based heartbeating, along with its tests. - Cap exponential backoff in send_periodic_heartbeat_sync at the heartbeat timeout (delay * 3) instead of 5x the interval. Previously backoff could reach 200s with a 120s timeout, meaning Temporal would kill the activity before the next retry attempt.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Contributor
📜 Docstring Coverage ReportRESULT: PASSED (minimum: 30.0%, actual: 80.1%) Detailed Coverage Report |
Contributor
📦 Trivy Vulnerability Scan Results
Report SummaryCould not generate summary table (data length mismatch: 9 vs 8). Scan Result Detailsrequirements.txtVulnerabilities
uv.lockVulnerabilities
|
Contributor
📦 Trivy Secret Scan Results
Report SummaryCould not generate summary table (data length mismatch: 9 vs 8). Scan Result Detailsrequirements.txtuv.lock |
Collaborator
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
Contributor
|
🛠 Full Test Coverage Report: https://k.atlan.dev/coverage/application-sdk/pr/1443 |
4 tasks
Member
|
v3 relevance check: ✅ Still relevant — resilient heartbeater addresses a real v3 issue (heartbeat failures during Dapr health checks). Please rebase on latest main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog
Additional context (e.g. screenshots, logs, links)
Checklist
Copyleft License Compliance