Fix encounter_list memory by using DB-level pagination#228
Open
alexiadowns-canvas wants to merge 1 commit intomainfrom
Open
Fix encounter_list memory by using DB-level pagination#228alexiadowns-canvas wants to merge 1 commit intomainfrom
alexiadowns-canvas wants to merge 1 commit intomainfrom
Conversation
…of loading all notes The encounter_list plugin was loading entire querysets into Python memory before paginating, causing 100-500MB+ memory spikes per request. This caused production outages on Doctronic (Pylon #22626) and affects all 65 instances running this plugin. Changes: - _sort_and_paginate_database: use Django queryset slicing (OFFSET/LIMIT) instead of list(queryset) followed by Python slicing - _sort_and_paginate_delegated_orders: replace full-queryset materialization + per-note Python loop with DB-level Subquery annotations on ImagingOrder and Referral, enabling DB-side sort/filter/pagination - Remove unused _apply_pagination helper The delegated orders DB annotation is approximate (does not check the JSON task_ids field for open tasks), but the exact count is still computed for the ~25 notes on each rendered page via _calculate_delegated_orders_count. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
alexiadowns-canvas
changed the title
djantzen
reviewed
Apr 6, 2026
| reverse=(sort_direction == "desc") | ||
| """Sort and paginate for delegated orders using database-level annotation. | ||
|
|
||
| Uses ImagingOrder and Referral FKs to Note to compute an approximate |
Contributor
There was a problem hiding this comment.
I gather that this change will result in a higher count on some views than on others, because it will be the superset of all tasks, not just those that are open. Is that change in behavior going to be acceptable to users?
djantzen
reviewed
Apr 6, 2026
| .annotate(cnt=Count("id")) | ||
| .values("cnt") | ||
| ) | ||
| note_queryset = note_queryset.annotate( |
Contributor
There was a problem hiding this comment.
We are missing indexes on forwarded and delegated, which might force a sequential scan. We need to look at an explain plan for this new query with the annotation and subqueries.
Contributor
There was a problem hiding this comment.
@claude What SQL will be generated by lines 259-264?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_sort_and_paginate_databaseand_sort_and_paginate_delegated_orders) were callinglist(note_queryset), loading the entire result set into memory before slicing for pagination. For large instances like Doctronic, this caused 100–500 MB memory spikes per request, leading to OOM-related 502 outages.list(queryset)+ Python slicing with Django queryset slicing (queryset[offset:offset+page_size]), which translates to SQLOFFSET/LIMIT.Subqueryannotations onImagingOrder.delegatedandReferral.forwarded, enabling database-side sort, filter, and pagination.task_idsfield) is still computed per-note in the render loop — but only for the ~25 notes on the displayed page, not all notes in the system._apply_paginationhelper.Test plan