fix: forkserver w/ preloaded modules/registry #1140
Conversation
nkemnitz
force-pushed
the
nkem/fix-multiproc
branch
2 times, most recently
from
ff96372 to
5cce2d3
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1140 +/- ##
========================================
Coverage 99.97% 99.97%
========================================
Files 190 195 +5
Lines 9675 9960 +285
========================================
+ Hits 9673 9958 +285
Misses 2 2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@nkemnitz my concern with this approach is that the the C extensions that you're trying to skip are the slow ones at importing, necessitating using |
|
What is the problem with using fork in |
At that point, one of the Google Python clients we use already called |
|
Do we know what is using gRPC? Maybe we can defer loading the module until it is truly necessary. Alternatively, all that is forked in Another alternative is rewriting the chunking process as a generator but that seems complex. |
|
Another thought is to write chunk generation in Cython. It then can use multiple cores within the same process without forking. |
nkemnitz
force-pushed
the
nkem/fix-multiproc
branch
from
5cce2d3 to
8e0a7d8
Compare
But I put gRPC only as example because it's the noisiest with the log messages. It's not guaranteed that gRPC is the culprit for the segmentation fault I encountered. Regarding performance: For the problematic inference spec that invokes these multiprocessing branches (bbox_strider and volumetric_apply_flow), I did measure no difference in speed (actually the forkserver forks were ~2ms faster compared to regular fork, for some reason). If you have another spec to test, I am happy to check. |
|
Ah, still need to check what necessitates the test change... |
|
Yea non inference code wont trigger the performance bug. The examlle specs
is in some old PRs or i can give tomorrow
…On Wed, Jan 21, 2026 at 7:35 PM Nico Kemnitz ***@***.***> wrote:
*nkemnitz* left a comment (ZettaAI/zetta_utils#1140)
<#1140 (comment)>
Ah, still need to check what necessitates the test change...
—
Reply to this email directly, view it on GitHub
<#1140 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAR5XOBESBTEQMZZTZYIERL4H5QBVAVCNFSM6AAAAACSGBWEF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTONZXGYYTCNZRGI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
nkemnitz
force-pushed
the
nkem/fix-multiproc
branch
from
8e0a7d8 to
3fbd2a6
Compare
|
@nkemnitz inference spec for testing performance: https://zettaai.slack.com/archives/C03TJ1FPJTD/p1769049979803059?thread_ts=1768337610.977159&cid=C03TJ1FPJTD |
|
Thanks, I took your script and increased This PR: 2026-01-22 10:59:13.473 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa/dryrun.py: 24
Starting dryrun....
2026-01-22 10:59:14.120 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 23409 tasks took 0.15s, total processing time 0.64s
2026-01-22 10:59:14.579 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.15s, total processing time 0.22s
2026-01-22 10:59:18.281 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.70s
2026-01-22 10:59:18.873 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.14s, total processing time 0.22s
2026-01-22 10:59:22.420 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.54s
2026-01-22 10:59:22.987 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.14s, total processing time 0.22s
2026-01-22 10:59:26.600 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.61s
2026-01-22 10:59:27.180 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.14s, total processing time 0.22s
2026-01-22 10:59:30.736 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.55s
2026-01-22 10:59:31.318 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.15s, total processing time 0.22s
2026-01-22 10:59:34.970 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.65s
2026-01-22 10:59:35.549 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.15s, total processing time 0.23s
2026-01-22 10:59:39.121 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.57s
2026-01-22 10:59:39.699 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.15s, total processing time 0.22s
2026-01-22 10:59:43.325 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.62s
2026-01-22 10:59:43.903 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.14s, total processing time 0.22s
2026-01-22 10:59:47.464 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.56s
2026-01-22 10:59:47.808 INFO [PID 13691] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 832 main: 2026-01-22 11:18:18.950 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa/dryrun.py: 24
Starting dryrun....
2026-01-22 11:18:20.077 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 23409 tasks took 0.33s, total processing time 1.12s
2026-01-22 11:18:20.700 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.28s, total processing time 0.34s
2026-01-22 11:18:25.404 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 4.70s
2026-01-22 11:18:26.183 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.30s, total processing time 0.37s
2026-01-22 11:18:30.311 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 4.12s
2026-01-22 11:18:30.930 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.20s, total processing time 0.25s
2026-01-22 11:18:34.529 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.59s
2026-01-22 11:18:35.152 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.22s, total processing time 0.26s
2026-01-22 11:18:38.708 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.55s
2026-01-22 11:18:39.316 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.20s, total processing time 0.25s
2026-01-22 11:18:42.954 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.63s
2026-01-22 11:18:43.574 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.21s, total processing time 0.27s
2026-01-22 11:18:47.100 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.52s
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1769077127.465597 18361 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
2026-01-22 11:18:47.713 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4698 tasks took 0.20s, total processing time 0.25s
2026-01-22 11:18:51.353 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.64s
2026-01-22 11:18:51.990 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/geometry/bbox_strider.py: 253
[get_all_chunk_bboxes]: mp setup for 4617 tasks took 0.22s, total processing time 0.27s
2026-01-22 11:18:55.517 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 636
[make_tasks_with_checkerboarding]: no mp branch for 9 tasks, total processing time 3.52s
2026-01-22 11:18:55.861 INFO [PID 18361] zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/mazepa_layer_processing/common/volumetric_apply_flow.py: 832 |
|
Are you actually running inference? If yes, can you try replicate the performance bug by removing the threshold for single threaded chunking. |
nkemnitz
force-pushed
the
nkem/fix-multiproc
branch
from
3fbd2a6 to
e7224f0
Compare
nkemnitz
force-pushed
the
nkem/fix-multiproc
branch
from
e7224f0 to
cb87b6a
Compare
Set up forkserver with all the fat modules imported and the slow registration done.
--> Stability from "spawn" (forkserver created before gRPC clients and other C extension stuff got initialized)
--> Speed from "fork"