Regarding the setting of the job's unfit status.

[Moxixis]

I noticed that in Paella, for non-memory jobs where GPU resources are insufficient, the job is marked as resource-unfit when num_outstanding_kernels_ < max_num_outstanding_kernels_.

However, both the block start and block finish methods include logic to clear the job's unfit flag. In the block start method, once all blocks are launched, it directly checks the job's unfit flag — if it's marked as resource-unfit, it clears the flag and decrements num_outstanding_kernels_. In the block finish method, once all blocks are completed, it checks the job's unfit flag again, clears it, and also decrements num_outstanding_kernels_.

I'm trying to understand the rationale behind this behavior — in particular, why the unfit flag is set in the first place, since in the schedule_job method, even after marking a job as unfit, one job still gets to run.

Also, the operations to clear the unfit flag and decrement num_outstanding_kernels_ are repeated in both the block start and block finish methods. If this has already been done in the block start method, wouldn’t the block finish method skip it?

Another question is about the setting of max_num_outstanding_kernels_ — what’s the reference or guideline used to determine its value?

[Kelvin-Ng]

The unset_unfit() in handle_block_end() was there only to workaround a bug in CUDA atomic. I forgot the details, but basically sometimes the messages for handle_block_start() will be missing due to a CUDA atomic bug and so I do the unset_unfit() at handle_block_end() as well (any block that reaches handle_block_end() should have never been unfit or should have unset unfit). I think the bug was actually fixed long time ago (like in CUDA 12) but I just did not remove this. Without that bug, the condition of if (job->is_unfit()) in handle_block_end() should never evaluate to true.

The unfit flag is set for kernels that does not fit, but we still send max_num_outstanding_kernels_ number of unfit kernels to the GPU. This is to hide kernel launch overheads. This number should be set to be the smallest number that is large enough to hide (most of the) kernel launch overheads. If the number is too large, it undermines the benefits of scheduling; if too small, kernel launch overheads will be too significant.

As we still send max_num_outstanding_kernels_ number of unfit kernels to the GPU, some kernels will be in the GPU but with unfit flag set to true. As a block from an originally unfit kernel starts, we know that this kernel becomes fit and so we unset its unfit flag.