Skip to content

Conversation

@li041
Copy link
Contributor

@li041 li041 commented Dec 11, 2025

This PR improves the SMP and IPI infrastructure and introduces per-task CPU residency
tracking to reduce unnecessary cross-core TLB shootdowns.

Background

Cross-core TLB flushes are expensive and should be avoided when possible.
Previously, the kernel lacked sufficient information to determine which CPUs
a task had actually run on, resulting in overly conservative TLB shootdowns.
In addition, the underlying IPI mechanism had several limitations that
prevented reliable cross-core synchronization.

What these PRs do

  • Fix and enhance the low-level IPI mechanism
    • Fix bugs in send_ipi where notifications could only target a single CPU
    • Improve IPI synchronization semantics to support reliable cross-core coordination
  • Enable cross-core TLB flush
    • Provide the required IPI-based infrastructure for remote TLB invalidation
    • Support lazy TLB flush when possible
  • Track per-task CPU residency
    • Record the set of CPUs a task has run on
    • Use this information to limit TLB shootdowns to relevant CPUs only

Result

  • Avoids unnecessary cross-core TLB shootdowns
  • Reduces IPI traffic and synchronization overhead
  • Improves scalability on SMP systems

Related work

Copilot AI review requested due to automatic review settings December 11, 2025 08:10
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds per-task CPU residency tracking to optimize TLB (Translation Lookaside Buffer) shootdown operations. By tracking which CPUs a process has run on, the system can avoid sending unnecessary cross-core TLB invalidation IPIs to CPUs that haven't executed the process.

Key changes:

  • Added AxCpuMask tracking to ProcessData to record CPU residency
  • Implemented on_enter/on_leave hooks to update CPU mask when tasks are scheduled on/off CPUs
  • Added on_cpu_mask() method to query which CPUs a task has run on

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

core/src/task.rs Outdated
Comment on lines 157 to 158
let mut on_cpu_mask = self.proc_data.on_cpu_mask.write();
on_cpu_mask.set(axhal::percpu::this_cpu_id(), false);
Copy link

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Race condition: Clearing the CPU bit in on_leave creates a race when multiple threads from the same process run concurrently. Since ProcessData is shared via Arc among all threads in a process, if Thread A from the process leaves CPU X while Thread B from the same process is still running (or about to run) on CPU X, the bit for CPU X will be incorrectly cleared. The comment on line 195 says "The CPUs on which the task has run" which suggests historical tracking, but the current implementation attempts to track "currently running" which requires more sophisticated synchronization or reference counting per CPU.

Copilot uses AI. Check for mistakes.
Comment on lines +195 to +194
/// The CPUs on which the task has run.
pub on_cpu_mask: RwLock<AxCpuMask>,
Copy link

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states "The CPUs on which the task has run" which implies historical tracking of all CPUs where the process has executed. However, the implementation in on_leave (line 157-158) clears CPU bits when leaving, suggesting it tracks "currently running CPUs" instead. This discrepancy between documentation and implementation needs to be resolved. Clarify whether this should track historical CPU usage (never clear bits) or current CPU presence (requires reference counting per CPU to handle multiple threads).

Copilot uses AI. Check for mistakes.
@li041 li041 marked this pull request as ready for review December 23, 2025 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant