Fix corruption in virtual methods cache table

[josesimoes]

Description

• Add bounds validation to entries array.
• Cache entries are now validated preventing uninitialized memory corruption.
• Add verbose diagnostics output to assist debug.

Motivation and Context

• When using nanoCLR in release build some core lib unit tests were causing the virtual device to crash with memory access exceptions. This was on RELEASE build only. DEGUB build was working fine.

How Has This Been Tested?

• Extensive running offending unit tests with both release and debug builds of nanoCLR.
• [tested against Merge generics support in mscorlib CoreLibrary#245].

Screenshots

Types of changes

• Improvement (non-breaking change that improves a feature, code or algorithm)
• Bug fix (non-breaking change which fixes an issue with code or algorithm)
• New feature (non-breaking change which adds functionality to code)
• Breaking change (fix or feature that would cause existing functionality to change)
• Config and build (change in the configuration and build system, has no impact on code or features)
• Dev Containers (changes related with Dev Containers, has no impact on code or features)
• Dependencies/declarations (update dependencies or assembly declarations and changes associated, has no impact on code or features)
• Documentation (changes or updates in the documentation, has no impact on code or features)

Checklist

• My code follows the code style of this project (only if there are changes in source code).
• My changes require an update to the documentation (there are changes that require the docs website to be updated).
• I have updated the documentation accordingly (the changes require an update on the docs in this repo).
• I have read the CONTRIBUTING document.
• I have tested everything locally and all new and existing tests passed (only if there are changes in source code).

Summary by CodeRabbit

• Bug Fixes
  • Improved cache operation reliability through enhanced error detection and recovery mechanisms, ensuring more robust handling of edge cases and invalid states.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Defensive programming enhancements added to Cache.cpp, including DEBUG diagnostics to detect memory overlaps in VirtualMethodTable structure initialization and runtime safety checks in the FindVirtualMethod hash lookup loop to validate indices, repair corrupted hash chains, and handle invalid payload entries.

Changes

Cohort / File(s)

Change Summary

Cache diagnostics and safety hardening src/CLR/Core/Cache.cpp

Added DEBUG-guarded diagnostics in VirtualMethodTable::Initialize to report structure layout, alignments, array sizes, and detect memory region overlaps. Introduced runtime safety checks in FindVirtualMethod hash lookup loop: index validation against LinkArraySize(), PayloadArraySize assertions, invalid payload handling, FindVirtualMethodDef failure safeguards, and new entry initialization with diagnostic logging. Mirrored changes for AVLTREE-enabled code path.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

• Review focus areas:
  • Correctness of memory overlap detection logic and address range comparisons
  • Hash chain corruption detection and repair mechanism to ensure it resolves corruption without side effects
  • Proper conditional compilation guards (DEBUG && _WIN64) and no unintended performance regressions in release builds
  • Validation that safety checks do not mask legitimate errors or cause false positives

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly summarizes the main change: fixing corruption in the virtual methods cache table through bounds validation and diagnostic improvements.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[josesimoes]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

src/CLR/Core/Cache.cpp (1)

199-250: Good defensive diagnostics; consider format specifier safety on 64-bit.

The diagnostic output is excellent for debugging memory layout issues. However, casting uintptr_t to unsigned int and using %08X will truncate addresses on 64-bit platforms.

Consider using %p or platform-appropriate format specifiers:

     CLR_Debug::Printf(
-        "m_entries: 0x%08X - 0x%08X (%u bytes)\r\n",
-        (unsigned int)entries_start,
-        (unsigned int)entries_end,
+        "m_entries: %p - %p (%u bytes)\r\n",
+        (void*)entries_start,
+        (void*)entries_end,
         (unsigned int)(entries_end - entries_start));

Apply similar changes to lines 223-232.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 542bc41 and 1f30b77.

📒 Files selected for processing (1)

• src/CLR/Core/Cache.cpp (4 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

src/CLR/Core/Cache.cpp (3)

src/CLR/Diagnostics/Info.cpp (2)

• Printf (256-269)
• Printf (256-256)

src/CLR/Core/TypeSystem.cpp (2)

• Link (6563-6580)
• Link (6563-6563)

src/CLR/Core/CLR_RT_RuntimeMemory.cpp (6)

• LinkArraySize (111-114)
• LinkArraySize (111-111)
• LinkMRUArraySize (115-118)
• LinkMRUArraySize (115-115)
• PayloadArraySize (119-122)
• PayloadArraySize (119-119)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)

• GitHub Check: nf-interpreter (Nightly build) (Build_STM32_targets MXCHIP_AZ3166)
• GitHub Check: nf-interpreter (Nightly build) (Build_STM32_targets ORGPAL_PALX)
• GitHub Check: nf-interpreter (Nightly build) (Build_STM32_targets ORGPAL_PALTHREE)
• GitHub Check: nf-interpreter (Check_Code_Style)
• GitHub Check: nf-interpreter (Nightly build) (Check_Build_Options)
• GitHub Check: nf-interpreter (Check_Build_Options)

🔇 Additional comments (4)

src/CLR/Core/Cache.cpp (4)

362-371: Excellent bounds validation to prevent memory corruption.

This runtime safety check validates the index before dereferencing arrays, addressing the memory access exceptions mentioned in the PR objectives. The repair logic correctly resets the corrupted hash chain and treats the lookup as a cache miss.

---

375-375: Good defensive assertion for payload index bounds.

This assertion ensures that indices in the hash chain fall within the payload array bounds, catching logic errors in debug builds.

---

379-382: Good addition of braces for clarity.

Adding braces around single-statement blocks improves consistency and prevents future bugs.

Also applies to: 384-387, 395-398

---

406-408: Critical fix: Initialize entry links before use.

This initialization is essential to prevent memory corruption. Without it, MoveEntryToTop at line 419 would use uninitialized m_next and m_prev values, potentially causing the crashes described in the PR objectives.

[josesimoes]

Unit tests failing because of features missing in EE and type engine. Addressed in #3242 . Merging anyway.