【PMEM 300】Memory Mode Unable (while AppDirect and everything else works fine) ！！！

[GohardSGG]

Hello to the community managers and all the helpful members here!

About a month ago, I built a system based on the 4th Gen Xeon Scalable platform, paired with Intel Optane Persistent Memory. The system is currently running perfectly in AppDirect mode, but I am unable to switch it to Memory Mode. I have tried numerous approaches without finding a clear solution, which is why I'm reaching out for your expertise.

System Specifications:

• CPU: Intel Xeon 8480+ ES (SKU: Q03K, Stepping 4)
• Memory: 4x 48GB DDR5 RDIMM 4800MT/s 2Rx4
• Optane PMem: 4x 512GB Intel PMem 300 Series (Part Number: NMC2XAD512GQS)
• Motherboard: Gigabyte MS03-CE0 (BIOS Ver: R12)
• OS: Windows 10 Pro for Workstations 22H2

Problem Details & Diagnostics

I can see the Optane configuration tab in the BIOS setup utility.

From there, I can successfully create goals and namespaces for AppDirect mode. However, within the goal creation menu, there is a grayed-out message stating: "Platform does not support Memory Mode." I am unsure of the reason for this limitation.

I then attempted to manage the modules using ipmctl. Here are the details of my Optane configuration:

Memory Resources:

PS C:\Windows\system2> ipmctl show -memoryresources
 MemoryType    | DDR         | PMemModule   | Total
==========================================================
 Volatile      | 192.000 GiB | 0.000 GiB    | 192.000 GiB
 AppDirect     | -           | 2036.000 GiB | 2036.000 GiB
 Cache         | 0.000 GiB   | -            | 0.000 GiB
 Inaccessible  | 0.000 GiB   | 2.969 GiB    | 2.969 GiB
 Physical      | 192.000 GiB | 2038.969 GiB | 2230.969 GiB

Platform Capabilities:

PS C:\Windows\system32> ipmctl show -system -capabilities
PlatformConfigSupported=1
Alignment=1.000 GiB
AllowedVolatileMode=1LM
CurrentVolatileMode=1LM
AllowedAppDirectMode=App Direct

I am particularly confused by AllowedVolatileMode=1LM. My understanding is that this should indicate support for Memory Mode, but perhaps I am misinterpreting it.

Memory Topology:

PS C:\Windows\system32> ipmctl show -topology
 DimmID | MemoryType                  | Capacity    | PhysicalID| DeviceLocator
================================================================================
 0x0000 | Logical Non-Volatile Device | 509.688 GiB | 0x0030    | DIMM_P0_A0
 0x0100 | Logical Non-Volatile Device | 509.688 GiB | 0x0032    | DIMM_P0_C0
 0x0200 | Logical Non-Volatile Device | 509.688 GiB | 0x0034    | DIMM_P0_E0
 0x0300 | Logical Non-Volatile Device | 509.688 GiB | 0x0036    | DIMM_P0_G0
 N/A    | DDR5                        | 48.000 GiB  | 0x0031    | DIMM_P0_B0
 N/A    | DDR5                        | 48.000 GiB  | 0x0033    | DIMM_P0_D0
 N/A    | DDR5                        | 48.000 GiB  | 0x0035    | DIMM_P0_F0
 N/A    | DDR5                        | 48.000 GiB  | 0x0037    | DIMM_P0_H0

DIMM Status:

PS C:\Windows\system32> ipmctl show -dimm
 DimmID | Capacity    | LockState | HealthState | FWVersion
===============================================================
 0x0000 | 509.742 GiB | Disabled  | Healthy     | 03.01.00.2769
 0x0100 | 509.742 GiB | Disabled  | Healthy     | 03.01.00.2716
 0x0200 | 509.742 GiB | Disabled  | Healthy     | 03.01.00.2769
 0x0300 | 509.742 GiB | Disabled  | Healthy     | 03.01.00.2769

When I attempt to create a goal for Memory Mode using ipmctl, I receive the following error, which is consistent with the BIOS message:

PS C:\Windows\system32> ipmctl create -goal memorymode=100
Create region configuration goal failed: Error 113 - Platform does not support Memory Mode

I have also reviewed my motherboard's user manual.

I've noticed other Optane users seem to have a "2LM" option. However, my manual states that this particular setting is only configurable when a Xeon Max series CPU (with HBM) is installed, which is why I cannot access it.

My Troubleshooting and Questions

I have a few theories about the potential cause of this issue and would appreciate your input:

1. DDR5 to Optane PMem Capacity Ratio:
   According to official documentation, a DRAM-to-PMem ratio between 1:4 and 1:16 should be viable. However, I'm unsure if Memory Mode has stricter requirements than AppDirect mode. I have seen some unofficial sources suggest a minimum ratio of 1:8 is needed for Memory Mode. My current ratio is 48GB:512GB (approximately 1:10.67). Could this be the reason for the "Platform does not support Memory Mode" error?

2. 1LM vs. 2LM Configuration:
   I am not sure if this inaccessible BIOS setting could be related to the problem. Resetting the motherboard to factory defaults did not change its availability, and the board was purchased new.

3. Other Potential Issues?
   I am open to any other perspectives or potential causes that I may have overlooked.

Below is the detailed PCAT information from my system, which may be helpful for diagnosis.

PS C:\Windows\system32> ipmctl show -system pcat
---Platform Configurations Attributes Table---
   Signature: PCAT
   Length: 236 bytes
   Revision: 0x32
   Checksum: 0x9
   OEMID: GBT
   OEMTableID: GBTUACPI
   OEMRevision: 0x2
   CreatorID: INTL
   CreatorRevision: 0x20091013
   ---TableType=0x0
      Length: 16 bytes
      TypeEquals: PlatformCapabilityInfoTable
      PMemModuleMgmtSWConfigInputSupport: 0x1 (Yes)
      MemoryModeCapabilities: 0x5 (1LM, AppDirect)
      CurrentMemoryMode: 0x4
         -Current Volatile Memory Mode: 1LM
         -Allowed Persistent Memory Mode: AppDirect
         -Allowed Volatile Memory Mode: 1LM
      MaxPMInterleaveSets: 0x28
         -Per CPU Die: 0x8
      CacheCapabilities: 0x1
         -MemoryMode: In-tile
      DDRCacheSize: 0 GiB per DDR DIMM

   ---TableType=0x1
      Length: 148 bytes
      TypeEquals: MemoryInterleaveCapabilityTable
      MemoryMode: 0x3 (AppDirect)
      NumberOfInterleaveFormatsSupported: 0x21
      InterleaveFormatSupported(0): 0x100ff
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 8-way
      InterleaveFormatSupported(1): 0x10055
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(2): 0x100aa
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(3): 0x33
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(4): 0xcc
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(5): 0xc3
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(6): 0x3c
         -Not recommended

         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(7): 0xf
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(8): 0xf0
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 4-way
      InterleaveFormatSupported(9): 0x10011
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(10): 0x10022
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(11): 0x10044
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(12): 0x10088
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(13): 0x10041
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(14): 0x10082
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(15): 0x10014
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(16): 0x10028
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(17): 0x5
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(18): 0xa
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(19): 0x50
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(20): 0xa0
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(21): 0x3
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(22): 0xc
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(23): 0x30
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(24): 0xc0
         -Not recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 2-way
      InterleaveFormatSupported(25): 0x10001
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(26): 0x10002
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(27): 0x10004
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(28): 0x10008
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(29): 0x10010
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(30): 0x10020
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(31): 0x10040
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveFormatSupported(32): 0x10080
         -Recommended
         -Channel interleave size: 256B
         -iMC interleave size: 256B
         -Channel ways: 1-way
      InterleaveAlignmentSize: 0x1e
      MaxPMInterleaveSetsPerMemType: 0x28
         -Per CPU Die: 0x8
         -Per PMem module: 0x2

   ---TableType=0x6
      Length: 32 bytes
      TypeEquals: SocketSkuInfoTable
      SocketID: 0x0
      DieID: 0x0
      MappedMemorySizeLimit: 18446744073709551615
      TotalMemorySizeMappedToSpa: 2392296783872
      CachingMemorySize: 0

I have also run the ipmctl diagnostic tool, which passed most tests but showed a warning for a non-optimal firmware version on one module. I have also tried configuring the system with only one stick of DDR5 and one Optane PMem module, with the same result.

PS C:\Windows\system32> ipmctl start -diagnostic
--Test = Quick
   State = Ok
   Message = The quick health check succeeded.
      State = Ok
   --SubTest = Boot status
      State = Ok
   --SubTest = Health
      State = Ok

--Test = Config
   State = Ok
   Message = The platform configuration check succeeded.
   --SubTest = PMem module specs
      State = Ok
   --SubTest = Duplicate PMem module
      State = Ok
   --SubTest = System Capability
      State = Ok
   --SubTest = Namespace LSA
      State = Ok
   --SubTest = PCD
      State = Ok

--Test = Security
   State = Ok
   Message = The security check succeeded.
   --SubTest = Encryption status
      State = Ok
   --SubTest = Inconsistency
      State = Ok

--Test = FW
   State = Warning
   --SubTest = FW Consistency
      State = Warning
      Message.1 = The firmware consistency and settings check detected that firmware version on PMem module 0x0100 with subsystem device ID 0x097c is nonoptimal, preferred version is 03.01.00.2769.
   --SubTest = Viral Policy
      State = Ok
   --SubTest = Threshold check
      State = Ok
   --SubTest = System Time
      State = Ok

I would be sincerely grateful for any guidance from the experts here. Thank you in advance for your time and assistance.

Best regards.

[j-warnke]

Hello,

2LM mode was not part of Intel's POR for PMEM 300 series. Intel Eagle Stream reference BIOS 2023 and later sets 1LM/App Direct as the default, and in later versions removes the 2LM (memory mode) option entirely for Optane.

This 'Memory Map' knob you show is BIOS vendor specific. I recommend contacting AMI for more details. Without knowing AMI BIOS code, my initial guess is, this knob controls HBM 2LM (HBM 2LM is separate from Optane 2LM).

[GohardSGG]

Hello,

2LM mode was not part of Intel's POR for PMEM 300 series. Intel Eagle Stream reference BIOS 2023 and later sets 1LM/App Direct as the default, and in later versions removes the 2LM (memory mode) option entirely for Optane.

This 'Memory Map' knob you show is BIOS vendor specific. I recommend contacting AMI for more details. Without knowing AMI BIOS code, my initial guess is, this knob controls HBM 2LM (HBM 2LM is separate from Optane 2LM).

Hi, thank you so much for your insightful reply!

You are right, I also discovered that the "Memory Map" feature is indeed for HBM configurations and seems unrelated to the Optane Memory Mode I'm trying to enable.

However, I'd like to clarify the main point to make sure I understand correctly: Does this mean that the entire PMem 300 series no longer supports being used in Memory Mode on this platform? That would be quite disappointing, as it would lose a significant amount of its practical value for many use cases.

Or, could this still be an issue specific to my hardware combination, such as this particular motherboard model or my DRAM-to-PMem ratio? I'm wondering if a different brand of motherboard or a different capacity ratio might yield a different result.

Thanks again for your help!

[j-warnke]

Hi,

This is not specifically an issue with your hardware configuration.

Crow Pass (PMem 300 series) was originally planned to support Memory Mode on release, but its scope was reduced in late 2022, with the 300 series being cancelled entirely in early 2023 (we have a public notice @ URL below).

https://www.intel.com/content/www/us/en/support/articles/000093792/memory-and-storage/intel-optane-persistent-memory.html

The last PMem product to support Memory Mode is the 200 series.

Probably not the info you were hoping for, but I hope it helps.