How to Enable Page Access Permission Menagement (S2AP) in RMM?

[bows7ring]

Hi there.

I'm working on some new Qemu features, and I guess I need RMM's support for Page Access Permission Menagement, by which I mean DBM & S2AP.

But I found that this is different from implementing page permission management (such as dirty log) in KVM.

Firstly, according to RMM's spec, stage 2 dirty logging is disabled.

Secondly, in the func s2tte_get_ripas in lib/realm/src/s2tt.c, there is a comment:

If valid s2tte descriptor is passed, then ensure S2AP[0] bit is 1 (S2AP is set to RW for lower EL), which corresponds to RIPAS_RAM (bits[6:5] = b01) on a valid descriptor.

But I don't understand why only S2AP[0] is set. Shouldn't RW be '11'? And shouldn't S2AP be pte<6:7>, which conflicts with RIPAS_RAM (bits<6:5>)?

My questions can be summarized as follows:

1. How does RMM disable DBM and S2AP? Is it implemented by masking off the PTE provided by the host?
2. Why are the definitions of S2AP and RIPAS contradictory? Does it mean that dirty logging can only be implemented in software?

[soby-mathew]

Hi @bows7ring ,

Yes, The RMM spec mandates that DBM and Hardware access is disabled at Stage 2.

some new Qemu features, and I guess I need RMM's support for Page Access Permission Menagement, by which I mean DBM & S2AP

Could you describe your new feature/usecase ? We might need changes to the RMM specification to cater for your usecase. We are aware that such a capability may be required for Live Migration but this is currently not in scope.

But I don't understand why only S2AP[0] is set. Shouldn't RW be '11'? And shouldn't S2AP be pte<6:7>, which conflicts with RIPAS_RAM (bits<6:5>)?

Yes indeed there is a mismatch. S2AP should be 0b11 and it seems the current TF-RMM code does not call s2tte_get_ripas() for valid pte. This is an error and needs to be corrected AFAICS.

We re-use some bits in pte (namely bits 5 and 6) for storing the RIPAS state when the pte is invalid. When the pte is valid, TF-RMM assumes that RIPAS is always RIPAS_RAM and hence we do not refer to these bits for a valid pte.

How does RMM disable DBM and S2AP? Is it implemented by masking off the PTE provided by the host?

RMM doesn't allow Host to specify the S2AP and DBM fields of pte in protected address space (eg: RMI_DATA_CREATE). For unprotected address space, it only allows host to specify S2AP but not allow access to DBM field (eg: RMI_RTT_MAP_UNPROTECTED).

Why are the definitions of S2AP and RIPAS contradictory? Does it mean that dirty logging can only be implemented in software?

For a valid pte, RIPAS can only be RIPAS_RAM and these bits (5 and 6) are not used for representing RIPAS state.
Hardware Dirty bit management can be possibly be enabled for RMM but as I mentioned , this will need update to RMM specification and some discussion on the usecase.

[bows7ring]

Hi @soby-mathew ,

Thank you for your explanation!
We are trying to implement Live Migration on Qemu to support the subsequent development of CCA on Huawei Cloud.
Since Realm VM should not expose memory dirty page tracking to the host, we plan to implement dirty page tracking ourselves in RMM, and we agree that RMM should mask-off host's configuration for DBM and S2AP in PTE update.

Our current design is similar to the solution opensourced by Intel TDX. TDX implements batched page import/export and dirty page tracking within its TDX Module, independent of KVM.

For a valid pte, RIPAS can only be RIPAS_RAM and these bits (5 and 6) are not used for representing RIPAS state.
Hardware Dirty bit management can be possibly be enabled for RMM but as I mentioned , this will need update to RMM specification and some discussion on the usecase.

Got it.
We will try to release the relevant bits.
As for enabling hardware support for dirty page tracking in RMM, we may need your further guidance in the future.

[soby-mathew]

Interesting, Good to know about this.

[Jingfanccc]

Can you tell us more details about how to enable hardware dirty bit management in RMM.
After reading a portion of RMM and Linux code, I only found that it is possible to mask dirty bits by writing val=~(HD | HA) in the function(handle-amperel_tcr)of switch.h.

And may add other translation regime state like TCR in the struct (realm s2 context) of realm. h

How can make PTE vaild to support our design work

[soby-mathew]

Hi @Jingfanccc ,

I did not understand the requirement your are trying to design for. Could you elaborate further ?

The function handle_ampere1_tcr() seems to be trapping tcr_el1 access from EL1 and preventing EL1 from changing the HD and HA bits.

RMM allows Realms to enable HD and HA and Realm EL1 software is allowed to make use of Hardware dirty bit mgmt and Hardware access features.

RMM does not enable DBM at Realm EL2 due to reasons mentioned above and only usecase we are aware is for Migration of Realm VMs. The Stage 2 PTE are managed by S2TT library and you would need to modify the code according to your requirement.

[Jingfanccc]

Yes，We want to implement dirty page tracking in RMM to support the live migration of Realm VM.

RMM does not enable DBM at Realm EL2 due to reasons mentioned above and only usecase we are aware is for Migration of Realm VMs. The Stage 2 PTE are managed by S2TT library and you would need to modify the code according to your requirement.

[soby-mathew]

This is an area which we have not investigated from our side yet. Essentially enabling the VTCR_EL2.HD bit and the corresponding DBM bit in Stage 2 PTE, the hardware will transition the PTE from writeable-clean to writeable-dirty state when an attempt to write is made. The S2AP[1] bit is the permission bit which will be set by hardware.

How this hardware mechanism is actually utilized in RMM will depend on the Live migration mechanics and this will eventually be specified by RMM specification (but not currently planned yet from our side).

[soby-mathew]

The bit managed by hardware as part of DBM is S2AP[1] (bit 7).

When the PTE is valid for protected IPA space, i.e when RIPAS=RAM and HIPAS=Assigned, these bits will be as used as per Arm ARM and RMM will not make use them for software state tracking. Since PTE has to be valid for Realms to access the memory, and DBM also needs PTE to be valid, there should not be any problem for use of S2AP[1] for hardware DBM mechanism (AFAICS).

For details on RIPAS and HIPAS and how they are changed, refer to the RMM specification.

[Jingfanccc]

it works！Thank you for your guidance.

[soby-mathew]

Good to hear. We are happy to receive any contribution or inputs wrt Live Migration.

[bows7ring]

We have successfully enabled and tested the basic functionality of dirty page tracking on the QemuCCA platform.

RMM dirty page tracking is feasible. But recently, we have not continued to push it forward due to another issue #27 .

Regarding Live Migration, we are likely to need assistance with a series of issues in the future. So, let us close this issue and move on to new ones.