RFC: SGX EPC limits enforcement using NRI to configure misc controller

[mythi]

Since the SGX device plugin (or SGX "support" for Kubernetes in general) was added, we've wanted to be able to enforce per-container sgx.intel.com/epc: <limit> values as the hard limits for how much each container gets to use Encrypted Page Cache (EPC) memory but the kernel has not provided the mechanisms for it.

Finally, the Linux kernel has work ongoing to use misc cgroup controller to limit sgx_epc usage. However, this work alone is not sufficient but we also need a mechanism to get sgx.intel.com/epc: <limit> values configured to containers' misc.max.

The chain is roughly:

workload with "sgx.intel.com/epc: 1Mi" -> kubelet -> CRI-O/containerd -> OCI/runc  ("sgx_epc 1048 576" > misc.max) 

To get "sgx.intel.com/epc: 1Mi" passed over CRI to CRI-O/containerd, annotations can be used. Most (all?) deployments using the setup from this repository, end up using the SGX mutating webhook that is responsible for pod mutations during CREATE. The existing webhook can be modified to add the necessary "EPC limit" values as annotations.

OCI (Linux) Runtime specification has limited configuration options to set misc.max values:

1. NRI: have an NRI plug-in that registers to 'create container' events and updates config.json's "unified": { ... } with the values defined by the annotations. This depends on cgroup-v2 / unified hierarchy to be used by the nodes.
2. OCI hooks: have a custom EPC hook that writes to misc.max before the container is started. The setup works for both cgroup-v1 and cgroup-v2 but containerd does not have proper support to enable OCI hooks. CDI (prepare device plugin API changes for CDI injection #1457) can be used but it's in early stages still.

The proposal is to start with 1.

For telemetry/monitoring, misc.current can be used to read per-container EPC statistics. For telemetry, the proposal is to start with cAdvisor+Prometheus+Gafana dashboards.

Tasks:

• integrate with plugins: add sgx-epc plugin. containers/nri-plugins#156: SGX: set EPC limits via NRI annotations #1582
• update pkg/webhooks/sgx to add NRI annotations: SGX: set EPC limits via NRI annotations #1582
• add e2e/sgx test cases
• (add pod ValidatingAdmissionPolicy using CEL language)
• get libct/cg/stats: support misc for cgroup v2 opencontainers/runc#3972 merged
• get add misc controller metrics google/cadvisor#3420 merged
• prepare cAdvisor deployment that enables misc stats and prometheus
• build Grafana dashboard for demonstrating per-container EPC usage

Future work:

• Pass down resources to CRI kubernetes/enhancements#4113 to get rid of annotations
• evaluate: SGX NRI plugin to replace SGX device plugin?

Testing:

• Follow this gist to set up the code