Quality of Service (QoS) is defined as the minimal end-to-end performance that is guaranteed in advance by a service level agreement (SLA) to a workload. A workload may be a single application, a group of applications, a virtual machine, a group of virtual machines, or a combination of those.
The RISC-V Capacity and Bandwidth QoS Register Interface (CBQRI) cite:[CBQRI] specification specifies mechanisms to a) associate QoS identifiers with requests made by workloads to resources controlled by capacity and bandwidth controllers, b) memory-mapped register interface for capacity allocation and capacity usage monitoring in capacity controllers in shared resources such as shared caches, c) memory-mapped register interface for bandwidth allocation and bandwidth usage monitoring in bandwidth controllers in shared resources such as memory controllers.
The Ssqosid extension cite:[SSQOSID] provides a read/write S/HS-mode register
(srmcfg) to configure QoS Identifiers to be used with requests made by the
hart to shared resources. The CBQRI specification specifies the IOMMU extension
to associate QoS Identifiers with requests made by devices to shared resources.
With supervisor domains, the resource accesses from an SD or the RDSM must not be observable by entities that are not within their TCB using the resource usage monitors. Similarly, the resource allocations for an SD or the RDSM must not be influenced by entities outside their TCB.
To support this security objective, the following capabilities are required:
-
Multiple QoS register interfaces (QRIs) in the capacity and bandwidth controllers within the SoC.
-
The RDSM may allocate a QRI exclusively to an SD or may mediate access to a QRI among multiple SDs. A non-ISA supervisor domain QRI extension is specified in Supervisor Domain QoS Register interfaces (QRI) to provide this capability.
-
-
Associating a QRI ID (QRID) with requests originating from SDs and RDSM.
-
The QRID, along with the
RCIDandAT, is used to identify the resource allocation configurations in a capacity or bandwidth controller. The QRID, along with theMCID, is used to identify the ID of the counter used to monitor resource usage in a capacity or bandwidth controller. -
The Smsdqosid Smsdqosid - Supervisor Domain Quality-of-Service Identifiers extension is specified to enable partitioning the RCID and MCID associated with a QRI among the SDs sharing a QRI.
-
The Smqosid extension Smqosid - Machine-Mode Quality-of-Service Identifiers is specified to configure QoS identifiers for requests made by a hart with effective privilege mode equal to M.
-
The Smqosid introduces a read/write M-mode register (mrmcfg) to configure
QoS identifiers for requests made by a hart with effective privilege mode equal
to M.
The mrmcfg register is an MXLEN-wide read/write machine-mode register used to
configure a Resource Control ID (RCID), Monitoring Counter ID (MCID), and a
QoS Register Interface ID (QRID). The QRID, RCID, and MCID are WARL
fields. The register is formatted as shown in Machine Resource Management Configuration (mrmcfg) register for MXLEN=64 when MXLEN=64 and
Machine Resource Management Configuration (mrmcfg) register for MXLEN=32 when MXLEN=32. The CSR number for mrmcfg is 0x381.
The QRID, RCID, and MCID values accompany each request made with effective
privilege mode equal to M by the hart to shared resource controllers. The QRID
and RCID values are used to determine the resource allocations (e.g., cache
occupancy limits, memory bandwidth limits, etc.) to enforce. The QRID and
MCID values are used to identify a counter to monitor resource usage.
|
Note
|
A suggested reset value of 0 for the The |
When Smqosid is implemented, the configurations in the srmcfg register
introduced by Ssqosid extension only apply to requests made with effective
privilege modes less than M.
mrmcfg) register for MXLEN=64{reg: [
{bits: 12, name: 'RCID'},
{bits: 4, name: 'WPRI'},
{bits: 12, name: 'MCID'},
{bits: 4, name: 'QRID'},
{bits: 32, name: 'WPRI'},
], config:{lanes: 2, hspace:1024}}
mrmcfg) register for MXLEN=32{reg: [
{bits: 12, name: 'RCID'},
{bits: 4, name: 'WPRI'},
{bits: 12, name: 'MCID'},
{bits: 4, name: 'QRID'},
], config:{lanes: 1, hspace:1024}}
The Smsdqosid extension enables the scenario where two or more supervisor domains
share a common set of capacity and bandwidth controllers. In this setup, access
to the QRI of these controllers is restricted to RDSM, which then provides
mediated access for these supervisor domains to configure capacity or bandwidth
allocation and read the corresponding monitoring counters. A common QRID is
associated with requests from these domains, and the RDSM allocates sets of
RCID and MCID for each domain’s use. Supervisor domains are then restricted
to selecting a RCID and MCID in srmcfg from the allocated range. Although
the RDSM could limit access to srmcfg to prevent direct domain configuration,
enabling domains to program srmcfg directly is preferred for performance
reasons.
To facilitate delegation of srmcfg to the SD, the Smsdqosid extension
introduces a MXLEN-wide read/write machine-mode register called mnrmcfg.
The register is formatted as shown in Machine Next-Privilege Resource Management Configuration (mnrmcfg) register for MXLEN=64 for MXLEN=64 and
Machine Next-Privilege Resource Management Configuration (mnrmcfg) register for MXLEN=32 for MXLEN=32. The CSR number for mnrmcfg is 0x781.
mnrmcfg) register for MXLEN=64{reg: [
{bits: 12, name: 'RCID'},
{bits: 4, name: 'WPRI'},
{bits: 12, name: 'MCID'},
{bits: 4, name: 'QRID'},
{bits: 32, name: 'WPRI'},
], config:{lanes: 2, hspace:1024}}
mnrmcfg) register for MXLEN=32{reg: [
{bits: 12, name: 'RCID'},
{bits: 4, name: 'WPRI'},
{bits: 12, name: 'MCID'},
{bits: 4, name: 'QRID'},
], config:{lanes: 2, hspace:1024}}
Additionally, the extension introduces the SRL, SML, SSRM, and SSMM
fields in the msdcfg [MSDCFG] register.
The SRL (supervisor-RCID-length) is a WARL field with legal values from 0 to
RCIDLEN, where RCIDLEN is the number of implemented bits of the RCID
field of srmcfg. The least-significant bits of RCID are implemented first.
That is, if RCIDLEN is nonzero, then RCID[RCIDLEN-1:0] is writable.
The SML (supervisor-MCID-length) is a WARL field with legal values from 0 to
MCIDLEN, where MCIDLEN is the number of implemented bits of the MCID
field of srmcfg. The least-significant bits of MCID are implemented first.
That is, if MCIDLEN is nonzero, then MCID[MCIDLEN-1:0] is writable.
The supervisor-srmcfg-rcid-mode (SSRM) and SRL fields in the msdcfg
determine the legal values for the srmcfg.RCID field. When SSRM is 0, the
legal values are between 0 and 2SRL - 1. When SSRM is 1, the legal values
are between 0 and 2RCIDLEN - 2SRL - 1, and in this mode SRL must satisfy
0 ≤ SRL < RCIDLEN.
The supervisor-srmcfg-mcid-mode (SSMM) and SML fields determine the legal
values for the srmcfg.MCID field. When SSMM is 0, the legal values are
between 0 and 2SML - 1. When SSMM is 1, the legal values are between
0 and 2MCIDLEN - 2SML - 1, and in this mode SML must satisfy
0 ≤ SML < MCIDLEN.
These ranges constrain only the values that may be written to and held in
srmcfg.RCID and srmcfg.MCID. The effective RCID and MCID values that
accompany requests to the shared resource controllers may differ when SSRM or
SSMM is 0, as defined by the following equations.
The mnrmcfg.RCID and SSRM determine the value of RCID that accompanies
requests made with effective privilege mode less than M by the hart to shared
resource controllers as follows:
SRL_MASK = (1 << SRL) - 1
RCID = (SSRM == 0) ? ((mnrmcfg.RCID & ~SRL_MASK) | (srmcfg.RCID & SRL_MASK)) :
srmcfg.RCID
The mnrmcfg.MCID and SSMM determine the value of MCID that accompanies
requests made with effective privilege mode less than M by the hart to shared
resource controllers as follows:
SML_MASK = (1 << SML) - 1
MCID = (SSMM == 0) ? ((mnrmcfg.MCID & ~SML_MASK) | (srmcfg.MCID & SML_MASK)) :
srmcfg.MCID
These RCID and MCID values along with mnrmcfg.QRID accompany each request
made with effective privilege mode less than M by the hart to shared resource
controllers. The QRID and RCID values are used to determine the resource
allocations (e.g., cache occupancy limits, memory bandwidth limits, etc.) to
enforce. The QRID and MCID values are used to identify a counter to monitor
resource usage.
|
Note
|
Consider a QRI that supports 32 RCIDs and is mediated by RDSM between two SDs.
The RDSM may allocate 24 RCIDs to the first SD and 8 RCIDs to the second SD. The
|
The RDSM is expected to configure the mnrmcfg fields and the msdcfg fields
such that each SD is associated with a disjoint range of values for RCID and
MCID.
|
Note
|
The When a QRI is shared among multiple supervisor domains, the RDSM must mediate
access to the memory-mapped QoS register interface provided by the capacity
and bandwidth controllers. This mediation may be implemented by the RDSM
either by emulation of a virtual QoS register interface—intercepting and
emulating loads and stores to these registers—or by providing a supervisor
binary interface (SBI) for QoS configuration. Programming of the QoS
controllers is typically performed at job startup and is infrequent compared
to the switching of RCID and/or MCID values in When a supervisor domain is allocated a dedicated QRI, the RDSM may permit
direct access from that domain to the QRI’s memory-mapped register interface.
In this case, the RDSM should set |
Capacity and bandwidth controllers that support supervisor domains provide one or more memory-mapped QoS register interfaces (QRI). A QRI may be made exclusively accessible to a supervisor domain or the RDSM may mediate access to the QRI using an SBI. The RDSM can control access to the QRI from supervisor domains using MPT and/or PMP. The RDSM controls access to the QRI from devices using IOMPT and/or IOPMP.
The number of RCID and MCID supported by the controllers for each QRI need
not be identical. For maximal flexibility in allocation of RCID and MCID
values, it is recommended that the number of RCID and MCID supported for a
given QRID be identical in all capacity and bandwidth controllers in the
system.
The capacity and bandwidth controllers use the configurations that were
established for the RCID and AT in the request through the QRI corresponding
to the QRID in the request. Likewise the counters to count resource usage are
selected using MCID and the QRID in the request and a QRI can be used to
access counters associated with the corresponding QRID.
When a controller supports only a single QRI, machine mode is required to mediate access to that QRI. The RDSM may implement this mediation either by emulating a virtual QoS register interface—intercepting and emulating loads and stores to those registers—or by providing a supervisor binary interface (SBI) for QoS configuration.
When a capacity or bandwidth controller supports multiple QRIs, all resources in the controller are, by default, available for allocation through any QRI. The controller may optionally support reservation of resources for exclusive use by a specific QRI. When such reservation is supported, capacity or bandwidth may be reserved for allocation only by that QRI. Support for resource reservation enables effective partitioning of shared resources among supervisor domains (SDs) that share the controller.
When multiple QRIs are implemented, one of them must be designated for use by machine mode. This QRI is known as the machine-mode QRI (MQRI).
The MQRI supports a new operation, CONFIG_QRI_LIMIT (OP=4), which enables
partitioning of shared resources—such as cache capacity or memory
bandwidth—among the QRIs by specifying limits on the amount of each resource
that may be allocated by the corresponding QRI. If this operation is requested
on any QRI other than the MQRI, it shall fail with STATUS=2.
The MQRI may not support any RCID or MCID, and when these are unsupported,
its primary role is to configure resource limits for other QRIs managed by
the controller. When RCID and/or MCID are supported, the MQRI may be used
exclusively by machine mode or may be shared with one or more supervisor
domains.
Bits 31:28 of the cc_alloc_ctl and the bc_alloc_ctl registers of an MQRI
are defined as the QRID field. This field is reserved in all other QRIs of the
controller. It is used by the CONFIG_QRI_LIMIT (OP=4) operation to specify
the QRID to which the limit applies and is ignored by all other operations.
To reserve capacity for a QRI, the RDSM may use the CONFIG_QRI_LIMIT operation
through the MQRI, with the QRID field identifying the target QRI. The capacity
to be reserved is specified using the cc_block_mask register. The AT and
RCID fields of the cc_alloc_ctl register are ignored by this operation.
The mask specified in cc_block_mask must contain a contiguous run of 1s, and
an implementation may require the mask to reserve at least one capacity block;
otherwise, the operation shall fail with STATUS=5.
The number of 1 bits in the cc_block_mask value specifies the number of
capacity blocks that may be allocated using the identified QRI.
On successful completion, the NCBLKS field in the cc_capabilities register
of the target QRI shall reflect the updated number of allocatable capacity blocks.
|
Note
|
The |
|
Note
|
Let’s consider a cache with
|
To reserve bandwidth for a QRI, the RDSM may use the CONFIG_QRI_LIMIT operation
through the MQRI, with the QRID field identifying the target QRI. The amount
of bandwidth to reserve is specified using the Rbwb and Mweight fields of the
bc_bw_alloc register. The AT and RCID fields of the bc_alloc_ctl register
are ignored by this operation. Similarly, the sharedAT and useShared fields of
the bc_bw_alloc register are also ignored.
When the specified Mweight is nonzero, it represents a proportional share
of non-reserved or unused bandwidth that may be used by RCIDs associated with
the QRI. When Mweight is zero, the value in Rbwb serves as a hard limit, and
RCIDs associated with the QRI are not eligible to use any non-reserved or unused
bandwidth.
On successful completion of the operation, the bc_capabilities.MRBWB field of
the target QRI shall reflect the value specified in Rbwb.
When the Mweight established for a QRI is nonzero, a two-level weighted
sharing of unused or non-reserved bandwidth occurs. In such cases, the unused
bandwidth is apportioned among all QRIs with Mweight != 0 that are contending
for it, based on their relative Mweight values. Specifically, the proportion
of unused bandwidth allocated to QRID q is determined by dividing its
Mweight by the sum of the Mweight values across all QRIs contending for
unused bandwidth, as shown in equation (1).
This proportion \(P_q\) of the unused bandwidth is then further distributed among the RCIDs associated with that QRI using the weights configured for the RCIDs.
|
Note
|
Consider a bandwidth controller that supports two QRIs. For brevity, this
example controller does not support bandwidth allocation by access-type
|
|
Note
|
The bandwidth enforcement is typically work-conserving, meaning that it allows
unused bandwidth to be used by QRIs enabled to use it even if they have consumed
their When contending for unused bandwidth, the weighted share is typically computed among the QRIs that are actively generating requests in that accounting interval and have a non-zero weight programmed. |
|
Note
|
Typical use cases may require only a small number of QRIs (e.g., 1 to 4). In confidential computing scenarios, two QRIs may suffice: one designated for use by the non-confidential host OS or hypervisor, and another retained under RDSM control to mediate access on behalf of a confidential OS or hypervisor. The number of RCIDs and MCIDs supported by the two QRIs does not need to be identical. All QoS controllers in the SoC are strongly encouraged to implement the same number of QRIs. For each QRID, the corresponding QRI should support the same number of RCIDs and MCIDs across all controllers. For example, if QRID=1 supports 8 RCIDs and 4 MCIDs in one controller, QRID=1 should support the same in every controller. QRID=2 may support a different number of RCIDs and MCIDs than QRID=1, but it should do so consistently across all controllers. Furthermore, the QRID assigned to the machine-mode QRI should be the same across all controllers. This uniformity facilitates efficient and predictable allocation of QRIs and RCIDs/MCIDs to supervisor domains (SDs). |