AI clusters have gotten a shared infrastructure. Neoclouds, enterprise AI platform groups, monetary companies organizations, life sciences groups, and analysis teams have to share GPU capability. This shared infrastructure can undergo from decrease monetization, elevated operational complexity, and restricted management and visibility throughout tenants, workloads, hosts, and the community cloth.
EVPN/VXLAN is the sensible community basis. It gives tenant-scoped overlay segmentation utilizing VRFs, VNIs, route distinguishers, and route targets. Nonetheless, tenant-aware segmentation isn’t job-aware segmentation. The scheduler understands jobs; the community sometimes understands routes, interfaces, queues, drops, and flows.
Why AI clusters want multitenancy
Devoted GPU clusters are easy to isolate, however they’re inefficient to function at scale. As GPU estates develop, organizations desire a shared useful resource pool that may serve a number of groups, prospects, and workload courses with out forcing each group into its personal bodily cluster. In any other case, one group can have stranded GPUs in a devoted island whereas one other waits in queue.
The requirement seems in a number of patterns:
- A GPU-as-a-Service supplier maps every tenant to an exterior buyer with its personal handle and coverage area (per-customer isolation whereas maintaining the GPU pool shareable).
- An enterprise platform staff maps tenants to growth, testing, manufacturing fine-tuning, mannequin analysis, or regulated analytics (constant setting boundaries with out constructing separate clusters).
- A monetary service division separates fraud analytics, threat modeling, and analysis workloads on one coaching cluster (stronger management boundaries and auditability with out duplicating GPU islands).
- A analysis group assigns shared GPU capability to impartial analysis teams (clearer quota, utilization, and troubleshooting accountability throughout competing initiatives).
Because of this multitenancy can’t cease at compute allocation. Distributed coaching depends upon east-west GPU communication, sometimes over Ethernet materials, so the community turns into an integral a part of the isolation and efficiency boundary.
How business solves it at this time
Present AI multitenancy is often carried out throughout three layers:
- Orchestration and scheduler layer. Kubernetes-based platforms, GPU cloud orchestration programs, and Slurm schedulers outline the logical possession mannequin for the cluster. They monitor tenants or initiatives, customers, queues or namespaces, job requests, node placement, and GPU allocation. For instance, Tenant A may submit Job 100 requesting eight GPUs throughout two servers, whereas Tenant B submits Job 200 requesting 4 GPUs on a unique set of nodes. As an example, in an orchestration platform like Rafay, the platform can personal tenant onboarding and infrastructure intent, whereas the precise job scheduling could occur in Kubernetes, Slurm, or a tenant-operated scheduler.
- Host isolation layer. The host enforces the native system boundary for every workload. If a tenant receives entire servers, isolation is easier as a result of the server, GPU set, and NIC set might be handled as one tenant-owned unit. If a number of tenants or jobs share GPUs throughout the identical server, the runtime should expose solely the assigned GPU units and bind the workload’s communication libraries, corresponding to NCCL or UCX, to the meant NICs. This host-side mapping issues as a result of a GPU server could have a number of NICs related to completely different switches or tenant-facing community segments. Cloth segmentation can isolate site visitors as soon as it enters the community, nevertheless it can’t appropriate an incorrect native project the place the workload is allowed to make use of the incorrect GPU or NIC.
- Community segmentation layer. EVPN/VXLAN gives scalable tenant segmentation throughout the material. VXLAN encapsulates tenant site visitors and makes use of VNIs to determine the overlay section or routing area. EVPN makes use of BGP to promote endpoint and prefix reachability and to regulate which VTEPs import a tenant’s routes by means of route targets. In a routed AI cloth, a tenant generally maps to a VRF and a number of VNIs, with route distinguishers maintaining tenant routes distinctive and route targets controlling import-export coverage. This enables overlapping tenant handle house and scoped reachability throughout a shared underlay.
ACLs or safety group ACLs can add supply and vacation spot coverage, particularly in brownfield L3 designs or the place the material can’t but devour richer workload id. Their limitation is operational scale: static or manually up to date ACL and VRF insurance policies don’t naturally comply with fast-changing AI job placement.
Collectively, these layers present a workable tenant-level mannequin. The remaining hole is job context: the community can often see tenant context, interfaces, routes, queues, and counters, however not the particular scheduler job working inside a tenant. Tenant segmentation itself doesn’t mechanically isolate Job 100 from Job 101 inside the identical tenant until job id can also be carried, derived, or programmed into community coverage.
Cisco Nexus One integration with AI iorchestration platforms
Cisco Nexus One is effectively positioned because the broader basis for making tenant-aware AI materials extra deterministic. On this structure, Nexus One is the whole cloth automation, integration, and visibility floor for the whole cloth.
Nexus One can present cloth topology context to an AI infrastructure orchestration platform corresponding to Rafay by means of integration workflows or APIs. That lets groups map tenant VRFs, VLANs, and port assignments on to a tenant, reasonably than managing them solely as an summary tenant label.
As well as, Nexus One extends the mannequin past provisioning. Tenant-level visibility can present the tenant’s cloth path and related well being indicators corresponding to congestion, drops, and so forth. This enhances AI job observability: job-aware views can correlate scheduler, topology, optics, GPU telemetry, analytics, and anomalies, whereas tenant-specific Job-ID enforcement stays a separate future-facing coverage functionality.
Tenant-aware isn’t job-aware
Tenant segmentation solutions the query, “Which buyer or group owns this site visitors?” AI operations typically want, “Which coaching job is creating or experiencing this site visitors inside a tenant?”
This distinction issues for segmentation in addition to throughout troubleshooting. A scheduler can determine the job, allotted nodes, GPUs, and runtime state. The community can determine interfaces, routes, queues, drops, ECN marks, PFC occasions, optics well being, and paths. With out correlation, operators should manually join these two views.
The result’s a standard operational downside: the material reveals a scorching uplink or lossy interface, whereas the platform staff sees a gradual coaching job. The lacking hyperlink is the workload id within the community working mannequin.
Future route: AI Job-ID-aware segmentation
Job-ID-aware segmentation route—patent-pending expertise from Cisco—is the logical subsequent step. (Notice that this describes our architectural route, not a transport function.) The aim is for infrastructure orchestrator (corresponding to Rafay) and scheduler (corresponding to Slurm) intent to hold each tenant id and job id into the material management and data-plane mannequin.
In that mannequin, the material controller interprets job intent into coverage. The change information aircraft carries or derives a job ID, for instance by means of VXLAN GPO bits, and enforces that solely endpoints in the identical licensed tenant and job can change RoCEv2 site visitors.
The anticipated advantages are operationally necessary:
- Less complicated operations, as a result of groups can cause in tenants and jobs as an alternative of translating each grow to be static community objects—essential for NetOps and cloth operations groups.
- Deeper visibility, as a result of drops, congestion, paths, and optics might be correlated to workload context reasonably than solely to interfaces or tenant VRFs—helpful for platform engineering and SRE groups.
- Extra granular segmentation, as a result of coverage can comply with the lifecycle of a job reasonably than stopping on the tenant boundary—necessary for safety, compliance, and tenant governance groups.
This method is constructed on open requirements—not a proprietary overlay. EVPN/VXLAN is IETF-defined, and the Group Coverage Choice (GPO) follows the identical path: an IETF-defined mechanism that encodes a gaggle/coverage identifier within the VXLAN header alongside the VNI, which Cisco NX-OS implements in alignment with the open specification. Tenant scope (VNI) and workload/job scope (GPO) are due to this fact expressed in constructs a standards-compliant cloth can interpret—letting operators evolve from tenant-aware to job-aware enforcement and not using a cloth forklift.
Technical instance: tenant and job boundaries
Think about a GPU-as-a-Service setting with two prospects, Tenant A and Tenant B. Every tenant is mapped to its personal VRF/VNI within the EVPN/VXLAN cloth. Tenant-level segmentation prevents Tenant B from reaching Tenant A.
Now assume Tenant A runs two concurrent coaching jobs. Job 100 makes use of GPUs on servers 1 and a pair of. Job 101 makes use of completely different GPUs on the identical shared cloth. Tenant-level EVPN/VXLAN nonetheless treats each jobs as Tenant A site visitors. Job-ID-aware segmentation would add one other enforcement dimension: Job 100 endpoints might change RoCEv2 site visitors with different Job 100 endpoints, however not with Job 101 endpoints, even inside the identical tenant.
That’s the architectural shift: EVPN/VXLAN stays the tenant basis, whereas Job ID turns into the long run workload-level coverage and observability attribute.
Advancing safety from tenant-level to job-level segmentation
AI information middle multitenancy begins with EVPN/VXLAN tenant segmentation, nevertheless it doesn’t finish there. The stronger working mannequin combines topology-aware provisioning, tenant-level enforcement, and end-to-end visibility at this time, then evolves towards Job-ID-aware segmentation as scheduler and orchestrator integration matures.
If you’re designing a shared AI cluster at this time, tenant-aware EVPN/VXLAN is the muse. Job-aware enforcement and observability are the subsequent frontier.
*Particular due to Ramesh Ponnapalli and his staff, whose engineering management has been instrumental in bringing this expertise to life.
Further assets:
