In an era where data-intensive applications demand both speed and efficiency, traditional CPU-driven networking can create performance bottlenecks. To address this, Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) offers a powerful solution—allowing data to be transferred directly between memory spaces, bypassing CPU overhead. At iWave, we have pushed the envelope by integrating AMD’s ERNIC IP on our Zynq UltraScale+ MPSoC development kit, delivering a fully hardware-offloaded 100 Gb/s Ethernet solution.
What Is RoCE and Why It Matters
RDMA enables peer-to-peer memory transfers without involving the host’s CPU. This not only reduces latency but significantly lowers CPU utilization, freeing up headroom for application-level tasks. RoCE, the protocol layer for RDMA over Ethernet, brings these benefits to standard Ethernet infrastructure, making it a compelling choice for high-performance embedded systems.
There are two major versions of RoCE:
- RoCE v1, which works within a single Ethernet broadcast domain, uses Ethertype 0x8915.
- RoCE v2, which adds IP and UDP headers to enable Layer-3 routing and operation across larger network topologies.
iWave’s RDMA-Enabled Implementation with ERNIC
We have successfully demonstrated a 100 Gb/s Ethernet solution on our Zynq UltraScale+ MPSoC (ZU19EG) development kit by integrating AMD’s ERNIC (Embedded RDMA NIC) IP core.
Key highlights of our implementation:
- ERNIC IP offloads the full RoCE v2 stack into programmable logic (PL), enabling high-throughput and low-latency data paths.
- A driver in the Processing System (PS) handles communication: posting incoming packets to DDR and managing “doorbell” signaling to the ERNIC core.
- We provide both kernel and user-space libraries optimized for RDMA SEND, READ, and WRITE operations, delivering efficient memory transfer mechanisms.
- Precise time synchronization is supported via IEEE 1588 PTP timestamping, ensuring deterministic performance for real-time applications.
Demonstration Setup
Our reference setup includes:
- iWave’s ZU19EG-based Zynq UltraScale+ MPSoC development board
- A Mellanox ConnectX-5 100 Gb/s NIC
- A 1 Gb/s PTP-enabled NIC to support time synchronization
- QSFP-28 modules, MTP cables, and standard RJ45 Cat-6 Ethernet
- Host system running Ubuntu 22.04
Performance and Validation
In our tests, the system reliably achieves full 100 Gb/s throughput, executing RDMA READ, WRITE, and SEND operations with minimal latency. We also used custom data-pattern generators (raw and video) to validate application-level performance, capturing 8K video at over 100 fps and 4K video at over 400 fps.
Applications & Use Cases
This RDMA-enabled architecture unlocks performance advantages across domains, including:
- Data Centers & Cloud: Accelerate server-to-server communication using high-speed, low-latency links.
- High-Performance Computing (HPC): Offload memory transfers to specialized hardware for compute-bound tasks.
- Real-Time Video Processing: Stream high-resolution video (8K/4K) at high frame rates with precise timing.
- Storage & Edge Solutions: Rapid data transfers in storage appliances or edge devices without taxing the CPU.
Conclusion
By combining iWave’s Zynq UltraScale+ MPSoC platform with AMD’s ERNIC IP, we deliver a high-performance, hardware-offloaded RDMA solution over 100 Gb Ethernet. This innovation greatly reduces latency, frees up CPU resources, and opens up new possibilities in networking, real-time processing, and next-generation embedded applications.
To learn more about how iWave can help you architect RDMA-enabled systems for your next-gen products, reach out to our team at mktg@iwave-global.com
