Remote Direct Memory Access (RDMA) is a key technology for modern high-performance systems across data centres, telecommunications, storage, and real-time data acquisition, enabling direct memory-to-memory data transfers between systems without CPU involvement. This approach significantly reduces latency, lowers system overhead, and delivers near line-rate performance.
iWave’s RoCEv2 (RDMA over Converged Ethernet) IP core extends these benefits to FPGA and SoC platforms, offering a complete, high-speed, and efficient data-movement engine. It combines deterministic performance with standards-based interoperability, making it well suited for scalable, Ethernet-based high-performance systems.
Key features of iWave RoCEV2 IP core includes,
- Compliant with RoCEv2 specification and RNIC products
- Hardware operated Reliable connection (RC) service
- Easy integration with 10G, 25G, 40/50G and 100G Ethernet MAC
- Supports RDMA SEND, RDMA RECEIVE, RDMA READ and RDMA WRITE operations
- Configurable Queue Pair support and QP1 Support for Connection Management
- Retransmission logic implemented in hardware
- Supports RoCEv2 packets and supports ARP and ICMP packets though sideband interface
- Support for L2 (Ethernet) / L3 (IP) RoCE v2 path modes and IP Layer (L3) support IPV4
In order to validate the performance, correctness, and scalability of a RoCEv2 IP core implemented on FPGA, multiple variants are employed and this article describes couple of commonly used variants of this IP core. Each variant targets a different validation objective—ranging from peak throughput demonstration to system-level memory interaction—while using the same RoCEv2 protocol stack and 100G Ethernet MAC.
Variant 1: Test Driver as Data Source
In this variant, a test driver module implemented in the FPGA Fabric acts as the data source and sink. The primary objective is to validate the RoCEv2 IP core’s protocol handling, data integrity, and maximum achievable throughput without being limited by system memory bandwidth. RoCEV2 IP core interfaces with the data source and sink using AXI-S interface.
The data path of this variant consists of:
- Test Driver in FPGA Fabric
- RoCEv2 IP Core
- 100G Ethernet MAC
This variant of the IP supports the AXI-Stream interface directly connected to the test driver, which includes both a data generator and a data checker. IP also interfaces with vendor specific Ethernet MAC over AXI-Stream interface. This variant of the IP proves,
- Protocol correctness and data integrity validation
- FPGA-to-FPGA or FPGA-to-NIC (e.g., Mellanox) interoperability testing
- Marketing and benchmark demonstrations showcasing maximum performance
Functional Flow
- Connection Management: Connection establishment and configuration are handled by the RoCEV2 driver running on ARM core.
- RDMA Write Operation: The test driver generates known data patterns, which are packetized by the RoCEv2 IP core and transmitted over the 100/200/400G MAC for direct placement into the remote system’s memory.
- RDMA Read Operation: Data is read back from the remote memory over the 100/200/400G MAC, de-packetized and integrity-checked by the RoCEv2 IP core, and verified against the reference data within the test driver.
Throughput Characteristics
- The AXI-Stream interface is driven directly by the test driver logic, eliminating DDR or DMA bottlenecks which is the main reason for the reduction in the performance
- Test driver logic generates the incremental/Random data whenever the IP is ready to receive the data over AXI stream interface and width of this AXI interface can be modified based on the ethernet speed
- This variant demonstrates peak throughput proving that the IP core itself is not a throughput-limiting factor and near line-rate performance can be achieved
Variant 2: DDR as Data Source
In this variant, DDR memory connected to the ARM core or FPGA fabric is used as the data source and destination. The focus shifts from raw IP throughput to system-level performance, including DMA behaviour, memory bandwidth limitations, and coexistence with other applications. RoCEV2 IP core interfaces with the data source and sink using AXI4 interface.
The data path of this variant consists of:
- DDR Memory
- DMA Engine integrated with RoCEv2 IP
- 100G Ethernet MAC
The RoCEv2 IP includes an internal DMA controller to move data between DDR and IP over AXI4 interface and IP also interfaces with vendor specific Ethernet MAC over AXI-Stream interface. This variant of the IP proves,
- End-to-end system validation
- Evaluating memory bandwidth constraints
- Multi-application scenarios where DDR memory is shared
- Understanding real-world performance impact in deployed systems
Functional Flow
- Connection Management: Connection establishment and configuration are handled by the RoCEV2 driver running on ARM core.
- RDMA Write Operation: Known data is preloaded into predefined DDR locations, continuously read by the DMA engine, packetized by the RoCEv2 IP core, and transmitted over the 100/200/400G MAC for direct placement into the remote system’s memory.
- RDMA Read Operation: Data is read back from the remote memory over the 100/200/400G MAC, de-packetized and integrity-checked by the RoCEv2 IP core, written to predefined DDR locations under IP-controlled DMA, and finally verified by the test application for data integrity.
Throughput Characteristics
- When an AXI4 interface is used between the DDR controller and the IP, the achievable throughput is constrained by the AXI data width and the AXI clock frequency.
- This bottleneck in throughput can be significantly addressed by implementing ping-pong buffering and using the FPGA-attached DDR as the data source and sink.
- In general, when DDR memory is used as the source and sink, the AXI4 interface becomes the system bottleneck as it will not be able to feed enough data that IP can handle underutilizing the bandwidth offered by the IP and MAC; however, this throughput limitation is not inherent to the RoCEv2 IP core itself.
Conclusion
Both IP variants are designed to complement each other, addressing different needs within a RoCEv2 IP core use case.
- The Test Driver variant proves that the IP core can sustain maximum throughput and operate correctly at line rate without internal bottlenecks.
- The DDR variant highlights realistic system constraints and provides insight into memory bandwidth limitations and application-level behaviour.
Together, these approaches provide a comprehensive validation strategy—ensuring both protocol excellence and system readiness for high-performance RDMA-based applications.
For more information or inquiries, reach out to mktg@iwave-global.com
