X410 GNURadio streaming underflows

[bastiber]

Hi there,

I want to use the full bandwidth of the X410 with the CG_400 FPGA image. I have tweaked all the network settings according to the kb article and also get no underflow errors with the benchmark_rate from the uhd examples at 500 MSps. (see below).

But with a simple Gnuradio flowchart (second screenshot) with a file source and a USRP sink, the required sample rate of 491.52 MSps cannot be achieved, in fact I only get ~365 MSps and therefore a lot of underflows from the USRP.

The problem is not the file source block: replacing the USRP sink with a null sink results in a sample rate of 564 MSps.

Does anyone have experience with such high sample rates and gnuradio?

Cheers,
Sebastian

---

$sudo ./benchmark_rate \                    
--args "type=x4xx,addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=172.24.12.67,master_clock_rate=500e6" \
--priority "high" \
--multi_streamer \
--duration 60 \
--channels "0" \
--rx_rate 500e6 \
--rx_subdev "B:1" \
--tx_rate 500e6 \
--tx_subdev "A:0"

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.6.0.0-0ubuntu1~focal1
[00:00:00.000055] Creating the usrp device with: type=x4xx,addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=172.24.12.67,master_clock_rate=500e6...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=172.24.12.67,type=x4xx,product=x410,serial=427B85F,name=ni-x4xx-327B85F,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,master_clock_rate=500e6
[WARNING] [MPM.RPCServer] A timeout event occured!
[INFO] [MPM.PeriphManager.ClkMgr] Using Clock Configuration:
DB0: Master Clock Rate: 500.0 MSps @Converter Rate 3.0 GHz
DB1: Master Clock Rate: 500.0 MSps @Converter Rate 3.0 GHz
[INFO] [MPM.PeriphManager] init() called with device args `fpga=CG_400,master_clock_rate=(500000000.0,),mgmt_addr=172.24.12.67,name=ni-x4xx-427B85F,product=x410,second_addr=192.168.20.2,clock_source=internal,time_source=internal,initializing=True'.
[INFO] [0/Radio#0] Clocking reconfigured, running ADC Self Cal on DB0...
[INFO] [0/Radio#0] Calibrating channel 0...
[INFO] [0/Radio#0] Calibrating channel 1...
[INFO] [0/Radio#1] Clocking reconfigured, running ADC Self Cal on DB1...
[INFO] [0/Radio#1] Calibrating channel 2...
[INFO] [0/Radio#1] Calibrating channel 3...
Using Device: Single USRP:
  Device: X400-Series Device
  Mboard 0: x410
  RX Channel: 0
    RX DSP: n/a
    RX Dboard: B
    RX Subdev: 1
  TX Channel: 0
    TX DSP: n/a
    TX Dboard: A
    TX Subdev: 0

[00:00:17.638519618] Setting device timestamp to 0...
Setting TX spb to 1984
[00:00:17.640044495] Testing receive rate 500.000000 Msps on 1 channels
[00:00:17.641169153] Testing transmit rate 500.000000 Msps on 1 channels
[00:01:17.641902988] Benchmark complete.


Benchmark rate summary:
  Num received samples:     29999545568
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  29999401344
  Num sequence errors (Tx): 0
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

[Tobi82]

Hi. I have exactly the same Problem. Benchmark_rate with 4 channels send and 4 channels receive result:

Benchmark rate summary:
  Num received samples:     20166667524
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  19664275136
  Num sequence errors (Tx): 0
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

But if i have a GNURadio flowchart like yours, i have only U´s and O´s. Have you solve this problem? What are your settings in USRP Sink (Adress, Arguments...)?

I see you don´t use dpdk? I think without dpdk its not possible to get the highest performance

[dhpancha]

Hi,

Not sure if you were able to resolve your issue in gnuradio. However, I'm trying to implement dpdk in order to transmit (as well as receive) at the high sample rate specified in the X410. I was wondering if you built dpdk from source to get it to work? If so, which version did you use? I tried building from source from multiple versions but had various problems. 
Apparently I need to enable this line: "CONFIG_RTE_LIBRTE_MLX5_PMD=y" in the build files but it doesn't seem to be available in the later versions (20.11 to recent versions).

Also, what NIC(s) did you use for this? Are you using the Mellanox drivers? 

I attempted to follow the instructions from here: https://kb.ettus.com/Getting_Started_with_DPDK_and_UHD

Your help with this is highly appreciated. Thanks

[dhpancha]

I ended up using NI Dual 100 Gigabit Ethernet PCIe and was able to run the uhd benchmark test. However, I keep getting sample drops and sequence errors and not sure how to resolve it. I'm using the B channels and also have it configured in loopback. Any ideas?

/usr/local/lib/uhd/examples$ sudo ./benchmark_rate --args "type=x4xx,product=x410,addr=192.168.20.3,mgmt_addr=192.168.1.3,use_dpdk=1" --priority "high" --multi_streamer --rx_rate 245.76e6 --rx_subdev "B:1" --tx_rate 245.76e6 --tx_subdev "B:0"

[INFO] [UHD] linux; GNU C++ version 11.4.0; Boost_107400; DPDK_21.11; UHD_4.7.0.HEAD-0-ga5ed1872
EAL: Detected CPU lcores: 32
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
[00:00:00.000109] Creating the usrp device with: type=x4xx,product=x410,addr=192.168.20.3,mgmt_addr=192.168.1.3,use_dpdk=1...
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.1.3,type=x4xx,product=x410,serial=328AFD7,name=ni-x4xx-328AFD7,fpga=UC_200,claimed=False,addr=192.168.20.3,use_dpdk=1
[INFO] [MPM.PeriphManager] init() called with device args `fpga=UC_200,mgmt_addr=192.168.1.3,name=ni-x4xx-328AFD7,product=x410,use_dpdk=1,clock_source=internal,time_source=internal,initializing=True'.
Using Device: Single USRP:
Device: X400-Series Device
Mboard 0: x410
RX Channel: 0
RX DSP: 0
RX Dboard: B
RX Subdev: 1
TX Channel: 0
TX DSP: 0
TX Dboard: B
TX Subdev: 0

[00:00:01.970153754] Setting device timestamp to 0...
[00:00:01.971248509] Testing receive rate 245.760000 Msps on 1 channels
Setting TX spb to 1992
[00:00:01.972147276] Testing transmit rate 245.760000 Msps on 1 channels
U[D00:00:02.502074084] Detected Rx sequence error.
U[D00:00:03.501866063] Detected Rx sequence error.
U[D00:00:04.501965973] Detected Rx sequence error.
U[D00:00:05.501905705] Detected Rx sequence error.
U[D00:00:06.501533956] Detected Rx sequence error.
U[D00:00:07.501567020] Detected Rx sequence error.
U[D00:00:08.501554331] Detected Rx sequence error.
U[D00:00:09.501610267] Detected Rx sequence error.
U[D00:00:10.501971471] Detected Rx sequence error.
U[D00:00:11.501931301] Detected Rx sequence error.
[00:00:11.973155250] Benchmark complete.

Benchmark rate summary:
Num received samples: 2344330478
Num dropped samples: 113209128
Num overruns detected: 0
Num transmitted samples: 2335492512
Num sequence errors (Tx): 0
Num sequence errors (Rx): 10
Num underruns detected: 10
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0

Done!

[BIXUT]

Hi,

I am working with the Mellanox network card and dpdk 23.11, I also have the same problem to set "CONFIG_RTE_LIBRTE_MLX5_PMD=y", may I ask you how did you solve this problem? Also, to do the NIC configuration, I think I should edit the uhd.conf file, did you also do this step and how did you find this file?

Thank you very much for your time!

[taks000]

I'm using the default Ubuntu 22.04 installed mlx5 driver, which works with 20.11 DPDK ("dpdk-devbind.py -s" shows it active) and both ConnectX-5 and ConnectX-6 NICs. However, I'm suffering from the same problem - occasional underflows. Unfortunately, an underflow doesn't just drop transmission for me, it slides my timing out requiring a restart to realign the timing.

I've done everything that has been advised regarding configuration, including isolating CPUs and explicitly preventing other applications from using them. There are things I can do in my code to minimize the frequency of an underflow, but not eliminate it.

[taks000]

Oh, note, I don't get overflows on the receivers, and they all run at higher rates.