Virtio net with bridge does not transmit traffic with queues >=4

General
1

I am configuring virtio net with bridge. Followed the general steps in the virsh xml . Specified driver to vhost and tried putting queues to 4 because I have 4 packet processing threads. But it does not work. No traffic is being seen on the nics. Not even drops. Is there some limitation I am not aware of

Tried putting the queue to 1,2,3 and they worked

2

Potential Reasons for the Issue:

  1. Virtio-net Driver Support:
  • Queue Configuration: Not all NIC drivers or virtual environments support multiple queues for Virtio-net. Although modern versions of KVM/QEMU and Virtio-net drivers generally support multiqueue operation, you should confirm that your host kernel, guest kernel, and NIC driver all support multiqueue.
  • Vhost-net: Vhost-net should theoretically handle multiple queues, but this requires specific driver support on both the host and guest sides.
  • Queue Limitations: The physical NIC on your host may have a limit to the number of queues it supports. When you try using 4 queues, if the hardware or driver does not support it, traffic might stop flowing or you might not see any drops.
  1. NIC Multiqueue Support:
  • Check if your physical NIC supports 4 queues. Many NICs do, but some have limitations (especially older models) and can only handle a certain number of queues. For instance, certain Intel and Realtek NICs may not support more than 1 or 2 queues, while more advanced NICs (like Mellanox) can handle multiple queues.To check the capabilities of your NIC, you can use ethtool on the host:
ethtool -l <nic_name>
  • This will tell you how many transmit and receive queues your NIC supports.
  • Guest Network Stack Limitations:
  • Some guest OSes may not handle multiqueue configurations well. Ensure that your guest OS supports multiqueue and that the Virtio-net driver is correctly installed in the guest. Check for the latest version of the Virtio-net drivers to see if they support multiqueue configurations.
  • Virtio-net Configuration in XML:
  • Your virsh XML configuration for Virtio-net should explicitly define multiqueue settings, including setting the correct driver (vhost) and enabling multiqueue support:Example:
<interface type='bridge'>
    <mac address='52:54:00:ab:cd:ef'/>
    <source bridge='br0'/>
    <model type='virtio'/>
    <driver name='vhost' queues='4'/>
</interface>
  1. Make sure that the queues='4' configuration is correctly set in the XML file.
  2. CPU Pinning and NUMA Considerations:
  • Multiqueue networking benefits from proper CPU pinning and NUMA awareness. Check if your virtual CPU threads are properly pinned to avoid cross-NUMA node traffic, which can result in poor performance and issues with traffic handling. Properly allocate vCPUs to match the number of queues you want to configure.
  1. Host Network Bridge Setup:
  • Verify that the network bridge setup on the host is properly configured. Sometimes, issues with the bridge (e.g., incorrect MTU settings or misconfigurations in the bridge setup) can prevent traffic from flowing correctly, especially when scaling to multiple queues.
  1. Resource Contention:
  • If the host or the VM is under heavy load (CPU or memory), resource contention could affect the ability of the NIC to handle multiqueue traffic processing. Monitor resource usage to ensure the system is not bottlenecked.

Debugging Steps:

  1. Check the Host NIC Multiqueue Support: Ensure that the NIC on the host supports 4 queues using:
ethtool -l <nic_name>