Conversation with Gemini

You said

How do I tune a FreeBSD machine to get maximum network throughput with high link latency?

Gemini said

Tuning FreeBSD for high-bandwidth, high-latency networks—often referred to as Long Fat Networks (LFNs)—requires optimizing the TCP stack to handle a large Bandwidth-Delay Product (BDP).

When latency is high, the "in-flight" data (unacknowledged packets) must be large enough to fill the pipe. If your buffers are too small, the sender stops and waits for ACKs, killing your throughput.

1. Expand TCP Window Limits

The default FreeBSD settings are usually too conservative for 10Gbps+ links with high RTT (Round Trip Time). You need to increase the maximum buffer sizes allowed by the kernel.

Add or update these in /etc/sysctl.conf:

  • Maximum Buffer Size: kern.ipc.maxsockbuf=16777216 (16MB, or higher for 40/100G links)

  • Send/Receive Space: net.inet.tcp.sendspace=262144 net.inet.tcp.recvspace=262144

  • Auto-Scaling Limits: net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216

  • Enable Auto-Tuning: Ensure net.inet.tcp.sendbuf_auto=1 and net.inet.tcp.recvbuf_auto=1 are active (default in modern FreeBSD).

2. Choose the Right Congestion Control

The default newreno algorithm is often too slow to recover throughput after a packet loss event on high-latency links. CUBIC is generally the better choice for LFNs.

  1. Load the module: kldload cc_cubic

  2. Make it permanent: Add cc_cubic_load="YES" to /boot/loader.conf.

  3. Set as default: sysctl net.inet.tcp.cc.algorithm=cubic

3. Network Interface Card (NIC) Offloading

Modern NICs can handle some of the processing overhead, freeing up the CPU. Ensure these are enabled on your interface (e.g., ix0, mlx0):

  • TSO (TCP Segmentation Offload): Allows the NIC to handle segmenting data.

  • LRO (Large Receive Offload): Aggregates incoming packets into larger buffers.

Check your settings with ifconfig. If disabled, enable them: ifconfig <interface> tso lro -vlanhwtso

4. Interrupt Moderation & Queues

On high-speed links, the CPU can be overwhelmed by per-packet interrupts.

  • RSS (Receive Side Scaling): Distributes incoming traffic across multiple CPU cores. Ensure your kernel supports RSS and that net.inet.rss.enabled=1.

  • Interrupt Rate: Limit how often the NIC interrupts the CPU. For Intel cards (em/igb/ix), you might adjust hw.ix.max_interrupt_rate=16000 in /boot/loader.conf.

5. Modern TCP Enhancements

FreeBSD supports several features that help mitigate the "high latency" penalty:

ParameterRecommended SettingPurpose
SACKnet.inet.tcp.sack.enable=1Selective ACKs allow retransmitting only lost segments.
Window Scalingnet.inet.tcp.rfc1323=1Required to support windows larger than 64KB.
ABCnet.inet.tcp.abc_version=2Appropriate Byte Counting helps increase the window faster.
ECNnet.inet.tcp.ecn.enable=1Explicit Congestion Notification reduces the need for retransmissions.

Summary Checklist for /etc/sysctl.conf

Bash
# General throughput optimizations
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.mssdflt=1460

# Congestion and Reliability
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.sack.enable=1
net.inet.tcp.ecn.enable=1

Note: Always monitor netstat -s -p tcp after making changes to see if you are hitting buffer limits or experiencing excessive retransmissions.

Are you dealing with a specific hardware setup, like 100G Chelsio or Mellanox cards, or is this for a general high-latency tunnel?

You said

What does vlanhwtso do?

Gemini said

In the context of FreeBSD and high-performance networking, vlanhwtso stands for VLAN Hardware TCP Segmentation Offload.

To understand what it does, it helps to break it down into its two core components: VLAN tagging and TSO.

1. The Core Concept: TCP Segmentation Offload (TSO)

Normally, when an application sends a large chunk of data (e.g., 64KB), the operating system's CPU has to do the heavy lifting of breaking that data into smaller segments (typically 1,460 bytes) that fit into standard Ethernet frames. This consumes significant CPU cycles.

With TSO, the CPU hands the entire large block of data to the Network Interface Card (NIC). The NIC's hardware then handles the segmentation, adding the necessary TCP/IP headers to each small packet. This significantly reduces CPU overhead and increases throughput.

2. The Challenge with VLANs

When you use VLANs (802.1Q), a 4-byte "tag" is inserted into the Ethernet header to identify which virtual network the traffic belongs to.

If you have TSO enabled but are also using VLANs, the NIC needs to be "VLAN-aware." It must know how to insert that 4-byte tag into every single small segment it creates from the large block of data.

3. What vlanhwtso specifically does

The vlanhwtso flag tells the FreeBSD kernel to offload TCP segmentation to the hardware specifically for traffic that is encapsulated in a VLAN tag.

  • When Enabled (vlanhwtso): The CPU sends one large 64KB chunk to the NIC along with the VLAN ID. The NIC hardware segments the data and stamps each packet with the VLAN tag. This is the most efficient method.

  • When Disabled (-vlanhwtso): If the NIC hardware is buggy or doesn't support this specific combination, the CPU must perform the segmentation manually for all VLAN traffic before sending it to the wire.

Why would you disable it?

While it sounds like a feature you would always want active, you will often see it disabled in troubleshooting guides (using -vlanhwtso in ifconfig). This is because:

  1. Driver Bugs: Some NIC drivers have historically struggled to calculate checksums correctly when both TSO and VLAN tagging are active simultaneously, leading to corrupted packets.

  2. Compatibility: Older hardware might support TSO for "untagged" traffic but fail when a VLAN tag is present.

  3. Packet Captures: If you are using tcpdump to debug network issues, hardware offloading can sometimes make the captured packets look "impossible" (e.g., seeing 64KB packets in the log), making it harder to see what is actually happening on the wire.

To check your current status: Run ifconfig -v <interface> (e.g., ifconfig -v ix0). Look for vlanhwtso in the options list. If it has a minus sign in front (-vlanhwtso), it is currently disabled.

profile picture


Gemini is AI and can make mistakes.