Tuning Linux: TCP hash entries

To contact us Click HERE
Tuning Linux for scalable network applications is hard, partly because it's not documented anywhere. Or if it is, all the top Google results are for old versions of Linux. For example, the TUNABLE file of (I supposed) tunable parameters goes back to the 1.x version of the Linux kernel, even though we are now on version 3.4.

One important parameter is the number of TCP hash entries. When a TCP packet arrives, the kernel needs to do a lookup to find which connection the packet belongs to. This is done with a typical hash table. If the hash table is too small, you'll get "hash collisions", requiring several memory lookups before you reach the correct entry. In a scalable system, each memory lookup costs 300 clock cycles.

The parameter that controls this is for Linux is "thash_entries=" which you set as a typical boot parameter. The default is 65536 entries for every gigabyte of memory. In other words, an 8-gigabyte system will have 512k hash entries. You can find out many your current system has by doing "dmesg | grep "TCP established hash table entries"

This is probably more than enough for even the most scalable systems, so I doubt that you need to change it. That's probably why it was so hard finding the stupid parameter -- while it's a common issue for most operating systems, Linux automatically chooses a number that's big enough, so nobody cares about configuring it on Linux.

Now, after rummaging around in the Linux stack, I feel dirty, and am off to go take a shower ...

---

Update: The equivalent Solaris parameter is tcp_conn_hash_size. I include the name here for people searching for the Linux equivalent of the Solaris parameter.Update: Other useful boot options:

hugepages

Reserves the specific number of 2-megabyte TLB pages. Allocating hugepages after boot can be difficult, because memory will be fragmented by allocating/deleting 4096-byte pages. Thus, reserve this number at boot time for use by your application when it starts after boot.