Here are notes I've found useful for managing routing on clusters of GNU/Linux boxes.
See more sections on Gnu/Linux .
Read some good free chapters on networking at http://www.networkinglinuxbook.com/
Make sure your interface card is bound to an IP address
You should be able to ping all nodes from each other.
do not show more hops than necessary.
Make sure you have direct routes to all other nodes. Any of the following print route tables, with IP addresses or hostnames:
$ route -n $ route or $ netstat -rn $ netstat -r
To diagnose problems with routing on a cluster, examine the following information on the mayor node and on a worker node.
Make sure that your default installation does not run
a firewall packet filter by default. If you have the
iptables packages installed, you can disable firewall
support by either uninstalling those packages, or running
selecting "no firewall".
See if you have identified hosts explicitly.
$ cat /etc/hosts
If so, make sure "hosts" are resolved by "files" first:
$ cat /etc/nsswitch.conf
We use the lookup order "hosts: files nis dns".
If you are not using DNS, then
must be empty.
If you are using DNS, then specify the DNS nameserver
by IP address, and specify default domains for completing host names.
DHCP may recreate this file for you every time you reboot.
Check DNS lookups from other machines with
nslookup hostname or
dig +short hostname
Check data transfer rates and lags between cluster nodes by
ttcp on two nodes. Here's the result of a good
100Mb/s connection. Start the receiver with
vacn2> ttcp -r -s -n 16384 ttcp-r: buflen=8192, nbuf=16384, align=16384/0, port=5001 tcp ttcp-r: socket ttcp-r: accept from 18.104.22.168 ttcp-r: 134217728 bytes in 11.41 real seconds = 11488.08 KB/sec +++ ttcp-r: 92640 I/O calls, msec/call = 0.13, calls/sec = 8119.62 ttcp-r: 0.1user 2.4sys 0:11real 22% 0i+0d 0maxrss 0+2pf 0+0csw
vacn7> ttcp -t -s vacn2 -n 16384 ttcp-t: buflen=8192, nbuf=16384, align=16384/0, port=5001 tcp -> vacn2 ttcp-t: socket ttcp-t: connect ttcp-t: 134217728 bytes in 11.40 real seconds = 11495.07 KB/sec +++ ttcp-t: 16384 I/O calls, msec/call = 0.71, calls/sec = 1436.88 ttcp-t: 0.0user 0.8sys 0:11real 7% 0i+0d 0maxrss 0+2pf 0+0csw
A special routing problem may occur if a Linux cluster mayor node has two ethernet interfaces --- one for an external address and one for an internal cluster IP address. If the mayor's hostname corresponds to the external address, then the machine will mis-identify itself to other cluster nodes. Those nodes will route through the external interface, if they are able to route at all. A conservative fix would use the internal address of the mayor node as the first-hop gateway for the external address of the mayor node.
$ route add -net 22.214.171.124 netmask 255.255.255.255 gw 172.16.0.1
172.16.0.1 is the internal IP address
of the mayor node and
126.96.36.199 is the external
address of the mayor node.
Even better, set the route on all cluster nodes to use the internal address of the mayor as the first-hop gateway for any unknown external address:
$ route add -net 0.0.0.0 netmask 0.0.0.0 gw 172.16.0.1
This fix makes the previous fix unnecessary.
Outside machines might not have a route to the cluster nodes either. To add a route to a PC that needs to see a cluster node, set the route to use the external address of the mayor node as the first-hop gateway to all cluster node addresses:
$ route add 172.16.0.0 mask 255.255.0.0 188.8.131.52
172.16.0.0 with a mask of
specifies the address range of the cluster nodes,
is the external address of the cluster mayor node.
Bill Harlan, 2002-2005
Return to parent directory.