EMC Isilon – Checking Network Connectivity Between Clusters

Introduction

If you work with EMC Isilon clusters you will likely have a requirement at some point to check network connectivity between clusters to ensure reachability for all nodes. Now you can do it manually if you have small clusters but of course once you get to any sort of size that becomes incredibly tedious. Often clusters have multiple networks configured so these checks can become a burden if an efficient way to check node to node reachability is not found.

Solution

The solution uses standard Unix commands with OneFS specific commands to get the results required.

Listing the interfaces / addresses across a cluster is quite simple:

isi_for_array -s 'ifconfig <interface>' | grep <network range>

This will return all of the addresses across the cluster nodes for the specified network adapter.

<Cluster Name>-1# isi_for_array -s ' ifconfig vlan1' | grep 'inet <network>'
<Cluster Name>-1: inet <network>.10 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-2: inet <network>.11 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-3: inet <network>.12 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-4: inet <network>.13 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-5: inet <network>.14 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-6: inet <network>.15 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-7: inet <network>.16 netmask 0xffffff00 broadcast <network>.255 zone 1 
<Cluster Name>-8: inet <network>.17 netmask 0xffffff00 broadcast <network>.255 zone 1

When you know what interfaces / addresses you want to confirm connectivity to, from another cluster, you can use a for loop to ping that range of addresses from every node across the cluster. Adding a grep to the command will find any misses with 100% packet loss.

for i in {150..182}; do isi_for_array -s "ping -c 1 -i 2 <network range>.$i"; done | grep 100.0%

You should expect to have nothing returned if all nodes can communicate with each other. If you find some nodes are unable to ping others then further investigation will be required.