Monitoring virtual FC Client Traffic

With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).

Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:

$ vios lsnports ms15-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78CB.001.XXXXXXX-P1-C5-T1  1       64      62      2032    2012
fcs1  U78CB.001.XXXXXXX-P1-C5-T2  1       64      62      2032    2012
fcs2  U78CB.001.XXXXXXX-P1-C5-T3  1       64      61      2032    1979
fcs3  U78CB.001.XXXXXXX-P1-C5-T4  1       64      61      2032    1979
fcs4  U78CB.001.XXXXXXX-P1-C3-T1  1       64      50      3088    3000
fcs5  U78CB.001.XXXXXXX-P1-C3-T2  1       64      63      3088    3077
$

We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:

$ vios fcstat ms15-vio1
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x210000XXXXX56EC5  fcs1   774.75/s  129.51 MB/s  1332.71/s   92.96 MB/s  20
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   318.10/s   83.39 MB/s  481.34/s   126.18 MB/s  0
ms15-vio1  fcs2     0x210000XXXXX56EC6  fcs2   318.10/s   83.39 MB/s  480.78/s   126.03 MB/s  0
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   583.98/s   60.35 MB/s  1835.17/s  124.86 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   583.70/s   60.27 MB/s  1836.21/s  124.92 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.19/s  165.08 MB/s  1032.81/s   17.25 MB/s  46
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   775.12/s  129.48 MB/s  1047.32/s   17.15 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   775.78/s  128.99 MB/s  1037.99/s   17.39 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s    0.00 B/s   290.39/s    76.12 MB/s  0
aixtsmp1   fcs0     0xC050760XXXXX0052  fcs4   142.89/s   36.12 MB/s    0.00/s     0.00 B/s   26
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   234.97/s    4.58 MB/s  621.78/s    11.12 MB/s  40
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   243.55/s    5.05 MB/s  432.33/s     9.95 MB/s  0
cus1dbi01  fcs4     0xC050760XXXXX0044  fcs1     0.94/s   10.42 KB/s   87.28/s   459.26 KB/s  0
...
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   1772.84/s  162.24 MB/s  1309.30/s   70.60 MB/s  68
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   1769.13/s  161.95 MB/s  1305.60/s   70.54 MB/s  68
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1   883.55/s   118.97 MB/s  1551.97/s  108.78 MB/s  43
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2   201.09/s    52.72 MB/s  497.26/s   130.35 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   201.09/s    52.72 MB/s  495.40/s   129.87 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.54/s   128.89 MB/s  1234.98/s   23.31 MB/s  65
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   876.93/s   118.93 MB/s  1234.98/s   23.32 MB/s  44
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   884.17/s   119.07 MB/s  1223.50/s   23.00 MB/s  43
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s     0.00 B/s   325.83/s    85.41 MB/s  0
...
^C
$

The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:

$ vios fcstat -t 10 ms15-vio1 2
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  1034.58/s   86.56 MB/s  2052.23/s  160.11 MB/s  20
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  1532.63/s  115.60 MB/s  1235.72/s  118.32 MB/s  40
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  1510.33/s  114.88 MB/s  1236.49/s  118.27 MB/s  40
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  1036.11/s   86.67 MB/s  1612.25/s   44.86 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  1031.50/s   86.29 MB/s  1588.02/s   44.27 MB/s  20
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  1029.58/s   86.31 MB/s  1567.63/s   43.65 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   436.52/s   114.43 MB/s  0
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2    0.00/s     0.00 B/s   435.75/s   114.23 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6    0.00/s     0.00 B/s   432.68/s   113.42 MB/s  0
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4  144.99/s     0.78 MB/s  478.83/s     2.22 MB/s  46
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  758.14/s   35.55 MB/s  1822.99/s  112.60 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  757.38/s   35.52 MB/s  1821.46/s  112.59 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  944.23/s   85.09 MB/s  1657.58/s   41.40 MB/s  2
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  943.47/s   85.15 MB/s  1636.90/s   40.68 MB/s  2
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  949.21/s   84.88 MB/s  1586.74/s   39.41 MB/s  2
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  946.53/s   84.64 MB/s  1584.83/s   39.40 MB/s  2
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   39.44/s  449.92 KB/s  676.97/s     3.63 MB/s  10
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   29.10/s  471.69 KB/s  310.92/s     1.28 MB/s  4
cus1mqp01  fcs4     0xC050760XXXXX002C  fcs0    1.91/s    4.71 KB/s  230.12/s     1.66 MB/s  0
cus2orap01 fcs4     0xC050760XXXXX000F  fcs0    0.77/s    4.31 KB/s   48.25/s   263.49 KB/s  0
^C
$

The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):

$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1858.72/s   51.14 MB/s  1231.82/s  104.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    6.94/s     1.82 MB/s    6.94/s     1.82 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.39/s     1.19 KB/s    0.39/s   395.05 B/s   0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.39/s     7.72 B/s     0.00/s     0.00 B/s   1
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1760.48/s  111.48 MB/s  1125.70/s   95.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    8.53/s     2.24 MB/s  484.61/s   127.04 MB/s  0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   469.04/s   122.96 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.37/s     1.14 KB/s    0.00/s     0.00 B/s   0
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
^C
$

With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.

If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.

The Impact of FC-Ports without a Link

FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).

(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)

Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):

$ lpar lsslot ms26-vio1
DRC_NAME                  DRC_INDEX  IOPOOL  DESCRIPTION
U78D3.001.XXXXXXX-P1-C49  21040015   none    PCIe3 x8 SAS RAID Internal Adapter 6Gb
U78D3.001.XXXXXXX-P1-C7   2103001C   none    PCIe3 4-Port 16Gb FC Adapter
U78D3.001.XXXXXXX-P1-C2   21010021   none    PCIe3 4-Port 16Gb FC Adapter
$
(HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)

However, only 2 ports of the 8 ports are cabled:

$ vios lsnports ms26-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78D3.001.XXXXXXX-P1-C2-T1  1       64      64      3072    3072
fcs4  U78D3.001.XXXXXXX-P1-C7-T1  1       64      64      3072    3072
$
(VIOS: lsnports)

When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000045

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m13.56s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m11.61s
user    0m0.01s
sys     0m0.00s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000045

real    0m11.31s
user    0m0.01s
sys     0m0.01s
(4)padmin@ms26-vio1:/home/padmin>

LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.

In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.

The following example shows how to reconfigure the fscsi1 device:

$ vios chdev ms26-vio1 fscsi1 autoconfig=defined
$
(VIOS: chdev -dev fscsi1 -attr autoconfig=defined)
$
$ vios rmdev ms26-vio1 fscsi1
$
(VIOS: rmdev -dev fscsi1 –ucfg)
$
$ vios lsdev ms26-vio1 fscsi1
NAME    STATUS   PHYSLOC                     PARENT  DESCRIPTION
fscsi1  Defined  U78D3.001.XXXXXXX-P1-C2-T2  fcs1    FC SCSI I/O Controller Protocol Device
$
(VIOS: lsdev -dev fscsi1)
$
$  vios lsattr ms26-vio1 fscsi1
ATTRIBUTE     VALUE      DESCRIPTION                            USER_SETTABLE
attach        none       How this adapter is CONNECTED          False
autoconfig    defined    Configuration State                    True
dyntrk        yes        Dynamic Tracking of FC Devices         True+
fc_err_recov  fast_fail  FC Fabric Event Error RECOVERY Policy  True+
scsi_id       Adapter    SCSI ID                                False
sw_fc_class   3          FC Class for Fabric                    True
$
(VIOS: lsdev -dev fscsi1 –attr)
$

With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!

If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m1.02s
user    0m0.04s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m9.70s
user    0m0.00s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.00s
user    0m0.02s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.

We now also set the autoconfig attribute to defined for all other unused FC ports:

$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7
> do
> vios chdev ms26-vio1 $fscsi autoconfig=defined
> vios rmdev ms26-vio1 $fscsi
> done
$

Now we repeat the runtime measurement of the commands again:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000005

Error opening device: /dev/fscsi3
errno: 00000005

Error opening device: /dev/fscsi5
errno: 00000005

Error opening device: /dev/fscsi6
errno: 00000005

Error opening device: /dev/fscsi7
errno: 00000005

real    0m0.81s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports         
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m0.00s
user    0m0.01s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1       

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.04s
user    0m0.00s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.

It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!