Monitoring virtual FC Client Traffic

With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).

Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:

$ vios lsnports ms15-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78CB.001.XXXXXXX-P1-C5-T1  1       64      62      2032    2012
fcs1  U78CB.001.XXXXXXX-P1-C5-T2  1       64      62      2032    2012
fcs2  U78CB.001.XXXXXXX-P1-C5-T3  1       64      61      2032    1979
fcs3  U78CB.001.XXXXXXX-P1-C5-T4  1       64      61      2032    1979
fcs4  U78CB.001.XXXXXXX-P1-C3-T1  1       64      50      3088    3000
fcs5  U78CB.001.XXXXXXX-P1-C3-T2  1       64      63      3088    3077
$

We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:

$ vios fcstat ms15-vio1
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x210000XXXXX56EC5  fcs1   774.75/s  129.51 MB/s  1332.71/s   92.96 MB/s  20
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   318.10/s   83.39 MB/s  481.34/s   126.18 MB/s  0
ms15-vio1  fcs2     0x210000XXXXX56EC6  fcs2   318.10/s   83.39 MB/s  480.78/s   126.03 MB/s  0
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   583.98/s   60.35 MB/s  1835.17/s  124.86 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   583.70/s   60.27 MB/s  1836.21/s  124.92 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.19/s  165.08 MB/s  1032.81/s   17.25 MB/s  46
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   775.12/s  129.48 MB/s  1047.32/s   17.15 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   775.78/s  128.99 MB/s  1037.99/s   17.39 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s    0.00 B/s   290.39/s    76.12 MB/s  0
aixtsmp1   fcs0     0xC050760XXXXX0052  fcs4   142.89/s   36.12 MB/s    0.00/s     0.00 B/s   26
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   234.97/s    4.58 MB/s  621.78/s    11.12 MB/s  40
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   243.55/s    5.05 MB/s  432.33/s     9.95 MB/s  0
cus1dbi01  fcs4     0xC050760XXXXX0044  fcs1     0.94/s   10.42 KB/s   87.28/s   459.26 KB/s  0
...
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   1772.84/s  162.24 MB/s  1309.30/s   70.60 MB/s  68
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   1769.13/s  161.95 MB/s  1305.60/s   70.54 MB/s  68
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1   883.55/s   118.97 MB/s  1551.97/s  108.78 MB/s  43
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2   201.09/s    52.72 MB/s  497.26/s   130.35 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   201.09/s    52.72 MB/s  495.40/s   129.87 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.54/s   128.89 MB/s  1234.98/s   23.31 MB/s  65
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   876.93/s   118.93 MB/s  1234.98/s   23.32 MB/s  44
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   884.17/s   119.07 MB/s  1223.50/s   23.00 MB/s  43
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s     0.00 B/s   325.83/s    85.41 MB/s  0
...
^C
$

The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:

$ vios fcstat -t 10 ms15-vio1 2
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  1034.58/s   86.56 MB/s  2052.23/s  160.11 MB/s  20
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  1532.63/s  115.60 MB/s  1235.72/s  118.32 MB/s  40
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  1510.33/s  114.88 MB/s  1236.49/s  118.27 MB/s  40
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  1036.11/s   86.67 MB/s  1612.25/s   44.86 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  1031.50/s   86.29 MB/s  1588.02/s   44.27 MB/s  20
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  1029.58/s   86.31 MB/s  1567.63/s   43.65 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   436.52/s   114.43 MB/s  0
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2    0.00/s     0.00 B/s   435.75/s   114.23 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6    0.00/s     0.00 B/s   432.68/s   113.42 MB/s  0
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4  144.99/s     0.78 MB/s  478.83/s     2.22 MB/s  46
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  758.14/s   35.55 MB/s  1822.99/s  112.60 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  757.38/s   35.52 MB/s  1821.46/s  112.59 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  944.23/s   85.09 MB/s  1657.58/s   41.40 MB/s  2
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  943.47/s   85.15 MB/s  1636.90/s   40.68 MB/s  2
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  949.21/s   84.88 MB/s  1586.74/s   39.41 MB/s  2
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  946.53/s   84.64 MB/s  1584.83/s   39.40 MB/s  2
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   39.44/s  449.92 KB/s  676.97/s     3.63 MB/s  10
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   29.10/s  471.69 KB/s  310.92/s     1.28 MB/s  4
cus1mqp01  fcs4     0xC050760XXXXX002C  fcs0    1.91/s    4.71 KB/s  230.12/s     1.66 MB/s  0
cus2orap01 fcs4     0xC050760XXXXX000F  fcs0    0.77/s    4.31 KB/s   48.25/s   263.49 KB/s  0
^C
$

The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):

$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1858.72/s   51.14 MB/s  1231.82/s  104.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    6.94/s     1.82 MB/s    6.94/s     1.82 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.39/s     1.19 KB/s    0.39/s   395.05 B/s   0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.39/s     7.72 B/s     0.00/s     0.00 B/s   1
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1760.48/s  111.48 MB/s  1125.70/s   95.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    8.53/s     2.24 MB/s  484.61/s   127.04 MB/s  0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   469.04/s   122.96 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.37/s     1.14 KB/s    0.00/s     0.00 B/s   0
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
^C
$

With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.

If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.

The “label” Attribute for FC Adapters

As of AIX 7.2 TL4 and VIOS 3.1.1.10 there is a new attribute “label” for physical FC adapters. The administrator can set this attribute to any character string (maximum 255 characters). Even if the attribute is only informative, it can be extremely useful in PowerVM virtualization environments. If you have a large number of managed systems, it is not always clear to which FC fabric a certain FC port is connected. This can of course be looked up in the documentation of your systems, but it does involve a certain amount of effort. It is easier if you link this information directly with the FC adapters, which is exactly what the new “label” attribute allows in a simple way. On AIX:

# chdev -l fcs0 -U -a label="Fabric_1"
fcs0 changed
# lsattr -El fcs0 -a label -F value
Fabric_1
#

On virtual I/O servers, the attribute can also be set using the padmin account:

/home/padmin> chdev -dev fcs1 -attr label="Fabric_2" -perm
fcs1 changed
/home/padmin> lsdev -dev fcs1 -attr label                
value

Fabric_2
/home/padmin>

The attribute is also defined for older FC adapters.

If the “label” attribute is consistently used, it is always possible to determine online for each FC adapter to which fabric the adapter is connected to. This information only needs to be stored once for each FC adapter.

(Note: The “label” attribute is not implemented for AIX 7.1, at least not until 7.1 TL5 SP6.)

HSCF0180E Operation failed for

When trying to update the system firmware of a managed system via the HMC Command Line, we encountered the following error message:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027
HSCF0180E Operation failed for ms26 (9009-22A*XXXXXXX).
Could not unpack the firmware update package.
Check the health and available disk space of the file system.
hmc01:~>

The error message that was displayed suggested checking the available space in the HMC file systems:

hmc01:~> lshmcfs
filesystem=/var,filesystem_size=7935,filesystem_avail=4955,temp_files_start_time=11/22/2018 12:59:00,temp_files_size=2011
filesystem=/dump,filesystem_size=60347,filesystem_avail=55935,temp_files_start_time=02/15/2021 10:21:00,temp_files_size=0
filesystem=/extra,filesystem_size=20030,filesystem_avail=15939,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15615,filesystem_avail=4369,temp_files_start_time=02/15/2021 06:05:00,temp_files_size=4
hmc01:~>

Actually, the available space should be sufficient, but to be on the safe side, we’ve cleaned up a bit with the temporary files:

hmc01:~> chhmcfs -o f -d 5
hmc01:~>

The updlic command was unimpressed and returned the same error message.

The removal of some old firmware versions from the HMC‘s local disk repository was also unsuccessful:

hmc01:~> updlic -o p --ecnumber 01AL740
hmc01:~> updlic -o p --ecnumber 01AL770
hmc01:~> updlic -o p --ecnumber 01AM740
hmc01:~>

The error message was still the same. Obviously, contrary to the information in the error message, the problem had nothing to do with the available space on the HMC!

We then took a closer look at the downloaded firmware. We downloaded the firmware as an ISO file H75557812_01VL940_071_027.iso from the IBM website and then mounted it on our NIM server using the loopmount command:

aixnim:/root> loopmount -i /tmp/H75557812_01VL940_071_027.iso -o "-o ro -V cdrfs" -m /mnt
aixnim:/root> ls -l /mnt
total 528296
-rw-r--r--    1 102010979 213            1860 Feb 04 09:08 01VL940071_special_instructs.xml.special.note.xml
-rw-r-----    1 102010979 210            7290 Feb 04 09:08 01VL940_071_027.dd.xml
-rw-r--r--    1 102010979 213           95687 Feb 04 09:07 01VL940_071_027.html
-rw-r-----    1 102010979 210            2971 Feb 04 09:08 01VL940_071_027.pd.sdd
-rw-r-----    1 102010979 210           67338 Feb 04 09:08 01VL940_071_027.readme.txt
-rw-r-----    1 102010979 210       134969022 Feb 04 09:08 01VL940_071_027.rpm
-rw-r-----    1 102010979 210       135328848 Feb 04 09:08 01VL940_071_027.tar.gz
-rw-r-----    1 102010979 210            9442 Feb 04 09:08 01VL940_071_027.xml
aixnim:/root>

What we didn’t notice when copying the files was the lack of read permissions on other for most files.

After we had assigned read permissions for all files, the next update attempt was successful:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027 
HSCF0179W Operation was partially successful for ms26 (9009-22A*XXXXXXX).
The following deferred fixes are present in the fix pack.  Deferred fixes will be activated after the next IPL of the system.
An immediate IPL is not required, unless you want to activate one of the fixes below now.
..
hmc01:~>

LPAR-Tool 1.6.0.0 is available now

Version 1.6.0.0 of our LPAR tool is now available in our download area!

New features are:

  • Online monitoring of SEA client statistics (vios help seastat)
  • Online monitoring of virtual FC client adapters (vios help fcstat)
  • Display of historical processor and memory data (lpar help lsmem, lpar help lsproc)

In the article Monitoring SEA Traffic the possibilities of calling up SEA client statistics are shown.

The Impact of FC-Ports without a Link

FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).

(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)

Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):

$ lpar lsslot ms26-vio1
DRC_NAME                  DRC_INDEX  IOPOOL  DESCRIPTION
U78D3.001.XXXXXXX-P1-C49  21040015   none    PCIe3 x8 SAS RAID Internal Adapter 6Gb
U78D3.001.XXXXXXX-P1-C7   2103001C   none    PCIe3 4-Port 16Gb FC Adapter
U78D3.001.XXXXXXX-P1-C2   21010021   none    PCIe3 4-Port 16Gb FC Adapter
$
(HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)

However, only 2 ports of the 8 ports are cabled:

$ vios lsnports ms26-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78D3.001.XXXXXXX-P1-C2-T1  1       64      64      3072    3072
fcs4  U78D3.001.XXXXXXX-P1-C7-T1  1       64      64      3072    3072
$
(VIOS: lsnports)

When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000045

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m13.56s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m11.61s
user    0m0.01s
sys     0m0.00s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000045

real    0m11.31s
user    0m0.01s
sys     0m0.01s
(4)padmin@ms26-vio1:/home/padmin>

LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.

In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.

The following example shows how to reconfigure the fscsi1 device:

$ vios chdev ms26-vio1 fscsi1 autoconfig=defined
$
(VIOS: chdev -dev fscsi1 -attr autoconfig=defined)
$
$ vios rmdev ms26-vio1 fscsi1
$
(VIOS: rmdev -dev fscsi1 –ucfg)
$
$ vios lsdev ms26-vio1 fscsi1
NAME    STATUS   PHYSLOC                     PARENT  DESCRIPTION
fscsi1  Defined  U78D3.001.XXXXXXX-P1-C2-T2  fcs1    FC SCSI I/O Controller Protocol Device
$
(VIOS: lsdev -dev fscsi1)
$
$  vios lsattr ms26-vio1 fscsi1
ATTRIBUTE     VALUE      DESCRIPTION                            USER_SETTABLE
attach        none       How this adapter is CONNECTED          False
autoconfig    defined    Configuration State                    True
dyntrk        yes        Dynamic Tracking of FC Devices         True+
fc_err_recov  fast_fail  FC Fabric Event Error RECOVERY Policy  True+
scsi_id       Adapter    SCSI ID                                False
sw_fc_class   3          FC Class for Fabric                    True
$
(VIOS: lsdev -dev fscsi1 –attr)
$

With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!

If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m1.02s
user    0m0.04s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m9.70s
user    0m0.00s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.00s
user    0m0.02s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.

We now also set the autoconfig attribute to defined for all other unused FC ports:

$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7
> do
> vios chdev ms26-vio1 $fscsi autoconfig=defined
> vios rmdev ms26-vio1 $fscsi
> done
$

Now we repeat the runtime measurement of the commands again:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000005

Error opening device: /dev/fscsi3
errno: 00000005

Error opening device: /dev/fscsi5
errno: 00000005

Error opening device: /dev/fscsi6
errno: 00000005

Error opening device: /dev/fscsi7
errno: 00000005

real    0m0.81s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports         
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m0.00s
user    0m0.01s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1       

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.04s
user    0m0.00s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.

It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!

 

MDS reports at your fingertips

Many AIX and Power System administrators use Microcode Discovery Services to regularly check the versions of adapter firmware and system firmware. The following steps are usually necessary:

– Download the current catalog file catalog.mic.

– Run Inventory Scout to generate the microcode upload file.

– Upload the microcode upload file to IBM http://www14.software.ibm.com/support/customercare/mds/mds

In many cases, the upload is carried out via a browser. The report is shown in the form of an HTML output. Alternatively, you can also upload e.g. with the help of curl and request the data in JSON format.

$ curl -F "mdsData=@ms01-vio1.mup;type=multipart/form" -F "format=json" -H "Expect:" http://www14.software.ibm.com/support/customercare/mds/mds -o ms01-vio1.mup

The returned JSON file contains all information that is otherwise displayed in the browser.

With a small script, the JSON file can be displayed relatively easily in readable ASCII form. We have created the script mds_report for this purpose and made it available in our download area (https://powercampus.de/download). The script expects a microcode upload file as an argument, here is a sample output:

$ mds_report ms01-vio1.mup
ms01-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms01-vio1
IP Addr: X.X.X.X
Model: 8205-E6D   Serial: XXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           AL770_126        AL770_126        None        8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware
sissas0          0422003f         0422003f         None        PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E)
ent0,1,2,3       10080180         10240310         Update      4-Port Gigabit Ethernet PCI-Express Adapter
ent4,5,6,7       0400401800007    0400401800009    Update      PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter
fcs0,1,2,3       210301           210313           Update      PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729
fcs4,5,6,7       0320080270       0325080271       Update      8Gb PCIe2 Low Profile 4-Port FC Adapter
hdisk0,1         37343138         37343139         Update      Savvio 15K.3 146/300GB SAS Disk Drive
cd0              RA65             RA65             None        SATA DVD-RAM Drive RMBO0140512

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Security      SPE         2018.05.27    AL770_126        8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware
Usability     ATT         2013.06.06    0422003f         PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E)
Usability     ATT         2019.06.20    10240310         4-Port Gigabit Ethernet PCI-Express Adapter
Usability     ATT         2016.11.14    0400401800009    PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter
Usability     ATT         2019.06.17    210313           PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729
Usability     ATT         2020.01.28    0325080271       8Gb PCIe2 Low Profile 4-Port FC Adapter
Function      ATT         2019.04.30    37343139         Savvio 15K.3 146/300GB SAS Disk Drive
New           NEW         2014.10.24    RA65             SATA DVD-RAM Drive RMBO0140512
$

The output is very similar to the output in the browser. In the first section “Microcode by Host” the update recommendations for the system firmware and adapter firmware are given. In the second section “Microcode by TypeImpact and Severity, as well as the release date of the last available firmware version are shown.

If access to the Internet is only possible via a proxy, the proxy can be specified using the -x argument, as shown in the following example:

$ mds_report -x http://10.0.0.217:1234 ms07-vio1.mup
ms07-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms07-vio1
IP Addr: X.X.X.X
Model: 8408-44E   Serial: XXXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           SV860_138        SV860_215        Update      8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60
sissas0          15511800         19512900         Update      PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
ses0,1,2,3       1D0B             1D0B             None        SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E
pdisk0,1         37363135         37363142         Update      BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive
fcs0,1           00010000020025201919  00012000040025700015  Update      PCIe2 2-Port 16Gb FC Adapter
fcs2,3,4,5       0320080270       0325080271       Update      8Gb PCIe2 Low Profile 4-Port FC Adapter

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Security      HIPER       2020.03.04    SV860_215        8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60
Availability  ATT         2020.02.25    19512900         PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
New           NEW         2015.06.03    1D0B             SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E
Function      ATT         2020.04.16    37363142         BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive
Usability     ATT         2020.02.18    00012000040025700015  PCIe2 2-Port 16Gb FC Adapter
Usability     ATT         2020.01.28    0325080271       8Gb PCIe2 Low Profile 4-Port FC Adapter
$

If you want to use the script more often, you should enter the proxy in the script itself, for this there is the PROXY variable, which can be set as follows:

$ grep ^PROXY mds_report
PROXY="http://10.0.0.217:1234"
$

(Where 10.0.0.217:1234 is just an example, you have to supply your own values here.)

It is then no longer necessary to specify a proxy using the -x option.

If the script is executed as root on an AIX system, the proxy configuration is automatically adopted from ESA (Electronic Service Agent).

If you need the URLs to download the firmware, you should use the option -u (show download URLs). The links for the firmware versions are then displayed at the end of the output, here is an example:

$ mds_report -u ms03-vio1.mup
/appdata/daten/fk450/aix/mds/virt-aix23-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms03-vio1
IP Addr: X.X.X.X
Model: 9009-22A   Serial: XXXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           VL910_144        VL940_050        Update      9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system
sissas0          19511400         19512900         Update      PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
pdisk0           36383035         36383035         None        AL14SE 600/1200/1800 GB 4K Hard Disk Drive
pdisk1,2         41374B30         41374B30         None        Ultrastar C15K600-5xx
fcs0,1,2,3,4,5,6,7  00011000040041500005  00012000040025700015  Update      PCIe3 4-Port 16Gb FC Adapter

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Availability  SPE         2020.05.21    VL940_050        9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system
Availability  ATT         2020.02.25    19512900         PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
Data          HIPER       2016.12.01    36383035         AL14SE 600/1200/1800 GB 4K Hard Disk Drive
Function      ATT         2015.08.18    41374B30         Ultrastar C15K600-5xx
Usability     ATT         2020.02.18    00012000040025700015  PCIe3 4-Port 16Gb FC Adapter

Downloads

http://www.ibm.com/support/fixcentral/quickorder?product=ibm/power/900922A&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=01VL940_050_027
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=40145679_20200224110413_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1354333840_20161130155709_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1448849004_20150813164908_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=427029183_20200213134040_GRP
$

The script generally takes less than 1 second to run!

We tested the script on AIX, Linux, and MacOS. Under MacOS there is usually no ksh93. But the installed ksh supports all the necessary features that are required by the mds_report script. If you change the interpreter in the first line of the script to ksh, the script will also run on a Mac.

A good description of Inventory Scout and MDS can be found here: http://gibsonnet.net/blog/cgaix/html/MDS%20reports.html (Chris Gibson)

You can find out how to automate Inventory Scout in our article Automating Inventory Scout

 

LPAR-Tool: Which LPARs have no RMC-Connection

Status and configuration of LPARs are regularly needed information in the administration of LPARs. With the LPAR tool, information such as status, RMC status, number of cores, size of RAM, OS version and other data can be easily and quickly determined, even with hundreds or thousands of LPARs. Which LPARs don’t have an RMC connection is shown as one of the examples.

All of the following examples were performed on an environment with 10 HMCs, 50 managed systems, and just over 500 LPARs. To determine how long the LPAR tool requires, the run times of the commands were measured and specified using time.

The names of the LPARs were manually changed in the outputs shown and replaced by generic names lparXX and aixYY.

First of all, the status of a single LPAR:

$ time lpar status aix01
NAME   LPAR_ID  LPAR_ENV   STATE     PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01  27       aixlinux   Running   standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614

real    0m0.210s
user    0m0.011s
sys     0m0.013s
$

Of course you can also specify multiple LPARs. If you want to know the status of all LPARs (in our case just over 500 LPARs), just leave out the argument:

$ time lpar status
NAME    LPAR_ID  LPAR_ENV   STATE     PROFILE      SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01   27       aixlinux   Running   standard     0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02   1        aixlinux   Running   standard     -     -         1      -           8320    Unknown
...
lpar01  6        aixlinux   Running   standard     0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m18.933s
user	0m3.819s
sys	0m3.789s
$

In the background, the LPAR tool executes more than 150 commands on the corresponding HMCs (lshwres and lssyscfg)!

The output should now be restricted to LPARs that are currently active (state=Running). There is the option “-s“, which can be used to specify criteria for attributes that must be met. Only LPARs meeting these criteria will be shown:

$ time lpar status -s state=Running
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01    27       aixlinux   Running  standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar01   6        aixlinux   Running  standard   0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m17.998s
user	0m3.692s
sys	0m3.647s
$

Now we want to know on which of these LPARs RMC is not working/not active. The option “-s” allows to combine any number of criteria. All specified criteria must then be met (logical AND). The RMC state can be found in the attribute rmc_state:

$ time lpar status -s state=Running,rmc_state!=active
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
aix03    2        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar07   4        aixlinux   Running  standard   0     none      1      1.0         4352    Unknown

real	0m19.057s
user	0m3.550s
sys	0m3.512s
$

As another example we want to know on which LPARs AIX 7.1 TL5 is installed. The os_version attribute contains the OS version. The ‘~‘ operator can be used to compare against a regular expression (similar to the grep command). We use the regular expression 7100-05:

$ time lpar status -s os_version~7100-05
NAME     LPAR_ID  LPAR_ENV  STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix14    14       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-02-1810
aix16    24       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-03-1846
...
lpar10   10       aixlinux  Running  standard   0     active    3      0.3         32768   AIX 7.1 7100-05-02-1810

real	0m18.212s
user	0m3.726s
sys	0m3.676s
$

So far, we have always used the default output format. Now we’d like to list all systems that still run on AIX 6.1, but this time only the LPAR name and the OS version will be output. For this there is the option “-F“, with which the desired output fields can be specified:

$ time lpar status -s os_version~6100 -F name:os_version
aix39:AIX 6.1 6100-07-04-1216
aix46:AIX 6.1 6100-07-04-1216
...
lpar35:AIX 6.1 6100-09-05-1524

real	0m18.041s
user	0m3.619s
sys	0m3.699s
$

If you prefer JSON output, you can easily do that with the option “-j“, here’s the same example with JSON output:

$ time lpar status -s os_version~6100 -F name:os_version -j
{
	"name": "aix39",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
{
	"name": "aix46",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
...
{
	"name": "lpar35",
	"os_version": "AIX 6.1 6100-09-05-1524"
}

real	0m21.247s
user	0m3.670s
sys	0m3.720s
$

Of course you can not know all attribute names by heart! But that’s not necessary, because you can easily see all attribute names. Use the option “-f” (stanza format) and specify any LPAR:

$ lpar status -f lpar19
lpar19:
	curr_lpar_proc_compat_mode = POWER7
	curr_mem = 8192
	curr_proc_mode = shared
	curr_proc_units = 0.3
	curr_procs = 2
	name = lpar19
	os_version = AIX 6.1 6100-09-05-1524
...
$

With the options “-h” and “-m” the LPARs can be selected depending on the associated HMC and/or managed system.

Status of all LPARs with associated HMC hmc01:

$ lpar -h hmc01 status

Status of all LPARs whose corresponding HMC has type 7042-CR6:

$ lpar -h 7042-CR6 status

Status of all LPARs whose associated HMC has type 7042-CR6 and whose name begins with lpar:

$ lpar -h 7042-CR6 status lpar*

Status of all LPARs on the managed system ms13:

$ lpar -m ms13 status

Status of all LPARs whose managed system is an S922:

$ lpar -m 9009-22A status

The presented selection and output options apply to all output commands of the LPAR tool (except the vios command).

The LPAR-Tool can be downloaded from our download area: https://powercampus.de/en/download-2

The LPAR-Tool contains a test license which is valid until the end of october.

FC NPIV client throughput

When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like

  • how much throughput is achieved by a particular LPAR
  • which LPARs have the highest throughput and produce the most FC traffic
  • are there resource bottlenecks

could be answered.

Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:

(0)padmin@aixvio1:/home/padmin> fcstat -client
              hostname   dev                wwpn     inreqs    outreqs ctrlreqs          inbytes         outbytes  DMA_errs Elem_errs Comm_errs

               aixvio1  fcs0  0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576         0         0         0
     C050760YYYYYYYYY
                                    0          0        0                0                0         0         0         0
     C050760ZZZZZZZZZ
                                    0          0        0                0                0         0         0         0
                 aix01  fcs0  0xC050760XXXXXXXXX   22685402  101956075 10065757     699512617896    1572578056704         0         0         0
                 aix02  fcs0  0xC050760XXXXXXXXX   28200473   82295158 12051365     387847746448     626772151808         0         0         0
                 aix03  fcs0  0xC050760XXXXXXXXX  376500672  255163053 21583628   22619424512608    3786990844928         0         0         0
                 aix04  fcs0  0xC050760XXXXXXXXX  116450405  504688524 14020031    4037786527400    9929289617408         0         0         0
          blbprodora22  fcs0  0xC050760XXXXXXXXX 1341092479  580673554 37458927   44288566807072   12166718497792         0         0         0
...
               aixvio1  fcs1  0x100000XXXXXXXXXX  391131484 1090556094 156294130   71031615240217   87642294572864         0         0         0
              aixtsm01  fcs2  0xC050760XXXXXXXXX  334020900  785597352 74659821   62072552942128   83284555980288         0         0         0
              aixtsm02  fcs0  0xC050760XXXXXXXXX    2943054   40921231 11617552     107317697968     289142333440         0         0         0

               aixvio1  fcs2  0x210000XXXXXXXXXX  403180246 5877180796   236998  105482699300998 1540608710446612         0         0         0
              aixtsm01  fcs6  0xC050760XXXXXXXXX  146492419  392365162    74250   38378099796342  102844775468007         0         0         0
              aixtsm02  fcs2  0xC050760XXXXXXXXX         19     192848       20             1090      50551063184         0         0         0

               aixvio1  fcs3  0x210000XXXXXXXXXX  405673338 7371951499   260575  105969796271246 1932388891128304         0         0         0
              aixtsm02  fcs3  0xC050760XXXXXXXXX          0          0        4                0                0         0         0         0
                 aix02  fcs7  0xC050760XXXXXXXXX      42624 2677470211    34211          2382280  701864613402184         0         0         0
...
Invalid initiator world wide name
Invalid initiator world wide name
(0)padmin@aixvio1:/home/padmin>

The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.

The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.

Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:

$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#

delta=5 # seconds

typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs

bc |& # start bc as coroutine
print -p "scale=2"

# get first sample

/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta

while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs

print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs

printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print

sleep $delta
done

$

The script ‘npivstat‘ is available for download in our download-area.

Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):

aixvio1 # ./npivstat
       hostname    dev              wwpn  inreqs  outreqs  inbytes  outbytes  DMA_errs  Elem_errs  Comm_errs
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1019.00     0.00    254.75      0.00       0.00       0.00
       aixtsm01   fcs6  0xC0507605E5890074    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC0507609A6C70004    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC0507609A6C7001C    0.00  1018.20     0.00    254.55      0.00       0.00       0.00
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1020.20     0.00    255.05      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00  1019.80     0.00    254.95      0.00       0.00       0.00
...                                                                                                           
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00   984.80     0.00    246.20      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00   985.00     0.00    246.25      0.00       0.00       0.00
...
^Caixvio1 # 

In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.

The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.