Tag: fcs

Posted on

Virtual FC Adapter and NPIV

Physical FC port with Virtual FC and NPIV

One possibility for the virtualization of storage under PowerVM is the use of virtual FC adapters. A virtual FC client adapter is connected to a virtual FC server adapter on a virtual I/O server via the POWER Hypervisor, as shown in figure 7.10. On the virtual I/O server, the virtual FC server adapter is then connected to one of the physical FC ports (mapping). Each of the connected virtual FC server adapters can log into the FC fabric of its own. Each virtual FC server adapter is assigned its own 24-bit FC address.

Communication path of the virtual FC client adapter to the SAN LUN.
Figure 7.10: Communication path of the virtual FC client adapter to the SAN LUN.

The advantage of virtual FC is that each virtual FC client adapter has its own N_Port and can therefore communicate directly with the storage in the FC fabric. The storage LUNs can be assigned directly to the virtual FC client adapter, without having to map each LUN individually on the virtual I/O server. The virtual I/O server itself, normally does not see the storage LUNs of the virtual FC clients. This makes administration much easier than with virtual SCSI, where each storage LUN has to be mapped on the virtual I/O server to a virtual SCSI server adapter (see next chapter).

Before a virtual FC adapter is created and mapped, the situation on a virtual I/O server is as shown in figure 7.11. The physical FC port is connected to a FC fabric and therefore configures an N_Port. The physical FC port logs into the fabric (FLOGI) and is assigned the unique N_Port ID 8c8240. The FC port then registers its WWPN (here 10:00:00:10:9b:ab:01:02) with the simple name server (SNS) of the fabric (PLOGI). The virtual I/O server can then communicate with other N_Ports in the fabric using the fcs0 device.

Physical FC port without virtual FC and NPIV
Figure 7.11: Physical FC Port without Virtual FC and NPIV

N_Port-ID virtualization or NPIV for short is an extension of the FC standard and allows more than one N_Port to log into the fabric using the same physical FC port. In principle, this option has always existed, but only in connection with FC Arbitrated Loop (FC-AL) and fabrics. With NPIV, multiple client LPARs can share a physical FC port. Each client has its own unique N_Port.

Figure 7.12 shows the situation with 2 virtual FC client adapters. Each of the client adapters has a unique WWPN. The WWPN is assigned by PowerVM when the virtual FC client adapter is created (in order to be able to support live partition mobility, 2 WWPNs are always assigned, whereby only one of the two WWPNs is active). Each virtual FC client adapter requires a partner adapter on a virtual I/O server, the virtual FC server adapter (or vfchost). One of the physical FC ports must be assigned to the virtual FC server adapter on the virtual I/O server. If the client LPAR is active, the virtual FC server adapter logs into the fabric (FDISC) and is assigned a unique N_Port ID. In the figure it is the 8c8268 for the blue adapter and the 8c8262 for the red adapter. Then the blue adapter registers its client WWPN (here c0:50:76:07:12:cd:00:16) with the simple name server (SNS) of the fabric (PLOGI). The red adapter does the same for its client WWPN (here c0:50:76:07:12:cd:00:09). Both virtual FC client adapters then have an N_Port with a unique 24-bit ID and can thus communicate with other N_Ports in the fabric.

Physical FC port with Virtual FC and NPIV
Figure 7.12: Physical FC port with Virtual FC and NPIV

The data is of course not copied between the virtual FC client adapter and the virtual FC server adapter by the hypervisor, as this would cost too much performance. The hypervisor only forwards the physical memory address at which the data is located and the physical FC port then uses DMA (Direct Memory Access) to access this data.

7.4.1. NPIV capable FC Adapter

7.4.2. Adding a Virtual FC Adapter

7.4.3. Assigning a Physical FC Port (Mapping)

7.4.4. Adding LUNs

7.4.5. Adding a Virtual FC Adapter with Mapping

7.4.6. Changing Attributes of a Virtual FC Adapter

7.4.7. Removing an NPIV Mapping

7.4.8. Changing an NPIV Mapping

7.4.9. Removing a Virtual FC Adapter

7.4.10. Assigning Predefined WWPNs

Posted on

Monitoring virtual FC Client Traffic

With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).

Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:

$ vios lsnports ms15-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78CB.001.XXXXXXX-P1-C5-T1  1       64      62      2032    2012
fcs1  U78CB.001.XXXXXXX-P1-C5-T2  1       64      62      2032    2012
fcs2  U78CB.001.XXXXXXX-P1-C5-T3  1       64      61      2032    1979
fcs3  U78CB.001.XXXXXXX-P1-C5-T4  1       64      61      2032    1979
fcs4  U78CB.001.XXXXXXX-P1-C3-T1  1       64      50      3088    3000
fcs5  U78CB.001.XXXXXXX-P1-C3-T2  1       64      63      3088    3077
$

We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:

$ vios fcstat ms15-vio1
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x210000XXXXX56EC5  fcs1   774.75/s  129.51 MB/s  1332.71/s   92.96 MB/s  20
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   318.10/s   83.39 MB/s  481.34/s   126.18 MB/s  0
ms15-vio1  fcs2     0x210000XXXXX56EC6  fcs2   318.10/s   83.39 MB/s  480.78/s   126.03 MB/s  0
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   583.98/s   60.35 MB/s  1835.17/s  124.86 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   583.70/s   60.27 MB/s  1836.21/s  124.92 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.19/s  165.08 MB/s  1032.81/s   17.25 MB/s  46
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   775.12/s  129.48 MB/s  1047.32/s   17.15 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   775.78/s  128.99 MB/s  1037.99/s   17.39 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s    0.00 B/s   290.39/s    76.12 MB/s  0
aixtsmp1   fcs0     0xC050760XXXXX0052  fcs4   142.89/s   36.12 MB/s    0.00/s     0.00 B/s   26
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   234.97/s    4.58 MB/s  621.78/s    11.12 MB/s  40
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   243.55/s    5.05 MB/s  432.33/s     9.95 MB/s  0
cus1dbi01  fcs4     0xC050760XXXXX0044  fcs1     0.94/s   10.42 KB/s   87.28/s   459.26 KB/s  0
...
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   1772.84/s  162.24 MB/s  1309.30/s   70.60 MB/s  68
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   1769.13/s  161.95 MB/s  1305.60/s   70.54 MB/s  68
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1   883.55/s   118.97 MB/s  1551.97/s  108.78 MB/s  43
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2   201.09/s    52.72 MB/s  497.26/s   130.35 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   201.09/s    52.72 MB/s  495.40/s   129.87 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.54/s   128.89 MB/s  1234.98/s   23.31 MB/s  65
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   876.93/s   118.93 MB/s  1234.98/s   23.32 MB/s  44
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   884.17/s   119.07 MB/s  1223.50/s   23.00 MB/s  43
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s     0.00 B/s   325.83/s    85.41 MB/s  0
...
^C
$

The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:

$ vios fcstat -t 10 ms15-vio1 2
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  1034.58/s   86.56 MB/s  2052.23/s  160.11 MB/s  20
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  1532.63/s  115.60 MB/s  1235.72/s  118.32 MB/s  40
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  1510.33/s  114.88 MB/s  1236.49/s  118.27 MB/s  40
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  1036.11/s   86.67 MB/s  1612.25/s   44.86 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  1031.50/s   86.29 MB/s  1588.02/s   44.27 MB/s  20
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  1029.58/s   86.31 MB/s  1567.63/s   43.65 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   436.52/s   114.43 MB/s  0
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2    0.00/s     0.00 B/s   435.75/s   114.23 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6    0.00/s     0.00 B/s   432.68/s   113.42 MB/s  0
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4  144.99/s     0.78 MB/s  478.83/s     2.22 MB/s  46
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  758.14/s   35.55 MB/s  1822.99/s  112.60 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  757.38/s   35.52 MB/s  1821.46/s  112.59 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  944.23/s   85.09 MB/s  1657.58/s   41.40 MB/s  2
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  943.47/s   85.15 MB/s  1636.90/s   40.68 MB/s  2
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  949.21/s   84.88 MB/s  1586.74/s   39.41 MB/s  2
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  946.53/s   84.64 MB/s  1584.83/s   39.40 MB/s  2
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   39.44/s  449.92 KB/s  676.97/s     3.63 MB/s  10
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   29.10/s  471.69 KB/s  310.92/s     1.28 MB/s  4
cus1mqp01  fcs4     0xC050760XXXXX002C  fcs0    1.91/s    4.71 KB/s  230.12/s     1.66 MB/s  0
cus2orap01 fcs4     0xC050760XXXXX000F  fcs0    0.77/s    4.31 KB/s   48.25/s   263.49 KB/s  0
^C
$

The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):

$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1858.72/s   51.14 MB/s  1231.82/s  104.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    6.94/s     1.82 MB/s    6.94/s     1.82 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.39/s     1.19 KB/s    0.39/s   395.05 B/s   0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.39/s     7.72 B/s     0.00/s     0.00 B/s   1
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1760.48/s  111.48 MB/s  1125.70/s   95.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    8.53/s     2.24 MB/s  484.61/s   127.04 MB/s  0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   469.04/s   122.96 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.37/s     1.14 KB/s    0.00/s     0.00 B/s   0
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
^C
$

With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.

If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.

Posted on

The Impact of FC-Ports without a Link

FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).

(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)

Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):

$ lpar lsslot ms26-vio1
DRC_NAME                  DRC_INDEX  IOPOOL  DESCRIPTION
U78D3.001.XXXXXXX-P1-C49  21040015   none    PCIe3 x8 SAS RAID Internal Adapter 6Gb
U78D3.001.XXXXXXX-P1-C7   2103001C   none    PCIe3 4-Port 16Gb FC Adapter
U78D3.001.XXXXXXX-P1-C2   21010021   none    PCIe3 4-Port 16Gb FC Adapter
$
(HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)

However, only 2 ports of the 8 ports are cabled:

$ vios lsnports ms26-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78D3.001.XXXXXXX-P1-C2-T1  1       64      64      3072    3072
fcs4  U78D3.001.XXXXXXX-P1-C7-T1  1       64      64      3072    3072
$
(VIOS: lsnports)

When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000045

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m13.56s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m11.61s
user    0m0.01s
sys     0m0.00s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000045

real    0m11.31s
user    0m0.01s
sys     0m0.01s
(4)padmin@ms26-vio1:/home/padmin>

LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.

In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.

The following example shows how to reconfigure the fscsi1 device:

$ vios chdev ms26-vio1 fscsi1 autoconfig=defined
$
(VIOS: chdev -dev fscsi1 -attr autoconfig=defined)
$
$ vios rmdev ms26-vio1 fscsi1
$
(VIOS: rmdev -dev fscsi1 –ucfg)
$
$ vios lsdev ms26-vio1 fscsi1
NAME    STATUS   PHYSLOC                     PARENT  DESCRIPTION
fscsi1  Defined  U78D3.001.XXXXXXX-P1-C2-T2  fcs1    FC SCSI I/O Controller Protocol Device
$
(VIOS: lsdev -dev fscsi1)
$
$  vios lsattr ms26-vio1 fscsi1
ATTRIBUTE     VALUE      DESCRIPTION                            USER_SETTABLE
attach        none       How this adapter is CONNECTED          False
autoconfig    defined    Configuration State                    True
dyntrk        yes        Dynamic Tracking of FC Devices         True+
fc_err_recov  fast_fail  FC Fabric Event Error RECOVERY Policy  True+
scsi_id       Adapter    SCSI ID                                False
sw_fc_class   3          FC Class for Fabric                    True
$
(VIOS: lsdev -dev fscsi1 –attr)
$

With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!

If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m1.02s
user    0m0.04s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m9.70s
user    0m0.00s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.00s
user    0m0.02s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.

We now also set the autoconfig attribute to defined for all other unused FC ports:

$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7
> do
> vios chdev ms26-vio1 $fscsi autoconfig=defined
> vios rmdev ms26-vio1 $fscsi
> done
$

Now we repeat the runtime measurement of the commands again:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000005

Error opening device: /dev/fscsi3
errno: 00000005

Error opening device: /dev/fscsi5
errno: 00000005

Error opening device: /dev/fscsi6
errno: 00000005

Error opening device: /dev/fscsi7
errno: 00000005

real    0m0.81s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports         
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m0.00s
user    0m0.01s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1       

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.04s
user    0m0.00s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.

It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!