When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like
- how much throughput is achieved by a particular LPAR
- which LPARs have the highest throughput and produce the most FC traffic
- are there resource bottlenecks
could be answered.
Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:
(0)padmin@aixvio1:/home/padmin> fcstat -client hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs aixvio1 fcs0 0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576 0 0 0 C050760YYYYYYYYY 0 0 0 0 0 0 0 0 C050760ZZZZZZZZZ 0 0 0 0 0 0 0 0 aix01 fcs0 0xC050760XXXXXXXXX 22685402 101956075 10065757 699512617896 1572578056704 0 0 0 aix02 fcs0 0xC050760XXXXXXXXX 28200473 82295158 12051365 387847746448 626772151808 0 0 0 aix03 fcs0 0xC050760XXXXXXXXX 376500672 255163053 21583628 22619424512608 3786990844928 0 0 0 aix04 fcs0 0xC050760XXXXXXXXX 116450405 504688524 14020031 4037786527400 9929289617408 0 0 0 blbprodora22 fcs0 0xC050760XXXXXXXXX 1341092479 580673554 37458927 44288566807072 12166718497792 0 0 0 ... aixvio1 fcs1 0x100000XXXXXXXXXX 391131484 1090556094 156294130 71031615240217 87642294572864 0 0 0 aixtsm01 fcs2 0xC050760XXXXXXXXX 334020900 785597352 74659821 62072552942128 83284555980288 0 0 0 aixtsm02 fcs0 0xC050760XXXXXXXXX 2943054 40921231 11617552 107317697968 289142333440 0 0 0 aixvio1 fcs2 0x210000XXXXXXXXXX 403180246 5877180796 236998 105482699300998 1540608710446612 0 0 0 aixtsm01 fcs6 0xC050760XXXXXXXXX 146492419 392365162 74250 38378099796342 102844775468007 0 0 0 aixtsm02 fcs2 0xC050760XXXXXXXXX 19 192848 20 1090 50551063184 0 0 0 aixvio1 fcs3 0x210000XXXXXXXXXX 405673338 7371951499 260575 105969796271246 1932388891128304 0 0 0 aixtsm02 fcs3 0xC050760XXXXXXXXX 0 0 4 0 0 0 0 0 aix02 fcs7 0xC050760XXXXXXXXX 42624 2677470211 34211 2382280 701864613402184 0 0 0 ... Invalid initiator world wide name Invalid initiator world wide name (0)padmin@aixvio1:/home/padmin>
The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.
The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.
Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:
$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#
delta=5 # seconds
typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs
bc |& # start bc as coroutine
print -p "scale=2"
# get first sample
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta
while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs
printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print
sleep $delta
done
$
The script ‘npivstat‘ is available for download in our download-area.
Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):
aixvio1 # ./npivstat
hostname dev wwpn inreqs outreqs inbytes outbytes DMA_errs Elem_errs Comm_errs
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 1019.00 0.00 254.75 0.00 0.00 0.00
aixtsm01 fcs6 0xC0507605E5890074 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC0507609A6C70004 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC0507609A6C7001C 0.00 1018.20 0.00 254.55 0.00 0.00 0.00
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 1020.20 0.00 255.05 0.00 0.00 0.00
aixtsm01 fcs6 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC050760XXXXXXXXX 0.00 1019.80 0.00 254.95 0.00 0.00 0.00
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 984.80 0.00 246.20 0.00 0.00 0.00
aixtsm01 fcs6 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC050760XXXXXXXXX 0.00 985.00 0.00 246.25 0.00 0.00 0.00
...
^Caixvio1 #
In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.
The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.