In the simplest case, a SEA (Shared Ethernet Adapter) can consist of just one trunking adapter. A SEA can have up to 16 trunking adapters, whereby each of the trunking adapters can have up to 20 additional VLANs in addition to the port VLAN ID.
CVE-2021-25220: AIX is vulnerable to cache poisoning due to ISC BIND
CVE-2021-25220 describes a vulnerability in ISC BIND. Using our tool “apar“, some questions are examined and answered below, such as: is my system affected by this vulnerability, where can I find a more detailed description of the vulnerability, where can I find a fix to close the vulnerability, are there other vulnerabilities of which my system is affected?
Note: The “apar” tool is available in our download area in versions for AIX (VIOS), Linux and MacOS. It includes a time-limited trial license. See the Manage and Access APARs for more information on using the tool.
Is my system affected by this vulnerability?
Information about the vulnerability can be displayed using the “apar show” command and the “CVE-2021-25220” argument:
$ apar show CVE-2021-25220 type: sec product: aix versions: 7300-00-01,7300-00-02 abstract: AIX is vulnerable to cache poisoning due to ISC BIND apars: CVE-2021-25220,IJ40614 fixedIn: 7300-00-03 ifixes: IJ40614m2b.220718.epkg.Z bulletinUrl: https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc filesets: bos.net.tcp.bind:7.3.0.0-7.3.0.1,bos.net.tcp.bind_utils:7.3.0.0-7.3.0.1 issued: 20220728 updated: siblings: download: https://aix.software.ibm.com/aix/efixes/security/bind_fix21.tar cvss: CVE-2021-25220:6.8 reboot: no … $
Multiple records are displayed. There are separate records for different AIX and VIOS versions. Each record contains a line with the associated AIX or VIOS versions (line “versions: …”). In addition, the affected filesets are listed, including the version (line “filesets: …”). If, for example, AIX 7300-00-01 or 7300-00-02 is installed on my system (command “oslevel –s”) and I have one of the fileset versions listed (command “lslpp –l bos.net.tcp.bind bos.net .tcp.bind_utils“), then my system is affected by the vulnerability.
Where can I find a more detailed description of the vulnerability?
IBM typically offers more detailed information about a vulnerability via a so-called bulletin. The URL for the bulletin is shown in the output of “apar show” (above) on the line beginning with “bulletinUrl: …”. In the case above, this is https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc . This URL can be specified in a browser. When using the “apar” command, the bulletin can also be displayed directly on the command line, this can be done with the command “apar bulletin” and the number of the CVE (here CVE-2021-25220) or the fix or APAR number ( e.g. IJ40614):
$ apar bulletin CVE-2021-25220 IBM SECURITY ADVISORY First Issued: Thu Jul 28 13:24:22 CDT 2022 The most recent version of this document is available here: http://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc ftp://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc Security Bulletin: AIX is vulnerable to cache poisoning due to ISC BIND (CVE-2021-25220) =============================================================================== SUMMARY: A vulnerability in ISC BIND could allow a remote attacker to poison the cache (CVE-2021-25220). AIX uses ISC BIND as part of its DNS functions. =============================================================================== VULNERABILITY DETAILS: CVEID: CVE-2021-25220 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25220 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25220 DESCRIPTION: ISC BIND could allow a remote attacker to bypass security restrictions, caused by an error when using DNS forwarders. An attacker could exploit this vulnerability to poison the cache with incorrect records leading to queries being made to the wrong servers, which might also result in false information being returned to clients. CVSS Base Score: 6.8 CVSS Temporal Score: See https://exchange.xforce.ibmcloud.com/vulnerabilities/221991 for the current score CVSS Environmental Score*: Undefined CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:C/C:N/I:H/A:N) … $
All associated APARs are usually listed in the bulletin. You will also find an overview of the fixes and corresponding versions.
Where can I find a fix to close the vulnerability?
In the records above, you will also find a listing of the associated fixes in the line beginning with “ifixes: …”. In the case mentioned, this is the fix IJ40614m2b.220718.epkg.Z. In many cases, several fixes are listed and you have to select the correct fix from the list. The description in the bulletin is helpful here, with a list of which fix is to be used for which version.
The URL for downloading the fix(s) is given in the line beginning with “download: …”, in the current case this is the following URL:
https://aix.software.ibm.com/aix/efixes/security/bind_fix21.tar
The fix can be downloaded with a browser, for example. When using the “apar” command, however, this is even easier using the command line. The “apar” command can be invoked with the argument “download” and the CVE number or fix number. Then it downloads the fix and stores it in the current working directory:
$ apar download CVE-2021-25220 downloading bind_fix21.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19.1M 100 19.1M 0 0 1480k 0 0:00:13 0:00:13 --:--:-- 1672k $
The fix is saved under the name used in the URL, here bind_fix21.tar.
Are there other vulnerabilities affecting my system?
The command “apar check” can be used to examine a system for known vulnerabilities. In order for the command to be able to access the information about installed fixes, the command must be started with root privileges.
Here is an example of a system with all relevant fixes installed:
aix01 # apar check SUMMARY: 2/2 fixes installed aix01 #
And below is an example of a system with only a few relevant fixes installed:
aix02 # apar check SUMMARY: 4/8 fixes installed (2 APARs have no fix specified) aix02 #
Of the 8 known (IBM disclosed) vulnerabilities, only 4 of the vulnerabilities have the associated fixes installed. If you want to know which vulnerabilities are open, one of the options “-b” (brief report) or “-l” (long report) can be used:
aix02 # apar check -b 20220817 sec aix CVE-2022-1292,CVE-2022-2068,CVE-2022-2097 AIX is vulnerable to arbitrary command execution due to OpenSSL INSTALLED: no fix installed 20220912 sec aix CVE-2022-36768 AIX is vulnerable to a privilege escalation vulnerability due to invscout INSTALLED: no fix installed 20220923 sec aix CVE-2021-20266,CVE-2021-20271,CVE-2021-3421 AIX is vulnerable to arbitrary code execution and RPM database corruption and denial of service due to RPM. INSTALLED: no fix installed 20220928 sec aix CVE-2018-25032 AIX is vulnerable to denial of service due to zlib and zlibNX INSTALLED: no fix installed SUMMARY: 4/8 fixes installed (2 APARs have no fix specified) aix02 #
The “apar” command supports further options, which are described in Manage and Access APARs.
Manage and Access APARs
Keeping AIX and virtual I/O servers up to date with regard to HIPER and SECURITY fixes has become enormously important in recent years. To do this, the systems must be checked regularly for any missing fixes. The appropriate fixes must be downloaded and then installed. Determining which fix needs to be installed on a particular system often involves viewing bulletins with a web browser. PowerCampus 01 provides the ‘apar‘ command to simplify the management of fixes. This makes working with fixes and APARs as well as CVEs much easier.
Some sample uses of the ‘apar‘ command
The ‘apar‘ command allows the download of HIPER and SECURITY fixes, the checking of systems (AIX and VIOS) for installed and missing fixes, as well as the display and targeted search for fixes. In order to be able to use all functionalities, a direct Internet connection or a connection via an http proxy server is required. The command is available in versions for AIX, Linux and MacOS. A number of example calls are shown below.
Example 1: What fixes have been released in the last 30 days?
$ apar last 20220817 sec aix CVE-2022-1292,CVE-2022-2068,CVE-2022-2097 AIX is vulnerable to arbitrary command execution due to OpenSSL 20220912 sec vios CVE-2022-29824,IJ42339,IJ42378,IJ42379 AIX is vulnerable to a denial of service due to libxml2 for VIOS 20220912 sec vios CVE-2022-29824,IJ42339,IJ42378,IJ42379 AIX is vulnerable to a denial of service due to libxml2 for VIOS 20220912 sec aix CVE-2022-29824,IJ42339,IJ42378,IJ42379 AIX is vulnerable to a denial of service due to libxml2 20220912 sec aix CVE-2022-29824,IJ42341 AIX is vulnerable to a denial of service due to libxml2 20220912 sec aix CVE-2022-29824,IJ42381 AIX is vulnerable to a denial of service due to libxml2 20220912 sec vios CVE-2022-29824,IJ42381 AIX is vulnerable to a denial of service due to libxml2 for VIOS 20220912 sec vios CVE-2022-34356,IJ41396,IJ41685,IJ41795 AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS 20220912 sec aix CVE-2022-34356,IJ41396,IJ41685,IJ41795 AIX kernel is vulnerable to a privilege escalation vulnerability 20220912 sec vios CVE-2022-34356,IJ41396,IJ41685,IJ41795 AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS 20220912 sec aix CVE-2022-34356,IJ41687 AIX kernel is vulnerable to a privilege escalation vulnerability 20220912 sec aix CVE-2022-34356,IJ41688 AIX kernel is vulnerable to a privilege escalation vulnerability 20220912 sec vios CVE-2022-34356,IJ41706 AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS 20220912 sec aix CVE-2022-34356,IJ41706 AIX kernel is vulnerable to a privilege escalation vulnerability 20220912 sec aix CVE-2022-36768 AIX is vulnerable to a privilege escalation vulnerability due to invscout $
Example 2: Displaying information about APAR ID IJ42341.
$ apar show IJ42341 type: sec product: aix versions: 7300-00-01,7300-00-02 abstract: AIX is vulnerable to a denial of service due to libxml2 apars: CVE-2022-29824,IJ42341 fixedIn: 7300-00-04 ifixes: IJ42341s2a.220907.epkg.Z bulletinUrl: https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc filesets: bos.rte.control:7.3.0.0-7.3.0.1 issued: 20220912 updated: siblings: download: https://aix.software.ibm.com/aix/efixes/security/libxml2_fix3.tar cvss: CVE-2022-29824:5.5 reboot: no $
Example 3: Viewing the bulletin for APAR ID IJ42341.
$ apar bulletin IJ42341
IBM SECURITY ADVISORY
First Issued: Mon Sep 12 15:07:01 CDT 2022
The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
ftp://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
Security Bulletin: AIX is vulnerable to a denial of service due to libxml2
(CVE-2022-29824)
…
REMEDIATION:
A. APARS
IBM has assigned the following APARs to this problem:
AIX Level APAR Availability SP KEY
-----------------------------------------------------
7.2.4 IJ42381 ** N/A key_w_apar
7.2.5 IJ42339 ** SP06 key_w_apar
7.3.0 IJ42341 ** SP04 key_w_apar
…
$
Example 4: Download the fix for APAR IJ42341.
$ apar download IJ42341 downloading libxml2_fix3.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 30.8M 100 30.8M 0 0 1522k 0 0:00:20 0:00:20 --:--:-- 1638k $
The fix is downloaded to the current working directory.
Example 5: Searching for fixes for the keywords ‘memory‘ and ‘leak‘.
$ apar search memory leak 20141029 CVE-2014-3513,CVE-2014-3566,CVE-2014-3567 AIX OpenSSL Denial of Service due to memory leak in DTLS / AIX OpenSSL Patch to mitigate CVE-2014-3566 / AIX OpenSSL Denial of Service due to memory consumption 20150319 IV71217 NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK 20150319 IV71217 NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK 20150319 IV71219 NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK $
Example 6: Checking the current system (AIX or VIOS).
# time apar check SUMMARY: 6/21 fixes installed (3 APARs have no fix specified) Real 2.00 User 0.40 System 0.23 #
To check a system for fixes, root privileges are required to determine the list of installed fixes.
The check took 2 seconds and determined that only 6 out of 21 of the existing fixes are installed.
The missing fixes can be displayed using the option ‘-b‘ (brief listing) or ‘-l‘ (long listing):
# time apar check -b 20210315 sec aix CVE-2020-14779,CVE-2020-14781,CVE-2020-14782,CVE-2020-14796,CVE-2020-14797,CVE-2020-14798,CVE-2020-14803,CVE-2020-27221,CVE-2020-2773 Multiple vulnerabilities in IBM Java SDK affect AIX INSTALLED: no fix installed 20210730 sec aix CVE-2021-29741,IJ30557 There is a vulnerability in Korn Shell (ksh) that affects AIX INSTALLED: no fix installed 20210819 hiper aix IJ34376 Applications can terminate on systems with active IPv6 traffic INSTALLED: no fix installed 20210825 sec aix CVE-2021-29727,CVE-2021-29801,CVE-2021-29862,IJ32631 There are multiple vulnerabilities in the AIX kernel INSTALLED: no fix installed 20210915 sec aix CVE-2021-2161,CVE-2021-2369,CVE-2021-2432 Multiple vulnerabilities in IBM Java SDK affect AIX INSTALLED: no fix installed 20211116 sec aix CVE-2021-29860,IJ32714,IJ32736 There is a vulnerability in the libc.a library that affects AIX INSTALLED: no fix installed 20211116 sec aix CVE-2021-29861,IJ35078,IJ35211 There is a vulnerability in EFS that affects AIX INSTALLED: no fix installed 20220106 sec aix CVE-2021-3712 There is a vulnerability in OpenSSL used by AIX. INSTALLED: no fix installed 20220106 sec aix CVE-2021-41617 Vulnerabilities in OpenSSH affect AIX. INSTALLED: no fix installed 20220223 sec aix CVE-2021-2341,CVE-2021-35556,CVE-2021-35559,CVE-2021-35560,CVE-2021-35564,CVE-2021-35565,CVE-2021-35578,CVE-2021-35586,CVE-2021-41035 Multiple vulnerabilities in IBM Java SDK affect AIX INSTALLED: no fix installed 20220223 sec aix CVE-2021-38994,CVE-2021-38995,IJ37012 There are multiple vulnerabilities in the AIX kernel. INSTALLED: no fix installed 20220228 sec aix CVE-2021-38955,IJ38117,IJ38119 There is a vulnerability in the AIX audit user commands. INSTALLED: no fix installed 20220301 sec aix CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512 There are multiple vulnerabilities in AIX CAA. INSTALLED: no fix installed 20220304 sec aix CVE-2021-38989,IJ37488,IJ37778 There is a vulnerability in the AIX pmsvcs kernel extension. INSTALLED: no fix installed 20220304 sec aix CVE-2022-22351,IJ36681,IJ37706 There is a vulnerability in the AIX nimsh daemon. INSTALLED: no fix installed SUMMARY: 6/21 fixes installed (3 APARs have no fix specified) Real 1.90 User 0.32 System 0.18 #
Example 7: Download all fixes for IOS version 3.1.3.21.
$ apar download 3.1.3.21 downloading lpd_fix2.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 270k 100 270k 0 0 197k 0 0:00:01 0:00:01 --:--:-- 197k downloading bind_fix21.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19.1M 100 19.1M 0 0 1498k 0 0:00:13 0:00:13 --:--:-- 1665k downloading vios_fix.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 32.7M 100 32.7M 0 0 1571k 0 0:00:21 0:00:21 --:--:-- 1750k downloading kernel_fix4.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 138M 100 138M 0 0 1618k 0 0:01:27 0:01:27 --:--:-- 1671k downloading libxml2_fix3.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 30.8M 100 30.8M 0 0 1537k 0 0:00:20 0:00:20 --:--:-- 1643k $ $ ls -l total 453952 -rw-r--r-- 1 user01 staff 20080640 Sep 17 10:48 bind_fix21.tar -rw-r--r-- 1 user01 staff 145326080 Sep 17 10:50 kernel_fix4.tar -rw-r--r-- 1 user01 staff 32378880 Sep 17 10:51 libxml2_fix3.tar -rw-r--r-- 1 user01 staff 276480 Sep 17 10:48 lpd_fix2.tar -rw-r--r-- 1 user01 staff 34355200 Sep 17 10:49 vios_fix.tar $
Similarly, all fixes for a specific AIX version can be downloaded by specifying the AIX version!
Example 8: Checking NIM clients for fixes
# apar check aix01 aix02 vios1 aix01: 13/16 fixes installed aix02: 4/12 fixes installed (1 APAR has no fix specified) vios1: 17/20 fixes installed (3 APARs have no fix specified) #
Any number of NIM clients can be specified. NIM groups (mac_group) can also be specified.
Example 9: Checking a NIM client and downloading missing fixes
# apar check -d aix07 aix07: 13/16 fixes installed downloading efs_fix.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5010k 100 5010k 0 0 1079k 0 0:00:04 0:00:04 --:--:-- 1241k downloading kernel_fix3.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 142M 100 142M 0 0 1637k 0 0:01:29 0:01:29 --:--:-- 1684k downloading bind_fix20.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19.1M 100 19.1M 0 0 1494k 0 0:00:13 0:00:13 --:--:-- 1596k #
The fixes are placed in the current directory.
Example 10: View fixes for a specific fileset
$ apar show bos.cluster.rte type: hiper product: vios versions: 2.2.3.80,2.2.3.90 abstract: CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLUSTER apars: IV97148 fixedIn: See Advisory ifixes: IV97148s8a.170613.61TL09SP08.epkg.Z,IV97148s8a.170613.epkg.Z,IV97148s9b.171030.61TL09SP09.epkg.Z,IV97148s9b.171030.epkg.Z bulletinUrl: http://www-01.ibm.com/support/docview.wss?uid=isg1IV97148 filesets: bos.cluster.rte:6.1.9.200-6.1.9.201 issued: 20171108 updated: siblings: 6100-09:IV97148 7100-04:IV97265 7200-01:IV97266 download: https://aix.software.ibm.com/aix/ifixes/iv97148/ cvss: reboot: yes … $
A version can also be specified:
$ apar show bos.cluster.rte:7.2.5.1 type: sec product: aix versions: 7200-05-01,7200-05-01-2038,7200-05-01-2039,7200-05-02,7200-05-02-2114,7200-05-03-2135,7200-05-03-2136,7200-05-03-2148 abstract: There are multiple vulnerabilities in AIX CAA. apars: CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512 fixedIn: 7200-05-04 ifixes: IJ36682s3a.220228.epkg.Z,IJ36682s3b.220228.epkg.Z,IJ37512s1a.220228.epkg.Z,IJ37512s2a.220228.epkg.Z bulletinUrl: https://aix.software.ibm.com/aix/efixes/security/caa_advisory2.asc filesets: bos.cluster.rte:7.2.5.0-7.2.5.1,bos.cluster.rte:7.2.5.100-7.2.5.101 issued: 20220301 updated: siblings: download: https://aix.software.ibm.com/aix/efixes/security/caa_fix2.tar cvss: CVE-2022-22350:6.2 / CVE-2021-38996:6.2 reboot: yes … $
Information about the ‘apar‘ command
The curl command is used to download files. It is available, for example, on the AIX toolbox. If curl is not installed or there is no connection to the Internet (with or without a proxy), then the download functionality of the ‘apar‘ command cannot be used. However, all other functions such as viewing APARs, checking for fixes, or searching for specific APARs can still be used without such a connection.
If a proxy is required, it can be configured using one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg, e.g.:
# The HTTP proxy to use # Default: (none) HttpProxy: http://172.168.10.12:3333
We recommend using the /opt/pwrcmps/etc/tools.cfg file for the proxy configuration, as this is valid for all users.
The ‘apar‘ command requires the CSV file apar.csv which contains data records of all HIPER and SECURITY fixes. This file is made available by IBM at the following URL:
https://esupport.ibm.com/customercare/flrt/doc?page=aparCSV
By default, the ‘apar‘ command first searches for this file in the user’s home directory and then under /opt/pwrcmps/etc. If the file is not available in both places, the file will be downloaded from IBM using the URL above. The behavior can be configured via one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg:
# The order of locations to look for the apar.csv file # Default: ~,/opt/pwrcmps/etc,ibmwebsite #AparCsvResolve:
We recommend downloading the file regularly using a crontab entry and storing it under /opt/pwrcmps/etc/apar.csv. The file can then be used by all users without having to download it again for each command call.
The download can be done using the following call:
$ apar getcsv % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2834k 0 2834k 0 0 1240k 0 --:--:-- 0:00:02 --:--:-- 1240k $
The file is stored in the current directory. A crontab call from root for regular download could look like this:
( cd /opt/pwrcmps/etc; apar getcsv )
The ‘apar‘ command can be downloaded from our download area, it includes a time-limited test license for evaluation purposes.
LPAR tool 1.7.0.1 is now available
Version 1.7.0.1 of the LPAR tool is now available in our download area.
The new version supports the following new features, among others:
-
- Installation of IFixes and updates on the HMC (hmc help updhmc)
- System firmware updates (and more) of managed systems (ms help updatelic)
- Display FLRT data with online query at IBM (hmc help flrt, ms help flrt, lpar help flrt)
- Configuration of NTP on HMCs (hmc help ntp)
Versions for Linux, AIX and Macos are available.
All versions include a test license valid until September 30th, 2022.
So download, install and then try it out!
Monitoring virtual FC Client Traffic
With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).
Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:
$ vios lsnports ms15-vio1 NAME PHYSLOC FABRIC TPORTS APORTS SWWPNS AWWPNS fcs0 U78CB.001.XXXXXXX-P1-C5-T1 1 64 62 2032 2012 fcs1 U78CB.001.XXXXXXX-P1-C5-T2 1 64 62 2032 2012 fcs2 U78CB.001.XXXXXXX-P1-C5-T3 1 64 61 2032 1979 fcs3 U78CB.001.XXXXXXX-P1-C5-T4 1 64 61 2032 1979 fcs4 U78CB.001.XXXXXXX-P1-C3-T1 1 64 50 3088 3000 fcs5 U78CB.001.XXXXXXX-P1-C3-T2 1 64 63 3088 3077 $
We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:
$ vios fcstat ms15-vio1 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS ms15-vio1 fcs1 0x210000XXXXX56EC5 fcs1 774.75/s 129.51 MB/s 1332.71/s 92.96 MB/s 20 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 318.10/s 83.39 MB/s 481.34/s 126.18 MB/s 0 ms15-vio1 fcs2 0x210000XXXXX56EC6 fcs2 318.10/s 83.39 MB/s 480.78/s 126.03 MB/s 0 aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 583.98/s 60.35 MB/s 1835.17/s 124.86 MB/s 0 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 583.70/s 60.27 MB/s 1836.21/s 124.92 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 923.19/s 165.08 MB/s 1032.81/s 17.25 MB/s 46 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 775.12/s 129.48 MB/s 1047.32/s 17.15 MB/s 20 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 775.78/s 128.99 MB/s 1037.99/s 17.39 MB/s 20 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 290.39/s 76.12 MB/s 0 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 142.89/s 36.12 MB/s 0.00/s 0.00 B/s 26 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 234.97/s 4.58 MB/s 621.78/s 11.12 MB/s 40 cus1dbp01 fcs4 0xC050760XXXXX0047 fcs0 243.55/s 5.05 MB/s 432.33/s 9.95 MB/s 0 cus1dbi01 fcs4 0xC050760XXXXX0044 fcs1 0.94/s 10.42 KB/s 87.28/s 459.26 KB/s 0 ... HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1772.84/s 162.24 MB/s 1309.30/s 70.60 MB/s 68 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 1769.13/s 161.95 MB/s 1305.60/s 70.54 MB/s 68 ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 883.55/s 118.97 MB/s 1551.97/s 108.78 MB/s 43 ms15-vio1 fcs2 0x21000024XXXXXEC6 fcs2 201.09/s 52.72 MB/s 497.26/s 130.35 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 201.09/s 52.72 MB/s 495.40/s 129.87 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 923.54/s 128.89 MB/s 1234.98/s 23.31 MB/s 65 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 876.93/s 118.93 MB/s 1234.98/s 23.32 MB/s 44 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 884.17/s 119.07 MB/s 1223.50/s 23.00 MB/s 43 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 325.83/s 85.41 MB/s 0 ... ^C $
The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:
$ vios fcstat -t 10 ms15-vio1 2 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 1034.58/s 86.56 MB/s 2052.23/s 160.11 MB/s 20 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 1532.63/s 115.60 MB/s 1235.72/s 118.32 MB/s 40 aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1510.33/s 114.88 MB/s 1236.49/s 118.27 MB/s 40 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 1036.11/s 86.67 MB/s 1612.25/s 44.86 MB/s 20 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 1031.50/s 86.29 MB/s 1588.02/s 44.27 MB/s 20 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 1029.58/s 86.31 MB/s 1567.63/s 43.65 MB/s 20 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 436.52/s 114.43 MB/s 0 ms15-vio1 fcs2 0x21000024XXXXXEC6 fcs2 0.00/s 0.00 B/s 435.75/s 114.23 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 0.00/s 0.00 B/s 432.68/s 113.42 MB/s 0 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 144.99/s 0.78 MB/s 478.83/s 2.22 MB/s 46 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 758.14/s 35.55 MB/s 1822.99/s 112.60 MB/s 0 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 757.38/s 35.52 MB/s 1821.46/s 112.59 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 944.23/s 85.09 MB/s 1657.58/s 41.40 MB/s 2 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 943.47/s 85.15 MB/s 1636.90/s 40.68 MB/s 2 ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 949.21/s 84.88 MB/s 1586.74/s 39.41 MB/s 2 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 946.53/s 84.64 MB/s 1584.83/s 39.40 MB/s 2 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 39.44/s 449.92 KB/s 676.97/s 3.63 MB/s 10 cus1dbp01 fcs4 0xC050760XXXXX0047 fcs0 29.10/s 471.69 KB/s 310.92/s 1.28 MB/s 4 cus1mqp01 fcs4 0xC050760XXXXX002C fcs0 1.91/s 4.71 KB/s 230.12/s 1.66 MB/s 0 cus2orap01 fcs4 0xC050760XXXXX000F fcs0 0.77/s 4.31 KB/s 48.25/s 263.49 KB/s 0 ^C $
The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):
$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1858.72/s 51.14 MB/s 1231.82/s 104.20 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 6.94/s 1.82 MB/s 6.94/s 1.82 MB/s 0 aixtsmp1 fcs4 0xC050760XXXXX0042 fcs2 0.39/s 1.19 KB/s 0.39/s 395.05 B/s 0 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.39/s 7.72 B/s 0.00/s 0.00 B/s 1 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs3 0xC050760XXXXX005A fcs7 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1760.48/s 111.48 MB/s 1125.70/s 95.20 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 8.53/s 2.24 MB/s 484.61/s 127.04 MB/s 0 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 469.04/s 122.96 MB/s 0 aixtsmp1 fcs4 0xC050760XXXXX0042 fcs2 0.37/s 1.14 KB/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs3 0xC050760XXXXX005A fcs7 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 ^C $
With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.
If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.
The “label” Attribute for FC Adapters
As of AIX 7.2 TL4 and VIOS 3.1.1.10 there is a new attribute “label” for physical FC adapters. The administrator can set this attribute to any character string (maximum 255 characters). Even if the attribute is only informative, it can be extremely useful in PowerVM virtualization environments. If you have a large number of managed systems, it is not always clear to which FC fabric a certain FC port is connected. This can of course be looked up in the documentation of your systems, but it does involve a certain amount of effort. It is easier if you link this information directly with the FC adapters, which is exactly what the new “label” attribute allows in a simple way. On AIX:
# chdev -l fcs0 -U -a label="Fabric_1" fcs0 changed # lsattr -El fcs0 -a label -F value Fabric_1 #
On virtual I/O servers, the attribute can also be set using the padmin account:
/home/padmin> chdev -dev fcs1 -attr label="Fabric_2" -perm fcs1 changed /home/padmin> lsdev -dev fcs1 -attr label value Fabric_2 /home/padmin>
The attribute is also defined for older FC adapters.
If the “label” attribute is consistently used, it is always possible to determine online for each FC adapter to which fabric the adapter is connected to. This information only needs to be stored once for each FC adapter.
(Note: The “label” attribute is not implemented for AIX 7.1, at least not until 7.1 TL5 SP6.)
LPAR-Tool 1.6.0.0 is available now
Version 1.6.0.0 of our LPAR tool is now available in our download area!
New features are:
- Online monitoring of SEA client statistics (vios help seastat)
- Online monitoring of virtual FC client adapters (vios help fcstat)
- Display of historical processor and memory data (lpar help lsmem, lpar help lsproc)
In the article Monitoring SEA Traffic the possibilities of calling up SEA client statistics are shown.
The Impact of FC-Ports without a Link
FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).
(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)
Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):
$ lpar lsslot ms26-vio1 DRC_NAME DRC_INDEX IOPOOL DESCRIPTION U78D3.001.XXXXXXX-P1-C49 21040015 none PCIe3 x8 SAS RAID Internal Adapter 6Gb U78D3.001.XXXXXXX-P1-C7 2103001C none PCIe3 4-Port 16Gb FC Adapter U78D3.001.XXXXXXX-P1-C2 21010021 none PCIe3 4-Port 16Gb FC Adapter $ (HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)
However, only 2 ports of the 8 ports are cabled:
$ vios lsnports ms26-vio1 NAME PHYSLOC FABRIC TPORTS APORTS SWWPNS AWWPNS fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 $ (VIOS: lsnports)
When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000045 Error opening device: /dev/fscsi2 errno: 00000045 Error opening device: /dev/fscsi3 errno: 00000045 Error opening device: /dev/fscsi5 errno: 00000045 Error opening device: /dev/fscsi6 errno: 00000045 Error opening device: /dev/fscsi7 errno: 00000045 real 1m13.56s user 0m0.03s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m11.61s user 0m0.01s sys 0m0.00s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000045 real 0m11.31s user 0m0.01s sys 0m0.01s (4)padmin@ms26-vio1:/home/padmin>
LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.
In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.
The following example shows how to reconfigure the fscsi1 device:
$ vios chdev ms26-vio1 fscsi1 autoconfig=defined $ (VIOS: chdev -dev fscsi1 -attr autoconfig=defined) $ $ vios rmdev ms26-vio1 fscsi1 $ (VIOS: rmdev -dev fscsi1 –ucfg) $ $ vios lsdev ms26-vio1 fscsi1 NAME STATUS PHYSLOC PARENT DESCRIPTION fscsi1 Defined U78D3.001.XXXXXXX-P1-C2-T2 fcs1 FC SCSI I/O Controller Protocol Device $ (VIOS: lsdev -dev fscsi1) $ $ vios lsattr ms26-vio1 fscsi1 ATTRIBUTE VALUE DESCRIPTION USER_SETTABLE attach none How this adapter is CONNECTED False autoconfig defined Configuration State True dyntrk yes Dynamic Tracking of FC Devices True+ fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True+ scsi_id Adapter SCSI ID False sw_fc_class 3 FC Class for Fabric True $ (VIOS: lsdev -dev fscsi1 –attr) $
With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!
If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000005 Error opening device: /dev/fscsi2 errno: 00000045 Error opening device: /dev/fscsi3 errno: 00000045 Error opening device: /dev/fscsi5 errno: 00000045 Error opening device: /dev/fscsi6 errno: 00000045 Error opening device: /dev/fscsi7 errno: 00000045 real 1m1.02s user 0m0.04s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m9.70s user 0m0.00s sys 0m0.01s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000005 real 0m0.00s user 0m0.02s sys 0m0.00s (4)padmin@ms26-vio1:/home/padmin>
The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.
We now also set the autoconfig attribute to defined for all other unused FC ports:
$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7 > do > vios chdev ms26-vio1 $fscsi autoconfig=defined > vios rmdev ms26-vio1 $fscsi > done $
Now we repeat the runtime measurement of the commands again:
(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats … Error opening device: /dev/fscsi1 errno: 00000005 Error opening device: /dev/fscsi2 errno: 00000005 Error opening device: /dev/fscsi3 errno: 00000005 Error opening device: /dev/fscsi5 errno: 00000005 Error opening device: /dev/fscsi6 errno: 00000005 Error opening device: /dev/fscsi7 errno: 00000005 real 0m0.81s user 0m0.03s sys 0m0.10s (0)padmin@ms26-vio1:/home/padmin> (0)padmin@ms26-vio1:/home/padmin> time lsnports name physloc fabric tports aports swwpns awwpns fcs0 U78D3.001.XXXXXXX-P1-C2-T1 1 64 64 3072 3072 fcs4 U78D3.001.XXXXXXX-P1-C7-T1 1 64 64 3072 3072 real 0m0.00s user 0m0.01s sys 0m0.01s (0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1 Error opening device: /dev/fscsi1 errno: 00000005 real 0m0.04s user 0m0.00s sys 0m0.00s (4)padmin@ms26-vio1:/home/padmin>
The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.
It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!
MDS reports at your fingertips
Many AIX and Power System administrators use Microcode Discovery Services to regularly check the versions of adapter firmware and system firmware. The following steps are usually necessary:
– Download the current catalog file catalog.mic.
– Run Inventory Scout to generate the microcode upload file.
– Upload the microcode upload file to IBM http://www14.software.ibm.com/support/customercare/mds/mds
In many cases, the upload is carried out via a browser. The report is shown in the form of an HTML output. Alternatively, you can also upload e.g. with the help of curl and request the data in JSON format.
$ curl -F "mdsData=@ms01-vio1.mup;type=multipart/form" -F "format=json" -H "Expect:" http://www14.software.ibm.com/support/customercare/mds/mds -o ms01-vio1.mup
The returned JSON file contains all information that is otherwise displayed in the browser.
With a small script, the JSON file can be displayed relatively easily in readable ASCII form. We have created the script mds_report for this purpose and made it available in our download area (https://powercampus.de/download). The script expects a microcode upload file as an argument, here is a sample output:
$ mds_report ms01-vio1.mup ms01-vio1.mup upload microcode upload file to IBM ... uploaded Microcode by Host ms01-vio1 IP Addr: X.X.X.X Model: 8205-E6D Serial: XXXXXX Microcode catalog: 2020.07.30 DEVICES INSTALLED LATEST RECOMMEND PKGNAME system AL770_126 AL770_126 None 8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware sissas0 0422003f 0422003f None PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E) ent0,1,2,3 10080180 10240310 Update 4-Port Gigabit Ethernet PCI-Express Adapter ent4,5,6,7 0400401800007 0400401800009 Update PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter fcs0,1,2,3 210301 210313 Update PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729 fcs4,5,6,7 0320080270 0325080271 Update 8Gb PCIe2 Low Profile 4-Port FC Adapter hdisk0,1 37343138 37343139 Update Savvio 15K.3 146/300GB SAS Disk Drive cd0 RA65 RA65 None SATA DVD-RAM Drive RMBO0140512 Microcode by Type IMPACT SEVERITY RELDATE LATEST PKGNAME Security SPE 2018.05.27 AL770_126 8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware Usability ATT 2013.06.06 0422003f PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E) Usability ATT 2019.06.20 10240310 4-Port Gigabit Ethernet PCI-Express Adapter Usability ATT 2016.11.14 0400401800009 PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter Usability ATT 2019.06.17 210313 PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729 Usability ATT 2020.01.28 0325080271 8Gb PCIe2 Low Profile 4-Port FC Adapter Function ATT 2019.04.30 37343139 Savvio 15K.3 146/300GB SAS Disk Drive New NEW 2014.10.24 RA65 SATA DVD-RAM Drive RMBO0140512 $
The output is very similar to the output in the browser. In the first section “Microcode by Host” the update recommendations for the system firmware and adapter firmware are given. In the second section “Microcode by Type” Impact and Severity, as well as the release date of the last available firmware version are shown.
If access to the Internet is only possible via a proxy, the proxy can be specified using the -x argument, as shown in the following example:
$ mds_report -x http://10.0.0.217:1234 ms07-vio1.mup ms07-vio1.mup upload microcode upload file to IBM ... uploaded Microcode by Host ms07-vio1 IP Addr: X.X.X.X Model: 8408-44E Serial: XXXXXXX Microcode catalog: 2020.07.30 DEVICES INSTALLED LATEST RECOMMEND PKGNAME system SV860_138 SV860_215 Update 8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60 sissas0 15511800 19512900 Update PCIe3 RAID SAS Adapter Quad-port 6Gb x8... ses0,1,2,3 1D0B 1D0B None SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E pdisk0,1 37363135 37363142 Update BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive fcs0,1 00010000020025201919 00012000040025700015 Update PCIe2 2-Port 16Gb FC Adapter fcs2,3,4,5 0320080270 0325080271 Update 8Gb PCIe2 Low Profile 4-Port FC Adapter Microcode by Type IMPACT SEVERITY RELDATE LATEST PKGNAME Security HIPER 2020.03.04 SV860_215 8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60 Availability ATT 2020.02.25 19512900 PCIe3 RAID SAS Adapter Quad-port 6Gb x8... New NEW 2015.06.03 1D0B SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E Function ATT 2020.04.16 37363142 BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive Usability ATT 2020.02.18 00012000040025700015 PCIe2 2-Port 16Gb FC Adapter Usability ATT 2020.01.28 0325080271 8Gb PCIe2 Low Profile 4-Port FC Adapter $
If you want to use the script more often, you should enter the proxy in the script itself, for this there is the PROXY variable, which can be set as follows:
$ grep ^PROXY mds_report PROXY="http://10.0.0.217:1234" $
(Where 10.0.0.217:1234 is just an example, you have to supply your own values here.)
It is then no longer necessary to specify a proxy using the -x option.
If the script is executed as root on an AIX system, the proxy configuration is automatically adopted from ESA (Electronic Service Agent).
If you need the URLs to download the firmware, you should use the option -u (show download URLs). The links for the firmware versions are then displayed at the end of the output, here is an example:
$ mds_report -u ms03-vio1.mup /appdata/daten/fk450/aix/mds/virt-aix23-vio1.mup upload microcode upload file to IBM ... uploaded Microcode by Host ms03-vio1 IP Addr: X.X.X.X Model: 9009-22A Serial: XXXXXXX Microcode catalog: 2020.07.30 DEVICES INSTALLED LATEST RECOMMEND PKGNAME system VL910_144 VL940_050 Update 9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system sissas0 19511400 19512900 Update PCIe3 RAID SAS Adapter Quad-port 6Gb x8... pdisk0 36383035 36383035 None AL14SE 600/1200/1800 GB 4K Hard Disk Drive pdisk1,2 41374B30 41374B30 None Ultrastar C15K600-5xx fcs0,1,2,3,4,5,6,7 00011000040041500005 00012000040025700015 Update PCIe3 4-Port 16Gb FC Adapter Microcode by Type IMPACT SEVERITY RELDATE LATEST PKGNAME Availability SPE 2020.05.21 VL940_050 9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system Availability ATT 2020.02.25 19512900 PCIe3 RAID SAS Adapter Quad-port 6Gb x8... Data HIPER 2016.12.01 36383035 AL14SE 600/1200/1800 GB 4K Hard Disk Drive Function ATT 2015.08.18 41374B30 Ultrastar C15K600-5xx Usability ATT 2020.02.18 00012000040025700015 PCIe3 4-Port 16Gb FC Adapter Downloads http://www.ibm.com/support/fixcentral/quickorder?product=ibm/power/900922A&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=01VL940_050_027 http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=40145679_20200224110413_GRP http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1354333840_20161130155709_GRP http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1448849004_20150813164908_GRP http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=427029183_20200213134040_GRP $
The script generally takes less than 1 second to run!
We tested the script on AIX, Linux, and MacOS. Under MacOS there is usually no ksh93. But the installed ksh supports all the necessary features that are required by the mds_report script. If you change the interpreter in the first line of the script to ksh, the script will also run on a Mac.
A good description of Inventory Scout and MDS can be found here: http://gibsonnet.net/blog/cgaix/html/MDS%20reports.html (Chris Gibson)
You can find out how to automate Inventory Scout in our article Automating Inventory Scout
FC NPIV client throughput
When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like
- how much throughput is achieved by a particular LPAR
- which LPARs have the highest throughput and produce the most FC traffic
- are there resource bottlenecks
could be answered.
Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:
(0)padmin@aixvio1:/home/padmin> fcstat -client hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs aixvio1 fcs0 0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576 0 0 0 C050760YYYYYYYYY 0 0 0 0 0 0 0 0 C050760ZZZZZZZZZ 0 0 0 0 0 0 0 0 aix01 fcs0 0xC050760XXXXXXXXX 22685402 101956075 10065757 699512617896 1572578056704 0 0 0 aix02 fcs0 0xC050760XXXXXXXXX 28200473 82295158 12051365 387847746448 626772151808 0 0 0 aix03 fcs0 0xC050760XXXXXXXXX 376500672 255163053 21583628 22619424512608 3786990844928 0 0 0 aix04 fcs0 0xC050760XXXXXXXXX 116450405 504688524 14020031 4037786527400 9929289617408 0 0 0 blbprodora22 fcs0 0xC050760XXXXXXXXX 1341092479 580673554 37458927 44288566807072 12166718497792 0 0 0 ... aixvio1 fcs1 0x100000XXXXXXXXXX 391131484 1090556094 156294130 71031615240217 87642294572864 0 0 0 aixtsm01 fcs2 0xC050760XXXXXXXXX 334020900 785597352 74659821 62072552942128 83284555980288 0 0 0 aixtsm02 fcs0 0xC050760XXXXXXXXX 2943054 40921231 11617552 107317697968 289142333440 0 0 0 aixvio1 fcs2 0x210000XXXXXXXXXX 403180246 5877180796 236998 105482699300998 1540608710446612 0 0 0 aixtsm01 fcs6 0xC050760XXXXXXXXX 146492419 392365162 74250 38378099796342 102844775468007 0 0 0 aixtsm02 fcs2 0xC050760XXXXXXXXX 19 192848 20 1090 50551063184 0 0 0 aixvio1 fcs3 0x210000XXXXXXXXXX 405673338 7371951499 260575 105969796271246 1932388891128304 0 0 0 aixtsm02 fcs3 0xC050760XXXXXXXXX 0 0 4 0 0 0 0 0 aix02 fcs7 0xC050760XXXXXXXXX 42624 2677470211 34211 2382280 701864613402184 0 0 0 ... Invalid initiator world wide name Invalid initiator world wide name (0)padmin@aixvio1:/home/padmin>
The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.
The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.
Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:
$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#
delta=5 # seconds
typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs
bc |& # start bc as coroutine
print -p "scale=2"
# get first sample
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta
while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs
printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print
sleep $delta
done
$
The script ‘npivstat‘ is available for download in our download-area.
Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):
aixvio1 # ./npivstat
hostname dev wwpn inreqs outreqs inbytes outbytes DMA_errs Elem_errs Comm_errs
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 1019.00 0.00 254.75 0.00 0.00 0.00
aixtsm01 fcs6 0xC0507605E5890074 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC0507609A6C70004 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC0507609A6C7001C 0.00 1018.20 0.00 254.55 0.00 0.00 0.00
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 1020.20 0.00 255.05 0.00 0.00 0.00
aixtsm01 fcs6 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC050760XXXXXXXXX 0.00 1019.80 0.00 254.95 0.00 0.00 0.00
...
aixvio1 fcs2 0x210000XXXXXXXXXX 0.00 984.80 0.00 246.20 0.00 0.00 0.00
aixtsm01 fcs6 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aixtsm02 fcs2 0xC050760XXXXXXXXX 0.00 0.00 0.00 0.00 0.00 0.00 0.00
aix05 fcs6 0xC050760XXXXXXXXX 0.00 985.00 0.00 246.25 0.00 0.00 0.00
...
^Caixvio1 #
In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.
The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.