Error: Mirror pools must be defined

The following error message occurred while creating a mirrored logical volume:

# mklv -c 2 -t jfs2 datavg01 10
0516-1814 lcreatelv: Mirror pools must be defined for each copy when strict mirror
        pools are enabled.
0516-822 mklv: Unable to create logical volume.
#

The reason for this is that the mirror pool strictness is set for the volume group. The article Mirror Pools: Understanding Mirror Pool Strictness examines and explains this in more detail.

Error: Every mirror pool must contain a copy of the logical volume

The attempt to create an unmirrored LV in a VG with mirror pools results in the following error:

# mklv -t jfs2 -p copy1=DC1 datavg01 10
0516-1829 mklv: Every mirror pool must contain a copy of
        the logical volume.
0516-822 mklv: Unable to create logical volume.
#

The cause is the VG‘s Mirror Pool Strictness attribute, which is set to ‘super-strict‘. The article Mirror Pools: Understanding Mirror Pool Strictness examines the importance of Mirror Pool Strictness using examples.

New Article Introduction to Mirror Pools

Many IT environments operate their systems in more than one data center. In order not to have any data loss in the event of a failure of an entire data center, the data is mirrored between 2 or more data centers. The mirroring can be realized by the storage (storage based mirroring) or by a volume manager (LVM in the case of AIX) on the server (host based mirroring). In the article Mirror Pools: An Introduction we look at mirroring using AIX Logical Volume Manager and Mirror Pools. The aim is to show how the correct mirroring of logical volumes can be implemented with the help of mirror pools. In larger environments with many physical volumes, maintaining correct mirroring without mirror pools is difficult and a challenge for the administrator.

The Impact of FC-Ports without a Link

FC ports that are not used and do not have a link should be deactivated, as these significantly extend the runtime of a series of commands and operations (e.g. LPM).

(Note: our LPAR tool is used in some examples, but the corresponding commands on the HMC or the virtual I / O server are always shown!)

Two 4-port FC adapters are in use on one of our virtual I / O servers (ms26-vio1):

$ lpar lsslot ms26-vio1
DRC_NAME                  DRC_INDEX  IOPOOL  DESCRIPTION
U78D3.001.XXXXXXX-P1-C49  21040015   none    PCIe3 x8 SAS RAID Internal Adapter 6Gb
U78D3.001.XXXXXXX-P1-C7   2103001C   none    PCIe3 4-Port 16Gb FC Adapter
U78D3.001.XXXXXXX-P1-C2   21010021   none    PCIe3 4-Port 16Gb FC Adapter
$
(HMC: lshwres -r io --rsubtype slot -m ms26 --filter lpar_names=ms26-vio1)

However, only 2 ports of the 8 ports are cabled:

$ vios lsnports ms26-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78D3.001.XXXXXXX-P1-C2-T1  1       64      64      3072    3072
fcs4  U78D3.001.XXXXXXX-P1-C7-T1  1       64      64      3072    3072
$
(VIOS: lsnports)

When working with the virtual I / O server, it is noticeable, that some of the commands have an unexpectedly long runtime and sometimes hang for a long time. Some example commands are given below, along with the measured runtime:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000045

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m13.56s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m11.61s
user    0m0.01s
sys     0m0.00s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000045

real    0m11.31s
user    0m0.01s
sys     0m0.01s
(4)padmin@ms26-vio1:/home/padmin>

LPM operations also take significantly longer, since all FC ports are examined when searching for suitable FC ports for the necessary NPIV mappings. This can lead to delays in the range of minutes before the migration is finally started.

In order to avoid these unnecessarily long runtimes, FC ports that are not wired should not be activated. The fscsi device has the attribute autoconfig, with the possible values defined and available. By default, the value available is used, which means that the kernel configures and activates the device, even if it has no link, which leads to the waiting times shown above. If the autoconfig attribute is set to defined, the fscsi device is not activated, it then remains in the defined state.

The following example shows how to reconfigure the fscsi1 device:

$ vios chdev ms26-vio1 fscsi1 autoconfig=defined
$
(VIOS: chdev -dev fscsi1 -attr autoconfig=defined)
$
$ vios rmdev ms26-vio1 fscsi1
$
(VIOS: rmdev -dev fscsi1 –ucfg)
$
$ vios lsdev ms26-vio1 fscsi1
NAME    STATUS   PHYSLOC                     PARENT  DESCRIPTION
fscsi1  Defined  U78D3.001.XXXXXXX-P1-C2-T2  fcs1    FC SCSI I/O Controller Protocol Device
$
(VIOS: lsdev -dev fscsi1)
$
$  vios lsattr ms26-vio1 fscsi1
ATTRIBUTE     VALUE      DESCRIPTION                            USER_SETTABLE
attach        none       How this adapter is CONNECTED          False
autoconfig    defined    Configuration State                    True
dyntrk        yes        Dynamic Tracking of FC Devices         True+
fc_err_recov  fast_fail  FC Fabric Event Error RECOVERY Policy  True+
scsi_id       Adapter    SCSI ID                                False
sw_fc_class   3          FC Class for Fabric                    True
$
(VIOS: lsdev -dev fscsi1 –attr)
$

With the autoconfig=defined attribute, the fscsi device remains defined even when the cfgmgr is run!

If one repeats the runtime measurement of the commands above, one can see that the runtime of the commands has already measurably improved:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000045

Error opening device: /dev/fscsi3
errno: 00000045

Error opening device: /dev/fscsi5
errno: 00000045

Error opening device: /dev/fscsi6
errno: 00000045

Error opening device: /dev/fscsi7
errno: 00000045

real    1m1.02s
user    0m0.04s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m9.70s
user    0m0.00s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.00s
user    0m0.02s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The running time of the netstat command was shortened by 12 seconds, the lsnports command was about 2 seconds faster.

We now also set the autoconfig attribute to defined for all other unused FC ports:

$ for fscsi in fscsi2 fscsi3 fscsi5 fscsi6 fscsi7
> do
> vios chdev ms26-vio1 $fscsi autoconfig=defined
> vios rmdev ms26-vio1 $fscsi
> done
$

Now we repeat the runtime measurement of the commands again:

(0)padmin@ms26-vio1:/home/padmin> time netstat –cdlistats
…
Error opening device: /dev/fscsi1
errno: 00000005

Error opening device: /dev/fscsi2
errno: 00000005

Error opening device: /dev/fscsi3
errno: 00000005

Error opening device: /dev/fscsi5
errno: 00000005

Error opening device: /dev/fscsi6
errno: 00000005

Error opening device: /dev/fscsi7
errno: 00000005

real    0m0.81s
user    0m0.03s
sys     0m0.10s
(0)padmin@ms26-vio1:/home/padmin>
(0)padmin@ms26-vio1:/home/padmin> time lsnports         
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U78D3.001.XXXXXXX-P1-C2-T1          1     64     64   3072    3072
fcs4             U78D3.001.XXXXXXX-P1-C7-T1          1     64     64   3072    3072

real    0m0.00s
user    0m0.01s
sys     0m0.01s
(0)padmin@ms26-vio1:/home/padmin> time fcstat fcs1       

Error opening device: /dev/fscsi1
errno: 00000005

real    0m0.04s
user    0m0.00s
sys     0m0.00s
(4)padmin@ms26-vio1:/home/padmin>

The netstat command now takes less than 1 second, the lsnports command only 0.1 seconds.

It is therefore worthwhile to set the autoconfig attribute for unused FC ports to defined!

 

Don’t Forget to Update AIX-rpm

Recently, when installing Python3 from the AIX toolbox using YUM, we encountered the following error situation:

# yum install python3
...
--> Finished Dependency Resolution
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libssl.a(libssl.so.1.0.2)
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libcrypto.a(libcrypto.so.1.0.2)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
#

Python3 requires version 1.0.2 of libssl.so and libcrypto.so, but the two libraries cannot be found. These two libraries can either be provided via the RPM package openssl, in the appropriate version, or via AIX. When the AIX toolbox is used, the openssl package is typically not installed, and the AIX OpenSSL libraries are used. Therefore, in our case, the OpenSSL RPM package is not installed. The libraries are then made available by BFF packages of the AIX operating system.

The information about which libraries are made available by AIX is provided via the virtual RPM package AIX-rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so.0.9.7
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

The output shows that versions 0.9.8 and 1.0.0 of the two required libraries are known to the AIX-rpm package, but the required version 1.0.2 is apparently not available. We take a quick look at the associated archive /usr/lib/libssl.a (or /usr/lib/libcrypto.a):

# ar -X any t /usr/lib/libssl.a
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl64.so
libssl64.so.0.9.8
libssl64.so.1.0.0
#

The archive obviously also provides version 1.0.2. This information is not provided by the AIX-rpm RPM package. This is because a newer version of openssl.base has obviously been installed, but the AIX-rpm package has not been updated. The RPM package currently has the following version:

# rpm -q AIX-rpm
AIX-rpm-7.1.5.15-6.ppc
#

We update the package now with the command updtvpkg (update virtual package):

# updtvpkg
Please wait...
# 
# rpm -q AIX-rpm
AIX-rpm-7.1.5.30-7.ppc
#

The version number has now changed from 7.1.5.15-6 to 7.1.5.30-7! We can again display which libraries are provided by AIX via AIX rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto.so.1.0.2)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so
libcrypto.so.0.9.7
libcrypto.so.1.0.0
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl.so.1.0.2)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl.so
libssl.so.1.0.0
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

After the update of the virtual package AIX-rpm, the RPM database now also contains the version 1.0.2 of libssl.so and libcrypto.so.

The installation of Python3 is now successful:

# yum install python3
...
Is this ok [y/N]: y
...
Installed:
  python3.ppc 0:3.7.1-1                                                                                       

Dependency Installed:
  bzip2.ppc 0:1.0.6-3  expat.ppc 0:2.2.4-1  info.ppc 0:6.4-1  libffi.ppc 0:3.2.1-3  libstdc++.ppc 0:6.3.0-2
  ncurses.ppc 0:6.1-2

Dependency Updated:
  readline.ppc 0:7.0-5                                                                                        

Complete!
#

Therefore, do not forget to update AIX-rpm when updating BFF packages!

MDS reports at your fingertips

Many AIX and Power System administrators use Microcode Discovery Services to regularly check the versions of adapter firmware and system firmware. The following steps are usually necessary:

– Download the current catalog file catalog.mic.

– Run Inventory Scout to generate the microcode upload file.

– Upload the microcode upload file to IBM http://www14.software.ibm.com/support/customercare/mds/mds

In many cases, the upload is carried out via a browser. The report is shown in the form of an HTML output. Alternatively, you can also upload e.g. with the help of curl and request the data in JSON format.

$ curl -F "mdsData=@ms01-vio1.mup;type=multipart/form" -F "format=json" -H "Expect:" http://www14.software.ibm.com/support/customercare/mds/mds -o ms01-vio1.mup

The returned JSON file contains all information that is otherwise displayed in the browser.

With a small script, the JSON file can be displayed relatively easily in readable ASCII form. We have created the script mds_report for this purpose and made it available in our download area (https://powercampus.de/download). The script expects a microcode upload file as an argument, here is a sample output:

$ mds_report ms01-vio1.mup
ms01-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms01-vio1
IP Addr: X.X.X.X
Model: 8205-E6D   Serial: XXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           AL770_126        AL770_126        None        8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware
sissas0          0422003f         0422003f         None        PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E)
ent0,1,2,3       10080180         10240310         Update      4-Port Gigabit Ethernet PCI-Express Adapter
ent4,5,6,7       0400401800007    0400401800009    Update      PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter
fcs0,1,2,3       210301           210313           Update      PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729
fcs4,5,6,7       0320080270       0325080271       Update      8Gb PCIe2 Low Profile 4-Port FC Adapter
hdisk0,1         37343138         37343139         Update      Savvio 15K.3 146/300GB SAS Disk Drive
cd0              RA65             RA65             None        SATA DVD-RAM Drive RMBO0140512

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Security      SPE         2018.05.27    AL770_126        8231-E1D; 8231-E2D; 8246-L1D; 8246-L1T; 8246-L2D; 8246-L2T; 8202-E4D; 8205-E6D; 8268-E1D; 8493-SV6 HV16 System Firmware
Usability     ATT         2013.06.06    0422003f         PCI Express x8 Ext Dual-x4 3Gb SAS RAID Adapter (CCIN: 574E)
Usability     ATT         2019.06.20    10240310         4-Port Gigabit Ethernet PCI-Express Adapter
Usability     ATT         2016.11.14    0400401800009    PCIe2 2-Port 10GbE SFP+Copper or 10GbE SR Adapter
Usability     ATT         2019.06.17    210313           PCIe2 4-Port 8Gb Fibre Channel Adapter, FC 5729
Usability     ATT         2020.01.28    0325080271       8Gb PCIe2 Low Profile 4-Port FC Adapter
Function      ATT         2019.04.30    37343139         Savvio 15K.3 146/300GB SAS Disk Drive
New           NEW         2014.10.24    RA65             SATA DVD-RAM Drive RMBO0140512
$

The output is very similar to the output in the browser. In the first section “Microcode by Host” the update recommendations for the system firmware and adapter firmware are given. In the second section “Microcode by TypeImpact and Severity, as well as the release date of the last available firmware version are shown.

If access to the Internet is only possible via a proxy, the proxy can be specified using the -x argument, as shown in the following example:

$ mds_report -x http://10.0.0.217:1234 ms07-vio1.mup
ms07-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms07-vio1
IP Addr: X.X.X.X
Model: 8408-44E   Serial: XXXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           SV860_138        SV860_215        Update      8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60
sissas0          15511800         19512900         Update      PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
ses0,1,2,3       1D0B             1D0B             None        SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E
pdisk0,1         37363135         37363142         Update      BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive
fcs0,1           00010000020025201919  00012000040025700015  Update      PCIe2 2-Port 16Gb FC Adapter
fcs2,3,4,5       0320080270       0325080271       Update      8Gb PCIe2 Low Profile 4-Port FC Adapter

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Security      HIPER       2020.03.04    SV860_215        8247-21L, 8247-22L, 8247-42L, 8284-21A, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 5148-21L, 5148-22L - system-v860.60
Availability  ATT         2020.02.25    19512900         PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
New           NEW         2015.06.03    1D0B             SAS Enclosure Services for Power 8 4U High Function DASD backplane 8408-E8E
Function      ATT         2020.04.16    37363142         BP5XX15KHDD 15KRPM 73/146/300/600GB SAS Disk Drive
Usability     ATT         2020.02.18    00012000040025700015  PCIe2 2-Port 16Gb FC Adapter
Usability     ATT         2020.01.28    0325080271       8Gb PCIe2 Low Profile 4-Port FC Adapter
$

If you want to use the script more often, you should enter the proxy in the script itself, for this there is the PROXY variable, which can be set as follows:

$ grep ^PROXY mds_report
PROXY="http://10.0.0.217:1234"
$

(Where 10.0.0.217:1234 is just an example, you have to supply your own values here.)

It is then no longer necessary to specify a proxy using the -x option.

If the script is executed as root on an AIX system, the proxy configuration is automatically adopted from ESA (Electronic Service Agent).

If you need the URLs to download the firmware, you should use the option -u (show download URLs). The links for the firmware versions are then displayed at the end of the output, here is an example:

$ mds_report -u ms03-vio1.mup
/appdata/daten/fk450/aix/mds/virt-aix23-vio1.mup upload microcode upload file to IBM ... uploaded

Microcode by Host

ms03-vio1
IP Addr: X.X.X.X
Model: 9009-22A   Serial: XXXXXXX
Microcode catalog: 2020.07.30

DEVICES          INSTALLED        LATEST           RECOMMEND   PKGNAME
system           VL910_144        VL940_050        Update      9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system
sissas0          19511400         19512900         Update      PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
pdisk0           36383035         36383035         None        AL14SE 600/1200/1800 GB 4K Hard Disk Drive
pdisk1,2         41374B30         41374B30         None        Ultrastar C15K600-5xx
fcs0,1,2,3,4,5,6,7  00011000040041500005  00012000040025700015  Update      PCIe3 4-Port 16Gb FC Adapter

Microcode by Type

IMPACT        SEVERITY    RELDATE       LATEST           PKGNAME
Availability  SPE         2020.05.21    VL940_050        9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H-system
Availability  ATT         2020.02.25    19512900         PCIe3 RAID SAS Adapter Quad-port 6Gb x8...
Data          HIPER       2016.12.01    36383035         AL14SE 600/1200/1800 GB 4K Hard Disk Drive
Function      ATT         2015.08.18    41374B30         Ultrastar C15K600-5xx
Usability     ATT         2020.02.18    00012000040025700015  PCIe3 4-Port 16Gb FC Adapter

Downloads

http://www.ibm.com/support/fixcentral/quickorder?product=ibm/power/900922A&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=01VL940_050_027
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=40145679_20200224110413_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1354333840_20161130155709_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=1448849004_20150813164908_GRP
http://www.ibm.com/support/fixcentral/quickorder?product=ibm/io&release=all&platform=all&function=fixId&includeSupersedes=0&source=fc&fixids=427029183_20200213134040_GRP
$

The script generally takes less than 1 second to run!

We tested the script on AIX, Linux, and MacOS. Under MacOS there is usually no ksh93. But the installed ksh supports all the necessary features that are required by the mds_report script. If you change the interpreter in the first line of the script to ksh, the script will also run on a Mac.

A good description of Inventory Scout and MDS can be found here: http://gibsonnet.net/blog/cgaix/html/MDS%20reports.html (Chris Gibson)

You can find out how to automate Inventory Scout in our article Automating Inventory Scout

 

nmon_printer: quick view into NMON files

Many administrators use NMON to collect performance data on AIX systems. The collected data can then be visualized graphically with the NMON analyzer for example. Alternatively, one can of course also use njmon. Sometimes you want to take a quick look into an NMON file on the AIX system itself. Of course, this can be done with grep, awk, and other standard UNIX utilities. However, the appropriate command line must always be considered and typed in and the output is not always very readable.

For such a quick look into an NMON file, we have provided the script nmon_printer in our download area. With the nmon_printer you can easily list which data records have been collected in the NMON file:

$ nmon_printer -l aix01_200718.nmon
CPU_ALL -     CPU Total aix01
CPU01 -       CPU 1 aix01
CPU02 -       CPU 2 aix01
CPU03 -       CPU 3 aix01
CPU04 -       CPU 4 aix01
CPU05 -       CPU 5 aix01
CPU06 -       CPU 6 aix01
CPU07 -       CPU 7 aix01
CPU08 -       CPU 8 aix01
MEM -         Memory aix01
MEMNEW -      Memory New aix01
MEMUSE -      Memory Use aix01
PAGE -        Paging aix01
...
$

To display the data records, in addition to the NMON file, you must also specify the desired data record to be displayed, upper or lower case is irrelevant. Here is an example of CPU_ALL:

$ nmon_printer aix01_200718.nmon cpu_all
  CPU_ALL             Timestamp  User%  Sys%  Wait%  Idle%  Busy  PhysicalCPUs 
  CPU_ALL  00:05:15 17-JUL-2020    1.2   3.0    0.1   95.7   4.2              8
  CPU_ALL  00:10:15 17-JUL-2020    1.3   2.7    0.0   96.0   3.9              8
  CPU_ALL  00:15:15 17-JUL-2020    1.0   2.3    0.1   96.7   3.3              8
  CPU_ALL  00:20:16 17-JUL-2020    5.1   3.0    0.1   91.8   8.2              8
  CPU_ALL  00:25:16 17-JUL-2020    1.2   2.7    0.0   96.0   3.9              8
  CPU_ALL  00:30:16 17-JUL-2020    1.1  17.5    0.1   81.3  18.5              8
  CPU_ALL  00:35:16 17-JUL-2020    1.1   5.8    0.0   93.1   6.9              8
...
$

In some cases, a data record has so many fields that long lines have to be broken and the output becomes somewhat confusing. The data record LPAR has e.g. 23 fields! In such cases, or if you are only interested in certain fields, you can specify the desired fields with the ‘-f‘ option. You can either specify the desired fields separated by commas:

$ nmon_printer -f 0,1,2,13,14,15 aix01_200718.nmon lpar
  LPAR             Timestamp  PhysicalCPU  EC_User%  EC_Sys%  EC_Wait%
  LPAR  00:05:15 17-JUL-2020        0.037      1.23     2.95      0.00
  LPAR  00:10:15 17-JUL-2020        0.034      1.28     2.66      0.00
  LPAR  00:15:15 17-JUL-2020        0.029      1.03     2.25      0.00
  LPAR  00:20:16 17-JUL-2020        0.064      5.14     3.01      0.00
  LPAR  00:25:16 17-JUL-2020        0.035      1.21     2.69      0.00
  LPAR  00:30:16 17-JUL-2020        0.150      1.08    17.46      0.02
  LPAR  00:35:16 17-JUL-2020        0.063      1.13     5.77      0.00
  LPAR  00:40:16 17-JUL-2020        0.030      1.13     2.24      0.00
  LPAR  00:45:16 17-JUL-2020        0.030      1.06     2.34      0.00
...
$

Or you can also specify ranges (e.g. 0-2 or 13-15):

$ nmon_printer -f 1,5-7 aix01_200718.nmon ioadapt
             Timestamp  fcs1_read-KB/s  fcs1_write-KB/s  fcs1_xfer-tps
  00:05:15 17-JUL-2020             0.0             27.3            3.0
  00:10:15 17-JUL-2020             0.0              0.0            2.6
  00:15:15 17-JUL-2020             0.0             13.6            2.8
  00:20:16 17-JUL-2020             0.0             13.6            2.3
  00:25:16 17-JUL-2020             0.0             13.7            2.3
  00:30:16 17-JUL-2020            27.3           4982.5           89.4
  00:35:16 17-JUL-2020            13.7             27.3            4.1
  00:40:16 17-JUL-2020             0.0             13.6            2.6
  00:45:16 17-JUL-2020             0.0             13.6            2.6
...
$

The fields are counted starting from 0.

HSCLB505 The partition cannot use hardware-accelerated encryption

When migrating LPARs using LPM onto a somewhat older hardware, the following error can occur:

HSCLB505 The partition cannot use hardware-accelerated encryption on the destination managed system because the destination managed system does not support hardware-accelerated encryption.

This means that hardware-accelerated encryption is activated for the LPAR, but is not supported on the destination managed system.

Disabling hardware-accelerated encryption using the LPAR-Tool is easy:

$ lpar -d chmem lpar01 hardware_mem_encryption=0
$

Without the LPAR-Tool this is of course also possible. Log into your HMC and use the following command from the commandoline:

chhwres -m ms01 -r mem -o s -p lpar01 -a 'hardware_mem_encryption=0'

Afterwards validation and migration using LPM should work.

HSCLB504 The migrating partition cannot use hardware-accelerated Active Memory Expansion

When migrating LPARs using LPM onto a somewhat older hardware, the following error can occur:

HSCLB504 The migrating partition cannot use hardware-accelerated Active Memory Expansion on the destination managed system because the destination managed system does not support hardware-accelerated Active Memory Expansion.

This means that Active Memory Erweiterung (AME) is activated for the LPAR, but is not supported on the destination managed system.

Disabling Active Memory Expansion using the LPAR-Tool is easy:

$ lpar -d chmem lpar01 hardware_mem_expansion=0
$

Without the LPAR-Tool this is of course also possible. Log into your HMC and use the following command from the commandoline:

chhwres -m ms01 -r mem -o s -p lpar01 -a 'hardware_mem_expansion=0'

Afterwards validation and migration using LPM should work.

AIX-Lifecycle at your fingertips

How often have you searched the lifecycle informations (start of general availability and end of support) for AIX, VIOS, PowerHA or HMC via Google?

This information can also be found easily using the command line of the IBM FLRT API (https://www14.software.ibm.com/webapp/set2/flrt/sas?page=jsonapi). All you need is:

  • a direct internet connection or connection via an HTTP proxy
  • an installed version of curl

You can get an overview of all AIX versions with the help of the following curl command:

$ curl "https://www14.software.ibm.com/support/customercare/flrt/liteTable?prodKey=aix&format=json"
{"results": [
{
"input": "7200-04-02",
"inputurl": "http://www.ibm.com/support/fixcentral/aix/quickorder?fixids=U887468&function=fixId&includeRequisites=1&includeSupersedes=0&release=7.2&source=flrt",
"ga": "2020.05.15",
"eosps": "2022.11.30"
},
{
"input": "7200-04-01",
"inputurl": "http://www.ibm.com/support/fixcentral/aix/quickorder?fixids=U886355&function=fixId&includeRequisites=1&includeSupersedes=0&release=7.2&source=flrt",
"ga": "2019.11.15",
"eosps": "2022.11.30"
},
...

The output is in JSON format and is greatly shortened in the example.

We packed everything necessary in a small shell script (show_life_cycle) and made it available in our download area  (https://powercampus.de/download).

Here is a sample run of the script without arguments:

$ show_life_cycle 
VERSION          GA          EOSPS       UPDATE           UPGRADE
7200-04-02       2020.05.15  2022.11.30  -                -
7200-04-01       2019.11.15  2022.11.30  -                -
7200-04-00       2019.11.15  2022.11.30  7200-04-01       -
7200-03-04       2020.02.14  2021.09.30  -                7200-04-01
7200-03-03       2019.05.10  2021.09.30  7200-03-04       7200-04-01 
7200-03-02       2018.11.16  2021.09.30  7200-03-04       7200-04-01
...

All AIX versions are listed. However, other products can be selected for display via an argument, e.g. RHEL versions, SLES versions, VIOS versions, HMC versions, PowerHA versions and even HMC models or power systems. A complete list can be found at https://www14.software.ibm.com/webapp/set2/flrt/doc?page=prodTable. Here is a list of the RHEL versions as an example:

$ show_life_cycle rhel
VERSION          GA          EOSPS       UPDATE           UPGRADE
8.0              2019.05.07  NA          -                -
7.7              2019.08.06  NA          -                -
7.6              2018.10.30  NA          7.7              -
7.5              2018.04.10  NA          7.7              -
7.4              2017.07.31  NA          7.7              -
7.3              2016.11.03  NA          7.7              -
7.2              2015.11.19  NA          7.7              -
7.1              2015.03.05  NA          7.7              -
7.0              2014.06.10  2015.03.05  7.7              -
6.10             2018.06.19  NA          -                7.6
6.9              2017.03.21  NA          6.10             7.6
6.8              2016.05.10  NA          6.10             7.6
6.7              2015.07.22  NA          6.10             7.6
6.6              2014.10.14  NA          6.10             7.6
6.5              2013.11.21  2014.10.14  6.10             7.6
...

The output is very long in some cases. An additional argument can be used to specify a prefix for the desired versions, shown here using PowerHA 7.2.2 as an example:

$ show_life_cycle hacmp 7.2.2
VERSION          GA          EOSPS       UPDATE           UPGRADE
7.2.2.4          2020.05.28  2021.04.30  -                7.2.3.2
7.2.2.3          2019.09.13  2021.04.30  -                7.2.3.2
7.2.2.2          2018.11.30  2021.04.30  7.2.2.3          7.2.3.2
7.2.2.1          2018.06.29  2021.04.30  7.2.2.3          7.2.3.2
7.2.2            2017.12.15  2021.04.30  7.2.2.3          7.2.3.2
$

If only a connection using a proxy is possible, then the following options are available:

1. The HTTP proxy is specified using the ‘-p‘ option, e.g..:

$ show_life_cycle -p http://10.0.0.3:8000 hmc V9
VERSION          GA          EOSPS       UPDATE           UPGRADE
V9 R1 M941       2020.05.22  2021.04.30  -                -
V9 R1 M940       2019.11.22  2021.04.30  -                -
V9 R1 M931       2019.09.11  2021.04.30  -                -
V9 R1 M930       2019.05.17  2021.04.30  V9 R1 M931       -
V9 R1 M921       2018.11.16  2021.04.30  V9 R1 M931       -
V9 R1 M920       2018.08.17  2021.04.30  V9 R1 M931       -
V9 R1 M911       2018.05.25  2021.04.30  V9 R1 M931       -
V9 R1 M910       2018.03.20  2021.04.30  V9 R1 M931       -
$

2. The HTTP proxy is set directly in the script, shell variable PROXY:

$ grep ^PROXY show_life_cycle 
PROXY="http://10.0.0.3:8000"
$

3. If the script is executed as root on AIX, the proxy configuration of the Electronic Service Agent (ESA) is automatically taken over. This can be overwritten by using the command line option ‘-p‘ or by setting the variable PROXY.

The script should run on any UNIX system with a Korn shell (including MacOS). This means that searching via Google for lifecycle data is no longer necessary.

On the subject of Inventory Scout we have an article Automating Inventory Scout and another script for download available!