Manage and Access APARs

Keeping AIX and virtual I/O servers up to date with regard to HIPER and SECURITY fixes has become enormously important in recent years. To do this, the systems must be checked regularly for any missing fixes. The appropriate fixes must be downloaded and then installed. Determining which fix needs to be installed on a particular system often involves viewing bulletins with a web browser. PowerCampus 01 provides the ‘apar‘ command to simplify the management of fixes. This makes working with fixes and APARs as well as CVEs much easier.

Some sample uses of the ‘apar‘ command

The ‘apar‘ command allows the download of HIPER and SECURITY fixes, the checking of systems (AIX and VIOS) for installed and missing fixes, as well as the display and targeted search for fixes. In order to be able to use all functionalities, a direct Internet connection or a connection via an http proxy server is required. The command is available in versions for AIX, Linux and MacOS. A number of example calls are shown below.

Example 1: What fixes have been released in the last 30 days?

$ apar last
20220817  sec  aix  CVE-2022-1292,CVE-2022-2068,CVE-2022-2097  AIX is vulnerable to arbitrary command execution due to OpenSSL
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  aix  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42341  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  vios  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41687  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-34356,IJ41688  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-36768  AIX is vulnerable to a privilege escalation vulnerability due to invscout
$

Example 2: Displaying information about APAR ID IJ42341.

$ apar show IJ42341
type:         sec
product:      aix
versions:     7300-00-01,7300-00-02
abstract:     AIX is vulnerable to a denial of service due to libxml2
apars:        CVE-2022-29824,IJ42341
fixedIn:      7300-00-04
ifixes:       IJ42341s2a.220907.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
filesets:     bos.rte.control:7.3.0.0-7.3.0.1
issued:       20220912
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/libxml2_fix3.tar
cvss:         CVE-2022-29824:5.5
reboot:       no
$

Example 3: Viewing the bulletin for APAR ID IJ42341.

$ apar bulletin IJ42341
IBM SECURITY ADVISORY

First Issued: Mon Sep 12 15:07:01 CDT 2022

The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
ftp://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc

Security Bulletin: AIX is vulnerable to a denial of service due to libxml2
    (CVE-2022-29824)
…

    REMEDIATION:

        A. APARS
           
            IBM has assigned the following APARs to this problem:

            AIX Level APAR     Availability  SP        KEY
            -----------------------------------------------------
            7.2.4     IJ42381  **            N/A       key_w_apar
            7.2.5     IJ42339  **            SP06      key_w_apar
            7.3.0     IJ42341  **            SP04      key_w_apar
…
$

Example 4: Download the fix for APAR IJ42341.

$ apar download IJ42341
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1522k      0  0:00:20  0:00:20 --:--:-- 1638k
$

The fix is downloaded to the current working directory.

Example 5: Searching for fixes for the keywords ‘memory‘ and ‘leak‘.

$ apar search memory leak
20141029  CVE-2014-3513,CVE-2014-3566,CVE-2014-3567  AIX OpenSSL Denial of Service due to memory leak in  DTLS / AIX OpenSSL Patch to mitigate CVE-2014-3566 / AIX OpenSSL Denial of Service due to memory consumption
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71219  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
$

Example 6: Checking the current system (AIX or VIOS).

# time apar check
SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   2.00
User   0.40
System 0.23
#

To check a system for fixes, root privileges are required to determine the list of installed fixes.

The check took 2 seconds and determined that only 6 out of 21 of the existing fixes are installed.

The missing fixes can be displayed using the option ‘-b‘ (brief listing) or ‘-l‘ (long listing):

# time apar check -b
20210315  sec  aix  CVE-2020-14779,CVE-2020-14781,CVE-2020-14782,CVE-2020-14796,CVE-2020-14797,CVE-2020-14798,CVE-2020-14803,CVE-2020-27221,CVE-2020-2773  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20210730  sec  aix  CVE-2021-29741,IJ30557  There is a vulnerability in Korn Shell (ksh) that affects AIX
INSTALLED: no fix installed

20210819  hiper  aix  IJ34376  Applications can terminate on systems with active IPv6 traffic
INSTALLED: no fix installed

20210825  sec  aix  CVE-2021-29727,CVE-2021-29801,CVE-2021-29862,IJ32631  There are multiple vulnerabilities in the AIX kernel
INSTALLED: no fix installed

20210915  sec  aix  CVE-2021-2161,CVE-2021-2369,CVE-2021-2432  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29860,IJ32714,IJ32736  There is a vulnerability in the libc.a library that affects AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29861,IJ35078,IJ35211  There is a vulnerability in EFS that affects AIX
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-3712  There is a vulnerability in OpenSSL used by AIX.
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-41617  Vulnerabilities in OpenSSH affect AIX.
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-2341,CVE-2021-35556,CVE-2021-35559,CVE-2021-35560,CVE-2021-35564,CVE-2021-35565,CVE-2021-35578,CVE-2021-35586,CVE-2021-41035  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-38994,CVE-2021-38995,IJ37012  There are multiple vulnerabilities in the AIX kernel.
INSTALLED: no fix installed

20220228  sec  aix  CVE-2021-38955,IJ38117,IJ38119  There is a vulnerability in the AIX audit user commands.
INSTALLED: no fix installed

20220301  sec  aix  CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512  There are multiple vulnerabilities in AIX CAA.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2021-38989,IJ37488,IJ37778  There is a vulnerability in the AIX pmsvcs kernel extension.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2022-22351,IJ36681,IJ37706  There is a vulnerability in the AIX nimsh daemon.
INSTALLED: no fix installed

SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   1.90
User   0.32
System 0.18
#

Example 7: Download all fixes for IOS version 3.1.3.21.

$ apar download 3.1.3.21
downloading lpd_fix2.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  270k  100  270k    0     0   197k      0  0:00:01  0:00:01 --:--:--  197k
downloading bind_fix21.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1498k      0  0:00:13  0:00:13 --:--:-- 1665k
downloading vios_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32.7M  100 32.7M    0     0  1571k      0  0:00:21  0:00:21 --:--:-- 1750k
downloading kernel_fix4.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  138M  100  138M    0     0  1618k      0  0:01:27  0:01:27 --:--:-- 1671k
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1537k      0  0:00:20  0:00:20 --:--:-- 1643k
$
$ ls -l
total 453952
-rw-r--r--    1 user01  staff   20080640 Sep 17 10:48 bind_fix21.tar
-rw-r--r--    1 user01  staff  145326080 Sep 17 10:50 kernel_fix4.tar
-rw-r--r--    1 user01  staff   32378880 Sep 17 10:51 libxml2_fix3.tar
-rw-r--r--    1 user01  staff     276480 Sep 17 10:48 lpd_fix2.tar
-rw-r--r--    1 user01  staff   34355200 Sep 17 10:49 vios_fix.tar
$

Similarly, all fixes for a specific AIX version can be downloaded by specifying the AIX version!

Example 8: Checking NIM clients for fixes

# apar check aix01 aix02 vios1
aix01: 13/16 fixes installed
aix02: 4/12 fixes installed (1 APAR has no fix specified)
vios1: 17/20 fixes installed (3 APARs have no fix specified)
#

Any number of NIM clients can be specified. NIM groups (mac_group) can also be specified.

Example 9: Checking a NIM client and downloading missing fixes

# apar check -d aix07
aix07: 13/16 fixes installed
downloading efs_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5010k  100 5010k    0     0  1079k      0  0:00:04  0:00:04 --:--:-- 1241k
downloading kernel_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  142M  100  142M    0     0  1637k      0  0:01:29  0:01:29 --:--:-- 1684k
downloading bind_fix20.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1494k      0  0:00:13  0:00:13 --:--:-- 1596k
#

The fixes are placed in the current directory.

Example 10: View fixes for a specific fileset

$ apar show bos.cluster.rte
type:         hiper
product:      vios
versions:     2.2.3.80,2.2.3.90
abstract:     CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLUSTER
apars:        IV97148
fixedIn:      See Advisory
ifixes:       IV97148s8a.170613.61TL09SP08.epkg.Z,IV97148s8a.170613.epkg.Z,IV97148s9b.171030.61TL09SP09.epkg.Z,IV97148s9b.171030.epkg.Z
bulletinUrl:  http://www-01.ibm.com/support/docview.wss?uid=isg1IV97148
filesets:     bos.cluster.rte:6.1.9.200-6.1.9.201
issued:       20171108
updated:      
siblings:     6100-09:IV97148 7100-04:IV97265 7200-01:IV97266
download:     https://aix.software.ibm.com/aix/ifixes/iv97148/
cvss:         
reboot:       yes
…
$

A version can also be specified:

$ apar show bos.cluster.rte:7.2.5.1
type:         sec
product:      aix
versions:     7200-05-01,7200-05-01-2038,7200-05-01-2039,7200-05-02,7200-05-02-2114,7200-05-03-2135,7200-05-03-2136,7200-05-03-2148
abstract:     There are multiple vulnerabilities in AIX CAA.
apars:        CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512
fixedIn:      7200-05-04
ifixes:       IJ36682s3a.220228.epkg.Z,IJ36682s3b.220228.epkg.Z,IJ37512s1a.220228.epkg.Z,IJ37512s2a.220228.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/caa_advisory2.asc
filesets:     bos.cluster.rte:7.2.5.0-7.2.5.1,bos.cluster.rte:7.2.5.100-7.2.5.101
issued:       20220301
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/caa_fix2.tar
cvss:         CVE-2022-22350:6.2 / CVE-2021-38996:6.2
reboot:       yes
…
$

Information about the ‘apar‘ command

The curl command is used to download files. It is available, for example, on the AIX toolbox. If curl is not installed or there is no connection to the Internet (with or without a proxy), then the download functionality of the ‘apar‘ command cannot be used. However, all other functions such as viewing APARs, checking for fixes, or searching for specific APARs can still be used without such a connection.

If a proxy is required, it can be configured using one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg, e.g.:

# The HTTP proxy to use
# Default: (none)
HttpProxy: http://172.168.10.12:3333

We recommend using the /opt/pwrcmps/etc/tools.cfg file for the proxy configuration, as this is valid for all users.

The ‘apar‘ command requires the CSV file apar.csv which contains data records of all HIPER and SECURITY fixes. This file is made available by IBM at the following URL:

https://esupport.ibm.com/customercare/flrt/doc?page=aparCSV

By default, the ‘apar‘ command first searches for this file in the user’s home directory and then under /opt/pwrcmps/etc. If the file is not available in both places, the file will be downloaded from IBM using the URL above. The behavior can be configured via one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg:

# The order of locations to look for the apar.csv file
# Default: ~,/opt/pwrcmps/etc,ibmwebsite
#AparCsvResolve:

We recommend downloading the file regularly using a crontab entry and storing it under /opt/pwrcmps/etc/apar.csv. The file can then be used by all users without having to download it again for each command call.

The download can be done using the following call:

$ apar getcsv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2834k    0 2834k    0     0  1240k      0 --:--:--  0:00:02 --:--:-- 1240k
$

The file is stored in the current directory. A crontab call from root for regular download could look like this:

( cd /opt/pwrcmps/etc; apar getcsv )

The ‘apar‘ command can be downloaded from our download area, it includes a time-limited test license for evaluation purposes.

Migration to AIX PCM combined with OS update using Alternate Disk Copy

On most AIX systems the SP or TL level is updated at regular intervals. It makes sense to perform the migration from SDDPCM to AIX PCM together with such an update. This saves time and some reboots, which otherwise have to be done because of the multipathing migration.

In our blog post “Migration from SDDPCM to AIX-PCM” we had already shown the migration for standalone systems.

Here, the migration from SDDPCM to AIX-PCM will be shown as part of an OS update, using the Alternate Disk Copy method. The procedure is roughly the following:

  1. Unmirroring the rootvg to get a free disk for Alternate Disk Copy.
  2. Change the Path Control Module (PCM) to AIX PCM.
  3. Creating the altinst_rootvg.
  4. Removal of fixes in the altinst_rootvg.
  5. Performing the OS update on the altinst_rootvg.
  6. Installing fixes in the altinst_rootvg.
  7. Adding a firstboot script to set disk attributes.
  8. Change the Path Control Module (PCM) back to SDDPCM.
  9. Booting from the altinst_rootvg.

On our example system AIX 7.1 TL5 SP2 is installed, the disks are SVC disks connected via virtual FC adapters. SDDPCM is the currently active multipathing driver:

# oslevel -s
7100-05-02-1810
# lsdev -l hdisk0 -F uniquetype
disk/fcp/2145
aix01:/root> lsattr -El hdisk0 -a PCM -F value
PCM/friend/sddpcm
#

As stated in the blog post above, some disk attributes change when migrating to AIX PCM. Therefore, you should take a close look at the current attributes in order to take them over later (at least partially). By way of example, we only look at the attribute queue_depth, which currently has the value 120:

# lsattr -El hdisk0 -a queue_depth -F value
120
#

Our system has a mirrored rootvg:

# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            active            399         232         00..01..71..80..80
hdisk0            active            399         240         00..01..79..80..80
#

The system was booted from the hdisk0:

# bootinfo -b
hdisk0
#

So we leave hdisk0 in the rootvg and remove  hdisk3 from the rootvg to get a free disk for Alternate Disk Copy.

# unmirrorvg rootvg hdisk3
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.
0516-1804 chvg: The quorum change takes effect immediately.
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk0.
# reducevg rootvg hdisk3
# chpv -c hdisk3
# bootlist -m normal hdisk0
#

Before we create a copy of the rootvg using Alternate Disk Copy, we temporarily change the system to AIX PCM without, however, rebooting. If then the altinst_rootvg is generated, the conversion to AIX PCM is already done in altinst_rootvg!

# manage_disk_drivers -d IBMSVC -o AIX_AAPCM
********************** ATTENTION *************************
  For the change to take effect the system must be rebooted
#

At the end of the OS update, we then undo this change on the rootvg to have the original state with SDDPCM.

After these preparations we start now the alt_disk_copy command:

# alt_disk_copy -d hdisk3 -B
Calling mkszfile to create new /image.data file.
Checking disk sizes.
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5.
Creating logical volume alt_hd6.
Creating logical volume alt_hd8.
…
#

Some fixes are installed on the system, which we remove from the altinst_rootvg before the update:

# emgr -l
ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ========== ======================================
1    S    102m_ifix  10/14/18 10:48:18            IFIX for Openssl CVE on 1.0.2m       
2    S    IJ03121s0a 10/14/18 10:49:04            IJ03121 for AIX 7.1 TL5 SP00         
3    S    IJ05822s2a 10/14/18 10:49:18            a potential security issue exists    
…
#

Activation of the altinst_rootvg:

# alt_rootvg_op -W -d hdisk3
Waking up altinst_rootvg volume group ...
#

And removal of the fixes:

# INUCLIENTS=1 /usr/sbin/chroot /alt_inst /usr/sbin/emgr –r -n 3
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Accessing efix metadata ...
Processing efix label "IJ05822s2a" ...
…
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log

EFIX NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 IJ05822s2a          REMOVE                 SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 2
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 1
…

(Note: please do not forget the variable INUCLIENTS, this signals that the operation is taking place in an alternate boot environment!)

Now we mount the LPP source for the OS update via NFS from our NIM server:

# mount aixnim:/export/nim/lpps/aix710503lpp /mnt
#

The OS update can now be done in the altinst_rootvg:

# alt_rootvg_op -C -b update_all -l /mnt
Installing optional filesets or updates into altinst_rootvg...
install_all_updates: Initializing system parameters.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Checking for updated install utilities on media.
…
installp:  * * * A T T E N T I O N ! ! !
        Software changes processed during this session require
        any diskless/dataless clients to which this SPOT is
        currently allocated to be rebooted.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS
#

Finally, we install some fixes. We first mount the directory /mnt with the fixes in the altinst_rootvg:

# mount -v namefs /mnt /alt_inst/mnt
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/102p_fix.181127.epkg.Z
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Efix package file is: /mnt/emgr/ppc/102p_fix.181127.epkg.Z
…
EPKG NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 102p_fix            INSTALL                SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ09621s3a.181001.epkg.Z
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ11545s0a.181127.epkg.Z
…
# umount /alt_inst/mnt
#

To set the desired disk attributes and uninstall SDDPCM we use a firstboot script:

# cat /alt_inst/etc/firstboot
#! /bin/ksh

print "INFO: adjust hdisk attributes"
chdev -Pl hdisk0 -a queue_depth=120

print "INFO: uninstalling SDDPCM"
installp -u devices.sddpcm.$(uname -v)$(uname -r).rte devices.fcp.disk.ibm.mpio.rte

print "INFO: perform reboot"
reboot

# chmod a+x /alt_inst/etc/firstboot
#

The script should, if used, be adapted to your own needs. There, you should customize all the desired disk attributes (queue_depth, reserve_policy, …). The sample script here is just to indicate what you could do!

The altinst_rootvg is now updated and converted to AIX PCM. We disable the altinst_rootvg so that it can be booted.

# alt_rootvg_op –S -t
Putting volume group altinst_rootvg to sleep ...
forced unmount of /alt_inst/var/adm/ras/livedump
…
forced unmount of /alt_inst
Fixing LV control blocks...
Fixing file system superblocks...
#

(Note: please do not forget the option “-t“, this creates a new boot image!)

But before we boot from the altinst_rootvg, we change the multipathing driver back to SDDPCM on the rootvg!

# manage_disk_drivers -d IBMSVC -o NO_OVERRIDE
********************** ATTENTION *************************
  For the change to take effect the system must be rebooted
#

Finally we change the bootlist to altinst_rootvg (hdisk3):

# bootlist -m normal hdisk3
#

And last but not least we reboot:

# shutdown –r now

SHUTDOWN PROGRAM
Tue Apr 16 19:49:08 CEST 2019

Broadcast message from root@aix01 (tty) at 19:49:08 ...

PLEASE LOG OFF NOW ! ! !
System maintenance in progress.
All processes will be killed now.
…

-------------------------------------------------------------------------------
                                Welcome to AIX.
                   boot image timestamp: 19:45:08 04/16/2019
                 The current time and date: 19:51:11 04/16/2019
        processor count: 2;  memory size: 4096MB;  kernel size: 36847630
       boot device: /vdevice/vfc-client@3000000a/disk@5005076XXXXXXXXX:2
-------------------------------------------------------------------------------
…
Multi-user initialization completed
INFO: adjust hdisk attributes
hdisk0 changed
INFO: uninstalling SDDPCM
…
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
devices.sddpcm.71.rte       2.7.1.1         ROOT        DEINSTALL   SUCCESS   
devices.sddpcm.71.rte       2.7.1.1         USR         DEINSTALL   SUCCESS   
devices.fcp.disk.ibm.mpio.r 1.0.0.25        USR         DEINSTALL   SUCCESS   
INFO: perform reboot
Rebooting . . .
…

AIX Version 7
Copyright IBM Corporation, 1982, 2018.
Console login:

(In the output you can see the actions of the firstboot script: changing disk attributes, uninstalling SDDPCM and rebooting.)

After logging in we check the OS version, the used multipathing driver and some disk attributes:

# oslevel -s
7100-05-03-1846
# lsdev -l hdisk0 -F uniquetype
disk/fcp/mpioosdisk
# lsattr -El hdisk0 -a PCM -F value
PCM/friend/fcpother
# lsattr -El hdisk0 -a queue_depth -F value
120
# genkex|grep pcm
         5ae0000    60000 /usr/lib/drivers/aixdiskpcmke
# lslpp -l|grep sddpcm
#

We have successfully completed the migration from SDDPCM to AIX PCM together with an OS update. Using scripts this can be automated further.

We have tested this procedure for AIX 7.1 and AIX 7.2. So far, we have not been able to carry out a test for PowerHA for reasons of time.

Fixes and Alternate Disk Copy

When using the Alternate Disk Copy method for AIX updates, it sometimes happens that the installed fixes prevent a successful update. In this case the installed fixes can be removed directly in the altinst_rootvg. For this the emgr command can be called directly in the altinst_rootvg using the chroot command.

After creating the altinst_rootvg, e.g. with the alt_disk_copy command, the altinst_rootvg must be activated first:

# alt_rootvg_op -W -d hdisk3
Waking up altinst_rootvg volume group ...
#

The installed fixes can be listed as follows:

# /usr/sbin/chroot /alt_inst /usr/sbin/emgr –l

ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ========== ======================================
1    S    102m_ifix  10/14/18 10:48:18            IFIX for Openssl CVE on 1.0.2m       
2    S    IJ03121s0a 10/14/18 10:49:04            IJ03121 for AIX 7.1 TL5 SP00         
3    S    IJ05822s2a 10/14/18 10:49:18            a potential security issue exists    
…

When removing fixes in altinst_rootvg, the environment variable INUCLIENTS is important. It signals the emgr command not to restart services and not to change devices dynamically. Without setting these variables, uninstalling some fixes will fail in the altinst_rootvg!

# INUCLIENTS=1 /usr/sbin/chroot /alt_inst /usr/sbin/emgr –r -n 3
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Accessing efix metadata ...
Processing efix label "IJ05822s2a" ...
...
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log

EFIX NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 IJ05822s2a          REMOVE                 SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 2
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 1
…

Now there are no fixes in the way of an OS update!

After the OS update, new fixes can be installed similarly to altinst_rootvg before rebooting. We first mount the directory with the fixes under /alt_inst/mnt:

# mount aixnim:/export/nim/lpps/aix710503lpp /alt_inst/mnt
#

And then we install the fixes directly in the altinst_rootvg, again with the help of chroot and INUCLIENTS:

# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/102p_fix.181127.epkg.Z
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Efix package file is: /mnt/emgr/ppc/102p_fix.181127.epkg.Z
…
EPKG NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 102p_fix            INSTALL                SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ09621s3a.181001.epkg.Z
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ11545s0a.181127.epkg.Z
…

Finally the altinst_rootvg has to be deactivated, and a new boot image should be created in the altinst_rootvg, otherwise the system will hang (depending on the fixes installed) when booting from altinst_rootvg!

# alt_rootvg_op –S -t
Putting volume group altinst_rootvg to sleep ...
forced unmount of /alt_inst/var/adm/ras/livedump
…
forced unmount of /alt_inst
Fixing LV control blocks...
Fixing file system superblocks...
#

(Note: The ‘-t‘ option forces the creation of a new boot image!)

Now, as usual, the boot list can be changed and the system can be rebooted after the update.