Manage and Access APARs

Keeping AIX and virtual I/O servers up to date with regard to HIPER and SECURITY fixes has become enormously important in recent years. To do this, the systems must be checked regularly for any missing fixes. The appropriate fixes must be downloaded and then installed. Determining which fix needs to be installed on a particular system often involves viewing bulletins with a web browser. PowerCampus 01 provides the ‘apar‘ command to simplify the management of fixes. This makes working with fixes and APARs as well as CVEs much easier.

Some sample uses of the ‘apar‘ command

The ‘apar‘ command allows the download of HIPER and SECURITY fixes, the checking of systems (AIX and VIOS) for installed and missing fixes, as well as the display and targeted search for fixes. In order to be able to use all functionalities, a direct Internet connection or a connection via an http proxy server is required. The command is available in versions for AIX, Linux and MacOS. A number of example calls are shown below.

Example 1: What fixes have been released in the last 30 days?

$ apar last
20220817  sec  aix  CVE-2022-1292,CVE-2022-2068,CVE-2022-2097  AIX is vulnerable to arbitrary command execution due to OpenSSL
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  aix  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42341  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  vios  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41687  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-34356,IJ41688  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-36768  AIX is vulnerable to a privilege escalation vulnerability due to invscout
$

Example 2: Displaying information about APAR ID IJ42341.

$ apar show IJ42341
type:         sec
product:      aix
versions:     7300-00-01,7300-00-02
abstract:     AIX is vulnerable to a denial of service due to libxml2
apars:        CVE-2022-29824,IJ42341
fixedIn:      7300-00-04
ifixes:       IJ42341s2a.220907.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
filesets:     bos.rte.control:7.3.0.0-7.3.0.1
issued:       20220912
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/libxml2_fix3.tar
cvss:         CVE-2022-29824:5.5
reboot:       no
$

Example 3: Viewing the bulletin for APAR ID IJ42341.

$ apar bulletin IJ42341
IBM SECURITY ADVISORY

First Issued: Mon Sep 12 15:07:01 CDT 2022

The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
ftp://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc

Security Bulletin: AIX is vulnerable to a denial of service due to libxml2
    (CVE-2022-29824)
…

    REMEDIATION:

        A. APARS
           
            IBM has assigned the following APARs to this problem:

            AIX Level APAR     Availability  SP        KEY
            -----------------------------------------------------
            7.2.4     IJ42381  **            N/A       key_w_apar
            7.2.5     IJ42339  **            SP06      key_w_apar
            7.3.0     IJ42341  **            SP04      key_w_apar
…
$

Example 4: Download the fix for APAR IJ42341.

$ apar download IJ42341
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1522k      0  0:00:20  0:00:20 --:--:-- 1638k
$

The fix is downloaded to the current working directory.

Example 5: Searching for fixes for the keywords ‘memory‘ and ‘leak‘.

$ apar search memory leak
20141029  CVE-2014-3513,CVE-2014-3566,CVE-2014-3567  AIX OpenSSL Denial of Service due to memory leak in  DTLS / AIX OpenSSL Patch to mitigate CVE-2014-3566 / AIX OpenSSL Denial of Service due to memory consumption
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71219  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
$

Example 6: Checking the current system (AIX or VIOS).

# time apar check
SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   2.00
User   0.40
System 0.23
#

To check a system for fixes, root privileges are required to determine the list of installed fixes.

The check took 2 seconds and determined that only 6 out of 21 of the existing fixes are installed.

The missing fixes can be displayed using the option ‘-b‘ (brief listing) or ‘-l‘ (long listing):

# time apar check -b
20210315  sec  aix  CVE-2020-14779,CVE-2020-14781,CVE-2020-14782,CVE-2020-14796,CVE-2020-14797,CVE-2020-14798,CVE-2020-14803,CVE-2020-27221,CVE-2020-2773  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20210730  sec  aix  CVE-2021-29741,IJ30557  There is a vulnerability in Korn Shell (ksh) that affects AIX
INSTALLED: no fix installed

20210819  hiper  aix  IJ34376  Applications can terminate on systems with active IPv6 traffic
INSTALLED: no fix installed

20210825  sec  aix  CVE-2021-29727,CVE-2021-29801,CVE-2021-29862,IJ32631  There are multiple vulnerabilities in the AIX kernel
INSTALLED: no fix installed

20210915  sec  aix  CVE-2021-2161,CVE-2021-2369,CVE-2021-2432  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29860,IJ32714,IJ32736  There is a vulnerability in the libc.a library that affects AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29861,IJ35078,IJ35211  There is a vulnerability in EFS that affects AIX
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-3712  There is a vulnerability in OpenSSL used by AIX.
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-41617  Vulnerabilities in OpenSSH affect AIX.
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-2341,CVE-2021-35556,CVE-2021-35559,CVE-2021-35560,CVE-2021-35564,CVE-2021-35565,CVE-2021-35578,CVE-2021-35586,CVE-2021-41035  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-38994,CVE-2021-38995,IJ37012  There are multiple vulnerabilities in the AIX kernel.
INSTALLED: no fix installed

20220228  sec  aix  CVE-2021-38955,IJ38117,IJ38119  There is a vulnerability in the AIX audit user commands.
INSTALLED: no fix installed

20220301  sec  aix  CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512  There are multiple vulnerabilities in AIX CAA.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2021-38989,IJ37488,IJ37778  There is a vulnerability in the AIX pmsvcs kernel extension.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2022-22351,IJ36681,IJ37706  There is a vulnerability in the AIX nimsh daemon.
INSTALLED: no fix installed

SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   1.90
User   0.32
System 0.18
#

Example 7: Download all fixes for IOS version 3.1.3.21.

$ apar download 3.1.3.21
downloading lpd_fix2.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  270k  100  270k    0     0   197k      0  0:00:01  0:00:01 --:--:--  197k
downloading bind_fix21.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1498k      0  0:00:13  0:00:13 --:--:-- 1665k
downloading vios_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32.7M  100 32.7M    0     0  1571k      0  0:00:21  0:00:21 --:--:-- 1750k
downloading kernel_fix4.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  138M  100  138M    0     0  1618k      0  0:01:27  0:01:27 --:--:-- 1671k
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1537k      0  0:00:20  0:00:20 --:--:-- 1643k
$
$ ls -l
total 453952
-rw-r--r--    1 user01  staff   20080640 Sep 17 10:48 bind_fix21.tar
-rw-r--r--    1 user01  staff  145326080 Sep 17 10:50 kernel_fix4.tar
-rw-r--r--    1 user01  staff   32378880 Sep 17 10:51 libxml2_fix3.tar
-rw-r--r--    1 user01  staff     276480 Sep 17 10:48 lpd_fix2.tar
-rw-r--r--    1 user01  staff   34355200 Sep 17 10:49 vios_fix.tar
$

Similarly, all fixes for a specific AIX version can be downloaded by specifying the AIX version!

Example 8: Checking NIM clients for fixes

# apar check aix01 aix02 vios1
aix01: 13/16 fixes installed
aix02: 4/12 fixes installed (1 APAR has no fix specified)
vios1: 17/20 fixes installed (3 APARs have no fix specified)
#

Any number of NIM clients can be specified. NIM groups (mac_group) can also be specified.

Example 9: Checking a NIM client and downloading missing fixes

# apar check -d aix07
aix07: 13/16 fixes installed
downloading efs_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5010k  100 5010k    0     0  1079k      0  0:00:04  0:00:04 --:--:-- 1241k
downloading kernel_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  142M  100  142M    0     0  1637k      0  0:01:29  0:01:29 --:--:-- 1684k
downloading bind_fix20.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1494k      0  0:00:13  0:00:13 --:--:-- 1596k
#

The fixes are placed in the current directory.

Example 10: View fixes for a specific fileset

$ apar show bos.cluster.rte
type:         hiper
product:      vios
versions:     2.2.3.80,2.2.3.90
abstract:     CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLUSTER
apars:        IV97148
fixedIn:      See Advisory
ifixes:       IV97148s8a.170613.61TL09SP08.epkg.Z,IV97148s8a.170613.epkg.Z,IV97148s9b.171030.61TL09SP09.epkg.Z,IV97148s9b.171030.epkg.Z
bulletinUrl:  http://www-01.ibm.com/support/docview.wss?uid=isg1IV97148
filesets:     bos.cluster.rte:6.1.9.200-6.1.9.201
issued:       20171108
updated:      
siblings:     6100-09:IV97148 7100-04:IV97265 7200-01:IV97266
download:     https://aix.software.ibm.com/aix/ifixes/iv97148/
cvss:         
reboot:       yes
…
$

A version can also be specified:

$ apar show bos.cluster.rte:7.2.5.1
type:         sec
product:      aix
versions:     7200-05-01,7200-05-01-2038,7200-05-01-2039,7200-05-02,7200-05-02-2114,7200-05-03-2135,7200-05-03-2136,7200-05-03-2148
abstract:     There are multiple vulnerabilities in AIX CAA.
apars:        CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512
fixedIn:      7200-05-04
ifixes:       IJ36682s3a.220228.epkg.Z,IJ36682s3b.220228.epkg.Z,IJ37512s1a.220228.epkg.Z,IJ37512s2a.220228.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/caa_advisory2.asc
filesets:     bos.cluster.rte:7.2.5.0-7.2.5.1,bos.cluster.rte:7.2.5.100-7.2.5.101
issued:       20220301
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/caa_fix2.tar
cvss:         CVE-2022-22350:6.2 / CVE-2021-38996:6.2
reboot:       yes
…
$

Information about the ‘apar‘ command

The curl command is used to download files. It is available, for example, on the AIX toolbox. If curl is not installed or there is no connection to the Internet (with or without a proxy), then the download functionality of the ‘apar‘ command cannot be used. However, all other functions such as viewing APARs, checking for fixes, or searching for specific APARs can still be used without such a connection.

If a proxy is required, it can be configured using one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg, e.g.:

# The HTTP proxy to use
# Default: (none)
HttpProxy: http://172.168.10.12:3333

We recommend using the /opt/pwrcmps/etc/tools.cfg file for the proxy configuration, as this is valid for all users.

The ‘apar‘ command requires the CSV file apar.csv which contains data records of all HIPER and SECURITY fixes. This file is made available by IBM at the following URL:

https://esupport.ibm.com/customercare/flrt/doc?page=aparCSV

By default, the ‘apar‘ command first searches for this file in the user’s home directory and then under /opt/pwrcmps/etc. If the file is not available in both places, the file will be downloaded from IBM using the URL above. The behavior can be configured via one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg:

# The order of locations to look for the apar.csv file
# Default: ~,/opt/pwrcmps/etc,ibmwebsite
#AparCsvResolve:

We recommend downloading the file regularly using a crontab entry and storing it under /opt/pwrcmps/etc/apar.csv. The file can then be used by all users without having to download it again for each command call.

The download can be done using the following call:

$ apar getcsv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2834k    0 2834k    0     0  1240k      0 --:--:--  0:00:02 --:--:-- 1240k
$

The file is stored in the current directory. A crontab call from root for regular download could look like this:

( cd /opt/pwrcmps/etc; apar getcsv )

The ‘apar‘ command can be downloaded from our download area, it includes a time-limited test license for evaluation purposes.

The Command chrctcp

chrctcp

On AIX, the System Resource Controller (SRC) is used to control subsystems and subservers. Every AIX administrator is surely familiar with the lssrc, startsrc and stopsrc commands. The chrctcp command, which is used to manage subsystems that are started via /etc/rc.tcpip, is not quite as well known. By using this command, editing the start script /etc/rc.tcpip can be avoided. Most administrators only know and use the “-a” (add/activate), “-d” (delete/deactivate) and “-S” (start, stop or restart subsystem) options. However, the chrctcp command offers even more functionality via the options “-c” (change) and “-s” (show), which SMIT also uses. (After all, chrctcp was probably written specifically for use by SMIT.)

A number of subsystems are started at boot time via /etc/rc.tcpip. However, only if there is an entry of the following form in the /etc/rc.tcpip script:

start /usr/sbin/inetd "$src_running"

If a specific subsystem is not to be started at boot time, the relevant subsystem must be commented out in /etc/rc.tcpip:

#start /usr/sbin/inetd "$src_running"

In principle, the change can be made with an editor. However, there is a much more elegant way using the chrctcp command. If a specific service should not be started at next boot, chrctcp can be used with the “-d” option to comment out the service:

# chrctcp -d inetd
# grep /usr/sbin/inet /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
#

This has no effect on the currently running subsystem. If you also want to stop the running subsystem, you can either use the “stopsrc –s” command or, which is much easier, simply use the “-S” option of chrctcp:

# chrctcp -S -d inetd
0513-044 The /usr/sbin/inetd Subsystem was requested to stop.
#

The subsystem is commented out in /etc/rc.tcpip and an eventually running subsystem is stopped as well.

You can proceed in a similar way if you want to use a previously unused service from /etc/rc.tcpip. The “-a” option can be used to activate the corresponding entry:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -a inetd
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

Note: The entry ‘#start /usr/sbin/inetd “$src_running”‘ must already be present and at the correct place in the file!

If not only the file /etc/rc.tcpip is to be adjusted, but the service is also to be started immediately, the “-S” option can be used again:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -S -a inetd
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636582.
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

In the following we will stick to inetd as an example. However, everything described applies in the same way to other subsystems.

Services such as inetd, syslogd, and others often have options and arguments that can be specified at startup. For example, inetd offers the following options and arguments:

-d   Send debugging information to syslogd

-6   IPv6 support

file  Specification of a configuration file to be used instead of /etc/inetd.conf.

The options and arguments used can be displayed with the chrctcp command and the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
no:no:
#

The sample output shows debugging and IPv6 disabled and no alternate configuration file specified.

With the option “-c” (change) you can change the options and arguments. Attributes can be specified with the “-f” option. As an example, let’s enable debugging and allow IPv6:

# chrctcp -c inetd -f debug=yes -f ipv6=yes
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running" " -d  -6 "
#

A look at /etc/rc.tcpip shows that the start of inetd has been supplemented with the options “-d” (debugging) and “-6” (IPv6)! Again, only the start file /etc/rc.tcpip has been changed, an inetd that may already be running remains unaffected by the change:

# ps -ef|grep ine[t]d
    root  5636582  4587924   0 18:34:47      -  0:00 /usr/sbin/inetd
#

As with enabling and disabling subsystems, the “-S” option can also be used when changing:

# chrctcp -S -c inetd -f debug=yes -f ipv6=yes
0513-044 The inetd Subsystem was requested to stop.
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636584.
#

The service is then stopped and restarted immediately. Then of course with the changed options and arguments:

# ps -ef|grep ine[t]d
    root  5636584  4587924   0 18:49:25      -  0:00 /usr/sbin/inetd -d -6
#

The options and arguments used at next boot can be listed with the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
yes:yes:
#

A bit unfortunate is the fact that the names shown in the header (debug_mode, ipv6_mode, config_file) do not quite match the attribute names to be used. The correct supported attribute names in the case of inetd are debug, ipv6 and file.

In one of the next posts we will list the supported attributes for further subsystems and make them available in a table.