Manage and Access APARs

Keeping AIX and virtual I/O servers up to date with regard to HIPER and SECURITY fixes has become enormously important in recent years. To do this, the systems must be checked regularly for any missing fixes. The appropriate fixes must be downloaded and then installed. Determining which fix needs to be installed on a particular system often involves viewing bulletins with a web browser. PowerCampus 01 provides the ‘apar‘ command to simplify the management of fixes. This makes working with fixes and APARs as well as CVEs much easier.

Some sample uses of the ‘apar‘ command

The ‘apar‘ command allows the download of HIPER and SECURITY fixes, the checking of systems (AIX and VIOS) for installed and missing fixes, as well as the display and targeted search for fixes. In order to be able to use all functionalities, a direct Internet connection or a connection via an http proxy server is required. The command is available in versions for AIX, Linux and MacOS. A number of example calls are shown below.

Example 1: What fixes have been released in the last 30 days?

$ apar last
20220817  sec  aix  CVE-2022-1292,CVE-2022-2068,CVE-2022-2097  AIX is vulnerable to arbitrary command execution due to OpenSSL
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  aix  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42341  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  vios  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41687  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-34356,IJ41688  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-36768  AIX is vulnerable to a privilege escalation vulnerability due to invscout
$

Example 2: Displaying information about APAR ID IJ42341.

$ apar show IJ42341
type:         sec
product:      aix
versions:     7300-00-01,7300-00-02
abstract:     AIX is vulnerable to a denial of service due to libxml2
apars:        CVE-2022-29824,IJ42341
fixedIn:      7300-00-04
ifixes:       IJ42341s2a.220907.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
filesets:     bos.rte.control:7.3.0.0-7.3.0.1
issued:       20220912
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/libxml2_fix3.tar
cvss:         CVE-2022-29824:5.5
reboot:       no
$

Example 3: Viewing the bulletin for APAR ID IJ42341.

$ apar bulletin IJ42341
IBM SECURITY ADVISORY

First Issued: Mon Sep 12 15:07:01 CDT 2022

The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
ftp://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc

Security Bulletin: AIX is vulnerable to a denial of service due to libxml2
    (CVE-2022-29824)
…

    REMEDIATION:

        A. APARS
           
            IBM has assigned the following APARs to this problem:

            AIX Level APAR     Availability  SP        KEY
            -----------------------------------------------------
            7.2.4     IJ42381  **            N/A       key_w_apar
            7.2.5     IJ42339  **            SP06      key_w_apar
            7.3.0     IJ42341  **            SP04      key_w_apar
…
$

Example 4: Download the fix for APAR IJ42341.

$ apar download IJ42341
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1522k      0  0:00:20  0:00:20 --:--:-- 1638k
$

The fix is downloaded to the current working directory.

Example 5: Searching for fixes for the keywords ‘memory‘ and ‘leak‘.

$ apar search memory leak
20141029  CVE-2014-3513,CVE-2014-3566,CVE-2014-3567  AIX OpenSSL Denial of Service due to memory leak in  DTLS / AIX OpenSSL Patch to mitigate CVE-2014-3566 / AIX OpenSSL Denial of Service due to memory consumption
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71219  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
$

Example 6: Checking the current system (AIX or VIOS).

# time apar check
SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   2.00
User   0.40
System 0.23
#

To check a system for fixes, root privileges are required to determine the list of installed fixes.

The check took 2 seconds and determined that only 6 out of 21 of the existing fixes are installed.

The missing fixes can be displayed using the option ‘-b‘ (brief listing) or ‘-l‘ (long listing):

# time apar check -b
20210315  sec  aix  CVE-2020-14779,CVE-2020-14781,CVE-2020-14782,CVE-2020-14796,CVE-2020-14797,CVE-2020-14798,CVE-2020-14803,CVE-2020-27221,CVE-2020-2773  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20210730  sec  aix  CVE-2021-29741,IJ30557  There is a vulnerability in Korn Shell (ksh) that affects AIX
INSTALLED: no fix installed

20210819  hiper  aix  IJ34376  Applications can terminate on systems with active IPv6 traffic
INSTALLED: no fix installed

20210825  sec  aix  CVE-2021-29727,CVE-2021-29801,CVE-2021-29862,IJ32631  There are multiple vulnerabilities in the AIX kernel
INSTALLED: no fix installed

20210915  sec  aix  CVE-2021-2161,CVE-2021-2369,CVE-2021-2432  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29860,IJ32714,IJ32736  There is a vulnerability in the libc.a library that affects AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29861,IJ35078,IJ35211  There is a vulnerability in EFS that affects AIX
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-3712  There is a vulnerability in OpenSSL used by AIX.
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-41617  Vulnerabilities in OpenSSH affect AIX.
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-2341,CVE-2021-35556,CVE-2021-35559,CVE-2021-35560,CVE-2021-35564,CVE-2021-35565,CVE-2021-35578,CVE-2021-35586,CVE-2021-41035  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-38994,CVE-2021-38995,IJ37012  There are multiple vulnerabilities in the AIX kernel.
INSTALLED: no fix installed

20220228  sec  aix  CVE-2021-38955,IJ38117,IJ38119  There is a vulnerability in the AIX audit user commands.
INSTALLED: no fix installed

20220301  sec  aix  CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512  There are multiple vulnerabilities in AIX CAA.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2021-38989,IJ37488,IJ37778  There is a vulnerability in the AIX pmsvcs kernel extension.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2022-22351,IJ36681,IJ37706  There is a vulnerability in the AIX nimsh daemon.
INSTALLED: no fix installed

SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   1.90
User   0.32
System 0.18
#

Example 7: Download all fixes for IOS version 3.1.3.21.

$ apar download 3.1.3.21
downloading lpd_fix2.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  270k  100  270k    0     0   197k      0  0:00:01  0:00:01 --:--:--  197k
downloading bind_fix21.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1498k      0  0:00:13  0:00:13 --:--:-- 1665k
downloading vios_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32.7M  100 32.7M    0     0  1571k      0  0:00:21  0:00:21 --:--:-- 1750k
downloading kernel_fix4.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  138M  100  138M    0     0  1618k      0  0:01:27  0:01:27 --:--:-- 1671k
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1537k      0  0:00:20  0:00:20 --:--:-- 1643k
$
$ ls -l
total 453952
-rw-r--r--    1 user01  staff   20080640 Sep 17 10:48 bind_fix21.tar
-rw-r--r--    1 user01  staff  145326080 Sep 17 10:50 kernel_fix4.tar
-rw-r--r--    1 user01  staff   32378880 Sep 17 10:51 libxml2_fix3.tar
-rw-r--r--    1 user01  staff     276480 Sep 17 10:48 lpd_fix2.tar
-rw-r--r--    1 user01  staff   34355200 Sep 17 10:49 vios_fix.tar
$

Similarly, all fixes for a specific AIX version can be downloaded by specifying the AIX version!

Example 8: Checking NIM clients for fixes

# apar check aix01 aix02 vios1
aix01: 13/16 fixes installed
aix02: 4/12 fixes installed (1 APAR has no fix specified)
vios1: 17/20 fixes installed (3 APARs have no fix specified)
#

Any number of NIM clients can be specified. NIM groups (mac_group) can also be specified.

Example 9: Checking a NIM client and downloading missing fixes

# apar check -d aix07
aix07: 13/16 fixes installed
downloading efs_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5010k  100 5010k    0     0  1079k      0  0:00:04  0:00:04 --:--:-- 1241k
downloading kernel_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  142M  100  142M    0     0  1637k      0  0:01:29  0:01:29 --:--:-- 1684k
downloading bind_fix20.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1494k      0  0:00:13  0:00:13 --:--:-- 1596k
#

The fixes are placed in the current directory.

Example 10: View fixes for a specific fileset

$ apar show bos.cluster.rte
type:         hiper
product:      vios
versions:     2.2.3.80,2.2.3.90
abstract:     CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLUSTER
apars:        IV97148
fixedIn:      See Advisory
ifixes:       IV97148s8a.170613.61TL09SP08.epkg.Z,IV97148s8a.170613.epkg.Z,IV97148s9b.171030.61TL09SP09.epkg.Z,IV97148s9b.171030.epkg.Z
bulletinUrl:  http://www-01.ibm.com/support/docview.wss?uid=isg1IV97148
filesets:     bos.cluster.rte:6.1.9.200-6.1.9.201
issued:       20171108
updated:      
siblings:     6100-09:IV97148 7100-04:IV97265 7200-01:IV97266
download:     https://aix.software.ibm.com/aix/ifixes/iv97148/
cvss:         
reboot:       yes
…
$

A version can also be specified:

$ apar show bos.cluster.rte:7.2.5.1
type:         sec
product:      aix
versions:     7200-05-01,7200-05-01-2038,7200-05-01-2039,7200-05-02,7200-05-02-2114,7200-05-03-2135,7200-05-03-2136,7200-05-03-2148
abstract:     There are multiple vulnerabilities in AIX CAA.
apars:        CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512
fixedIn:      7200-05-04
ifixes:       IJ36682s3a.220228.epkg.Z,IJ36682s3b.220228.epkg.Z,IJ37512s1a.220228.epkg.Z,IJ37512s2a.220228.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/caa_advisory2.asc
filesets:     bos.cluster.rte:7.2.5.0-7.2.5.1,bos.cluster.rte:7.2.5.100-7.2.5.101
issued:       20220301
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/caa_fix2.tar
cvss:         CVE-2022-22350:6.2 / CVE-2021-38996:6.2
reboot:       yes
…
$

Information about the ‘apar‘ command

The curl command is used to download files. It is available, for example, on the AIX toolbox. If curl is not installed or there is no connection to the Internet (with or without a proxy), then the download functionality of the ‘apar‘ command cannot be used. However, all other functions such as viewing APARs, checking for fixes, or searching for specific APARs can still be used without such a connection.

If a proxy is required, it can be configured using one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg, e.g.:

# The HTTP proxy to use
# Default: (none)
HttpProxy: http://172.168.10.12:3333

We recommend using the /opt/pwrcmps/etc/tools.cfg file for the proxy configuration, as this is valid for all users.

The ‘apar‘ command requires the CSV file apar.csv which contains data records of all HIPER and SECURITY fixes. This file is made available by IBM at the following URL:

https://esupport.ibm.com/customercare/flrt/doc?page=aparCSV

By default, the ‘apar‘ command first searches for this file in the user’s home directory and then under /opt/pwrcmps/etc. If the file is not available in both places, the file will be downloaded from IBM using the URL above. The behavior can be configured via one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg:

# The order of locations to look for the apar.csv file
# Default: ~,/opt/pwrcmps/etc,ibmwebsite
#AparCsvResolve:

We recommend downloading the file regularly using a crontab entry and storing it under /opt/pwrcmps/etc/apar.csv. The file can then be used by all users without having to download it again for each command call.

The download can be done using the following call:

$ apar getcsv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2834k    0 2834k    0     0  1240k      0 --:--:--  0:00:02 --:--:-- 1240k
$

The file is stored in the current directory. A crontab call from root for regular download could look like this:

( cd /opt/pwrcmps/etc; apar getcsv )

The ‘apar‘ command can be downloaded from our download area, it includes a time-limited test license for evaluation purposes.

Path to the executable of a running AIX process

If a process was not started with an absolute path, it is surprisingly difficult to find out the absolute path for the associated executable.

We demonstrate this using Splunk as an example:

$ ps -ef |grep splun[k]d
    root  7143802  5702116   0   Apr 23      - 23:26 splunkd --nodaemon -p 8089 _internal_exec_splunkd
    root 31916484  7143802   0   Apr 23      -  0:00 [splunkd pid=7143802] splunkd --nodaemon -p 8089 _internal_exec_splunkd [process-runner]
$

When starting the process with the PID 31916484, the argument 0 was also changed.

Part of the information about a process is available via the process file system /proc. Since Splunk runs as root, root privileges are required to access the information in /proc about the process.

Below /proc there is a subdirectory for each running process, with the PID as the directory name.

# ls -l /proc/31916484
total 32
-rw-------    1 root     system            0 Apr 28 15:17 as
-r--------    1 root     system          128 Apr 28 15:17 cred
--w-------    1 root     system            0 Apr 28 15:17 ctl
lr-x------   38 root     system            0 Apr 28 13:31 cwd -> /root/
dr-x------    1 root     system            0 Apr 28 15:17 fd
dr-xr-xr-x    1 root     system            0 Apr 28 15:17 lwp
-r--------    1 root     system            0 Apr 28 15:17 map
-r--------    1 root     system            0 Apr 28 15:17 mmap
dr-x------    1 root     system            0 Apr 28 15:17 object
-r--r--r--    1 root     system          448 Apr 28 15:17 psinfo
lr-x------   48 root     system            0 Apr 28 09:02 root -> /
-r--------    1 root     system        12288 Apr 28 15:17 sigact
-r--------    1 root     system         1520 Apr 28 15:17 status
-r--r--r--    1 root     system            0 Apr 28 15:17 sysent
#

In the subdirectory object, in addition to the open files, the executable is also available:

# ls -l /proc/31916484/object
total 854216
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  a.out
-r-xr-xr-x    1 bin      bin           10749 Sep 21 2015  jfs2.10.5.103039
-r--r--r--    1 bin      bin        12858422 May 28 2019  jfs2.10.5.225767
-r-xr-xr-x    1 bin      bin           77411 Mar 25 2021  jfs2.10.5.4157
-r-xr-xr-x    1 bin      bin        13438344 Jul 27 2021  jfs2.10.5.4205
-r--r--r--    1 bin      bin         1351386 Mar 09 2021  jfs2.10.5.4209
-r--r--r--    1 bin      bin           80450 Jul 27 2021  jfs2.10.5.4220
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  jfs2.10.9.139342
-r-xr-xr-x    1 root     system      3487967 Nov 12 2019  jfs2.10.9.155672
-r-xr-xr-x    1 root     system       228087 Nov 12 2019  jfs2.10.9.155673
-r-xr-xr-x    1 root     system      2688333 Nov 16 2019  jfs2.10.9.155675
-r-xr-xr-x    1 root     system       901800 Nov 12 2019  jfs2.10.9.155677
-r-xr-xr-x    1 root     system      4256593 Nov 12 2019  jfs2.10.9.155679
-r-xr-xr-x    1 root     system       568791 Nov 16 2019  jfs2.10.9.155684
-r-xr-xr-x    1 root     system      6166802 Nov 12 2019  jfs2.10.9.155685
-r-xr-xr-x    1 root     system      1239656 Nov 12 2019  jfs2.10.9.155686
-r-xr-xr-x    1 root     system       124218 Nov 16 2019  jfs2.10.9.155690
#

The file a.out represents the executable. Access to this file is redirected to the executable’s file. Information such as the inode number and device (file system) of the executable can be found with the istat command:

# istat /proc/31916484/object/a.out
Inode 139342 on device 10/9     File
Protection: r-xr-xr-x  
Owner: 0(root)          Group: 0(system)
Link count:   1         Length 194980846 bytes

Last updated:   Tue Jul 27 07:15:40 CEST 2021
Last modified:  Sat Nov 16 01:23:05 CET 2019
Last accessed:  Thu Apr 28 10:38:13 CEST 2022

#

The inode number of the executable is 139342. The file system is the file system on the device with major number 10 and minor number 9 (“device 10/9“).

You could now first find out the device by searching under /dev:

$ ls -l /dev | grep "10,  9"
brw-rw----    1 root     system       10,  9 Jul 27 2021  hd10opt
crw-rw----    1 root     system       10,  9 Apr 05 13:36 rhd10opt
$

and then determine the associated file system via df:

$ df | grep hd10opt
/dev/hd10opt    12582912   7527920   41%    33349     4% /opt
$

However, this is even easier. You can simply specify the path for the a.out file in df:

# df /proc/31916484/object/a.out
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd10opt    12582912   7527912   41%    33349     4% /opt
#

The file system we are looking for is the /opt file system. Now you can find out the absolute path to the executable by searching for the inode number 139342 in /opt:

# find /opt -inum 139342
/opt/splunkforwarder/bin/splunkd
#

The process with the PID 31916484 runs the executable /opt/splunkforwarder/bin/splunkd.

With a little trick you can also shorten the search significantly. For this you need a shell as user root. In this shell you open the a.out explicitly with exec and a (free) descriptor number:

# exec 5</proc/31916484/object/a.out
#

Our current shell has thus opened the executable (via the file descriptor 5)! Now you can use the procfiles command to display the open files in this shell. Using the “-n” option, which displays the absolute paths of files that belong to a file descriptor:

# procfiles -n $$
19136808 : ksh
  Current rlimit: 10000 file descriptors
   0: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   1: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   2: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   5: S_IFREG mode:0555 dev:10,9 ino:139342 uid:0 gid:0 rdev:0,0
      O_RDONLY size:194980846  name:/opt/splunkforwarder/bin/splunkd
   10: S_IFREG mode:0444 dev:10,5 ino:124151 uid:0 gid:0 rdev:0,0
      O_RDONLY size:5875  name:/usr/lib/nls/msg/EN_US/ksh.cat
   63: S_IFREG mode:0600 dev:10,4 ino:41933 uid:0 gid:0 rdev:0,0
      O_RDWR | O_APPEND size:5494  name://root/.sh_history
#

Note: The special variable $$ is replaced by the shell with the shell’s PID.

For file descriptor 5, the absolute path /opt/splunkforwarder/bin/splunkd is displayed.

The file descriptor should of course be closed again:

# exec 5<&-
#