CVE-2021-25220: AIX is vulnerable to cache poisoning due to ISC BIND

CVE-2021-25220 describes a vulnerability in ISC BIND. Using our tool “apar“, some questions are examined and answered below, such as: is my system affected by this vulnerability, where can I find a more detailed description of the vulnerability, where can I find a fix to close the vulnerability, are there other vulnerabilities of which my system is affected?

Note: The “apar” tool is available in our download area in versions for AIX (VIOS), Linux and MacOS. It includes a time-limited trial license. See the Manage and Access APARs for more information on using the tool.

Is my system affected by this vulnerability?

Information about the vulnerability can be displayed using the “apar show” command and the “CVE-2021-25220” argument:

$ apar show CVE-2021-25220
type:         sec
product:      aix
versions:     7300-00-01,7300-00-02
abstract:     AIX is vulnerable to cache poisoning due to ISC BIND
apars:        CVE-2021-25220,IJ40614
fixedIn:      7300-00-03
ifixes:       IJ40614m2b.220718.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc
filesets:     bos.net.tcp.bind:7.3.0.0-7.3.0.1,bos.net.tcp.bind_utils:7.3.0.0-7.3.0.1
issued:       20220728
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/bind_fix21.tar
cvss:         CVE-2021-25220:6.8
reboot:       no
…
$

Multiple records are displayed. There are separate records for different AIX and VIOS versions. Each record contains a line with the associated AIX or VIOS versions (line “versions: …”). In addition, the affected filesets are listed, including the version (line “filesets: …”). If, for example, AIX 7300-00-01 or 7300-00-02 is installed on my system (command “oslevel –s”) and I have one of the fileset versions listed (command “lslpp –l bos.net.tcp.bind bos.net .tcp.bind_utils“), then my system is affected by the vulnerability.

Where can I find a more detailed description of the vulnerability?

IBM typically offers more detailed information about a vulnerability via a so-called bulletin. The URL for the bulletin is shown in the output of “apar show” (above) on the line beginning with “bulletinUrl: …”. In the case above, this is https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc . This URL can be specified in a browser. When using the “apar” command, the bulletin can also be displayed directly on the command line, this can be done with the command “apar bulletin” and the number of the CVE (here CVE-2021-25220) or the fix or APAR number ( e.g. IJ40614):

$ apar bulletin CVE-2021-25220
IBM SECURITY ADVISORY

First Issued: Thu Jul 28 13:24:22 CDT 2022

The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc
https://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc
ftp://aix.software.ibm.com/aix/efixes/security/bind_advisory21.asc

Security Bulletin: AIX is vulnerable to cache poisoning due to ISC BIND
    (CVE-2021-25220)

===============================================================================

SUMMARY:

    A vulnerability in ISC BIND could allow a remote attacker to poison the
    cache (CVE-2021-25220). AIX uses ISC BIND as part of its DNS functions.


===============================================================================

VULNERABILITY DETAILS:

    CVEID: CVE-2021-25220
        http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25220
        https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25220
    DESCRIPTION: ISC BIND could allow a remote attacker to bypass security
        restrictions, caused by an error when using DNS forwarders. An
        attacker could exploit this vulnerability to poison the cache with
        incorrect records leading to queries being made to the wrong servers,
        which might also result in false information being returned to
        clients.
    CVSS Base Score: 6.8
    CVSS Temporal Score: See
        https://exchange.xforce.ibmcloud.com/vulnerabilities/221991
        for the current score
    CVSS Environmental Score*: Undefined
    CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:C/C:N/I:H/A:N)
…
$

All associated APARs are usually listed in the bulletin. You will also find an overview of the fixes and corresponding versions.

Where can I find a fix to close the vulnerability?

In the records above, you will also find a listing of the associated fixes in the line beginning with “ifixes: …”. In the case mentioned, this is the fix IJ40614m2b.220718.epkg.Z. In many cases, several fixes are listed and you have to select the correct fix from the list. The description in the bulletin is helpful here, with a list of which fix is to be used for which version.

The URL for downloading the fix(s) is given in the line beginning with “download: …”, in the current case this is the following URL:

https://aix.software.ibm.com/aix/efixes/security/bind_fix21.tar

The fix can be downloaded with a browser, for example. When using the “apar” command, however, this is even easier using the command line. The “apar” command can be invoked with the argument “download” and the CVE number or fix number. Then it downloads the fix and stores it in the current working directory:

$ apar download CVE-2021-25220
downloading bind_fix21.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1480k      0  0:00:13  0:00:13 --:--:-- 1672k
$

The fix is saved under the name used in the URL, here bind_fix21.tar.

Are there other vulnerabilities affecting my system?

The command “apar check” can be used to examine a system for known vulnerabilities. In order for the command to be able to access the information about installed fixes, the command must be started with root privileges.

Here is an example of a system with all relevant fixes installed:

aix01 # apar check
SUMMARY: 2/2 fixes installed
aix01 #

And below is an example of a system with only a few relevant fixes installed:

aix02 # apar check
SUMMARY: 4/8 fixes installed (2 APARs have no fix specified)
aix02 #

Of the 8 known (IBM disclosed) vulnerabilities, only 4 of the vulnerabilities have the associated fixes installed. If you want to know which vulnerabilities are open, one of the options “-b” (brief report) or “-l” (long report) can be used:

aix02 # apar check -b
20220817  sec  aix  CVE-2022-1292,CVE-2022-2068,CVE-2022-2097  AIX is vulnerable to arbitrary command execution due to OpenSSL
INSTALLED: no fix installed

20220912  sec  aix  CVE-2022-36768  AIX is vulnerable to a privilege escalation vulnerability due to invscout
INSTALLED: no fix installed

20220923  sec  aix  CVE-2021-20266,CVE-2021-20271,CVE-2021-3421  AIX is vulnerable to arbitrary code execution and RPM database corruption and denial of service due to RPM.
INSTALLED: no fix installed

20220928  sec  aix  CVE-2018-25032  AIX is vulnerable to denial of service due to zlib and zlibNX
INSTALLED: no fix installed

SUMMARY: 4/8 fixes installed (2 APARs have no fix specified)
aix02 #

The “apar” command supports further options, which are described in Manage and Access APARs.

Manage and Access APARs

Keeping AIX and virtual I/O servers up to date with regard to HIPER and SECURITY fixes has become enormously important in recent years. To do this, the systems must be checked regularly for any missing fixes. The appropriate fixes must be downloaded and then installed. Determining which fix needs to be installed on a particular system often involves viewing bulletins with a web browser. PowerCampus 01 provides the ‘apar‘ command to simplify the management of fixes. This makes working with fixes and APARs as well as CVEs much easier.

Some sample uses of the ‘apar‘ command

The ‘apar‘ command allows the download of HIPER and SECURITY fixes, the checking of systems (AIX and VIOS) for installed and missing fixes, as well as the display and targeted search for fixes. In order to be able to use all functionalities, a direct Internet connection or a connection via an http proxy server is required. The command is available in versions for AIX, Linux and MacOS. A number of example calls are shown below.

Example 1: What fixes have been released in the last 30 days?

$ apar last
20220817  sec  aix  CVE-2022-1292,CVE-2022-2068,CVE-2022-2097  AIX is vulnerable to arbitrary command execution due to OpenSSL
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  aix  CVE-2022-29824,IJ42339,IJ42378,IJ42379  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42341  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  aix  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2
20220912  sec  vios  CVE-2022-29824,IJ42381  AIX is vulnerable to a denial of service due to libxml2 for VIOS
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41396,IJ41685,IJ41795  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41687  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-34356,IJ41688  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  vios  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability for VIOS
20220912  sec  aix  CVE-2022-34356,IJ41706  AIX kernel is vulnerable to a privilege escalation vulnerability
20220912  sec  aix  CVE-2022-36768  AIX is vulnerable to a privilege escalation vulnerability due to invscout
$

Example 2: Displaying information about APAR ID IJ42341.

$ apar show IJ42341
type:         sec
product:      aix
versions:     7300-00-01,7300-00-02
abstract:     AIX is vulnerable to a denial of service due to libxml2
apars:        CVE-2022-29824,IJ42341
fixedIn:      7300-00-04
ifixes:       IJ42341s2a.220907.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
filesets:     bos.rte.control:7.3.0.0-7.3.0.1
issued:       20220912
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/libxml2_fix3.tar
cvss:         CVE-2022-29824:5.5
reboot:       no
$

Example 3: Viewing the bulletin for APAR ID IJ42341.

$ apar bulletin IJ42341
IBM SECURITY ADVISORY

First Issued: Mon Sep 12 15:07:01 CDT 2022

The most recent version of this document is available here:
http://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
https://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc
ftp://aix.software.ibm.com/aix/efixes/security/libxml2_advisory3.asc

Security Bulletin: AIX is vulnerable to a denial of service due to libxml2
    (CVE-2022-29824)
…

    REMEDIATION:

        A. APARS
           
            IBM has assigned the following APARs to this problem:

            AIX Level APAR     Availability  SP        KEY
            -----------------------------------------------------
            7.2.4     IJ42381  **            N/A       key_w_apar
            7.2.5     IJ42339  **            SP06      key_w_apar
            7.3.0     IJ42341  **            SP04      key_w_apar
…
$

Example 4: Download the fix for APAR IJ42341.

$ apar download IJ42341
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1522k      0  0:00:20  0:00:20 --:--:-- 1638k
$

The fix is downloaded to the current working directory.

Example 5: Searching for fixes for the keywords ‘memory‘ and ‘leak‘.

$ apar search memory leak
20141029  CVE-2014-3513,CVE-2014-3566,CVE-2014-3567  AIX OpenSSL Denial of Service due to memory leak in  DTLS / AIX OpenSSL Patch to mitigate CVE-2014-3566 / AIX OpenSSL Denial of Service due to memory consumption
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71217  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
20150319  IV71219  NODE DOWN IN CAA CLUSTER DUE TO CONFIGRM MEMORY LEAK
$

Example 6: Checking the current system (AIX or VIOS).

# time apar check
SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   2.00
User   0.40
System 0.23
#

To check a system for fixes, root privileges are required to determine the list of installed fixes.

The check took 2 seconds and determined that only 6 out of 21 of the existing fixes are installed.

The missing fixes can be displayed using the option ‘-b‘ (brief listing) or ‘-l‘ (long listing):

# time apar check -b
20210315  sec  aix  CVE-2020-14779,CVE-2020-14781,CVE-2020-14782,CVE-2020-14796,CVE-2020-14797,CVE-2020-14798,CVE-2020-14803,CVE-2020-27221,CVE-2020-2773  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20210730  sec  aix  CVE-2021-29741,IJ30557  There is a vulnerability in Korn Shell (ksh) that affects AIX
INSTALLED: no fix installed

20210819  hiper  aix  IJ34376  Applications can terminate on systems with active IPv6 traffic
INSTALLED: no fix installed

20210825  sec  aix  CVE-2021-29727,CVE-2021-29801,CVE-2021-29862,IJ32631  There are multiple vulnerabilities in the AIX kernel
INSTALLED: no fix installed

20210915  sec  aix  CVE-2021-2161,CVE-2021-2369,CVE-2021-2432  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29860,IJ32714,IJ32736  There is a vulnerability in the libc.a library that affects AIX
INSTALLED: no fix installed

20211116  sec  aix  CVE-2021-29861,IJ35078,IJ35211  There is a vulnerability in EFS that affects AIX
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-3712  There is a vulnerability in OpenSSL used by AIX.
INSTALLED: no fix installed

20220106  sec  aix  CVE-2021-41617  Vulnerabilities in OpenSSH affect AIX.
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-2341,CVE-2021-35556,CVE-2021-35559,CVE-2021-35560,CVE-2021-35564,CVE-2021-35565,CVE-2021-35578,CVE-2021-35586,CVE-2021-41035  Multiple vulnerabilities in IBM Java SDK affect AIX
INSTALLED: no fix installed

20220223  sec  aix  CVE-2021-38994,CVE-2021-38995,IJ37012  There are multiple vulnerabilities in the AIX kernel.
INSTALLED: no fix installed

20220228  sec  aix  CVE-2021-38955,IJ38117,IJ38119  There is a vulnerability in the AIX audit user commands.
INSTALLED: no fix installed

20220301  sec  aix  CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512  There are multiple vulnerabilities in AIX CAA.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2021-38989,IJ37488,IJ37778  There is a vulnerability in the AIX pmsvcs kernel extension.
INSTALLED: no fix installed

20220304  sec  aix  CVE-2022-22351,IJ36681,IJ37706  There is a vulnerability in the AIX nimsh daemon.
INSTALLED: no fix installed

SUMMARY: 6/21 fixes installed (3 APARs have no fix specified)

Real   1.90
User   0.32
System 0.18
#

Example 7: Download all fixes for IOS version 3.1.3.21.

$ apar download 3.1.3.21
downloading lpd_fix2.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  270k  100  270k    0     0   197k      0  0:00:01  0:00:01 --:--:--  197k
downloading bind_fix21.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1498k      0  0:00:13  0:00:13 --:--:-- 1665k
downloading vios_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32.7M  100 32.7M    0     0  1571k      0  0:00:21  0:00:21 --:--:-- 1750k
downloading kernel_fix4.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  138M  100  138M    0     0  1618k      0  0:01:27  0:01:27 --:--:-- 1671k
downloading libxml2_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.8M  100 30.8M    0     0  1537k      0  0:00:20  0:00:20 --:--:-- 1643k
$
$ ls -l
total 453952
-rw-r--r--    1 user01  staff   20080640 Sep 17 10:48 bind_fix21.tar
-rw-r--r--    1 user01  staff  145326080 Sep 17 10:50 kernel_fix4.tar
-rw-r--r--    1 user01  staff   32378880 Sep 17 10:51 libxml2_fix3.tar
-rw-r--r--    1 user01  staff     276480 Sep 17 10:48 lpd_fix2.tar
-rw-r--r--    1 user01  staff   34355200 Sep 17 10:49 vios_fix.tar
$

Similarly, all fixes for a specific AIX version can be downloaded by specifying the AIX version!

Example 8: Checking NIM clients for fixes

# apar check aix01 aix02 vios1
aix01: 13/16 fixes installed
aix02: 4/12 fixes installed (1 APAR has no fix specified)
vios1: 17/20 fixes installed (3 APARs have no fix specified)
#

Any number of NIM clients can be specified. NIM groups (mac_group) can also be specified.

Example 9: Checking a NIM client and downloading missing fixes

# apar check -d aix07
aix07: 13/16 fixes installed
downloading efs_fix.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5010k  100 5010k    0     0  1079k      0  0:00:04  0:00:04 --:--:-- 1241k
downloading kernel_fix3.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  142M  100  142M    0     0  1637k      0  0:01:29  0:01:29 --:--:-- 1684k
downloading bind_fix20.tar ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  1494k      0  0:00:13  0:00:13 --:--:-- 1596k
#

The fixes are placed in the current directory.

Example 10: View fixes for a specific fileset

$ apar show bos.cluster.rte
type:         hiper
product:      vios
versions:     2.2.3.80,2.2.3.90
abstract:     CAA:SLOW GOSSIP RECEIPT ON BOOT MAY CAUSE PARTITIONED CLUSTER
apars:        IV97148
fixedIn:      See Advisory
ifixes:       IV97148s8a.170613.61TL09SP08.epkg.Z,IV97148s8a.170613.epkg.Z,IV97148s9b.171030.61TL09SP09.epkg.Z,IV97148s9b.171030.epkg.Z
bulletinUrl:  http://www-01.ibm.com/support/docview.wss?uid=isg1IV97148
filesets:     bos.cluster.rte:6.1.9.200-6.1.9.201
issued:       20171108
updated:      
siblings:     6100-09:IV97148 7100-04:IV97265 7200-01:IV97266
download:     https://aix.software.ibm.com/aix/ifixes/iv97148/
cvss:         
reboot:       yes
…
$

A version can also be specified:

$ apar show bos.cluster.rte:7.2.5.1
type:         sec
product:      aix
versions:     7200-05-01,7200-05-01-2038,7200-05-01-2039,7200-05-02,7200-05-02-2114,7200-05-03-2135,7200-05-03-2136,7200-05-03-2148
abstract:     There are multiple vulnerabilities in AIX CAA.
apars:        CVE-2021-38996,CVE-2022-22350,IJ36682,IJ37512
fixedIn:      7200-05-04
ifixes:       IJ36682s3a.220228.epkg.Z,IJ36682s3b.220228.epkg.Z,IJ37512s1a.220228.epkg.Z,IJ37512s2a.220228.epkg.Z
bulletinUrl:  https://aix.software.ibm.com/aix/efixes/security/caa_advisory2.asc
filesets:     bos.cluster.rte:7.2.5.0-7.2.5.1,bos.cluster.rte:7.2.5.100-7.2.5.101
issued:       20220301
updated:      
siblings:    
download:     https://aix.software.ibm.com/aix/efixes/security/caa_fix2.tar
cvss:         CVE-2022-22350:6.2 / CVE-2021-38996:6.2
reboot:       yes
…
$

Information about the ‘apar‘ command

The curl command is used to download files. It is available, for example, on the AIX toolbox. If curl is not installed or there is no connection to the Internet (with or without a proxy), then the download functionality of the ‘apar‘ command cannot be used. However, all other functions such as viewing APARs, checking for fixes, or searching for specific APARs can still be used without such a connection.

If a proxy is required, it can be configured using one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg, e.g.:

# The HTTP proxy to use
# Default: (none)
HttpProxy: http://172.168.10.12:3333

We recommend using the /opt/pwrcmps/etc/tools.cfg file for the proxy configuration, as this is valid for all users.

The ‘apar‘ command requires the CSV file apar.csv which contains data records of all HIPER and SECURITY fixes. This file is made available by IBM at the following URL:

https://esupport.ibm.com/customercare/flrt/doc?page=aparCSV

By default, the ‘apar‘ command first searches for this file in the user’s home directory and then under /opt/pwrcmps/etc. If the file is not available in both places, the file will be downloaded from IBM using the URL above. The behavior can be configured via one of the two files /opt/pwrcmps/etc/tools.cfg or ~/.tools.cfg:

# The order of locations to look for the apar.csv file
# Default: ~,/opt/pwrcmps/etc,ibmwebsite
#AparCsvResolve:

We recommend downloading the file regularly using a crontab entry and storing it under /opt/pwrcmps/etc/apar.csv. The file can then be used by all users without having to download it again for each command call.

The download can be done using the following call:

$ apar getcsv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2834k    0 2834k    0     0  1240k      0 --:--:--  0:00:02 --:--:-- 1240k
$

The file is stored in the current directory. A crontab call from root for regular download could look like this:

( cd /opt/pwrcmps/etc; apar getcsv )

The ‘apar‘ command can be downloaded from our download area, it includes a time-limited test license for evaluation purposes.

SSHD: Effective Values of Keywords

By default, the sshd service uses the configuration file /etc/ssh/sshd_config. There the service can be configured in detail using keyword/value pairs. There are around 80 such keywords (depending on the SSH version). If a keyword is not listed in the configuration file, then a default value applies. However, this can depend on the version of SSH used. Then it is not always clear what value a certain keyword has. But that can be found out very easily with the help of sshd itself! There is an “extended test mode” for this, which can be started with the “-T” option. Sshd then checks the validity of the current configuration, prints the effective configuration, and then exits:

# sshd -T
port 22
addressfamily inet
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
x11maxdisplays 1000
maxauthtries 6
maxsessions 1024
clientaliveinterval 60
clientalivecountmax 3
streamlocalbindmask 0177
permitrootlogin forced-commands-only
…
#

Root privileges are required.

This makes it very easy to determine the effective value of any keyword.

NTP Configuration on HMC with LPAR tool

NTP-Konfiguration HMC

The current status of NTP on an HMC can then be displayed using “hmc lsntp“:

$ hmc lsntp
NAME   XNTP     XNTPSTATUS    XNTPSERVER
hmc01  disable  -             -
hmc02  enable   synchronized  192.168.189.77,192.168.189.78
hmc03  enable   synchronized  192.168.189.77,192.168.189.78
$

Another NTP server can be added to the NTP configuration of an HMC using the “hmc addntpserver” command:

$ hmc addntpserver hmc01 192.168.189.77
$ hmc addntpserver hmc01 192.168.189.78
$

A check with “hmc lsntp” shows that two NTP servers are now configured, but NTP is still not activated:

$ hmc lsntp hmc01
NAME   XNTP     XNTPSTATUS  XNTPSERVER
hmc01  disable  -           192.168.189.77,192.168.189.78
$

NTP can now be activated with the command “hmc enablentp“:

$ hmc enablentp hmc01
$

The first sync may take a while:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS      XNTPSERVER
hmc01  enable  unsynchronized  192.168.189.77,192.168.189.78
$

The time on the HMC is not yet synchronized immediately after enabling NTP (XNTPSTATUS: unsynchronized).

A detailed status for each NTP server can be obtained by using the “-a” option (all NTP servers):

$ hmc lsntp –a hmc01
NAME    SERVER         STATE          POLL_FREQ_SECONDS  SECONDS_SINCE_LAST_POLL
hmc01  192.168.189.77  not connected  64                 0
hmc01  192.168.189.78  not connected  64                 0
$

As soon as synchronization with one of the NTP servers is achieved, the overall status is synchronized:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS    XNTPSERVER
hmc01  enable  synchronized  192.168.189.77,192.168.189.78
$

A more detailed description can be found here: NTP Configuration on the HMC

The LPAR tool can be downloaded for testing here: Download

History Expansion bash

Drawing Shell

Many AIX and UNIX users use bash as their preferred shell. Navigating in the history with the cursor keys is certainly used countless times a day by all users. As long as the interesting commands are only a short time ago, this works very well. However, for commands longer in the past, access using the cursor keys is relatively time-consuming. Who wants to press the cursor keys 50 times to access a command?

The bash history expansion mechanism offers a much more efficient option here. Previous commands can be accessed using the history expansion character “!“. The commands can be specified in different ways:

    • The number of the command: !31
    • The nth previous command: !-n (e.g. !-3 for the third last command)
    • The last command that begins with a specific character string: !ca
    • The last command that has a specific character string anywhere: !?ca

However, the possibilities of the bash are far from exhausted. You can specifically access individual arguments of a previous command and even make changes.

Here are a few of those options:

    • !! (run the last command again)
    • ^op^art (run the last command again, but replace “op” with “art“)
    • cat !?sam?:% (run the cat command on the last argument containing the string “sam“)
    • vi !$ (Run vi on the last argument of the last command)

A description of these and other bash options can be found here:

The bash History Expansion

The Command chrctcp

chrctcp

On AIX, the System Resource Controller (SRC) is used to control subsystems and subservers. Every AIX administrator is surely familiar with the lssrc, startsrc and stopsrc commands. The chrctcp command, which is used to manage subsystems that are started via /etc/rc.tcpip, is not quite as well known. By using this command, editing the start script /etc/rc.tcpip can be avoided. Most administrators only know and use the “-a” (add/activate), “-d” (delete/deactivate) and “-S” (start, stop or restart subsystem) options. However, the chrctcp command offers even more functionality via the options “-c” (change) and “-s” (show), which SMIT also uses. (After all, chrctcp was probably written specifically for use by SMIT.)

A number of subsystems are started at boot time via /etc/rc.tcpip. However, only if there is an entry of the following form in the /etc/rc.tcpip script:

start /usr/sbin/inetd "$src_running"

If a specific subsystem is not to be started at boot time, the relevant subsystem must be commented out in /etc/rc.tcpip:

#start /usr/sbin/inetd "$src_running"

In principle, the change can be made with an editor. However, there is a much more elegant way using the chrctcp command. If a specific service should not be started at next boot, chrctcp can be used with the “-d” option to comment out the service:

# chrctcp -d inetd
# grep /usr/sbin/inet /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
#

This has no effect on the currently running subsystem. If you also want to stop the running subsystem, you can either use the “stopsrc –s” command or, which is much easier, simply use the “-S” option of chrctcp:

# chrctcp -S -d inetd
0513-044 The /usr/sbin/inetd Subsystem was requested to stop.
#

The subsystem is commented out in /etc/rc.tcpip and an eventually running subsystem is stopped as well.

You can proceed in a similar way if you want to use a previously unused service from /etc/rc.tcpip. The “-a” option can be used to activate the corresponding entry:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -a inetd
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

Note: The entry ‘#start /usr/sbin/inetd “$src_running”‘ must already be present and at the correct place in the file!

If not only the file /etc/rc.tcpip is to be adjusted, but the service is also to be started immediately, the “-S” option can be used again:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -S -a inetd
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636582.
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

In the following we will stick to inetd as an example. However, everything described applies in the same way to other subsystems.

Services such as inetd, syslogd, and others often have options and arguments that can be specified at startup. For example, inetd offers the following options and arguments:

-d   Send debugging information to syslogd

-6   IPv6 support

file  Specification of a configuration file to be used instead of /etc/inetd.conf.

The options and arguments used can be displayed with the chrctcp command and the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
no:no:
#

The sample output shows debugging and IPv6 disabled and no alternate configuration file specified.

With the option “-c” (change) you can change the options and arguments. Attributes can be specified with the “-f” option. As an example, let’s enable debugging and allow IPv6:

# chrctcp -c inetd -f debug=yes -f ipv6=yes
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running" " -d  -6 "
#

A look at /etc/rc.tcpip shows that the start of inetd has been supplemented with the options “-d” (debugging) and “-6” (IPv6)! Again, only the start file /etc/rc.tcpip has been changed, an inetd that may already be running remains unaffected by the change:

# ps -ef|grep ine[t]d
    root  5636582  4587924   0 18:34:47      -  0:00 /usr/sbin/inetd
#

As with enabling and disabling subsystems, the “-S” option can also be used when changing:

# chrctcp -S -c inetd -f debug=yes -f ipv6=yes
0513-044 The inetd Subsystem was requested to stop.
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636584.
#

The service is then stopped and restarted immediately. Then of course with the changed options and arguments:

# ps -ef|grep ine[t]d
    root  5636584  4587924   0 18:49:25      -  0:00 /usr/sbin/inetd -d -6
#

The options and arguments used at next boot can be listed with the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
yes:yes:
#

A bit unfortunate is the fact that the names shown in the header (debug_mode, ipv6_mode, config_file) do not quite match the attribute names to be used. The correct supported attribute names in the case of inetd are debug, ipv6 and file.

In one of the next posts we will list the supported attributes for further subsystems and make them available in a table.

Path to the executable of a running AIX process

If a process was not started with an absolute path, it is surprisingly difficult to find out the absolute path for the associated executable.

We demonstrate this using Splunk as an example:

$ ps -ef |grep splun[k]d
    root  7143802  5702116   0   Apr 23      - 23:26 splunkd --nodaemon -p 8089 _internal_exec_splunkd
    root 31916484  7143802   0   Apr 23      -  0:00 [splunkd pid=7143802] splunkd --nodaemon -p 8089 _internal_exec_splunkd [process-runner]
$

When starting the process with the PID 31916484, the argument 0 was also changed.

Part of the information about a process is available via the process file system /proc. Since Splunk runs as root, root privileges are required to access the information in /proc about the process.

Below /proc there is a subdirectory for each running process, with the PID as the directory name.

# ls -l /proc/31916484
total 32
-rw-------    1 root     system            0 Apr 28 15:17 as
-r--------    1 root     system          128 Apr 28 15:17 cred
--w-------    1 root     system            0 Apr 28 15:17 ctl
lr-x------   38 root     system            0 Apr 28 13:31 cwd -> /root/
dr-x------    1 root     system            0 Apr 28 15:17 fd
dr-xr-xr-x    1 root     system            0 Apr 28 15:17 lwp
-r--------    1 root     system            0 Apr 28 15:17 map
-r--------    1 root     system            0 Apr 28 15:17 mmap
dr-x------    1 root     system            0 Apr 28 15:17 object
-r--r--r--    1 root     system          448 Apr 28 15:17 psinfo
lr-x------   48 root     system            0 Apr 28 09:02 root -> /
-r--------    1 root     system        12288 Apr 28 15:17 sigact
-r--------    1 root     system         1520 Apr 28 15:17 status
-r--r--r--    1 root     system            0 Apr 28 15:17 sysent
#

In the subdirectory object, in addition to the open files, the executable is also available:

# ls -l /proc/31916484/object
total 854216
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  a.out
-r-xr-xr-x    1 bin      bin           10749 Sep 21 2015  jfs2.10.5.103039
-r--r--r--    1 bin      bin        12858422 May 28 2019  jfs2.10.5.225767
-r-xr-xr-x    1 bin      bin           77411 Mar 25 2021  jfs2.10.5.4157
-r-xr-xr-x    1 bin      bin        13438344 Jul 27 2021  jfs2.10.5.4205
-r--r--r--    1 bin      bin         1351386 Mar 09 2021  jfs2.10.5.4209
-r--r--r--    1 bin      bin           80450 Jul 27 2021  jfs2.10.5.4220
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  jfs2.10.9.139342
-r-xr-xr-x    1 root     system      3487967 Nov 12 2019  jfs2.10.9.155672
-r-xr-xr-x    1 root     system       228087 Nov 12 2019  jfs2.10.9.155673
-r-xr-xr-x    1 root     system      2688333 Nov 16 2019  jfs2.10.9.155675
-r-xr-xr-x    1 root     system       901800 Nov 12 2019  jfs2.10.9.155677
-r-xr-xr-x    1 root     system      4256593 Nov 12 2019  jfs2.10.9.155679
-r-xr-xr-x    1 root     system       568791 Nov 16 2019  jfs2.10.9.155684
-r-xr-xr-x    1 root     system      6166802 Nov 12 2019  jfs2.10.9.155685
-r-xr-xr-x    1 root     system      1239656 Nov 12 2019  jfs2.10.9.155686
-r-xr-xr-x    1 root     system       124218 Nov 16 2019  jfs2.10.9.155690
#

The file a.out represents the executable. Access to this file is redirected to the executable’s file. Information such as the inode number and device (file system) of the executable can be found with the istat command:

# istat /proc/31916484/object/a.out
Inode 139342 on device 10/9     File
Protection: r-xr-xr-x  
Owner: 0(root)          Group: 0(system)
Link count:   1         Length 194980846 bytes

Last updated:   Tue Jul 27 07:15:40 CEST 2021
Last modified:  Sat Nov 16 01:23:05 CET 2019
Last accessed:  Thu Apr 28 10:38:13 CEST 2022

#

The inode number of the executable is 139342. The file system is the file system on the device with major number 10 and minor number 9 (“device 10/9“).

You could now first find out the device by searching under /dev:

$ ls -l /dev | grep "10,  9"
brw-rw----    1 root     system       10,  9 Jul 27 2021  hd10opt
crw-rw----    1 root     system       10,  9 Apr 05 13:36 rhd10opt
$

and then determine the associated file system via df:

$ df | grep hd10opt
/dev/hd10opt    12582912   7527920   41%    33349     4% /opt
$

However, this is even easier. You can simply specify the path for the a.out file in df:

# df /proc/31916484/object/a.out
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd10opt    12582912   7527912   41%    33349     4% /opt
#

The file system we are looking for is the /opt file system. Now you can find out the absolute path to the executable by searching for the inode number 139342 in /opt:

# find /opt -inum 139342
/opt/splunkforwarder/bin/splunkd
#

The process with the PID 31916484 runs the executable /opt/splunkforwarder/bin/splunkd.

With a little trick you can also shorten the search significantly. For this you need a shell as user root. In this shell you open the a.out explicitly with exec and a (free) descriptor number:

# exec 5</proc/31916484/object/a.out
#

Our current shell has thus opened the executable (via the file descriptor 5)! Now you can use the procfiles command to display the open files in this shell. Using the “-n” option, which displays the absolute paths of files that belong to a file descriptor:

# procfiles -n $$
19136808 : ksh
  Current rlimit: 10000 file descriptors
   0: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   1: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   2: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   5: S_IFREG mode:0555 dev:10,9 ino:139342 uid:0 gid:0 rdev:0,0
      O_RDONLY size:194980846  name:/opt/splunkforwarder/bin/splunkd
   10: S_IFREG mode:0444 dev:10,5 ino:124151 uid:0 gid:0 rdev:0,0
      O_RDONLY size:5875  name:/usr/lib/nls/msg/EN_US/ksh.cat
   63: S_IFREG mode:0600 dev:10,4 ino:41933 uid:0 gid:0 rdev:0,0
      O_RDWR | O_APPEND size:5494  name://root/.sh_history
#

Note: The special variable $$ is replaced by the shell with the shell’s PID.

For file descriptor 5, the absolute path /opt/splunkforwarder/bin/splunkd is displayed.

The file descriptor should of course be closed again:

# exec 5<&-
#

Change passwords non-interactively

With the chpasswd command, AIX offers the option of changing passwords both interactively and non-interactively. However, the use of the command is reserved for the root user.

In the simplest case, the administrator can start the command without arguments. The interactive input of the user name and associated password, separated by a colon “:”, is then expected. One user name and a plain text password are specified per line. The input must be terminated with Control-D:

# chpasswd
user01:hello19
<Control>-<D>
#

The ADMCHG flag for the user account is set by default:

# pwdadm -q user01
user01:
        lastupdate = 1650438240
        flags = ADMCHG

#

The user is aked to change the password the next time he logs in.

If you want to set the password non-interactively via a script, you can use a so-called “here” document, for example:

# chpasswd -c <<EOF
> user02:hello20
> EOF
#

This variant no longer requires manual input. The users and passwords to be entered can be specified directly in the script. To prevent users from being prompted to change their password at the next login, we used the “-c” (clear all password flags) option.

Alternatively, you could also use a pipe with an echo command, for example:

# echo user03:hello21 | chpasswd -c
#

If you use a bash, then there is the particularly elegant option of setting up an input redirection to a character string. For this purpose, 3 less-than characters “<<<” followed by a character string are used:

(bash)# chpasswd -c <<<user04:hello22
(bash)#

In all of the above examples, the password was given in clear text. This is generally not desired when setting the password non-interactively. You can also specify the password in encrypted form. All you have to do is to use the “-e” (encrypted password) option. The chpasswd command does not check whether the specified encrypted password has the correct length and syntax, or whether it is even valid!

However, one must now note that the encrypted password may contain special characters such as “$” or “!“, which are evaluated and possibly replaced by the shell. When using a “here” document, special characters in the input are interpreted by the shell. We demonstrate this by setting a variable VAR, which is then used in the encrypted password:

# VAR=hello
# chpasswd -e -c <<EOF
> user02:{ssha512}06TQ.$VAR
EOF
#

The specified encrypted password is far too short and therefore not valid, but there is no error message. The “$VAR” part is replaced by the shell with the value “hello“, as shown by the displaying the password set:

# lsuser -a spassword user02
user02 spassword={ssha512}06TQ.hello
#

Shell substitution can be avoided by enclosing the word “EOF” in quotation marks:

# VAR=hello
# chpasswd -e -c <<”EOF”
> user02:{ssha512}06TQ.$VAR
EOF
#

This time “$VAR” has not been replaced:

# lsuser -a spassword user02
user02 spassword={ssha512}06TQ.$VAR
#

(However, the encrypted password is still too short and therefore invalid.)

Manage group membership on AIX with chgrpmem

AIX provides an elegant command to change user group membership: chgrpmem.

As an example we use the users user01, user02, …, and the group mygroup:

$ lsgroup mygroup
mygroup id=225 admin=false users= registry=files
$

The group mygroup currently has no members (users=””).

To add the two local users user01 and user02 to the group mygroup, the “-m” (member) option must be used. Then follows a plus sign “+” for add and a comma-separated list of user names. The last argument is the group:

# chgrpmem -m + user01,user02 mygroup
#
# lsgroup mygroup
mygroup id=225 admin=false users=user01,user02 registry=files
#

Using the equal sign “=” instead of the plus sign “+” overwrites the current list of users with the given list of user names:

# chgrpmem -m = user03,user04,user05 mygroup
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user04,user05 registry=files
#

Removing users is done by using a minus sign “” e.g. removing user04:

# chgrpmem -m - user04 mygroup
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user05 registry=files
#

However, removing a user from the member list of a group does not always have to be successful! We create the user user06 with primary group mygroup:

# mkuser pgrp=mygroup user06
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user05,user06 registry=files
#

The output of lsgroup shows that the user06 is also a member of the group mygroup. However, membership cannot be revoked in this case:

# chgrpmem -m - user06 mygroup
Cannot drop "user06" from primary group "mygroup".
#

A user must always have a primary group! The chgrpmem command can only be used to manage users’ additional memberships. The primary group can only be changed with the chuser command.

Note: The chgrpmem command and the “-a” option can also be used to change the administrators of a group. However, this is rarely used in practice and is therefore not addressed here.

Changing the PVID of a Physical Volume

Each physical volume used by AIX LVM has a unique physical volume ID, or PVID for short. The PVID is a software-generated ID that is stored in the header area of a disk (block 0). When a new disk is added to an AIX system, the new physical volume does not yet have a PVID. As soon as a physical volume is added to a volume group, a PVID is automatically generated if the physical volume should not already have a PVID. An already existing PVID is adopted.

A PVID can also be created manually, using the chdev command. The pv attribute is set to the value yes:

# chdev -l hdisk3 -a pv=yes
hdisk3 changed
#

The set PVID can be displayed either with the lsattr command or simply with lspv:

$ lsattr -El hdisk3 -a pvid -F value
00c276b0084049750000000000000000
$
$ lspv |grep hdisk3
hdisk3          00c276b008404975                    None                       
$

A PVID can also be removed again. However, the physical volume must not be in use for this (e.g. as part of a volume group).

To clear a PVID of a physical volume, the pv attribute can be set to the value clear:

# chdev -l hdisk3 -a pv=clear
hdisk3 changed
#

The PVID has been removed as shown by the following outputs:

$ lsattr -El hdisk3 -a pvid -F value
none
$
$ lspv |grep hdisk3
hdisk3          none                                None                       
$

Attempting to delete the PVID of a physical volume that is in use, results in the following error message:

# chdev -l hdisk0 -a pv=clear
Method error (/usr/lib/methods/chgdisk):
        0514-062 Cannot perform the requested function because the
                 specified device is busy.
     pv    

#