SSHD: Effective Values of Keywords

By default, the sshd service uses the configuration file /etc/ssh/sshd_config. There the service can be configured in detail using keyword/value pairs. There are around 80 such keywords (depending on the SSH version). If a keyword is not listed in the configuration file, then a default value applies. However, this can depend on the version of SSH used. Then it is not always clear what value a certain keyword has. But that can be found out very easily with the help of sshd itself! There is an “extended test mode” for this, which can be started with the “-T” option. Sshd then checks the validity of the current configuration, prints the effective configuration, and then exits:

# sshd -T
port 22
addressfamily inet
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
x11maxdisplays 1000
maxauthtries 6
maxsessions 1024
clientaliveinterval 60
clientalivecountmax 3
streamlocalbindmask 0177
permitrootlogin forced-commands-only
…
#

Root privileges are required.

This makes it very easy to determine the effective value of any keyword.

NTP Configuration on HMC with LPAR tool

NTP-Konfiguration HMC

The current status of NTP on an HMC can then be displayed using “hmc lsntp“:

$ hmc lsntp
NAME   XNTP     XNTPSTATUS    XNTPSERVER
hmc01  disable  -             -
hmc02  enable   synchronized  192.168.189.77,192.168.189.78
hmc03  enable   synchronized  192.168.189.77,192.168.189.78
$

Another NTP server can be added to the NTP configuration of an HMC using the “hmc addntpserver” command:

$ hmc addntpserver hmc01 192.168.189.77
$ hmc addntpserver hmc01 192.168.189.78
$

A check with “hmc lsntp” shows that two NTP servers are now configured, but NTP is still not activated:

$ hmc lsntp hmc01
NAME   XNTP     XNTPSTATUS  XNTPSERVER
hmc01  disable  -           192.168.189.77,192.168.189.78
$

NTP can now be activated with the command “hmc enablentp“:

$ hmc enablentp hmc01
$

The first sync may take a while:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS      XNTPSERVER
hmc01  enable  unsynchronized  192.168.189.77,192.168.189.78
$

The time on the HMC is not yet synchronized immediately after enabling NTP (XNTPSTATUS: unsynchronized).

A detailed status for each NTP server can be obtained by using the “-a” option (all NTP servers):

$ hmc lsntp –a hmc01
NAME    SERVER         STATE          POLL_FREQ_SECONDS  SECONDS_SINCE_LAST_POLL
hmc01  192.168.189.77  not connected  64                 0
hmc01  192.168.189.78  not connected  64                 0
$

As soon as synchronization with one of the NTP servers is achieved, the overall status is synchronized:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS    XNTPSERVER
hmc01  enable  synchronized  192.168.189.77,192.168.189.78
$

A more detailed description can be found here: NTP Configuration on the HMC

The LPAR tool can be downloaded for testing here: Download

History Expansion bash

Drawing Shell

Many AIX and UNIX users use bash as their preferred shell. Navigating in the history with the cursor keys is certainly used countless times a day by all users. As long as the interesting commands are only a short time ago, this works very well. However, for commands longer in the past, access using the cursor keys is relatively time-consuming. Who wants to press the cursor keys 50 times to access a command?

The bash history expansion mechanism offers a much more efficient option here. Previous commands can be accessed using the history expansion character “!“. The commands can be specified in different ways:

    • The number of the command: !31
    • The nth previous command: !-n (e.g. !-3 for the third last command)
    • The last command that begins with a specific character string: !ca
    • The last command that has a specific character string anywhere: !?ca

However, the possibilities of the bash are far from exhausted. You can specifically access individual arguments of a previous command and even make changes.

Here are a few of those options:

    • !! (run the last command again)
    • ^op^art (run the last command again, but replace “op” with “art“)
    • cat !?sam?:% (run the cat command on the last argument containing the string “sam“)
    • vi !$ (Run vi on the last argument of the last command)

A description of these and other bash options can be found here:

The bash History Expansion

The Command chrctcp

chrctcp

On AIX, the System Resource Controller (SRC) is used to control subsystems and subservers. Every AIX administrator is surely familiar with the lssrc, startsrc and stopsrc commands. The chrctcp command, which is used to manage subsystems that are started via /etc/rc.tcpip, is not quite as well known. By using this command, editing the start script /etc/rc.tcpip can be avoided. Most administrators only know and use the “-a” (add/activate), “-d” (delete/deactivate) and “-S” (start, stop or restart subsystem) options. However, the chrctcp command offers even more functionality via the options “-c” (change) and “-s” (show), which SMIT also uses. (After all, chrctcp was probably written specifically for use by SMIT.)

A number of subsystems are started at boot time via /etc/rc.tcpip. However, only if there is an entry of the following form in the /etc/rc.tcpip script:

start /usr/sbin/inetd "$src_running"

If a specific subsystem is not to be started at boot time, the relevant subsystem must be commented out in /etc/rc.tcpip:

#start /usr/sbin/inetd "$src_running"

In principle, the change can be made with an editor. However, there is a much more elegant way using the chrctcp command. If a specific service should not be started at next boot, chrctcp can be used with the “-d” option to comment out the service:

# chrctcp -d inetd
# grep /usr/sbin/inet /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
#

This has no effect on the currently running subsystem. If you also want to stop the running subsystem, you can either use the “stopsrc –s” command or, which is much easier, simply use the “-S” option of chrctcp:

# chrctcp -S -d inetd
0513-044 The /usr/sbin/inetd Subsystem was requested to stop.
#

The subsystem is commented out in /etc/rc.tcpip and an eventually running subsystem is stopped as well.

You can proceed in a similar way if you want to use a previously unused service from /etc/rc.tcpip. The “-a” option can be used to activate the corresponding entry:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -a inetd
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

Note: The entry ‘#start /usr/sbin/inetd “$src_running”‘ must already be present and at the correct place in the file!

If not only the file /etc/rc.tcpip is to be adjusted, but the service is also to be started immediately, the “-S” option can be used again:

# grep /usr/sbin/inetd /etc/rc.tcpip
#start /usr/sbin/inetd "$src_running"
# chrctcp -S -a inetd
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636582.
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running"
#

In the following we will stick to inetd as an example. However, everything described applies in the same way to other subsystems.

Services such as inetd, syslogd, and others often have options and arguments that can be specified at startup. For example, inetd offers the following options and arguments:

-d   Send debugging information to syslogd

-6   IPv6 support

file  Specification of a configuration file to be used instead of /etc/inetd.conf.

The options and arguments used can be displayed with the chrctcp command and the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
no:no:
#

The sample output shows debugging and IPv6 disabled and no alternate configuration file specified.

With the option “-c” (change) you can change the options and arguments. Attributes can be specified with the “-f” option. As an example, let’s enable debugging and allow IPv6:

# chrctcp -c inetd -f debug=yes -f ipv6=yes
# grep /usr/sbin/inetd /etc/rc.tcpip
start /usr/sbin/inetd "$src_running" " -d  -6 "
#

A look at /etc/rc.tcpip shows that the start of inetd has been supplemented with the options “-d” (debugging) and “-6” (IPv6)! Again, only the start file /etc/rc.tcpip has been changed, an inetd that may already be running remains unaffected by the change:

# ps -ef|grep ine[t]d
    root  5636582  4587924   0 18:34:47      -  0:00 /usr/sbin/inetd
#

As with enabling and disabling subsystems, the “-S” option can also be used when changing:

# chrctcp -S -c inetd -f debug=yes -f ipv6=yes
0513-044 The inetd Subsystem was requested to stop.
0513-059 The inetd Subsystem has been started. Subsystem PID is 5636584.
#

The service is then stopped and restarted immediately. Then of course with the changed options and arguments:

# ps -ef|grep ine[t]d
    root  5636584  4587924   0 18:49:25      -  0:00 /usr/sbin/inetd -d -6
#

The options and arguments used at next boot can be listed with the “-s” option:

# chrctcp -s inetd
#debug_mode:ipv6_mode:config_file
yes:yes:
#

A bit unfortunate is the fact that the names shown in the header (debug_mode, ipv6_mode, config_file) do not quite match the attribute names to be used. The correct supported attribute names in the case of inetd are debug, ipv6 and file.

In one of the next posts we will list the supported attributes for further subsystems and make them available in a table.

Path to the executable of a running AIX process

If a process was not started with an absolute path, it is surprisingly difficult to find out the absolute path for the associated executable.

We demonstrate this using Splunk as an example:

$ ps -ef |grep splun[k]d
    root  7143802  5702116   0   Apr 23      - 23:26 splunkd --nodaemon -p 8089 _internal_exec_splunkd
    root 31916484  7143802   0   Apr 23      -  0:00 [splunkd pid=7143802] splunkd --nodaemon -p 8089 _internal_exec_splunkd [process-runner]
$

When starting the process with the PID 31916484, the argument 0 was also changed.

Part of the information about a process is available via the process file system /proc. Since Splunk runs as root, root privileges are required to access the information in /proc about the process.

Below /proc there is a subdirectory for each running process, with the PID as the directory name.

# ls -l /proc/31916484
total 32
-rw-------    1 root     system            0 Apr 28 15:17 as
-r--------    1 root     system          128 Apr 28 15:17 cred
--w-------    1 root     system            0 Apr 28 15:17 ctl
lr-x------   38 root     system            0 Apr 28 13:31 cwd -> /root/
dr-x------    1 root     system            0 Apr 28 15:17 fd
dr-xr-xr-x    1 root     system            0 Apr 28 15:17 lwp
-r--------    1 root     system            0 Apr 28 15:17 map
-r--------    1 root     system            0 Apr 28 15:17 mmap
dr-x------    1 root     system            0 Apr 28 15:17 object
-r--r--r--    1 root     system          448 Apr 28 15:17 psinfo
lr-x------   48 root     system            0 Apr 28 09:02 root -> /
-r--------    1 root     system        12288 Apr 28 15:17 sigact
-r--------    1 root     system         1520 Apr 28 15:17 status
-r--r--r--    1 root     system            0 Apr 28 15:17 sysent
#

In the subdirectory object, in addition to the open files, the executable is also available:

# ls -l /proc/31916484/object
total 854216
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  a.out
-r-xr-xr-x    1 bin      bin           10749 Sep 21 2015  jfs2.10.5.103039
-r--r--r--    1 bin      bin        12858422 May 28 2019  jfs2.10.5.225767
-r-xr-xr-x    1 bin      bin           77411 Mar 25 2021  jfs2.10.5.4157
-r-xr-xr-x    1 bin      bin        13438344 Jul 27 2021  jfs2.10.5.4205
-r--r--r--    1 bin      bin         1351386 Mar 09 2021  jfs2.10.5.4209
-r--r--r--    1 bin      bin           80450 Jul 27 2021  jfs2.10.5.4220
-r-xr-xr-x    1 root     system    194980846 Nov 16 2019  jfs2.10.9.139342
-r-xr-xr-x    1 root     system      3487967 Nov 12 2019  jfs2.10.9.155672
-r-xr-xr-x    1 root     system       228087 Nov 12 2019  jfs2.10.9.155673
-r-xr-xr-x    1 root     system      2688333 Nov 16 2019  jfs2.10.9.155675
-r-xr-xr-x    1 root     system       901800 Nov 12 2019  jfs2.10.9.155677
-r-xr-xr-x    1 root     system      4256593 Nov 12 2019  jfs2.10.9.155679
-r-xr-xr-x    1 root     system       568791 Nov 16 2019  jfs2.10.9.155684
-r-xr-xr-x    1 root     system      6166802 Nov 12 2019  jfs2.10.9.155685
-r-xr-xr-x    1 root     system      1239656 Nov 12 2019  jfs2.10.9.155686
-r-xr-xr-x    1 root     system       124218 Nov 16 2019  jfs2.10.9.155690
#

The file a.out represents the executable. Access to this file is redirected to the executable’s file. Information such as the inode number and device (file system) of the executable can be found with the istat command:

# istat /proc/31916484/object/a.out
Inode 139342 on device 10/9     File
Protection: r-xr-xr-x  
Owner: 0(root)          Group: 0(system)
Link count:   1         Length 194980846 bytes

Last updated:   Tue Jul 27 07:15:40 CEST 2021
Last modified:  Sat Nov 16 01:23:05 CET 2019
Last accessed:  Thu Apr 28 10:38:13 CEST 2022

#

The inode number of the executable is 139342. The file system is the file system on the device with major number 10 and minor number 9 (“device 10/9“).

You could now first find out the device by searching under /dev:

$ ls -l /dev | grep "10,  9"
brw-rw----    1 root     system       10,  9 Jul 27 2021  hd10opt
crw-rw----    1 root     system       10,  9 Apr 05 13:36 rhd10opt
$

and then determine the associated file system via df:

$ df | grep hd10opt
/dev/hd10opt    12582912   7527920   41%    33349     4% /opt
$

However, this is even easier. You can simply specify the path for the a.out file in df:

# df /proc/31916484/object/a.out
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd10opt    12582912   7527912   41%    33349     4% /opt
#

The file system we are looking for is the /opt file system. Now you can find out the absolute path to the executable by searching for the inode number 139342 in /opt:

# find /opt -inum 139342
/opt/splunkforwarder/bin/splunkd
#

The process with the PID 31916484 runs the executable /opt/splunkforwarder/bin/splunkd.

With a little trick you can also shorten the search significantly. For this you need a shell as user root. In this shell you open the a.out explicitly with exec and a (free) descriptor number:

# exec 5</proc/31916484/object/a.out
#

Our current shell has thus opened the executable (via the file descriptor 5)! Now you can use the procfiles command to display the open files in this shell. Using the “-n” option, which displays the absolute paths of files that belong to a file descriptor:

# procfiles -n $$
19136808 : ksh
  Current rlimit: 10000 file descriptors
   0: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   1: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   2: S_IFCHR mode:00 dev:10,4 ino:4463 uid:0 gid:0 rdev:21,3
      O_RDWR | O_NOCTTY  name://dev/pts/3
   5: S_IFREG mode:0555 dev:10,9 ino:139342 uid:0 gid:0 rdev:0,0
      O_RDONLY size:194980846  name:/opt/splunkforwarder/bin/splunkd
   10: S_IFREG mode:0444 dev:10,5 ino:124151 uid:0 gid:0 rdev:0,0
      O_RDONLY size:5875  name:/usr/lib/nls/msg/EN_US/ksh.cat
   63: S_IFREG mode:0600 dev:10,4 ino:41933 uid:0 gid:0 rdev:0,0
      O_RDWR | O_APPEND size:5494  name://root/.sh_history
#

Note: The special variable $$ is replaced by the shell with the shell’s PID.

For file descriptor 5, the absolute path /opt/splunkforwarder/bin/splunkd is displayed.

The file descriptor should of course be closed again:

# exec 5<&-
#

Change passwords non-interactively

With the chpasswd command, AIX offers the option of changing passwords both interactively and non-interactively. However, the use of the command is reserved for the root user.

In the simplest case, the administrator can start the command without arguments. The interactive input of the user name and associated password, separated by a colon “:”, is then expected. One user name and a plain text password are specified per line. The input must be terminated with Control-D:

# chpasswd
user01:hello19
<Control>-<D>
#

The ADMCHG flag for the user account is set by default:

# pwdadm -q user01
user01:
        lastupdate = 1650438240
        flags = ADMCHG

#

The user is aked to change the password the next time he logs in.

If you want to set the password non-interactively via a script, you can use a so-called “here” document, for example:

# chpasswd -c <<EOF
> user02:hello20
> EOF
#

This variant no longer requires manual input. The users and passwords to be entered can be specified directly in the script. To prevent users from being prompted to change their password at the next login, we used the “-c” (clear all password flags) option.

Alternatively, you could also use a pipe with an echo command, for example:

# echo user03:hello21 | chpasswd -c
#

If you use a bash, then there is the particularly elegant option of setting up an input redirection to a character string. For this purpose, 3 less-than characters “<<<” followed by a character string are used:

(bash)# chpasswd -c <<<user04:hello22
(bash)#

In all of the above examples, the password was given in clear text. This is generally not desired when setting the password non-interactively. You can also specify the password in encrypted form. All you have to do is to use the “-e” (encrypted password) option. The chpasswd command does not check whether the specified encrypted password has the correct length and syntax, or whether it is even valid!

However, one must now note that the encrypted password may contain special characters such as “$” or “!“, which are evaluated and possibly replaced by the shell. When using a “here” document, special characters in the input are interpreted by the shell. We demonstrate this by setting a variable VAR, which is then used in the encrypted password:

# VAR=hello
# chpasswd -e -c <<EOF
> user02:{ssha512}06TQ.$VAR
EOF
#

The specified encrypted password is far too short and therefore not valid, but there is no error message. The “$VAR” part is replaced by the shell with the value “hello“, as shown by the displaying the password set:

# lsuser -a spassword user02
user02 spassword={ssha512}06TQ.hello
#

Shell substitution can be avoided by enclosing the word “EOF” in quotation marks:

# VAR=hello
# chpasswd -e -c <<”EOF”
> user02:{ssha512}06TQ.$VAR
EOF
#

This time “$VAR” has not been replaced:

# lsuser -a spassword user02
user02 spassword={ssha512}06TQ.$VAR
#

(However, the encrypted password is still too short and therefore invalid.)

Manage group membership on AIX with chgrpmem

AIX provides an elegant command to change user group membership: chgrpmem.

As an example we use the users user01, user02, …, and the group mygroup:

$ lsgroup mygroup
mygroup id=225 admin=false users= registry=files
$

The group mygroup currently has no members (users=””).

To add the two local users user01 and user02 to the group mygroup, the “-m” (member) option must be used. Then follows a plus sign “+” for add and a comma-separated list of user names. The last argument is the group:

# chgrpmem -m + user01,user02 mygroup
#
# lsgroup mygroup
mygroup id=225 admin=false users=user01,user02 registry=files
#

Using the equal sign “=” instead of the plus sign “+” overwrites the current list of users with the given list of user names:

# chgrpmem -m = user03,user04,user05 mygroup
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user04,user05 registry=files
#

Removing users is done by using a minus sign “” e.g. removing user04:

# chgrpmem -m - user04 mygroup
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user05 registry=files
#

However, removing a user from the member list of a group does not always have to be successful! We create the user user06 with primary group mygroup:

# mkuser pgrp=mygroup user06
# 
# lsgroup mygroup
mygroup id=225 admin=false users=user03,user05,user06 registry=files
#

The output of lsgroup shows that the user06 is also a member of the group mygroup. However, membership cannot be revoked in this case:

# chgrpmem -m - user06 mygroup
Cannot drop "user06" from primary group "mygroup".
#

A user must always have a primary group! The chgrpmem command can only be used to manage users’ additional memberships. The primary group can only be changed with the chuser command.

Note: The chgrpmem command and the “-a” option can also be used to change the administrators of a group. However, this is rarely used in practice and is therefore not addressed here.

Changing the PVID of a Physical Volume

Each physical volume used by AIX LVM has a unique physical volume ID, or PVID for short. The PVID is a software-generated ID that is stored in the header area of a disk (block 0). When a new disk is added to an AIX system, the new physical volume does not yet have a PVID. As soon as a physical volume is added to a volume group, a PVID is automatically generated if the physical volume should not already have a PVID. An already existing PVID is adopted.

A PVID can also be created manually, using the chdev command. The pv attribute is set to the value yes:

# chdev -l hdisk3 -a pv=yes
hdisk3 changed
#

The set PVID can be displayed either with the lsattr command or simply with lspv:

$ lsattr -El hdisk3 -a pvid -F value
00c276b0084049750000000000000000
$
$ lspv |grep hdisk3
hdisk3          00c276b008404975                    None                       
$

A PVID can also be removed again. However, the physical volume must not be in use for this (e.g. as part of a volume group).

To clear a PVID of a physical volume, the pv attribute can be set to the value clear:

# chdev -l hdisk3 -a pv=clear
hdisk3 changed
#

The PVID has been removed as shown by the following outputs:

$ lsattr -El hdisk3 -a pvid -F value
none
$
$ lspv |grep hdisk3
hdisk3          none                                None                       
$

Attempting to delete the PVID of a physical volume that is in use, results in the following error message:

# chdev -l hdisk0 -a pv=clear
Method error (/usr/lib/methods/chgdisk):
        0514-062 Cannot perform the requested function because the
                 specified device is busy.
     pv    

#

Determining the Size of a Physical Volume

There are a number of different ways to determine the size of a physical volume (disk, LUN) under AIX.

If you have root privileges, you can use the bootinfo command with the “-s” (size) option:

#  bootinfo -s hdisk0
51200
#

The size of the physical volume is given in MB. In the example 51,200 MB or about 50 GB.

Without root privileges, the getconf command can be used. With this command, system-wide configuration parameters as well as device-specific variables can be displayed. The device-specific variable DISK_SIZE can be used to display the size of a physical volume. The physical volume in question is specified by the absolute path of the physical volume’s block or character device file:

$ getconf DISK_SIZE /dev/hdisk0
51200
$

Here, too, the size is given in MB.

Another option, which again requires root privileges, is to use the lsmpio command. The command offers the option “-q” (query) to display data about an MPIO storage device:

# lsmpio -ql hdisk0
Device:  hdisk0
…
           Capacity:  50.00GiB
…
#

The size is directly displayed in GB (GiB) this time.

If the physical volume is part of a volume group, the lspv command can also be used to at least estimate the size:

$ lspv hdisk0
…
TOTAL PPs:          199 (50944 megabytes)    VG DESCRIPTORS:   2
…                                      
$

The area that can be used for data is specified here (50,944 MB), the physical volume itself is somewhat larger, since space is also used for administrative information.

LPAR tool 1.7.0.1 is now available

Version 1.7.0.1 of the LPAR tool is now available in our download area.

The new version supports the following new features, among others:

    • Installation of IFixes and updates on the HMC (hmc help updhmc)
    • System firmware updates (and more) of managed systems (ms help updatelic)
    • Display FLRT data with online query at IBM (hmc help flrt, ms help flrt, lpar help flrt)
    • Configuration of NTP on HMCs (hmc help ntp)

Versions for Linux, AIX and Macos are available.

All versions include a test license valid until September 30th, 2022.

So download, install and then try it out!