LPAR-Tool: Which LPARs have no RMC-Connection

Status and configuration of LPARs are regularly needed information in the administration of LPARs. With the LPAR tool, information such as status, RMC status, number of cores, size of RAM, OS version and other data can be easily and quickly determined, even with hundreds or thousands of LPARs. Which LPARs don’t have an RMC connection is shown as one of the examples.

All of the following examples were performed on an environment with 10 HMCs, 50 managed systems, and just over 500 LPARs. To determine how long the LPAR tool requires, the run times of the commands were measured and specified using time.

The names of the LPARs were manually changed in the outputs shown and replaced by generic names lparXX and aixYY.

First of all, the status of a single LPAR:

$ time lpar status aix01
NAME   LPAR_ID  LPAR_ENV   STATE     PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01  27       aixlinux   Running   standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614

real    0m0.210s
user    0m0.011s
sys     0m0.013s
$

Of course you can also specify multiple LPARs. If you want to know the status of all LPARs (in our case just over 500 LPARs), just leave out the argument:

$ time lpar status
NAME    LPAR_ID  LPAR_ENV   STATE     PROFILE      SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01   27       aixlinux   Running   standard     0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02   1        aixlinux   Running   standard     -     -         1      -           8320    Unknown
...
lpar01  6        aixlinux   Running   standard     0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m18.933s
user	0m3.819s
sys	0m3.789s
$

In the background, the LPAR tool executes more than 150 commands on the corresponding HMCs (lshwres and lssyscfg)!

The output should now be restricted to LPARs that are currently active (state=Running). There is the option “-s“, which can be used to specify criteria for attributes that must be met. Only LPARs meeting these criteria will be shown:

$ time lpar status -s state=Running
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01    27       aixlinux   Running  standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar01   6        aixlinux   Running  standard   0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m17.998s
user	0m3.692s
sys	0m3.647s
$

Now we want to know on which of these LPARs RMC is not working/not active. The option “-s” allows to combine any number of criteria. All specified criteria must then be met (logical AND). The RMC state can be found in the attribute rmc_state:

$ time lpar status -s state=Running,rmc_state!=active
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
aix03    2        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar07   4        aixlinux   Running  standard   0     none      1      1.0         4352    Unknown

real	0m19.057s
user	0m3.550s
sys	0m3.512s
$

As another example we want to know on which LPARs AIX 7.1 TL5 is installed. The os_version attribute contains the OS version. The ‘~‘ operator can be used to compare against a regular expression (similar to the grep command). We use the regular expression 7100-05:

$ time lpar status -s os_version~7100-05
NAME     LPAR_ID  LPAR_ENV  STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix14    14       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-02-1810
aix16    24       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-03-1846
...
lpar10   10       aixlinux  Running  standard   0     active    3      0.3         32768   AIX 7.1 7100-05-02-1810

real	0m18.212s
user	0m3.726s
sys	0m3.676s
$

So far, we have always used the default output format. Now we’d like to list all systems that still run on AIX 6.1, but this time only the LPAR name and the OS version will be output. For this there is the option “-F“, with which the desired output fields can be specified:

$ time lpar status -s os_version~6100 -F name:os_version
aix39:AIX 6.1 6100-07-04-1216
aix46:AIX 6.1 6100-07-04-1216
...
lpar35:AIX 6.1 6100-09-05-1524

real	0m18.041s
user	0m3.619s
sys	0m3.699s
$

If you prefer JSON output, you can easily do that with the option “-j“, here’s the same example with JSON output:

$ time lpar status -s os_version~6100 -F name:os_version -j
{
	"name": "aix39",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
{
	"name": "aix46",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
...
{
	"name": "lpar35",
	"os_version": "AIX 6.1 6100-09-05-1524"
}

real	0m21.247s
user	0m3.670s
sys	0m3.720s
$

Of course you can not know all attribute names by heart! But that’s not necessary, because you can easily see all attribute names. Use the option “-f” (stanza format) and specify any LPAR:

$ lpar status -f lpar19
lpar19:
	curr_lpar_proc_compat_mode = POWER7
	curr_mem = 8192
	curr_proc_mode = shared
	curr_proc_units = 0.3
	curr_procs = 2
	name = lpar19
	os_version = AIX 6.1 6100-09-05-1524
...
$

With the options “-h” and “-m” the LPARs can be selected depending on the associated HMC and/or managed system.

Status of all LPARs with associated HMC hmc01:

$ lpar -h hmc01 status

Status of all LPARs whose corresponding HMC has type 7042-CR6:

$ lpar -h 7042-CR6 status

Status of all LPARs whose associated HMC has type 7042-CR6 and whose name begins with lpar:

$ lpar -h 7042-CR6 status lpar*

Status of all LPARs on the managed system ms13:

$ lpar -m ms13 status

Status of all LPARs whose managed system is an S922:

$ lpar -m 9009-22A status

The presented selection and output options apply to all output commands of the LPAR tool (except the vios command).

The LPAR-Tool can be downloaded from our download area: https://powercampus.de/en/download-2

The LPAR-Tool contains a test license which is valid until the end of october.

LPAR tool 1.4.0.1 available (including a valid test license)!

In our download area, version 1.4.0.1 of our LPAR tool, including a valid test license (valid until 31th october 2019) is available for download. The license is contained directly in the binaries, so no license key must be entered. The included trial license allows use of the LPAR tool for up to 10 HMCs, 100 managed systems and 1000 LPARs.

LPAR console using Virtual I/O Server

Typically, a console for an LPAR is launched via an HMC, via GUI or CLI (vtmenu or mkvterm). A console depends on the availability of an HMC. During an HMC update or problems with the HMC, you may not be able to connect to an LPAR console.

Relatively unknown is the ability to configure a console to an LPAR via a virtual I/O server. If the HMC is not available, then a console can be started via the virtual I/O server. No configuration is required on the client LPAR! By default, each client LPAR has 2 virtual serial server adapters (slots 0 and 1). If you configure an associated client adapter on a virtual I/O server, you can use it for a console connection.

On the virtual I/O server one needs only an unused virtual slot (here slot 45). The client LPAR has the LPAR ID 39. The virtual serial client adapter can be created with the following command:

hmc01 $ chhwres -m ms02 -r virtualio --rsubtype serial -o a -p ms02-vio1 -s 45 -a adapter_type=client,remote_lpar_name=aix02,remote_slot_num=0,supports_hmc=0
hmc01 $

Now you can always start a console for the LPAR via the virtual I/O server:

ms02-vio1 :/home/padmin> mkvt -id 39
AIX Version 7
Copyright IBM Corporation, 1982, 2018.
Console login: root
root's Password: XXXXXX


aix02  AIX 7.2         powerpc


Last unsuccessful login: Mon Mar 18 23:14:26 2019 on ssh from N.N.N.N
Last login: Wed Mar 27 20:19:22 2019 on /dev/pts/0 from M.M.M.M
[YOU HAVE NEW MAIL]
aix02:/root> hostname
aix02
aix02:/root>

The command mkvt on the virtual I/O server corresponds to the command mkvterm on the HMC. Here the desired partition must be specified by the LPAR-ID. Terminating the console works as usual with “~.“, Or if you are logged in via SSH on the virtual I/O server with “~~.“.

Alternatively, you can also end a console session with the command rmvt:

ms02-vio1:/home/padmin> rmvt -id 39
ms02-vio1:/home/padmin>

The following message appears in the console and the console is closed:

Virtual terminal has been disconnected.

$

With the LPAR tool, the console can of course be set up even easier. The virtual serial adapter on the virtual I/O server can be created with the command “lpar addserial“, a manual login to the HMC is not necessary for this to work:

$ lpar addscsi -c ms02-vio1 45 aix02 0
$

The “-c” option means “create client adapter”. The command also creates the adapter in the profile. The success of the action can be checked by “lpar vslots“, showing all virtual adapters of an LPAR:

$ lpar vslots ms02-vio1
SLOT  REQ  TYPE           DATA
0     1    serial/server  remote: -(any)/any status=unavailable hmc=1
1     1    serial/server  remote: -(any)/any status=unavailable hmc=1
2     0    eth            PVID=1 VLANS=- XXXXXXXXXXXX ETHERNET0
3     1    eth            TRUNK(1) IEEE PVID=1 VLANS=201 XXXXXXXXXXXXX ETHERNET0
...
45     0   serial/client  remote: aix02(39)/0 status=unavailable hmc=0
...
$

Starting the console then proceeds as usual by logging in as padmin on the virtual I/O server and the command mkvt.

Caution: The console session through the virtual I/O server should always be terminated when it is no longer needed. You can not terminate it from the HMC! Here is the attempt to start a console using the HMC, while the console is already active using the virtual I/O server:

$ lpar console aix02

Open in progress 

A terminal session is already open for this partition. 
Only one open session is allowed for a partition. 
Exiting.... 
Attempts to open the session failed. Please close the terminal and retry the open at a later time. 
If the problem persists, Please contact IBM support. 
Received end of file, Exiting.
Connection to X.X.X.X closed.
$

Even rmvterm does not help:

$ lpar rmvterm aix02
/bin/stty: standard input: Inappropriate ioctl for device
$

Conversely, no console can be started using the virtual I/O server if a console is active using the HMC:

ms02-vio1:/home/padmin> mkvt -id 39
Virtual terminal is already connected.

ms02-vio1:/home/padmin>

So always make sure that the console is terminated.

 

Migration from SDDPCM to AIX PCM

Many AIX systems still use SDDPCM as a multipathing solution. However, SDDPCM is no longer supported on POWER 9 hardware from IBM.

The following is the migration from SDDPCM to AIX PCM. On our example system we have the following physical volumes:

$ lspv
hdisk0          00abcdefabcde000                    datavg          active     
hdisk1          00abcdefabcde001                    datavg          active     
hdisk2          none                                None                       
hdisk3          00abcdefabcde003                    altinst_rootvg             
hdisk4          00abcdefabcde004                    rootvg          active     
$

The Physical Volumes are disks that are made available through an SVC:

$ lsdev -l hdisk0 -F uniquetype
disk/fcp/2145
$

The Path Control Module (PCM) uses SDDPCM:

$ lsattr -El hdisk0 -a PCM -F value
PCM/friend/sddpcm
$

You can also see this when looking at the list of kernel extensions:

$ genkex | grep pcm
f1000000c012a000    af000 /usr/lib/drivers/sddpcmke
$

Which PCM driver is used for which disk types can be easily viewd with the command “manage_disk_drivers”:

$ manage_disk_drivers -l
Device              Present Driver        Driver Options    
2810XIV             AIX_AAPCM             AIX_AAPCM,AIX_non_MPIO
DS4100              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS4200              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS4300              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS4500              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS4700              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS4800              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS3950              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS5020              AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DCS3700             AIX_APPCM             AIX_APPCM
DCS3860             AIX_APPCM             AIX_APPCM
DS5100/DS5300       AIX_SDDAPPCM          AIX_APPCM,AIX_SDDAPPCM
DS3500              AIX_APPCM             AIX_APPCM
XIVCTRL             MPIO_XIVCTRL          MPIO_XIVCTRL,nonMPIO_XIVCTRL
2107DS8K            NO_OVERRIDE           AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
IBMFlash            NO_OVERRIDE           AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
IBMSVC              NO_OVERRIDE           AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$

In our case, SVC disks, the last line is relevant (IBMSVC). As current PCM driver NO_OVERRIDE is listed here, possible other drivers are AIX_AAPCM (AIX PCM for active / active and ALUASystems) and AIX_non_MPIO (drives without multi-pathing). The value NO_OVERRIDE means that if no multipathing driver is explicitly specified, a multipathing driver is used if possible (if available), otherwise no multipathing driver is used. If more than one multipathing driver is available (in our case AIX PCM and SDDPCM, then SDDPCM has priority).

In a subsequent blog entry, we will take a closer look at the possible values, as well as the point in AIX where the selection is made.

Before we change the driver for IBMSVC disks (a reboot is necessary), let’s take a look at the attributes of our disks, here an example for the hdisk0:

$ lsattr -El hdisk0
PCM             PCM/friend/sddpcm                                   PCM                                     True
...
algorithm       load_balance                                        Algorithm                               True+
...
queue_depth     120                                                 Queue DEPTH                             True+
...
reserve_policy  no_reserve                                          Reserve Policy                          True+
...
$

Changing the driver will cause the values of some set attributes to be lost or replaced by new default values of the new driver. This is especially true for the queue_depth (here: 120), the reserve_policy (here: no_reserve) and the load-balancing policy (algorithm). The current values should be noted, then after the conversion to the AIX PCM driver then adjust accordingly.

Switching to AIX PCM can be done with the command “manage_disk_drivers”. For this, the command is given the disk type (here IBMSVC) with the option “-d” and the desired driver (here AIX_AAPCM for the AIX PCM driver) with the option “-o”:

# manage_disk_drivers -d IBMSVC -o AIX_AAPCM
********************** ATTENTION *************************
  For the change to take effect the system must be rebooted
#

The changed configuration can be listed directly with “manage_disk_drivers -l”:

$ manage_disk_drivers -l
Device              Present Driver        Driver Options    
...
IBMSVC              AIX_AAPCM             AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$

To make the change, the system must now be rebooted:

# shutdown -r now

SHUTDOWN PROGRAM
Thu Feb  7 09:43:38 CET 2019
...

We execute the 3 commands from the beginning again (lspv, lsdevund lsattr):

$ lspv
hdisk0          00abcdefabcde000                    datavg          active     
hdisk1          00abcdefabcde001                    datavg          active     
hdisk2          none                                None                       
hdisk3          00abcdefabcde003                    altinst_rootvg             
hdisk4          00abcdefabcde004                    rootvg          active     
$

The physical volumes are unchanged.

$ lsdev -l hdisk0 -F uniquetype
disk/fcp/mpioosdisk
$

The type of disks has changed from disk / fcp / 2145 to disk / fcp / mpioosdisk. This already indicates that the multipathing driver has changed.

$ lsattr -El hdisk0 -a PCM -F value
PCM/friend/fcpother
$

The Path Control Module (PCM) has also changed. The guy is no longer sddpcm but fcpother. That does not look directly after AIX PCM. However, a look at the corresponding driver shows immediately that AIX PCM is in use here:

$ lsdev -P -c PCM -s friend -t fcpother -F DvDr
aixdiskpcmke
$

The associated kernel extension aixdiskpcmke is also currently loaded and in use:

$ genkex | grep pcm
         73e2000    57000 /usr/lib/drivers/aixdiskpcmke
$

Let’s take a look at the attributes of hdisk0 again. We expect changed values for some attributes here:

$ lsattr -El hdisk0
PCM             PCM/friend/fcpother                                 Path Control Module              False
...
algorithm       fail_over                                           Algorithm                        True+
...
queue_depth     20                                                  Queue DEPTH                      True+
...
reserve_policy  single_path                                         Reserve Policy                   True+
...
$

The value 120 for the queue_depth has been lost and has been replaced by the default value 20. The reserve_policy has changed to single_path and the load-balancing algorithm is now fail_over, i. only one path is used at a time.

We change the settings to a configuration that corresponds to the initial situation:

# chdev -P -l hdisk0 -a algorithm=shortest_queue -a queue_depth=120 -a reserve_policy=no_reserve
hdisk0 changed
#

Since the Physical Volume is in use, the setting can only be changed in the ODM and a further reboot is necessary.

After all disks have been reconfigured via the ODM, the system must be rebooted a second time:

# shutdown -r now

SHUTDOWN PROGRAM
Thu Feb  6 20:07:12 CET 2019
...

After the reboot SDDPCM can be uninstalled:

# installp -u devices.fcp.disk.ibm.mpio.rte devices.sddpcm.72.rte
+-----------------------------------------------------------------------------+
                    Pre-deinstall Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
...
0503-292 This update will not fully take effect until after a
        system reboot.

    * * *  A T T E N T I O N  * * *
    System boot image has been updated. You should reboot the
    system as soon as possible to properly integrate the changes
    and to avoid disruption of current functionality.

installp:  bosboot process completed.
+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
devices.sddpcm.72.rte       2.7.1.1         ROOT        DEINSTALL   SUCCESS   
devices.sddpcm.72.rte       2.7.1.1         USR         DEINSTALL   SUCCESS   
devices.fcp.disk.ibm.mpio.r 1.0.0.25        USR         DEINSTALL   SUCCESS   
#

The SDDPCM fileset, as well as the associated host-attachment fileset, were successfully uninstalled.

Since the SDDPCM driver was not loaded, and thus no changes have been made to the kernel, actually another reboot should not be necessary. However, since it is explicitly pointed out a quick reboot, and it is also a good idea to do a reboot test with the final configuration, we reboot the system a third and final time:

# shutdown -r now

SHUTDOWN PROGRAM
Thu Feb  6 20:17:21 CET 2019
...

After the reboot, we check the disk attributes again:

$ lsattr -El hdisk0
PCM             PCM/friend/fcpother                                 Path Control Module              False
...
algorithm       shortest_queue                                      Algorithm                        True+
...
queue_depth     120                                                 Queue DEPTH                      True+
...
reserve_policy  no_reserve                                          Reserve Policy                   True+
...
$

The system now uses the AIX PCM driver for multipathing:

$ manage_disk_drivers -l
Device              Present Driver        Driver Options    
...
IBMSVC              AIX_AAPCM             AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$

Migrating from SDDPCM to AIX PCM is pretty easy to do.