A PDF on PowerVM is now available in our download area (approx. 260 pages, with approx. 75 figures):
LPAR-Tool 1.4.2.0 is available
Version 1.4.2.0 of our LPAR-Tool is available now for download. It includes a test license with validity until 30th june 2020!
LPAR-Tool: Which LPARs have no RMC-Connection
Status and configuration of LPARs are regularly needed information in the administration of LPARs. With the LPAR tool, information such as status, RMC status, number of cores, size of RAM, OS version and other data can be easily and quickly determined, even with hundreds or thousands of LPARs. Which LPARs don’t have an RMC connection is shown as one of the examples.
All of the following examples were performed on an environment with 10 HMCs, 50 managed systems, and just over 500 LPARs. To determine how long the LPAR tool requires, the run times of the commands were measured and specified using time.
The names of the LPARs were manually changed in the outputs shown and replaced by generic names lparXX and aixYY.
First of all, the status of a single LPAR:
$ time lpar status aix01 NAME LPAR_ID LPAR_ENV STATE PROFILE SYNC RMC PROCS PROC_UNITS MEM OS_VERSION aix01 27 aixlinux Running standard 0 active 1 0.1 8192 AIX 7.1 7100-04-02-1614 real 0m0.210s user 0m0.011s sys 0m0.013s $
Of course you can also specify multiple LPARs. If you want to know the status of all LPARs (in our case just over 500 LPARs), just leave out the argument:
$ time lpar status NAME LPAR_ID LPAR_ENV STATE PROFILE SYNC RMC PROCS PROC_UNITS MEM OS_VERSION aix01 27 aixlinux Running standard 0 active 1 0.1 8192 AIX 7.1 7100-04-02-1614 aix02 1 aixlinux Running standard - - 1 - 8320 Unknown ... lpar01 6 aixlinux Running standard 0 active 1 0.4 20480 AIX 7.1 7100-04-05-1720 real 0m18.933s user 0m3.819s sys 0m3.789s $
In the background, the LPAR tool executes more than 150 commands on the corresponding HMCs (lshwres and lssyscfg)!
The output should now be restricted to LPARs that are currently active (state=Running). There is the option “-s“, which can be used to specify criteria for attributes that must be met. Only LPARs meeting these criteria will be shown:
$ time lpar status -s state=Running NAME LPAR_ID LPAR_ENV STATE PROFILE SYNC RMC PROCS PROC_UNITS MEM OS_VERSION aix01 27 aixlinux Running standard 0 active 1 0.1 8192 AIX 7.1 7100-04-02-1614 aix02 1 aixlinux Running standard - - 1 - 8320 Unknown ... lpar01 6 aixlinux Running standard 0 active 1 0.4 20480 AIX 7.1 7100-04-05-1720 real 0m17.998s user 0m3.692s sys 0m3.647s $
Now we want to know on which of these LPARs RMC is not working/not active. The option “-s” allows to combine any number of criteria. All specified criteria must then be met (logical AND). The RMC state can be found in the attribute rmc_state:
$ time lpar status -s state=Running,rmc_state!=active NAME LPAR_ID LPAR_ENV STATE PROFILE SYNC RMC PROCS PROC_UNITS MEM OS_VERSION aix02 1 aixlinux Running standard - - 1 - 8320 Unknown aix03 2 aixlinux Running standard - - 1 - 8320 Unknown ... lpar07 4 aixlinux Running standard 0 none 1 1.0 4352 Unknown real 0m19.057s user 0m3.550s sys 0m3.512s $
As another example we want to know on which LPARs AIX 7.1 TL5 is installed. The os_version attribute contains the OS version. The ‘~‘ operator can be used to compare against a regular expression (similar to the grep command). We use the regular expression 7100-05:
$ time lpar status -s os_version~7100-05 NAME LPAR_ID LPAR_ENV STATE PROFILE SYNC RMC PROCS PROC_UNITS MEM OS_VERSION aix14 14 aixlinux Running standard 0 active 2 0.2 16384 AIX 7.1 7100-05-02-1810 aix16 24 aixlinux Running standard 0 active 2 0.2 16384 AIX 7.1 7100-05-03-1846 ... lpar10 10 aixlinux Running standard 0 active 3 0.3 32768 AIX 7.1 7100-05-02-1810 real 0m18.212s user 0m3.726s sys 0m3.676s $
So far, we have always used the default output format. Now we’d like to list all systems that still run on AIX 6.1, but this time only the LPAR name and the OS version will be output. For this there is the option “-F“, with which the desired output fields can be specified:
$ time lpar status -s os_version~6100 -F name:os_version aix39:AIX 6.1 6100-07-04-1216 aix46:AIX 6.1 6100-07-04-1216 ... lpar35:AIX 6.1 6100-09-05-1524 real 0m18.041s user 0m3.619s sys 0m3.699s $
If you prefer JSON output, you can easily do that with the option “-j“, here’s the same example with JSON output:
$ time lpar status -s os_version~6100 -F name:os_version -j
{
"name": "aix39",
"os_version": "AIX 6.1 6100-07-04-1216"
}
{
"name": "aix46",
"os_version": "AIX 6.1 6100-07-04-1216"
}
...
{
"name": "lpar35",
"os_version": "AIX 6.1 6100-09-05-1524"
}
real 0m21.247s
user 0m3.670s
sys 0m3.720s
$
Of course you can not know all attribute names by heart! But that’s not necessary, because you can easily see all attribute names. Use the option “-f” (stanza format) and specify any LPAR:
$ lpar status -f lpar19 lpar19: curr_lpar_proc_compat_mode = POWER7 curr_mem = 8192 curr_proc_mode = shared curr_proc_units = 0.3 curr_procs = 2 name = lpar19 os_version = AIX 6.1 6100-09-05-1524 ... $
With the options “-h” and “-m” the LPARs can be selected depending on the associated HMC and/or managed system.
Status of all LPARs with associated HMC hmc01:
$ lpar -h hmc01 status
Status of all LPARs whose corresponding HMC has type 7042-CR6:
$ lpar -h 7042-CR6 status
Status of all LPARs whose associated HMC has type 7042-CR6 and whose name begins with lpar:
$ lpar -h 7042-CR6 status lpar*
Status of all LPARs on the managed system ms13:
$ lpar -m ms13 status
Status of all LPARs whose managed system is an S922:
$ lpar -m 9009-22A status
The presented selection and output options apply to all output commands of the LPAR tool (except the vios command).
The LPAR-Tool can be downloaded from our download area: https://powercampus.de/en/download-2
The LPAR-Tool contains a test license which is valid until the end of october.
LPAR tool 1.4.0.1 available (including a valid test license)!
In our download area, version 1.4.0.1 of our LPAR tool, including a valid test license (valid until 31th october 2019) is available for download. The license is contained directly in the binaries, so no license key must be entered. The included trial license allows use of the LPAR tool for up to 10 HMCs, 100 managed systems and 1000 LPARs.
LPAR tool 1.3.0.2 with statically linked C++ libraries
In case somebody should have problems with the dynamically linked libstdc++.so and libgcc_s.so libraries, version 1.3.0.2 of the LPAR tool with statically linked libraries is available on our download page.
LPAR console using Virtual I/O Server
Typically, a console for an LPAR is launched via an HMC, via GUI or CLI (vtmenu or mkvterm). A console depends on the availability of an HMC. During an HMC update or problems with the HMC, you may not be able to connect to an LPAR console.
Relatively unknown is the ability to configure a console to an LPAR via a virtual I/O server. If the HMC is not available, then a console can be started via the virtual I/O server. No configuration is required on the client LPAR! By default, each client LPAR has 2 virtual serial server adapters (slots 0 and 1). If you configure an associated client adapter on a virtual I/O server, you can use it for a console connection.
On the virtual I/O server one needs only an unused virtual slot (here slot 45). The client LPAR has the LPAR ID 39. The virtual serial client adapter can be created with the following command:
hmc01 $ chhwres -m ms02 -r virtualio --rsubtype serial -o a -p ms02-vio1 -s 45 -a adapter_type=client,remote_lpar_name=aix02,remote_slot_num=0,supports_hmc=0 hmc01 $
Now you can always start a console for the LPAR via the virtual I/O server:
ms02-vio1 :/home/padmin> mkvt -id 39 AIX Version 7 Copyright IBM Corporation, 1982, 2018. Console login: root root's Password: XXXXXX aix02 AIX 7.2 powerpc Last unsuccessful login: Mon Mar 18 23:14:26 2019 on ssh from N.N.N.N Last login: Wed Mar 27 20:19:22 2019 on /dev/pts/0 from M.M.M.M [YOU HAVE NEW MAIL] aix02:/root> hostname aix02 aix02:/root>
The command mkvt on the virtual I/O server corresponds to the command mkvterm on the HMC. Here the desired partition must be specified by the LPAR-ID. Terminating the console works as usual with “~.“, Or if you are logged in via SSH on the virtual I/O server with “~~.“.
Alternatively, you can also end a console session with the command rmvt:
ms02-vio1:/home/padmin> rmvt -id 39 ms02-vio1:/home/padmin>
The following message appears in the console and the console is closed:
Virtual terminal has been disconnected. $
With the LPAR tool, the console can of course be set up even easier. The virtual serial adapter on the virtual I/O server can be created with the command “lpar addserial“, a manual login to the HMC is not necessary for this to work:
$ lpar addserial -c ms02-vio1 45 aix02 0 $
The “-c” option means “create client adapter”. The command also creates the adapter in the profile. The success of the action can be checked by “lpar vslots“, showing all virtual adapters of an LPAR:
$ lpar vslots ms02-vio1 SLOT REQ TYPE DATA 0 1 serial/server remote: -(any)/any status=unavailable hmc=1 1 1 serial/server remote: -(any)/any status=unavailable hmc=1 2 0 eth PVID=1 VLANS=- XXXXXXXXXXXX ETHERNET0 3 1 eth TRUNK(1) IEEE PVID=1 VLANS=201 XXXXXXXXXXXXX ETHERNET0 ... 45 0 serial/client remote: aix02(39)/0 status=unavailable hmc=0 ... $
Starting the console then proceeds as usual by logging in as padmin on the virtual I/O server and the command mkvt.
Caution: The console session through the virtual I/O server should always be terminated when it is no longer needed. You can not terminate it from the HMC! Here is the attempt to start a console using the HMC, while the console is already active using the virtual I/O server:
$ lpar console aix02 Open in progress A terminal session is already open for this partition. Only one open session is allowed for a partition. Exiting.... Attempts to open the session failed. Please close the terminal and retry the open at a later time. If the problem persists, Please contact IBM support. Received end of file, Exiting. Connection to X.X.X.X closed. $
Even rmvterm does not help:
$ lpar rmvterm aix02 /bin/stty: standard input: Inappropriate ioctl for device $
Conversely, no console can be started using the virtual I/O server if a console is active using the HMC:
ms02-vio1:/home/padmin> mkvt -id 39 Virtual terminal is already connected. ms02-vio1:/home/padmin>
So always make sure that the console is terminated.
Migration from SDDPCM to AIX PCM
Many AIX systems still use SDDPCM as a multipathing solution. However, SDDPCM is no longer supported on POWER 9 hardware from IBM.
The following is the migration from SDDPCM to AIX PCM. On our example system we have the following physical volumes:
$ lspv
hdisk0 00abcdefabcde000 datavg active
hdisk1 00abcdefabcde001 datavg active
hdisk2 none None
hdisk3 00abcdefabcde003 altinst_rootvg
hdisk4 00abcdefabcde004 rootvg active
$The Physical Volumes are disks that are made available through an SVC:
$ lsdev -l hdisk0 -F uniquetype
disk/fcp/2145
$The Path Control Module (PCM) uses SDDPCM:
$ lsattr -El hdisk0 -a PCM -F value
PCM/friend/sddpcm
$You can also see this when looking at the list of kernel extensions:
$ genkex | grep pcm
f1000000c012a000 af000 /usr/lib/drivers/sddpcmke
$Which PCM driver is used for which disk types can be easily viewd with the command “manage_disk_drivers”:
$ manage_disk_drivers -l
Device Present Driver Driver Options
2810XIV AIX_AAPCM AIX_AAPCM,AIX_non_MPIO
DS4100 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS4200 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS4300 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS4500 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS4700 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS4800 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS3950 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS5020 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DCS3700 AIX_APPCM AIX_APPCM
DCS3860 AIX_APPCM AIX_APPCM
DS5100/DS5300 AIX_SDDAPPCM AIX_APPCM,AIX_SDDAPPCM
DS3500 AIX_APPCM AIX_APPCM
XIVCTRL MPIO_XIVCTRL MPIO_XIVCTRL,nonMPIO_XIVCTRL
2107DS8K NO_OVERRIDE AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
IBMFlash NO_OVERRIDE AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
IBMSVC NO_OVERRIDE AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$In our case, SVC disks, the last line is relevant (IBMSVC). As current PCM driver NO_OVERRIDE is listed here, possible other drivers are AIX_AAPCM (AIX PCM for active / active and ALUASystems) and AIX_non_MPIO (drives without multi-pathing). The value NO_OVERRIDE means that if no multipathing driver is explicitly specified, a multipathing driver is used if possible (if available), otherwise no multipathing driver is used. If more than one multipathing driver is available (in our case AIX PCM and SDDPCM, then SDDPCM has priority).
In a subsequent blog entry, we will take a closer look at the possible values, as well as the point in AIX where the selection is made.
Before we change the driver for IBMSVC disks (a reboot is necessary), let’s take a look at the attributes of our disks, here an example for the hdisk0:
$ lsattr -El hdisk0
PCM PCM/friend/sddpcm PCM True
...
algorithm load_balance Algorithm True+
...
queue_depth 120 Queue DEPTH True+
...
reserve_policy no_reserve Reserve Policy True+
...
$Changing the driver will cause the values of some set attributes to be lost or replaced by new default values of the new driver. This is especially true for the queue_depth (here: 120), the reserve_policy (here: no_reserve) and the load-balancing policy (algorithm). The current values should be noted, then after the conversion to the AIX PCM driver then adjust accordingly.
Switching to AIX PCM can be done with the command “manage_disk_drivers”. For this, the command is given the disk type (here IBMSVC) with the option “-d” and the desired driver (here AIX_AAPCM for the AIX PCM driver) with the option “-o”:
# manage_disk_drivers -d IBMSVC -o AIX_AAPCM
********************** ATTENTION *************************
For the change to take effect the system must be rebooted
#The changed configuration can be listed directly with “manage_disk_drivers -l”:
$ manage_disk_drivers -l
Device Present Driver Driver Options
...
IBMSVC AIX_AAPCM AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$To make the change, the system must now be rebooted:
# shutdown -r now
SHUTDOWN PROGRAM
Thu Feb 7 09:43:38 CET 2019
...We execute the 3 commands from the beginning again (lspv, lsdevund lsattr):
$ lspv
hdisk0 00abcdefabcde000 datavg active
hdisk1 00abcdefabcde001 datavg active
hdisk2 none None
hdisk3 00abcdefabcde003 altinst_rootvg
hdisk4 00abcdefabcde004 rootvg active
$The physical volumes are unchanged.
$ lsdev -l hdisk0 -F uniquetype
disk/fcp/mpioosdisk
$The type of disks has changed from disk / fcp / 2145 to disk / fcp / mpioosdisk. This already indicates that the multipathing driver has changed.
$ lsattr -El hdisk0 -a PCM -F value
PCM/friend/fcpother
$The Path Control Module (PCM) has also changed. The guy is no longer sddpcm but fcpother. That does not look directly after AIX PCM. However, a look at the corresponding driver shows immediately that AIX PCM is in use here:
$ lsdev -P -c PCM -s friend -t fcpother -F DvDr
aixdiskpcmke
$The associated kernel extension aixdiskpcmke is also currently loaded and in use:
$ genkex | grep pcm
73e2000 57000 /usr/lib/drivers/aixdiskpcmke
$Let’s take a look at the attributes of hdisk0 again. We expect changed values for some attributes here:
$ lsattr -El hdisk0
PCM PCM/friend/fcpother Path Control Module False
...
algorithm fail_over Algorithm True+
...
queue_depth 20 Queue DEPTH True+
...
reserve_policy single_path Reserve Policy True+
...
$The value 120 for the queue_depth has been lost and has been replaced by the default value 20. The reserve_policy has changed to single_path and the load-balancing algorithm is now fail_over, i. only one path is used at a time.
We change the settings to a configuration that corresponds to the initial situation:
# chdev -P -l hdisk0 -a algorithm=shortest_queue -a queue_depth=120 -a reserve_policy=no_reserve
hdisk0 changed
#Since the Physical Volume is in use, the setting can only be changed in the ODM and a further reboot is necessary.
After all disks have been reconfigured via the ODM, the system must be rebooted a second time:
# shutdown -r now
SHUTDOWN PROGRAM
Thu Feb 6 20:07:12 CET 2019
...After the reboot SDDPCM can be uninstalled:
# installp -u devices.fcp.disk.ibm.mpio.rte devices.sddpcm.72.rte
+-----------------------------------------------------------------------------+
Pre-deinstall Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
SUCCESSES
...
0503-292 This update will not fully take effect until after a
system reboot.
* * * A T T E N T I O N * * *
System boot image has been updated. You should reboot the
system as soon as possible to properly integrate the changes
and to avoid disruption of current functionality.
installp: bosboot process completed.
+-----------------------------------------------------------------------------+
Summaries:
+-----------------------------------------------------------------------------+
Installation Summary
--------------------
Name Level Part Event Result
-------------------------------------------------------------------------------
devices.sddpcm.72.rte 2.7.1.1 ROOT DEINSTALL SUCCESS
devices.sddpcm.72.rte 2.7.1.1 USR DEINSTALL SUCCESS
devices.fcp.disk.ibm.mpio.r 1.0.0.25 USR DEINSTALL SUCCESS
#The SDDPCM fileset, as well as the associated host-attachment fileset, were successfully uninstalled.
Since the SDDPCM driver was not loaded, and thus no changes have been made to the kernel, actually another reboot should not be necessary. However, since it is explicitly pointed out a quick reboot, and it is also a good idea to do a reboot test with the final configuration, we reboot the system a third and final time:
# shutdown -r now
SHUTDOWN PROGRAM
Thu Feb 6 20:17:21 CET 2019
...After the reboot, we check the disk attributes again:
$ lsattr -El hdisk0
PCM PCM/friend/fcpother Path Control Module False
...
algorithm shortest_queue Algorithm True+
...
queue_depth 120 Queue DEPTH True+
...
reserve_policy no_reserve Reserve Policy True+
...
$The system now uses the AIX PCM driver for multipathing:
$ manage_disk_drivers -l
Device Present Driver Driver Options
...
IBMSVC AIX_AAPCM AIX_AAPCM,AIX_non_MPIO,NO_OVERRIDE
$Migrating from SDDPCM to AIX PCM is pretty easy to do.