LPAR-Tool: Which LPARs have no RMC-Connection

Status and configuration of LPARs are regularly needed information in the administration of LPARs. With the LPAR tool, information such as status, RMC status, number of cores, size of RAM, OS version and other data can be easily and quickly determined, even with hundreds or thousands of LPARs. Which LPARs don’t have an RMC connection is shown as one of the examples.

All of the following examples were performed on an environment with 10 HMCs, 50 managed systems, and just over 500 LPARs. To determine how long the LPAR tool requires, the run times of the commands were measured and specified using time.

The names of the LPARs were manually changed in the outputs shown and replaced by generic names lparXX and aixYY.

First of all, the status of a single LPAR:

$ time lpar status aix01
NAME   LPAR_ID  LPAR_ENV   STATE     PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01  27       aixlinux   Running   standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614

real    0m0.210s
user    0m0.011s
sys     0m0.013s
$

Of course you can also specify multiple LPARs. If you want to know the status of all LPARs (in our case just over 500 LPARs), just leave out the argument:

$ time lpar status
NAME    LPAR_ID  LPAR_ENV   STATE     PROFILE      SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01   27       aixlinux   Running   standard     0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02   1        aixlinux   Running   standard     -     -         1      -           8320    Unknown
...
lpar01  6        aixlinux   Running   standard     0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m18.933s
user	0m3.819s
sys	0m3.789s
$

In the background, the LPAR tool executes more than 150 commands on the corresponding HMCs (lshwres and lssyscfg)!

The output should now be restricted to LPARs that are currently active (state=Running). There is the option “-s“, which can be used to specify criteria for attributes that must be met. Only LPARs meeting these criteria will be shown:

$ time lpar status -s state=Running
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01    27       aixlinux   Running  standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar01   6        aixlinux   Running  standard   0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m17.998s
user	0m3.692s
sys	0m3.647s
$

Now we want to know on which of these LPARs RMC is not working/not active. The option “-s” allows to combine any number of criteria. All specified criteria must then be met (logical AND). The RMC state can be found in the attribute rmc_state:

$ time lpar status -s state=Running,rmc_state!=active
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
aix03    2        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar07   4        aixlinux   Running  standard   0     none      1      1.0         4352    Unknown

real	0m19.057s
user	0m3.550s
sys	0m3.512s
$

As another example we want to know on which LPARs AIX 7.1 TL5 is installed. The os_version attribute contains the OS version. The ‘~‘ operator can be used to compare against a regular expression (similar to the grep command). We use the regular expression 7100-05:

$ time lpar status -s os_version~7100-05
NAME     LPAR_ID  LPAR_ENV  STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix14    14       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-02-1810
aix16    24       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-03-1846
...
lpar10   10       aixlinux  Running  standard   0     active    3      0.3         32768   AIX 7.1 7100-05-02-1810

real	0m18.212s
user	0m3.726s
sys	0m3.676s
$

So far, we have always used the default output format. Now we’d like to list all systems that still run on AIX 6.1, but this time only the LPAR name and the OS version will be output. For this there is the option “-F“, with which the desired output fields can be specified:

$ time lpar status -s os_version~6100 -F name:os_version
aix39:AIX 6.1 6100-07-04-1216
aix46:AIX 6.1 6100-07-04-1216
...
lpar35:AIX 6.1 6100-09-05-1524

real	0m18.041s
user	0m3.619s
sys	0m3.699s
$

If you prefer JSON output, you can easily do that with the option “-j“, here’s the same example with JSON output:

$ time lpar status -s os_version~6100 -F name:os_version -j
{
	"name": "aix39",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
{
	"name": "aix46",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
...
{
	"name": "lpar35",
	"os_version": "AIX 6.1 6100-09-05-1524"
}

real	0m21.247s
user	0m3.670s
sys	0m3.720s
$

Of course you can not know all attribute names by heart! But that’s not necessary, because you can easily see all attribute names. Use the option “-f” (stanza format) and specify any LPAR:

$ lpar status -f lpar19
lpar19:
	curr_lpar_proc_compat_mode = POWER7
	curr_mem = 8192
	curr_proc_mode = shared
	curr_proc_units = 0.3
	curr_procs = 2
	name = lpar19
	os_version = AIX 6.1 6100-09-05-1524
...
$

With the options “-h” and “-m” the LPARs can be selected depending on the associated HMC and/or managed system.

Status of all LPARs with associated HMC hmc01:

$ lpar -h hmc01 status

Status of all LPARs whose corresponding HMC has type 7042-CR6:

$ lpar -h 7042-CR6 status

Status of all LPARs whose associated HMC has type 7042-CR6 and whose name begins with lpar:

$ lpar -h 7042-CR6 status lpar*

Status of all LPARs on the managed system ms13:

$ lpar -m ms13 status

Status of all LPARs whose managed system is an S922:

$ lpar -m 9009-22A status

The presented selection and output options apply to all output commands of the LPAR tool (except the vios command).

The LPAR-Tool can be downloaded from our download area: https://powercampus.de/en/download-2

The LPAR-Tool contains a test license which is valid until the end of october.

FC NPIV client throughput

When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like

  • how much throughput is achieved by a particular LPAR
  • which LPARs have the highest throughput and produce the most FC traffic
  • are there resource bottlenecks

could be answered.

Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:

(0)padmin@aixvio1:/home/padmin> fcstat -client
              hostname   dev                wwpn     inreqs    outreqs ctrlreqs          inbytes         outbytes  DMA_errs Elem_errs Comm_errs

               aixvio1  fcs0  0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576         0         0         0
     C050760YYYYYYYYY
                                    0          0        0                0                0         0         0         0
     C050760ZZZZZZZZZ
                                    0          0        0                0                0         0         0         0
                 aix01  fcs0  0xC050760XXXXXXXXX   22685402  101956075 10065757     699512617896    1572578056704         0         0         0
                 aix02  fcs0  0xC050760XXXXXXXXX   28200473   82295158 12051365     387847746448     626772151808         0         0         0
                 aix03  fcs0  0xC050760XXXXXXXXX  376500672  255163053 21583628   22619424512608    3786990844928         0         0         0
                 aix04  fcs0  0xC050760XXXXXXXXX  116450405  504688524 14020031    4037786527400    9929289617408         0         0         0
          blbprodora22  fcs0  0xC050760XXXXXXXXX 1341092479  580673554 37458927   44288566807072   12166718497792         0         0         0
...
               aixvio1  fcs1  0x100000XXXXXXXXXX  391131484 1090556094 156294130   71031615240217   87642294572864         0         0         0
              aixtsm01  fcs2  0xC050760XXXXXXXXX  334020900  785597352 74659821   62072552942128   83284555980288         0         0         0
              aixtsm02  fcs0  0xC050760XXXXXXXXX    2943054   40921231 11617552     107317697968     289142333440         0         0         0

               aixvio1  fcs2  0x210000XXXXXXXXXX  403180246 5877180796   236998  105482699300998 1540608710446612         0         0         0
              aixtsm01  fcs6  0xC050760XXXXXXXXX  146492419  392365162    74250   38378099796342  102844775468007         0         0         0
              aixtsm02  fcs2  0xC050760XXXXXXXXX         19     192848       20             1090      50551063184         0         0         0

               aixvio1  fcs3  0x210000XXXXXXXXXX  405673338 7371951499   260575  105969796271246 1932388891128304         0         0         0
              aixtsm02  fcs3  0xC050760XXXXXXXXX          0          0        4                0                0         0         0         0
                 aix02  fcs7  0xC050760XXXXXXXXX      42624 2677470211    34211          2382280  701864613402184         0         0         0
...
Invalid initiator world wide name
Invalid initiator world wide name
(0)padmin@aixvio1:/home/padmin>

The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.

The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.

Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:

$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#

delta=5 # seconds

typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs

bc |& # start bc as coroutine
print -p "scale=2"

# get first sample

/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta

while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs

print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs

printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print

sleep $delta
done

$

The script ‘npivstat‘ is available for download in our download-area.

Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):

aixvio1 # ./npivstat
       hostname    dev              wwpn  inreqs  outreqs  inbytes  outbytes  DMA_errs  Elem_errs  Comm_errs
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1019.00     0.00    254.75      0.00       0.00       0.00
       aixtsm01   fcs6  0xC0507605E5890074    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC0507609A6C70004    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC0507609A6C7001C    0.00  1018.20     0.00    254.55      0.00       0.00       0.00
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1020.20     0.00    255.05      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00  1019.80     0.00    254.95      0.00       0.00       0.00
...                                                                                                           
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00   984.80     0.00    246.20      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00   985.00     0.00    246.25      0.00       0.00       0.00
...
^Caixvio1 # 

In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.

The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.

LPAR console using Virtual I/O Server

Typically, a console for an LPAR is launched via an HMC, via GUI or CLI (vtmenu or mkvterm). A console depends on the availability of an HMC. During an HMC update or problems with the HMC, you may not be able to connect to an LPAR console.

Relatively unknown is the ability to configure a console to an LPAR via a virtual I/O server. If the HMC is not available, then a console can be started via the virtual I/O server. No configuration is required on the client LPAR! By default, each client LPAR has 2 virtual serial server adapters (slots 0 and 1). If you configure an associated client adapter on a virtual I/O server, you can use it for a console connection.

On the virtual I/O server one needs only an unused virtual slot (here slot 45). The client LPAR has the LPAR ID 39. The virtual serial client adapter can be created with the following command:

hmc01 $ chhwres -m ms02 -r virtualio --rsubtype serial -o a -p ms02-vio1 -s 45 -a adapter_type=client,remote_lpar_name=aix02,remote_slot_num=0,supports_hmc=0
hmc01 $

Now you can always start a console for the LPAR via the virtual I/O server:

ms02-vio1 :/home/padmin> mkvt -id 39
AIX Version 7
Copyright IBM Corporation, 1982, 2018.
Console login: root
root's Password: XXXXXX


aix02  AIX 7.2         powerpc


Last unsuccessful login: Mon Mar 18 23:14:26 2019 on ssh from N.N.N.N
Last login: Wed Mar 27 20:19:22 2019 on /dev/pts/0 from M.M.M.M
[YOU HAVE NEW MAIL]
aix02:/root> hostname
aix02
aix02:/root>

The command mkvt on the virtual I/O server corresponds to the command mkvterm on the HMC. Here the desired partition must be specified by the LPAR-ID. Terminating the console works as usual with “~.“, Or if you are logged in via SSH on the virtual I/O server with “~~.“.

Alternatively, you can also end a console session with the command rmvt:

ms02-vio1:/home/padmin> rmvt -id 39
ms02-vio1:/home/padmin>

The following message appears in the console and the console is closed:

Virtual terminal has been disconnected.

$

With the LPAR tool, the console can of course be set up even easier. The virtual serial adapter on the virtual I/O server can be created with the command “lpar addserial“, a manual login to the HMC is not necessary for this to work:

$ lpar addscsi -c ms02-vio1 45 aix02 0
$

The “-c” option means “create client adapter”. The command also creates the adapter in the profile. The success of the action can be checked by “lpar vslots“, showing all virtual adapters of an LPAR:

$ lpar vslots ms02-vio1
SLOT  REQ  TYPE           DATA
0     1    serial/server  remote: -(any)/any status=unavailable hmc=1
1     1    serial/server  remote: -(any)/any status=unavailable hmc=1
2     0    eth            PVID=1 VLANS=- XXXXXXXXXXXX ETHERNET0
3     1    eth            TRUNK(1) IEEE PVID=1 VLANS=201 XXXXXXXXXXXXX ETHERNET0
...
45     0   serial/client  remote: aix02(39)/0 status=unavailable hmc=0
...
$

Starting the console then proceeds as usual by logging in as padmin on the virtual I/O server and the command mkvt.

Caution: The console session through the virtual I/O server should always be terminated when it is no longer needed. You can not terminate it from the HMC! Here is the attempt to start a console using the HMC, while the console is already active using the virtual I/O server:

$ lpar console aix02

Open in progress 

A terminal session is already open for this partition. 
Only one open session is allowed for a partition. 
Exiting.... 
Attempts to open the session failed. Please close the terminal and retry the open at a later time. 
If the problem persists, Please contact IBM support. 
Received end of file, Exiting.
Connection to X.X.X.X closed.
$

Even rmvterm does not help:

$ lpar rmvterm aix02
/bin/stty: standard input: Inappropriate ioctl for device
$

Conversely, no console can be started using the virtual I/O server if a console is active using the HMC:

ms02-vio1:/home/padmin> mkvt -id 39
Virtual terminal is already connected.

ms02-vio1:/home/padmin>

So always make sure that the console is terminated.