LPAR-Tool in Action: Examples

The LPAR tool can administer HMCs, managed systems, LPARs and virtual-I/O-servers via the command line. The current version of the LPAR tool (currently 1.4.0.2) can be downloaded from our download page https://powercampus.de/download. A trial license, valid until October 31, is included. This article will show you some simple but useful applications of the LPAR tool.

A common question in larger environments (multiple HMCs, many managed systems) is: where is a particular LPAR? This question can easily be answered with the LPAR tool, by using the command “lpar show“:

$ lpar show lpar02
NAME    ID  SERIAL     LPAR_ENV  MS    HMCS
lpar02  39  123456789  aixlinux  ms21  hmc01,hmc02
$

In addition to the name, the LPAR-ID and the serial number, the managed system, here ms21, and the associated HMCs, here hmc01 and hmc02, are also shown. You can also specify multiple LPARs and/or wildcards:

$ lpar show lpar02 lpar01
...
$ lpar show lpar*
...
$

If no argument is given, all LPARs are listed.

 

Another question that frequently arises is the status of an LPAR or multiple LPARs. Again, this can be easily answered, this time with the command “lpar status“:

$ lpar status lpar02
NAME    LPAR_ID  LPAR_ENV  STATE    PROFILE   SYNC  RMC     PROCS  PROC_UNITS  MEM   OS_VERSION
lpar02  39       aixlinux  Running  standard  0     active  1      0.7         7168  AIX 7.2 7200-03-02-1846
$

The LPAR lpar02 is Running, the profile used is standard, the RMC connection is active and the LPAR is running AIX 7.2 (TL3 SP2). The LPAR has 1 processor core, with 0.7 processing units and 7 GB RAM. The column SYNC indicates whether the current configuration is synchronized with the profile (attribute sync_curr_profile).

Of course, several LPARs or even all LPARs can be specified here.

If you want to see what the LPAR tool does in the background: for most commands you can specify the option “-v” for verbose-only. The HMC commands will then be listed, but no changes will be made to the HMC. Here are the HMC commands that are issued for the status output:

$ lpar status -v lpar02
hmc01: lssyscfg -r lpar -m ms21
hmc01: lshwres -r mem -m ms21 --level lpar
hmc01: lshwres -r proc -m ms21 --level lpar
$

 

Next, the addition of additional RAM will be shown. We start with the status of the LPAR:

$ lpar status lpar02
NAME    LPAR_ID  LPAR_ENV  STATE    PROFILE   SYNC  RMC     PROCS  PROC_UNITS  MEM   OS_VERSION
lpar02  39       aixlinux  Running  standard  0     active  1      0.7         7168  AIX 7.2 7200-03-02-1846
$

The LPAR is running and RMC is active, so a DLPAR operation should be possible. We will first check if the maximum memory size is already in use:

$ lpar lsmem lpar02
            MEMORY         MEMORY         HUGE_PAGES 
LPAR_NAME  MODE  AME  MIN   CURR  MAX   MIN  CURR  MAX
lpar02     ded   0.0  2048  7168  8192  0    0     0
$

Currently the LPAR uses 7 GB and a maximum of 8 GB are possible. Extending the memory by 1 GB (1024 MB) should be possible. We add the memory by using the command “lpar addmem“:

$ lpar addmem lpar02 1024
$

We check the success by starting the command “lpar lsmem” again:

$ lpar lsmem lpar02
           MEMORY         MEMORY         HUGE_PAGES 
LPAR_NAME  MODE  AME  MIN   CURR  MAX   MIN  CURR  MAX
lpar02     ded   0.0  2048  8192  8192  0    0     0
$

(By the way: if the current configuration is not synchronized with the current profile, attribute sync_curr_profile, then the LPAR tool also updates the profile!)

 

Virtual adapters can be listed using “lpar lsvslot“:

$ lpar lsvslot lpar02
SLOT  REQ  ADAPTER_TYPE   STATE  DATA
0     Yes  serial/server  1      remote: (any)/any connect_status=unavailable hmc=1
1     Yes  serial/server  1      remote: (any)/any connect_status=unavailable hmc=1
2     No   eth            1      PVID=123 VLANS= ETHERNET0 XXXXXXXXXXXX
6     No   vnic           -      PVID=1234 VLANS=none XXXXXXXXXXXX failover sriov/ms21-vio1/1/3/0/2700c003/2.0/2.0/20/100.0/100.0,sriov/ms21-vio2/2/1/0/27004004/2.0/2.0/10/100.0/100.0
10    No   fc/client      1      remote: ms21-vio1(1)/47 c050760XXXXX0016,c050760XXXXX0017
20    No   fc/client      1      remote: ms21-vio2(2)/25 c050760XXXXX0018,c050760XXXXX0019
21    No   scsi/client    1      remote: ms21-vio2(2)/20
$

The example shows virtual FC and SCSI adapters as well as a vNIC adapter in slot 6.

 

Finally, we’ll show how to start a console for an LPAR:

$ lpar console lpar02

Open in progress 

 Open Completed.

…

AIX Version 7

Copyright IBM Corporation, 1982, 2018.

Console login:

…

The console can be terminated with “~.“.

 

Of course, the LPAR tool can do much more.

To be continued.

 

LPAR tool 1.4.0.1 available (including a valid test license)!

In our download area, version 1.4.0.1 of our LPAR tool, including a valid test license (valid until 31th october 2019) is available for download. The license is contained directly in the binaries, so no license key must be entered. The included trial license allows use of the LPAR tool for up to 10 HMCs, 100 managed systems and 1000 LPARs.

LPAR tool with test license until 15th september 2019

In our download area, version 1.3.0.2 of our LPAR tool, including a valid test license (valid until 15th september 2019) is available for download. The license is contained directly in the binaries, so no license key must be entered. The included trial license allows use of the LPAR tool for up to 10 HMCs, 100 managed systems and 1000 LPARs.

Resources of not activated LPARs and Memory Affinity

When an LPAR is shut down, resources such as processors, memory, and I/O slots are not automatically released by the LPAR. The resources remain assigned to the LPAR and are then reused on the next activation (with the current configuration). In the first part of the article Resources of not activated LPARs we had already looked at this.

(Note: In the example output, we use version 1.4 of the LPAR tool, but in all cases we show the underlying commands on the HMC command line, so you can try everything without using the LPAR tool.)

The example LPAR lpar1 was shut down, but currently still occupies 100 GB of memory:

linux $ lpar status lpar1
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  39       aixlinux  Not Activated  standard  0     inactive  1      0.2         102400  Unknown
linux $

The following commands for the output above were executed on the corresponding HMC hmc01:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1
hmc01: lshwres -r mem -m ms09 --level lpar --filter lpar_names=lpar1
hmc01: lshwres -r proc -m ms09 --level lpar --filter lpar_names=lpar1

As the output shows, the LPAR lpar1 has still allocated its resources (processors, memory, I/O adapters).

In order to understand why deactivating an LPAR does not release the resources, you have to look at the “Memory Affinity Score”:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      100   0
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

The Memory Affinity Score describes how close processors and memory are, the closer the memory to the memory is, the better is the throughput to the memory. The command above indicates, with a value between 1 and 100, how big the affinity between processors and LPARs is. Our LPAR lpar1 currently has a value of 100, which means the best possible affinity of memory and processors. If the resources were freed when deactivating an LPAR, then the LPAR would lose this Memory Affinity Score. The next time you enable the LPAR, it then depends on the memory and processors available then how good the memory affinity will be then. We release the resources once:

linux $ lpar -d rmprocs lpar1 1
linux $

HMC command line:

hmc01: chhwres -m ms09 -r proc  -o r -p lpar1 --procs 1

No more score will be given, since the LPAR has no longer allocated any resources:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      none  none
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

Now we allocate resources again and look at the effect this has on memory affinity:

linux $ lpar applyprof lpar1 standard
linux $

HMC command line:

hmc01: chsyscfg -r lpar -m ms09 -o apply -p lpar1 -n standard

We again determine the Memory Affinity Score:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      53    0
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

The score is now only 53, the performance of the LPAR has become worse. Whether and how much this is noticeable, depends ultimately on the applications on the LPAR.

The fact that the resources are not released when deactivating an LPAR, thus guarantees the next time you activate (with the current configuration) the memory affinity remains the same and thus the performance should be the same.

If you release the resources of an LPAR (manually or automatically), then you have to realize that this has an effect on the LPAR if it is later activated again, because then the resources are reassigned and a worse (but possibly also a better) Memory Affinity Score can result.

Conversely, before activating a new LPAR you can also make sure that there is a good chance for a high memory affinity score for the new LPAR by releasing resources of inactive LPARs.

(Note: resource distribution can be changed and improved at runtime using the Dynamic Platform Optimizer DPO. DPO is supported as of POWER8.)

 

Resources of not activated LPARs

When an LPAR is shutdown, resources such as processors, memory, and I/O slots are not automatically released by the LPAR. The resources remain assigned to the LPAR and are reused on the next activation (with the current configuration).

The article will show how such resources are automatically released and, if desired, how to manually release resources of an inactive LPAR.

(Note: In the example output, we use version 1.4 of the LPAR tool, but in all cases we show the underlying commands on the HMC command line, so you can try everything without using the LPAR tool.)

The example LPAR lpar1 was shut down, but currently still occupies 100 GB of memory:

linux $ lpar status lpar1
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  39       aixlinux  Not Activated  standard  0     inactive  1      0.2         102400  Unknown
linux $

The following commands for the output above were executed on the corresponding HMC hmc01:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1
hmc01: lshwres -r mem -m ms09 --level lpar --filter lpar_names=lpar1
hmc01: lshwres -r proc -m ms09 --level lpar --filter lpar_names=lpar1

The resource_config attribute of an LPAR indicates whether the LPAR has currently allocated resources (resource_config=1) or not (resource_config=0):

linux $ lpar status -F resource_config lpar1
1
linux $

Or on the HMC command line:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1 –F resource_config

The resources allocated by an not activated LPAR can be released in 2 different ways:

  1. Automatic: The resources used are needed by another LPAR, e.g. because memory is expanded dynamically or an LPAR is activated that does not have sufficient resources. In this case, resources are automatically removed from a not activated LPAR. We will show this below with an example.
  2. Manual: The allocated resources are explicitly released by the administrator. This is also shown below in an example.

First we show an example in which resources are automatically taken away from an not activated LPAR.

The managed system ms09 currently has about 36 GB free memory:

linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     33792     786432        36352  256
linux $

HMC command line:

hmc01: lshwres -r mem -m ms09 --level sys

We start an LPAR (lpar2) which was configured with 100 GB of RAM. The managed system has only 36 GB of RAM and is therefore forced to take resources from inactive LPARs in order to provide the required 100 GB. We start lpar2 with the profile standard and look at the memory relations:

linux $ lpar activate -b sms -p standard lpar2
linux $

HMC command line:

hmc01: chsysstate -m ms09 -r lpar -o on -n lpar2 -b sms -f standard

Overview of the storage relations of lpar1 and lpar2:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  1      0.2         60160   Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     35584     786432        0      256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The LPAR lpar2 has 100 GB RAM, the managed system has no more free memory and the memory allocated by LPAR lpar1 has been reduced to about 60 GB. Allocated resources from non-activated LPARs are automatically released, when needed and assigned to other LPARs.

But you can of course also release the resources manually. This is also shown briefly here. We are reducing the memory of LPAR lpar1 by 20 GB:

linux $ lpar -d rmmem lpar1 20480
linux $

HMC command line:

hmc01: chhwres -m ms09 -r mem  -o r -p lpar1 -q 20480

As stated, the allocated memory has been reduced by 20 GB:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  1      0.2         39680   Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     35584     786432        20480  256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The 20 GB are immediately available to the managed system as free memory. If you remove the entire memory or all processors (or processor units), then all resources of an inactive LPAR are released:

linux $ lpar -d rmmem lpar1 39680
linux $

HMC command line:

hmc01: chhwres -m ms09 -r mem  -o r -p lpar1 -q 39680

Here are the resulting memory relations:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  0      0.0         0       Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME        INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     31232     786432        64512  256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The LPAR lpar1 now has 0 processors, 0.0 processor units and 0 MB of memory! In addition, the resource_config attribute now has the value 0, which indicates that the LPAR no longer has any resources configured!

linux $ lpar status -F resource_config lpar1
0
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1 –F resource_config

Finally, the question arises as to why you should release resources manually if they are automatically released by the managed system when needed?

We will answer this question in a second article.

 

Accessing the Update Access Key Expiration Date from AIX

As part of the introduction of POWER8 systems, IBM has also introduced the “Update Access Key”, which is necessary to perform firmware updates of the managed system. By default, newly delivered systems have an update access key that usually expires after 3 years. Thereafter, the Update Access Key can has to be extended every 6 month, but only if a maintenance contract exists(https://www.ibm.com/servers/eserver/ess/index.wss).

Of course, it is easy to find out when the current Update Access Key runs through the HMC, GUI or CLI. But you can also display the expiration date via the lscfg command from AIX:

In the case of AIX 7.1, this looks like this:

$ lscfg -vpl sysplanar0 | grep -p "System Firmware"
      System Firmware:
...
        Microcode Image.............SV860_138 SV860_103 SV860_138
        Microcode Level.............FW860.42 FW860.30 FW860.42
        Microcode Build Date........20180101 20170628 20180101
        Microcode Entitlement Date..20190825
        Hardware Location Code......U8284.22A.XXXXXXX-Y1
      Physical Location: U8284.22A.XXXXXXX-Y1

In the case of AIX 7.2, the output is slightly different:

$ lscfg -vpl sysplanar0 |grep -p "System Firmware"
      System Firmware:
...
        Microcode Image.............SV860_138 SV860_103 SV860_138
        Microcode Level.............FW860.42 FW860.30 FW860.42
        Microcode Build Date........20180101 20170628 20180101
        Update Access Key Exp Date..20190825
        Hardware Location Code......U8284.22A.XXXXXXX-Y1
      Physical Location: U8284.22A.XXXXXXX-Y1

Relevant are the lines “Microcode Entitlement Date” respectively “Update Access Key Exp Date“.

FC NPIV client throughput

When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like

  • how much throughput is achieved by a particular LPAR
  • which LPARs have the highest throughput and produce the most FC traffic
  • are there resource bottlenecks

could be answered.

Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:

(0)padmin@aixvio1:/home/padmin> fcstat -client
              hostname   dev                wwpn     inreqs    outreqs ctrlreqs          inbytes         outbytes  DMA_errs Elem_errs Comm_errs

               aixvio1  fcs0  0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576         0         0         0
     C050760YYYYYYYYY
                                    0          0        0                0                0         0         0         0
     C050760ZZZZZZZZZ
                                    0          0        0                0                0         0         0         0
                 aix01  fcs0  0xC050760XXXXXXXXX   22685402  101956075 10065757     699512617896    1572578056704         0         0         0
                 aix02  fcs0  0xC050760XXXXXXXXX   28200473   82295158 12051365     387847746448     626772151808         0         0         0
                 aix03  fcs0  0xC050760XXXXXXXXX  376500672  255163053 21583628   22619424512608    3786990844928         0         0         0
                 aix04  fcs0  0xC050760XXXXXXXXX  116450405  504688524 14020031    4037786527400    9929289617408         0         0         0
          blbprodora22  fcs0  0xC050760XXXXXXXXX 1341092479  580673554 37458927   44288566807072   12166718497792         0         0         0
...
               aixvio1  fcs1  0x100000XXXXXXXXXX  391131484 1090556094 156294130   71031615240217   87642294572864         0         0         0
              aixtsm01  fcs2  0xC050760XXXXXXXXX  334020900  785597352 74659821   62072552942128   83284555980288         0         0         0
              aixtsm02  fcs0  0xC050760XXXXXXXXX    2943054   40921231 11617552     107317697968     289142333440         0         0         0

               aixvio1  fcs2  0x210000XXXXXXXXXX  403180246 5877180796   236998  105482699300998 1540608710446612         0         0         0
              aixtsm01  fcs6  0xC050760XXXXXXXXX  146492419  392365162    74250   38378099796342  102844775468007         0         0         0
              aixtsm02  fcs2  0xC050760XXXXXXXXX         19     192848       20             1090      50551063184         0         0         0

               aixvio1  fcs3  0x210000XXXXXXXXXX  405673338 7371951499   260575  105969796271246 1932388891128304         0         0         0
              aixtsm02  fcs3  0xC050760XXXXXXXXX          0          0        4                0                0         0         0         0
                 aix02  fcs7  0xC050760XXXXXXXXX      42624 2677470211    34211          2382280  701864613402184         0         0         0
...
Invalid initiator world wide name
Invalid initiator world wide name
(0)padmin@aixvio1:/home/padmin>

The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.

The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.

Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:

$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#

delta=5 # seconds

typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs

bc |& # start bc as coroutine
print -p "scale=2"

# get first sample

/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta

while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs

print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs

printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print

sleep $delta
done

$

The script ‘npivstat‘ is available for download in our download-area.

Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):

aixvio1 # ./npivstat
       hostname    dev              wwpn  inreqs  outreqs  inbytes  outbytes  DMA_errs  Elem_errs  Comm_errs
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1019.00     0.00    254.75      0.00       0.00       0.00
       aixtsm01   fcs6  0xC0507605E5890074    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC0507609A6C70004    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC0507609A6C7001C    0.00  1018.20     0.00    254.55      0.00       0.00       0.00
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1020.20     0.00    255.05      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00  1019.80     0.00    254.95      0.00       0.00       0.00
...                                                                                                           
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00   984.80     0.00    246.20      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00   985.00     0.00    246.25      0.00       0.00       0.00
...
^Caixvio1 # 

In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.

The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.

Did you know that state and configuration change information is available on the HMC for about 2 months?

Status and configuration changes of LPARs and managed systems are stored on the HMCs for about 2 months. This can be used to find out, when a managed system was shut down, when a service processor failover took place, or when the memory of an LPAR was expanded, at least if the event is no more than 2 months ago.

The status changes of a managed system can be listed with the command “lslparutil -r sys -m <managed-system> -sh –startyear 1970 –filter event_types = state_change“, or alternatively with the LPAR-Tool command “ms history <managed -system> “.

linux $ ms history ms04
TIME                  PRIMARY_STATE         DETAILED_STATE
03/14/2019 08:45:13   Started               None
03/14/2019 08:36:52   Not Available         Unknown
02/17/2019 01:51:55   Started               None
02/17/2019 01:44:00   Not Available         Unknown
02/12/2019 09:32:57   Started               None
02/12/2019 09:28:02   Started               Service Processor Failover
02/12/2019 09:27:07   Started               None
02/12/2019 09:24:42   Standby               None
02/12/2019 09:21:25   Starting              None
02/12/2019 09:22:59   Stopped               None
02/12/2019 09:21:58   Not Available         Unknown
02/12/2019 09:09:45   Stopped               None
02/12/2019 09:07:53   Stopping              None
linux $

Configuration changes (processor, memory) of a managed system can be displayed with “lslparutil -r sys -m <managed-system> -s h –startyear 1970 –filter event_types = config_change“, or alternatively again with the LPAR tool:

linux $ ms history -c ms02
                                PROCUNIS              MEMORY
TIME                  CONFIGURABLE  AVAILABLE  CONFIGURABLE  AVAILABLE  FIRMWARE
04/16/2019 12:15:51      20.0          5.05       1048576       249344     25856
04/11/2019 11:17:39      20.0          5.25       1048576       253696     25600
04/02/2019 13:24:35      20.0          4.85       1048576       249344     25856
03/29/2019 14:29:14      20.0          5.25       1048576       253696     25600
03/15/2019 15:37:08      20.0          4.85       1048576       249344     25856
03/15/2019 11:36:57      20.0          4.95       1048576       249344     25856
...
linux $

The same information can also be displayed for LPARs.

The last status changes of an LPAR can be listed with “lpar history <lpar>“:

linux $ lpar history lpar02
TIME                  PRIMARY_STATE         DETAILED_STATE
04/17/2019 05:42:43   Started               None
04/17/2019 05:41:24   Waiting For Input     Open Firmware
04/16/2019 12:01:54   Started               None
04/16/2019 12:01:29   Stopped               None
02/15/2019 11:30:48   Stopped               None
02/01/2019 12:23:34   Not Available         Unknown
02/01/2019 12:22:50   Relocating            None
...

This corresponds to the command “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = state_change, lpar_names = lpar02” on the HMC command line.

From the output it can be seen that the LPAR has been relocated using LPM, was stopped and restartet and has been in Open Firmware mode.

And finally you can look at the last configuration changes of an LPAR using the command on the HMC CLI “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = config_change, lpar_names = lpar02“. The output of the LPAR tool is a bit clearer:

linux $ lpar history -c lpar02
TIME                  PROC_MODE  PROCS  PROCUNITS  SHARING  UNCAP_WEIGHT  PROCPOOL         MEM_MODE  MEM
04/23/2019 18:49:43   shared    1      0.7        uncap    10          DefaultPool      ded       4096
04/23/2019 18:49:17   shared    1      0.7        uncap    5           DefaultPool      ded       4096
04/23/2019 18:48:44   shared    1      0.3        uncap    5           DefaultPool      ded       4096
04/09/2019 08:04:25   shared    1      0.3        uncap    5           DefaultPool      ded       3072
03/14/2019 12:37:32   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/26/2019 09:34:28   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/20/2019 06:51:57   shared    1      0.3        uncap    5           DefaultPool      ded       3072
01/31/2019 08:12:58   shared    1      0.3        uncap    5           DefaultPool      ded       3072
..

From the output you can see that the number of processing units were changed several time, the uncapped weight was changed and the memory has been expanded.

Changes of the last two months are available at any time!

PowerVM: Do you know the Profile “last*valid*configuration”?

Maybe one or the other has ever wondered how and where the current configuration of an LPAR is stored. If the current configuration and profile are not synchronized with each other, differences will quickly arise. When an LPAR is shut down and deactivated, the last current configuration is retained. When activating the LPAR, this configuration is available in addition to the profiles of the LPAR as the “current configuration” in the GUI. If one selects the current configuration, then the LPAR has the same configuration after activation as before deactivation. For a newly created LPAR, however, this selection is not available on activation. The difference also manifests itself on the HMC command line: the already activated LPAR can be activated without specifying a profile, the newly created LPAR can only be activated by specifying a profile. Let’s take a closer look.

(Short note: The commands on the HMC command line were executed directly on the HMC hmc01. In the example outputs with the LPAR tool, the commands were started from a Linux jump server. All commands are always shown with both variants!)

We have activated and booted the LPAR aix01 with the profile “standard“. We have not made any dynamic changes yet. We briefly look at the status of the LPAR and check if there is an RMC connection to the HMC:

hscroot@hmc01:~> lssyscfg -m p710 -r lpar --filter lpar_names=aix01 --header -F name lpar_env state curr_profile rmc_state os_version
name lpar_env state curr_profile rmc_state os_version
aix01 aixlinux Running standard active "AIX 7.1 7100-04-00-0000"
hscroot@hmc01:~>
linux $ lpar status aix01
NAME  ID      TYPE   STATUS  PROFILE    RMC   PROCS  PROCUNITS MEMORY  OS
aix01  5  aixlinux  Running  standard  active   1       -      3072    AIX 7.1 7100-04-00-0000
linux $

To see the effect of a dynamic change, let’s take a look at the actual state and the profile “standard“:

hscroot@hmc01:~> lshwres -m p710 -r mem --level lpar --filter lpar_names=aix01 -F curr_mem
3072
hscroot@hmc01:~> lssyscfg -m p710 -r prof --filter profile_names=standard,lpar_names=aix01 -F desired_mem
3072
hscroot@hmc01:~>
linux $ lpar mem aix01
      MEMORY            MEMORY           HUGEPAGES
NAME   MODE  AME   MIN   CURR   MAX   MIN  CURR  MAX
aix01  ded    -   2048   3072  8192    0     0    0
linux $ lpar -p standard mem aix01
      MEMORY            MEMORY           HUGEPAGES
NAME   MODE  AME   MIN   CURR   MAX   MIN  CURR  MAX
aix01  ded    -   2048   3072  8192    0     0    0
linux $

The LPAR has currently 3072 MB of main memory, which are also stored in the “standard” profile.

Now we add 1024 MB of main memory dynamically (DLPAR):

hscroot@hmc01:~> chhwres -m p710 -r mem -o a -p aix01 -q 1024
hscroot@hmc01:~>
linux $ lpar -d addmem aix01 1024
linux $

Now let’s look at the resulting memory resources of the LPAR:

hscroot@hmc01:~> lshwres -m p710 -r mem --level lpar --filter lpar_names=aix01 -F curr_mem
4096
hscroot@hmc01:~>
linux $ lpar mem aix01
     MEMORY            MEMORY          HUGEPAGES
NAME  MODE  AME   MIN   CURR   MAX   MIN  CURR  MAX
aix01  ded   -   2048   4096  8192    0     0    0
linux $

As expected, the LPAR now has 4096 MB of RAM. But what does the profile “standard” looks like?

hscroot@hmc01:~> lssyscfg -m p710 -r prof --filter profile_names=standard,lpar_names=aix01 -F desired_mem
3072
hscroot@hmc01:~>
linux $ lpar -p standard mem aix01
     MEMORY            MEMORY          HUGEPAGES
NAME  MODE  AME   MIN   CURR   MAX   MIN  CURR  MAX
aix01  ded   -   2048   3072  8192    0     0    0
linux $

The profile has not changed, activating the LPAR with this profile would result in 3072 MB of main memory.

The current configuration is always saved in the special profile “last*valid*configuration“:

hscroot@hmc01:~> lssyscfg -m p710 -r prof --filter profile_names=last*valid*configuration,lpar_names=aix01 -F desired_mem
4096
hscroot@hmc01:~>
linux $ lpar -p last*valid*configuration mem aix01
     MEMORY            MEMORY          HUGEPAGES
NAME  MODE  AME   MIN   CURR   MAX   MIN  CURR  MAX
aix01  ded   -   2048   4096  8192    0     0    0
linux $

Here the value of 4096 MB is consistent with the currently available memory in the LPAR.

Every dynamic change to an LPAR is performed on the LPAR via a DLPAR operation and by updating the special profile! If a profile is synchronized manually or automatically, then this special profile is ultimately synchronized with the desired profile.

The existence and handling of the special profile “last*valid*configuration” also makes some LPM possibilities easier to understand. We will deal with this in a later blog post.

We want your feedback!

The new PowerCampus “LPAR tool” is available for download! Much revised and written in C ++. It supports output in various formats: JSON + YAML!

The first 100 feedbacks get two licenses (for 2 LPARS) for free! Forever!

So, download and give feedback, just send an e-mail to info@powercampus.de!

The integrated test license supports without further registration one HMC and two complete managed systems! For an extended trial version for 4 HMC’s and unlimited MS just send an email to info@powercampus.de.

Download “LPAR tool”: https://powercampus.de/en/download-2/