HMC: Which network interfaces have a link

The HMC allows the network interface configuration to be displayed using the “lshmc -n” command. It cannot be determined from the output whether a network interface currently has a link or not. However, this can be determined relatively easily via the sysfs file system, mounted under /sys. First, you can use the following command to determine which network interfaces the HMC has:

hscroot@hmc02:~> ls -1d /sys/class/net/eth*
/sys/class/net/eth0
/sys/class/net/eth1
/sys/class/net/eth2
/sys/class/net/eth3
/sys/class/net/eth4
/sys/class/net/eth5
hscroot@hmc02:~>

(Output is from a 7063-CR2.)

The link status is recorded in the operstate file and can be shown with a simple cat command:

hscroot@hmc02:~> cat /sys/class/net/eth*/operstate
up
up
down
down
down
down
hscroot@hmc02:~>

In the example case shown, the first two network interfaces (eth0 and eth1) have the link status up.

The current speed can be displayed by showing the content of the speed file:

hscroot@hmc02:~> cat /sys/class/net/eth*/speed
1000
1000
-1
-1
-1
-1
hscroot@hmc02:~>

There are a number of additional files for each network interface with additional information.

IBM PowerVM: Add a virtual Ethernet adapter to an LPAR

A virtual Ethernet adapter is to be added to the LPAR aix01 with IBM PowerVM. The data in detail:

    • HMC: hmc01
    • managed system: ms25
    • LPAR: aix01
    • profile: standard
    • virtual slot number: 4
    • Port-VLAN-ID: 900
    • virtual Ethernet switch: ETHERNET0(default)
    • additional VLANs: none

The command on the associated HMC hmc01 is:

hscroot@hmc01:~> chhwres -m ms25 -r virtualio --rsubtype eth -o a -p aix01 -s 2 -a 'ieee_virtual_eth=0,port_vlan_id=900'
hscroot@hmc01:~>

If the currently used profile of the LPAR is not automatically synchronized, then the additional virtual Ethernet adapter should also be added to the profile:

hscroot@hmc01:~> chsyscfg -r prof -m ms25 -i 'lpar_name=aix01,name=standard,"virtual_eth_adapters+=""4/0/900///0"""'
hscroot@hmc01:~>

With our LPAR tool, the command to use looks like this:

$ lpar addeth aix01 4 900
$

The current profile is automatically adjusted.

Detailed information on the LPAR tool and virtual Ethernet adapters can be found here: Virtual Ethernet

NTP Configuration on HMC with LPAR tool

NTP-Konfiguration HMC

The current status of NTP on an HMC can then be displayed using “hmc lsntp“:

$ hmc lsntp
NAME   XNTP     XNTPSTATUS    XNTPSERVER
hmc01  disable  -             -
hmc02  enable   synchronized  192.168.189.77,192.168.189.78
hmc03  enable   synchronized  192.168.189.77,192.168.189.78
$

Another NTP server can be added to the NTP configuration of an HMC using the “hmc addntpserver” command:

$ hmc addntpserver hmc01 192.168.189.77
$ hmc addntpserver hmc01 192.168.189.78
$

A check with “hmc lsntp” shows that two NTP servers are now configured, but NTP is still not activated:

$ hmc lsntp hmc01
NAME   XNTP     XNTPSTATUS  XNTPSERVER
hmc01  disable  -           192.168.189.77,192.168.189.78
$

NTP can now be activated with the command “hmc enablentp“:

$ hmc enablentp hmc01
$

The first sync may take a while:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS      XNTPSERVER
hmc01  enable  unsynchronized  192.168.189.77,192.168.189.78
$

The time on the HMC is not yet synchronized immediately after enabling NTP (XNTPSTATUS: unsynchronized).

A detailed status for each NTP server can be obtained by using the “-a” option (all NTP servers):

$ hmc lsntp –a hmc01
NAME    SERVER         STATE          POLL_FREQ_SECONDS  SECONDS_SINCE_LAST_POLL
hmc01  192.168.189.77  not connected  64                 0
hmc01  192.168.189.78  not connected  64                 0
$

As soon as synchronization with one of the NTP servers is achieved, the overall status is synchronized:

$ hmc lsntp hmc01
NAME   XNTP    XNTPSTATUS    XNTPSERVER
hmc01  enable  synchronized  192.168.189.77,192.168.189.78
$

A more detailed description can be found here: NTP Configuration on the HMC

The LPAR tool can be downloaded for testing here: Download

LPAR tool 1.7.0.1 is now available

Version 1.7.0.1 of the LPAR tool is now available in our download area.

The new version supports the following new features, among others:

    • Installation of IFixes and updates on the HMC (hmc help updhmc)
    • System firmware updates (and more) of managed systems (ms help updatelic)
    • Display FLRT data with online query at IBM (hmc help flrt, ms help flrt, lpar help flrt)
    • Configuration of NTP on HMCs (hmc help ntp)

Versions for Linux, AIX and Macos are available.

All versions include a test license valid until September 30th, 2022.

So download, install and then try it out!

HSCF0180E Operation failed for

When trying to update the system firmware of a managed system via the HMC Command Line, we encountered the following error message:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027
HSCF0180E Operation failed for ms26 (9009-22A*XXXXXXX).
Could not unpack the firmware update package.
Check the health and available disk space of the file system.
hmc01:~>

The error message that was displayed suggested checking the available space in the HMC file systems:

hmc01:~> lshmcfs
filesystem=/var,filesystem_size=7935,filesystem_avail=4955,temp_files_start_time=11/22/2018 12:59:00,temp_files_size=2011
filesystem=/dump,filesystem_size=60347,filesystem_avail=55935,temp_files_start_time=02/15/2021 10:21:00,temp_files_size=0
filesystem=/extra,filesystem_size=20030,filesystem_avail=15939,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15615,filesystem_avail=4369,temp_files_start_time=02/15/2021 06:05:00,temp_files_size=4
hmc01:~>

Actually, the available space should be sufficient, but to be on the safe side, we’ve cleaned up a bit with the temporary files:

hmc01:~> chhmcfs -o f -d 5
hmc01:~>

The updlic command was unimpressed and returned the same error message.

The removal of some old firmware versions from the HMC‘s local disk repository was also unsuccessful:

hmc01:~> updlic -o p --ecnumber 01AL740
hmc01:~> updlic -o p --ecnumber 01AL770
hmc01:~> updlic -o p --ecnumber 01AM740
hmc01:~>

The error message was still the same. Obviously, contrary to the information in the error message, the problem had nothing to do with the available space on the HMC!

We then took a closer look at the downloaded firmware. We downloaded the firmware as an ISO file H75557812_01VL940_071_027.iso from the IBM website and then mounted it on our NIM server using the loopmount command:

aixnim:/root> loopmount -i /tmp/H75557812_01VL940_071_027.iso -o "-o ro -V cdrfs" -m /mnt
aixnim:/root> ls -l /mnt
total 528296
-rw-r--r--    1 102010979 213            1860 Feb 04 09:08 01VL940071_special_instructs.xml.special.note.xml
-rw-r-----    1 102010979 210            7290 Feb 04 09:08 01VL940_071_027.dd.xml
-rw-r--r--    1 102010979 213           95687 Feb 04 09:07 01VL940_071_027.html
-rw-r-----    1 102010979 210            2971 Feb 04 09:08 01VL940_071_027.pd.sdd
-rw-r-----    1 102010979 210           67338 Feb 04 09:08 01VL940_071_027.readme.txt
-rw-r-----    1 102010979 210       134969022 Feb 04 09:08 01VL940_071_027.rpm
-rw-r-----    1 102010979 210       135328848 Feb 04 09:08 01VL940_071_027.tar.gz
-rw-r-----    1 102010979 210            9442 Feb 04 09:08 01VL940_071_027.xml
aixnim:/root>

What we didn’t notice when copying the files was the lack of read permissions on other for most files.

After we had assigned read permissions for all files, the next update attempt was successful:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027 
HSCF0179W Operation was partially successful for ms26 (9009-22A*XXXXXXX).
The following deferred fixes are present in the fix pack.  Deferred fixes will be activated after the next IPL of the system.
An immediate IPL is not required, unless you want to activate one of the fixes below now.
..
hmc01:~>

LPAR-Tool: Which LPARs have no RMC-Connection

Status and configuration of LPARs are regularly needed information in the administration of LPARs. With the LPAR tool, information such as status, RMC status, number of cores, size of RAM, OS version and other data can be easily and quickly determined, even with hundreds or thousands of LPARs. Which LPARs don’t have an RMC connection is shown as one of the examples.

All of the following examples were performed on an environment with 10 HMCs, 50 managed systems, and just over 500 LPARs. To determine how long the LPAR tool requires, the run times of the commands were measured and specified using time.

The names of the LPARs were manually changed in the outputs shown and replaced by generic names lparXX and aixYY.

First of all, the status of a single LPAR:

$ time lpar status aix01
NAME   LPAR_ID  LPAR_ENV   STATE     PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01  27       aixlinux   Running   standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614

real    0m0.210s
user    0m0.011s
sys     0m0.013s
$

Of course you can also specify multiple LPARs. If you want to know the status of all LPARs (in our case just over 500 LPARs), just leave out the argument:

$ time lpar status
NAME    LPAR_ID  LPAR_ENV   STATE     PROFILE      SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01   27       aixlinux   Running   standard     0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02   1        aixlinux   Running   standard     -     -         1      -           8320    Unknown
...
lpar01  6        aixlinux   Running   standard     0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m18.933s
user	0m3.819s
sys	0m3.789s
$

In the background, the LPAR tool executes more than 150 commands on the corresponding HMCs (lshwres and lssyscfg)!

The output should now be restricted to LPARs that are currently active (state=Running). There is the option “-s“, which can be used to specify criteria for attributes that must be met. Only LPARs meeting these criteria will be shown:

$ time lpar status -s state=Running
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix01    27       aixlinux   Running  standard   0     active    1      0.1         8192    AIX 7.1 7100-04-02-1614
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar01   6        aixlinux   Running  standard   0     active    1      0.4         20480   AIX 7.1 7100-04-05-1720

real	0m17.998s
user	0m3.692s
sys	0m3.647s
$

Now we want to know on which of these LPARs RMC is not working/not active. The option “-s” allows to combine any number of criteria. All specified criteria must then be met (logical AND). The RMC state can be found in the attribute rmc_state:

$ time lpar status -s state=Running,rmc_state!=active
NAME     LPAR_ID  LPAR_ENV   STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix02    1        aixlinux   Running  standard   -     -         1      -           8320    Unknown
aix03    2        aixlinux   Running  standard   -     -         1      -           8320    Unknown
...
lpar07   4        aixlinux   Running  standard   0     none      1      1.0         4352    Unknown

real	0m19.057s
user	0m3.550s
sys	0m3.512s
$

As another example we want to know on which LPARs AIX 7.1 TL5 is installed. The os_version attribute contains the OS version. The ‘~‘ operator can be used to compare against a regular expression (similar to the grep command). We use the regular expression 7100-05:

$ time lpar status -s os_version~7100-05
NAME     LPAR_ID  LPAR_ENV  STATE    PROFILE    SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
aix14    14       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-02-1810
aix16    24       aixlinux  Running  standard   0     active    2      0.2         16384   AIX 7.1 7100-05-03-1846
...
lpar10   10       aixlinux  Running  standard   0     active    3      0.3         32768   AIX 7.1 7100-05-02-1810

real	0m18.212s
user	0m3.726s
sys	0m3.676s
$

So far, we have always used the default output format. Now we’d like to list all systems that still run on AIX 6.1, but this time only the LPAR name and the OS version will be output. For this there is the option “-F“, with which the desired output fields can be specified:

$ time lpar status -s os_version~6100 -F name:os_version
aix39:AIX 6.1 6100-07-04-1216
aix46:AIX 6.1 6100-07-04-1216
...
lpar35:AIX 6.1 6100-09-05-1524

real	0m18.041s
user	0m3.619s
sys	0m3.699s
$

If you prefer JSON output, you can easily do that with the option “-j“, here’s the same example with JSON output:

$ time lpar status -s os_version~6100 -F name:os_version -j
{
	"name": "aix39",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
{
	"name": "aix46",
	"os_version": "AIX 6.1 6100-07-04-1216"
}
...
{
	"name": "lpar35",
	"os_version": "AIX 6.1 6100-09-05-1524"
}

real	0m21.247s
user	0m3.670s
sys	0m3.720s
$

Of course you can not know all attribute names by heart! But that’s not necessary, because you can easily see all attribute names. Use the option “-f” (stanza format) and specify any LPAR:

$ lpar status -f lpar19
lpar19:
	curr_lpar_proc_compat_mode = POWER7
	curr_mem = 8192
	curr_proc_mode = shared
	curr_proc_units = 0.3
	curr_procs = 2
	name = lpar19
	os_version = AIX 6.1 6100-09-05-1524
...
$

With the options “-h” and “-m” the LPARs can be selected depending on the associated HMC and/or managed system.

Status of all LPARs with associated HMC hmc01:

$ lpar -h hmc01 status

Status of all LPARs whose corresponding HMC has type 7042-CR6:

$ lpar -h 7042-CR6 status

Status of all LPARs whose associated HMC has type 7042-CR6 and whose name begins with lpar:

$ lpar -h 7042-CR6 status lpar*

Status of all LPARs on the managed system ms13:

$ lpar -m ms13 status

Status of all LPARs whose managed system is an S922:

$ lpar -m 9009-22A status

The presented selection and output options apply to all output commands of the LPAR tool (except the vios command).

The LPAR-Tool can be downloaded from our download area: https://powercampus.de/en/download-2

The LPAR-Tool contains a test license which is valid until the end of october.

Did you know that state and configuration change information is available on the HMC for about 2 months?

Status and configuration changes of LPARs and managed systems are stored on the HMCs for about 2 months. This can be used to find out, when a managed system was shut down, when a service processor failover took place, or when the memory of an LPAR was expanded, at least if the event is no more than 2 months ago.

The status changes of a managed system can be listed with the command “lslparutil -r sys -m <managed-system> -sh –startyear 1970 –filter event_types = state_change“, or alternatively with the LPAR-Tool command “ms history <managed -system> “.

linux $ ms history ms04
TIME                  PRIMARY_STATE         DETAILED_STATE
03/14/2019 08:45:13   Started               None
03/14/2019 08:36:52   Not Available         Unknown
02/17/2019 01:51:55   Started               None
02/17/2019 01:44:00   Not Available         Unknown
02/12/2019 09:32:57   Started               None
02/12/2019 09:28:02   Started               Service Processor Failover
02/12/2019 09:27:07   Started               None
02/12/2019 09:24:42   Standby               None
02/12/2019 09:21:25   Starting              None
02/12/2019 09:22:59   Stopped               None
02/12/2019 09:21:58   Not Available         Unknown
02/12/2019 09:09:45   Stopped               None
02/12/2019 09:07:53   Stopping              None
linux $

Configuration changes (processor, memory) of a managed system can be displayed with “lslparutil -r sys -m <managed-system> -s h –startyear 1970 –filter event_types = config_change“, or alternatively again with the LPAR tool:

linux $ ms history -c ms02
                                PROCUNIS              MEMORY
TIME                  CONFIGURABLE  AVAILABLE  CONFIGURABLE  AVAILABLE  FIRMWARE
04/16/2019 12:15:51      20.0          5.05       1048576       249344     25856
04/11/2019 11:17:39      20.0          5.25       1048576       253696     25600
04/02/2019 13:24:35      20.0          4.85       1048576       249344     25856
03/29/2019 14:29:14      20.0          5.25       1048576       253696     25600
03/15/2019 15:37:08      20.0          4.85       1048576       249344     25856
03/15/2019 11:36:57      20.0          4.95       1048576       249344     25856
...
linux $

The same information can also be displayed for LPARs.

The last status changes of an LPAR can be listed with “lpar history <lpar>“:

linux $ lpar history lpar02
TIME                  PRIMARY_STATE         DETAILED_STATE
04/17/2019 05:42:43   Started               None
04/17/2019 05:41:24   Waiting For Input     Open Firmware
04/16/2019 12:01:54   Started               None
04/16/2019 12:01:29   Stopped               None
02/15/2019 11:30:48   Stopped               None
02/01/2019 12:23:34   Not Available         Unknown
02/01/2019 12:22:50   Relocating            None
...

This corresponds to the command “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = state_change, lpar_names = lpar02” on the HMC command line.

From the output it can be seen that the LPAR has been relocated using LPM, was stopped and restartet and has been in Open Firmware mode.

And finally you can look at the last configuration changes of an LPAR using the command on the HMC CLI “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = config_change, lpar_names = lpar02“. The output of the LPAR tool is a bit clearer:

linux $ lpar history -c lpar02
TIME                  PROC_MODE  PROCS  PROCUNITS  SHARING  UNCAP_WEIGHT  PROCPOOL         MEM_MODE  MEM
04/23/2019 18:49:43   shared    1      0.7        uncap    10          DefaultPool      ded       4096
04/23/2019 18:49:17   shared    1      0.7        uncap    5           DefaultPool      ded       4096
04/23/2019 18:48:44   shared    1      0.3        uncap    5           DefaultPool      ded       4096
04/09/2019 08:04:25   shared    1      0.3        uncap    5           DefaultPool      ded       3072
03/14/2019 12:37:32   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/26/2019 09:34:28   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/20/2019 06:51:57   shared    1      0.3        uncap    5           DefaultPool      ded       3072
01/31/2019 08:12:58   shared    1      0.3        uncap    5           DefaultPool      ded       3072
..

From the output you can see that the number of processing units were changed several time, the uncapped weight was changed and the memory has been expanded.

Changes of the last two months are available at any time!