oslevel -s does not show latest AIX version

The command “oslevel –s” shows the highest fully installed service pack on an AIX system, e.g .:

$ oslevel -s
7100-05-05-1939
$

However, a higher service pack can easily be installed (but not completely). The easiest way to check this, is with “oslevel –qs“:

$ oslevel -qs
Known Service Packs
-------------------
7100-05-06-2028
7100-05-06-2016
7100-05-06-2015
7100-05-05-1939
7100-05-05-1938
7100-05-05-1937
7100-05-04-1914
7100-05-04-1913
7100-05-03-1846
7100-05-03-1838
…
$

The output shows that 7100-05-06-2028 is installed. However, there are filesets that have not been updated to 7100-05-06-2028, which is why a lower version is displayed under “oslevel –s“.

If you want to know which filesets are lower than 7100-05-06-2028, “oslevel –s –l” can be used:

$ oslevel -s -l 7100-05-06-2028
Fileset                                 Actual Level       Service Pack Level
-----------------------------------------------------------------------------
openssl.base                            1.0.2.1801         1.0.2.2002    
openssl.license                         1.0.2.1801         1.0.2.2002    
openssl.man.en_US                       1.0.2.1801         1.0.2.2002    
$

Here, the OpenSSL filesets are still installed in an older version (1.0.2.1801) and would have to be updated to 1.0.2.2002 in order that the service pack 7100-05-06-2028 is completely installed.

0516-404 allocp: This system cannot fulfill the allocation request

When increasing logical volumes or file systems, the error “0516-404 allocp: This system cannot fulfill the allocation request” often occurs. Many AIX administrators are at first a little at a loss as to what the cause of the problem is: is it due to max LPs or upper bound, the chosen strictness or does the problem have completely different causes. Unfortunately, the error message is only generic and does not say what the real problem is. A good way to get to the bottom of the problem is the alog lvmt, in which all LVM actions, especially the low-level commands, including error messages are logged.

Our article Troubleshooting problems with extendlv and file system extension uses a series of examples to show how the cause of the problem can be found in many cases.

0516-787 extendlv: Maximum allocation for logical volume lv01 is 512

If the following message occurs while expanding a logical volume or file system:

# chfs -a size=+5G /fs01
0516-787 extendlv: Maximum allocation for logical volume fslv01
        is 512.
#

Then the cause is a limitation of the logical volume. The size of a logical volume is limited by the maximum number of LPs (logical partitions) that can be allocated for the logical volume. The error message says exactly that and even gives the current value for the maximum number of LPs. This is a changeable attribute of the logical volume and can be displayed with the lslv command:

$ lslv fslv01
...MAX LPs:            512                    PP SIZE:        8 megabyte(s)
...$

The attribute can be changed with the command chlv:

# chlv -x 768 fslv01
#

With a PP (Physical Partition) size of 8 MB, 768 PPs are sufficient for exactly 6 GB. Of course, you could consider the next expansion and set the value accordingly higher.

If there are enough 0516-787 extendlv: Maximum allocation for logical volumes free in the underlying volume group and no other limits are exceeded, the file system or logical volume should now be able to be expanded:

# chfs -a size=+5G /fs01
Filesystem size changed to 12582912
Inlinelog size changed to 24 MB.
#

Monitoring virtual FC Client Traffic

With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).

Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:

$ vios lsnports ms15-vio1
NAME  PHYSLOC                     FABRIC  TPORTS  APORTS  SWWPNS  AWWPNS
fcs0  U78CB.001.XXXXXXX-P1-C5-T1  1       64      62      2032    2012
fcs1  U78CB.001.XXXXXXX-P1-C5-T2  1       64      62      2032    2012
fcs2  U78CB.001.XXXXXXX-P1-C5-T3  1       64      61      2032    1979
fcs3  U78CB.001.XXXXXXX-P1-C5-T4  1       64      61      2032    1979
fcs4  U78CB.001.XXXXXXX-P1-C3-T1  1       64      50      3088    3000
fcs5  U78CB.001.XXXXXXX-P1-C3-T2  1       64      63      3088    3077
$

We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:

$ vios fcstat ms15-vio1
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x210000XXXXX56EC5  fcs1   774.75/s  129.51 MB/s  1332.71/s   92.96 MB/s  20
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   318.10/s   83.39 MB/s  481.34/s   126.18 MB/s  0
ms15-vio1  fcs2     0x210000XXXXX56EC6  fcs2   318.10/s   83.39 MB/s  480.78/s   126.03 MB/s  0
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   583.98/s   60.35 MB/s  1835.17/s  124.86 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   583.70/s   60.27 MB/s  1836.21/s  124.92 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.19/s  165.08 MB/s  1032.81/s   17.25 MB/s  46
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   775.12/s  129.48 MB/s  1047.32/s   17.15 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   775.78/s  128.99 MB/s  1037.99/s   17.39 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s    0.00 B/s   290.39/s    76.12 MB/s  0
aixtsmp1   fcs0     0xC050760XXXXX0052  fcs4   142.89/s   36.12 MB/s    0.00/s     0.00 B/s   26
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   234.97/s    4.58 MB/s  621.78/s    11.12 MB/s  40
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   243.55/s    5.05 MB/s  432.33/s     9.95 MB/s  0
cus1dbi01  fcs4     0xC050760XXXXX0044  fcs1     0.94/s   10.42 KB/s   87.28/s   459.26 KB/s  0
...
HOSTNAME   PHYSDEV  WWPN                DEV    INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0   1772.84/s  162.24 MB/s  1309.30/s   70.60 MB/s  68
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5   1769.13/s  161.95 MB/s  1305.60/s   70.54 MB/s  68
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1   883.55/s   118.97 MB/s  1551.97/s  108.78 MB/s  43
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2   201.09/s    52.72 MB/s  497.26/s   130.35 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6   201.09/s    52.72 MB/s  495.40/s   129.87 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0   923.54/s   128.89 MB/s  1234.98/s   23.31 MB/s  65
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1   876.93/s   118.93 MB/s  1234.98/s   23.32 MB/s  44
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0   884.17/s   119.07 MB/s  1223.50/s   23.00 MB/s  43
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5     0.00/s     0.00 B/s   325.83/s    85.41 MB/s  0
...
^C
$

The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:

$ vios fcstat -t 10 ms15-vio1 2
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  1034.58/s   86.56 MB/s  2052.23/s  160.11 MB/s  20
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  1532.63/s  115.60 MB/s  1235.72/s  118.32 MB/s  40
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  1510.33/s  114.88 MB/s  1236.49/s  118.27 MB/s  40
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  1036.11/s   86.67 MB/s  1612.25/s   44.86 MB/s  20
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  1031.50/s   86.29 MB/s  1588.02/s   44.27 MB/s  20
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  1029.58/s   86.31 MB/s  1567.63/s   43.65 MB/s  20
aixtsmp1   fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   436.52/s   114.43 MB/s  0
ms15-vio1  fcs2     0x21000024XXXXXEC6  fcs2    0.00/s     0.00 B/s   435.75/s   114.23 MB/s  0
aixtsmp1   fcs2     0xC050760XXXXX0058  fcs6    0.00/s     0.00 B/s   432.68/s   113.42 MB/s  0
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4  144.99/s     0.78 MB/s  478.83/s     2.22 MB/s  46
HOSTNAME   PHYSDEV  WWPN                DEV   INREQS    INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1   fcs5     0xC050760XXXXX003E  fcs0  758.14/s   35.55 MB/s  1822.99/s  112.60 MB/s  0
ms15-vio1  fcs5     0x10000090XXXXX12D  fcs5  757.38/s   35.52 MB/s  1821.46/s  112.59 MB/s  0
ms15-vio1  fcs0     0x21000024XXXXXEC4  fcs0  944.23/s   85.09 MB/s  1657.58/s   41.40 MB/s  2
aixtsmp3   fcs0     0xC050760XXXXX00DE  fcs1  943.47/s   85.15 MB/s  1636.90/s   40.68 MB/s  2
ms15-vio1  fcs1     0x21000024XXXXXEC5  fcs1  949.21/s   84.88 MB/s  1586.74/s   39.41 MB/s  2
aixtsmp3   fcs1     0xC050760XXXXX00E4  fcs0  946.53/s   84.64 MB/s  1584.83/s   39.40 MB/s  2
ms15-vio1  fcs4     0x10000090XXXXX12C  fcs4   39.44/s  449.92 KB/s  676.97/s     3.63 MB/s  10
cus1dbp01  fcs4     0xC050760XXXXX0047  fcs0   29.10/s  471.69 KB/s  310.92/s     1.28 MB/s  4
cus1mqp01  fcs4     0xC050760XXXXX002C  fcs0    1.91/s    4.71 KB/s  230.12/s     1.66 MB/s  0
cus2orap01 fcs4     0xC050760XXXXX000F  fcs0    0.77/s    4.31 KB/s   48.25/s   263.49 KB/s  0
^C
$

The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):

$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1858.72/s   51.14 MB/s  1231.82/s  104.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    6.94/s     1.82 MB/s    6.94/s     1.82 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.39/s     1.19 KB/s    0.39/s   395.05 B/s   0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.39/s     7.72 B/s     0.00/s     0.00 B/s   1
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
HOSTNAME  PHYSDEV  WWPN                DEV   INREQS     INBYTES      OUTREQS    OUTBYTES     CTRLREQS
aixtsmp1  fcs5     0xC050760XXXXX003E  fcs0  1760.48/s  111.48 MB/s  1125.70/s   95.20 MB/s  0
aixtsmp1  fcs2     0xC050760XXXXX0058  fcs6    8.53/s     2.24 MB/s  484.61/s   127.04 MB/s  0
aixtsmp1  fcs1     0xC050760XXXXX0056  fcs5    0.00/s     0.00 B/s   469.04/s   122.96 MB/s  0
aixtsmp1  fcs4     0xC050760XXXXX0042  fcs2    0.37/s     1.14 KB/s    0.00/s     0.00 B/s   0
aixtsmp1  fcs0     0xC050760XXXXX0052  fcs4    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
aixtsmp1  fcs3     0xC050760XXXXX005A  fcs7    0.00/s     0.00 B/s     0.00/s     0.00 B/s   0
^C
$

With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.

If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.

secldapclntd: LDAP failed to bind to server

We recently had a minor outage on one of our systems. Users could no longer log in and users could no longer use a web GUI. The problem occurred sporadically and then disappeared again. During these times there were a lot of messages of the following types in the syslog:

Mar  4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11.
Mar  4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12.
Mar  4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11.
Mar  4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12.
...

These messages indicated that both LDAP servers were temporarily unavailable. Investigations of the logs on the two LDAP servers did not reveal any connection attempts that were rejected. An examination of the firewall logs also showed that the firewall did not block any packets to the LDAP servers.

We then opened a case at IBM and promptly received instructions back for setting up special LDAP tracing. However, the problem then no longer occurred and therefore the tracing could not determine the cause.

The system in question can be reached via the Internet and therefore the number of ephemeral ports has been restricted for security reasons. Only about 1500 ephemeral ports are allowed through the firewall and configured on AIX (tcp_ephemeral_high and tcp_ephemeral_low). We then simulated on a test system what happens when the available ephemeral ports are exhausted. The above messages came immediately as soon as the LDAP client tried to open a new connection.

One reason for the LDAP client error message “LDAP failed to bind to server” can be the unavailability of ephemeral ports!

The “label” Attribute for FC Adapters

As of AIX 7.2 TL4 and VIOS 3.1.1.10 there is a new attribute “label” for physical FC adapters. The administrator can set this attribute to any character string (maximum 255 characters). Even if the attribute is only informative, it can be extremely useful in PowerVM virtualization environments. If you have a large number of managed systems, it is not always clear to which FC fabric a certain FC port is connected. This can of course be looked up in the documentation of your systems, but it does involve a certain amount of effort. It is easier if you link this information directly with the FC adapters, which is exactly what the new “label” attribute allows in a simple way. On AIX:

# chdev -l fcs0 -U -a label="Fabric_1"
fcs0 changed
# lsattr -El fcs0 -a label -F value
Fabric_1
#

On virtual I/O servers, the attribute can also be set using the padmin account:

/home/padmin> chdev -dev fcs1 -attr label="Fabric_2" -perm
fcs1 changed
/home/padmin> lsdev -dev fcs1 -attr label                
value

Fabric_2
/home/padmin>

The attribute is also defined for older FC adapters.

If the “label” attribute is consistently used, it is always possible to determine online for each FC adapter to which fabric the adapter is connected to. This information only needs to be stored once for each FC adapter.

(Note: The “label” attribute is not implemented for AIX 7.1, at least not until 7.1 TL5 SP6.)

Correcting “wrong” LVM mirroring

Mirrored logical volume from practice

In Part III of our article series “AIX LVM: Mechanics of migratepv” we show how incorrect mirroring of logical volumes can be corrected online. All that is required is the migratelp and migratepv commands and a good understanding of how these commands work.

Here are the links to the articles in the series:

AIX LVM: Mechanics of migratepv (Part I)

AIX LVM: Mechanics of migratepv (Part II)

AIX LVM: Mechanics of migratepv (Part III)

HSCF0180E Operation failed for

When trying to update the system firmware of a managed system via the HMC Command Line, we encountered the following error message:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027
HSCF0180E Operation failed for ms26 (9009-22A*XXXXXXX).
Could not unpack the firmware update package.
Check the health and available disk space of the file system.
hmc01:~>

The error message that was displayed suggested checking the available space in the HMC file systems:

hmc01:~> lshmcfs
filesystem=/var,filesystem_size=7935,filesystem_avail=4955,temp_files_start_time=11/22/2018 12:59:00,temp_files_size=2011
filesystem=/dump,filesystem_size=60347,filesystem_avail=55935,temp_files_start_time=02/15/2021 10:21:00,temp_files_size=0
filesystem=/extra,filesystem_size=20030,filesystem_avail=15939,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15615,filesystem_avail=4369,temp_files_start_time=02/15/2021 06:05:00,temp_files_size=4
hmc01:~>

Actually, the available space should be sufficient, but to be on the safe side, we’ve cleaned up a bit with the temporary files:

hmc01:~> chhmcfs -o f -d 5
hmc01:~>

The updlic command was unimpressed and returned the same error message.

The removal of some old firmware versions from the HMC‘s local disk repository was also unsuccessful:

hmc01:~> updlic -o p --ecnumber 01AL740
hmc01:~> updlic -o p --ecnumber 01AL770
hmc01:~> updlic -o p --ecnumber 01AM740
hmc01:~>

The error message was still the same. Obviously, contrary to the information in the error message, the problem had nothing to do with the available space on the HMC!

We then took a closer look at the downloaded firmware. We downloaded the firmware as an ISO file H75557812_01VL940_071_027.iso from the IBM website and then mounted it on our NIM server using the loopmount command:

aixnim:/root> loopmount -i /tmp/H75557812_01VL940_071_027.iso -o "-o ro -V cdrfs" -m /mnt
aixnim:/root> ls -l /mnt
total 528296
-rw-r--r--    1 102010979 213            1860 Feb 04 09:08 01VL940071_special_instructs.xml.special.note.xml
-rw-r-----    1 102010979 210            7290 Feb 04 09:08 01VL940_071_027.dd.xml
-rw-r--r--    1 102010979 213           95687 Feb 04 09:07 01VL940_071_027.html
-rw-r-----    1 102010979 210            2971 Feb 04 09:08 01VL940_071_027.pd.sdd
-rw-r-----    1 102010979 210           67338 Feb 04 09:08 01VL940_071_027.readme.txt
-rw-r-----    1 102010979 210       134969022 Feb 04 09:08 01VL940_071_027.rpm
-rw-r-----    1 102010979 210       135328848 Feb 04 09:08 01VL940_071_027.tar.gz
-rw-r-----    1 102010979 210            9442 Feb 04 09:08 01VL940_071_027.xml
aixnim:/root>

What we didn’t notice when copying the files was the lack of read permissions on other for most files.

After we had assigned read permissions for all files, the next update attempt was successful:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027 
HSCF0179W Operation was partially successful for ms26 (9009-22A*XXXXXXX).
The following deferred fixes are present in the fix pack.  Deferred fixes will be activated after the next IPL of the system.
An immediate IPL is not required, unless you want to activate one of the fixes below now.
..
hmc01:~>

LPAR-Tool 1.6.0.0 is available now

Version 1.6.0.0 of our LPAR tool is now available in our download area!

New features are:

  • Online monitoring of SEA client statistics (vios help seastat)
  • Online monitoring of virtual FC client adapters (vios help fcstat)
  • Display of historical processor and memory data (lpar help lsmem, lpar help lsproc)

In the article Monitoring SEA Traffic the possibilities of calling up SEA client statistics are shown.

Error: Mirror pools must be defined

The following error message occurred while creating a mirrored logical volume:

# mklv -c 2 -t jfs2 datavg01 10
0516-1814 lcreatelv: Mirror pools must be defined for each copy when strict mirror
        pools are enabled.
0516-822 mklv: Unable to create logical volume.
#

The reason for this is that the mirror pool strictness is set for the volume group. The article Mirror Pools: Understanding Mirror Pool Strictness examines and explains this in more detail.