oslevel -s does not show latest AIX version

The command “oslevel –s” shows the highest fully installed service pack on an AIX system, e.g .:

$ oslevel -s
7100-05-05-1939
$

However, a higher service pack can easily be installed (but not completely). The easiest way to check this, is with “oslevel –qs“:

$ oslevel -qs
Known Service Packs
-------------------
7100-05-06-2028
7100-05-06-2016
7100-05-06-2015
7100-05-05-1939
7100-05-05-1938
7100-05-05-1937
7100-05-04-1914
7100-05-04-1913
7100-05-03-1846
7100-05-03-1838
…
$

The output shows that 7100-05-06-2028 is installed. However, there are filesets that have not been updated to 7100-05-06-2028, which is why a lower version is displayed under “oslevel –s“.

If you want to know which filesets are lower than 7100-05-06-2028, “oslevel –s –l” can be used:

$ oslevel -s -l 7100-05-06-2028
Fileset                                 Actual Level       Service Pack Level
-----------------------------------------------------------------------------
openssl.base                            1.0.2.1801         1.0.2.2002    
openssl.license                         1.0.2.1801         1.0.2.2002    
openssl.man.en_US                       1.0.2.1801         1.0.2.2002    
$

Here, the OpenSSL filesets are still installed in an older version (1.0.2.1801) and would have to be updated to 1.0.2.2002 in order that the service pack 7100-05-06-2028 is completely installed.

0516-404 allocp: This system cannot fulfill the allocation request

When increasing logical volumes or file systems, the error “0516-404 allocp: This system cannot fulfill the allocation request” often occurs. Many AIX administrators are at first a little at a loss as to what the cause of the problem is: is it due to max LPs or upper bound, the chosen strictness or does the problem have completely different causes. Unfortunately, the error message is only generic and does not say what the real problem is. A good way to get to the bottom of the problem is the alog lvmt, in which all LVM actions, especially the low-level commands, including error messages are logged.

Our article Troubleshooting problems with extendlv and file system extension uses a series of examples to show how the cause of the problem can be found in many cases.

0516-787 extendlv: Maximum allocation for logical volume lv01 is 512

If the following message occurs while expanding a logical volume or file system:

# chfs -a size=+5G /fs01
0516-787 extendlv: Maximum allocation for logical volume fslv01
        is 512.
#

Then the cause is a limitation of the logical volume. The size of a logical volume is limited by the maximum number of LPs (logical partitions) that can be allocated for the logical volume. The error message says exactly that and even gives the current value for the maximum number of LPs. This is a changeable attribute of the logical volume and can be displayed with the lslv command:

$ lslv fslv01
...MAX LPs:            512                    PP SIZE:        8 megabyte(s)
...$

The attribute can be changed with the command chlv:

# chlv -x 768 fslv01
#

With a PP (Physical Partition) size of 8 MB, 768 PPs are sufficient for exactly 6 GB. Of course, you could consider the next expansion and set the value accordingly higher.

If there are enough 0516-787 extendlv: Maximum allocation for logical volumes free in the underlying volume group and no other limits are exceeded, the file system or logical volume should now be able to be expanded:

# chfs -a size=+5G /fs01
Filesystem size changed to 12582912
Inlinelog size changed to 24 MB.
#

secldapclntd: LDAP failed to bind to server

We recently had a minor outage on one of our systems. Users could no longer log in and users could no longer use a web GUI. The problem occurred sporadically and then disappeared again. During these times there were a lot of messages of the following types in the syslog:

Mar  4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11.
Mar  4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12.
Mar  4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11.
Mar  4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12.
...

These messages indicated that both LDAP servers were temporarily unavailable. Investigations of the logs on the two LDAP servers did not reveal any connection attempts that were rejected. An examination of the firewall logs also showed that the firewall did not block any packets to the LDAP servers.

We then opened a case at IBM and promptly received instructions back for setting up special LDAP tracing. However, the problem then no longer occurred and therefore the tracing could not determine the cause.

The system in question can be reached via the Internet and therefore the number of ephemeral ports has been restricted for security reasons. Only about 1500 ephemeral ports are allowed through the firewall and configured on AIX (tcp_ephemeral_high and tcp_ephemeral_low). We then simulated on a test system what happens when the available ephemeral ports are exhausted. The above messages came immediately as soon as the LDAP client tried to open a new connection.

One reason for the LDAP client error message “LDAP failed to bind to server” can be the unavailability of ephemeral ports!

The “label” Attribute for FC Adapters

As of AIX 7.2 TL4 and VIOS 3.1.1.10 there is a new attribute “label” for physical FC adapters. The administrator can set this attribute to any character string (maximum 255 characters). Even if the attribute is only informative, it can be extremely useful in PowerVM virtualization environments. If you have a large number of managed systems, it is not always clear to which FC fabric a certain FC port is connected. This can of course be looked up in the documentation of your systems, but it does involve a certain amount of effort. It is easier if you link this information directly with the FC adapters, which is exactly what the new “label” attribute allows in a simple way. On AIX:

# chdev -l fcs0 -U -a label="Fabric_1"
fcs0 changed
# lsattr -El fcs0 -a label -F value
Fabric_1
#

On virtual I/O servers, the attribute can also be set using the padmin account:

/home/padmin> chdev -dev fcs1 -attr label="Fabric_2" -perm
fcs1 changed
/home/padmin> lsdev -dev fcs1 -attr label                
value

Fabric_2
/home/padmin>

The attribute is also defined for older FC adapters.

If the “label” attribute is consistently used, it is always possible to determine online for each FC adapter to which fabric the adapter is connected to. This information only needs to be stored once for each FC adapter.

(Note: The “label” attribute is not implemented for AIX 7.1, at least not until 7.1 TL5 SP6.)

Correcting “wrong” LVM mirroring

Mirrored logical volume from practice

In Part III of our article series “AIX LVM: Mechanics of migratepv” we show how incorrect mirroring of logical volumes can be corrected online. All that is required is the migratelp and migratepv commands and a good understanding of how these commands work.

Here are the links to the articles in the series:

AIX LVM: Mechanics of migratepv (Part I)

AIX LVM: Mechanics of migratepv (Part II)

AIX LVM: Mechanics of migratepv (Part III)

Error: Mirror pools must be defined

The following error message occurred while creating a mirrored logical volume:

# mklv -c 2 -t jfs2 datavg01 10
0516-1814 lcreatelv: Mirror pools must be defined for each copy when strict mirror
        pools are enabled.
0516-822 mklv: Unable to create logical volume.
#

The reason for this is that the mirror pool strictness is set for the volume group. The article Mirror Pools: Understanding Mirror Pool Strictness examines and explains this in more detail.

Error: Every mirror pool must contain a copy of the logical volume

The attempt to create an unmirrored LV in a VG with mirror pools results in the following error:

# mklv -t jfs2 -p copy1=DC1 datavg01 10
0516-1829 mklv: Every mirror pool must contain a copy of
        the logical volume.
0516-822 mklv: Unable to create logical volume.
#

The cause is the VG‘s Mirror Pool Strictness attribute, which is set to ‘super-strict‘. The article Mirror Pools: Understanding Mirror Pool Strictness examines the importance of Mirror Pool Strictness using examples.

New Article Introduction to Mirror Pools

Many IT environments operate their systems in more than one data center. In order not to have any data loss in the event of a failure of an entire data center, the data is mirrored between 2 or more data centers. The mirroring can be realized by the storage (storage based mirroring) or by a volume manager (LVM in the case of AIX) on the server (host based mirroring). In the article Mirror Pools: An Introduction we look at mirroring using AIX Logical Volume Manager and Mirror Pools. The aim is to show how the correct mirroring of logical volumes can be implemented with the help of mirror pools. In larger environments with many physical volumes, maintaining correct mirroring without mirror pools is difficult and a challenge for the administrator.

Don’t Forget to Update AIX-rpm

Recently, when installing Python3 from the AIX toolbox using YUM, we encountered the following error situation:

# yum install python3
...
--> Finished Dependency Resolution
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libssl.a(libssl.so.1.0.2)
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libcrypto.a(libcrypto.so.1.0.2)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
#

Python3 requires version 1.0.2 of libssl.so and libcrypto.so, but the two libraries cannot be found. These two libraries can either be provided via the RPM package openssl, in the appropriate version, or via AIX. When the AIX toolbox is used, the openssl package is typically not installed, and the AIX OpenSSL libraries are used. Therefore, in our case, the OpenSSL RPM package is not installed. The libraries are then made available by BFF packages of the AIX operating system.

The information about which libraries are made available by AIX is provided via the virtual RPM package AIX-rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so.0.9.7
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

The output shows that versions 0.9.8 and 1.0.0 of the two required libraries are known to the AIX-rpm package, but the required version 1.0.2 is apparently not available. We take a quick look at the associated archive /usr/lib/libssl.a (or /usr/lib/libcrypto.a):

# ar -X any t /usr/lib/libssl.a
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl64.so
libssl64.so.0.9.8
libssl64.so.1.0.0
#

The archive obviously also provides version 1.0.2. This information is not provided by the AIX-rpm RPM package. This is because a newer version of openssl.base has obviously been installed, but the AIX-rpm package has not been updated. The RPM package currently has the following version:

# rpm -q AIX-rpm
AIX-rpm-7.1.5.15-6.ppc
#

We update the package now with the command updtvpkg (update virtual package):

# updtvpkg
Please wait...
# 
# rpm -q AIX-rpm
AIX-rpm-7.1.5.30-7.ppc
#

The version number has now changed from 7.1.5.15-6 to 7.1.5.30-7! We can again display which libraries are provided by AIX via AIX rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto.so.1.0.2)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so
libcrypto.so.0.9.7
libcrypto.so.1.0.0
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl.so.1.0.2)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl.so
libssl.so.1.0.0
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

After the update of the virtual package AIX-rpm, the RPM database now also contains the version 1.0.2 of libssl.so and libcrypto.so.

The installation of Python3 is now successful:

# yum install python3
...
Is this ok [y/N]: y
...
Installed:
  python3.ppc 0:3.7.1-1                                                                                       

Dependency Installed:
  bzip2.ppc 0:1.0.6-3  expat.ppc 0:2.2.4-1  info.ppc 0:6.4-1  libffi.ppc 0:3.2.1-3  libstdc++.ppc 0:6.3.0-2
  ncurses.ppc 0:6.1-2

Dependency Updated:
  readline.ppc 0:7.0-5                                                                                        

Complete!
#

Therefore, do not forget to update AIX-rpm when updating BFF packages!