A PDF on PowerVM is now available in our download area (approx. 260 pages, with approx. 75 figures):
oslevel -s does not show latest AIX version
The command “oslevel –s” shows the highest fully installed service pack on an AIX system, e.g .:
$ oslevel -s 7100-05-05-1939 $
However, a higher service pack can easily be installed (but not completely). The easiest way to check this, is with “oslevel –qs“:
$ oslevel -qs Known Service Packs ------------------- 7100-05-06-2028 7100-05-06-2016 7100-05-06-2015 7100-05-05-1939 7100-05-05-1938 7100-05-05-1937 7100-05-04-1914 7100-05-04-1913 7100-05-03-1846 7100-05-03-1838 … $
The output shows that 7100-05-06-2028 is installed. However, there are filesets that have not been updated to 7100-05-06-2028, which is why a lower version is displayed under “oslevel –s“.
If you want to know which filesets are lower than 7100-05-06-2028, “oslevel –s –l” can be used:
$ oslevel -s -l 7100-05-06-2028 Fileset Actual Level Service Pack Level ----------------------------------------------------------------------------- openssl.base 1.0.2.1801 1.0.2.2002 openssl.license 1.0.2.1801 1.0.2.2002 openssl.man.en_US 1.0.2.1801 1.0.2.2002 $
Here, the OpenSSL filesets are still installed in an older version (1.0.2.1801) and would have to be updated to 1.0.2.2002 in order that the service pack 7100-05-06-2028 is completely installed.
Monitoring virtual FC Client Traffic
With the LPAR tool, statistics for all virtual FC clients can be displayed at any time using the “vios fcstat” command. This allows you to determine at any time which client LPARs have which I/O throughput (when using NPIV).
Which NPIV-capable FC adapters are available on a virtual I/O server can easily be found out with “vios lsnports“:
$ vios lsnports ms15-vio1 NAME PHYSLOC FABRIC TPORTS APORTS SWWPNS AWWPNS fcs0 U78CB.001.XXXXXXX-P1-C5-T1 1 64 62 2032 2012 fcs1 U78CB.001.XXXXXXX-P1-C5-T2 1 64 62 2032 2012 fcs2 U78CB.001.XXXXXXX-P1-C5-T3 1 64 61 2032 1979 fcs3 U78CB.001.XXXXXXX-P1-C5-T4 1 64 61 2032 1979 fcs4 U78CB.001.XXXXXXX-P1-C3-T1 1 64 50 3088 3000 fcs5 U78CB.001.XXXXXXX-P1-C3-T2 1 64 63 3088 3077 $
We display the FC client statistics with the command “vios fcstat”. By default, the data for all virtual FC clients of the specified virtual I/O server are shown every 10 seconds:
$ vios fcstat ms15-vio1 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS ms15-vio1 fcs1 0x210000XXXXX56EC5 fcs1 774.75/s 129.51 MB/s 1332.71/s 92.96 MB/s 20 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 318.10/s 83.39 MB/s 481.34/s 126.18 MB/s 0 ms15-vio1 fcs2 0x210000XXXXX56EC6 fcs2 318.10/s 83.39 MB/s 480.78/s 126.03 MB/s 0 aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 583.98/s 60.35 MB/s 1835.17/s 124.86 MB/s 0 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 583.70/s 60.27 MB/s 1836.21/s 124.92 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 923.19/s 165.08 MB/s 1032.81/s 17.25 MB/s 46 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 775.12/s 129.48 MB/s 1047.32/s 17.15 MB/s 20 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 775.78/s 128.99 MB/s 1037.99/s 17.39 MB/s 20 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 290.39/s 76.12 MB/s 0 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 142.89/s 36.12 MB/s 0.00/s 0.00 B/s 26 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 234.97/s 4.58 MB/s 621.78/s 11.12 MB/s 40 cus1dbp01 fcs4 0xC050760XXXXX0047 fcs0 243.55/s 5.05 MB/s 432.33/s 9.95 MB/s 0 cus1dbi01 fcs4 0xC050760XXXXX0044 fcs1 0.94/s 10.42 KB/s 87.28/s 459.26 KB/s 0 ... HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1772.84/s 162.24 MB/s 1309.30/s 70.60 MB/s 68 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 1769.13/s 161.95 MB/s 1305.60/s 70.54 MB/s 68 ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 883.55/s 118.97 MB/s 1551.97/s 108.78 MB/s 43 ms15-vio1 fcs2 0x21000024XXXXXEC6 fcs2 201.09/s 52.72 MB/s 497.26/s 130.35 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 201.09/s 52.72 MB/s 495.40/s 129.87 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 923.54/s 128.89 MB/s 1234.98/s 23.31 MB/s 65 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 876.93/s 118.93 MB/s 1234.98/s 23.32 MB/s 44 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 884.17/s 119.07 MB/s 1223.50/s 23.00 MB/s 43 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 325.83/s 85.41 MB/s 0 ... ^C $
The LPAR name, the physical FC port (PHYSDEV) on the virtual I/O server, the WWPN of the client adapter, the virtual FC client port (DEV), as well as the number of requests (INREQS and OUTREQS) and thereby transferred bytes (INBYTES and OUTBYTES). The transfer rates are output in KB/s, MB/s or GB/s. The output can be very long on larger systems! The output is sorted according to throughput, i.e. the most active virtual client adapters are output first. With the option ‘-t‘ (top) the output can be restricted to a desired number of data records: e.g. with ‘-t 10‘ only the top ten adapters with the highest throughput are shown. In addition, the interval length (in seconds) can be specified via a further argument, here is a short example:
$ vios fcstat -t 10 ms15-vio1 2 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 1034.58/s 86.56 MB/s 2052.23/s 160.11 MB/s 20 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 1532.63/s 115.60 MB/s 1235.72/s 118.32 MB/s 40 aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1510.33/s 114.88 MB/s 1236.49/s 118.27 MB/s 40 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 1036.11/s 86.67 MB/s 1612.25/s 44.86 MB/s 20 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 1031.50/s 86.29 MB/s 1588.02/s 44.27 MB/s 20 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 1029.58/s 86.31 MB/s 1567.63/s 43.65 MB/s 20 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 436.52/s 114.43 MB/s 0 ms15-vio1 fcs2 0x21000024XXXXXEC6 fcs2 0.00/s 0.00 B/s 435.75/s 114.23 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 0.00/s 0.00 B/s 432.68/s 113.42 MB/s 0 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 144.99/s 0.78 MB/s 478.83/s 2.22 MB/s 46 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 758.14/s 35.55 MB/s 1822.99/s 112.60 MB/s 0 ms15-vio1 fcs5 0x10000090XXXXX12D fcs5 757.38/s 35.52 MB/s 1821.46/s 112.59 MB/s 0 ms15-vio1 fcs0 0x21000024XXXXXEC4 fcs0 944.23/s 85.09 MB/s 1657.58/s 41.40 MB/s 2 aixtsmp3 fcs0 0xC050760XXXXX00DE fcs1 943.47/s 85.15 MB/s 1636.90/s 40.68 MB/s 2 ms15-vio1 fcs1 0x21000024XXXXXEC5 fcs1 949.21/s 84.88 MB/s 1586.74/s 39.41 MB/s 2 aixtsmp3 fcs1 0xC050760XXXXX00E4 fcs0 946.53/s 84.64 MB/s 1584.83/s 39.40 MB/s 2 ms15-vio1 fcs4 0x10000090XXXXX12C fcs4 39.44/s 449.92 KB/s 676.97/s 3.63 MB/s 10 cus1dbp01 fcs4 0xC050760XXXXX0047 fcs0 29.10/s 471.69 KB/s 310.92/s 1.28 MB/s 4 cus1mqp01 fcs4 0xC050760XXXXX002C fcs0 1.91/s 4.71 KB/s 230.12/s 1.66 MB/s 0 cus2orap01 fcs4 0xC050760XXXXX000F fcs0 0.77/s 4.31 KB/s 48.25/s 263.49 KB/s 0 ^C $
The option ‘-s‘ (select) can be used to select and show only data records from a specific client (‘-s hostname = aixtsmp1‘) or only data records from a specific physical port (‘-s physdev = fcs1‘):
$ vios fcstat -s hostname=aixtsmp1 ms15-vio1 2 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1858.72/s 51.14 MB/s 1231.82/s 104.20 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 6.94/s 1.82 MB/s 6.94/s 1.82 MB/s 0 aixtsmp1 fcs4 0xC050760XXXXX0042 fcs2 0.39/s 1.19 KB/s 0.39/s 395.05 B/s 0 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.39/s 7.72 B/s 0.00/s 0.00 B/s 1 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs3 0xC050760XXXXX005A fcs7 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 HOSTNAME PHYSDEV WWPN DEV INREQS INBYTES OUTREQS OUTBYTES CTRLREQS aixtsmp1 fcs5 0xC050760XXXXX003E fcs0 1760.48/s 111.48 MB/s 1125.70/s 95.20 MB/s 0 aixtsmp1 fcs2 0xC050760XXXXX0058 fcs6 8.53/s 2.24 MB/s 484.61/s 127.04 MB/s 0 aixtsmp1 fcs1 0xC050760XXXXX0056 fcs5 0.00/s 0.00 B/s 469.04/s 122.96 MB/s 0 aixtsmp1 fcs4 0xC050760XXXXX0042 fcs2 0.37/s 1.14 KB/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs0 0xC050760XXXXX0052 fcs4 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 aixtsmp1 fcs3 0xC050760XXXXX005A fcs7 0.00/s 0.00 B/s 0.00/s 0.00 B/s 0 ^C $
With the “vios fcstat” command, FC throughput of any LPAR can be shown at any time in an extremely simple way, at the push of a button, so to speak.
If the intervals are smaller, the accuracy of the displayed values suffers. At 2 second intervals the inaccuracy is approx. 10%. However, the relationship between the displayed values is still correct.
secldapclntd: LDAP failed to bind to server
We recently had a minor outage on one of our systems. Users could no longer log in and users could no longer use a web GUI. The problem occurred sporadically and then disappeared again. During these times there were a lot of messages of the following types in the syslog:
Mar 4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11. Mar 4 07:56:05 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12. Mar 4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp11. Mar 4 07:56:10 aix01 daemon:err|error secldapclntd: LDAP failed to bind to server aixldapp12. ...
These messages indicated that both LDAP servers were temporarily unavailable. Investigations of the logs on the two LDAP servers did not reveal any connection attempts that were rejected. An examination of the firewall logs also showed that the firewall did not block any packets to the LDAP servers.
We then opened a case at IBM and promptly received instructions back for setting up special LDAP tracing. However, the problem then no longer occurred and therefore the tracing could not determine the cause.
The system in question can be reached via the Internet and therefore the number of ephemeral ports has been restricted for security reasons. Only about 1500 ephemeral ports are allowed through the firewall and configured on AIX (tcp_ephemeral_high and tcp_ephemeral_low). We then simulated on a test system what happens when the available ephemeral ports are exhausted. The above messages came immediately as soon as the LDAP client tried to open a new connection.
One reason for the LDAP client error message “LDAP failed to bind to server” can be the unavailability of ephemeral ports!
The “label” Attribute for FC Adapters
As of AIX 7.2 TL4 and VIOS 3.1.1.10 there is a new attribute “label” for physical FC adapters. The administrator can set this attribute to any character string (maximum 255 characters). Even if the attribute is only informative, it can be extremely useful in PowerVM virtualization environments. If you have a large number of managed systems, it is not always clear to which FC fabric a certain FC port is connected. This can of course be looked up in the documentation of your systems, but it does involve a certain amount of effort. It is easier if you link this information directly with the FC adapters, which is exactly what the new “label” attribute allows in a simple way. On AIX:
# chdev -l fcs0 -U -a label="Fabric_1" fcs0 changed # lsattr -El fcs0 -a label -F value Fabric_1 #
On virtual I/O servers, the attribute can also be set using the padmin account:
/home/padmin> chdev -dev fcs1 -attr label="Fabric_2" -perm fcs1 changed /home/padmin> lsdev -dev fcs1 -attr label value Fabric_2 /home/padmin>
The attribute is also defined for older FC adapters.
If the “label” attribute is consistently used, it is always possible to determine online for each FC adapter to which fabric the adapter is connected to. This information only needs to be stored once for each FC adapter.
(Note: The “label” attribute is not implemented for AIX 7.1, at least not until 7.1 TL5 SP6.)
Correcting “wrong” LVM mirroring
In Part III of our article series “AIX LVM: Mechanics of migratepv” we show how incorrect mirroring of logical volumes can be corrected online. All that is required is the migratelp and migratepv commands and a good understanding of how these commands work.
Here are the links to the articles in the series:
AIX LVM: Mechanics of migratepv (Part I)
LPAR-Tool 1.6.0.0 is available now
Version 1.6.0.0 of our LPAR tool is now available in our download area!
New features are:
- Online monitoring of SEA client statistics (vios help seastat)
- Online monitoring of virtual FC client adapters (vios help fcstat)
- Display of historical processor and memory data (lpar help lsmem, lpar help lsproc)
In the article Monitoring SEA Traffic the possibilities of calling up SEA client statistics are shown.