ProbeVue in Action: Identifying a crashing Process

Recently, our monitoring reported a full /var file system on one of our systems. We detected that core files in the directory /var/adm/core had filled the file system. It quickly turned out that all core files came from perl. However, based on the core files we could not determine which perl script had caused the crash of perl. A look at the timestamps of the core files unfortunately showed no pattern:

-bash-4.4$ ls -ltr /var/adm/core
total 2130240
drwxr-xr-x    2 root     system          256 Jan 29 10:20 lost+found/
-rw-------    1 root     system    100137039 Jun 26 04:51 core.22610328.26025105.Z
-rw-------    1 root     system     99054991 Jun 26 06:21 core.21102892.26042104.Z
-rw-------    1 root     system     99068916 Jun 26 08:06 core.18153840.26060607.Z
-rw-------    1 root     system    100132866 Jun 26 08:21 core.19005848.26062105.Z
-rw-------    1 root     system     97986020 Jun 26 16:36 core.15270246.26143608.Z
-rw-------    1 root     system     99208958 Jun 26 22:21 core.22675838.26202106.Z
-rw-------    1 root     system     97557063 Jun 27 01:06 core.5505292.26230604.Z
-rw-------    1 root     system     98962499 Jun 27 10:06 core.8257960.27080603.Z
-rw-------    1 root     system     99804173 Jun 27 14:51 core.18940202.27125107.Z
-rw-------    1 root     system     99633676 Jun 28 03:21 core.17563960.28012107.Z
-rw-------    1 root     system     99116032 Jun 28 19:06 core.8651210.28170608.Z
-bash-4.4$

Also, the entries in the error report provided no information about the crashing perl script and how it was started.

-bash-4.4$ sudo errpt -j A924A5FC –a
...
---------------------------------------------------------------------------
LABEL:          CORE_DUMP
IDENTIFIER:     A924A5FC

Date/Time:       Wed May 29 15:21:25 CEST 2019
Sequence Number: 17548
Machine Id:      XXXXXXXXXXXX
Node Id:         XXXXXXXX
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSPROC        

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

        Recommended Actions
        CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
         11
USER'S PROCESS ID:
              13369662
FILE SYSTEM SERIAL NUMBER
           1
INODE NUMBER
                 69639
CORE FILE NAME
/var/adm/core/core.13369662.29132106
PROGRAM NAME
perl
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER

PROCESSOR ID
  hw_fru_id: 1
  hw_cpu_id: 19

ADDITIONAL INFORMATION

Unable to generate symptom string.
Too many stack elements.
-bash-4.4$

The only information that could be found was that the processes were terminated with signal 11 (SIGSEGV), that is, due to an access to an invalid memory address.

The question arose: how to determine which perl script was crashing and how it is started.

This should be a task for ProbeVue.

The sysproc provider, which generates an event in case of an exit of a process, seemed to be the right probe to use. The special built-in variable __exitinfo provides more detailed information about the exit, such as exit status or the signal number that terminated the process. This can be used to write the following clause:

1: @@sysproc:exit:*
2: when ( __exitinfo->signo == 11 )
3: {
4:         printf( "%llu:  %s\n" , __pid , __pname );
5:         ptree(10);
6: }

The 6 lines are briefly explained here:

  1. The probe point: provider is sysproc, event is exit, * means any process.
  2. By using the above predicate, the subsequent action block is executed only when the process was terminated with signal 11 (SIGSEGV).
  3. Start of the action block.
  4. Output of the PID and the program name of the process.
  5. The function ptree outputs the father, grandfather, etc. (up to 10 levels) of the process.
  6. End of the action block.

Unfortunately, no program arguments can be listed here, which in our case would have given the name of the perl script. But at least we get informations from the function ptree about the way the program was called, which is sufficient in some cases to ultimately identify the program.

For identification we would like to have the information about the arguments with which perl was called. This information is provided by the syscall provider for the system call execve with which the program is started. The probe point is thus syscall: *: execve: entry, since the arguments are known when entering the function. The signature of execve for ProbeVue looks like this:

int execve( char* , struct arg_t* args , char* );

Here, the first argument (provided by ProbeVue as __arg1) is the program name. The second argument is a structure with the arguments in question (provided by __arg2). The third argument gives access to environment variables, which is not important in our case. The structure struct arg_t looks like this for 5 arguments:

struct arg_t
{
        union
        {
                char* arg[5];
                int num[5];
        } u;
};

This structure and the signature of execve must be declared in the ProbeVue script before they can be used.

When accessing the arguments, there is another small problem: if the action block for our sample is addressed, we are in kernel mode, but the arguments themselves are addresses in the user mode of the process. The data (in this case character strings) must be copied out of the user address space. This is done by the function get_userstring.

For every execve we get the PID, the program name, the command and up to 5 arguments. This is implemented in the following program:

#! /usr/bin/probevue

struct arg_t
{
        union
        {
                char* arg[5];
                int num[5];
        } u;
};

int execve( char* , struct arg_t* args , char* );

@@syscall:*:execve:entry
{
        __auto String command[128];
        __auto String argument[128];
        __auto struct arg_t argv;
        copy_userdata( __arg2 , argv );
        command = get_userstring( __arg1 , -1 );
        argument = get_userstring( argv.u.arg[0] , -1 );
        printf( "%llu: %s called execve(%s) with arguments: %s " , __pid , __pname , command , argument )
;
        if ( argv.u.num[1] != 0 )
        {
                argument = get_userstring( argv.u.arg[1] , -1 );
                printf( "%s " , argument );
                if ( argv.u.num[2] != 0 )
                {
                        argument = get_userstring( argv.u.arg[2] , -1 );
                        printf( "%s " , argument );
                        if ( argv.u.num[3] != 0 )
                        {
                                argument = get_userstring( argv.u.arg[3] , -1 );
                                printf( "%s " , argument );
                                if ( argv.u.num[4] != 0 )
                                {
                                        argument = get_userstring( argv.u.arg[4] , -1 );
                                        printf( "%s " , argument );
                                }
                        }
                }
        }
        printf( "\n" );
}

@@sysproc:exit:*
when ( __exitinfo->signo == 11 )
{
        printf( "%llu:  %s\n" , __pid , __pname );
        ptree(10);
}

We called the script capture_segv.e and made it executable.

In theory, after startup the program should output all starting programs with PID, name and up to 5 arguments. In addition, an output occurs when a process is terminated with signal 11 (SIGSEGV). The corresponding PID can then be matched further up in the output and thus the program with arguments can be identified.

Unfortunately, the following small problem arises in practice: if a program is terminated very quickly after the execve, before ProbeVue can copy the arguments with get_userstring, get_userstring accesses a no longer existing address and the ProbeVue script is aborted. We bypassed this by simply starting the ProbeVue script over and over in an infinite loop:

# while true; do ./capture_segv.e >>/tmp/wait_for_segv ; done

We then ran the ProbeVue script for a few hours, until the next crash of the perl script. The file /tmp/wait_for_segv contained about 23,000 lines! We have listed only the relevant lines here:

# cat /tmp/wait_for_segv
…
8651210: ksh called execve(xxxx_hacheck.pl) with arguments: xxxx_hacheck.pl -c
8651210: ksh called execve(/var/opt/OV/bin/instrumentation/xxxx_hacheck.pl) with arguments: xxxx_hacheck
.pl -c
20054518: ksh called execve(/bin/proc2mon.pl) with arguments: proc2mon.pl
…
8651210:  perl

     PID              CMD
       1              init
                        |
                        V
9634196              ovcd
                        |
                        V
9765232              opcacta
                        |
                        V
8651210              perl    <=======
…

One can see that perl was started via the program opcacta, which was started by ovcd. These processes belong to HP OpenView that is in use here. At the top of the output one can see that the perl script /var/opt/OV/bin/instrumentation/xxxx_hacheck.pl has been started. So we found the script that generates the many core files.

The script was written recently and obviously needs to be re-examined and reworked.

With the help of ProbeVue a short script and several hours of waiting was enough to find the cause of the problem! ProbeVue is not only useful for investigating problems; also in performance monitoring, ProbeVue proves to be extremely helpful.

Resources of not activated LPARs and Memory Affinity

When an LPAR is shut down, resources such as processors, memory, and I/O slots are not automatically released by the LPAR. The resources remain assigned to the LPAR and are then reused on the next activation (with the current configuration). In the first part of the article Resources of not activated LPARs we had already looked at this.

(Note: In the example output, we use version 1.4 of the LPAR tool, but in all cases we show the underlying commands on the HMC command line, so you can try everything without using the LPAR tool.)

The example LPAR lpar1 was shut down, but currently still occupies 100 GB of memory:

linux $ lpar status lpar1
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  39       aixlinux  Not Activated  standard  0     inactive  1      0.2         102400  Unknown
linux $

The following commands for the output above were executed on the corresponding HMC hmc01:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1
hmc01: lshwres -r mem -m ms09 --level lpar --filter lpar_names=lpar1
hmc01: lshwres -r proc -m ms09 --level lpar --filter lpar_names=lpar1

As the output shows, the LPAR lpar1 has still allocated its resources (processors, memory, I/O adapters).

In order to understand why deactivating an LPAR does not release the resources, you have to look at the “Memory Affinity Score”:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      100   0
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

The Memory Affinity Score describes how close processors and memory are, the closer the memory to the memory is, the better is the throughput to the memory. The command above indicates, with a value between 1 and 100, how big the affinity between processors and LPARs is. Our LPAR lpar1 currently has a value of 100, which means the best possible affinity of memory and processors. If the resources were freed when deactivating an LPAR, then the LPAR would lose this Memory Affinity Score. The next time you enable the LPAR, it then depends on the memory and processors available then how good the memory affinity will be then. We release the resources once:

linux $ lpar -d rmprocs lpar1 1
linux $

HMC command line:

hmc01: chhwres -m ms09 -r proc  -o r -p lpar1 --procs 1

No more score will be given, since the LPAR has no longer allocated any resources:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      none  none
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

Now we allocate resources again and look at the effect this has on memory affinity:

linux $ lpar applyprof lpar1 standard
linux $

HMC command line:

hmc01: chsyscfg -r lpar -m ms09 -o apply -p lpar1 -n standard

We again determine the Memory Affinity Score:

linux $ lpar lsmemopt lpar1
             LPAR_SCORE  
LPAR_NAME  CURR  PREDICTED
lpar1      53    0
linux $

HMC command line:

hmc01: lsmemopt -m ms09 -r lpar -o currscore –filter lpar_names=lpar1

The score is now only 53, the performance of the LPAR has become worse. Whether and how much this is noticeable, depends ultimately on the applications on the LPAR.

The fact that the resources are not released when deactivating an LPAR, thus guarantees the next time you activate (with the current configuration) the memory affinity remains the same and thus the performance should be the same.

If you release the resources of an LPAR (manually or automatically), then you have to realize that this has an effect on the LPAR if it is later activated again, because then the resources are reassigned and a worse (but possibly also a better) Memory Affinity Score can result.

Conversely, before activating a new LPAR you can also make sure that there is a good chance for a high memory affinity score for the new LPAR by releasing resources of inactive LPARs.

(Note: resource distribution can be changed and improved at runtime using the Dynamic Platform Optimizer DPO. DPO is supported as of POWER8.)

 

Resources of not activated LPARs

When an LPAR is shutdown, resources such as processors, memory, and I/O slots are not automatically released by the LPAR. The resources remain assigned to the LPAR and are reused on the next activation (with the current configuration).

The article will show how such resources are automatically released and, if desired, how to manually release resources of an inactive LPAR.

(Note: In the example output, we use version 1.4 of the LPAR tool, but in all cases we show the underlying commands on the HMC command line, so you can try everything without using the LPAR tool.)

The example LPAR lpar1 was shut down, but currently still occupies 100 GB of memory:

linux $ lpar status lpar1
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  39       aixlinux  Not Activated  standard  0     inactive  1      0.2         102400  Unknown
linux $

The following commands for the output above were executed on the corresponding HMC hmc01:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1
hmc01: lshwres -r mem -m ms09 --level lpar --filter lpar_names=lpar1
hmc01: lshwres -r proc -m ms09 --level lpar --filter lpar_names=lpar1

The resource_config attribute of an LPAR indicates whether the LPAR has currently allocated resources (resource_config=1) or not (resource_config=0):

linux $ lpar status -F resource_config lpar1
1
linux $

Or on the HMC command line:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1 –F resource_config

The resources allocated by an not activated LPAR can be released in 2 different ways:

  1. Automatic: The resources used are needed by another LPAR, e.g. because memory is expanded dynamically or an LPAR is activated that does not have sufficient resources. In this case, resources are automatically removed from a not activated LPAR. We will show this below with an example.
  2. Manual: The allocated resources are explicitly released by the administrator. This is also shown below in an example.

First we show an example in which resources are automatically taken away from an not activated LPAR.

The managed system ms09 currently has about 36 GB free memory:

linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     33792     786432        36352  256
linux $

HMC command line:

hmc01: lshwres -r mem -m ms09 --level sys

We start an LPAR (lpar2) which was configured with 100 GB of RAM. The managed system has only 36 GB of RAM and is therefore forced to take resources from inactive LPARs in order to provide the required 100 GB. We start lpar2 with the profile standard and look at the memory relations:

linux $ lpar activate -b sms -p standard lpar2
linux $

HMC command line:

hmc01: chsysstate -m ms09 -r lpar -o on -n lpar2 -b sms -f standard

Overview of the storage relations of lpar1 and lpar2:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  1      0.2         60160   Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     35584     786432        0      256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The LPAR lpar2 has 100 GB RAM, the managed system has no more free memory and the memory allocated by LPAR lpar1 has been reduced to about 60 GB. Allocated resources from non-activated LPARs are automatically released, when needed and assigned to other LPARs.

But you can of course also release the resources manually. This is also shown briefly here. We are reducing the memory of LPAR lpar1 by 20 GB:

linux $ lpar -d rmmem lpar1 20480
linux $

HMC command line:

hmc01: chhwres -m ms09 -r mem  -o r -p lpar1 -q 20480

As stated, the allocated memory has been reduced by 20 GB:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  1      0.2         39680   Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME  INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     35584     786432        20480  256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The 20 GB are immediately available to the managed system as free memory. If you remove the entire memory or all processors (or processor units), then all resources of an inactive LPAR are released:

linux $ lpar -d rmmem lpar1 39680
linux $

HMC command line:

hmc01: chhwres -m ms09 -r mem  -o r -p lpar1 -q 39680

Here are the resulting memory relations:

linux $ lpar status lpar\*
NAME   LPAR_ID  LPAR_ENV  STATE          PROFILE   SYNC  RMC       PROCS  PROC_UNITS  MEM     OS_VERSION
lpar1  4        aixlinux  Not Activated  standard  0     inactive  0      0.0         0       Unknown
lpar2  8        aixlinux  Open Firmware  standard  0     inactive  1      0.2         102400  Unknown
linux $ ms lsmem ms09
NAME        INSTALLED  FIRMWARE  CONFIGURABLE  AVAIL  MEM_REGION_SIZE
ms09  786432     31232     786432        64512  256
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09
hmc01: lshwres -r mem -m ms09 --level lpar
hmc01: lshwres -r proc -m ms09 --level lpar
hmc01: lshwres -r mem -m ms09 --level sys

The LPAR lpar1 now has 0 processors, 0.0 processor units and 0 MB of memory! In addition, the resource_config attribute now has the value 0, which indicates that the LPAR no longer has any resources configured!

linux $ lpar status -F resource_config lpar1
0
linux $

HMC command line:

hmc01: lssyscfg -r lpar -m ms09 --filter lpar_names=lpar1 –F resource_config

Finally, the question arises as to why you should release resources manually if they are automatically released by the managed system when needed?

We will answer this question in a second article.

 

Accessing the Update Access Key Expiration Date from AIX

As part of the introduction of POWER8 systems, IBM has also introduced the “Update Access Key”, which is necessary to perform firmware updates of the managed system. By default, newly delivered systems have an update access key that usually expires after 3 years. Thereafter, the Update Access Key can has to be extended every 6 month, but only if a maintenance contract exists(https://www.ibm.com/servers/eserver/ess/index.wss).

Of course, it is easy to find out when the current Update Access Key runs through the HMC, GUI or CLI. But you can also display the expiration date via the lscfg command from AIX:

In the case of AIX 7.1, this looks like this:

$ lscfg -vpl sysplanar0 | grep -p "System Firmware"
      System Firmware:
...
        Microcode Image.............SV860_138 SV860_103 SV860_138
        Microcode Level.............FW860.42 FW860.30 FW860.42
        Microcode Build Date........20180101 20170628 20180101
        Microcode Entitlement Date..20190825
        Hardware Location Code......U8284.22A.XXXXXXX-Y1
      Physical Location: U8284.22A.XXXXXXX-Y1

In the case of AIX 7.2, the output is slightly different:

$ lscfg -vpl sysplanar0 |grep -p "System Firmware"
      System Firmware:
...
        Microcode Image.............SV860_138 SV860_103 SV860_138
        Microcode Level.............FW860.42 FW860.30 FW860.42
        Microcode Build Date........20180101 20170628 20180101
        Update Access Key Exp Date..20190825
        Hardware Location Code......U8284.22A.XXXXXXX-Y1
      Physical Location: U8284.22A.XXXXXXX-Y1

Relevant are the lines “Microcode Entitlement Date” respectively “Update Access Key Exp Date“.

Special features of NFSv4 Mounts

Many AIX administrators use NFSv4 in the same way as NFSv3 and NFSv2 before. When exporting and mounting version 4 is specified, otherwise everything else is done as before. While this works in most cases, it also prevents the use of some interesting features of NFSv4.

The first significant difference between NFSv4 and its predecessors already shows up when mounting. NFSv2 and NFSv3 use a separate MOUNT protocol. For example, to run the following NFS mount:

clientv3 # mount aixnim:/export/data /mnt
clientv3 #

an RPC request is sent to the rpc.mountd on the NFS server to get an NFS filehandle for the file system/directory /export/data. All NFS operations must always specify an NFS filehandle that uniquely identifies a file or directory on the NFS server. The very first filehandle is requested via the MOUNT protocol.

In the case of NFSv4 this looks quite different, no separate MOUNT protocol is needed. Instead, a so-called root filehandle is used, which is defined by the NFS standard and need not be learned through a separate protocol. At the following NFSv4 mount

clientv4 # mount –o vers=4 aixnim:/export/data /mnt
clientv4 #

the client starts an NFS LOOKUP specifying the well-known (NFS default) root filehandle and relative path “export/data“, then the NFS server returns the associated filehandle. This can easily be tracked with the help of tcpdump, which we did not do here for reasons of space.

A side effect that is surprising for many administrators (and maybe not always understood) is that you can not see the filesystems exported with NFSv4 with “showmount -e“. This is simply because there is no MOUNT protocol for NFSv4. So you can not easily find out on the NFS client which file systems the NFS server has exported for NFSv4.

clientv4 # showmount -e aixnim
no exported file systems for aixnim
clientv4 #

The command “showmount -e” does not show exported file systems, although we were able to mount successfully using NFSv4. We’ll come back to that later.

The second significant difference is that for NFSv4 the NFS server generates a pseudo-file system for each NFSv4 client. This file system starts with the nfsroot directory (by default this is /) and contains all underlying directories and files exported to the client. The pseudo-file system is also created if there is no exported file system for the client!

For demonstration purposes, we have un-exported all file systems on our NFS server aixnim:

aixnim # lsnfsexp
aixnim #

Although nothing has been exported yet for the NFS client, the per client-generated pseudo-file system can be mounted using NFSv4:

clientv4 # mount -o vers=4 aixnim:/ /mnt
clientv4 # ls -il /mnt
total 0
clientv4 #

The mounted file system is of course empty, since nothing has been exported yet. We unmount the mounted file system again (not shown here) and export the directory /var/adm on the NFS server aixnim:

aixnim # mknfsexp -d /var/adm -v 4 -r clientv4
aixnim # lsnfsexp
/var/adm -vers=4,root=clientv4
aixnim #

We now mount the pseudo-file system / again:

clientv4 # mount -o vers=4 aixnim:/ /mnt
clientv4 #

In order to illustrate the differences to NFSv2 and NFSv3 more easily, we briefly show the useful command nfs4cl for the NFSv4 client:

clientv4 # nfs4cl showfs

Server      Remote Path          fsid                 Local Path        
--------    ---------------      ---------------      ---------------   
aixnim    /                    0:42949672964        /mnt              
clientv4 #

The command shows the pseudo-file system / from aixnim, which is mounted under /mnt. We now look briefly with the command ls in the directory /mnt

clientv4 # ls -il /mnt
total 1
    2 dr-xr-xr-x    2 root     system            3 May 21 07:34 var
clientv4 #

In the pseudo-file system generated by the NFS server, only the path component /var is visible. This path component is a prefix of the exported directory /var/adm. Other directories such as /opt or /usr are not visible in the pseudo-file system, because they are not prefixes of an exported path. We take a look at /mnt/var:

clientv4 # ls -il /mnt/var   
total 8
   32 drwxrwxr-x   15 root     adm            4096 May  2 11:30 adm
clientv4 #

Also under directory var, only the directory adm is visible, since only /var/adm is a prefix of an exported path. The pseudo-file system is of course unchangeable at the places that were not exported, as the attempt to create a file under /mnt/var shows:

clientv4 # touch /mnt/var/file
touch: /mnt/var/file cannot create
clientv4 #

Starting with /mnt/var/adm everything looks like we know from NFSv2 and NFSv3, one has access to the exported data:

clientv4 # ls -il /mnt/var/adm
total 704
  110 drw-r-----    2 root     system          256 May 20 14:33 SRC
4165 drwxrwxr-x    2 adm      adm             256 Apr 17 08:07 acct
   70 drwx------    4 root     system          256 Apr 17 07:50 config
4133 drwx------    2 root     system          256 Apr 17 08:03 corrals
...
4   33 -rw-rw-r--    1 adm      adm          337608 May 20 09:30 wtmp
clientv4 #

Now let’s look again at the output of the command “nfs4cl showfs“:

Clientv4 # nfs4cl showfs

Server      Remote Path          fsid                 Local Path        
--------    ---------------      ---------------      ---------------   
aixnim   /var                 0:42949672966        /mnt/var          
aixnim   /                    0:42949672964        /mnt        
clientv4 #

For each physical file system on the server, a separate pseudo-file system is created. The respective pseudo-file system grants access to exported directories of the underlying physical file system and generates read-only directories for path prefixes of exported directories.

On the NFS server aixnim we export the directory /usr/share for the client:

aixnim # mknfsexp -d /usr/share -v 4 -r clientv4
aixnim # lsnfsexp
/var/adm -vers=4,root=clientv4
/usr/share -vers=4,root=clientv4
aixnim #

On the client this time we do not umount and re-mount, but simply use ls to access the mount point /mnt again:

clientv4 # ls -il /mnt
total 2
    2 dr-xr-xr-x    2 root     system            3 May 21 08:13 usr
    2 dr-xr-xr-x    2 root     system            3 May 21 07:34 var
clientv4 #

The path prefix usr of the currently exported directory /usr/share appears on the client without having explicitly performed a mount. A look at /mnt/usr shows that the directory share appears as expected:

clientv4 # ls -il /mnt/usr

total 0

16390 drwxr-xr-x    8 bin      bin             256 Apr 17 08:31 share

clientv4 #

And under /mnt/usr/share are, as expected, the exported data:

clientv4 # ls -il /mnt/usr/share
total 24
74212 drwxr-xr-x    2 bin      bin             256 Apr 17 08:24 X11
49162 drwxr-xr-x    2 bin      bin             256 Nov  3 2015  dict
16391 drwxr-xr-x   12 bin      bin            4096 Apr 17 08:22 lib
17762 lrwxrwxrwx    1 root     system           26 Apr 17 08:31 locale -> /opt/freeware/share/locale
20653 drwxr-xr-x    5 root     system          256 Apr 24 15:46 lpp
16911 drwxr-xr-x   11 bin      bin            4096 May 20 14:25 man
45096 drwxr-xr-x    2 bin      bin            4096 Apr 17 08:03 modems
clientv4 #

The command “nfs4cl showfs” now shows 3 file systems:

Clientv4 # nfs4cl showfs

Server      Remote Path          fsid                 Local Path        
--------    ---------------      ---------------      ---------------   
aixnim /usr                 0:42949672965        /mnt/usr          
aixnim /var                 0:42949672966        /mnt/var          
aixnim /                    0:42949672964        /mnt              
clientv4 #

The last example shows that in the case of NFSv4 new file systems do not necessarily have to be manually mounted on the client. If further file systems are exported on the NFS server for an NFSv4 client, the client can simply access the new file system. Prerequisite, however, is that the path for the new file system is part of the client-mounted pseudo-file system. Since we had the entire pseudo-file system starting from the nfsroot mounted, this is trivially the case. But if we had only mounted /var from the NFS server, then /usr/share would not be part of the pseudo-file system of /var and a separate mount would have been necessary.

That additional file systems can be accessed without explicit mount, as just shown, comes from a third difference of NFSv4 to its predecessors. For NFSv2 and NFSv3, all filehandles are persistent, that is, immutable. NFSv4 introduced volatile filehandles in addition to persistent filehandles. The file handles used by the pseudo-file system are volatile. That means the client must expect that such a filehandle can change. This is the case if another path is exported on the NFS server, the filehandles for the path prefixes in the pseudo-file system change then, the client recognizes this after a short time and responds accordingly.

Finally, a small problem should be noted: we had 1 mount done, but the command “nfs4cl showfs” showed that 3 filesystems were involved. But there is only one file system mounted, as df shows:

clientv4 # df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           1.00      0.86   15%     8367     5% /
/dev/hd2           6.12      2.90   53%    65563     9% /usr
/dev/hd9var        2.00      1.81   10%     1770     1% /var
/dev/hd3           2.00      1.89    6%      452     1% /tmp
/dev/hd1           1.00      0.88   12%      454     1% /home
/dev/hd11admin      0.50      0.50    1%       15     1% /admin
/proc                 -         -    -        -      - /proc
/dev/hd10opt       2.00      0.94   54%    22460    10% /opt
/dev/livedump      1.00      1.00    1%        4     1% /var/adm/ras/livedump
/dev/varadmloglv      2.00      1.85    8%      275     1% /var/adm/log
aixnim:/       0.38      0.20   47%    12644    22% /mnt
clientv4 #

The file system mounted under /mnt is 0.38 GB in size. On the NFS server, /usr/share and /var/adm were exported, a df shows the following sizes:

aixnim # df –g / /usr/share /var/adm
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           0.38      0.20   47%    12644    22% /
/dev/hd2           2.06      0.23   89%    39236    41% /usr
/dev/hd9var        0.50      0.45   10%      614     1% /var
aixnim #

Obviously, the values of the file system / of the NFS server are used on the client! Under /usr and thus also /usr/share would be available free space of close to 2 GB, but this is not reflected in the df output on the client. Of course it is difficult to specify values on the client, because multiple file systems are involved on the NFS server. The df command simply displays the data of the physical file system underlying the mounted pseudo-file system. In our case, this is the root file system of the NFS server. Here again you can use the command nfs4cl, which has a subcommand for displaying file system information similar to df:

clientv4 # nfs4cl showstat

Filesystem       512-blocks        Free  %Used       Iused %Iused  Mounted on
aixnim:/usr     4325376      482752    89%       39236    41% /mnt/usr  
aixnim:/var     1048576      947064    10%         614     1% /mnt/var  
aixnim:/      786432      417944    47%       12644    22% /mnt     
clientv4 #

This is identical to the values displayed by df on the NFS server.

But even with the standard df of AIX you can get this information, as the following output shows:

clientv4 # df -g /mnt /mnt/usr /mnt/usr/share /mnt/var /mnt/var/adm
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
aixnim:/           0.38      0.20   47%    12644    22% /mnt
[NFSv4]            2.06      0.23   89%    39236    41% /mnt/usr
[NFSv4]            2.06      0.23   89%    39236    41% /mnt/usr/share
[NFSv4]            0.50      0.45   10%      614     1% /mnt/var
[NFSv4]            0.50      0.45   10%      614     1% /mnt/var/adm
clientv4 #

Of course there are a number of other differences of NFSv4 and previous versions, but they should not be discussed here. Maybe there will be another article on the subject.

The features presented above are especially useful for hierarchical mounts. With NFSv4, you only need to mount one filesystem and you need not worry about the order of mounts.

Understanding how the pseudo-file system works for the NFSv4 client helps determine the exported file systems on the client. Instead of using “showmount -e” as before with NFSv2 and NFSv3 (which does not get results with NFSv4), you can simply mount everything starting with / and then use cd and ls to find out what the NFS server has exported.

Extremly fast growing /var/adm/wtmp

Recently, we had a full /var file system on one of our AIX SAP systems. It turned out that a 1.9 GB grown /var/adm/wtmp file was the cause. This file has grown within a short time to almost 2 GB. The question arose what produced the extreme number of entries. To determine this, the contents of the file was displayed in ASCII form:

# cat /var/adm/wtmp  | /usr/sbin/acct/fwtmp
         ac02                         8 25690134 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac01                         8 27525310 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac00                         8 27525308 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac00     ac00                         5 7864366 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac01     ac01                         5 7864368 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac02     ac02                         5 7864370 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac01                         8 7864368 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac00                         8 7864366 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
…
#

These entries repeated themselves endlessly, sometimes there were more than 50 entries within one second! The strings “ac00“, “ac01” and “ac02” are IDs from /etc/inittab. Column 2 respectively 3 shows the type of entry, here 5 and 8. The meaning can be found out in the header file /usr/include/utmp.h:

# cat /usr/include/utmp.h
…
/*      Definitions for ut_type                                         */
…
#define INIT_PROCESS    5       /* Process spawned by "init" */
…
#define DEAD_PROCESS    8
…

The processes were started by /etc/init and then died immediately. It looks like processes with the action “respawn” are started here, which are immediately terminated due to an error. We look at the corresponding inittab entries:

#  lsitab ac00    
ac00:2345:respawn:/oracle/NW1/acs/acsgen -D
#  lsitab ac01
ac01:2345:respawn:/oracle/NW1/acs/acsd
#  lsitab ac02
ac02:2345:respawn:/oracle/NW1/acs/fcmcli -D
#

These are Oracle entries that obviously do not work as intended.

In our case, the binaries simply did not exist at the specified location:

#  ls -l /oracle/NW1/acs/acsgen /oracle/NW1/acs/acsd /oracle/NW1/acs/fcmcli
ls: 0653-341 The file /oracle/NW1/acs/acsgen does not exist.
ls: 0653-341 The file /oracle/NW1/acs/acsd does not exist.
ls: 0653-341 The file /oracle/NW1/acs/fcmcli does not exist.
#

In consultation with the Oracle colleagues, the entries in /etc/inittab were removed, which resolved the problem:

# rmitab ac00
# rmitab ac01
# rmitab ac02
#

Incorrect entries in /etc/inittab can result in a fast growing /var/adm/wtmp file.

 

AIX: Applications of the namefs-Filesystem

Occasionally one needs a directory (or a file system) somewhere else in the file system or maybe even at several different places in the file system. Instead of solving the problem with symbolic links, you can elegantly use the namefs file system.

In the following example /data/in is required elsewhere:

# ls -l /data/in
total 16
-rw-r--r--    1 root     system          554 May 14 16:10 file1
-rw-r--r--    1 root     system          381 May 14 16:10 file2
# ls -l /other/place
total 0
#

Mounting the directory to the desired location /other/place:

# mount -v namefs /data/in /other/place
# ls -l /other/place
total 16
-rw-r--r--    1 root     system          554 May 14 16:10 file1
-rw-r--r--    1 root     system          381 May 14 16:10 file2
#

The mount with the namefs file system additionally offers the possibility to specify mount options, which then only apply to the directory. One can do so to mount a directory with Direct-I/O, even though the original directory was not mounted with Direct-I/O:

# mount -v namefs -o dio /data/in /other/place
# mount
  node       mounted        mounted over    vfs       date        options     
-------- ---------------  ---------------  ------ ------------ ---------------
         /dev/hd4         /                jfs2   May 02 11:30 rw,log=/dev/hd8
...
         /data/in         /other/place     namefs May 14 16:14 rw,dio         
#

When accessing the files below /other/place, Direct-I / O will be used. When accessing via the “originals” under /data/in, no Direct-I / O will be used!

However, access to files is limited to the underlying physical file system, as with NFS. This can easily be demonstrated by the file system /. We mount / using namefs onto /mnt and look at /mnt/usr and /mnt/var:

# mount -v namefs / /mnt
# ls -l /mnt/usr /mnt/var
/mnt/usr:
total 0
lrwxrwxrwx    1 root     system           11 Apr 17 07:49 lib -> /../usr/lib

/mnt/var:
total 0
#

The directories are empty or contain a symbolic link, /usr and /var clearly look different!

Of course, this can also be exploited, e.g. in cases where interesting data has been over-mounted. We dropped a file below /home before /dev/hd1 was mounted onto /home. The root file system currently mounted on /mnt allows access to this over-mounted data:

# ls -l /mnt/home
total 0
-rw-r--r--    1 root     system            0 May 14 17:48 overmounted_file
#

Another application is to protect a directory against overwriting. We demonstrate this for the directory /data with 2 test files:

# ls -l /data
total 16
-rw-r--r--    1 root     system          554 May 14 17:52 file1
-rw-r--r--    1 root     system          381 May 14 17:52 file2
# cp /etc/hosts /data
# ls -l /data
total 24
-rw-r--r--    1 root     system          554 May 14 17:52 file1
-rw-r--r--    1 root     system          381 May 14 17:52 file2
-rw-r--r--    1 root     system         2075 May 14 17:54 hosts
#

Overwriting or changing data is currently still possible, as shown by the successful cp command. Now we protect the data by doing a mount with the namefs file system and the option ro (read-only):

# mount -v namefs -o ro /data /data
# cp /etc/services /data
cp: /data/services: Read-only file system
#

The data obviously can not be changed anymore. Here we have over-mounted /data with a read-only version of itself!

Mounts with the namefs pseudo-file system can not only be done on jfs2 file systems, but also for NFS file systems or the procfs file system.

Finally we show the mounting of a file to some other place in the file system. We want to make the file /etc/hosts available via the name /hosts. To do this, we first create an empty file /hosts and then mount the file /etc/hosts onto this empty file:

# touch /hosts
# ls -l /hosts
-rw-r--r--    1 root     system            0 May 14 17:59 /hosts
# mount -v namefs /etc/hosts /hosts
# ls -l /hosts
-rw-rw-r--    1 root     system         2075 Apr 26 10:47 /hosts
#

Before the mount, /hosts was 0 bytes in size, after the mount 2075 bytes!

The namefs file system thus offers some interesting possibilities, that can be useful for some problems.

 

FC NPIV client throughput

When using NPIV, multiple client LPARs share a physical FC port of a virtual I/O server. Of course, for performance investigations, it would be nice to be able to easily determine the throughput of each client LPAR and to look at the througputs comparatively. Thus, questions like

  • how much throughput is achieved by a particular LPAR
  • which LPARs have the highest throughput and produce the most FC traffic
  • are there resource bottlenecks

could be answered.

Of course, there are several ways to gain this data. A particularly simple option is provided by the virtual I/O server via the padmin command ‘fcstat‘. The command allows to show NPIV client statistics, using the ‘-client‘ option:

(0)padmin@aixvio1:/home/padmin> fcstat -client
              hostname   dev                wwpn     inreqs    outreqs ctrlreqs          inbytes         outbytes  DMA_errs Elem_errs Comm_errs

               aixvio1  fcs0  0x100000XXXXXXXXXX 49467894179 50422150679 947794529 1861712755360927 1451335312750576         0         0         0
     C050760YYYYYYYYY
                                    0          0        0                0                0         0         0         0
     C050760ZZZZZZZZZ
                                    0          0        0                0                0         0         0         0
                 aix01  fcs0  0xC050760XXXXXXXXX   22685402  101956075 10065757     699512617896    1572578056704         0         0         0
                 aix02  fcs0  0xC050760XXXXXXXXX   28200473   82295158 12051365     387847746448     626772151808         0         0         0
                 aix03  fcs0  0xC050760XXXXXXXXX  376500672  255163053 21583628   22619424512608    3786990844928         0         0         0
                 aix04  fcs0  0xC050760XXXXXXXXX  116450405  504688524 14020031    4037786527400    9929289617408         0         0         0
          blbprodora22  fcs0  0xC050760XXXXXXXXX 1341092479  580673554 37458927   44288566807072   12166718497792         0         0         0
...
               aixvio1  fcs1  0x100000XXXXXXXXXX  391131484 1090556094 156294130   71031615240217   87642294572864         0         0         0
              aixtsm01  fcs2  0xC050760XXXXXXXXX  334020900  785597352 74659821   62072552942128   83284555980288         0         0         0
              aixtsm02  fcs0  0xC050760XXXXXXXXX    2943054   40921231 11617552     107317697968     289142333440         0         0         0

               aixvio1  fcs2  0x210000XXXXXXXXXX  403180246 5877180796   236998  105482699300998 1540608710446612         0         0         0
              aixtsm01  fcs6  0xC050760XXXXXXXXX  146492419  392365162    74250   38378099796342  102844775468007         0         0         0
              aixtsm02  fcs2  0xC050760XXXXXXXXX         19     192848       20             1090      50551063184         0         0         0

               aixvio1  fcs3  0x210000XXXXXXXXXX  405673338 7371951499   260575  105969796271246 1932388891128304         0         0         0
              aixtsm02  fcs3  0xC050760XXXXXXXXX          0          0        4                0                0         0         0         0
                 aix02  fcs7  0xC050760XXXXXXXXX      42624 2677470211    34211          2382280  701864613402184         0         0         0
...
Invalid initiator world wide name
Invalid initiator world wide name
(0)padmin@aixvio1:/home/padmin>

The line with WWPN C050760YYYYYYYYY and C050760ZZZZZZZZZ belongs to NPIV adapters of non-activated LPARs. Therefore, only zeros are displayed as counters. For each virtual (NPIV-enabled) FC port of the virtual I/O server, the physical FC port and the NPIV client LPARs are displayed. Based on the bold-marked block, the output will be briefly described here. First, the physical port of the virtual I/O server is always shown, here aixvio1 and FC port fcs1. In the following lines, the NPIV clients will be shown, each with the LPAR name and the associated virtual FC port of the LPAR, here aixtsm01 and aixtsm02. The virtual FC ports of the LPARs fcs2 (aixtsm01) and fcs0 (aixtsm02) are mapped to the physical FC port fcs1 of aixvio1. After a blank line comes the next physical FC port of the virtual I/O server.

The WWPN of the physical or virtual FC ports are listed in the columns. In addition, the number of incoming and outgoing requests, as well as the transferred bytes, also incoming and outgoing, are listed. Errors are listed in the 3 remaining columns. If there is no DMA buffer available for a request, DMA_errs is incremented, if the queue of the FC adapter is full, Elem_errs is incremented, in the case of transmission errors, Comm_errs is incremented. Regular increasing counters on DMA_errs or Elem_errs may be an indication of too small values for some tuning attributes.

Due to the length of the output and the absolute counters being output, the output is somewhat confusing. But with a small script, you can easily calculate delta values and scale the output to MB per second. With the following example script we have done this:

$ cat npivstat
#! /bin/ksh93
#
# Copyright (c) 2019 by PowerCampus 01 GmbH
# Author: Dr. Armin Schmidt
#

delta=5 # seconds

typeset -A dataInreqs
typeset -A dataOutreqs
typeset -A dataInbytes
typeset -A dataOutbytes
typeset -A dataDMA_errs
typeset -A dataElem_errs
typeset -A dataComm_errs

bc |& # start bc as coroutine
print -p "scale=2"

# get first sample

/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs
;;
esac
done
sleep $delta

while true
do
/usr/ios/cli/ioscli fcstat -client 2>/dev/null | \
while read hostname dev wwpn inreqs outreqs ctrlreqs inbytes outbytes DMA_errs Elem_errs Comm_errs rest
do
case "$wwpn" in
0x*)
prevInreqs=${dataInreqs[${hostname}_${dev}]}
prevOutreqs=${dataOutreqs[${hostname}_${dev}]}
prevInbytes=${dataInbytes[${hostname}_${dev}]}
prevOutbytes=${dataOutbytes[${hostname}_${dev}]}
prevDMA_errs=${dataDMA_errs[${hostname}_${dev}]}
prevElem_errs=${dataElem_errs[${hostname}_${dev}]}
prevComm_errs=${dataComm_errs[${hostname}_${dev}]}
dataInreqs[${hostname}_${dev}]=$inreqs
dataOutreqs[${hostname}_${dev}]=$outreqs
dataInbytes[${hostname}_${dev}]=$inbytes
dataOutbytes[${hostname}_${dev}]=$outbytes
dataDMA_errs[${hostname}_${dev}]=$DMA_errs
dataElem_errs[${hostname}_${dev}]=$Elem_errs
dataComm_errs[${hostname}_${dev}]=$Comm_errs

print -p "(${inreqs}-${prevInreqs})/$delta"
read -p inreqs
print -p "(${outreqs}-${prevOutreqs})/$delta"
read -p outreqs
print -p "(${inbytes}-${prevInbytes})/${delta}/1024/1024"
read -p inbytes
print -p "(${outbytes}-${prevOutbytes})/${delta}/1024/1024"
read -p outbytes
print -p "(${DMA_errs}-${prevDMA_errs})/$delta"
read -p DMA_errs
print -p "(${Elem_errs}-${prevElem_errs})/$delta"
read -p Elem_errs
print -p "(${Comm_errs}-${prevComm_errs})/$delta"
read -p Comm_errs

printf "%15s %5s %16s %6.2f %7.2f %7.2f %8.2f %8.2f %9.2f %9.2f\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"wwpn")
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
"")
[ -n "$hostname" ] && continue
printf "%15s %5s %16s %6s %7s %7s %8s %8s %9s %9s\n" "$hostname" "$dev" "$wwpn" "$inreqs" "$outreqs" \
"$inbytes" "$outbytes" "$DMA_errs" "$Elem_errs" "$Comm_errs"
;;
esac
done
print

sleep $delta
done

$

The script ‘npivstat‘ is available for download in our download-area.

Here is an excerpt from a run of the script (much shortened, only one of the physical ports is shown):

aixvio1 # ./npivstat
       hostname    dev              wwpn  inreqs  outreqs  inbytes  outbytes  DMA_errs  Elem_errs  Comm_errs
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1019.00     0.00    254.75      0.00       0.00       0.00
       aixtsm01   fcs6  0xC0507605E5890074    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC0507609A6C70004    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC0507609A6C7001C    0.00  1018.20     0.00    254.55      0.00       0.00       0.00
...                                                                                                          
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00  1020.20     0.00    255.05      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00  1019.80     0.00    254.95      0.00       0.00       0.00
...                                                                                                           
        aixvio1   fcs2  0x210000XXXXXXXXXX    0.00   984.80     0.00    246.20      0.00       0.00       0.00
       aixtsm01   fcs6  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
       aixtsm02   fcs2  0xC050760XXXXXXXXX    0.00     0.00     0.00      0.00      0.00       0.00       0.00
          aix05   fcs6  0xC050760XXXXXXXXX    0.00   985.00     0.00    246.25      0.00       0.00       0.00
...
^Caixvio1 # 

In the example above, the NPIV client aix05 generates approximately 250 MB/s of data, while the other two NPIV clients aixtsm01 and aixtsm02 have not produced FC traffic during this time.

The script must be started as root on a virtual I/O server. Of course you can customize the script to your own needs.

Migration to AIX PCM combined with OS update using Alternate Disk Copy

On most AIX systems the SP or TL level is updated at regular intervals. It makes sense to perform the migration from SDDPCM to AIX PCM together with such an update. This saves time and some reboots, which otherwise have to be done because of the multipathing migration.

In our blog post “Migration from SDDPCM to AIX-PCM” we had already shown the migration for standalone systems.

Here, the migration from SDDPCM to AIX-PCM will be shown as part of an OS update, using the Alternate Disk Copy method. The procedure is roughly the following:

  1. Unmirroring the rootvg to get a free disk for Alternate Disk Copy.
  2. Change the Path Control Module (PCM) to AIX PCM.
  3. Creating the altinst_rootvg.
  4. Removal of fixes in the altinst_rootvg.
  5. Performing the OS update on the altinst_rootvg.
  6. Installing fixes in the altinst_rootvg.
  7. Adding a firstboot script to set disk attributes.
  8. Change the Path Control Module (PCM) back to SDDPCM.
  9. Booting from the altinst_rootvg.

On our example system AIX 7.1 TL5 SP2 is installed, the disks are SVC disks connected via virtual FC adapters. SDDPCM is the currently active multipathing driver:

# oslevel -s
7100-05-02-1810
# lsdev -l hdisk0 -F uniquetype
disk/fcp/2145
aix01:/root> lsattr -El hdisk0 -a PCM -F value
PCM/friend/sddpcm
#

As stated in the blog post above, some disk attributes change when migrating to AIX PCM. Therefore, you should take a close look at the current attributes in order to take them over later (at least partially). By way of example, we only look at the attribute queue_depth, which currently has the value 120:

# lsattr -El hdisk0 -a queue_depth -F value
120
#

Our system has a mirrored rootvg:

# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            active            399         232         00..01..71..80..80
hdisk0            active            399         240         00..01..79..80..80
#

The system was booted from the hdisk0:

# bootinfo -b
hdisk0
#

So we leave hdisk0 in the rootvg and remove  hdisk3 from the rootvg to get a free disk for Alternate Disk Copy.

# unmirrorvg rootvg hdisk3
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.
0516-1804 chvg: The quorum change takes effect immediately.
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk0.
# reducevg rootvg hdisk3
# chpv -c hdisk3
# bootlist -m normal hdisk0
#

Before we create a copy of the rootvg using Alternate Disk Copy, we temporarily change the system to AIX PCM without, however, rebooting. If then the altinst_rootvg is generated, the conversion to AIX PCM is already done in altinst_rootvg!

# manage_disk_drivers -d IBMSVC -o AIX_AAPCM
********************** ATTENTION *************************
  For the change to take effect the system must be rebooted
#

At the end of the OS update, we then undo this change on the rootvg to have the original state with SDDPCM.

After these preparations we start now the alt_disk_copy command:

# alt_disk_copy -d hdisk3 -B
Calling mkszfile to create new /image.data file.
Checking disk sizes.
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5.
Creating logical volume alt_hd6.
Creating logical volume alt_hd8.
…
#

Some fixes are installed on the system, which we remove from the altinst_rootvg before the update:

# emgr -l
ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
=== ===== ========== ================= ========== ======================================
1    S    102m_ifix  10/14/18 10:48:18            IFIX for Openssl CVE on 1.0.2m       
2    S    IJ03121s0a 10/14/18 10:49:04            IJ03121 for AIX 7.1 TL5 SP00         
3    S    IJ05822s2a 10/14/18 10:49:18            a potential security issue exists    
…
#

Activation of the altinst_rootvg:

# alt_rootvg_op -W -d hdisk3
Waking up altinst_rootvg volume group ...
#

And removal of the fixes:

# INUCLIENTS=1 /usr/sbin/chroot /alt_inst /usr/sbin/emgr –r -n 3
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Accessing efix metadata ...
Processing efix label "IJ05822s2a" ...
…
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log

EFIX NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 IJ05822s2a          REMOVE                 SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 2
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -r -n 1
…

(Note: please do not forget the variable INUCLIENTS, this signals that the operation is taking place in an alternate boot environment!)

Now we mount the LPP source for the OS update via NFS from our NIM server:

# mount aixnim:/export/nim/lpps/aix710503lpp /mnt
#

The OS update can now be done in the altinst_rootvg:

# alt_rootvg_op -C -b update_all -l /mnt
Installing optional filesets or updates into altinst_rootvg...
install_all_updates: Initializing system parameters.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Checking for updated install utilities on media.
…
installp:  * * * A T T E N T I O N ! ! !
        Software changes processed during this session require
        any diskless/dataless clients to which this SPOT is
        currently allocated to be rebooted.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS
#

Finally, we install some fixes. We first mount the directory /mnt with the fixes in the altinst_rootvg:

# mount -v namefs /mnt /alt_inst/mnt
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/102p_fix.181127.epkg.Z
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
Efix package file is: /mnt/emgr/ppc/102p_fix.181127.epkg.Z
…
EPKG NUMBER       LABEL               OPERATION              RESULT           
===========       ==============      =================      ==============   
1                 102p_fix            INSTALL                SUCCESS          

Return Status = SUCCESS
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ09621s3a.181001.epkg.Z
…
# INUCLIENTS=1 chroot /alt_inst /usr/sbin/emgr -e /mnt/emgr/ppc/IJ11545s0a.181127.epkg.Z
…
# umount /alt_inst/mnt
#

To set the desired disk attributes and uninstall SDDPCM we use a firstboot script:

# cat /alt_inst/etc/firstboot
#! /bin/ksh

print "INFO: adjust hdisk attributes"
chdev -Pl hdisk0 -a queue_depth=120

print "INFO: uninstalling SDDPCM"
installp -u devices.sddpcm.$(uname -v)$(uname -r).rte devices.fcp.disk.ibm.mpio.rte

print "INFO: perform reboot"
reboot

# chmod a+x /alt_inst/etc/firstboot
#

The script should, if used, be adapted to your own needs. There, you should customize all the desired disk attributes (queue_depth, reserve_policy, …). The sample script here is just to indicate what you could do!

The altinst_rootvg is now updated and converted to AIX PCM. We disable the altinst_rootvg so that it can be booted.

# alt_rootvg_op –S -t
Putting volume group altinst_rootvg to sleep ...
forced unmount of /alt_inst/var/adm/ras/livedump
…
forced unmount of /alt_inst
Fixing LV control blocks...
Fixing file system superblocks...
#

(Note: please do not forget the option “-t“, this creates a new boot image!)

But before we boot from the altinst_rootvg, we change the multipathing driver back to SDDPCM on the rootvg!

# manage_disk_drivers -d IBMSVC -o NO_OVERRIDE
********************** ATTENTION *************************
  For the change to take effect the system must be rebooted
#

Finally we change the bootlist to altinst_rootvg (hdisk3):

# bootlist -m normal hdisk3
#

And last but not least we reboot:

# shutdown –r now

SHUTDOWN PROGRAM
Tue Apr 16 19:49:08 CEST 2019

Broadcast message from root@aix01 (tty) at 19:49:08 ...

PLEASE LOG OFF NOW ! ! !
System maintenance in progress.
All processes will be killed now.
…

-------------------------------------------------------------------------------
                                Welcome to AIX.
                   boot image timestamp: 19:45:08 04/16/2019
                 The current time and date: 19:51:11 04/16/2019
        processor count: 2;  memory size: 4096MB;  kernel size: 36847630
       boot device: /vdevice/vfc-client@3000000a/disk@5005076XXXXXXXXX:2
-------------------------------------------------------------------------------
…
Multi-user initialization completed
INFO: adjust hdisk attributes
hdisk0 changed
INFO: uninstalling SDDPCM
…
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
devices.sddpcm.71.rte       2.7.1.1         ROOT        DEINSTALL   SUCCESS   
devices.sddpcm.71.rte       2.7.1.1         USR         DEINSTALL   SUCCESS   
devices.fcp.disk.ibm.mpio.r 1.0.0.25        USR         DEINSTALL   SUCCESS   
INFO: perform reboot
Rebooting . . .
…

AIX Version 7
Copyright IBM Corporation, 1982, 2018.
Console login:

(In the output you can see the actions of the firstboot script: changing disk attributes, uninstalling SDDPCM and rebooting.)

After logging in we check the OS version, the used multipathing driver and some disk attributes:

# oslevel -s
7100-05-03-1846
# lsdev -l hdisk0 -F uniquetype
disk/fcp/mpioosdisk
# lsattr -El hdisk0 -a PCM -F value
PCM/friend/fcpother
# lsattr -El hdisk0 -a queue_depth -F value
120
# genkex|grep pcm
         5ae0000    60000 /usr/lib/drivers/aixdiskpcmke
# lslpp -l|grep sddpcm
#

We have successfully completed the migration from SDDPCM to AIX PCM together with an OS update. Using scripts this can be automated further.

We have tested this procedure for AIX 7.1 and AIX 7.2. So far, we have not been able to carry out a test for PowerHA for reasons of time.

Did you know that state and configuration change information is available on the HMC for about 2 months?

Status and configuration changes of LPARs and managed systems are stored on the HMCs for about 2 months. This can be used to find out, when a managed system was shut down, when a service processor failover took place, or when the memory of an LPAR was expanded, at least if the event is no more than 2 months ago.

The status changes of a managed system can be listed with the command “lslparutil -r sys -m <managed-system> -sh –startyear 1970 –filter event_types = state_change“, or alternatively with the LPAR-Tool command “ms history <managed -system> “.

linux $ ms history ms04
TIME                  PRIMARY_STATE         DETAILED_STATE
03/14/2019 08:45:13   Started               None
03/14/2019 08:36:52   Not Available         Unknown
02/17/2019 01:51:55   Started               None
02/17/2019 01:44:00   Not Available         Unknown
02/12/2019 09:32:57   Started               None
02/12/2019 09:28:02   Started               Service Processor Failover
02/12/2019 09:27:07   Started               None
02/12/2019 09:24:42   Standby               None
02/12/2019 09:21:25   Starting              None
02/12/2019 09:22:59   Stopped               None
02/12/2019 09:21:58   Not Available         Unknown
02/12/2019 09:09:45   Stopped               None
02/12/2019 09:07:53   Stopping              None
linux $

Configuration changes (processor, memory) of a managed system can be displayed with “lslparutil -r sys -m <managed-system> -s h –startyear 1970 –filter event_types = config_change“, or alternatively again with the LPAR tool:

linux $ ms history -c ms02
                                PROCUNIS              MEMORY
TIME                  CONFIGURABLE  AVAILABLE  CONFIGURABLE  AVAILABLE  FIRMWARE
04/16/2019 12:15:51      20.0          5.05       1048576       249344     25856
04/11/2019 11:17:39      20.0          5.25       1048576       253696     25600
04/02/2019 13:24:35      20.0          4.85       1048576       249344     25856
03/29/2019 14:29:14      20.0          5.25       1048576       253696     25600
03/15/2019 15:37:08      20.0          4.85       1048576       249344     25856
03/15/2019 11:36:57      20.0          4.95       1048576       249344     25856
...
linux $

The same information can also be displayed for LPARs.

The last status changes of an LPAR can be listed with “lpar history <lpar>“:

linux $ lpar history lpar02
TIME                  PRIMARY_STATE         DETAILED_STATE
04/17/2019 05:42:43   Started               None
04/17/2019 05:41:24   Waiting For Input     Open Firmware
04/16/2019 12:01:54   Started               None
04/16/2019 12:01:29   Stopped               None
02/15/2019 11:30:48   Stopped               None
02/01/2019 12:23:34   Not Available         Unknown
02/01/2019 12:22:50   Relocating            None
...

This corresponds to the command “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = state_change, lpar_names = lpar02” on the HMC command line.

From the output it can be seen that the LPAR has been relocated using LPM, was stopped and restartet and has been in Open Firmware mode.

And finally you can look at the last configuration changes of an LPAR using the command on the HMC CLI “lslparutil -r lpar -m ms03 -s h –startyear 1970 –filter event_types = config_change, lpar_names = lpar02“. The output of the LPAR tool is a bit clearer:

linux $ lpar history -c lpar02
TIME                  PROC_MODE  PROCS  PROCUNITS  SHARING  UNCAP_WEIGHT  PROCPOOL         MEM_MODE  MEM
04/23/2019 18:49:43   shared    1      0.7        uncap    10          DefaultPool      ded       4096
04/23/2019 18:49:17   shared    1      0.7        uncap    5           DefaultPool      ded       4096
04/23/2019 18:48:44   shared    1      0.3        uncap    5           DefaultPool      ded       4096
04/09/2019 08:04:25   shared    1      0.3        uncap    5           DefaultPool      ded       3072
03/14/2019 12:37:32   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/26/2019 09:34:28   shared    1      0.1        uncap    5           DefaultPool      ded       3072
02/20/2019 06:51:57   shared    1      0.3        uncap    5           DefaultPool      ded       3072
01/31/2019 08:12:58   shared    1      0.3        uncap    5           DefaultPool      ded       3072
..

From the output you can see that the number of processing units were changed several time, the uncapped weight was changed and the memory has been expanded.

Changes of the last two months are available at any time!