Extremly fast growing /var/adm/wtmp

Recently, we had a full /var file system on one of our AIX SAP systems. It turned out that a 1.9 GB grown /var/adm/wtmp file was the cause. This file has grown within a short time to almost 2 GB. The question arose what produced the extreme number of entries. To determine this, the contents of the file was displayed in ASCII form:

# cat /var/adm/wtmp  | /usr/sbin/acct/fwtmp
         ac02                         8 25690134 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac01                         8 27525310 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac00                         8 27525308 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac00     ac00                         5 7864366 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac01     ac01                         5 7864368 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
ac02     ac02                         5 7864370 0000 0000 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac01                         8 7864368 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
         ac00                         8 7864366 0000 0177 1558338990                                  Mon May 20 09:56:30 DFT 2019
…
#

These entries repeated themselves endlessly, sometimes there were more than 50 entries within one second! The strings “ac00“, “ac01” and “ac02” are IDs from /etc/inittab. Column 2 respectively 3 shows the type of entry, here 5 and 8. The meaning can be found out in the header file /usr/include/utmp.h:

# cat /usr/include/utmp.h
…
/*      Definitions for ut_type                                         */
…
#define INIT_PROCESS    5       /* Process spawned by "init" */
…
#define DEAD_PROCESS    8
…

The processes were started by /etc/init and then died immediately. It looks like processes with the action “respawn” are started here, which are immediately terminated due to an error. We look at the corresponding inittab entries:

#  lsitab ac00    
ac00:2345:respawn:/oracle/NW1/acs/acsgen -D
#  lsitab ac01
ac01:2345:respawn:/oracle/NW1/acs/acsd
#  lsitab ac02
ac02:2345:respawn:/oracle/NW1/acs/fcmcli -D
#

These are Oracle entries that obviously do not work as intended.

In our case, the binaries simply did not exist at the specified location:

#  ls -l /oracle/NW1/acs/acsgen /oracle/NW1/acs/acsd /oracle/NW1/acs/fcmcli
ls: 0653-341 The file /oracle/NW1/acs/acsgen does not exist.
ls: 0653-341 The file /oracle/NW1/acs/acsd does not exist.
ls: 0653-341 The file /oracle/NW1/acs/fcmcli does not exist.
#

In consultation with the Oracle colleagues, the entries in /etc/inittab were removed, which resolved the problem:

# rmitab ac00
# rmitab ac01
# rmitab ac02
#

Incorrect entries in /etc/inittab can result in a fast growing /var/adm/wtmp file.