HSCF0180E Operation failed for

When trying to update the system firmware of a managed system via the HMC Command Line, we encountered the following error message:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027
HSCF0180E Operation failed for ms26 (9009-22A*XXXXXXX).
Could not unpack the firmware update package.
Check the health and available disk space of the file system.
hmc01:~>

The error message that was displayed suggested checking the available space in the HMC file systems:

hmc01:~> lshmcfs
filesystem=/var,filesystem_size=7935,filesystem_avail=4955,temp_files_start_time=11/22/2018 12:59:00,temp_files_size=2011
filesystem=/dump,filesystem_size=60347,filesystem_avail=55935,temp_files_start_time=02/15/2021 10:21:00,temp_files_size=0
filesystem=/extra,filesystem_size=20030,filesystem_avail=15939,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15615,filesystem_avail=4369,temp_files_start_time=02/15/2021 06:05:00,temp_files_size=4
hmc01:~>

Actually, the available space should be sufficient, but to be on the safe side, we’ve cleaned up a bit with the temporary files:

hmc01:~> chhmcfs -o f -d 5
hmc01:~>

The updlic command was unimpressed and returned the same error message.

The removal of some old firmware versions from the HMC‘s local disk repository was also unsuccessful:

hmc01:~> updlic -o p --ecnumber 01AL740
hmc01:~> updlic -o p --ecnumber 01AL770
hmc01:~> updlic -o p --ecnumber 01AM740
hmc01:~>

The error message was still the same. Obviously, contrary to the information in the error message, the problem had nothing to do with the available space on the HMC!

We then took a closer look at the downloaded firmware. We downloaded the firmware as an ISO file H75557812_01VL940_071_027.iso from the IBM website and then mounted it on our NIM server using the loopmount command:

aixnim:/root> loopmount -i /tmp/H75557812_01VL940_071_027.iso -o "-o ro -V cdrfs" -m /mnt
aixnim:/root> ls -l /mnt
total 528296
-rw-r--r--    1 102010979 213            1860 Feb 04 09:08 01VL940071_special_instructs.xml.special.note.xml
-rw-r-----    1 102010979 210            7290 Feb 04 09:08 01VL940_071_027.dd.xml
-rw-r--r--    1 102010979 213           95687 Feb 04 09:07 01VL940_071_027.html
-rw-r-----    1 102010979 210            2971 Feb 04 09:08 01VL940_071_027.pd.sdd
-rw-r-----    1 102010979 210           67338 Feb 04 09:08 01VL940_071_027.readme.txt
-rw-r-----    1 102010979 210       134969022 Feb 04 09:08 01VL940_071_027.rpm
-rw-r-----    1 102010979 210       135328848 Feb 04 09:08 01VL940_071_027.tar.gz
-rw-r-----    1 102010979 210            9442 Feb 04 09:08 01VL940_071_027.xml
aixnim:/root>

What we didn’t notice when copying the files was the lack of read permissions on other for most files.

After we had assigned read permissions for all files, the next update attempt was successful:

hmc01:~> updlic -o a -t all -l latest -m ms26 -r sftp -h X.X.X.X -u XXXXXX --passwd XXXXXXXX -d /firmware/system/01VL940_071_027 
HSCF0179W Operation was partially successful for ms26 (9009-22A*XXXXXXX).
The following deferred fixes are present in the fix pack.  Deferred fixes will be activated after the next IPL of the system.
An immediate IPL is not required, unless you want to activate one of the fixes below now.
..
hmc01:~>

Don’t Forget to Update AIX-rpm

Recently, when installing Python3 from the AIX toolbox using YUM, we encountered the following error situation:

# yum install python3
...
--> Finished Dependency Resolution
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libssl.a(libssl.so.1.0.2)
Error: Package: python3-3.7.1-1.ppc (AIX_Toolbox)
           Requires: libcrypto.a(libcrypto.so.1.0.2)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
#

Python3 requires version 1.0.2 of libssl.so and libcrypto.so, but the two libraries cannot be found. These two libraries can either be provided via the RPM package openssl, in the appropriate version, or via AIX. When the AIX toolbox is used, the openssl package is typically not installed, and the AIX OpenSSL libraries are used. Therefore, in our case, the OpenSSL RPM package is not installed. The libraries are then made available by BFF packages of the AIX operating system.

The information about which libraries are made available by AIX is provided via the virtual RPM package AIX-rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so.0.9.7
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

The output shows that versions 0.9.8 and 1.0.0 of the two required libraries are known to the AIX-rpm package, but the required version 1.0.2 is apparently not available. We take a quick look at the associated archive /usr/lib/libssl.a (or /usr/lib/libcrypto.a):

# ar -X any t /usr/lib/libssl.a
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0
libssl.so.1.0.2
libssl64.so
libssl64.so.0.9.8
libssl64.so.1.0.0
#

The archive obviously also provides version 1.0.2. This information is not provided by the AIX-rpm RPM package. This is because a newer version of openssl.base has obviously been installed, but the AIX-rpm package has not been updated. The RPM package currently has the following version:

# rpm -q AIX-rpm
AIX-rpm-7.1.5.15-6.ppc
#

We update the package now with the command updtvpkg (update virtual package):

# updtvpkg
Please wait...
# 
# rpm -q AIX-rpm
AIX-rpm-7.1.5.30-7.ppc
#

The version number has now changed from 7.1.5.15-6 to 7.1.5.30-7! We can again display which libraries are provided by AIX via AIX rpm:

# rpm -q --provides AIX-rpm | egrep "lib(ssl|crypto)"
libcrypto.a(libcrypto.so)
libcrypto.a(libcrypto.so.0.9.8)
libcrypto.a(libcrypto.so.1.0.0)
libcrypto.a(libcrypto.so.1.0.2)
libcrypto.a(libcrypto64.so)
libcrypto.a(libcrypto64.so.0.9.8)
libcrypto.a(libcrypto64.so.1.0.0)
libcrypto.so
libcrypto.so.0.9.7
libcrypto.so.1.0.0
libcrypto_compat.a(libcrypto.so)
libcrypto_compat.a(libcrypto.so.0.9.8)
libcrypto_compat.a(libcrypto64.so)
libcrypto_compat.a(libcrypto64.so.0.9.8)
libssl.a(libssl.so)
libssl.a(libssl.so.0.9.8)
libssl.a(libssl.so.1.0.0)
libssl.a(libssl.so.1.0.2)
libssl.a(libssl64.so)
libssl.a(libssl64.so.0.9.8)
libssl.a(libssl64.so.1.0.0)
libssl.so
libssl.so.1.0.0
libssl3.a(libssl3.so)
libssl3.so
libssl_compat.a(libssl.so)
libssl_compat.a(libssl.so.0.9.8)
libssl_compat.a(libssl64.so)
libssl_compat.a(libssl64.so.0.9.8)
#

After the update of the virtual package AIX-rpm, the RPM database now also contains the version 1.0.2 of libssl.so and libcrypto.so.

The installation of Python3 is now successful:

# yum install python3
...
Is this ok [y/N]: y
...
Installed:
  python3.ppc 0:3.7.1-1                                                                                       

Dependency Installed:
  bzip2.ppc 0:1.0.6-3  expat.ppc 0:2.2.4-1  info.ppc 0:6.4-1  libffi.ppc 0:3.2.1-3  libstdc++.ppc 0:6.3.0-2
  ncurses.ppc 0:6.1-2

Dependency Updated:
  readline.ppc 0:7.0-5                                                                                        

Complete!
#

Therefore, do not forget to update AIX-rpm when updating BFF packages!

Building your own AIX installp packages (Part 1)

This blog post will show you how to build installp packages on AIX. The use of the command mkinstallp is demonstrated by a simple example. More complex packages will be covered in a later article.

It is relatively easy to build installp packages under AIX. This requires the command /usr/sbin/mkinstallp. If the command is not installed, the package bos.adt.insttools must be installed. Building a package requires root privileges.

First of all, we create a directory somewhere, in which the package is to be built:

# mkdir pwrcmps.installp.example
#

The package should contain a small shell script:

# cat <<EOF >hello
#! /bin/ksh
print "hello world"
EOF
# chmod a+x hello
#

The script will later be installed under /usr/local/bin. Files to be installed must be in the same place relative to the build directory (pwrcmps.installp.example in our case), as later relative to the root directory. We therefore create the necessary directory structure and copy the hello script to the appropriate location:

# mkdir –p pwrcmps.installp.example/usr/local/bin
# cp hello pwrcmps.installp.example/usr/local/bin
#

We change to the build directory and start the command mkinstallp to build the package:

# cd pwrcmps.installp.example
# mkinstallp
Using /export/src/installp/pwrcmps.installp.example as the base package directory.
Cannot find /export/src/installp/pwrcmps.installp.example/.info. Attempting to create.
Using /export/src/installp/pwrcmps.installp.example/.info to store package control files.
Cleaning intermediate files from /export/src/installp/pwrcmps.installp.example/.info.

************************************************************
|            Beginning interactive package input           |
|   * - required; () - example value; [] - default value   |
************************************************************

* Package Name (xyz.net) []: pwrcmps.installp
* Package VRMF (1.0.0.0) []: 1.0.0.0
Update (Y/N) [N]:
Number of filesets in pwrcmps.installp (1) [1]:

Gathering info for new fileset (1 remaining)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
* Fileset Name (pwrcmps.installp.rte) []: pwrcmps.installp.example
* Fileset VRMF (1.0.0.0) []: 1.0.0.0
* Fileset Description (some text) []: Example Fileset
Do you want to include a copyright file for this fileset? (Y/N) [N]:
Entering information for the USER part liblpp files
Do you want to include an installp pre_i/u script for this fileset? (Y/N) [N]:
Do you want to include an installp post_i/u script for this fileset? (Y/N) [N]:
Do you want to include a pre-deinstall Script for this fileset? (Y/N) [N]:
Do you want to include an installp pre_rm script for this fileset? (Y/N) [N]:
Do you want to include an installp config script for this fileset? (Y/N) [N]:
Would you like to create ROOT part? (Y/N) [N]:
Bosboot required (Y/N) [N]:
License agreement acceptance required (Y/N) [N]:
Include license files for pwrcmps.installp.example in this package (Y/N) [N]:
Do you want to specify Requisites using a file for this fileset? (Y/N) [N]:
Number of Requisites for pwrcmps.installp.example (1) [0]:
Number of filesystems requiring additional space for pwrcmps.installp.example [0]:

You should include any directories that you are creating in the file count.
(ie: For /usr/proj/myFile, enter 2; 1 for /usr/proj and 1 for /usr/proj/myFile)
Number of USR part files in pwrcmps.installp.example (1) [0]: 1
* 1 of 1. Directory or File Path (/usr/proj/myFile) []: /usr/local/bin/hello
Do you want to make this fileset relocatable? (Y/N) [N]:
Do you want to include an override inventory file for this fileset? (Y/N) [N]:
Do you want to add WPAR attributes to your fileset? (Y/N) [N]:


Using /export/src/installp/pwrcmps.installp.example/.info/pwrcmps.installp.template as the template file.
pwrcmps.installp 1.0.0.0 I
processing pwrcmps.installp.example
creating ./.info/liblpp.a
creating ./tmp/pwrcmps.installp.1.0.0.0.bff
#

The built end product can be found in the subdirectory tmp:

# ls –l tmp
total 8
-rw-r--r--    1 root     system         2560 Sep 25 09:49 pwrcmps.installp.1.0.0.0.bff
#

When creating the package, we have always confirmed the default values for simplicity. The product name is pwrcmps.installp and the file set name is pwrcmps.installp.example, each with version 1.0.0.0. All files and directories of the package must be listed explicitly with absolute path! This can be a bit impractical for a few hundred paths, but it can be simplified and automated by using scripts.

We install the newly generated package once, and check if our shell script is also installed:

# installp -ad tmp/pwrcmps.installp.1.0.0.0.bff all
+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.

  Selected Filesets
  -----------------
  pwrcmps.installp.example 1.0.0.0            # Example Fileset

  << End of Success Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    1  Selected to be installed, of which:
        1  Passed pre-installation verification
 ----
    1  Total to be installed

+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+

installp:  APPLYING software for:
        pwrcmps.installp.example 1.0.0.0

Finished processing all filesets.  (Total time:  1 secs).

+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
pwrcmps.installp.example    1.0.0.0         USR         APPLY       SUCCESS   
# which hello
/usr/local/bin/hello
# hello
hello world
#

We are now extending our script by printing the message “hello world, how are you“.

# vi usr/local/bin/hello
…
print "hello world, how are you"
#

The package should now be rebuilt, before we want to increase the version to 1.1.0.0. Of course, we could start the command mkinstallp interactively again and enter all the necessary information again, but this is very time consuming and not necessary. The command mkinstallp supports the specification of a template file with all necessary information. When building the first version of our package, such a template file was generated under .info/pwrcmps.installp.template. We copy this template file directly into the build root directory and rename it to template:

# cp .info/pwrcmps.installp.template template
# cat template
Package Name: pwrcmps.installp
Package VRMF: 1.0.0.0
Update: N
Fileset
  Fileset Name: pwrcmps.installp.example
  Fileset VRMF: 1.0.0.0
  Fileset Description: Example Fileset
  USRLIBLPPFiles
  EOUSRLIBLPPFiles
  Bosboot required: N
  License agreement acceptance required: N
  Include license files in this package: N
  Requisites:
  USRFiles
    /usr/local/bin/hello
  EOUSRFiles
  ROOT Part: N
  ROOTFiles
  EOROOTFiles
  Relocatable: N
EOFileset
#

The template file contains the information that we gave interactively. In this template file we change the version number from 1.0.0.0 to 1.1.0.0:

# vi template
…
Package VRMF: 1.1.0.0
…
  Fileset VRMF: 1.1.0.0
…
#

We now try to build the version 1.1.0.0 by starting mkinstallp with the option -T (for template) and the template file template:

# mkinstallp -T template
Using /export/src/installp/pwrcmps.installp.example as the base package directory.
Using /export/src/installp/pwrcmps.installp.example/.info to store package control files.
Cleaning intermediate files from /export/src/installp/pwrcmps.installp.example/.info.
0503-844 mkinstallp: Cannot overwrite existing
        /export/src/installp/pwrcmps.installp.example/.info/pwrcmps.installp.template file.
#

An error message appears, the template file under .info can not be overwritten. The .info directory is created again and again during the build process and can therefore simply be deleted before the next build process is started:

# rm -rf .info
# mkinstallp -T template
Using /export/src/installp/pwrcmps.installp.example as the base package directory.
Cannot find /export/src/installp/pwrcmps.installp.example/.info. Attempting to create.
Using /export/src/installp/pwrcmps.installp.example/.info to store package control files.
Cleaning intermediate files from /export/src/installp/pwrcmps.installp.example/.info.

Using template as the template file.
0503-880 mkinstallp: This file /usr/local/bin/hello
        already exists as a system file.
pwrcmps.installp 1.1.0.0 I
processing pwrcmps.installp.example
creating ./.info/liblpp.a
creating ./tmp/pwrcmps.installp.1.1.0.0.bff
#

The new version of the package can be found under tmp as before:

# ls -l tmp
total 16
-rw-r--r--    1 root     system         2560 Sep 25 10:08 pwrcmps.installp.1.0.0.0.bff
-rw-r--r--    1 root     system         2560 Sep 25 10:35 pwrcmps.installp.1.1.0.0.bff
#

This makes it easy to make changes and then package them.