Many IT environments operate their systems in more than one data center. In order not to have any data loss in the event of a failure of an entire data center, the data is mirrored between 2 or more data centers. The mirroring can be realized by the storage (storage based mirroring) or by a volume manager (LVM in the case of AIX) on the server (host based mirroring). In the following we only consider mirroring using the AIX Logical Volume Manager. The aim is to show how the correct mirroring of logical volumes can be implemented with the help of mirror pools. In larger environments with many physical volumes, maintaining correct mirroring without mirror pools is difficult and a challenge for the administrator.
1. Our Test Environment
In our test environment we have created a scalable volume group with 4 physical volumes:
$ lsvg -p datavg01 datavg01: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk4 active 39 39 08..08..07..08..08 hdisk5 active 39 39 08..08..07..08..08 hdisk8 active 39 39 08..08..07..08..08 hdisk9 active 39 39 08..08..07..08..08 $
The 4 physical volumes come from 2 different data centers, hdisk4 and hdisk5 come from the data center DC1 and hdisk8 and hdisk9 from the data center DC2.
2. The Problem
If a mirrored logical volume is created without a mapping file and without specifying the physical volumes to be used, then LVM allocates space on the physical volumes in the order in which the physical volumes are listed above.
# mklv -c 2 datavg01 10 lv00 #
Since hdisk4 and hdisk5 are the first physical volumes, these are used for the mirrored LV:
$ lslv -m lv00 lv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0009 hdisk4 0009 hdisk5 0002 0010 hdisk4 0010 hdisk5 0003 0011 hdisk4 0011 hdisk5 0004 0012 hdisk4 0012 hdisk5 0005 0013 hdisk4 0013 hdisk5 0006 0014 hdisk4 0014 hdisk5 0007 0015 hdisk4 0015 hdisk5 0008 0016 hdisk4 0016 hdisk5 0009 0017 hdisk4 0017 hdisk5 0010 0018 hdisk4 0018 hdisk5 $
Both mirror copies are now in the DC1 data center! If the data center fails, the entire mirrored LV fails!
This can of course easily be done better by specifying the desired physical volumes when creating the LV:
# mklv -c 2 -t jfs2 datavg01 10 hdisk4 hdisk8 fslv00 #
Now the LV is correctly mirrored between the two data centers:
$ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 $
However, the administrator must know for each physical volume which data center it belongs to. This isn’t always easy to see.
What about expanding a LV?
To do this, we first investigate how much free space is available on which physical volumes:
$ lsvg -p datavg01 datavg01: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk4 active 39 19 08..00..00..03..08 hdisk5 active 39 29 08..00..05..08..08 hdisk8 active 39 29 08..00..05..08..08 hdisk9 active 39 39 08..08..07..08..08 $
We extend the second LV (fslv00) by 20 LPs, initially without specifying physical volumes:
# extendlv fslv00 20 #
The mapping for the extended LV then looks like this:
$ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 0011 0001 hdisk4 0009 hdisk9 0012 0002 hdisk4 0010 hdisk9 0013 0003 hdisk4 0011 hdisk9 0014 0004 hdisk4 0012 hdisk9 0015 0005 hdisk4 0013 hdisk9 0016 0006 hdisk4 0014 hdisk9 0017 0007 hdisk4 0015 hdisk9 0018 0008 hdisk4 0016 hdisk9 0019 0029 hdisk4 0017 hdisk9 0020 0030 hdisk4 0018 hdisk9 0021 0031 hdisk4 0019 hdisk9 0022 0032 hdisk4 0020 hdisk9 0023 0033 hdisk4 0021 hdisk9 0024 0034 hdisk4 0022 hdisk9 0025 0035 hdisk4 0023 hdisk9 0026 0036 hdisk4 0001 hdisk9 0027 0037 hdisk4 0002 hdisk9 0028 0038 hdisk4 0003 hdisk9 0029 0039 hdisk4 0004 hdisk9 0030 0019 hdisk8 0005 hdisk9 $
The first mirror copy is now on hdisk4 and hdisk8 and is thus distributed over 2 data centers. The second mirror copy is on hdisk8 and hdisk9 and therefore entirely in the second data center. If the second data center fails, access to LP (Logical Partition) 30 is lost!
Again, by specifying the desired physical volumes when extendeding the LV should correctly mirror across the two data centers. For this we have deleted the LV fslv00 and created it again as originally:
# rmlv -f fslv00 rmlv: Logical volume fslv00 is removed. # mklv -c 2 -t jfs2 datavg01 10 hdisk4 hdisk8 fslv00 #
We extend the LV again and explicitly specify hdisk5 (DC1) and hdisk9 (DC2) as physical volumes for the expansion of the LV. There are enough free PPs (physical partitions) available on both physical volumes:
# extendlv fslv00 20 hdisk5 hdisk9 #
A quick look at the resulting mapping unfortunately shows that this did not had the desired effect:
$ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 0011 0009 hdisk9 0019 hdisk5 0012 0010 hdisk9 0020 hdisk5 0013 0011 hdisk9 0021 hdisk5 0014 0012 hdisk9 0022 hdisk5 0015 0013 hdisk9 0023 hdisk5 0016 0014 hdisk9 0001 hdisk5 0017 0015 hdisk9 0002 hdisk5 0018 0016 hdisk9 0003 hdisk5 0019 0017 hdisk9 0004 hdisk5 0020 0018 hdisk9 0005 hdisk5 0021 0019 hdisk9 0006 hdisk5 0022 0020 hdisk9 0007 hdisk5 0023 0021 hdisk9 0008 hdisk5 0024 0022 hdisk9 0024 hdisk5 0025 0023 hdisk9 0025 hdisk5 0026 0001 hdisk9 0026 hdisk5 0027 0002 hdisk9 0027 hdisk5 0028 0003 hdisk9 0028 hdisk5 0029 0004 hdisk9 0029 hdisk5 0030 0005 hdisk9 0030 hdisk5 $
Unfortunately the hdisk5 was not used for the first copy but for the second copy as expected and the hdisk9 was used for the second copy rather than the first copy. This means that both copies are distributed across both data centers! A failure of one of the two data centers would have fatal effects on the availability of the LV.
But that means that even in such a simple case you have to work with a mapping file in order to achieve correct mirroring between the data centers. For the sake of completeness, we have deleted the LV and created it again. We created the following mapping file for the extension:
$ cat mapfile hdisk5:19-38 hdisk9:19-38 $
If you now extend the LV with the mapping file, the correct mirroring between the data centers is retained:
# extendlv -m mapfile fslv00 20 # $ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 0011 0019 hdisk5 0019 hdisk9 0012 0020 hdisk5 0020 hdisk9 0013 0021 hdisk5 0021 hdisk9 0014 0022 hdisk5 0022 hdisk9 0015 0023 hdisk5 0023 hdisk9 0016 0024 hdisk5 0024 hdisk9 0017 0025 hdisk5 0025 hdisk9 0018 0026 hdisk5 0026 hdisk9 0019 0027 hdisk5 0027 hdisk9 0020 0028 hdisk5 0028 hdisk9 0021 0029 hdisk5 0029 hdisk9 0022 0030 hdisk5 0030 hdisk9 0023 0031 hdisk5 0031 hdisk9 0024 0032 hdisk5 0032 hdisk9 0025 0033 hdisk5 0033 hdisk9 0026 0034 hdisk5 0034 hdisk9 0027 0035 hdisk5 0035 hdisk9 0028 0036 hdisk5 0036 hdisk9 0029 0037 hdisk5 0037 hdisk9 0030 0038 hdisk5 0038 hdisk9 $
As expected, the mirror copy 1 is on hdisk4 and hdisk5 and thus in the data center DC1 and the copy 2 on hdisk8 and hdisk9 and thus in DC2. However, it is very time-consuming to create a mapping file for each extension.
Finally, let’s look at the case of a filesystem extension. To do this, we delete the LV again and recreate it and then create a file system on the LV:
# mklv -c 2 -t jfs2 datavg01 10 hdisk4 hdisk8 fslv00 # crfs -v jfs2 -d /dev/fslv00 -m /fs00 -a log=INLINE File system created successfully. 2610916 kilobytes total disk space. New File System size is 5242880 # mount /fs00 #
We now extend the mounted file system by 5 GB:
# chfs -a size=+5G /fs00 Filesystem size changed to 15728640 Inlinelog size changed to 30 MB. #
Neither physical volumes nor a mapping file can be specified with the chfs command. This means that it is not possible to specifically determine which physical volumes are used for the extension when expanding the file system. Accordingly, after the file system expansion, both mirror copies are again located in both data centers:
$ lslv -m fslv00 fslv00:/fs00 LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 0011 0001 hdisk4 0009 hdisk9 0012 0002 hdisk4 0010 hdisk9 0013 0003 hdisk4 0011 hdisk9 0014 0004 hdisk4 0012 hdisk9 0015 0005 hdisk4 0013 hdisk9 0016 0006 hdisk4 0014 hdisk9 0017 0007 hdisk4 0015 hdisk9 0018 0008 hdisk4 0016 hdisk9 0019 0029 hdisk4 0017 hdisk9 0020 0030 hdisk4 0018 hdisk9 0021 0031 hdisk4 0019 hdisk9 0022 0032 hdisk4 0020 hdisk9 0023 0033 hdisk4 0021 hdisk9 0024 0034 hdisk4 0022 hdisk9 0025 0035 hdisk4 0023 hdisk9 0026 0036 hdisk4 0001 hdisk9 0027 0037 hdisk4 0002 hdisk9 0028 0038 hdisk4 0003 hdisk9 0029 0039 hdisk4 0004 hdisk9 0030 0019 hdisk8 0005 hdisk9 $
As a workaround, you can first use extendlv with a mapping file and then run chfs for the file system extension.
Conclusion: Correct mirroring between data centers is very difficult to achieve without mirror pools. Even simple LVM operations can lead to surprising results, as some of the examples have shown.
3. Using Mirror Pools
The idea with mirror pools is to organize the physical volumes in pools and then to assign the mirror copies to these pools. Mirror pools can only be configured for scalable VGs.
First of all, the physical volumes must be assigned to the mirror pools. In our case, the physical volumes come from the two data centers DC1 and DC2, which suggests creating 2 mirror pools, one for the physical volumes from DC1 and one for the physical volumes from DC2.
# chpv -p DC1 hdisk4 # chpv -p DC1 hdisk5 # chpv -p DC2 hdisk8 # chpv -p DC2 hdisk9 #
Which physical volume belongs to which mirror pool can easily be found out using the ‘-P‘ option with the lsvg command:
# lsvg -P datavg01 Physical Volume Mirror Pool hdisk4 DC1 hdisk5 DC1 hdisk8 DC2 hdisk9 DC2 #
A mirror pool always belongs to a VG. Each VG has mirror pools that are independent of other VGs.
Whether and how mirror pools are used in a VG depends on the mirror pool strictness. The mirror pool strictness is an attribute of the VG and defines for the entire VG how mirror pools are to be used. The attribute can be set using the ‘-M‘ option when creating a VG with mkvg or subsequently with chvg. The possible values are the following:
n – No, Mirror pools can be used, but do not have to be used (default) y – Yes, Mirror pools must be used; a mirror pool must be assigned to each mirror copy s – Super-strict: Mirror pools must be used; each mirror pool must have at least one mirror copy of each LV.
By default, the mirror pool strictness of a VG is set to ‘n‘. It is therefore not required to use mirror pools.
When creating an LV, you can assign a mirror pool to each mirror copy:
# mklv -c 2 -t jfs2 -p copy1=DC1 -p copy2=DC2 datavg01 10 fslv00 #
The mirror copies are now in the corresponding mirror pools as specified:
$ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 $
An extension of the LV, with extendlv or chfs, is now carried out for each mirror copy only by using physical volumes of the associated mirror pool:
$ lslv -m fslv00 fslv00:N/A LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0019 hdisk4 0009 hdisk8 0002 0020 hdisk4 0010 hdisk8 0003 0021 hdisk4 0011 hdisk8 0004 0022 hdisk4 0012 hdisk8 0005 0023 hdisk4 0013 hdisk8 0006 0024 hdisk4 0014 hdisk8 0007 0025 hdisk4 0015 hdisk8 0008 0026 hdisk4 0016 hdisk8 0009 0027 hdisk4 0017 hdisk8 0010 0028 hdisk4 0018 hdisk8 0011 0001 hdisk4 0019 hdisk8 0012 0002 hdisk4 0020 hdisk8 0013 0003 hdisk4 0021 hdisk8 0014 0004 hdisk4 0022 hdisk8 0015 0005 hdisk4 0023 hdisk8 0016 0006 hdisk4 0001 hdisk8 0017 0007 hdisk4 0002 hdisk8 0018 0008 hdisk4 0003 hdisk8 0019 0029 hdisk4 0004 hdisk8 0020 0030 hdisk4 0005 hdisk8 0021 0031 hdisk4 0006 hdisk8 0022 0032 hdisk4 0007 hdisk8 0023 0033 hdisk4 0008 hdisk8 0024 0034 hdisk4 0024 hdisk8 0025 0035 hdisk4 0025 hdisk8 0026 0036 hdisk4 0026 hdisk8 0027 0037 hdisk4 0027 hdisk8 0028 0038 hdisk4 0028 hdisk8 0029 0039 hdisk4 0029 hdisk8 0030 0019 hdisk5 0030 hdisk8 $
Since intra and inter physical volume allocation policies are still taken into account, this can lead to different mappings for the mirror copies (mirror copy 1 uses hdisk4 and hdisk5 and thus 2 physical volumes, mirror copy 2 only uses hdisk8 and therefore only 1 physical volume), but physical volumes from 2 different data centers are no longer used within a mirror copy! By specifying the desired physical volumes, you can of course achieve that both mirror copies use PPs from specific physical volumes and thus both mirror copies use 2 physical volumes:
# extendlv fslv00 20 hdisk5 hdisk9 #
If physical volumes and mirror copies are assigned to mirror pools, it is very easy to maintain correct mirroring between data centers.
See also:
Mirror Pools: Understanding Mirror Pool Strictness
Michael Perzl: Introduction to AIX Mirror Pools
AIX LVM: Mechanics of migratepv (Part I)
AIX LVM: Mechanics of migratepv (Part II)