AIX LVM: Mechanics of migratepv (Part III)

The AIX LVM command migratepv can be used to exchange physical volumes in mirror copies of logical volumes. The third part is about the correction of “wrong” mirroring. The possibilities that arise are discussed. The possibilities shown are demonstrated using a larger example from practice.

In practice it occasionally happens that mirrored logical volumes are “incorrectly” mirrored. In the following, “wrong” mirroring is understood to mean that a mirror copy that should only contain physical volumes from one data center also contains physical volumes from another data center. This can happen relatively quickly if no mirror pools are used and logical volumes are extended frequently.

In this article we only consider the case of mirrored logical volumes with 2 mirror copies. In all cases considered, each logical partition has copies in both data centers. In particular, this means that if a data center fails, a valid copy is still available for each logical partition. The aim is to correct this in all of the following examples so that in the end each mirror copy only contains physical volumes of one data center. When correcting the mirroring, it is ensured that copies are always available in both data centers and thus redundancy is provided in the event of failure of one of the two data centers.

The Target State

Only mirrored logical volumes with 2 mirror copies are considered. The mirror copy 1 should be completely on physical volumes from the data center DC01 (green). The mirror copy 2 should lie completely on physical volumes from the data center DC02 (red).

Case Distinction

When examining the way migratepv works, we saw that it depends on which of the two mirror copies contains the physical volume to be replaced. The highest logical partition number in which the physical volume occurs is decisive. We therefore differentiate as follows:

- Case A: The highest occurrence of the physical volume is in the “correct” mirror copy.
- Case B: The highest occurrence of the physical volume is in the “wrong” mirror copy.

Case A: Highest Occurrence in “correct” Mirror Copy

In this case, the physical volume to be replaced must appear at least twice in the logical volume, first in the “wrong” mirror copy and last in the “correct” mirror copy. In the following we briefly demonstrate the two possible subcases:

- Case A1: The physical volume to be replaced appears last in mirror copy 1.
- Case A2: The physical volume to be replaced appears last in mirror copy 2.

Case A1: Highest Occurrence in “correct” Mirror Copy 1

The physical volume to be replaced, hdisk11 (DC01), occurs 4 times in the logical volume, 2 times in both mirror copies. The highest occurrence of hdisk11 (DC01) in a logical partition is in mirror copy 1. The replacement therefore takes place in mirror copy 1. The target physical volume must therefore also come from the data center DC01.

We use the physical volume hdisk12 (DC01) as the target physical volume:

# migratepv -l lv00 hdisk11 hdisk12

The logical volume is now correctly mirrored. The target physical volume hdisk12 (DC01) is only used in mirror copy 1.

Case A2: Highest Occurrence in “correct” Mirror Copy 2

The physical volume to be replaced, hdisk14 (DC02), occurs 4 times in the logical volume, 2 times in both mirror copies. The highest occurrence of hdisk14 (DC02) in a logical partition is in mirror copy 2. The replacement therefore takes place in mirror copy 2. The target physical volume must therefore be from the data center DC02.

We use the physical volume hdisk15 (DC02) as the target physical volume:

# migratepv -l lv00 hdisk14 hdisk15

The logical volume is now correctly mirrored. The target physical volume hdisk15 (DC02) is only used in mirror copy 2.

Case B: Highest Occurrence in “wrong” Mirror Copy

This case is much more complex. Simply replacing the physical volume hdiskY with another physical volume does not lead to correct mirroring! The case also includes the situation in which the physical volume hdiskY only occurs once, but in the “wrong” mirror copy. There are two subcases:

- Case B1: The physical volume to be replaced appears last in mirror copy 1.
- Case B2: The physical volume to be replaced appears last in mirror copy 2.

We discuss case B1 in detail. The same applies to case B2.

Case B1: Highest Occurrence in “wrong” Mirror Copy 1

The physical volume to be replaced, hdisk14 (DC02), occurs 4 times in the logical volume, 2 times in both mirror copies. The highest occurrence of hdisk14 (DC02) in a logical partition is in mirror copy 1. If you were to exchange hdisk14 (DC02) for a physical volume from DC01, both mirror copies would be in DC01. If the DC01 data center were to fail, both copies would be lost. If you exchange hdisk14 (DC02) by a physical volume from DC02, the two copies would be distributed across both data centers, but mirror copy 1 would be in DC02 (now for all 4 occurrences of hdisk14) and mirror copy 2 in DC01. The situation has become more worse.

In order to be able to correct the mirroring, one has to convert case B to case A. The physical volume hdisk14 (DC02) must appear last in mirror copy 2, then you would be in case A2. This could be achieved e.g. by copying a physical partition of hdisk13 (after the last occurrence of hdisk14) to hdisk14. Requirements are:

- There is at least one free physical partition on hdisk14.
- After the last occurrence of hdisk14 there is at least one logical partition that is correctly mirrored between the data centers.

We move the last physical partition of mirror copy 2 from hdisk13 (DC02) to hdisk14 (DC02).

# migratelp lv00/45/2 hdisk14

So case B1 is reduced to case A2!

As in case A2 above, we now exchange hdisk14 (DC02) by hdisk15 (DC02):

# migratepv -l lv00 hdisk14 hdisk15

The logical volume is now correctly mirrored between the two data centers! We move the physical partition that was moved in the first step back to the original physical volume hdisk13 (DC02):

# migratelp lv00/45/2 hdisk13/25

The following steps were necessary for case B1:

1. Moving a physical partition in mirror copy 2 to the source physical volume, with the aim of having the highest logical partition containing the source physical volume in mirror copy 2. With which the case is then reduced to case A2.
2. Exchanging the source physical volume for the target physical volume, with the target physical volume then only occurring in mirror copy 2.
3. Optional: Undo step 1 by moving the physical partition back to the original physical volume.

Case B1: No free Physical Partition on Source Physical Volume

In order to reduce case B1 to case A2 using migratelp, a free physical partition was necessary on the source physical volume (hdisk14). In order to gain a free physical partition on the source physical volume, one of the physical partitions could be moved to another physical volume from the same data center (migratelp). A physical partition from mirror copy 2 should be selected!

Case B1: There is no Physical Partition that can be moved

For the first step with migratelp it was necessary to find a logical partition after the last occurrence of the source physical volume (in our case hdisk14), which is correctly mirrored between the two data centers. If there is no such logical partition, e.g. because the source physical volume occurs in the highest logical partition, case B1 cannot be reduced to case A2 with a simple migratelp. If extending the logical volume is an option, extendlv could be used to add at least one logical partition, whereby first a physical volume from the data center DC01 and second the source physical volume from the data center DC02 must be specified.

We extend our starting logical volume (with incorrect mirroring) using extendlv:

# extendlv lv00 1 hdisk10 hdisk14

The following steps were necessary for case B1:

1. Extending the logical volume to a physical volume from data center DC01 (in mirror copy 1) and the source physical volume (in mirror copy 2).
2. Replacing the source physical volume by the target physical volume. The target physical volume then only appears in mirror copy 2.

Case B1: The Logical Volume cannot/must not be increased

If it is not possible to extend the logical volume, there is still the option of swapping the two mirror copies in the highest logical partition in which the source physical volume occurs. To do this, proceed as follows:

1. Copying the physical partition from mirror copy 1 to another physical volume (temporarily).
2. Migrate the physical partition from mirror copy 2 to the physical partition in mirror copy 1 that became free in step 1.
3. Migrate the temporary physical partition to the freed physical partition of mirror copy 2.

Disadvantage of the process: During the migrate operations, the logical partition is temporarily only in one data center! The redundancy between the data centers is lost for the duration of the migrate operations.

For our starting logical volume (with incorrect mirroring) the steps would be the following:

1. Migrating from LP 40 mirror copy 1 to the available physical volume hdisk12, this frees the PP 20 on hdisk14:
```
# migratelp lv00/40/1 hdisk12
```
2. Migrate mirror copy 2 of the source physical volume hdisk14, this frees PP 40 on hdisk10:
```
# migratelp lv00/40/2 hdisk14/20
```
3. Migrate the temporary copy from hdisk12 to hdisk10:
```
# migratelp lv00/40/1 hdisk10/40
```

Step 2 reduces case B1 again to case A2!

Case B1: Conclusion

Case B1 can be reduced to case A2 using one of the following options:

1. Moving a physical partition in mirror copy 2 to the source physical volume, with the aim of having the highest logical partition containing the source physical volume in mirror copy 2. With which the case is then reduced to case A2.
2. Extending the logical volume to a physical volume from data center DC01 (in mirror copy 1) and the source physical volume (in mirror copy 2).
3. Swap the mirror copies in the highest logical partition in which the source physical volume occurs. (Temporary loss of redundancy!)

Case B2

Case B2 can be reduced to case A1 in an analogous manner. There are the same cases and possibilities as in case B1!

An Example from Practice

A mirrored logical volume has grown to around 10 TB over time. The logical volume is distributed over 22 physical volumes. Partly, the mirror copies are not correctly distributed between the two computer centers. In the following we show how the mirroring can be corrected online.

The logical volume was mirrored “incorrectly” in 3 places. The physical volumes hdisk38, hdisk39, hdisk59 and hdisk60 appear only once in a mirror copy, but then in the mirror copy of the other data center. This is the case B1 or B2. The physical volumes hdisk42 and hdisk43 appear in both mirror copies, with the last occurrence in each case in the mirror copy of the correct data center. This is the case A1 or A2.

Optional: Introduction of Mirror Pools

In order to prevent “wrong” mirroring in the future, we first introduce mirror pools.

We create the mirror pool DC01 with the physical volumes from the data center DC01 and accordingly the mirror pool DC02.

# chpv -P DC01 hdisk30
…

Then we assign the mirror copies of the logical volume to the two mirror pools:

# chlv -m copy1=DC01 -m copy2=DC02 lv_scc
0516-1812 lchangelv: Warning, existing allocation violates mirror pools.
    Consider reorganizing the logical volume to bring it into compliance.
#

Replacement of hdisk43 (case A2)

We will first look at the simpler case with hdisk42 and hdisk43. You can either replace hdisk42 (DC01) (case A1) or replace hdisk43 (DC02) (case A2). We have decided to replace hdisk43 (DC02) with a new physical volume (hdisk71) of sufficient capacity from the data center DC02:

# chpv -P DC02 hdisk71 # optional
# migratepv -l lv_scc hdisk43 hdisk71
#

Migration of a Physical Partition to hdisk39 (DC02)

Next, let’s look at the case with hdisk38 (DC01) and hdisk39 (DC02). Both are in the “wrong” mirror copy, which brings us to case B. To reduce this to case A, we have to migrate one of the two physical partitions in a logical partition that occurs in the logical volume after the highest occurrence of hdisk38 or hdisk39 to hdisk38 or hdisk39. This logical partition must be correctly mirrored between the data centers!

There are still free physical partitions on both hdisk38 (DC01) and hdisk39 (DC02). We have decided to temporarily migrate mirror copy 2 of logical partition 20427 from hdisk55 (DC02) to hdisk39 (DC02):

# migratelp lv_scc/20427/2 hdisk39
0516-1291 migratelp: Mirror copy 2 of logical partition 20427 of logical volume
   lv_scc migrated to physical partition 8000 of hdisk39.
#

By migrating mirror copy 2 of logical partition 20427, we have reduced case B1 to case A2.

Replacement of hdisk39 (DC02)

The physical volume hdisk39 (DC02) can now be replaced by another physical volume from data center DC02.
For this we have provided a new physical volume hdisk72 with sufficient capacity in the data center DC02:

# chpv -p DC02 hdisk72 # optional
0516-1812 lchangepv: Warning, existing allocation violates mirror pools.
    Consider reorganizing the logical volume to bring it into compliance.
# migratepv -l lv_scc hdisk39 hdisk72
#

Migration of the Physical Partition back to hdisk55 (DC02)

We migrate the temporarily moved physical partition back to hdisk55 (DC02). This step is not absolutely necessary, because the logical partition 20427 is already correctly mirrored between the data centers. But we want to keep the layout of the two mirror copies in the logical volume as similar as possible. Therefore we migrate the mirror copy 2 of the logical partition 20427 back to hdisk55 (DC02) to the original physical partition 1:

# migratelp lv_scc/20427/2 hdisk55/1
0516-1291 migratelp: Mirror copy 2 of logical partition 20427 of logical volume
    lv_scc migrated to physical partition 1 of hdisk55.
#

Now the logical volume is only “incorrectly” mirrored with regard to hdisk59 and hdisk60. The situation is identical to hdisk38 and hdisk39. However, there is an additional complication here: this time there is no free physical partition on either physical volume available.

We consider here hdisk59 (DC01). The case for hdisk60 (DC02) could be used analogously. In the data center DC01 there is the physical volume hdisk27, which has a few free physical partitions. The physical volume hdisk27 is currently not used by our logical volume! We temporarily migrate one physical partition from hdisk59 (DC01) to hdisk27 (DC01):

# migratelp lv_scc/6895/2 hdisk27
0516-1291 migratelp: Mirror copy 2 of logical partition 6895 of logical volume
    lv_scc migrated to physical partition 2036 of hdisk27.
#

We have thus gained a free physical partition on hdisk59 (DC01).

To reduce this to case A1, we have to migrate at least one physical partition of mirror copy 1 to hdisk59 in a logical partition that occurs in the logical volume after the highest occurrence of hdisk59. This logical partition must be correctly mirrored between the data centers! We use the highest logical partition (40117) and migrate the physical partition from mirror copy 1 from hdisk45 (DC01) to hdisk59 (DC01):

# migratelp lv_scc/40117/1 hdisk59
0516-1291 migratelp: Mirror copy 1 of logical partition 40117 of logical volume
    lv_scc migrated to physical partition 1 of hdisk59.
#

We have thus reduced the situation to case A1. The physical volume hdisk59 (DC01) can thus be migrated to another physical volume from the data center DC01. The physical volume hdisk74 (DC01) with sufficient free capacity is available for this purpose:

# migratepv -l lv_scc hdisk59 hdisk74
#

Now only a single logical partition is “incorrectly” mirrored any more! The newly used physical volume hdisk27 (DC01) is in the “wrong” mirror copy 2, which brings us back to case B2. If you copy the mirror copy 1 of the last logical partition to hdisk27 (DC01) you are again at case A1 and hdisk27 can then simply be replaced. We migrate the mirror copy 1 of the logical partition 40117 to hdisk27 (DC01):

# migratelp lv_scc/40117/1 hdisk27
0516-1291 migratelp: Mirror copy 1 of logical partition 40117 of logical volume
    lv_scc migrated to physical partition 2037 of hdisk27.
#

Now the temporarily used physical volume hdisk27 (DC01) can be replaced by hdisk74 (DC01):

# migratepv -l lv_scc hdisk27 hdisk74
#

The logical volume is now correctly mirrored between the two data centers! Nevertheless we migrate the last physical partition back to hdisk45 (DC01):

# migratelp lv_scc/40117/1 hdisk45/6079
0516-1291 migratelp: Mirror copy 1 of logical partition 40117 of logical volume
    lv_scc migrated to physical partition 6079 of hdisk45.
#

Conclusion

The correction of “wrong” mirroring is possible with the help of migratepv and migratelp without loss of redundancy. This requires a good knowledge of how migratepv works. In addition, it must be cleverly selected which physical partitions have to be moved with migratelp in order to reduce case B into case A. With the exception of a few additional physical partitions, only the physical partitions affected by the “wrong” mirroring had to be migrated.