Replacing failed / missing PV on rootvg AIX

Replacing a failed PV when the configuration does not allow the use of the replacepv command (for hdisk in rootvg)

AIX boot process:
http://praveen4aix.blogspot.com/2009/08/boot-process-in-aix.html

Assume that the failed physical volume, hdisk0, and its mirror, hdisk1, are part of the yourvg volume group.

1. To remove mirror copies from the failed PV, run the following command:

   unmirrorvg yourvg hdisk0

2. If the PV failure occurred on rootvg, remove hdisk0 from the boot list by running the following command:
   Note: If your configuration uses boot devices other than hdisk0 and hdisk1, add them to the command syntax.

   bootlist -om normal hdisk1

   This step requires that hdisk1 remains a bootable device in rootvg. After completing this step, ensure that hdisk0 does not appear in output.
3. If the PV failure occurred on rootvg, recreate any dedicated dump devices from the failed PV.
   If you have a dedicated dump device that was on the failed PV, you can use the mklv command to create a new logical volume on an existing PV. Use the sysdumpdev command to set the new logical volume as the primary dump device.
4. To undefine the failed PV, run the following command:
   Note: Removing the disk device entry will also remove the /dev/ipldevice hard link if the failed PV is the PV used to boot the system.

   reducevg yourvg hdisk0
   rmdev -dl hdisk0 --- Do this step in cooperation with IBM support, technician

   needs to prepare disk for hot removal. 

5. If the failed PV is the most recently used boot device, recreate the /dev/ipldevice hard link that was removed in Step 4 by running the following command:

   ln /dev/rhdisk1 /dev/ipldevice

   Note the r prefixed to the PV name.
   To verify that your /dev/ipldevice hard link has been recreated, run the following command:
   

   ls /dev/ipldevice

6. Replace the failed disk.
7. To define the new PV, run the following command:

   cfgmgr

   The cfgmgr command assigns a PV name to the replacement PV. The assigned PV name is likely to be the same as the PV name previously assigned to the failed PV. In this example, assume that the device hdisk0 is assigned to the replacement PV.
8. To add the new PV to the volume group, run the following command:

   extendvg yourvg hdisk0

   You might encounter the following error message:

   0516-050 Not enough descriptor space left in this volume group.
   Either try adding a smaller PV or use another volume group.

   If you encounter this error and cannot add the PV to the volume group, you can try to mirror logical volumes to another PV that already exists in the volume group or add a smaller PV. If neither option is possible, you can try to bypass this limitation by upgrading the volume group to a Big-type or Scalable-type volume group using the chvg command.
9. Mirror the volume group.
   Note: The mirrorvg command cannot be used if all of the following conditions exist:
   

   • The target system is a logical partition (LPAR).
   • A copy of the boot logical volume (by default, hd5) resides on the failed PV.
   • The replacement PV's adapter was dynamically configured into the LPAR since the last cold boot.

   If all of the above conditions exist, use the mklvcopy command to recreate mirror copies for each logical volume as follows:
   

   1. Create copies of the boot logical volume to ensure that it is allocated to a contiguous series of physical partitions.
   2. Create copies of the remaining logical volumes, and synchronize the copies using the syncvg command.
   3. Make the disk bootable by shutting down the LPAR and activating it instead of rebooting using the shutdown or reboot commands. This shutdown does not have to be done immediately, but it is necessary for the system to boot from the new PV.
   Otherwise, create new copies of logical volumes in the volume group using the new PV with the following command:
   

   Note: The mirrorvg command disables quorum by default. For rootvg, you will want to use the -m option to ensure that the new logical volume copies are mapped to hdisk0 in the same way as the working disk.

   mirrorvg yourvg hdisk0

10. If your configuration holds third copies of some logical volumes, you might need to recreate those copies with the following command:

    mklvcopy -k

11. If the PV failure occurred on rootvg, initialize the boot record by running the following command:

    bosboot -a

12. If the PV failure occurred on rootvg, update the boot list by running the following command:
    Note: If your configuration uses boot devices other than hdisk0 and hdisk1, add them to the command.

    bootlist -om normal hdisk0 hdisk1

13. Verify that the procedure was successful.
    • To verify that all logical volumes are mirrored to the new PV, run the following command:

      lslv lvname

      Check the COPIES attribute of each logical volume affected by the failed PV to ensure that the desired number of copies now exist. If the number of copies of the logical volume is below the desired number, use the mklvcopy command to create additional copies.
    • To verify that all the logical volume partitions are synchronized, check that there are no stale partitions by running the following command:

      lspv hdisk0

      Check the STALE PARTITIONS attribute of the replaced PV to ensure that the count is zero. If there are stale partitions use the syncvg command to synchronize the partitions.
    If the PV failure occurred on rootvg, use the following steps to verify other aspects of this procedure:
    • To verify the boot list, run the following command:

      bootlist -om normal

    • To verify the dump device, run the following command:

      sysdumpdev -l

    • To verify the list of bootable PVs, run the following command:

      ipl_varyon -i

    • To verify the /dev/ipl_device, run the following command:

      ls -i /dev/rhdisk1 /dev/ipldevice

      Ensure the output of the ls command has the same i-node number for both entries.
    This step completes the procedure.

For more information

• AIX Logical Volume Manager from A to Z: Introduction and Concepts, available at the Redbooks® Web site at www.redbooks.ibm.com.
• AIX Storage Management, available at the Redbooks Web site at www.redbooks.ibm.com.