{"action":"created","issue":{"url":"https://api.github.com/repos/rear/rear/issues/572","labels_url":"https://api.github.com/repos/rear/rear/issues/572/labels{/name}","comments_url":"https://api.github.com/repos/rear/rear/issues/572/comments","events_url":"https://api.github.com/repos/rear/rear/issues/572/events","html_url":"https://github.com/rear/rear/issues/572","id":65205404,"number":572,"title":"multipath support/boot from san for ReaR -- i.e. recovery technique","user":{"login":"multipathmaster","id":11719398,"avatar_url":"https://avatars.githubusercontent.com/u/11719398?v=3","gravatar_id":"","url":"https://api.github.com/users/multipathmaster","html_url":"https://github.com/multipathmaster","followers_url":"https://api.github.com/users/multipathmaster/followers","following_url":"https://api.github.com/users/multipathmaster/following{/other_user}","gists_url":"https://api.github.com/users/multipathmaster/gists{/gist_id}","starred_url":"https://api.github.com/users/multipathmaster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/multipathmaster/subscriptions","organizations_url":"https://api.github.com/users/multipathmaster/orgs","repos_url":"https://api.github.com/users/multipathmaster/repos","events_url":"https://api.github.com/users/multipathmaster/events{/privacy}","received_events_url":"https://api.github.com/users/multipathmaster/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rear/rear/labels/enhancement","name":"enhancement","color":"02e10c"},{"url":"https://api.github.com/repos/rear/rear/labels/support","name":"support","color":"02d7e1"}],"state":"open","locked":false,"assignee":{"login":"gdha","id":888633,"avatar_url":"https://avatars.githubusercontent.com/u/888633?v=3","gravatar_id":"","url":"https://api.github.com/users/gdha","html_url":"https://github.com/gdha","followers_url":"https://api.github.com/users/gdha/followers","following_url":"https://api.github.com/users/gdha/following{/other_user}","gists_url":"https://api.github.com/users/gdha/gists{/gist_id}","starred_url":"https://api.github.com/users/gdha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdha/subscriptions","organizations_url":"https://api.github.com/users/gdha/orgs","repos_url":"https://api.github.com/users/gdha/repos","events_url":"https://api.github.com/users/gdha/events{/privacy}","received_events_url":"https://api.github.com/users/gdha/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-03-30T12:57:34Z","updated_at":"2015-05-04T12:49:45Z","closed_at":null,"body":"- ReaR version 1.17.\r\n- RHEL 7.X\r\n- EMC SAN/BOOT FROM SAN \r\n- DELL BLADE w/iDRAC\r\n- Multipath enabled\r\n- Using LVM2 as well (VG's)\r\n\r\nHey gents, I know y'all are really busy, however that being said, I have some information for you guys(as well as a bug, but I will get to the bug at the end of this rough draft instruction manual for restoring to a system that is boot from san as well as using multipath).\r\n\r\nLate on Friday I was able to successfully restore a system that uses boot from SAN, multipath, and LVM2 for RHEL 7.X.\r\n\r\nI have written a script that does some translations over multipath to specific block devices that are pertaining to multipath.  But to start off, I will start this thread from scratch.\r\n\r\n1.  You have a blade that is boot from SAN and attached to EMC/3PAR/IBM XIV/Hitachi storage.\r\n\r\n2.  The rest of the disks that are assigned to the system are also on the above mentioned SAN array.\r\n\r\n3.  Multipath is enabled.\r\n\r\n4.  Using a recovery method of TSM (Tivoli).\r\n\r\n5.  You also have LVM/LVM2 enabled and are using volume groups to control system wide sizes (i.e. size of /var, /opt, /tmp, /root, /home, etc...).\r\n\r\nThe first thing that you will want to do before creating a new recovery image via 'rear mkrescue' will be to run this script, from which I have provided...\r\n\r\n*NOTE* --> You will have to edit this script for your output of multipath -ll and exclude the model from field 5.  You can elect to keep the model name within the contents, however, being an admin/engineer, you should know what your environment is using and how it is utilizing these technologies. \r\n\r\n````\r\n#BEGIN#######\r\n#!/bin/bash\r\n#INITIAL VARIABLES\r\nMLIST=`multipath -ll | grep mpath | awk '{ print $1 }'`\r\nROLIST=`multipath -ll | egrep '(mpath|size|wp)' | awk '{ print $5 }' | grep -v SYMMETRIX` #exclude your model of SAN that will pull with field 5\r\nVGLIST=`vgdisplay | grep \"VG Name\" | awk '{ print $3 }' | sort -rn`\r\nPVLIST=`vgdisplay -v | grep \"PV Name\" | awk '{ print $3 }' | sort -rn`\r\nSDLIST=`multipath -ll | grep sd | awk '{ print $3 }'`\r\n#############\r\n#GUTS########\r\n\r\n        for x in `echo ${MLIST[@]}`; do echo $x >> MLIST.$$; done\r\n\r\n        for x in `echo ${ROLIST[@]}`; do echo $x >> ROLIST.$$; done\r\n\r\n        for x in `echo ${VGLIST[@]}`; do echo $x >> VGLIST.$$; done\r\n\r\n        for x in `echo ${PVLIST[@]}`; do echo $x >> PVLIST.$$; done\r\n\r\n                pr -mts MLIST.$$ ROLIST.$$ >> ROCHECK.$$\r\n\r\n                cat ROCHECK.$$ | grep -v ro >> ROCHECK2.$$\r\n\r\n                        for x in `cat ROCHECK2.$$ | awk '{ print $1 }'`; do multipath -ll $x | grep sd | awk '{ print $3 }' | head -1; done >> SDLIST.$$\r\n\r\n                                #DEBUG#pr -mts ROCHECK2.$$ VGLIST.$$ PVLIST.$$ SDLIST.$$\r\n\r\n                                pr -mts ROCHECK2.$$ VGLIST.$$ PVLIST.$$ SDLIST.$$ > output.out\r\n\r\n                                for x in `cat output.out | awk '{ print $5 }'`; do fdisk -l /dev/$x; done >> output.out\r\n\r\n                                        echo \"#################################################################\" >> output.out\r\n\r\n                                        echo \"/dev/sdX boot device will be /dev/sdX1, the rest will be on /dev/sdX2.  Customer data will be on /dev/sdY1.\" >> output.out\r\n\r\n                                        echo \"Please keep the fact that there are partitions in mind when you attempt to recover this system.\" >> output.out\r\n\r\n                                        echo \"######################CHECK output.out###########################\"\r\n\r\n                                        cat output.out\r\n\r\n                                        echo \"######################END OF output.out##########################\"\r\n\r\n                        rm MLIST.$$ ROLIST.$$ VGLIST.$$ PVLIST.$$ SDLIST.$$ ROCHECK.$$ ROCHECK2.$$\r\n\r\n#FIN########\r\n````\r\nThe above script will put your block devices on a line with the VG that is associated with that disk, as well as the read/write permission (i.e. assuming that you may have gatekeepers that are presented to the blade that would be read only).\r\n\r\nSample output from the above script:\r\n````\r\n[root@XXXXXXXXXXXX tmp]# ./test.sh\r\n    Finding all volume groups\r\n    Finding volume group \"vg00\"\r\n    Finding volume group \"vg01\"\r\n######################CHECK output.out###########################\r\nmpathb  wp=rw   vg01    /dev/mapper/mpathb1     sdb  (VG01 will be mpathb1, or /dev/sdb1)\r\nmpatha  wp=rw   vg00    /dev/mapper/mpatha2     sda  (VG00 will be mpatha2, or /dev/sda2)\r\n\r\nDisk /dev/sdb: 21.6 GB, 21626880000 bytes, 42240000 sectors\r\nUnits = sectors of 1 * 512 = 512 bytes\r\nSector size (logical/physical): 512 bytes / 512 bytes\r\nI/O size (minimum/optimal): 512 bytes / 512 bytes\r\nDisk label type: dos\r\nDisk identifier: 0x000c090f\r\n\r\n   Device Boot      Start         End      Blocks   Id  System\r\n/dev/sdb1            2048    42239999    21118976   8e  Linux LVM\r\n\r\nDisk /dev/sda: 21.6 GB, 21626880000 bytes, 42240000 sectors\r\nUnits = sectors of 1 * 512 = 512 bytes\r\nSector size (logical/physical): 512 bytes / 512 bytes\r\nI/O size (minimum/optimal): 512 bytes / 512 bytes\r\nDisk label type: dos\r\nDisk identifier: 0x0006efa7\r\n\r\n   Device Boot      Start         End      Blocks   Id  System\r\n/dev/sda1   *        2048     1050623      524288   83  Linux   (BOOT PARTITION!!!!)\r\n/dev/sda2         1050624    42239999    20594688   8e  Linux LVM\r\n#################################################################\r\n/dev/sdX boot device will be /dev/sdX1, the rest will be on /dev/sdX2.  Customer data will be on /dev/sdY1.\r\nPlease keep the fact that there are partitions in mind when you attempt to recover this system.\r\n######################END OF output.out##########################\r\n````\r\n\r\nNow what does this do for ReaR?  It at least sets an example on what volume groups are associated with what multipath disks, and what block devices are associated with that multipath disk number.  It will also eliminate any read only disks that may show up due to gatekeepers being presented, or any number of read only associated disks that may be presented to the system over fibre channel.\r\n\r\n\r\nOnce you have taken note on what volume groups are associated with what drives, go into the disklayout.conf (/var/lib/rear/layout/.) section and uncomment the multipath entries that are associated with the set of block devices that your volume group is a member of.  In the case above, if we were only to restore VG00 (i.e. the root volume group), we would find the entries for /dev/sda that are associated with /dev/mapper/mpatha(yes it is actually mpatha2, but we aren't talking about partitions, YET...) within the disklayout.conf file.\r\nAs shown here:\r\n\r\n````\r\ndisk /dev/sda 21626880000 msdos\r\n#disk /dev/sdb 21626880000 msdos\r\n#disk /dev/sdc 5898240 unknown\r\ndisk /dev/sdd 21626880000 msdos\r\n#disk /dev/sde 21626880000 msdos\r\n#disk /dev/sdf 5898240 unknown\r\ndisk /dev/sdg 21626880000 msdos\r\n#disk /dev/sdh 21626880000 msdos\r\n#disk /dev/sdi 5898240 unknown\r\ndisk /dev/sdj 21626880000 msdos\r\n#disk /dev/sdk 21626880000 msdos\r\n#disk /dev/sdl 5898240 unknown\r\n#disk /dev/sdm 0\r\nmultipath /dev/mapper/mpatha /dev/sda,/dev/sdd,/dev/sdg,/dev/sdj\r\n````\r\nAlso, head into /etc/rear and look at the `site.conf` file.\r\n\r\nEnsure in the `REQUIRED_PROGS=(` that `multipathd` and `multipath` are there as well.\r\n\r\nAn example would be the following:\r\n````\r\nREQUIRED_PROGS=(\r\n\"$SCRIPT_FILE\"\r\nbash\r\nawk\r\ncpio\r\ndd\r\ndf\r\ngetopt\r\nsed\r\nvi\r\nip\r\ntar\r\nsort\r\nmv\r\nuniq\r\ngrep\r\nloadkeys\r\nkbd_mode\r\ndumpkeys\r\ndiff\r\njoin\r\npwd\r\nfile\r\npidof\r\nsync\r\nmultipath    (i have included this binary to pull into the mkrescue image)\r\nmultipathd  (This one is the actual daemon, I haven't noticed an issue with order)\r\nparted  (need to include this one to rebuild any drives with partitions and what not)\r\ngrub2   (will need this as well to re-install grub to the main boot device MBR)\r\ngrub2-install\r\ngrub2-mkconfig\r\n)\r\n````\r\nAlso set your `BOOT_OVER_SAN=Y` in this file as well as\r\n`AUTOEXCLUDE_MULTIPATH=n`  or leave it blank as well so it is unset.\r\n\r\n\r\nOnce these steps are done, go ahead and make the rescue image.\r\n\r\n`rear mkrescue ` (this takes some time).\r\n\r\nOnce complete, the rescue image will be in `/var/lib/rear/output/rear-<servername>.iso`\r\n\r\nFTP this image to your laptop, or whatever device you will be using to go into the console.\r\n\r\nIn this particular case, this server had a DELL iDRAC connected, from which we will use the CD/DVD/ISO virtual media manager to mount the image and force the system to boot to the rescue image.\r\n\r\nOnce at the boot menu, DO NOT SELECT AUTOMATIC, select MANUAL recovery mode (there are some steps that must be completed first and foremost).\r\n\r\nOnce booted into the rescue image after selecting manual recovery method, the first, and THE MOST IMPORTANT step to do while in this environment is to turn on multipath(assuming you didn't have to dhclient your interface to obtain an IP address from DHCP).\r\n\r\n`multipathd`    (this will turn on the multipath daemon)\r\n\r\n`multipath -ll`   (this will spit out and confirm whether or not multipath is working and can translate the block devices into selective paths via the WWID on each LUN from the SAN).\r\n````\r\nmpathc (360000970000192600732533030303445) dm-0 EMC     ,SYMMETRIX\r\nsize=5.6M features='1 queue_if_no_path' hwhandler='0' wp=ro\r\n`-+- policy='service-time 0' prio=1 status=active\r\n  |- 3:0:0:240 sdc 8:32  active ready running\r\n  |- 3:0:1:240 sdf 8:80  active ready running\r\n  |- 0:0:0:240 sdi 8:128 active ready running\r\n  `- 0:0:1:240 sdl 8:176 active ready running\r\nmpathb (360000970000192600732533030313238) dm-2 EMC     ,SYMMETRIX\r\nsize=20G features='1 queue_if_no_path' hwhandler='0' wp=rw\r\n`-+- policy='service-time 0' prio=1 status=active\r\n  |- 3:0:0:1   sdb 8:16  active ready running\r\n  |- 3:0:1:1   sde 8:64  active ready running\r\n  |- 0:0:0:1   sdh 8:112 active ready running\r\n  `- 0:0:1:1   sdk 8:160 active ready running\r\nmpatha (360000970000192600732533030313237) dm-1 EMC     ,SYMMETRIX\r\nsize=20G features='1 queue_if_no_path' hwhandler='0' wp=rw\r\n`-+- policy='service-time 0' prio=1 status=active\r\n  |- 3:0:0:0   sda 8:0   active ready running\r\n  |- 3:0:1:0   sdd 8:48  active ready running\r\n  |- 0:0:0:0   sdg 8:96  active ready running\r\n  `- 0:0:1:0   sdj 8:144 active ready running\r\n````\r\nIf you notice above, there is a disk that is presented that is only 5.6M in size, the `wp` flag is set to *ro* (read only) which means this LUN is in actuality a gatekeeper from the EMC SAN.  If you followed along correctly above, you would know to ignore this particular device and shouldn't have un-commented this entry in the `disklayout.conf` file in the previous step (prior to mkrescue).\r\n\r\nNow, here is where the \"bug\" comes into play, however I have figured out a workaround for this particular bug (the bug that says, NO FILE SYSTEMS MOUNTED TO /mnt/local).\r\n\r\nThis isn't due to a bug within multipath or a bug within anything else, except for a bug within the rear code and how it pertains to LVM2 while having a VG associated with the mpath disk.  I want it to be understood that this is in no way anyone's fault.  The ideas and concepts around boot from SAN and multipath can differ greatly from environment to environment.  Some choose to use LVM and LVM2 to control the growth of data, other environments choose to not use LVM and to go with straight block devices (mpath numbers) instead.  Boot from SAN can also vary greatly depending on the environment in question as well, so my goal with this document was to try and show all the mechanics behind several different methodologies and to try and get around certain bugs that may be presented because there isn't a huge group of folks that are necessarily going about this method like anyone else would (i.e. there is no RFC white paper on the proper way to do boot from SAN with multipathing enabled while using LVM2 for data size control).\r\n\r\nNow that being said, let's get around this bug where it cannot mount logical volumes to /mnt/local.\r\n\r\nrun a `vgchange -a y`   (that's right, do a blanket activation of all volume groups that the system can see, even if the contents of the LVOLS within this volume group have been deleted, the volume group and subsequent LVOLS will still be there, and still able to be activated).  Now you will see why running my script at the very beginning, prior to mkrescue is so important.\r\n\r\nOnce the volume groups, or volume group (you could theoretically just have vg00) are activated, mount the lvols from the volume group you are using into /mnt/local.  Yes, I understand that I am doing this manually, but the rear scripts themselves are failing because of some error where it isn't fully understanding that from this point forward, because multipath is enabled and LVM2 is up and operational, you don't need to worry about disk names (and that is the whole point of multipath, so that you do not have to worry about what block devices your data is on).\r\n````\r\nmount /dev/mapper/vg00-lv_root    /mnt/local/.\r\nmount /dev/mapper/vg00-lv_tmp    /mnt/local/tmp\r\nmount /dev/mapper/vg00-lv_var    /mnt/local/var\r\n````\r\n\r\nDO NOT FORGET ABOUT /boot!!!!\r\n````\r\nmount /dev/mapper/mpatha1    /mnt/local/boot\r\nchmod 755 /mnt/local/boot\r\n````\r\nThis is because 9 times out of 10, boot will be at 555, and thus so, TSM or any other backup/restore utility will not be able to write to this directory due to permissions, please check the permissions, or if so, chmod them 755, after the restore, you can chmod it back to 555.\r\n\r\nThe boot information was gathered again from the original script you run before you even mkrescue, and more specifically in the fdisk area that is printed at the bottom, the asterisk marked block device will specify the boot partition, and since we know that mpatha is our primary disk, we know that mpatha1 will be our boot disk, and mpatha2 will be our vg00 disk.  As shown here:\r\n\r\n````\r\nDisk /dev/sda: 21.6 GB, 21626880000 bytes, 42240000 sectors\r\nUnits = sectors of 1 * 512 = 512 bytes\r\nSector size (logical/physical): 512 bytes / 512 bytes\r\nI/O size (minimum/optimal): 512 bytes / 512 bytes\r\nDisk label type: dos\r\nDisk identifier: 0x0006efa7\r\n\r\n   Device Boot      Start         End      Blocks   Id  System\r\n**** /dev/sda1   *        2048     1050623      524288   83  Linux   (BOOT PARTITION!!!!) ****\r\n/dev/sda2         1050624    42239999    20594688   8e  Linux LVM\r\n````\r\n\r\nNow, if everything is okay, and the sun and planets are aligned just perfectly, go ahead and do the recover from rear.\r\n\r\n`rear recover`\r\n\r\neverything should be fine with the configuration of the `disklayout.conf` as well as default.conf and site.conf.\r\n\r\nSelect the correct numbers for all the filesystems you wish to restore.  i.e. 1 2 3 4 5 6 7 8 9 10, etc...\r\n\r\nit will be normal to see some errors that say \"no code added to for filesystem blah blah blah\"\r\n1.  Continue\r\n2.  Abort\r\n\r\nSelect option 1 for every question that asks this.\r\n\r\nOnce it goes through these questions, it will then check that there is something mounted in /mnt/local.   SURPRISE!!!!  Your lvols are already mounted there via multipath because of the steps you just followed, so if files are missing or directories are missing(i.e. if someone tried a 'rm -Rf * /'), TSM will now pick up where you left off and begin recovery.\r\n\r\nWhen the above is completed, if you are utilizing selinux, you need to go into /mnt/local/etc/selinux and set enforce mode to permissive mode for selinux.  Then touch /mnt/local/.autorelabel.  This is to combat a minor issue when SELINUX blocks all new scanned FC devices during reboot and therefore your devices will not be mounted (your kernel will panic, or go into the mode where it kills off udev every so often).\r\n\r\nChange /mnt/local/boot back to permission level of 555 (note:  You may still encounter the error that rear is unable to find any /boot partitions.  From a theoretical standpoint, if something was wiped from /boot, you could use the permissions work around to restore from backup.)\r\n\r\nReboot.  And wait for the system to come back up.\r\n\r\nOnce the system comes back up, run `multipath -r`, then `multipath -v3`, then reboot one more time (sometimes only 3 of the 4 paths come back on the initial reboot, and doing a refresh, followed by a rescan (v3) will fix this issue and voila, you will have your multipath, boot from SAN, restored from backup machine.\r\n\r\n\r\nFOOD FOR THOUGHT:\r\nIt might be worth saying that for this type of recovery it is best to get your root filesystem recovered first and foremost.  Once you can boot normally because / is working correctly, then you could theoretically kick off a real TSM session with a system that is up in order to recover any additional volume groups that you have (i.e. oracle database, some application, or customer data).\r\n\r\nI HOPE TO GOD THAT THIS HELPS THE REAR TEAM GET CLOSER TO COMPLETING THIS PROJECT WITH FULLY AUTOMATED SUPPORT for BOOT FROM SAN as well as MULTIPATHD.\r\n\r\n\r\n####TOTAL DISASTER NOTE ####\r\n  If the lvols that are associated with the VG are completely gone (i.e. `vgchange -ay` completely fails out due to bad/broken disks).  What you could do prior to the recover, is to use the RHEL rescue image (i.e. boot into rescue mode with multipath enabled), go ahead and re-create your disks (i.e. pvcreate, kpartx -a, etc...), then go ahead and create your VG's as well as your LVOLS(use the df.txt file that ReaR has coded into their recovery agent if you need to take a quick look at how you had once setup your filesystems).  Once this is complete, stop right there, boot into the rescue image you created with ReaR and then follow the documented guide I have above.  This will mean that all LVOLS are completely empty and that TSM will then restore EVERYTHING that you mount into /mnt/local/.\r\n####TOTAL DISASTER NOTE####\r\n"},"comment":{"url":"https://api.github.com/repos/rear/rear/issues/comments/98698078","html_url":"https://github.com/rear/rear/issues/572#issuecomment-98698078","issue_url":"https://api.github.com/repos/rear/rear/issues/572","id":98698078,"user":{"login":"multipathmaster","id":11719398,"avatar_url":"https://avatars.githubusercontent.com/u/11719398?v=3","gravatar_id":"","url":"https://api.github.com/users/multipathmaster","html_url":"https://github.com/multipathmaster","followers_url":"https://api.github.com/users/multipathmaster/followers","following_url":"https://api.github.com/users/multipathmaster/following{/other_user}","gists_url":"https://api.github.com/users/multipathmaster/gists{/gist_id}","starred_url":"https://api.github.com/users/multipathmaster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/multipathmaster/subscriptions","organizations_url":"https://api.github.com/users/multipathmaster/orgs","repos_url":"https://api.github.com/users/multipathmaster/repos","events_url":"https://api.github.com/users/multipathmaster/events{/privacy}","received_events_url":"https://api.github.com/users/multipathmaster/received_events","type":"User","site_admin":false},"created_at":"2015-05-04T12:49:45Z","updated_at":"2015-05-04T12:49:45Z","body":"Just to provide an update.  I have opened a case with redhat regarding not having standard grub included with RHEL7.  This whole grub thing can be sorted out if the standard packaged grub comes with the system out of the box (can script some answers and pipe them into grub in order to re-write the MBR and boot partition).  I will updated you guys whenever I have heard some information back from redhat.\r\n\r\nThanks -- MultipathMaster"}}