Bug when resizing disk online
Today happened following error when resizing ekobal-mail.feldhost.cz
(one-97) running on node1. After issuing multipath -rr 360002ac0000000000000013a0001ec48
paths was again available. Guest FS remains write-protected and needs to reboot or shutdown and start. If VM is just rebooted, I have to finish resize of disk manually using virsh blockresize one-97 sda 51200M
command. I have to fix the size stored in the Opennebula database to reflect the real size.
UPDATE: I try to update the size in Opennebula by resizing again to the same size and it works.
We should implement check if multipathd -k"resize map $DEV"
returns non-zero code in which case we run multipath -rr $DEV
Sun Nov 15 18:04:52 2020 [Z0][VM][I]: New state is ACTIVE
Sun Nov 15 18:04:52 2020 [Z0][VM][I]: New LCM state is DISK_RESIZE
Sun Nov 15 18:05:01 2020 [Z0][VMM][I]: Command execution failed (exit code: 1): /var/lib/one/remotes/tm/3par/resize node1.feldcloud.net:/var/lib/one//datastores/100/97/disk.0 51200 97 101
Sun Nov 15 18:05:02 2020 [Z0][VMM][I]: resize: Resizing disk feldcloud.one.65.vv by 10240 MB
Sun Nov 15 18:05:03 2020 [Z0][VMM][E]: resize: Command " set -e
Sun Nov 15 18:05:03 2020 [Z0][VMM][I]: DEV="360002ac0000000000000013a0001ec48"
Sun Nov 15 18:05:04 2020 [Z0][VMM][I]:
Sun Nov 15 18:05:05 2020 [Z0][VMM][I]: HOSTS=$(cat /proc/scsi/scsi | awk -v RS="Type:" '$0 ~ "Vendor: 3PARdata" {print $0}' |grep -Po "scsi[0-9]+"|grep -Eo "[0-9]+" |sort|uniq|paste -sd "," -)
Sun Nov 15 18:05:05 2020 [Z0][VMM][I]: sudo /usr/bin/rescan-scsi-bus.sh --hosts=$HOSTS --luns=20 --nooptscan --forcerescan
Sun Nov 15 18:05:07 2020 [Z0][VMM][I]: sudo multipathd -k"resize map $DEV"" failed:
Sun Nov 15 18:05:07 2020 [Z0][VMM][E]: Error rescaning for new size
Sun Nov 15 18:05:08 2020 [Z0][VMM][I]: Failed to execute transfer manager driver operation: tm_resize.
Sun Nov 15 18:05:08 2020 [Z0][VMM][E]: Error resizing disk: Error rescaning for new size
Sun Nov 15 18:05:10 2020 [Z0][VM][I]: New LCM state is RUNNING
Sun Nov 15 18:05:10 2020 [Z0][LCM][I]: VM disk resize operation completed.
Nov 16 00:14:51 node2 multipathd: sdacg: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 7 1 135:656 1 135:672 1 135:688 1 135:720 1 135:704 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdacg [135:640]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:51 node2 multipathd: sdag: add path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdag, device size mismatch
Nov 16 00:14:51 node2 multipathd: uevent trigger error
Nov 16 00:14:51 node2 multipathd: sdach: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 6 1 135:672 1 135:688 1 135:720 1 135:704 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdach [135:656]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:51 node2 multipathd: sdah: add path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdah, device size mismatch
Nov 16 00:14:51 node2 multipathd: uevent trigger error
Nov 16 00:14:51 node2 multipathd: sdaci: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 5 1 135:688 1 135:720 1 135:704 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdaci [135:672]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:51 node2 multipathd: sdai: add path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdai, device size mismatch
Nov 16 00:14:51 node2 multipathd: uevent trigger error
Nov 16 00:14:51 node2 multipathd: sdacj: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 4 1 135:720 1 135:704 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdacj [135:688]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:51 node2 multipathd: sdaj: add path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdaj, device size mismatch
Nov 16 00:14:51 node2 multipathd: uevent trigger error
Nov 16 00:14:51 node2 multipathd: sdack: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 3 1 135:720 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdack [135:704]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:51 node2 multipathd: sdak: add path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdak, device size mismatch
Nov 16 00:14:51 node2 multipathd: uevent trigger error
Nov 16 00:14:51 node2 multipathd: sdacl: remove path (uevent)
Nov 16 00:14:51 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 135:736 1 135:752 1]
Nov 16 00:14:51 node2 multipathd: sdacl [135:720]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:52 node2 multipathd: sdal: add path (uevent)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdal, device size mismatch
Nov 16 00:14:52 node2 multipathd: uevent trigger error
Nov 16 00:14:52 node2 multipathd: sdacm: remove path (uevent)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 135:752 1]
Nov 16 00:14:52 node2 multipathd: sdacm [135:736]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:52 node2 multipathd: sdam: add path (uevent)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdam, device size mismatch
Nov 16 00:14:52 node2 multipathd: uevent trigger error
Nov 16 00:14:52 node2 multipathd: sdacn: remove path (uevent)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: map in use
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: can't flush
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: load table [0 20971520 multipath 1 queue_if_no_path 1 alua 0 0]
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: Entering recovery mode: max_retries=18
Nov 16 00:14:52 node2 multipathd: sdacn [135:752]: path removed from map 360002ac000000000000000d30001ec48
Nov 16 00:14:52 node2 multipathd: sdan: add path (uevent)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: failed to add new path sdan, device size mismatch
Nov 16 00:14:52 node2 multipathd: uevent trigger error
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: resize map (operator)
Nov 16 00:14:52 node2 multipathd: 360002ac000000000000000d30001ec48: couldn't get path group. cannot resize
Nov 16 00:17:53 node2 multipathd: 360002ac000000000000000d30001ec48: Disable queueing