issues with live migration on centos (was: Trillian test failures)

[weizhouapache]

@DaanHoogland @rhtyd
this needs research.

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	Error	63.19	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	Error	48.09	test_vm_life_cycle.py

#4736 (comment)

management logs

2021-03-02 18:50:44,203 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-106:ctx-a6be1c7c job-2200/job-2201 ctx-10684199) (logid:f366d768) Failed to copy volume
com.cloud.utils.exception.CloudRuntimeException: Copy operation failed in 'StorageSystemDataMotionStrategy.copyAsync': Exception during migrate: org.libvirt.LibvirtException: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported
	at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:1901)
	at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:119)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1645)
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1181)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3000)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5524)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-03-02 18:50:44,203 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-106:ctx-a6be1c7c job-2200/job-2201 ctx-10684199) (logid:f366d768) Failed to migrated vm VM[User|i-191-245-VM] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Copy operation failed in 'StorageSystemDataMotionStrategy.copyAsync': Exception during migrate: org.libvirt.LibvirtException: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported

ISSUE TYPE

• Bug Report
• Improvement Request
• Enhancement Request
• Feature Idea
• Documentation Report
• Other

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[DaanHoogland]

@weizhouapache the centos7 templates for kvm do not support life migration on ubuntu it should work fine. in #4727 I added an explicit test for host os in the test. this one is 13000 lines so not a pleasure to walkthrough, so
874a142 (#4727) should do it in future trillian runs.
Also I'm looking into adding the test data on ubuntu but it gets drowned a bit.

[DaanHoogland]

So the real issue is life migration on centos, @weizhouapache , not the tests.

[weizhouapache]

So the real issue is life migration on centos, @weizhouapache , not the tests.

@DaanHoogland yeah, I think so.

[yadvr]

I think it's because of stock CentO7 libvirtd that does not support it, I think with qemu-kvm-ev this may work (like Ubuntu) [1]
cc @shwstppr @Pearl1594 (any ideas for CentOS8?)

[1] https://wiki.centos.org/SpecialInterestGroup/Virtualization

[DaanHoogland]

I'm not sure about the milestone, but I don't think this was introduced after 4.15

[weizhouapache]

I'm not sure about the milestone, but I don't think this was introduced after 4.15

@DaanHoogland @rhtyd
as Rohit said, this might not be caused by cloudstack changes, but by some recent changes in centos7 template/qemu/libvirt.
no idea if it is good idea to skip the testings.

[Pearl1594]

@rhtyd Live migration on centos8 seems to work fine. However, there is an issue noticed during live migration of VMs with data disks (test_02_migrate_VM_with_two_data_disks) that aren't created from snapshots (i.e., don't have a template id) - this has been addressed with - #4698 cc @DaanHoogland @weizhouapache