Fix data volume import / export functionality when using swift as secondary storage#4875
Fix data volume import / export functionality when using swift as secondary storage#4875skattoju4 wants to merge 21 commits intoapache:mainfrom
Conversation
skattoju4
force-pushed
the
volume_import_export_swift
branch
from
178a25c to
8eb2133
Compare
| } | ||
|
|
||
| File destFile = SwiftUtil.getObject(swiftTO, downloadDirectory, srcData.getPath()); | ||
| String filePath = downloadPath + File.separator + destData.getName(); |
There was a problem hiding this comment.
Are you not using the getPath to find the path of the object anymore? does downloadPath consider the difference between snapshot and template and now volume?
There was a problem hiding this comment.
Hey Syed, thanks for the review. We do call getPath() on line 766 so the downloadPath will contain the path to folder depending on whether its a volume, snapshot or template. Here i'm skipping the copy from swift if the file is already there since we don't upload volumes to swift just to the staging nfs.
| } else { | ||
| hostVO = _hostDao.findById(vm.getLastHostId()); | ||
| } | ||
| destPrimaryStorage.setClusterId(hostVO.getClusterId()); |
There was a problem hiding this comment.
Do we want to set the clusterID for zonewide storage too?
There was a problem hiding this comment.
This was added because we were encountering a null pointer exception in some scenarios I don't recall exactly. I added the bit to get the last host id of the vm as attaching a disk to a stopped vm would fail. Have not encountered any side effects with regards to zone wide storage so far..
| @@ -0,0 +1,397 @@ | |||
| // | |||
There was a problem hiding this comment.
Where would this run? On the mgmt server or on the SSVM?
There was a problem hiding this comment.
This would run on the SSVM
|
@skattoju4 is this ready, how do we test it or can you provide test results? |
|
@rhtyd it should be good to go. A CloudStack setup that uses swift as secondary storage is needed to test. This was tested manually with the following high level steps:
Its been in production for some time with no side effects so far @pdion891 can attest. |
|
The travis ci failures are intermittent. |
|
@blueorangutan package |
|
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
Packaging result: ✖️ centos7 ✔️ centos8 ✔️ debian. SL-JID 268 |
|
@blueorangutan package |
|
@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. |
|
@blueorangutan package |
|
@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4875 +/- ##
============================================
+ Coverage 28.57% 29.23% +0.65%
- Complexity 29784 30501 +717
============================================
Files 5100 5101 +1
Lines 358565 358815 +250
Branches 52316 52342 +26
============================================
+ Hits 102464 104887 +2423
+ Misses 241968 239541 -2427
- Partials 14133 14387 +254
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 7044 |
|
@Sjnas can you please check the failures? Should we consider this for 4.19.0.0? |
|
@blueorangutan package |
|
@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 7212 |
|
@blueorangutan package |
|
@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7220 |
|
@blueorangutan test |
|
@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-7835)
|
|
@blueorangutan test |
|
@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-7854)
|
|
@skattoju4 the usage test failure seem consistent: 'Command failed due to Internal Server Error' can you check it locally? |
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
|
@skattoju4 are you still concerned with this? |
|
@skattoju4 , closing this due to inactivity. please feel free to reopen. |
12 participants
Description
This PR is a fix for the data volume import and export functionality when using swift as secondary storage.
Background:
Secondary storage is used for templates, snapshots and iso's. A data disk is a volume and is meant to reside on primary storage. However, when importing a volume it is first uploaded to secondary storage and then copied to over to primary storage when attached to a VM. Similarly when exporting a volume, it is copied to secondary storage before it is made available to the user for download. When using swift as a secondary storage, an intermediate staging nfs store is needed when copying volumes between primary and secondary storage. This staging nfs store is leveraged for the volume import and export functionality since volumes need not have a footprint on secondary storage.
Currently this functionality does not work:
When importing a volume the following error is observed:
org.apache.cloudstack.storage.to.VolumeObjectTO cannot be cast to org.apache.cloudstack.storage.to.TemplateObjectTOFurther investigation reveals that there is no provision in the code to handle import of volumes. Currently only template import is supported.
When exporting a volume a URL to the secondary storage is generated but it clicking it does not initiate a download of the exported volume. Further investigation revealed that the url points to a broken symlink which points to an invalid path on the SSVM
Use Cases
Types of changes
Feature/Enhancement Scale or Bug Severity
This addresses #4874
Feature/Enhancement Scale
Bug Severity
How Has This Been Tested?