Create profiles to download systemvm-templates by GutoVeronezi · Pull Request #5602 · apache/cloudstack

GutoVeronezi · 2021-10-21T13:07:21Z

Description

As I pointed on thread [VOTE] Apache Cloudstack 4.16.0.0 RC1, regarding the auto upgrading systemvms-templates; the code will autoupgrade the systemvm-templates if the templates are found locally. To generate the template during the build, one has to activate the noredist profile when building the packages, which is necessary (normally used to) only to VMWare.

Some people, might not want the noredist JARs from VMware, and therefore will not build with this profile. Furthermore, package cloudstack-management goes from ~100Mb to more than 1.5Gb when using the noredist while executing the 4.16.0.0 build. It happens due to cloudstack/engine/schema/pom.xml automatically downloading XenServer, KVM, and VMware systemvms-templates. Every time that we build the packages, it will download all these systemvms-templates, and we are unable to decide which one is necessary.

This PR intends to create specific profiles for each systemvm-template. Therefore, one can build ACS only for the hypervisor it needs, and if it needs to use the auto update of the system VM templates; one can decide not to use this feature and keep manually registering the templates only when they are needed.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

How Has This Been Tested?

I built the packages with all the profiles:

-DdownloadKvmSystemVmTemplate
-DdownloadHypervSystemVmTemplate
-DdownloadVmwareSystemVmTemplate
-DdownloadXenserverSystemVmTemplate
-DdownloadOvmSystemVmTemplate

And then I built them separately.

yadvr · 2021-10-21T14:56:33Z

@GutoVeronezi good to see this PR however if we move to some new ideas around systemvmtemplate (read-only iso) or other optimisations this may not be needed. Let me review this tomorrow and get back to you, I'll reply and explain why we need to bundle all 3 hypervisors templates over the email thread.

yadvr · 2021-10-21T16:21:58Z

            <activation>
                <property>
-                    <name>noredist</name>
+                    <name>downloadKvmSystemVmTemplate</name>


nit - this may be difficult to remember, should be simply named as systemvm-, say -D systemvm-kvm, -D systemvm-xen, -D systemvm-vmware...

@rhtyd done.

yadvr · 2021-10-21T16:22:12Z

@blueorangutan package

blueorangutan · 2021-10-21T16:23:04Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-10-21T16:50:24Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1600

nvazquez · 2021-10-21T17:00:50Z

@blueorangutan test matrix

blueorangutan · 2021-10-21T17:01:03Z

@nvazquez a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

yadvr · 2021-10-22T05:49:48Z

@blueorangutan package

blueorangutan · 2021-10-22T05:50:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2021-10-22T05:52:07Z

@GutoVeronezi lgtm, can we make mvn enable all main templates (at least kvm, vmware and xen) to be enabled when noredist is activated?

blueorangutan · 2021-10-22T06:37:19Z

Packaging result: ✖️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1603

yadvr

Guto I tested the pkgs, when -Dnoredist is passed it doesn't bundle all the three hypervisor. Can you explore if we can make it bundle all 3 when noredist is passed; and when not then you pass individual -D systemvm-kvm...

GutoVeronezi · 2021-10-22T13:04:42Z

Guto I tested the pkgs, when -Dnoredist is passed it doesn't bundle all the three hypervisor. Can you explore if we can make it bundle all 3 when noredist is passed; and when not then you pass individual -D systemvm-kvm...

@rhtdy bundling the hypervisors with the profile noredist would be counterproductive. This PR intends to make the downloading feature more granular and to allow us to decide which template to download (or none). Futhermore, the noredist profile is/was dedicated to non-redistributable packages (as the name says), which is not the case of the systemvm templates. Therefore, the goal is to fix this inconsistency as well.

To bundle the 3 hypervisors into the package, we would have to add these options to your maven command line:

-Dsystemvm-kvm
-Dsystemvm-vmware
-Dsystemvm-xen

yadvr · 2021-10-22T13:10:20Z

As I've tried to explain on the email thread, the noredist build profile is supposed to build turnkey packages. Other than very few people who like to customise their builds, the standard practice has been to build and publish turnkey all components included pkgs for years. That's why the use case would be to mix the new build profiles you've added when - Dnoredist is not passed, i.e. someone's building non-turnkey custom pkgs. I've mentioned on the ML thread if noredist need to be renamed, but that's the historical context.

GabrielBrascher · 2021-10-22T17:12:37Z

I like the idea of making it configurable and fitting all the tastes.
Also, making it a hypervisor configuration instead of noredist is clear and avoids any confusion; users can build the packages according to hypervisors they have in the Zone.

The noredist packages definition requires users to understand that in case XenServer or VMware they need the noredist.

I would be OK with having such a procedure of building described in the documentation.
On the packages that are build for convenience we can still keep building aiming all the hypervisors.

yadvr · 2021-10-26T11:45:13Z

@blueorangutan package

blueorangutan · 2021-10-26T11:46:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-10-26T13:00:36Z

Packaging result: ✖️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1619

yadvr · 2021-11-15T10:30:30Z

My point being, people building noredist pkgs - get all-inclusive pkgs. We can discuss and change the noredist build profile name, however, what I'm asking is a build profile that allows bundilng of all hypervisors - maybe consider a new profile like -D systemvm-all that is triggered/activated when -D noredist is passed?

PaulAngus · 2021-11-15T17:18:30Z

I'm not sure where this fits into the conversation, but i think i'd be a good idea to have a separate (and optional) cloudstack-systemvms rpm.
ie
yum install cloudstack-management cloudstack-systemvms

it stops the 'standard' install bloating massively while still simplifying the installation.

GabrielBrascher · 2021-11-15T17:22:07Z

@rhtyd I agree, it makes total sense to have a build profile that allows bundling of packages that would work on all/any hypervisors. The system VM packaging also looks nice; for many users, it would be easier to upgrade/deploy.

But I think that it would be great if the project offers flexibility in regards to such profiles.
I Imagine that's what @GutoVeronezi is aiming for.

As an example, users would by default have SystemVM in the package and the noredist set in order to pack in VMware or any other hypervisor.

However, "advanced users" could do a fine setup that would match their requirements.
For instance, build for a specific hypervisor and choose if the System VM template will be automatically injected or manually placed.

yadvr · 2021-11-25T12:14:01Z

We're coming with a new design doc that will reduce the cloudstack-management pkg by more than 1GB in size and introduce a single mvn flag that will bundle a template for only one type of hypervisor. (we tested and found we don't need to bundle all three hypervisor templates, as qemu-img can be used to convert formats on the fly which is what we currently, for example qemu-img takes 10-20s to convert say a vmdk to qcow2 or vmdk to vhd).
Early draft here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVM+Optimisations

Pearl1594 · 2022-03-10T14:30:45Z

@blueorangutan package kvm,xen

blueorangutan · 2022-03-10T14:30:58Z

@Pearl1594 a Jenkins job has been kicked to build packages. Package will be bundled with kvm,xen.I'll keep you posted as I make progress.

Pearl1594 · 2022-03-10T14:37:00Z

Was able to finally get bo to pass the extended choice parameters -
centos7 packaging job output - which will be same for centos8/rocky8 and suse:

Adding flags to package requested templates
++ rpm --eval '-Dsystemvm-kvm -Dsystemvm-xen'
+ FLAGS=' -Dnoredist -Dsystemvm-kvm -Dsystemvm-xen'
+ mvn -Psystemvm,developer -Dnoredist -Dsystemvm-kvm -Dsystemvm-xen clean package
/root/.mavenrc: line 1: /usr/libexec/java_home: No such file or directory
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0

[INFO] Scanning for projects...

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]

debian pkg output:

make[1]: Entering directory '/jenkins/workspace/deb-pkg-flags'
mvn clean package -Psystemvm,developer -Dsystemvm \
    -Dcs.replace.properties=replace.properties.tmp \
    -Dmaven.repo.local=/root/.m2/repository \
     -Dnoredist -Dnonoss  -Dsystemvm-kvm -Dsystemvm-xen
/usr/bin/mvn: 1: /root/.mavenrc: /usr/libexec/java_home: not found
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0

[INFO] Scanning for projects...
[INFO] ------------------------------------------------

FYI, failures you see in centos8 and suse packaging jobs are expected as they aren't created - for testing

blueorangutan · 2022-03-10T15:04:54Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✖️ suse15. SL-JID 11

GutoVeronezi · 2022-03-10T16:37:57Z

Thanks for testing it, @Pearl1594.

Pearl1594 · 2022-03-11T05:13:42Z

@blueorangutan package

blueorangutan · 2022-03-11T05:13:56Z

@Pearl1594 a Jenkins job has been kicked to build packages. Package will be bundled with.I'll keep you posted as I make progress.

blueorangutan · 2022-03-11T05:35:41Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✖️ suse15. SL-JID 12

yadvr · 2022-03-11T08:45:25Z

@blueorangutan package

blueorangutan · 2022-03-11T08:47:02Z

@rohityadavcloud a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-03-11T09:12:57Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2842

Pearl1594 · 2022-03-11T09:28:14Z

@blueorangutan package

blueorangutan · 2022-03-11T09:28:27Z

@Pearl1594 a Jenkins job has been kicked to build packages. Package will be bundled with KVM, XenServer and VMware SystemVM templates.I'll keep you posted as I make progress.

blueorangutan · 2022-03-11T10:18:43Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✖️ suse15. SL-JID 13

Pearl1594 · 2022-03-11T12:06:27Z

@blueorangutan package

blueorangutan · 2022-03-11T12:07:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-11T13:12:32Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2847

nvazquez · 2022-03-16T03:11:30Z

@blueorangutan test

blueorangutan · 2022-03-16T03:12:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-16T12:42:59Z

Trillian test result (tid-3620)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32999 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5602-t3620-kvm-centos7.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_nic	`Error`	173.54	test_nic.py

nvazquez · 2022-03-17T03:55:22Z

Merging based on approvals and test results (failure not related to this PR)

Create profiles to download systemvm-templates

703f46e

GutoVeronezi requested review from Pearl1594 and nvazquez October 21, 2021 13:07

yadvr reviewed Oct 21, 2021

View reviewed changes

Rename profiles

b081109

yadvr approved these changes Oct 22, 2021

View reviewed changes

yadvr requested changes Oct 22, 2021

View reviewed changes

nvazquez added this to the 4.16.1.0 milestone Oct 25, 2021

nvazquez added component:build type:enhancement labels Oct 25, 2021

apache deleted a comment from blueorangutan Mar 10, 2022

apache deleted a comment from blueorangutan Mar 11, 2022

nvazquez approved these changes Mar 17, 2022

View reviewed changes

nvazquez merged commit c146449 into apache:main Mar 17, 2022

Conversation

GutoVeronezi commented Oct 21, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

How Has This Been Tested?

Uh oh!

yadvr commented Oct 21, 2021

Uh oh!

yadvr Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

yadvr commented Oct 21, 2021

Uh oh!

blueorangutan commented Oct 21, 2021

Uh oh!

blueorangutan commented Oct 21, 2021

Uh oh!

nvazquez commented Oct 21, 2021

Uh oh!

blueorangutan commented Oct 21, 2021

Uh oh!

yadvr commented Oct 22, 2021

Uh oh!

blueorangutan commented Oct 22, 2021

Uh oh!

yadvr commented Oct 22, 2021

Uh oh!

blueorangutan commented Oct 22, 2021

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi commented Oct 22, 2021

Uh oh!

yadvr commented Oct 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GabrielBrascher commented Oct 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Oct 26, 2021

Uh oh!

blueorangutan commented Oct 26, 2021

Uh oh!

blueorangutan commented Oct 26, 2021

Uh oh!

yadvr commented Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PaulAngus commented Nov 15, 2021

Uh oh!

GabrielBrascher commented Nov 15, 2021

Uh oh!

yadvr commented Nov 25, 2021

Uh oh!

Pearl1594 commented Mar 10, 2022

Uh oh!

blueorangutan commented Mar 10, 2022

Uh oh!

Pearl1594 commented Mar 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Mar 10, 2022

Uh oh!

GutoVeronezi commented Mar 10, 2022

Uh oh!

Pearl1594 commented Mar 11, 2022

Uh oh!

blueorangutan commented Mar 11, 2022

Uh oh!

blueorangutan commented Mar 11, 2022

Uh oh!

yadvr commented Mar 11, 2022

Uh oh!

blueorangutan commented Mar 11, 2022

yadvr commented Oct 22, 2021 •

edited

Loading

GabrielBrascher commented Oct 22, 2021 •

edited

Loading

yadvr commented Nov 15, 2021 •

edited

Loading

Pearl1594 commented Mar 10, 2022 •

edited

Loading