Fix template registration error

[nvazquez]

Description

After forward merge from branch 4.13 template registration fails with NPE:

2020-04-01 17:53:54,306 INFO  [storage.template.HttpTemplateDownloader] (agentRequest-Handler-9:null) No credentials configured for host=dl.openvm.eu:80
2020-04-01 17:53:54,347 INFO  [storage.template.HttpTemplateDownloader] (pool-1-thread-4:null) Starting download from http://dl.openvm.eu/cloudstack/macchinina/x86_64/macchinina-kvm.qcow2.bz2 to /mnt/SecStorage/e5b31dbe-98f0-35b5-b5d6-fe87b26cc133/template/tmpl/2/203/dnld11734576344067057786tmp_ remoteSize=20456842 , max size=53687091200
2020-04-01 17:53:56,877 INFO  [storage.template.DownloadManagerImpl] (pool-1-thread-4:null) Download Completion for jobId: ec049522-34e6-4c98-b954-aaba8443fe6f, status=IN_PROGRESS
2020-04-01 17:53:56,878 INFO  [storage.template.DownloadManagerImpl] (pool-1-thread-4:null) local: /mnt/SecStorage/e5b31dbe-98f0-35b5-b5d6-fe87b26cc133/template/tmpl/2/203/dnld11734576344067057786tmp_, bytes=1049522, error= , pct=5
2020-04-01 17:53:56,878 INFO  [storage.template.DownloadManagerImpl] (pool-1-thread-4:null) Resuming jobId: ec049522-34e6-4c98-b954-aaba8443fe6f, status=IN_PROGRESS
2020-04-01 17:53:56,879 WARN  [storage.template.HttpTemplateDownloader] (pool-1-thread-4:null) Caught exception during download null
java.lang.NullPointerException
	at org.apache.cloudstack.utils.imagestore.ImageStoreUtil.isCorrectExtension(ImageStoreUtil.java:97)
	at org.apache.cloudstack.utils.imagestore.ImageStoreUtil.checkTemplateFormat(ImageStoreUtil.java:61)
	at com.cloud.storage.template.HttpTemplateDownloader$VerifyFormat.invoke(HttpTemplateDownloader.java:523)
	at com.cloud.storage.template.HttpTemplateDownloader.copyBytes(HttpTemplateDownloader.java:245)
	at com.cloud.storage.template.HttpTemplateDownloader.download(HttpTemplateDownloader.java:209)
	at com.cloud.storage.template.HttpTemplateDownloader.runInContext(HttpTemplateDownloader.java:426)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

The data structure for supported formats does not include img as a key, it is instead a value for key raw:

private final static Map<String, Set<String>> SUPPORTED_EXTENSIONS_BY_FORMAT =
            ImmutableMap.<String, Set<String>>builder()
                        .put("vhd", buildExtensionSet(false, "vhd"))
                        .put("vhdx", buildExtensionSet(false, "vhdx"))
                        .put("qcow2", buildExtensionSet(true, "qcow2"))
                        .put("ova", buildExtensionSet(true, "ova"))
                        .put("tar", buildExtensionSet(false, "tar"))
                        .put("raw", buildExtensionSet(false, "img", "raw"))
                        .put("vmdk", buildExtensionSet(false, "vmdk"))
                        .put("iso", buildExtensionSet(true, "iso"))
            .build();

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Before:
Register template -> Failure

After the fix:
Templates registered properly

[nvazquez]

@blueorangutan package

[nvazquez]

@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[weizhouapache]

code LGTM

[GabrielBrascher]

Code LGTM!

[blueorangutan]

Packaging result: ✔centos7 ✔debian. JID-1116

[nvazquez]

@blueorangutan test

[blueorangutan]

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[DaanHoogland]

@andrijapanicsb this is most definitely a merge booboo. needs to go in.

[andrijapanicsb]

yes @DaanHoogland, need to poke the author on his first master-down accomplishment :))) cc @Spaceman1984 ;)

[blueorangutan]

Trillian test result (tid-1345)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38019 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4008-t1345-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_deploy_and_upgrade_kubernetes_cluster	Failure	1052.41	test_kubernetes_clusters.py

[shwstppr]

@andrijapanicsb I've checked into the logs for failing CKS test and it seems an intermittent issue while performing the upgrade on a k8s cluster. It fails to upgrade one of the cluster nodes in the desired time. I'm sure this has nothing to do with PR changes.
cc @nvazquez

[andrijapanicsb]

LGTM

tested manually registering vmware/xs templates (single compressed/uncompressed variations) and also KVM with both qcow2.bz2 and qcow2.

.img extension is still accepted for RAW templates only, although i.e. Canonical distributes their qcow2 images with that (weird) .img extension. We can leave this improvement for later.

[nvazquez]

@andrijapanicsb img extension added for qcow2 format in the last commit
@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✖centos7 ✖debian. JID-1120

[nvazquez]

@andrijapanicsb reverted
@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[andrijapanicsb]

the last commit reverted, so nothing to test again - previous tests are valid (we need new packages, since old ones are wiped).

Merging based on 3 x LGTMs/Approvals, manual functional testing and automated regression testing (the single failing test is not related to the PR).

[blueorangutan]

Packaging result: ✔centos7 ✔debian. JID-1122