VMware: Improve error messaging / logs when starting non-user VMs, and secondary storage not available or doesn't have enough capacity by sureshanaparti · Pull Request #9207 · apache/cloudstack

sureshanaparti · 2024-06-10T14:55:57Z

Description

This PR improves error messaging / logs when starting non-user VMs in VMware, and secondary storage not available or doesn't have enough capacity. It addresses inappropriate messages part 2, 3 in #8390.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Set the config secondary.storage.capacity.thresholdto below the available secondary storage capacity, stop SSVM and start it. SSVM will fail to start with the below logs.

2024-06-24 21:24:59,152 WARN  [c.c.s.StatsCollector] (DirectAgent-35:ctx-1a4ece48 10.0.35.32, job-65/job-66, cmd: StartCommand) (logid:1c0be4ca) Image storage [1] has not enough capacity. Capacity: total=[1 TB], used=[1 TB], threshold=[20.000000298023224%].
2024-06-24 21:24:59,152 INFO  [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-35:ctx-1a4ece48 10.0.35.32, job-65/job-66, cmd: StartCommand) (logid:1c0be4ca) Secondary storage is either not having free capacity or not NFS, then use cache/staging storage instead
2024-06-24 21:24:59,154 WARN  [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-35:ctx-1a4ece48 10.0.35.32, job-65/job-66, cmd: StartCommand) (logid:1c0be4ca) No cache/staging storage found when NFS secondary storage with free capacity not available or non-NFS secondary storage is used
2024-06-24 21:24:59,155 INFO  [c.c.h.v.u.VmwareHelper] (DirectAgent-35:ctx-1a4ece48 10.0.35.32, job-65/job-66, cmd: StartCommand) (logid:1c0be4ca) [ignored]failed to get message for exception: NFS secondary or cache storage of dc 1 either doesn't have enough capacity (has reached 20% usage threshold) or not ready yet, or non-NFS secondary storage is used
2024-06-24 21:24:59,155 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-35:ctx-1a4ece48 10.0.35.32, job-65/job-66, cmd: StartCommand) (logid:1c0be4ca) StartCommand failed due to [Exception: java.lang.Exception
Message: NFS secondary or cache storage of dc 1 either doesn't have enough capacity (has reached 20% usage threshold) or not ready yet, or non-NFS secondary storage is used
].
java.lang.Exception: NFS secondary or cache storage of dc 1 either doesn't have enough capacity (has reached 20% usage threshold) or not ready yet, or non-NFS secondary storage is used
	at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2286)
	at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:566)

How did you try to break this feature and the system with this change?

sureshanaparti · 2024-06-10T14:58:53Z

@blueorangutan package

blueorangutan · 2024-06-10T15:00:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-06-10T15:04:46Z

Codecov Report

Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 14.95%. Comparing base (0f77019) to head (fd46dac).
Report is 71 commits behind head on 4.19.

Files	Patch %	Lines
...oud/hypervisor/vmware/resource/VmwareResource.java	0.00%	9 Missing ⚠️
...d/hypervisor/vmware/manager/VmwareManagerImpl.java	0.00%	5 Missing ⚠️
...e/image/manager/ImageStoreProviderManagerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.19    #9207       +/-   ##
============================================
+ Coverage     4.30%   14.95%   +10.64%     
- Complexity       0    11013    +11013     
============================================
  Files          363     5387     +5024     
  Lines        29312   470348   +441036     
  Branches      5118    61109    +55991     
============================================
+ Hits          1261    70319    +69058     
- Misses       27908   392237   +364329     
- Partials       143     7792     +7649

Flag	Coverage Δ
uitests	`4.28% <ø> (-0.02%)`	⬇️
unittests	`15.66% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-06-10T16:02:41Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9849

sureshanaparti · 2024-06-10T20:07:04Z

@blueorangutan test

blueorangutan · 2024-06-10T20:08:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr

LGTM - didn't verify it

blueorangutan · 2024-06-11T10:53:20Z

[SF] Trillian test result (tid-10408)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 51619 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9207-t10408-kvm-centos7.zip
Smoke tests completed. 128 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	341.56	test_safe_shutdown.py
test_01_verify_ipv6_vpc	`Failure`	569.16	test_vpc_ipv6.py
test_02_redundant_VPC_default_routes	`Failure`	361.38	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	712.59	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	712.60	test_vpc_redundant.py

DaanHoogland

clgtm

…d secondary storage not available or doesn't have enough capacity

sureshanaparti · 2024-06-12T09:10:45Z

@blueorangutan package

blueorangutan · 2024-06-12T09:12:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-12T10:35:19Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9891

sureshanaparti · 2024-06-13T12:50:03Z

@blueorangutan package

blueorangutan · 2024-06-13T12:52:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

andrijapanicsb

LGTM

blueorangutan · 2024-06-13T13:54:43Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9932

DaanHoogland · 2024-06-13T15:37:35Z

@blueorangutan test alma9 vmware-70u2

blueorangutan · 2024-06-13T15:38:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + vmware-70u2) has been kicked to run smoke tests

blueorangutan · 2024-06-13T15:41:03Z

[SF] Trillian Build Failed (tid-10437)

DaanHoogland · 2024-06-13T15:43:49Z

@blueorangutan test alma9 vmware-70u3

blueorangutan · 2024-06-13T15:46:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2024-06-14T07:12:45Z

[SF] Trillian test result (tid-10439)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server a9
Total time taken: 53804 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9207-t10439-vmware-70u3.zip
Smoke tests completed. 129 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_balanced_drs_algorithm	`Failure`	132.20	test_cluster_drs.py
test_05_rvpc_multi_tiers	`Error`	580.79	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	580.82	test_vpc_redundant.py

sureshanaparti · 2024-06-25T05:33:51Z

@blueorangutan package

blueorangutan · 2024-06-25T05:36:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-25T06:42:48Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10109

…d secondary storage not available or doesn't have enough capacity (apache#9207)

boring-cyborg bot added component:storage component:vmware labels Jun 10, 2024

sureshanaparti mentioned this pull request Jun 10, 2024

VMware: System VMs can't be started when no secondary storage with enough capacity + wrong messages around secondary storage #8390

Closed

sureshanaparti requested review from DaanHoogland and andrijapanicsb June 10, 2024 14:59

yadvr approved these changes Jun 11, 2024

View reviewed changes

sureshanaparti added the status:needs-testing label Jun 11, 2024

DaanHoogland approved these changes Jun 12, 2024

View reviewed changes

Comment thread .../hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/manager/VmwareManagerImpl.java Outdated

Comment thread .../hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/manager/VmwareManagerImpl.java Outdated

sureshanaparti added 2 commits June 12, 2024 14:36

VMware: Improve error messaging / logs when starting non-user VMs, an…

aa3121b

…d secondary storage not available or doesn't have enough capacity

Addressed review comments

6557da6

sureshanaparti force-pushed the vmware-sec-storage-check-improvements branch from 68ffea9 to 6557da6 Compare June 12, 2024 09:09

sureshanaparti marked this pull request as ready for review June 12, 2024 09:10

sureshanaparti added this to the 4.19.1.0 milestone Jun 12, 2024

yadvr linked an issue Jun 13, 2024 that may be closed by this pull request

VMware: System VMs can't be started when no secondary storage with enough capacity + wrong messages around secondary storage #8390

Closed

andrijapanicsb reviewed Jun 13, 2024

View reviewed changes

Comment thread ...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Outdated

andrijapanicsb reviewed Jun 13, 2024

View reviewed changes

Comment thread ...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Outdated

andrijapanicsb reviewed Jun 13, 2024

View reviewed changes

Comment thread ...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Outdated

Addressed review comments

eddbce4

andrijapanicsb approved these changes Jun 13, 2024

View reviewed changes

minor log update

fd46dac

vishesh92 merged commit 620ed16 into apache:4.19 Jun 25, 2024

vishesh92 deleted the vmware-sec-storage-check-improvements branch June 25, 2024 06:55

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jul 2, 2024

VMware: Improve error messaging / logs when starting non-user VMs, an…

a5c0f5d

…d secondary storage not available or doesn't have enough capacity (apache#9207)

DaanHoogland removed the status:needs-testing label Jan 28, 2026

Conversation

sureshanaparti commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Jun 10, 2024

Uh oh!

blueorangutan commented Jun 10, 2024

Uh oh!

codecov bot commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jun 10, 2024

Uh oh!

sureshanaparti commented Jun 10, 2024

Uh oh!

blueorangutan commented Jun 10, 2024

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 11, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 12, 2024

Uh oh!

blueorangutan commented Jun 12, 2024

Uh oh!

blueorangutan commented Jun 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

andrijapanicsb left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

DaanHoogland commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

DaanHoogland commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 14, 2024

Uh oh!

sureshanaparti commented Jun 25, 2024

Uh oh!

blueorangutan commented Jun 25, 2024

Uh oh!

blueorangutan commented Jun 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sureshanaparti commented Jun 10, 2024 •

edited

Loading

codecov bot commented Jun 10, 2024 •

edited

Loading