Management server metrics only shows on 1 GUI at a time

[Tbaugus44]

ISSUE TYPE

• Bug Report

COMPONENT NAME

GUI

CLOUDSTACK VERSION

4.18.1

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

When you go to the infrastructure>management server section and toggle on the metrics switch. We can only see the current metrics of the management vm that you are currently logged into. Both management servers are connected to the same mysql DB.

STEPS TO REPRODUCE

EXPECTED RESULTS

We excepted to see both management server metrics from either managements VMs GUI 

ACTUAL RESULTS

The expected result do not happen

[rajujith]

@Tbaugus44 In my tests I do see the metrics for both management servers.

[DaanHoogland]

my view @Tbaugus44 :

Might there be some connection problem between your MSses?

If you investigate the logs, anything related that you see?

[Tbaugus44]

Could you tell me what logs i need to look at. I am very new to Apachecloudstack and especially mysql. thanks

[rajujith]

@Tbaugus44 You can look for error messages like "Failed to generate zone metrics response".
You can find the below messages otherwise:

2023-12-07 07:31:54,172 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-17:ctx-d36e58f5 ctx-b91e5e56) (logid:13334424) Getting stats for mgmt2.shapeblue.com[c40c8a5b-06a5-45db-aebf-e195f09811b0]
2023-12-07 07:31:54,173 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-17:ctx-d36e58f5 ctx-b91e5e56) (logid:13334424) Status object found for MS mgmt2.shapeblue.com  - com.cloud.server.ManagementServerHostStatsEntry@7df6e68c[agentCount=0,availableProcessors=2,collectionTime=Thu Dec 07 07:31:37 UTC 2023,cpuUtilization=0.0,dbLocal=false,freeJvmMemoryBytes=548813448,heapMemoryTotal=1908932608,heapMemoryUsed=216256336,jvmName=16961@mgmt2,jvmStartTime=1701931499082,jvmUptime=2797982,jvmVendor=Ubuntu,jvmVersion=11.0.21+9-post-Ubuntu-0ubuntu122.04,kernelVersion=5.15.0-35-generic,loadAverage=0.0,logInfo=/var/log/cloudstack/management/management-server.log using: 2.2M

This feature is implemented by #5588 for more information.

[Tbaugus44]

@rajujith this is what i see in the /var/log/cloudstack/management/management-server.log

2023-12-19 16:33:05,892 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2023-12-19 16:33:05,893 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) API rate limiting is disabled. We will not use ApiRateLimitService.
2023-12-19 16:33:05,895 WARN  [c.c.a.d.ParamGenericValidationWorker] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Received unknown parameters for command listManagementServersMetrics. Unknown parameters : listall
2023-12-19 16:33:05,897 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-758e48ce) (logid:816b1eb7) HA health check task is running...
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Getting metrics for 2 MS hosts.
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Processing metrics for MS hosts 75ca0d4d-0ff1-4d61-af60-3d556099cc2e.
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Getting stats for erl-cs-mgt-vm-01[75ca0d4d-0ff1-4d61-af60-3d556099cc2e]
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Status object found for MS erl-cs-mgt-vm-01 - com.cloud.server.ManagementServerHostStatsEntry@39c57a65[agentCount=3,availableProcessors=8,collectionTime=Tue Dec 19 16:32:22 EST 2023,cpuUtilization=0.0,dbLocal=false,freeJvmMemoryBytes=225172320,heapMemoryTotal=1908932608,heapMemoryUsed=538380584,jvmName=87196@ERL-CS-MGT-VM-01,jvmStartTime=1703021457674,jvmUptime=84386,jvmVendor=Ubuntu,jvmVersion=11.0.21+9-post-Ubuntu-0ubuntu122.04,kernelVersion=5.15.0-91-generic,loadAverage=0.603515625,logInfo=/var/log/cloudstack/management/management-server.log using: 78M
on disk /dev/mapper/ubuntu--vg-ubuntu--lv mounted on / (8% full),managementServerHostId=1,managementServerHostUuid=75ca0d4d-0ff1-4d61-af60-3d556099cc2e,maxJvmMemoryBytes=1908932607,osDistribution=Ubuntu 22.04.3 LTS,pid=87196,processJvmMemoryBytes=0,sessions=2,systemBootTime=Mon Dec 18 10:40:57 EST 2023,systemCyclesUsage={224316,62928,85523366},systemLoadAverages={0.6,0.32,0.11},systemMemoryFree=13519716352,systemMemoryTotal=16764178432,systemMemoryUsed=1239116,systemMemoryVirtualSize=8340467712,systemTotalCpuCycles=24741.92,threadsBlockedCount=0,threadsDaemonCount=13,threadsRunnableCount=17,threadsTerminatedCount=0,threadsTotalCount=203,threadsWaitingCount=100,totalCommitted=947888128,totalInit=269811712,totalJvmMemoryBytes=762314752,totalUsed=716017328,usageLocal=false].
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Processing metrics for MS hosts 10defb5c-f546-434f-ac9e-f4f69eb334e9.
2023-12-19 16:33:05,898 DEBUG [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) Getting stats for ERL-CS-MGT-VM-02[10defb5c-f546-434f-ac9e-f4f69eb334e9]
2023-12-19 16:33:05,899 INFO  [o.a.c.m.MetricsServiceImpl] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) No status object found for MS ERL-CS-MGT-VM-02 - 10defb5c-f546-434f-ac9e-f4f69eb334e9.
2023-12-19 16:33:05,901 DEBUG [c.c.a.ApiServlet] (qtp1550261631-319:ctx-203a91a8 ctx-feb9f116) (logid:54559778) ===END===  10.10.34.90 -- GET  listall=true&page=1&pagesize=20&command=listManagementServersMetrics&response=json

[DaanHoogland]

this indeed means there is no data for "the other" host. One thing springs out: erl-cs-mgt-vm-01 vs. ERL-CS-MGT-VM-02 the two MS names are in different case. This would not be an issue in DNS but might be somewhere in the code.

• Can you check the other one to see if you see the same difference?
• Also, can you check what is in the mshosts table?

[rajujith]

@Tbaugus44 like @DaanHoogland mentioned there could be some stale records for the same management server, try this:

select id,msid,name,state,last_update from mshost where removed is null;