Console proxy timeouts when there are multiple subnets

[ravening]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Console Proxy (NoVNC)

CLOUDSTACK VERSION

4.14 onwards

CONFIGURATION

Advanced networking
Multiple subnets for management cidr (172.30.34.* and 172.30.36.*)

OS / ENVIRONMENT

Ubuntu 16 hosts

SUMMARY

The console for a vm timeouts when there are multiple subnets configured.
If we try to access the console of the vm which is running on the host with primary subnet then everything works fine.
If we try to open the console of the vm which is running on the host with second subnet then the console hangs

Looks like console proxy creates a static route for the second subnet which breaks the connection

Moving the vm back to the hypervisor which has the primary subnet, fixes the issue.
This issue was not present in cloudstack 4.7 version but the issues exists in 4.14 onwards. It probably might exist in 4.11 also

STEPS TO REPRODUCE

1. Configure multiple subnets for the management cidr (172.30.34.* and 172.30.26.*)
2. In our case few hypervisors had 172.30.34* subnet configured and few others had 172.30.36.*
3. Here 172.30.34.* is the main subnet and 172.30.26.* is second subnet
4. Deploy a VM on host with 172.30.34* and 172.30.36.* subnets
5. open the console for the console proxy vm
6. Now try to open the console for the vm running on host with subnet 172.30.34.* This works fine
7. Now open the console for the vm running on the host with 172.30.36.* as the subnet
8. The console hangs

Below is the routing table

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         37.48.69.14     0.0.0.0         UG    0      0        0 eth2
10.0.0.0        172.30.34.254   255.0.0.0       UG    0      0        0 eth1
37.48.69.0      0.0.0.0         255.255.255.240 U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.16.0.0      172.30.34.254   255.240.0.0     UG    0      0        0 eth1
172.30.34.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
172.30.36.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
192.168.0.0     172.30.34.254   255.255.0.0     UG    0      0        0 eth1


Below are the Ip address of the hosts where vm is running and these routes are created the moment we tried to access console of that vm. 

So these routes should not be created. When we delete this route manually, it works fine.

If we do a refresh, these routes are added again and it breaks

172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1

EXPECTED RESULTS

The console for all vm's running on all hosts should work fine without any timeout

ACTUAL RESULTS

The console for the vm's running on the hosts with secondary subnet timeouts/hangs

[ravening]

@weizhouapache @rhtyd have you guys faces this issue?

[weizhouapache]

@ravening is it related to your pr #4678 ?

[ravening]

@ravening is it related to your pr #4678 ?

@weizhouapache no... because that pr adds the cidr to the route but not the specific ip address.
if you see the route -n output, we can see that only specific ip routes are added when we access the console

[ravening]

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

[weizhouapache]

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

@ravening
it is added by java

    public void ensureRoute(String address) {
        if (_localgw != null) {
            if (s_logger.isDebugEnabled())
                s_logger.debug("Ensure route for " + address + " via " + _localgw);

            // this method won't be called in high frequency, serialize access
            // to script execution
            synchronized (this) {
                try {
                    addRouteToInternalIpOrCidr(_localgw, _eth1ip, _eth1mask, address);
                } catch (Throwable e) {
                    s_logger.warn("Unexpected exception while adding internal route to " + address, e);
                }
            }
        }
    }

[weizhouapache]

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

@ravening
I think you could consult networking engineers how to solve this issue in switch configurations

[ravening]

@weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also

[yadvr]

Never seen/heard it @ravening

[weizhouapache]

@weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also

@ravening
the issue might exist with legacy vnc console and new novnc console introduced in 4.14

ConsoleProxy.ensureRoute(uri.getHost());

You did not see the issue because you did not use legacy vnc console but use a different novnc implementation.

I suggest you to fix it by network configuration changes.

[nvazquez]

Hi @ravening is this issue still valid?

[ravening]

@nvazquez no.. I will close this