[Bug report] LDAP timeout making users unable to login until full restart #201
Description
Currently using Confighub configured with LDAP deployed in a container.
We have several other tools working great with LDAP configured with same LDAP server and same network.
Only tool misbehaving is Confighub that needs several restarts daily causing downtime.
We have a workaround for LDAP timeout that restarts Confighub used in a container with liveness probe that validates that LDAP login works in order for LDAP logins to work.
LDAP users and passwords aren't changing for the relevant users. LDAP is configured correctly.
Login with local users and token is ok.
After timing out once, LDAP users are not able to login and logs for these failed attempts do not appear.
Doing the connectivity and login test in admin panel shows that LDAP login work (this appears in log) but users are not able to login and these attempts do not show in logs at all.
We would like to at least have the connection timeout be configurable in here:
core/src/main/java/com/confighub/core/auth/LdapConnector.java
private final int connectionTimeout = 5000;
Not sure what improvement in LDAP code or configuration is necessary for logins to work after a timeout without needing a Confighub restart.
Log of relevant error:
2023-05-28 20:01:55,543 INFO Binding DN ---snip--- did not throw, connection authenticated: true
2023-05-28 20:02:25,663 INFO Connecting to LDAP server ---snip---, binding with user ---snip---
2023-05-28 20:02:30,665 ERROR Timed out connecting to LDAP server
com.google.common.util.concurrent.UncheckedTimeoutException: java.util.concurrent.TimeoutException
at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:133) ~[guava-23.0.jar:?]
at com.google.common.util.concurrent.SimpleTimeLimiter.access$100(SimpleTimeLimiter.java:47) ~[guava-23.0.jar:?]
at com.google.common.util.concurrent.SimpleTimeLimiter$1.invoke(SimpleTimeLimiter.java:101) ~[guava-23.0.jar:?]
at com.sun.proxy.$Proxy167.call(Unknown Source) ~[?:?]
at com.confighub.core.auth.LdapConnector.connect(LdapConnector.java:77) [ConfigHub-Core-1.8.3.jar:?]
at com.confighub.core.auth.Auth.getLdapNetworkConnection(Auth.java:118) [ConfigHub-Core-1.8.3.jar:?]
at com.confighub.core.auth.Auth.ldapAuth(Auth.java:170) [ConfigHub-Core-1.8.3.jar:?]
at com.confighub.api.auth.Login.login(Login.java:79) [Login.class:?]
at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]
at org.apache.openejb.server.cxf.rs.PojoInvoker.performInvocation(PojoInvoker.java:43) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.6.16.jar:2.6.16]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89) [cxf-rt-frontend-jaxrs-2.6.16.jar:2.6.16]
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:68) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:240) [cxf-rt-transports-http-2.6.16.jar:2.6.16]
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.doInvoke(CxfRsHttpListener.java:227) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.tomee.webservices.CXFJAXRSFilter.doFilter(CXFJAXRSFilter.java:94) [tomee-jaxrs-1.7.3.jar:1.7.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at com.confighub.api.server.filters.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:66) [UrlRewriteFilter.class:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.63]
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44) [tomee-catalina-1.7.3.jar:1.7.3]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.63]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.63]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.63]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) [catalina.jar:7.0.63]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) [tomcat-coyote.jar:7.0.63]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) [tomcat-coyote.jar:7.0.63]
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) [tomcat-coyote.jar:7.0.63]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111-internal]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.63]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111-internal]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_111-internal]
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:183) ~[guava-23.0.jar:?]
at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:127) ~[guava-23.0.jar:?]
... 46 more
2023-05-28 20:02:30,680 ERROR Failed to connect to LDAP: Could not connect to LDAP server
2023-05-28 20:02:30,680 INFO Search LDAP for (&(objectClass=person)(corpUidLocal=---snip---)), starting at ---snip---
2023-05-28 20:02:30,681 ERROR Failed to auth username: ---snip--- to LDAP
java.lang.NullPointerException: null
at com.confighub.core.auth.LdapConnector.search(LdapConnector.java:114) ~[ConfigHub-Core-1.8.3.jar:?]
at com.confighub.core.auth.Auth.ldapAuth(Auth.java:172) [ConfigHub-Core-1.8.3.jar:?]
at com.confighub.api.auth.Login.login(Login.java:79) [Login.class:?]
at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111-internal]
at org.apache.openejb.server.cxf.rs.PojoInvoker.performInvocation(PojoInvoker.java:43) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.6.16.jar:2.6.16]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89) [cxf-rt-frontend-jaxrs-2.6.16.jar:2.6.16]
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:68) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api-2.6.16.jar:2.6.16]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:240) [cxf-rt-transports-http-2.6.16.jar:2.6.16]
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.doInvoke(CxfRsHttpListener.java:227) [openejb-cxf-rs-4.7.3.jar:4.7.3]
at org.apache.tomee.webservices.CXFJAXRSFilter.doFilter(CXFJAXRSFilter.java:94) [tomee-jaxrs-1.7.3.jar:1.7.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at com.confighub.api.server.filters.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:66) [UrlRewriteFilter.class:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.63]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.63]
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44) [tomee-catalina-1.7.3.jar:1.7.3]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.63]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.63]
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957) [catalina.jar:7.0.63]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.63]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) [catalina.jar:7.0.63]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079) [tomcat-coyote.jar:7.0.63]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620) [tomcat-coyote.jar:7.0.63]
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) [tomcat-coyote.jar:7.0.63]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111-internal]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.63]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111-internal]
Screenshot after first timeout with LDAP login:
Screenshot from admin panel for LDAP:
Sorry for the long post.
Feel free to ask more details or possibly some pull request.
Appreciate any help.