Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.14.20, 2.14.21
-
3
-
3
Description
We are seeing instances of Nexus 2.x hanging and becoming unresponsive, requiring a restart. In all reported cases so far two or more of the following conditions were present:
- HTTPS connector being used in Eclipse Jetty
- older versions of Java 8 were being used
- Nexus Repository versions using Eclipse Jetty 8 ( 2.14.* and earlier )
Looking at thread dumps taken during time of issue, we see a large number of runnable threads like below:
"qtp2074358686-380177" #380177 prio=5 os_prio=0 tid=0x00007ff89260b800 nid=0x4a7b runnable [0x00007ff7ebaf9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.preClose0(Native Method) at sun.nio.ch.SocketDispatcher.preClose(Unknown Source) at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown Source) - locked <0x0000000658795570> (a java.lang.Object) at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown Source) at java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown Source) - locked <0x00000006587954a8> (a java.lang.Object) at org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:210) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:796) at org.eclipse.jetty.io.nio.ChannelEndPoint.shutdownChannelOutput(ChannelEndPoint.java:179) at org.eclipse.jetty.io.nio.ChannelEndPoint.shutdownOutput(ChannelEndPoint.java:191) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:412) - locked <0x00000006587956e0> (a org.eclipse.jetty.io.nio.SslConnection) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:48) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:678) at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1044) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source)
The above is related to below jetty issue:
https://github.com/eclipse/jetty.project/issues/4717
We have seen this issue before and the fix seems to require using Java 11 to run Jetty.
There is a plan to get Nexus Repo 3.x working with Java 11, but there are currently no plans to get Nexus Repo 2 working with Java 11.
Suggested Mitigation
It is advisable if such an issue is encountered, that these steps be taken by customers.
- upgrade to the most recent version of Java 8 to help reduce the likelihood of the issue.
- upgrade to Nexus Repo 2.15.1 or newer so that the latest Jetty 9 version is used ( see
NEXUS-21370) which is the same version of Jetty that Nexus Repo 3 uses and where this issue has NOT been noticed - terminate HTTPS connections outside Eclipse Jetty, for example using a reverse proxy to do that