Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-30865

Runnable Jetty threads on "sun.nio.ch.FileDispatcherImpl.preClose0"

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.14.20, 2.14.21
    • Fix Version/s: 3.0.0, 2.15.1
    • Component/s: Transport
    • Labels:
    • Notability:
      3

      Description

      We are seeing instances of Nexus 2.x hanging and becoming unresponsive, requiring a restart. In all reported cases so far two or more of the following conditions were present:

      • HTTPS connector being used in Eclipse Jetty
      • older versions of Java 8 were being used
      • Nexus Repository versions using Eclipse Jetty 8 ( 2.14.* and earlier )

      Looking at thread dumps taken during time of issue, we see a large number of runnable threads like below:

      "qtp2074358686-380177" #380177 prio=5 os_prio=0 tid=0x00007ff89260b800 nid=0x4a7b runnable [0x00007ff7ebaf9000]
      java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.FileDispatcherImpl.preClose0(Native Method)
      at sun.nio.ch.SocketDispatcher.preClose(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown Source)
      - locked <0x0000000658795570> (a java.lang.Object)
      at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown Source)
      at java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown Source)
      - locked <0x00000006587954a8> (a java.lang.Object)
      at org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:210)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:796)
      at org.eclipse.jetty.io.nio.ChannelEndPoint.shutdownChannelOutput(ChannelEndPoint.java:179)
      at org.eclipse.jetty.io.nio.ChannelEndPoint.shutdownOutput(ChannelEndPoint.java:191)
      at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:412)
      - locked <0x00000006587956e0> (a org.eclipse.jetty.io.nio.SslConnection)
      at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:48)
      at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:678)
      at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1044)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      at java.lang.Thread.run(Unknown Source)
      

       

      The above is related to below jetty issue:

      https://github.com/eclipse/jetty.project/issues/4717

      We have seen this issue before and the fix seems to require using Java 11 to run Jetty.

      There is a plan to get Nexus Repo 3.x working with Java 11, but there are currently no plans to get Nexus Repo 2 working with Java 11.  

      Suggested Mitigation

      It is advisable if such an issue is encountered, that these steps be taken by customers.

      • upgrade to the most recent version of Java 8 to help reduce the likelihood of the issue.
      • upgrade to Nexus Repo 2.15.1 or newer so that the latest Jetty 9 version is used ( see NEXUS-21370 ) which is the same version of Jetty that Nexus Repo 3 uses and where this issue has NOT been noticed
      • terminate HTTPS connections outside Eclipse Jetty, for example using a reverse proxy to do that
         

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            msurani Mahendra Surani
            Last Updated By:
            Peter Lynch Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title