Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18952

High CPU usage observed caused by a bug in older Jetty version

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.14.0, 3.15.1
    • Fix Version/s: 3.16.0
    • Component/s: Transport
    • Labels:
    • Story Points:
      1

      Description

      Customer complained of high CPU usage in Nexus Repo 3.14.0.  A thread dump from their instance showed the following active Jetty threads:

      qtp2102447597-846 id=846 state=RUNNABLE
       at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:547)
       at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:54)
       at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:332)
       at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
       at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:804)
       at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
       at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
       at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538)
       at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:795)
       at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:846)
       at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240)
       at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216)
       at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:515)
       at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167)
       at org.eclipse.jetty.server.Utf8HttpWriter.write(Utf8HttpWriter.java:183)
       at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:71)
       at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:65)
       at org.eclipse.jetty.server.ResponseWriter.write(ResponseWriter.java:231)
       - locked org.eclipse.jetty.server.Utf8HttpWriter@fe3f522
       at org.eclipse.jetty.server.ResponseWriter.write(ResponseWriter.java:248)
       at com.softwarementors.extjs.djn.router.processor.poll.PollRequestProcessor.process(PollRequestProcessor.java:157)
       at org.sonatype.nexus.extdirect.internal.ExtDirectServlet$4.processPollRequest(ExtDirectServlet.java:385)
       at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.processRequest(DirectJNgineServlet.java:636)
       at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.doPost(DirectJNgineServlet.java:595)
       at org.sonatype.nexus.extdirect.internal.ExtDirectServlet.doPost(ExtDirectServlet.java:155)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      
      qtp2102447597-971 id=971 state=RUNNABLE
       at java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
       at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
       at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
       at java.lang.ThreadLocal.get(ThreadLocal.java:163)
       at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:547)
       at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:54)
       at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:332)
       at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
       at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:804)
       at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
       at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
       at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538)
       at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:795)
      

       

      This is due to a bug in Jetty 9.4.11:

      https://github.com/eclipse/jetty.project/issues/2233#issuecomment-404082685

      This has been fixed in 9.4.12:

      https://github.com/eclipse/jetty.project/commit/17b6eee5aca00460913a2b7847325b6e3df39fd2

      So bumping our Jetty version should prevent future occurrences of this.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch Peter Lynch
              Team:
              NXRM - Morpheus
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title