Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: 1.0.1
    • Fix Version/s: None
    • Component/s: Transport
    • Labels:
      None
    • Global Rank:
      3979

      Activity

      Hide
      Sebastien Tardif added a comment -

      This is the worst Nexus bug for us. It breaks our build, we cannot build without avoiding calling Nexus for some of the artifact.

      From the profiling below we know that if we would be reading the file at the same time we are writing, that should execute at least 30% faster.

      This is the code profiling against Nexus version 1.9.2.3 running on Tomcat 7.0.22, on Windows Server sp2 Version 6.0.6002. The client is on Linux.

      The stats are:

      3 times on 10, we are reading from the file:
      "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4e000]
      java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:198)
      at org.sonatype.nexus.util.WrappingInputStream.read(WrappingInputStream.java:58)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
      at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
      at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
      at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)

      7 times on 10, we are writing to the output stream:
      "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4d000]
      java.lang.Thread.State: RUNNABLE
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:216)
      at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
      at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
      at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:241)
      at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
      at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
      at org.apache.coyote.Response.doWrite(Response.java:533)
      at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
      at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
      at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
      at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
      at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
      at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:190)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
      at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
      at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
      at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
      at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421)
      at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:392)
      at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
      at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

      Show
      Sebastien Tardif added a comment - This is the worst Nexus bug for us. It breaks our build, we cannot build without avoiding calling Nexus for some of the artifact. From the profiling below we know that if we would be reading the file at the same time we are writing, that should execute at least 30% faster. This is the code profiling against Nexus version 1.9.2.3 running on Tomcat 7.0.22, on Windows Server sp2 Version 6.0.6002. The client is on Linux. The stats are: 3 times on 10, we are reading from the file: "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:198) at org.sonatype.nexus.util.WrappingInputStream.read(WrappingInputStream.java:58) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174) at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78) at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492) at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428) 7 times on 10, we are writing to the output stream: "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4d000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:216) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:241) at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84) at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192) at org.apache.coyote.Response.doWrite(Response.java:533) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:190) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174) at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78) at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492) at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428) at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421) at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:392) at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148) at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
      Hide
      Sebastien Tardif added a comment -

      Beside doing parallel read/write, it seems that the main fix would be to call http://tomcat.apache.org/tomcat-5.5-doc/servletapi/javax/servlet/ServletResponse.html#setBufferSize(int) with a much bigger number than the default.

      The code that should be doing this could be com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421):

      Current version just take the OutputStream:
      public OutputStream getResponseEntityStream() {
      try

      { return getResponse().getOutputStream(); }

      catch (IOException e)

      { return null; }

      }

      Show
      Sebastien Tardif added a comment - Beside doing parallel read/write, it seems that the main fix would be to call http://tomcat.apache.org/tomcat-5.5-doc/servletapi/javax/servlet/ServletResponse.html#setBufferSize(int ) with a much bigger number than the default. The code that should be doing this could be com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421): Current version just take the OutputStream: public OutputStream getResponseEntityStream() { try { return getResponse().getOutputStream(); } catch (IOException e) { return null; } }
      Hide
      Rich Seddon added a comment -

      When this was tested by the Jetty team they found a smaller buffer size actually works best: http://docs.codehaus.org/display/JETTY/Slow+Win2003

      If you can switch to Jetty then you can fix this as described in that link.

      For Jetty 7 the equivalent configuration is:

          <Call name="addConnector">
              <Arg>
                  <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
                    <Set name="host">${application-host}</Set>
                    <Set name="port">${application-port}</Set>
                    
                    <!-- Documentation for this fix can be found at:
                         http://docs.codehaus.org/display/JETTY/Slow+Win2003
                    -->
                    <Set name="responseBufferSize">512</Set>
                  </New>
              </Arg>
          </Call>
      
      Show
      Rich Seddon added a comment - When this was tested by the Jetty team they found a smaller buffer size actually works best: http://docs.codehaus.org/display/JETTY/Slow+Win2003 If you can switch to Jetty then you can fix this as described in that link. For Jetty 7 the equivalent configuration is: <Call name= "addConnector" > <Arg> <New class= "org.eclipse.jetty.server.nio.SelectChannelConnector" > <Set name= "host" > ${application-host} </Set> <Set name= "port" > ${application-port} </Set> <!-- Documentation for this fix can be found at: http://docs.codehaus.org/display/JETTY/Slow+Win2003 --> <Set name= "responseBufferSize" > 512 </Set> </New> </Arg> </Call>
      Hide
      Sebastien Tardif added a comment -

      I just tried the equivalent settings for Tomcat (smaller buffer), and it's improving the performance. See socketBuffer at http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Java_TCP_socket_attributes

      Still, Nexus is about streaming artifact. It's kind of pathetic that this bug is still not fixed after all this years.

      It's the responsibility of Nexus to set the right buffer size using api ServletResponse.setBufferSize. It's a common practice with JSP. The right buffer size is use case specific.

      The code should also be optimized to read and write at the same time. From previous profiling I have provided, where the slowness side was writing. After I optimize writing using more appropriate buffer size, I raise the potential improvement that will bring having concurrent read/write.

      Show
      Sebastien Tardif added a comment - I just tried the equivalent settings for Tomcat (smaller buffer), and it's improving the performance. See socketBuffer at http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Java_TCP_socket_attributes Still, Nexus is about streaming artifact. It's kind of pathetic that this bug is still not fixed after all this years. It's the responsibility of Nexus to set the right buffer size using api ServletResponse.setBufferSize. It's a common practice with JSP. The right buffer size is use case specific. The code should also be optimized to read and write at the same time. From previous profiling I have provided, where the slowness side was writing. After I optimize writing using more appropriate buffer size, I raise the potential improvement that will bring having concurrent read/write.
      Hide
      Rich Seddon added a comment -

      No idea why this is in such an odd state. I'm closing this out, most of the problems on this issue were due to a windows bug, and the workaround is above.

      There is an issue open against 2.7 for a potential regression here: NEXUS-5790

      Show
      Rich Seddon added a comment - No idea why this is in such an odd state. I'm closing this out, most of the problems on this issue were due to a windows bug, and the workaround is above. There is an issue open against 2.7 for a potential regression here: NEXUS-5790

        People

        • Assignee:
          Rich Seddon
          Reporter:
          Brian Fox
          Last Updated By:
          Rich Seddon
        • Votes:
          4 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:
            Date of First Response: