Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: 1.0.1
    • Fix Version/s: None
    • Component/s: Transport
    • Labels:
      None
    • Global Rank:
      3979

      Activity

      Hide
      rschlege added a comment -

      Summary of thread:

      streaming speed seems to be depending on the environment. I used

      Windows Server 2003 Standard Edition
      Intel Core 2 Duo 2.4GHz
      2GB RAM
      java version "1.5.0_12"
      

      Maximum streaming speed (according to Firefox): only about ~260KB/s
      Compared to copying the artifact in the webapp folder and downloading from there (Firefox): ~5MB/s


      Performance with

      MacBook Pro / Mac OSX
      Java 1.5
      

      according to Tamas Cservenak:

      cstamas@Marvin tmp]$ curl -O http://localhost:8081/nexus/content/r...
      % Total % Received % Xferd Average Speed Time Time Time Current
      Dload Upload Total Spent Left Speed
      100 120M 100 120M 0 0 38.5M 0 0:00:03 0:00:03 --:--:-- 38.6M
      [cstamas@Marvin tmp]$
      
      Show
      rschlege added a comment - Summary of thread: streaming speed seems to be depending on the environment. I used Windows Server 2003 Standard Edition Intel Core 2 Duo 2.4GHz 2GB RAM java version "1.5.0_12" Maximum streaming speed (according to Firefox): only about ~260KB/s Compared to copying the artifact in the webapp folder and downloading from there (Firefox): ~5MB/s Performance with MacBook Pro / Mac OSX Java 1.5 according to Tamas Cservenak: cstamas@Marvin tmp]$ curl -O http://localhost:8081/nexus/content/r... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 120M 100 120M 0 0 38.5M 0 0:00:03 0:00:03 --:--:-- 38.6M [cstamas@Marvin tmp]$
      Hide
      rschlege added a comment - - edited

      Update:

      If I work with http://localhost:8081/nexus/content or with my own IP I also get the same speed like from the webapp directory. Only when downloading from a different machine from where I can't use localhost it's really slow!

      Show
      rschlege added a comment - - edited Update: If I work with http://localhost:8081/nexus/content or with my own IP I also get the same speed like from the webapp directory. Only when downloading from a different machine from where I can't use localhost it's really slow!
      Hide
      rschlege added a comment -

      Tested now on Solaris. From that machine, speed is excellent!
      So it definitely seems to be an M$ related issue.

      But still, in my case this is blocking the usage of Nexus because the storage is on a Windows server and currently can't be mounted to the Solaris machine.

      Show
      rschlege added a comment - Tested now on Solaris. From that machine, speed is excellent! So it definitely seems to be an M$ related issue. But still, in my case this is blocking the usage of Nexus because the storage is on a Windows server and currently can't be mounted to the Solaris machine.
      Hide
      rschlege added a comment -

      Ok, it's definitely something which is installed on our Windows machines. Tried at home on Win XP and that works just fine. Don't know what it is exactly, but I guess you can't do anything about it on Nexus side.

      Show
      rschlege added a comment - Ok, it's definitely something which is installed on our Windows machines. Tried at home on Win XP and that works just fine. Don't know what it is exactly, but I guess you can't do anything about it on Nexus side.
      Show
      Brian Fox added a comment - This has been reported again here: http://www.nabble.com/Nexus-artifact-downloads-are-slow-%28throttled--%29-tt20137516ef34838.html
      Hide
      rschlege added a comment - - edited

      Copying the speed test information:

      server client speed
      XP notebook XP box fast
      XP box XP notebook slow
      XP notebook linux box fast
      windows 2003 XP notebook slow
      windows 2003 XP box slow
      windows 2003 linux box fast
      XP box linux box fast
      solaris windows 2003 fast
      solaris XP notebook fast
      solaris XP box fast

      Notes:

      • all tests have been on the same subnetwork.
      • "fast" means ~10MB/s (which is ~10'240 KB/s)
      • "slow" means ~300KB/s or ~800KB/s when tweaking the buffers
      Show
      rschlege added a comment - - edited Copying the speed test information: server client speed XP notebook XP box fast XP box XP notebook slow XP notebook linux box fast windows 2003 XP notebook slow windows 2003 XP box slow windows 2003 linux box fast XP box linux box fast solaris windows 2003 fast solaris XP notebook fast solaris XP box fast Notes: all tests have been on the same subnetwork. "fast" means ~10MB/s (which is ~10'240 KB/s) "slow" means ~300KB/s or ~800KB/s when tweaking the buffers
      Hide
      Eric Miles added a comment -

      We too are having this issue with 1.4.0 hosted on a Win2k3 server using the embedded Jetty server. All our clients are XP workstations, so downloads are generally in the ~150KB/s range. Any solution for this?

      Show
      Eric Miles added a comment - We too are having this issue with 1.4.0 hosted on a Win2k3 server using the embedded Jetty server. All our clients are XP workstations, so downloads are generally in the ~150KB/s range. Any solution for this?
      Hide
      Duane Zamrok added a comment -

      Eric, you might want to look at http://nexus.sonatype.org/jetty-configuration.html#10. I don't know if this will solve your problem, but it might help.

      Show
      Duane Zamrok added a comment - Eric, you might want to look at http://nexus.sonatype.org/jetty-configuration.html#10 . I don't know if this will solve your problem, but it might help.
      Hide
      Sebastien Tardif added a comment -

      This is the worst Nexus bug for us. It breaks our build, we cannot build without avoiding calling Nexus for some of the artifact.

      From the profiling below we know that if we would be reading the file at the same time we are writing, that should execute at least 30% faster.

      This is the code profiling against Nexus version 1.9.2.3 running on Tomcat 7.0.22, on Windows Server sp2 Version 6.0.6002. The client is on Linux.

      The stats are:

      3 times on 10, we are reading from the file:
      "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4e000]
      java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:198)
      at org.sonatype.nexus.util.WrappingInputStream.read(WrappingInputStream.java:58)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
      at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
      at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
      at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)

      7 times on 10, we are writing to the output stream:
      "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4d000]
      java.lang.Thread.State: RUNNABLE
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:216)
      at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
      at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
      at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:241)
      at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
      at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
      at org.apache.coyote.Response.doWrite(Response.java:533)
      at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
      at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
      at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
      at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
      at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
      at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:190)
      at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
      at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
      at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
      at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
      at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421)
      at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:392)
      at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
      at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

      Show
      Sebastien Tardif added a comment - This is the worst Nexus bug for us. It breaks our build, we cannot build without avoiding calling Nexus for some of the artifact. From the profiling below we know that if we would be reading the file at the same time we are writing, that should execute at least 30% faster. This is the code profiling against Nexus version 1.9.2.3 running on Tomcat 7.0.22, on Windows Server sp2 Version 6.0.6002. The client is on Linux. The stats are: 3 times on 10, we are reading from the file: "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:198) at org.sonatype.nexus.util.WrappingInputStream.read(WrappingInputStream.java:58) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174) at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78) at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492) at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428) 7 times on 10, we are writing to the output stream: "http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4d000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:216) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:241) at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84) at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192) at org.apache.coyote.Response.doWrite(Response.java:533) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:190) at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174) at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78) at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492) at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428) at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421) at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:392) at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148) at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
      Hide
      Sebastien Tardif added a comment -

      Beside doing parallel read/write, it seems that the main fix would be to call http://tomcat.apache.org/tomcat-5.5-doc/servletapi/javax/servlet/ServletResponse.html#setBufferSize(int) with a much bigger number than the default.

      The code that should be doing this could be com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421):

      Current version just take the OutputStream:
      public OutputStream getResponseEntityStream() {
      try

      { return getResponse().getOutputStream(); }

      catch (IOException e)

      { return null; }

      }

      Show
      Sebastien Tardif added a comment - Beside doing parallel read/write, it seems that the main fix would be to call http://tomcat.apache.org/tomcat-5.5-doc/servletapi/javax/servlet/ServletResponse.html#setBufferSize(int ) with a much bigger number than the default. The code that should be doing this could be com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421): Current version just take the OutputStream: public OutputStream getResponseEntityStream() { try { return getResponse().getOutputStream(); } catch (IOException e) { return null; } }
      Hide
      Rich Seddon added a comment -

      When this was tested by the Jetty team they found a smaller buffer size actually works best: http://docs.codehaus.org/display/JETTY/Slow+Win2003

      If you can switch to Jetty then you can fix this as described in that link.

      For Jetty 7 the equivalent configuration is:

          <Call name="addConnector">
              <Arg>
                  <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
                    <Set name="host">${application-host}</Set>
                    <Set name="port">${application-port}</Set>
                    
                    <!-- Documentation for this fix can be found at:
                         http://docs.codehaus.org/display/JETTY/Slow+Win2003
                    -->
                    <Set name="responseBufferSize">512</Set>
                  </New>
              </Arg>
          </Call>
      
      Show
      Rich Seddon added a comment - When this was tested by the Jetty team they found a smaller buffer size actually works best: http://docs.codehaus.org/display/JETTY/Slow+Win2003 If you can switch to Jetty then you can fix this as described in that link. For Jetty 7 the equivalent configuration is: <Call name= "addConnector" > <Arg> <New class= "org.eclipse.jetty.server.nio.SelectChannelConnector" > <Set name= "host" > ${application-host} </Set> <Set name= "port" > ${application-port} </Set> <!-- Documentation for this fix can be found at: http://docs.codehaus.org/display/JETTY/Slow+Win2003 --> <Set name= "responseBufferSize" > 512 </Set> </New> </Arg> </Call>
      Hide
      Sebastien Tardif added a comment -

      I just tried the equivalent settings for Tomcat (smaller buffer), and it's improving the performance. See socketBuffer at http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Java_TCP_socket_attributes

      Still, Nexus is about streaming artifact. It's kind of pathetic that this bug is still not fixed after all this years.

      It's the responsibility of Nexus to set the right buffer size using api ServletResponse.setBufferSize. It's a common practice with JSP. The right buffer size is use case specific.

      The code should also be optimized to read and write at the same time. From previous profiling I have provided, where the slowness side was writing. After I optimize writing using more appropriate buffer size, I raise the potential improvement that will bring having concurrent read/write.

      Show
      Sebastien Tardif added a comment - I just tried the equivalent settings for Tomcat (smaller buffer), and it's improving the performance. See socketBuffer at http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Java_TCP_socket_attributes Still, Nexus is about streaming artifact. It's kind of pathetic that this bug is still not fixed after all this years. It's the responsibility of Nexus to set the right buffer size using api ServletResponse.setBufferSize. It's a common practice with JSP. The right buffer size is use case specific. The code should also be optimized to read and write at the same time. From previous profiling I have provided, where the slowness side was writing. After I optimize writing using more appropriate buffer size, I raise the potential improvement that will bring having concurrent read/write.
      Hide
      Rich Seddon added a comment -

      No idea why this is in such an odd state. I'm closing this out, most of the problems on this issue were due to a windows bug, and the workaround is above.

      There is an issue open against 2.7 for a potential regression here: NEXUS-5790

      Show
      Rich Seddon added a comment - No idea why this is in such an odd state. I'm closing this out, most of the problems on this issue were due to a windows bug, and the workaround is above. There is an issue open against 2.7 for a potential regression here: NEXUS-5790

        People

        • Assignee:
          Rich Seddon
          Reporter:
          Brian Fox
          Last Updated By:
          Rich Seddon
        • Votes:
          4 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:
            Date of First Response: