[NEXUS-812] Improve artifact streaming speed Created: 09/17/08  Updated: 08/09/16  Resolved: 08/13/13

Status: Closed
Project: Dev - Nexus Repo
Component/s: Transport
Affects Version/s: 1.0.1
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: Brian Fox Assignee: Rich Seddon
Resolution: Done Votes: 4
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relates
Sub-Tasks:
Key
Summary
Type
Status
Assignee
NEXUS-960 switch to newer version of jetty Sub-task Closed  
Last Updated By: Peter Lynch
Date of First Response:

 Description   

See this thread for more details: http://getsatisfaction.com/sonatype/topics/very_bad_performance_for_large_artifacts



 Comments   
Comment by rschlege [ 09/23/08 ]

Summary of thread:

streaming speed seems to be depending on the environment. I used

Windows Server 2003 Standard Edition
Intel Core 2 Duo 2.4GHz
2GB RAM
java version "1.5.0_12"

Maximum streaming speed (according to Firefox): only about ~260KB/s
Compared to copying the artifact in the webapp folder and downloading from there (Firefox): ~5MB/s


Performance with

MacBook Pro / Mac OSX
Java 1.5

according to Tamas Cservenak:

cstamas@Marvin tmp]$ curl -O http://localhost:8081/nexus/content/r...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 120M 100 120M 0 0 38.5M 0 0:00:03 0:00:03 --:--:-- 38.6M
[cstamas@Marvin tmp]$
Comment by rschlege [ 09/24/08 ]

Update:

If I work with http://localhost:8081/nexus/content or with my own IP I also get the same speed like from the webapp directory. Only when downloading from a different machine from where I can't use localhost it's really slow!

Comment by rschlege [ 09/24/08 ]

Tested now on Solaris. From that machine, speed is excellent!
So it definitely seems to be an M$ related issue.

But still, in my case this is blocking the usage of Nexus because the storage is on a Windows server and currently can't be mounted to the Solaris machine.

Comment by rschlege [ 09/24/08 ]

Ok, it's definitely something which is installed on our Windows machines. Tried at home on Win XP and that works just fine. Don't know what it is exactly, but I guess you can't do anything about it on Nexus side.

Comment by Brian Fox [ 10/23/08 ]

This has been reported again here: http://www.nabble.com/Nexus-artifact-downloads-are-slow-%28throttled--%29-tt20137516ef34838.html

Comment by rschlege [ 11/27/08 ]

Copying the speed test information:

server client speed
XP notebook XP box fast
XP box XP notebook slow
XP notebook linux box fast
windows 2003 XP notebook slow
windows 2003 XP box slow
windows 2003 linux box fast
XP box linux box fast
solaris windows 2003 fast
solaris XP notebook fast
solaris XP box fast

Notes:

  • all tests have been on the same subnetwork.
  • "fast" means ~10MB/s (which is ~10'240 KB/s)
  • "slow" means ~300KB/s or ~800KB/s when tweaking the buffers
Comment by Eric Miles [ 12/03/09 ]

We too are having this issue with 1.4.0 hosted on a Win2k3 server using the embedded Jetty server. All our clients are XP workstations, so downloads are generally in the ~150KB/s range. Any solution for this?

Comment by Duane Zamrok [ 02/11/10 ]

Eric, you might want to look at http://nexus.sonatype.org/jetty-configuration.html#10. I don't know if this will solve your problem, but it might help.

Comment by Sebastien Tardif [ 02/22/12 ]

This is the worst Nexus bug for us. It breaks our build, we cannot build without avoiding calling Nexus for some of the artifact.

From the profiling below we know that if we would be reading the file at the same time we are writing, that should execute at least 30% faster.

This is the code profiling against Nexus version 1.9.2.3 running on Tomcat 7.0.22, on Windows Server sp2 Version 6.0.6002. The client is on Linux.

The stats are:

3 times on 10, we are reading from the file:
"http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4e000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:198)
at org.sonatype.nexus.util.WrappingInputStream.read(WrappingInputStream.java:58)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)

7 times on 10, we are writing to the output stream:
"http-bio-8080-exec-9" daemon prio=6 tid=0x0000000007e90000 nid=0x17d4 runnable [0x0000000013d4d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:216)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:241)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:533)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:93)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:190)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
at org.sonatype.nexus.rest.StorageFileItemRepresentation.write(StorageFileItemRepresentation.java:78)
at com.noelios.restlet.http.HttpServerCall.writeResponseBody(HttpServerCall.java:492)
at com.noelios.restlet.http.HttpServerCall.sendResponse(HttpServerCall.java:428)
at com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421)
at com.noelios.restlet.http.HttpServerConverter.commit(HttpServerConverter.java:392)
at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:148)
at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

Comment by Sebastien Tardif [ 02/22/12 ]

Beside doing parallel read/write, it seems that the main fix would be to call http://tomcat.apache.org/tomcat-5.5-doc/servletapi/javax/servlet/ServletResponse.html#setBufferSize(int) with a much bigger number than the default.

The code that should be doing this could be com.noelios.restlet.ext.servlet.ServletCall.sendResponse(ServletCall.java:421):

Current version just take the OutputStream:
public OutputStream getResponseEntityStream() {
try

{ return getResponse().getOutputStream(); }

catch (IOException e)

{ return null; }

}

Comment by Rich Seddon [ 02/22/12 ]

When this was tested by the Jetty team they found a smaller buffer size actually works best: http://docs.codehaus.org/display/JETTY/Slow+Win2003

If you can switch to Jetty then you can fix this as described in that link.

For Jetty 7 the equivalent configuration is:

    <Call name="addConnector">
        <Arg>
            <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
              <Set name="host">${application-host}</Set>
              <Set name="port">${application-port}</Set>
              
              <!-- Documentation for this fix can be found at:
                   http://docs.codehaus.org/display/JETTY/Slow+Win2003
              -->
              <Set name="responseBufferSize">512</Set>
            </New>
        </Arg>
    </Call>
Comment by Sebastien Tardif [ 02/24/12 ]

I just tried the equivalent settings for Tomcat (smaller buffer), and it's improving the performance. See socketBuffer at http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Java_TCP_socket_attributes

Still, Nexus is about streaming artifact. It's kind of pathetic that this bug is still not fixed after all this years.

It's the responsibility of Nexus to set the right buffer size using api ServletResponse.setBufferSize. It's a common practice with JSP. The right buffer size is use case specific.

The code should also be optimized to read and write at the same time. From previous profiling I have provided, where the slowness side was writing. After I optimize writing using more appropriate buffer size, I raise the potential improvement that will bring having concurrent read/write.

Comment by Rich Seddon [ 08/13/13 ]

No idea why this is in such an odd state. I'm closing this out, most of the problems on this issue were due to a windows bug, and the workaround is above.

There is an issue open against 2.7 for a potential regression here: NEXUS-5790

Generated at Tue Oct 23 20:00:34 UTC 2018 using JIRA 7.3.3#73014-sha1:d5be8da522213be2ca9ad7b043c51da6e4cc9754.