Slow performance pushing large images to S3 based blobstore

Customer originally reported this using docker. They were also able to reproduce slow performance using raw repository.

The logs show examples of failed uploads due to "Early EOF":

2020-07-29 01:50:09,797+0000 WARN [qtp1982132812-85841] US002SH org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT /repository/PBRawDev/com/pb/miraster/Runtime-MiPro/Trunk\6.0/2020-07-29/Runtime-MiPro.zip

org.sonatype.nexus.blobstore.api.BlobStoreException: Error executing parallel requests for bucket:nexus-dev-blob key:content/tmp/tmp$e7e066a5-0543-4f1e-b229-9f57fcefbb23.bytes with uploadId:8serHhUMlcrmMGyvhMgkjebIJ2xxW9TVE_xon9VqOmnGjT0_ymtFht9XuJTqFNAzAFI6wp2M.rH3MO3L24_1uVCUbnlVa0BQ0UZAbpR1d4aPjFfvRTmwyny31vGH6TGz, Cause: org.eclipse.jetty.io.EofException: Early EOF​​

Which means that an upload was abruptly ended midstream/upload and indicates something outside of Nexus i.e. ELB is closing the connection.

Looking at the thread dump in the support.zip, there are a number of threads waiting on S3 blob uploads e.g.

"qtp1982132812-87101" id=87101 state=WAITING

• waiting on <0x2ce7efc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

• locked <0x2ce7efc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at sun.misc.Unsafe.park(Native Method)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)

at org.sonatype.nexus.blobstore.s3.internal.ParallelRequester.parallelRequests(ParallelRequester.java:98)

at org.sonatype.nexus.blobstore.s3.internal.ParallelUploader.upload(ParallelUploader.java:80)

at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.lambda$1(S3BlobStore.java:233)

at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$$Lambda$852/2129009552.ingestTo(Unknown Source)

at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.create(S3BlobStore.java:268)

at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.doCreate(S3BlobStore.java:230)

at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:127)