-
Type:
Bug
-
Status: Open
-
Priority:
Minor
-
Resolution: Unresolved
-
Affects Version/s: 3.14.0, 3.15.2
-
Fix Version/s: None
-
Component/s: Documentation, git-lfs
-
Labels:
-
Story Points:0.5
Git-lfs repository configured on nexus 3.14.0-04 behind nginx. The only additional configuration on reverse proxy is client_max_body_size increase to accomodate larger uploads. Git lfs push seems to loop on same file. Nexus log shows the following exceptions:
2019-02-06 14:42:33,961+0100 WARN [qtp985482502-15536] <REDACTED> org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT <REDACTED>/462f0c6ad4dc618b4317660e3f582d14ae8137ad98519cca9dafb2e3ffa1501c org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: tmp$71123292-cc1e-47a5-b62b-cf48d231dbff, org.eclipse.jetty.io.EofException: Early EOF, Cause: Early EOF at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:376) at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:312) at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:288) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:215) at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:228) at org.sonatype.nexus.repository.gitlfs.internal.GitLfsHostedFacetImpl.put(GitLfsHostedFacetImpl.java:82) at org.sonatype.nexus.repository.gitlfs.internal.GitLfsHostedUploadHandler.handle(GitLfsHostedUploadHandler.java:43) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.assetdownloadcount.internal.AssetDownloadCountContributedHandler.handle(AssetDownloadCountContributedHandler.java:53) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at com.sonatype.nexus.clm.internal.FirewallContributedHandler.handle(FirewallContributedHandler.java:87) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.Context.start(Context.java:114) at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:210) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:172) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Unknown Source) Caused by: org.eclipse.jetty.io.EofException: Early EOF at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1138) at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1126) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313) at org.sonatype.nexus.common.hash.MultiHashingInputStream.read(MultiHashingInputStream.java:66) at com.google.common.io.CountingInputStream.read(CountingInputStream.java:63) at java.security.DigestInputStream.read(Unknown Source) at java.io.FilterInputStream.read(Unknown Source) at java.io.FilterInputStream.read(Unknown Source) at com.google.common.io.ByteStreams.copy(ByteStreams.java:109) at org.sonatype.nexus.blobstore.file.internal.SimpleFileOperations.create(SimpleFileOperations.java:61) at org.sonatype.nexus.blobstore.file.FileBlobStore.lambda$0(FileBlobStore.java:288) at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:346) ... 120 common frames omitted
Turns out nginx proxy request buffering was an issue (similar issue is reported against git-lfs backend in gitlab).
Fix is to add the following to location section describing the the proxy definition:
proxy_request_buffering off
Applying this configuration fixed the issue.
Since git-lfs is commonly used for large files, you may want to amend nexus documentation for reverse proxy configuration.