Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19056

Early EOF exception when pushing large files to LFS repository behind nginx

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.14.0, 3.15.2
    • Fix Version/s: None
    • Component/s: Documentation, git-lfs
    • Labels:
    • Story Points:
      0.5

      Description

      Git-lfs repository configured on nexus 3.14.0-04 behind nginx. The only additional configuration on reverse proxy is client_max_body_size increase to accomodate larger uploads. Git lfs push seems to loop on same file. Nexus log shows the following exceptions:

       

      2019-02-06 14:42:33,961+0100 WARN  [qtp985482502-15536]  <REDACTED> org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT <REDACTED>/462f0c6ad4dc618b4317660e3f582d14ae8137ad98519cca9dafb2e3ffa1501c
      org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: tmp$71123292-cc1e-47a5-b62b-cf48d231dbff, org.eclipse.jetty.io.EofException: Early EOF, Cause: Early EOF
          at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:376)
          at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:312)
          at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:288)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:215)
          at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:228)
          at org.sonatype.nexus.repository.gitlfs.internal.GitLfsHostedFacetImpl.put(GitLfsHostedFacetImpl.java:82)
          at org.sonatype.nexus.repository.gitlfs.internal.GitLfsHostedUploadHandler.handle(GitLfsHostedUploadHandler.java:43)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.assetdownloadcount.internal.AssetDownloadCountContributedHandler.handle(AssetDownloadCountContributedHandler.java:53)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at com.sonatype.nexus.clm.internal.FirewallContributedHandler.handle(FirewallContributedHandler.java:87)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
          at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64)
          at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
          at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
          at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:210)
          at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:172)
          at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
          at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
          at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
          at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
          at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
          at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
          at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
          at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
          at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
          at org.eclipse.jetty.server.Server.handle(Server.java:531)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
          at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
          at java.lang.Thread.run(Unknown Source)
      Caused by: org.eclipse.jetty.io.EofException: Early EOF
          at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1138)
          at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1126)
          at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
          at org.sonatype.nexus.common.hash.MultiHashingInputStream.read(MultiHashingInputStream.java:66)
          at com.google.common.io.CountingInputStream.read(CountingInputStream.java:63)
          at java.security.DigestInputStream.read(Unknown Source)
          at java.io.FilterInputStream.read(Unknown Source)
          at java.io.FilterInputStream.read(Unknown Source)
          at com.google.common.io.ByteStreams.copy(ByteStreams.java:109)
          at org.sonatype.nexus.blobstore.file.internal.SimpleFileOperations.create(SimpleFileOperations.java:61)
          at org.sonatype.nexus.blobstore.file.FileBlobStore.lambda$0(FileBlobStore.java:288)
          at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:346)
          ... 120 common frames omitted

       

      Turns out nginx proxy request buffering was an issue (similar issue is reported against git-lfs backend in gitlab).

      Fix is to add the following to location section describing the the proxy definition:

      proxy_request_buffering off

      Applying this configuration fixed the issue.

       

      Since git-lfs is commonly used for large files, you may want to amend nexus documentation for reverse proxy configuration.

       

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            Nemes Antal Nemes
            Last Updated By:
            Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Date of First Response:

                tigCommentSecurity.panel-title