Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-17095

Upload handlers (re)create temporary blobs inside transactions

    XMLWordPrintable

    Details

      Description

      Several upload handlers currently follow this anti-pattern:

      • start transaction
      • do some validation
      • create temporary blobs
      • create components/assets
      • commit transaction

      The problem with this approach is that it leaves the transaction open for a long time. If the transaction needs to be retried then it will repeat the same checks and create more temporary blobs.

      EDIT: The other issue with starting the storage transaction early on is that it breaks the later call to check content selector permissions (but only if content selector permissions apply to the upload). This is because the current active transaction is for the component DB, but the content selector check wants to query the config DB. We don't support nested active transactions, so the content selector basically ends up querying the current DB (ie. component) for config related information - this leads to schema problems and other exceptions.

      Ideally the transaction should only be started just before it's needed, ie. calling into the facet to create the component/assets, and closed as soon as possible. This also solves the content selector issue because the call to check permissions is now made outside of an active transaction, before the storage transaction is started.

      This was recently done for the Maven upload handler:

      https://github.com/sonatype/nexus-internal/blob/901ff3ce36d3f81bda9d8d245ec492a8bc1da39f/plugins/nexus-repository-maven/src/main/java/org/sonatype/nexus/repository/maven/MavenUploadHandler.java

      here's the original which followed the anti-pattern:

      https://github.com/sonatype/nexus-internal/blob/22c3750b4d598681c864ba94b88195fa7ae0faa4/plugins/nexus-repository-maven/src/main/java/org/sonatype/nexus/repository/maven/MavenUploadHandler.java

      For simple handlers you may not even need to specify any transaction because the facet will do that for you when you call their "put" method. But to do this you may need to setup a unit-of-work which the facet can then use - because I don't believe the resource endpoint exposed for upload has an equivalent to UnitOfWorkHandler, which sets it up for incoming repository requests. (See the updated Maven upload handler for an example.)

      I've listed the formats that need their upload handlers checking/reworking in the components field.

      You've hit this problem if on upload from a non-admin user account, you see this error in nexus.log:

      2018-08-02 10:40:20,631+0200 WARN  [qtp2023704419-47590]  *UNKNOWN org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID 6366c419-bc9e-44ff-9259-3feda5834aa9) Unexpected exception: java.lang.IllegalArgumentException: Class 'selector_selector' not found in current database
java.lang.IllegalArgumentException: Class 'selector_selector' not found in current database
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.browseClass(ODatabaseDocumentTx.java:2589)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.browseClass(ODatabaseDocumentTx.java:2581)
	at org.sonatype.nexus.orient.entity.EntityAdapter.browseDocuments(EntityAdapter.java:244)
	at org.sonatype.nexus.orient.entity.action.BrowseEntitiesAction.execute(BrowseEntitiesAction.java:40)
	at org.sonatype.nexus.orient.entity.IterableEntityAdapter.browse(IterableEntityAdapter.java:96)
	at org.sonatype.nexus.internal.selector.SelectorConfigurationStoreImpl.lambda$0(SelectorConfigurationStoreImpl.java:73)
	at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$1(OrientOperations.java:56)
	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:196)
	at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
	at org.sonatype.nexus.orient.transaction.OrientOperations.call(OrientOperations.java:56)
	at org.sonatype.nexus.internal.selector.SelectorConfigurationStoreImpl.browse(SelectorConfigurationStoreImpl.java:73)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
	at org.sonatype.nexus.internal.selector.SelectorManagerImpl.browse(SelectorManagerImpl.java:108)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
	at org.sonatype.nexus.repository.security.internal.ContentPermissionCheckerImpl.isPermitted(ContentPermissionCheckerImpl.java:98)
	at org.sonatype.nexus.repository.upload.UploadHandler.ensurePermitted(UploadHandler.java:74)
	at org.sonatype.nexus.repository.maven.MavenUploadHandler.createAssets(MavenUploadHandler.java:215)
	at org.sonatype.nexus.repository.maven.MavenUploadHandler.lambda$0(MavenUploadHandler.java:168)
	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
	at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
	at org.sonatype.nexus.repository.maven.MavenUploadHandler.doUpload(MavenUploadHandler.java:151)
	at org.sonatype.nexus.repository.maven.MavenUploadHandler.handle(MavenUploadHandler.java:139)
	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:71)
	at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.uploadComponent(ComponentsResource.java:255)
	at sun.reflect.GeneratedMethodAccessor3050.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:294)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:248)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:235)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at org.sonatype.nexus.siesta.internal.resteasy.ComponentContainerImpl.service(ComponentContainerImpl.java:106)
	at org.sonatype.nexus.siesta.SiestaServlet.service(SiestaServlet.java:137)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at com.sonatype.nexus.analytics.internal.RestRequestCollector.doFilter(RestRequestCollector.java:86)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)
2018-08-02 10:40:20,631+0200 WARN  [qtp2023704419-47590]  *UNKNOWN org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID 6366c419-bc9e-44ff-9259-3feda5834aa9) Response: [500] 'ERROR: (ID 6366c419-bc9e-44ff-9259-3feda5834aa9) java.lang.IllegalArgumentException: Class 'selector_selector' not found in current database'; mapped from: java.lang.IllegalArgumentException: Class 'selector_selector' not found in current database

       

      Resolution

      This issue describes a general component upload problem which affected all repository formats.

      For Maven, the problem was fixed by NEXUS-16740 changes and shipped in 3.12.0.

      Other formats have linked issues on this ticket and were fixed as of 3.14.0.

        Attachments

          Issue Links

          is superceded by

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-16740 Nexus Upload Interface doesnt update or create metadata after upload file

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-17735 NPM Upload Transaction Too Long

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-17736 NuGet Upload Transaction Too Long

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-17737 Ruby Gems Transaction Too Long

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-17738 Raw Upload Transaction Too Long

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-18481 PyPI Upload Transaction Too Long

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mcculls Stuart McCulloch
              Last Updated By:
              Peter Lynch Peter Lynch
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title