Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28891

avoid building browse feature information during repository 2 to 3 migration to improve migration performance

    XMLWordPrintable

    Details

      Description

      During Migration of repository 2 to 3.34.0 Postgres, the database tables that store Browse feature information are populated in real time. Thread dumps can contain many RUNNABLE threads like this:

      "periodic-8-thread-1" id=590 state=RUNNABLE (running in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:443)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:153)
    at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
    at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
    at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
    at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
    at org.sonatype.nexus.datastore.mybatis.EntityExecutor.update(EntityExecutor.java:81)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194)
    at org.sonatype.nexus.datastore.mybatis.DataAccessSqlSession.update(DataAccessSqlSession.java:76)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62)
    at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
    at com.sun.proxy.$Proxy161.mergeBrowseNode(Unknown Source)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore.mergeBrowseNode(BrowseNodeStore.java:90)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635.GUICE$TRAMPOLINE(<generated>)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635$$Lambda$614/236206295.apply(Unknown Source)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
    at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:49)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
    at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635.mergeBrowseNode(<generated>)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.doCreateBrowseNodes(BrowseNodeManager.java:109)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.lambda$0(BrowseNodeManager.java:89)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager$$Lambda$2156/1859220116.run(Unknown Source)
    at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
    at org.sonatype.nexus.transaction.OperationPoint$$Lambda$2157/672517590.call(Unknown Source)
    at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
    at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
    at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
    at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
    at org.sonatype.nexus.transaction.Operations.run(Operations.java:175)
    at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.createBrowseNodes(BrowseNodeManager.java:89)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.lambda$6(BrowseFacetImpl.java:196)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$Lambda$2159/471373833.accept(Unknown Source)
    at java.util.Optional.ifPresent(Optional.java:159)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.createBrowseNodes(BrowseFacetImpl.java:193)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$Lambda$2150/1787441353.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.addPathsToAssets(BrowseFacetImpl.java:123)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658.GUICE$TRAMPOLINE(<generated>)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658$$Lambda$524/1536530867.apply(Unknown Source)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
    at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
    at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
    at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658.addPathsToAssets(<generated>)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.lambda$3(BrowseEventHandler.java:289)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$2146/1045492744.accept(Unknown Source)
    at java.util.Optional.ifPresent(Optional.java:159)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.lambda$2(BrowseEventHandler.java:288)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$2144/1060102093.accept(Unknown Source)
    at java.util.Map.forEach(Map.java:630)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.flushPageOfAssets(BrowseEventHandler.java:287)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.pollBrowseUpdateRequests(BrowseEventHandler.java:260)
    at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$1314/2028225422.run(Unknown Source)
    at org.sonatype.nexus.scheduling.internal.PeriodicJobServiceImpl.lambda$1(PeriodicJobServiceImpl.java:90)
    at org.sonatype.nexus.scheduling.internal.PeriodicJobServiceImpl$$Lambda$1315/454526372.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

      This threads contribute noticeable overhead to the overall migration process.

      Expected

      Do not build browse feature information during the repository 2 to 3 migration process. Instead execute a scheduled task to rebuild all repository browse nodes at the end of migration, after the user explicitly Finishes migration. This will mean for a time period after migration is complete, the Browse view of repository information will take some time to populate fully. However since the Browse feature is mostly a UI treatment, this is a warranted tradeoff in most cases.

        Attachments

          Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NEXUS-28896 improve INSERT performance into format specific browse_node tables

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              dbradicich Damian Bradicich
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Rich Seddon Rich Seddon
              Team:
              NXRM - Neo
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title