Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.34.0
-
Fix Version/s: 3.35.0
-
Component/s: Browse Storage, Migration
-
Story Points:3
-
Epic Link:
-
Sprint:NXRM Neo Sprint 17, NXRM Neo Sprint 16
Description
During Migration of repository 2 to 3.34.0 Postgres, the database tables that store Browse feature information are populated in real time. Thread dumps can contain many RUNNABLE threads like this:
"periodic-8-thread-1" id=590 state=RUNNABLE (running in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.receiveChar(PGStream.java:443) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164) at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:153) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.sonatype.nexus.datastore.mybatis.EntityExecutor.update(EntityExecutor.java:81) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194) at org.sonatype.nexus.datastore.mybatis.DataAccessSqlSession.update(DataAccessSqlSession.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86) at com.sun.proxy.$Proxy161.mergeBrowseNode(Unknown Source) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore.mergeBrowseNode(BrowseNodeStore.java:90) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635.GUICE$TRAMPOLINE(<generated>) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635$$Lambda$614/236206295.apply(Unknown Source) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:49) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75) at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeStore$$EnhancerByGuice$$706683635.mergeBrowseNode(<generated>) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.doCreateBrowseNodes(BrowseNodeManager.java:109) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.lambda$0(BrowseNodeManager.java:89) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager$$Lambda$2156/1859220116.run(Unknown Source) at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53) at org.sonatype.nexus.transaction.OperationPoint$$Lambda$2157/672517590.call(Unknown Source) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57) at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223) at org.sonatype.nexus.transaction.Operations.run(Operations.java:175) at org.sonatype.nexus.repository.content.browse.store.BrowseNodeManager.createBrowseNodes(BrowseNodeManager.java:89) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.lambda$6(BrowseFacetImpl.java:196) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$Lambda$2159/471373833.accept(Unknown Source) at java.util.Optional.ifPresent(Optional.java:159) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.createBrowseNodes(BrowseFacetImpl.java:193) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$Lambda$2150/1787441353.accept(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl.addPathsToAssets(BrowseFacetImpl.java:123) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658.GUICE$TRAMPOLINE(<generated>) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658$$Lambda$524/1536530867.apply(Unknown Source) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75) at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55) at org.sonatype.nexus.repository.content.browse.BrowseFacetImpl$$EnhancerByGuice$$376960658.addPathsToAssets(<generated>) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.lambda$3(BrowseEventHandler.java:289) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$2146/1045492744.accept(Unknown Source) at java.util.Optional.ifPresent(Optional.java:159) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.lambda$2(BrowseEventHandler.java:288) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$2144/1060102093.accept(Unknown Source) at java.util.Map.forEach(Map.java:630) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.flushPageOfAssets(BrowseEventHandler.java:287) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler.pollBrowseUpdateRequests(BrowseEventHandler.java:260) at org.sonatype.nexus.repository.content.browse.BrowseEventHandler$$Lambda$1314/2028225422.run(Unknown Source) at org.sonatype.nexus.scheduling.internal.PeriodicJobServiceImpl.lambda$1(PeriodicJobServiceImpl.java:90) at org.sonatype.nexus.scheduling.internal.PeriodicJobServiceImpl$$Lambda$1315/454526372.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
This threads contribute noticeable overhead to the overall migration process.
Expected
Do not build browse feature information during the repository 2 to 3 migration process. Instead execute a scheduled task to rebuild all repository browse nodes at the end of migration, after the user explicitly Finishes migration. This will mean for a time period after migration is complete, the Browse view of repository information will take some time to populate fully. However since the Browse feature is mostly a UI treatment, this is a warranted tradeoff in most cases.
Attachments
Issue Links
- is related to
-
NEXUS-28896 improve INSERT performance into format specific browse_node tables
-
- Closed
-