[NEXUS-16950] org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap permission bottleneck for large numbers of repositories in nested group repos - Sonatype JIRA

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.14.3, 2.14.8
Fix Version/s: 2.14.9
Component/s: Repository
Labels:

Story Points:
1

Description

org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap is a synchronized method allowing only one thread access at a time.

There are several triggers to get a repositories map per HTTP request into Nexus.

Example thread stacks which confluence around this method are:

/service/local/data_index

qtp162721995-13491 id=13491 state=BLOCKED
    - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
     owned by qtp162721995-5951 id=5951
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
    at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.index.DefaultIndexArtifactFilter.filterArtifactInfo(DefaultIndexArtifactFilter.java:85)
    at org.sonatype.nexus.index.DefaultIndexerManager$18.accepts(DefaultIndexerManager.java:1854)
    at org.apache.maven.index.AndMultiArtifactInfoFilter.accepts(AndMultiArtifactInfoFilter.java:46)
    at org.apache.maven.index.AbstractMultiArtifactInfoFilter.accepts(AbstractMultiArtifactInfoFilter.java:80)
    at org.apache.maven.index.DefaultIteratorResultSet.createNextAi(DefaultIteratorResultSet.java:245)
    at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:153)
    at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:51)
    at org.sonatype.nexus.rest.index.AbstractIndexerNexusPlexusResource.ai2NaColl(AbstractIndexerNexusPlexusResource.java:67)
    at org.sonatype.nexus.rest.index.AbstractIndexPlexusResource.get(AbstractIndexPlexusResource.java:165)
    at org.sonatype.nexus.rest.index.DefaultIndexPlexusResource.get(DefaultIndexPlexusResource.java:86)
    at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:233)

/service/local/lucene/search

qtp162721995-414 id=414 state=BLOCKED
    - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
     owned by qtp162721995-5951 id=5951
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
    at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.index.DefaultIndexArtifactFilter.filterArtifactInfo(DefaultIndexArtifactFilter.java:85)
    at org.sonatype.nexus.index.DefaultIndexerManager$18.accepts(DefaultIndexerManager.java:1854)
    at org.apache.maven.index.AndMultiArtifactInfoFilter.accepts(AndMultiArtifactInfoFilter.java:46)
    at org.apache.maven.index.AbstractMultiArtifactInfoFilter.accepts(AbstractMultiArtifactInfoFilter.java:80)
    at org.apache.maven.index.DefaultIteratorResultSet.createNextAi(DefaultIteratorResultSet.java:245)
    at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:153)
    at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:51)
    at org.sonatype.nexus.index.LockingIteratorResultSet.next(LockingIteratorResultSet.java:46)
    at org.sonatype.nexus.index.LockingIteratorResultSet.next(LockingIteratorResultSet.java:25)
    at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.packSearchNGResponse(SearchNGIndexPlexusResource.java:338)
    at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.get(SearchNGIndexPlexusResource.java:256)
    at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.get(SearchNGIndexPlexusResource.java:66)
    at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:233)
    at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:39)

NexusTargetMappingAuthorizationFilter

qtp162721995-408 id=408 state=BLOCKED
    - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
     owned by qtp162721995-5951 id=5951
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
    at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.authorizePath(DefaultRepositoryRouter.java:619)
    at org.sonatype.nexus.security.filter.authz.NexusTargetMappingAuthorizationFilter.isAccessAllowed(NexusTargetMappingAuthorizationFilter.java:160)
    at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
    at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
    at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.sonatype.nexus.web.internal.SecurityFilter.executeChain(SecurityFilter.java:90)

ContentServlet authorization

qtp162721995-1056 id=1056 state=BLOCKED
    - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
     owned by qtp162721995-5951 id=5951
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
    at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
    at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
    at org.sonatype.nexus.proxy.access.DefaultAccessManager.decide(DefaultAccessManager.java:46)
    at org.sonatype.nexus.proxy.repository.AbstractRepository.checkConditions(AbstractRepository.java:1194)
    at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:590)
    at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155)
    at org.sonatype.nexus.content.internal.ContentServlet.doGet(ContentServlet.java:386)

Smart Proxy event publishing

qtp162721995-1015 id=1015 state=BLOCKED
    - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
     owned by qtp162721995-5951 id=5951
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
    at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
    at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:268)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.recipients(PublishEventInspector.java:253)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.maybeFireItemUpdated(PublishEventInspector.java:354)
    at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.on(PublishEventInspector.java:182)
    at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
    at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
    at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
    at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
    at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
    at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
    at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:1032)
    at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.storeItem(AbstractMavenRepository.java:466)
    at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:661)
    at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.storeItem(DefaultRepositoryRouter.java:179)
    at org.sonatype.nexus.content.internal.ContentServlet.doPut(ContentServlet.java:602)
    at org.sonatype.nexus.content.internal.ContentServlet.service(ContentServlet.java:357)

Since the method is synchronized and this method copies the entire list of repository members which may be large, in a heavily loaded system receiving combinations of the mentioned code paths, threads may backup to the point where Nexus may appear non-responsive due to a high number of blocked threads on the subject method.

This performance problem is most likely to surface under two primary conditions:

1) nested group repo hierarchy
2) group repos containing thousands of repositories

Attachments

Activity

People

Assignee:: Brad Beck

Reporter:: Peter Lynch

Last Updated By:: Peter Lynch

Team:: Nexus - Core

Votes:: 1 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 04/25/18 07:09 PM

Updated:: 05/17/18 07:20 PM

Resolved:: 05/17/18 07:19 PM

Date of First Response:: 25/Apr/18 8:03 PM