Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16950

org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap permission bottleneck for large numbers of repositories in nested group repos

    XMLWordPrintable

    Details

    • Story Points:
      1

      Description

      org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap is a synchronized method allowing only one thread access at a time.

      There are several triggers to get a repositories map per HTTP request into Nexus.

      Example thread stacks which confluence around this method are:

      /service/local/data_index
      qtp162721995-13491 id=13491 state=BLOCKED
          - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
           owned by qtp162721995-5951 id=5951
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
          at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.index.DefaultIndexArtifactFilter.filterArtifactInfo(DefaultIndexArtifactFilter.java:85)
          at org.sonatype.nexus.index.DefaultIndexerManager$18.accepts(DefaultIndexerManager.java:1854)
          at org.apache.maven.index.AndMultiArtifactInfoFilter.accepts(AndMultiArtifactInfoFilter.java:46)
          at org.apache.maven.index.AbstractMultiArtifactInfoFilter.accepts(AbstractMultiArtifactInfoFilter.java:80)
          at org.apache.maven.index.DefaultIteratorResultSet.createNextAi(DefaultIteratorResultSet.java:245)
          at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:153)
          at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:51)
          at org.sonatype.nexus.rest.index.AbstractIndexerNexusPlexusResource.ai2NaColl(AbstractIndexerNexusPlexusResource.java:67)
          at org.sonatype.nexus.rest.index.AbstractIndexPlexusResource.get(AbstractIndexPlexusResource.java:165)
          at org.sonatype.nexus.rest.index.DefaultIndexPlexusResource.get(DefaultIndexPlexusResource.java:86)
          at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:233)
      
      /service/local/lucene/search
      qtp162721995-414 id=414 state=BLOCKED
          - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
           owned by qtp162721995-5951 id=5951
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
          at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.index.DefaultIndexArtifactFilter.filterArtifactInfo(DefaultIndexArtifactFilter.java:85)
          at org.sonatype.nexus.index.DefaultIndexerManager$18.accepts(DefaultIndexerManager.java:1854)
          at org.apache.maven.index.AndMultiArtifactInfoFilter.accepts(AndMultiArtifactInfoFilter.java:46)
          at org.apache.maven.index.AbstractMultiArtifactInfoFilter.accepts(AbstractMultiArtifactInfoFilter.java:80)
          at org.apache.maven.index.DefaultIteratorResultSet.createNextAi(DefaultIteratorResultSet.java:245)
          at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:153)
          at org.apache.maven.index.DefaultIteratorResultSet.next(DefaultIteratorResultSet.java:51)
          at org.sonatype.nexus.index.LockingIteratorResultSet.next(LockingIteratorResultSet.java:46)
          at org.sonatype.nexus.index.LockingIteratorResultSet.next(LockingIteratorResultSet.java:25)
          at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.packSearchNGResponse(SearchNGIndexPlexusResource.java:338)
          at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.get(SearchNGIndexPlexusResource.java:256)
          at org.sonatype.nexus.rest.indexng.SearchNGIndexPlexusResource.get(SearchNGIndexPlexusResource.java:66)
          at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:233)
          at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:39)
      
      NexusTargetMappingAuthorizationFilter
      qtp162721995-408 id=408 state=BLOCKED
          - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
           owned by qtp162721995-5951 id=5951
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
          at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.authorizePath(DefaultRepositoryRouter.java:619)
          at org.sonatype.nexus.security.filter.authz.NexusTargetMappingAuthorizationFilter.isAccessAllowed(NexusTargetMappingAuthorizationFilter.java:160)
          at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
          at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
          at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.sonatype.nexus.web.internal.SecurityFilter.executeChain(SecurityFilter.java:90)
      
      ContentServlet authorization
      qtp162721995-1056 id=1056 state=BLOCKED
          - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
           owned by qtp162721995-5951 id=5951
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
          at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:69)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePathCascade(DefaultNexusItemAuthorizer.java:71)
          at org.sonatype.nexus.proxy.access.DefaultNexusItemAuthorizer.authorizePath(DefaultNexusItemAuthorizer.java:63)
          at org.sonatype.nexus.proxy.access.DefaultAccessManager.decide(DefaultAccessManager.java:46)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.checkConditions(AbstractRepository.java:1194)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:590)
          at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155)
          at org.sonatype.nexus.content.internal.ContentServlet.doGet(ContentServlet.java:386)
      
      Smart Proxy event publishing
      qtp162721995-1015 id=1015 state=BLOCKED
          - waiting to lock <0x0e70f442> (a org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry)
           owned by qtp162721995-5951 id=5951
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepositoriesMap(DefaultRepositoryRegistry.java:207)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getRepository(DefaultRepositoryRegistry.java:118)
          at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.getMemberRepositories(AbstractGroupRepository.java:390)
          at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.getGroupsOfRepository(DefaultRepositoryRegistry.java:175)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:268)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.includeRecipients(PublishEventInspector.java:271)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.recipients(PublishEventInspector.java:253)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.maybeFireItemUpdated(PublishEventInspector.java:354)
          at com.sonatype.nexus.plugins.smartproxy.event.internal.PublishEventInspector.on(PublishEventInspector.java:182)
          at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
          at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
          at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
          at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
          at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
          at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:1032)
          at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.storeItem(AbstractMavenRepository.java:466)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:661)
          at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.storeItem(DefaultRepositoryRouter.java:179)
          at org.sonatype.nexus.content.internal.ContentServlet.doPut(ContentServlet.java:602)
          at org.sonatype.nexus.content.internal.ContentServlet.service(ContentServlet.java:357)
      

      Since the method is synchronized and this method copies the entire list of repository members which may be large, in a heavily loaded system receiving combinations of the mentioned code paths, threads may backup to the point where Nexus may appear non-responsive due to a high number of blocked threads on the subject method.

      This performance problem is most likely to surface under two primary conditions:

      1) nested group repo hierarchy
      2) group repos containing thousands of repositories

        Attachments

          Activity

            People

            Assignee:
            bradbeck Brad Beck
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch
            Team:
            Nexus - Core
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title