Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18245

AnonymousFilter uses EvictingQueue in non thread-safe manner possibly leading to HTTP 500 error responses

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.2, 3.10.0, 3.12.0, 3.13.0, 3.14.0
    • Fix Version/s: 3.15.0
    • Component/s: Security
    • Labels:

      Description

      Some instances under high-concurrent load by anonymous user access have encountered 500 responses from Nexus. If the offending URL is loaded in a web page, a 500 error page is displayed with a message such as the following:

      HTTP ERROR 500
      
      Problem accessing /repo/error.html. Reason: 
          Server Error
      
      
      Caused by:
      javax.servlet.ServletException: java.util.NoSuchElementException
       at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
       at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
       at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
       at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
       at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
       at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
       at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
       at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
       at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
       at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
       at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
       at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
       at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:203)
       at org.eclipse.jetty.server.Dispatcher.error(Dispatcher.java:81)
       at org.eclipse.jetty.server.handler.ErrorHandler.doError(ErrorHandler.java:119)
       at org.eclipse.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:78)
       at org.eclipse.jetty.server.Response.sendError(Response.java:655)
       at org.eclipse.jetty.server.Response.sendError(Response.java:590)
       at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:72)
       at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
       at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
       at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
       at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:531)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.NoSuchElementException
       at java.util.ArrayDeque.removeFirst(ArrayDeque.java:280)
       at java.util.ArrayDeque.remove(ArrayDeque.java:447)
       at com.google.common.collect.EvictingQueue.add(EvictingQueue.java:111)
       at org.sonatype.nexus.security.anonymous.AnonymousFilter.preHandle(AnonymousFilter.java:90)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
       ... 79 more
      
      
      Caused by:
      java.util.NoSuchElementException
       at java.util.ArrayDeque.removeFirst(ArrayDeque.java:280)
       at java.util.ArrayDeque.remove(ArrayDeque.java:447)
       at com.google.common.collect.EvictingQueue.add(EvictingQueue.java:111)
       at org.sonatype.nexus.security.anonymous.AnonymousFilter.preHandle(AnonymousFilter.java:90)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
       at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
       at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
       at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
       at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
       at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
       at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
       at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
       at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
       at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
       at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
       at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
       at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:203)
       at org.eclipse.jetty.server.Dispatcher.error(Dispatcher.java:81)
       at org.eclipse.jetty.server.handler.ErrorHandler.doError(ErrorHandler.java:119)
       at org.eclipse.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:78)
       at org.eclipse.jetty.server.Response.sendError(Response.java:655)
       at org.eclipse.jetty.server.Response.sendError(Response.java:590)
       at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:72)
       at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
       at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
       at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
       at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:531)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
       at java.lang.Thread.run(Thread.java:745)
      
      
      Powered by Jetty:// 9.4.11.v20180605
      

      Meanwhile the nexus.log may become filled with log messages such as:

      2018-10-20 22:19:53,183-0400 WARN  [qtp1323178108-14234]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html
      
      2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14261]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html
      
      2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14252]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html
      
      2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14267]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html
      
      2018-10-20 22:19:53,183-0400 WARN  [qtp1323178108-14269]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html
      

      The above message at this line, indicates a critical loop was entered inside the Jetty ErrorHandler - https://github.com/eclipse/jetty.project/blob/jetty-9.4.11.v20180605/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java#L106

      Once in the error page loop state, a restart of Nexus Repository Manager appears to be required to exit the page loop state.

      Cause

      The stack in the error page appears to be caused by use of a non thread-safe queue inside AnonymousFilter.

      Expected

      1. AnonymousFilter should use any queues in a thread safe manner
      2. Internal errors bubbled to ErrorHandler should conform to Exception instances that comply with the Servlet Filter specification ( AnonymousFilter uses Shiro's AdviceFilter, which is not expecting such violation )

        Attachments

        1. nexus.log
          184 kB
          Joe Tom
        2. Screen Shot 2018-10-23 at 10.30.50 PM.png
          639 kB
          Joe Tom
        3. Screen Shot 2018-10-23 at 10.38.20 PM.png
          847 kB
          Joe Tom

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch Peter Lynch
              Team:
              NXRM - Cypher
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title