Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18245

AnonymousFilter uses EvictingQueue in non thread-safe manner possibly leading to HTTP 500 error responses

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.2, 3.10.0, 3.12.0, 3.13.0, 3.14.0
    • Fix Version/s: 3.15.0
    • Component/s: Security
    • Labels:

      Description

      Some instances under high-concurrent load by anonymous user access have encountered 500 responses from Nexus. If the offending URL is loaded in a web page, a 500 error page is displayed with a message such as the following:

      HTTP ERROR 500

Problem accessing /repo/error.html. Reason: 
    Server Error


Caused by:
javax.servlet.ServletException: java.util.NoSuchElementException
 at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
 at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
 at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
 at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
 at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
 at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
 at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
 at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
 at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
 at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
 at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
 at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
 at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:203)
 at org.eclipse.jetty.server.Dispatcher.error(Dispatcher.java:81)
 at org.eclipse.jetty.server.handler.ErrorHandler.doError(ErrorHandler.java:119)
 at org.eclipse.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:78)
 at org.eclipse.jetty.server.Response.sendError(Response.java:655)
 at org.eclipse.jetty.server.Response.sendError(Response.java:590)
 at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:72)
 at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
 at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
 at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
 at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.Server.handle(Server.java:531)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
 at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException
 at java.util.ArrayDeque.removeFirst(ArrayDeque.java:280)
 at java.util.ArrayDeque.remove(ArrayDeque.java:447)
 at com.google.common.collect.EvictingQueue.add(EvictingQueue.java:111)
 at org.sonatype.nexus.security.anonymous.AnonymousFilter.preHandle(AnonymousFilter.java:90)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
 ... 79 more


Caused by:
java.util.NoSuchElementException
 at java.util.ArrayDeque.removeFirst(ArrayDeque.java:280)
 at java.util.ArrayDeque.remove(ArrayDeque.java:447)
 at com.google.common.collect.EvictingQueue.add(EvictingQueue.java:111)
 at org.sonatype.nexus.security.anonymous.AnonymousFilter.preHandle(AnonymousFilter.java:90)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
 at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
 at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
 at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
 at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
 at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
 at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
 at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
 at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
 at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
 at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
 at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
 at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:203)
 at org.eclipse.jetty.server.Dispatcher.error(Dispatcher.java:81)
 at org.eclipse.jetty.server.handler.ErrorHandler.doError(ErrorHandler.java:119)
 at org.eclipse.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:78)
 at org.eclipse.jetty.server.Response.sendError(Response.java:655)
 at org.eclipse.jetty.server.Response.sendError(Response.java:590)
 at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:72)
 at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
 at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
 at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
 at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.Server.handle(Server.java:531)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
 at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
 at java.lang.Thread.run(Thread.java:745)


Powered by Jetty:// 9.4.11.v20180605

      Meanwhile the nexus.log may become filled with log messages such as:

      2018-10-20 22:19:53,183-0400 WARN  [qtp1323178108-14234]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html

2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14261]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html

2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14252]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html

2018-10-20 22:19:53,215-0400 WARN  [qtp1323178108-14267]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html

2018-10-20 22:19:53,183-0400 WARN  [qtp1323178108-14269]  *SYSTEM org.eclipse.jetty.server.handler.ErrorHandler - Error page loop /error.html

      The above message at this line, indicates a critical loop was entered inside the Jetty ErrorHandler - https://github.com/eclipse/jetty.project/blob/jetty-9.4.11.v20180605/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java#L106

      Once in the error page loop state, a restart of Nexus Repository Manager appears to be required to exit the page loop state.

      Cause

      The stack in the error page appears to be caused by use of a non thread-safe queue inside AnonymousFilter.

      Expected

      1. AnonymousFilter should use any queues in a thread safe manner
      2. Internal errors bubbled to ErrorHandler should conform to Exception instances that comply with the Servlet Filter specification ( AnonymousFilter uses Shiro's AdviceFilter, which is not expecting such violation )

        Attachments

          Issue Links

          discovered while testing

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-18265 Anonymous user cannot perform certain tasks despite having functional buttons

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-14593 Errors reported when accessing NXRM3 via index.html

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch Peter Lynch
              Team:
              NXRM - Cypher
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title