Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-13566

OOM may lead to blobstore/database inconsistencies

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.2
    • Fix Version/s: None
    • Component/s: Blobstore, Database
    • Labels:
    • Story Points:
      5
    • Sprint:
      Core Sprint 102

      Description

      We had a user here who had thousands of blobs that were referenced in their database, but are not in blob storage.

      The cause seems to be that they ran out of heap space, and this caused CPU starvation of threads writing to the blob store. These threads reached their timeout, and the blobs were not written.  An example of this is below.

      2017-06-27 20:17:41,269-0500 ERROR [OrientDB Write Cache Flush Task (accesslog)] *SYSTEM java.lang.Throwable - java.lang.OutOfMemoryError: Java heap space
      2017-06-27 20:17:41,270-0500 WARN [qtp1144406047-17181] *SYSTEM org.eclipse.jetty.util.thread.QueuedThreadPool - Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@4e132759 in qtp1144406047{STARTED,8<=47<=200,i=5,q=1}
      2017-06-27 20:17:41,270-0500 WARN [qtp1144406047-17186] *SYSTEM org.eclipse.jetty.util.thread.QueuedThreadPool - Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@4e132759 in qtp1144406047{STARTED,8<=47<=200,i=5,q=1}
      2017-06-27 20:17:41,270-0500 ERROR [nexus_QuartzSchedulerThread] *SYSTEM java.lang.Throwable - java.lang.OutOfMemoryError: Java heap space
      2017-06-27 20:17:41,271-0500 ERROR [WAL Closer Task (audit)] *SYSTEM java.lang.Throwable - java.lang.OutOfMemoryError: Java heap space
      2017-06-27 20:17:41,285-0500 ERROR [nexus-httpclient-eviction-thread] *SYSTEM java.lang.Throwable - java.lang.OutOfMemoryError: Java heap space
      2017-06-27 20:26:58,929-0500 WARN [qtp1144406047-17177] deployment org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service failure
      org.sonatype.nexus.blobstore.api.BlobStoreException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 31711/30000 ms
      at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.create(FileBlobStore.java:290) [na:na]
      at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.create(FileBlobStore.java:225) [na:na]
      at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:220) [na:na]
      at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:232) [na:na]
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:199) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.doPut(HostedHandler.java:87) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.handle(HostedHandler.java:61) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      at org.sonatype.nexus.repository.maven.internal.VersionPolicyHandler.handle(VersionPolicyHandler.java:50) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) [na:na]
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            rseddon Rich Seddon
            Last Updated By:
            Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title