Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19618

Expensive, error prone check done for content validation of checksums

    XMLWordPrintable

    Details

    • Story Points:
      0

      Description

      Currently, the validation of checksum file content is going through the tika mime type evaluation layer.  This is an expensive check, and is error prone for checksums, since they can potentially start with the same magic byte patterns as many different file formats.  See NEXUS-19018 for a specific example of this.

      We should simply do a check that the file contains only hex digits, and the right number of them.  A simple regex match would work for this.

      Example: When a Remove Maven snapshots task runs ( or any administrative task like Cleanup Policies) , rebuilds the checksum for a rebuilt maven-metadata.xml file, the generated md5, can fail the entire task:

      2019-04-06 04:33:07,677-0400 ERROR [quartz-2-thread-13]  *SYSTEM org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask - Failed to run task 'Remove Maven snapshots from maven-snapshots' on repository 'maven-snapshots'
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: com/example/csi/callback/engine/4.6.1.0-GA-SNAPSHOT/maven-metadata.xml.md5
      	at org.sonatype.nexus.repository.storage.DefaultContentValidator.determineContentType(DefaultContentValidator.java:95)
      	at org.sonatype.nexus.repository.maven.internal.MavenContentValidator.determineContentType(MavenContentValidator.java:79)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.determineContentType(StorageTxImpl.java:961)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.buildStorageHeaders(StorageTxImpl.java:707)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.createBlob(StorageTxImpl.java:679)
      	at sun.reflect.GeneratedMethodAccessor329.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      	at com.sun.proxy.$Proxy232.createBlob(Unknown Source)
      	at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.doPut(MavenFacetImpl.java:238)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:45)
      	at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:202)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUtils.write(MetadataUtils.java:109)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.write(MetadataUpdater.java:202)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.lambda$0(MetadataUpdater.java:105)
      	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:196)
      	at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.update(MetadataUpdater.java:89)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.processMetadata(MetadataUpdater.java:72)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$2(MetadataRebuilder.java:461)
      	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      	at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:415)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:382)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:120)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:240)
      	at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:129)
      	at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl.removeSnapshots(RemoveSnapshotsFacetImpl.java:141)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      	at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:72)
      	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
      	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
      	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
      	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
      	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
      	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
      	at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:61)
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:73)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      2019-04-06 04:33:07,678-0400 WARN  [quartz-2-thread-13]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 10f37d14-fdbc-44bd-8d8f-459c68d1238e : 'remove snapshots (keep 2)' [repository.maven.remove-snapshots] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Remove Maven snapshots from maven-snapshots'; 1 failure
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:84)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: com/example/csi/callback/engine/4.6.1.0-GA-SNAPSHOT/maven-metadata.xml.md5
      		at org.sonatype.nexus.repository.storage.DefaultContentValidator.determineContentType(DefaultContentValidator.java:95)
      		at org.sonatype.nexus.repository.maven.internal.MavenContentValidator.determineContentType(MavenContentValidator.java:79)
      		at org.sonatype.nexus.repository.storage.StorageTxImpl.determineContentType(StorageTxImpl.java:961)
      		at org.sonatype.nexus.repository.storage.StorageTxImpl.buildStorageHeaders(StorageTxImpl.java:707)
      		at org.sonatype.nexus.repository.storage.StorageTxImpl.createBlob(StorageTxImpl.java:679)
      		at sun.reflect.GeneratedMethodAccessor329.invoke(Unknown Source)
      		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		at java.lang.reflect.Method.invoke(Method.java:498)
      		at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      		at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      		at com.sun.proxy.$Proxy232.createBlob(Unknown Source)
      		at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.doPut(MavenFacetImpl.java:238)
      		at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:45)
      		at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:202)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUtils.write(MetadataUtils.java:109)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.write(MetadataUpdater.java:202)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.lambda$0(MetadataUpdater.java:105)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:196)
      		at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.update(MetadataUpdater.java:89)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater.processMetadata(MetadataUpdater.java:72)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$2(MetadataRebuilder.java:461)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      		at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:415)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:382)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:120)
      		at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:240)
      		at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:129)
      		at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl.removeSnapshots(RemoveSnapshotsFacetImpl.java:141)
      		at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      		at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      		at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      		at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:72)
      		at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
      		at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
      		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
      		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
      		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      		at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
      		at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
      		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      		at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
      		at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:61)
      		at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:73)
      		... 12 common frames omitted
      

      Expected

      • A failure processing a single asset inside the RemoveSnapshotsTask should not stop/fail the entire task. It should keep going and process as many assets as it can ( possible separate issue ).
      • We certainly should not be performing any validation of our own generated hash files.
      • do not do mime type of any hash files, even proxied remotes or uploaded into NXRM - instead use a simpler hash file format detection validation algorithm using a regex or similar - NXRM 2 did that, NXRM 3 should adopt something similar

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mdodgson Mark Dodgson
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch
              Team:
              NXRM - Cypher
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title