Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-26355

sha512 or sha256 hashes are processed by expensive repository content validation

    XMLWordPrintable

    Details

      Description

      Start with a maven format source directory containing artifacts what when the sha512 sum is computed, results in a sha512sum that starts with the text "caff". When the import task tries to generate the sha512 sum, the sum fails content validation and thus never gets created.

      2021-01-09 08:17:30,470-0400 WARN  [quartz-9-thread-2]  *SYSTEM org.sonatype.nexus.repository.storage.StorageTxImpl - An exception occurred determining the content type of asset pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar.sha512 in repository central-hosted
      2021-01-09 08:17:30,472-0400 ERROR [quartz-9-thread-2]  *SYSTEM com.sonatype.nexus.exportimport.orient.internal.importtask.OrientRepositoryImportService - Import of file /import-data/central-releases/pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar into repository central-hosted failed
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar.sha512
      	at org.sonatype.nexus.repository.mime.DefaultContentValidator.determineContentType(DefaultContentValidator.java:95)
      	at org.sonatype.nexus.repository.maven.internal.MavenContentValidator.determineContentType(MavenContentValidator.java:85)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.determineContentType(StorageTxImpl.java:1019)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.buildStorageHeaders(StorageTxImpl.java:760)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.createBlob(StorageTxImpl.java:719)
      	at sun.reflect.GeneratedMethodAccessor271.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      	at com.sun.proxy.$Proxy237.createBlob(Unknown Source)
      	at org.sonatype.nexus.repository.maven.internal.orient.MavenFacetImpl.doPut(MavenFacetImpl.java:342)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:49)
      	at org.sonatype.nexus.repository.maven.internal.orient.MavenFacetImpl.put(MavenFacetImpl.java:306)
      	at org.sonatype.nexus.repository.maven.internal.orient.MavenFacetUtils.addHashes(MavenFacetUtils.java:189)
      	at org.sonatype.nexus.repository.maven.internal.orient.MavenUploadHandler.putChecksumFiles(MavenUploadHandler.java:130)
      	at org.sonatype.nexus.repository.maven.internal.orient.MavenUploadHandler.doPut(MavenUploadHandler.java:112)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandlerSupport.handle(MavenUploadHandlerSupport.java:190)
      	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:136)
      	at com.sonatype.nexus.exportimport.orient.internal.importtask.OrientRepositoryImportService.lambda$1(OrientRepositoryImportService.java:166)
      	at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
      	at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
      	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
      	at org.sonatype.nexus.transaction.Operations.run(Operations.java:175)
      	at com.sonatype.nexus.exportimport.orient.internal.importtask.OrientRepositoryImportService.lambda$0(OrientRepositoryImportService.java:162)
      	at com.sonatype.nexus.exportimport.internal.importtask.RepositoryImportSource$RepositoryImportFileVisitor.visitFile(RepositoryImportSource.java:83)
      	at com.sonatype.nexus.exportimport.internal.importtask.RepositoryImportSource$RepositoryImportFileVisitor.visitFile(RepositoryImportSource.java:1)
      	at java.nio.file.Files.walkFileTree(Files.java:2670)
      	at com.sonatype.nexus.exportimport.internal.importtask.RepositoryImportSource.walk(RepositoryImportSource.java:45)
      	at com.sonatype.nexus.exportimport.orient.internal.importtask.OrientRepositoryImportService.walkImportDirectory(OrientRepositoryImportService.java:140)
      	at com.sonatype.nexus.exportimport.orient.internal.importtask.OrientRepositoryImportService.doImport(OrientRepositoryImportService.java:125)
      	at com.sonatype.nexus.exportimport.internal.importtask.RepositoryImportTask.execute(RepositoryImportTask.java:64)
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:79)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	
      
      sha512sum pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar
      caff198f65d21d25d582f467d1ac78dc72903a7aace9a5380503839032cf663dd5871221f6488123b5fa2a4b07b728ba1827657d052889d65fa874e93fc368cf  ../pmd-test/pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar
      

      A sample partial 56GB import from Central repository found several artifacts which trigger this problem:

      > grep 'Detected content type' tasks/repository.import-20210108175634047.log | grep -E "256|512"
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: org/mortbay/jetty/jetty-util/6.1.15.rc2/jetty-util-6.1.15.rc2.jar.sha512
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: org/codehaus/fabric3/fabric3-management-api/0.6/fabric3-management-api-0.6.jar.sha256
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: org/apache/apache-jar-resource-bundle/1.4/apache-jar-resource-bundle-1.4-sources.jar.asc.sha256
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: woodstox/wstx-asl/1.8.2/wstx-asl-1.8.2.pom.sha256
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: pmd/pmd-jdk14/4.2/pmd-jdk14-4.2.jar.sha512
      org.sonatype.nexus.repository.InvalidContentException: Detected content type [audio/x-caf], but expected [text/plain]: turbine/turbine/2.2b1/turbine-2.2b1.jar.sha512
      

      This appears to be a variation of issues previously seen where checksums are being processed by content validation algorithms. Now that sha512 and sha256 sums are being generated and supported by NXRM, the requirement to not process these sum files by content validation does not appear to be implemented.

      Expected

      As in NEXUS-20340, avoid complex mime-type content validation for Maven hashes, including sha256 and sha512. Instead use a much simpler algorithm for proxying content validation. For import where NXRM generates these checksums, use no validation since it is implied the hash will be valid. Make sure all uses of content validation are fixed, including proxy repo processing and hosted repo import.

        Attachments

          Activity

            People

            Assignee:
            mbucher Michael Bucher
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Dawid Sawa Dawid Sawa
            Team:
            NXRM - Groot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title