Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-30692

Copying multi-arch manifest fails if docker registry doesn't allow redeploy

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.34.0
    • Fix Version/s: None
    • Component/s: Docker
    • Labels:

      Description

      Hi,

      when we try to copy a container manifest (in our case, generated by podman and copied with skopeo) to a different tag it fails with the following error (mind internal-registry.mydomain.com url):

      skopeo copy --all docker://internal-registry.mydomain.com/python-template/python-template:200 docker://internal-registry.mydomain.com/python-template/python-template:ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd
      ...
      Error: Uploading manifest list failed, attempted the following formats: application/vnd.oci.image.index.v1+json(uploading manifest ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd to internal-registry.mydomain.com/python-template/python-template: received unexpected HTTP status: 500 Server Error), application/vnd.docker.distribution.manifest.list.v2+json(uploading manifest ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd to internal-registry.mydomain.com/python-template/python-template: blob upload invalid: blob upload invalid)

      log error in nexus

      2021-12-22 15:33:49,378+0000 WARN  [qtp687201090-393402]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/python-template/python-template/manifests/ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd2021-12-22 15:33:49,378+0000 WARN  [qtp687201090-393402]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/python-template/python-template/manifests/ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bdjava.lang.NullPointerException: Cannot get property 'digest' on null object at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60) at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:190) at org.codehaus.groovy.runtime.DefaultGroovyMethods.getAt(DefaultGroovyMethods.java:342) at org.codehaus.groovy.runtime.dgm$242.invoke(Unknown Source) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:251) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:58) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128) at org.sonatype.nexus.repository.docker.internal.V2Manifest.referencedDigests(V2Manifest.groovy:88) at org.sonatype.nexus.repository.docker.internal.V2Manifest$referencedDigests$0.call(Unknown Source) at org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl.validateManifest(V2ManifestUtilImpl.groovy:259) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:49) at org.sonatype.nexus.repository.docker.internal.orient.DockerHostedFacetImpl.doPutManifestByDigest(DockerHostedFacetImpl.java:698) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:49) at org.sonatype.nexus.repository.docker.internal.orient.DockerHostedFacetImpl.doPutManifestByTag(DockerHostedFacetImpl.java:667) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57) at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55) at org.sonatype.nexus.repository.docker.internal.orient.DockerHostedFacetImpl.putManifestByTag(DockerHostedFacetImpl.java:652) at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$putManifestByTag$8.call(Unknown Source) at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure8.doCall(V2Handlers.groovy:197) at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.Closure.call(Closure.java:420) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) at com.sun.proxy.$Proxy198.handle(Unknown Source) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:46) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at com.sonatype.nexus.clm.internal.orient.FirewallContributedHandler.handle(FirewallContributedHandler.java:104) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source) at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305) at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.Closure.call(Closure.java:420) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) at com.sun.proxy.$Proxy198.handle(Unknown Source) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source) at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure16.doCall(V2Handlers.groovy:280) at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.Closure.call(Closure.java:420) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) at com.sun.proxy.$Proxy198.handle(Unknown Source) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source) at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93) at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.Closure.call(Closure.java:420) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) at com.sun.proxy.$Proxy198.handle(Unknown Source) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at com.sonatype.analytics.internal.handler.AnalyticsMeteringHandler.handle(AnalyticsMeteringHandler.java:69) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88) at org.sonatype.nexus.repository.view.Context.start(Context.java:179) at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:213) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450) at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang.Thread.run(Thread.java:748)2021-12-22 15:33:49,532+0000 WARN  [qtp687201090-393391]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/python-template/python-template/manifests/ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd: 400 - Repository does not allow updating assets: docker-hosted
      

       It seems that is trying to update a previous manifest when it shouldn't.

      How to reproduce it

      Create a multi-arch manifest (we're using podman and skopeo, but any tool will do it): 

      podman manifest create python-template-multiarch
      podman manifest add python-template-multiarch internal-registry.mydomain.com/python-template/python-template-amd64:200
      podman manifest add python-template-multiarch internal-registry.mydomain.com/python-template/python-template-arm64:200
      podman manifest push --all python-template-multiarch internal-registry.mydomain.com/python-template/python-template:200
      

      As soon as we have our multi-arch manifest (internal-registry.mydomain.com/python-template/python-template:200 with both amd and arm), if we copy it to another tag and push it again, it fails

      skopeo copy --all docker://internal-registry.mydomain.com/python-template/python-template:200 docker://internal-registry.mydomain.com/python-template/python-template:ca9ad327553fbd7a2f3d8ac7df735b837aa3c5bd
      

       

      Note: worth mentioning if we don't use --all option (which copy all the platforms in the source manifest) it creates the second manifest successfully, but it only contains the image for the platform we ran copy command.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jccastillocano Juan Carlos Castillo Cano
            Last Updated By:
            Benjamin Ruland Benjamin Ruland
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Date of First Response:

                tigCommentSecurity.panel-title