Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-10817

publishing npm packages with wrongly encoded ISO-8859-1 JSON fails with 400

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.1, 3.0.2
    • Fix Version/s: 3.1.0
    • Component/s: NPM
    • Labels:
    • Story Points:
      1
    • Sprint:
      Sprint 79

      Description

      It is common knowledge that the official NPM registry contains packages that have wrongly encoded ( non UTF-8 ) pakage.json.

      Nexus tries to handle JSON wrongly encoded as ISO-8859-1, by falling back to that encoding when parsing JSON as UTF-8 fails.

      It seems that the npm client may allow you to attempt publishing package.json that is not encoded properly - need to verify this

      The code from here shows how Nexus performs a fallback when UTF-8 parsing fails:

      @Nonnull
        static NestedAttributesMap parse(final Supplier<InputStream> streamSupplier) throws IOException {
          try {
            final Map<String, Object> backing =
                mapper.<Map<String, Object>>readValue(streamSupplier.get(), rawMapJsonTypeRef);
            return new NestedAttributesMap(String.valueOf(backing.get(NpmMetadataUtils.NAME)), backing);
          }
          catch (JsonParseException e) {
            // fallback
            if (e.getMessage().contains("Invalid UTF-8 middle byte")) {
              // try again, but assume ISO8859-1 encoding now, that is illegal for JSON
              final Map<String, Object> backing =
                  mapper.<Map<String, Object>>readValue(
                      new InputStreamReader(streamSupplier.get(), Charsets.ISO_8859_1),
                      rawMapJsonTypeRef
                  );
              return new NestedAttributesMap(String.valueOf(backing.get(NpmMetadataUtils.NAME)), backing);
            }
            throw new InvalidContentException("Invalid JSON input", e);
          }
        }
      
      

      However, parsing can fail with at least one other variation of exception message ( excluding any localized messages ):

      2016-09-02 10:17:31,147+0200 WARN  [qtp2014951231-9618] deploy.milestones.npm com.sonatype.nexus.repository.npm.internal.NpmHandlers - Error: PUT /0126af95c0e2d9b0a7c78738c4c00a860b04acc8: Status{successful=false, code=400, message='null'} - Invalid JSON input
      org.sonatype.nexus.repository.InvalidContentException: Invalid JSON input
      	at com.sonatype.nexus.repository.npm.internal.NpmJsonUtils.parse(NpmJsonUtils.java:84) [na:na]
      	at com.sonatype.nexus.repository.npm.internal.NpmFacetUtils.parse(NpmFacetUtils.java:93) [na:na]
      	at com.sonatype.nexus.repository.npm.internal.NpmHostedFacetImpl.putPackageRoot(NpmHostedFacetImpl.java:98) [na:na]
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:54) [org.sonatype.nexus.transaction:3.0.1.01]
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:53) [org.sonatype.nexus.transaction:3.0.1.01]
      	at com.sonatype.nexus.repository.npm.internal.NpmHostedFacetImpl.putPackage(NpmHostedFacetImpl.java:81) [na:na]
      	at com.sonatype.nexus.repository.npm.internal.NpmHandlers$3.handle(NpmHandlers.java:172) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at com.sonatype.nexus.repository.npm.internal.NpmHandlers$1.handle(NpmHandlers.java:110) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:45) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:79) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Context.start(Context.java:102) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:58) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43) [org.sonatype.nexus.repository:3.0.1.01]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:198) [org.sonatype.nexus.plugins.nexus-repository-httpbridge:3.0.1.01]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:160) [org.sonatype.nexus.plugins.nexus-repository-httpbridge:3.0.1.01]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:117) [org.sonatype.nexus.plugins.nexus-repository-httpbridge:3.0.1.01]
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api:3.1.0]
      	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [com.google.inject:4.0.0]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) [org.apache.shiro.web:1.2.4]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) [org.apache.shiro.web:1.2.4]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [org.apache.shiro.web:1.2.4]
      	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85) [org.sonatype.nexus.security:3.0.1.01]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [org.apache.shiro.web:1.2.4]
      	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [org.apache.shiro.core:1.2.4]
      	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [org.apache.shiro.core:1.2.4]
      	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [org.apache.shiro.core:1.2.4]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [org.apache.shiro.web:1.2.4]
      	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101) [org.sonatype.nexus.security:3.0.1.01]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.2.4]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) [com.codahale.metrics.servlet:3.0.2]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:63) [org.sonatype.nexus.base:3.0.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:97) [org.sonatype.nexus.base:3.0.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) [com.google.inject:4.0.0]
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) [com.google.inject:4.0.0]
      	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) [org.sonatype.nexus.bootstrap:3.0.1.01]
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [org.eclipse.jetty.security:9.3.7.v20160115]
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [com.codahale.metrics.jetty9:3.0.2]
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.Server.handle(Server.java:517) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at java.lang.Thread.run(Thread.java:812) [na:1.8.0_72]
      Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xa9
       at [Source: sun.nio.ch.ChannelInputStream@3400c0ff; line: 1, column: 3880]
      	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586) [na:na]
      	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521) [na:na]
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3482) [na:na]
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3476) [na:na]
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2503) [na:na]
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2429) [na:na]
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:318) [na:na]
      	at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:514) [na:na]
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:495) [na:na]
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) [na:na]
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:26) [na:na]
      	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3788) [com.fasterxml.jackson.core.jackson-databind:2.7.1]
      	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2856) [com.fasterxml.jackson.core.jackson-databind:2.7.1]
      	at com.sonatype.nexus.repository.npm.internal.NpmJsonUtils.parse(NpmJsonUtils.java:70) [na:na]
      	... 95 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                plynch Peter Lynch
                Last Updated By:
                Michael Prescott
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Date of First Response: