Dev - Nexus
  1. Dev - Nexus
  2. NEXUS-3743

Bad SHA1 created for ivy.xml artifact

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not A Bug
    • Affects Version/s: 1.7.2
    • Fix Version/s: None
    • Component/s: Repository
    • Labels:
      None
    • Environment:
      Uname -a: 5.11 snv_111b i86pc i386 i86pc Solaris

      Description

      Artifact 'ivy.xml' has a SHA1 checksum of: 'e779e3e74363e546fc22e1550b3655fd6684d4a3' when created by the Ant task '<checksum algorithm="sha1">' This checksum is also recognized by Ivy (2.1.0) as correct.

      When 'Rebuild Metadata' is called from the Nexus GUI, a bad SHA1 checksum is created: 'b3357dd9e151664bdea85112311448957ec8f51f'

      This problem ONLY occurs if the artifact is named 'ivy.xml'. If you rename the file to anything else the checksum is correctly generated by the 'Rebuild Metadata' call in Nexus.

      The ivy.xml file is attached.

      1. ivy.xml
        0.9 kB
        Todd Merrill
      2. test.ivy.xml
        1 kB
        Christian Höfig

        Activity

        Hide
        Randy Reddekopp added a comment -

        I have noticed the same issue with one minor difference, I have the version number within the ivy.xml file. For example "ivy-0.0.2.xml".

        Nexus calculates the wrong sha1 AND md5 values. In turn this breaks our dependency management in our projects.

        Show
        Randy Reddekopp added a comment - I have noticed the same issue with one minor difference, I have the version number within the ivy.xml file. For example "ivy-0.0.2.xml". Nexus calculates the wrong sha1 AND md5 values. In turn this breaks our dependency management in our projects.
        Hide
        Gustavo Chaves added a comment -

        I'm facing the same problem with lots of POM files in my repository. My build log has lots of messages like this:

        11:13:46  Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom
        11:13:46  4K downloaded  (cpqd-commons-i18n-ejb-6.5.2.pom)
        11:13:46  [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - RETRYING
        11:13:46  Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom
        11:13:46  4K downloaded  (cpqd-commons-i18n-ejb-6.5.2.pom)
        11:13:46  [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - IGNORING
        

        I've confirmed that the http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom.sha1 file in Nexus has the wrong contents, i.e., the one refereed to as "remote" above. (The SHA1 calculated by sha1sum is the one referred to as "local" above.)

        I've run manually a "Rebuild Maven Metadata Files" in this repository. The SHA1 files were regenerated but the newer ones have the same wrong contents.

        I'm using Maven 2.2.1, Nexus 1.9.1.1 and JDK 1.6.0_23.

        Show
        Gustavo Chaves added a comment - I'm facing the same problem with lots of POM files in my repository. My build log has lots of messages like this: 11:13:46 Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom 11:13:46 4K downloaded (cpqd-commons-i18n-ejb-6.5.2.pom) 11:13:46 [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - RETRYING 11:13:46 Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom 11:13:46 4K downloaded (cpqd-commons-i18n-ejb-6.5.2.pom) 11:13:46 [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - IGNORING I've confirmed that the http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom.sha1 file in Nexus has the wrong contents, i.e., the one refereed to as "remote" above. (The SHA1 calculated by sha1sum is the one referred to as "local" above.) I've run manually a "Rebuild Maven Metadata Files" in this repository. The SHA1 files were regenerated but the newer ones have the same wrong contents. I'm using Maven 2.2.1, Nexus 1.9.1.1 and JDK 1.6.0_23.
        Hide
        Brian Demers added a comment -

        Todd, after downloading the attached ivy.xml the sha1 is: 8cf3988f697168f860918bf48136228ec0d5bce8
        this doesn't match either of sha1's in the description

        Randy, Gustavo, can you provide more details on where your files came from? Where they deployed to nexus, if so how? Is this from a proxy repository? How was the original sha1's calculated and uploaded?

        Show
        Brian Demers added a comment - Todd, after downloading the attached ivy.xml the sha1 is: 8cf3988f697168f860918bf48136228ec0d5bce8 this doesn't match either of sha1's in the description Randy, Gustavo, can you provide more details on where your files came from? Where they deployed to nexus, if so how? Is this from a proxy repository? How was the original sha1's calculated and uploaded?
        Hide
        Gustavo Chaves added a comment -

        Brian, I've just found NEXUS-4347 which is probably what's causing my problems. I'm using maven 2.2.1. I'm gonna try the solution of prefixing our Nexus URLs with "dav:". I'll let you know how that goes. (It will take a while since I depend on other to perform the test for me.)

        Show
        Gustavo Chaves added a comment - Brian, I've just found NEXUS-4347 which is probably what's causing my problems. I'm using maven 2.2.1. I'm gonna try the solution of prefixing our Nexus URLs with "dav:". I'll let you know how that goes. (It will take a while since I depend on other to perform the test for me.)
        Hide
        Christian Höfig added a comment - - edited

        After upgrading to 1.9.2.4, (mvn 2.2.1) I ran into the wrong ivy.xml.sha1 issue again.

        What I do is:

        go to the nexus gui, browse repository, select create metadata => ivy.xml.sha1 gets (re)created BUT IS WRONG.

        I can fix this with an ant call that creates the correct checksum, but I have to remember to do it everytime, when recreate metadata was initiated.

        -work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
        8639c9fd19e8b36e30164b993d05d88f35d9eb05

        wrong value, after recreate metadata

        fixed with ant call to:

        nexus@server:/var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
        139b246c832c52a706fe554f9d96e8c039fd775c

        if recreate metadata is called again, the value is wrong again

        Show
        Christian Höfig added a comment - - edited After upgrading to 1.9.2.4, (mvn 2.2.1) I ran into the wrong ivy.xml.sha1 issue again. What I do is: go to the nexus gui, browse repository, select create metadata => ivy.xml.sha1 gets (re)created BUT IS WRONG. I can fix this with an ant call that creates the correct checksum, but I have to remember to do it everytime, when recreate metadata was initiated. -work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1 8639c9fd19e8b36e30164b993d05d88f35d9eb05 wrong value, after recreate metadata fixed with ant call to: nexus@server:/var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1 139b246c832c52a706fe554f9d96e8c039fd775c if recreate metadata is called again, the value is wrong again
        Hide
        Christian Höfig added a comment -

        using digest under solaris gives the same sha1 value as does ant, so nexus gui definitely produces the wrong sha1 value

        /var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)digest -a sha1 ivy.xml
        139b246c832c52a706fe554f9d96e8c039fd775c

        Show
        Christian Höfig added a comment - using digest under solaris gives the same sha1 value as does ant, so nexus gui definitely produces the wrong sha1 value /var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)digest -a sha1 ivy.xml 139b246c832c52a706fe554f9d96e8c039fd775c
        Hide
        Tamás Cservenák added a comment -

        @Todd: as Brian noted, the attached file is not the one you talk about, none of the checksums matches yours. I got same SHA1 as he did: 8cf3988f697168f860918bf48136228ec0d5bce8

        @Randy, @Gustavo: you did not ping back did you fix client side checksumming problems or not.

        @Christian: I am unable to reproduce any of these problems using attached ivy.xml deployed to "Third Party" repository as you have. Both, Nexus and openssl sha1 CLI tool creates same SHA1 for attached file. Are you sure it is not something external deploying this file? Can you give me a third checksum of same file (that once has 8639c9fd19e8b36e30164b993d05d88f35d9eb05 and once has 139b246c832c52a706fe554f9d96e8c039fd775c checksum) calculated by openssl sha1 (or alike, but not Nexus nor Ant)? Are you sure the file does not changes (the timestamp of it should be older or equal to timestamps of the SHA1/MD5 files)?

        For example, if you do deploy this file, but you do not deploy it's checksums, Nexus will not recreate them unless asked for. Hence, after something deploys this file, the hashes of previous file might be wrong for you.

        Show
        Tamás Cservenák added a comment - @Todd: as Brian noted, the attached file is not the one you talk about, none of the checksums matches yours. I got same SHA1 as he did: 8cf3988f697168f860918bf48136228ec0d5bce8 @Randy, @Gustavo: you did not ping back did you fix client side checksumming problems or not. @Christian: I am unable to reproduce any of these problems using attached ivy.xml deployed to "Third Party" repository as you have. Both, Nexus and openssl sha1 CLI tool creates same SHA1 for attached file. Are you sure it is not something external deploying this file? Can you give me a third checksum of same file (that once has 8639c9fd19e8b36e30164b993d05d88f35d9eb05 and once has 139b246c832c52a706fe554f9d96e8c039fd775c checksum) calculated by openssl sha1 (or alike, but not Nexus nor Ant)? Are you sure the file does not changes (the timestamp of it should be older or equal to timestamps of the SHA1/MD5 files)? For example, if you do deploy this file, but you do not deploy it's checksums, Nexus will not recreate them unless asked for. Hence, after something deploys this file, the hashes of previous file might be wrong for you.
        Hide
        Christian Höfig added a comment -

        the ivy.xml file, where nexus creates wrong ivy.xml.sha1 file

        Show
        Christian Höfig added a comment - the ivy.xml file, where nexus creates wrong ivy.xml.sha1 file
        Hide
        Christian Höfig added a comment -

        just to be sure, I download test.ivy.xml from this page again, transferred it to my solaris machine and did the digest -a sha1 (sorry, no openssl on that machine) on the file, I get

        139b246c832c52a706fe554f9d96e8c039fd775c

        which is the correct value for ivy to process, and the same that ant produces.

        When I ran rebuild metadata on my directory xp3_min/1.1.4c, the ivy.xml file does NOT change (time stamp is from last year). But all the md5 and sha1 files change - this is the expected behaviour. That way, I know that nexus created the md5 and sha1 files:

        rw-rr- 1 nexus was 1137 Sep 22 2011 ivy.xml
        rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.md5
        rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.sha1
        rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.pom.md5
        rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.pom.sha1
        rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.jar.md5
        rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.jar.sha1
        rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.md5
        rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.sha1
        rw-rr- 1 nexus was 32 Apr 11 13:41 ivy.xml.md5
        rw-rr- 1 nexus was 40 Apr 11 13:41 ivy.xml.sha1

        immediately after doing the recreate metadata for that directory, the ivy.xml.sha1 contents is wrong:

        /var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
        8639c9fd19e8b36e30164b993d05d88f35d9eb05n

        (and I go and fix it with my create-ivy-sha1 ant tool).

        Show
        Christian Höfig added a comment - just to be sure, I download test.ivy.xml from this page again, transferred it to my solaris machine and did the digest -a sha1 (sorry, no openssl on that machine) on the file, I get 139b246c832c52a706fe554f9d96e8c039fd775c which is the correct value for ivy to process, and the same that ant produces. When I ran rebuild metadata on my directory xp3_min/1.1.4c, the ivy.xml file does NOT change (time stamp is from last year). But all the md5 and sha1 files change - this is the expected behaviour. That way, I know that nexus created the md5 and sha1 files: rw-r r - 1 nexus was 1137 Sep 22 2011 ivy.xml rw-r r - 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.md5 rw-r r - 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.sha1 rw-r r - 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.pom.md5 rw-r r - 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.pom.sha1 rw-r r - 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.jar.md5 rw-r r - 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.jar.sha1 rw-r r - 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.md5 rw-r r - 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.sha1 rw-r r - 1 nexus was 32 Apr 11 13:41 ivy.xml.md5 rw-r r - 1 nexus was 40 Apr 11 13:41 ivy.xml.sha1 immediately after doing the recreate metadata for that directory, the ivy.xml.sha1 contents is wrong: /var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1 8639c9fd19e8b36e30164b993d05d88f35d9eb05n (and I go and fix it with my create-ivy-sha1 ant tool).
        Hide
        Christian Höfig added a comment -

        I think do to a transfer issue (dos \r\n versus just \n) the sha1sum under windows will show a different value for the files. If I look at the files on my PC, using cygwin, /usr/bin/sha1sum it will show a new, confusingly different value (7589a00c50f69777931a86ab26784b0bc674040b). But if I do a dos2unix (which replaces \r\n to \n) the sha1sum is:

        139b246c832c52a706fe554f9d96e8c039fd775c

        Do we have a dos2unix issue on the checksum calculation ?

        Show
        Christian Höfig added a comment - I think do to a transfer issue (dos \r\n versus just \n) the sha1sum under windows will show a different value for the files. If I look at the files on my PC, using cygwin, /usr/bin/sha1sum it will show a new, confusingly different value (7589a00c50f69777931a86ab26784b0bc674040b). But if I do a dos2unix (which replaces \r\n to \n) the sha1sum is: 139b246c832c52a706fe554f9d96e8c039fd775c Do we have a dos2unix issue on the checksum calculation ?
        Hide
        Christian Höfig added a comment -

        typo, I meant to write: "I think due to a transfer issue..."

        Show
        Christian Höfig added a comment - typo, I meant to write: "I think due to a transfer issue..."
        Hide
        Christian Höfig added a comment - - edited

        found openssl, same sha1 value on the unix system:
        nexus@server:~)/opt/CollabNet_Subversion/bin/openssl sha1 test.ivy.xml
        SHA1(test.ivy.xml)= 139b246c832c52a706fe554f9d96e8c039fd775c

        Show
        Christian Höfig added a comment - - edited found openssl, same sha1 value on the unix system: nexus@server:~)/opt/CollabNet_Subversion/bin/openssl sha1 test.ivy.xml SHA1(test.ivy.xml)= 139b246c832c52a706fe554f9d96e8c039fd775c
        Hide
        Randy Reddekopp added a comment -

        @Tamas: Sorry, I have since changed employers and have not worked on the problem. From what I recall I used some kind of work around by deleting the files with the Nexus GUI, rebuilt the cache and metadata, then re-uploaded the ivy.xml. If I have time in the future I may look into debugging the issue, but time is quite tight at the moment.

        Show
        Randy Reddekopp added a comment - @Tamas: Sorry, I have since changed employers and have not worked on the problem. From what I recall I used some kind of work around by deleting the files with the Nexus GUI, rebuilt the cache and metadata, then re-uploaded the ivy.xml. If I have time in the future I may look into debugging the issue, but time is quite tight at the moment.
        Hide
        Rich Seddon added a comment -

        Just to be clear on this:

        Nexus does not create the checksum file during a deploy, this is computed and uploaded by the client. If you think about it, that is the way it has to be, the checksum file is used to verify the artifact wasn't corrupted during upload and storage.

        Not sure exactly what is going wrong here, but it's not unknown for there to be client bugs that affect checksum calculation. See here for example:

        http://jira.codehaus.org/browse/WAGON-277

        Nexus will recompute the checksum if you run a "rebuild metadata" operation. This is not intended for normal use though, it's best thought of as a repair operation.

        Show
        Rich Seddon added a comment - Just to be clear on this: Nexus does not create the checksum file during a deploy, this is computed and uploaded by the client. If you think about it, that is the way it has to be, the checksum file is used to verify the artifact wasn't corrupted during upload and storage. Not sure exactly what is going wrong here, but it's not unknown for there to be client bugs that affect checksum calculation. See here for example: http://jira.codehaus.org/browse/WAGON-277 Nexus will recompute the checksum if you run a "rebuild metadata" operation. This is not intended for normal use though, it's best thought of as a repair operation.
        Hide
        Todd Merrill added a comment -

        @Rich: This problem ONLY appears when "rebuild metadata" is called from the GUI on existing artifacts. No deploying is involved. If "rebuild metadata" is accidentally clicked by an admin, all our ivy builds begin crashing and we have to regenerate the .SHA1 files ourselves with a script.

        Show
        Todd Merrill added a comment - @Rich: This problem ONLY appears when "rebuild metadata" is called from the GUI on existing artifacts. No deploying is involved. If "rebuild metadata" is accidentally clicked by an admin, all our ivy builds begin crashing and we have to regenerate the .SHA1 files ourselves with a script.
        Hide
        Todd Merrill added a comment -

        @Tamás: The problem with my uploaded ivy.xml was because I had to use a Windows client to upload it, so the file endings are wrong. Sorry, but I actually have no direct internet access from a Unix-based client here. If you just run the dos2unix util that comes with most distribs, you should see the described results.

        Show
        Todd Merrill added a comment - @Tamás: The problem with my uploaded ivy.xml was because I had to use a Windows client to upload it, so the file endings are wrong. Sorry, but I actually have no direct internet access from a Unix-based client here. If you just run the dos2unix util that comes with most distribs, you should see the described results.
        Hide
        Rich Seddon added a comment -

        The original sha1sum for test.ivy.xml is 7589a00c50f69777931a86ab26784b0bc674040b

        If I run dos2unix against the attached test.ivy.xml the sha1 is 139b246c832c52a706fe554f9d96e8c039fd775c

        It looks to me like the content-type header is be getting set incorrectly when this file is uploaded, which is causing the line endings to be translated.

        Show
        Rich Seddon added a comment - The original sha1sum for test.ivy.xml is 7589a00c50f69777931a86ab26784b0bc674040b If I run dos2unix against the attached test.ivy.xml the sha1 is 139b246c832c52a706fe554f9d96e8c039fd775c It looks to me like the content-type header is be getting set incorrectly when this file is uploaded, which is causing the line endings to be translated.
        Hide
        Todd Merrill added a comment -

        @Rich: My file is 'ivy.xml'. 'test.ivy.xml' is the one uploaded by Christian Höfig.

        Show
        Todd Merrill added a comment - @Rich: My file is 'ivy.xml'. 'test.ivy.xml' is the one uploaded by Christian Höfig.
        Hide
        Christian Höfig added a comment -

        @Tamas: can I reach you by mail ? my mail is c.hoefig@ing-diba.de

        Show
        Christian Höfig added a comment - @Tamas: can I reach you by mail ? my mail is c.hoefig@ing-diba.de
        Hide
        Tamás Cservenák added a comment -

        Ok, so to recap: I cannot reproduce this. Here is what I did:

        I downloaded now both attached files from this issue. First, I did confirm with Hex editor, they do have DOS line endings (verified sequence of 0x0D 0x0A). Then I converted the two files (by creating two new files) to have Unix line endings. Then checksummed all the 4 files using openssl, and got checksums that do match all of those written in all comments. Then I put these files "under" Nexus (into thirdparty repository), and made Nexus to "Recreate (Maven) metadata". And finally, compared what Nexus did with hashes got from openssl. No difference, all are as they should. Here is the transcript:

        [cstamas@Marvin NEXUS-3743]$ l
        total 32
        drwxr-xr-x   6 cstamas  staff   204 Ápr 12 14:07 .
        drwxr-xr-x  40 cstamas  staff  1360 Ápr 12 14:01 ..
        -rw-r--r--@  1 cstamas  staff   970 Ápr 11 14:17 ivy.xml
        -rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:01 test.ivy.xml
        [cstamas@Marvin NEXUS-3743]$ openssl sha1 *
        SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8
        SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b
        [cstamas@Marvin NEXUS-3743]$ cat ivy.xml | tr -d '\r' > ivy.xml.unix
        [cstamas@Marvin NEXUS-3743]$ cat test.ivy.xml | tr -d '\r' > test.ivy.xml.unix
        [cstamas@Marvin NEXUS-3743]$ l
        total 32
        drwxr-xr-x   6 cstamas  staff   204 Ápr 12 14:07 .
        drwxr-xr-x  40 cstamas  staff  1360 Ápr 12 14:01 ..
        -rw-r--r--@  1 cstamas  staff   970 Ápr 11 14:17 ivy.xml
        -rw-r--r--   1 cstamas  staff   951 Ápr 12 14:06 ivy.xml.unix
        -rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:01 test.ivy.xml
        -rw-r--r--   1 cstamas  staff  1137 Ápr 12 14:07 test.ivy.xml.unix
        [cstamas@Marvin NEXUS-3743]$ openssl sha1 *
        SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8
        SHA1(ivy.xml.unix)= e779e3e74363e546fc22e1550b3655fd6684d4a3
        SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b
        SHA1(test.ivy.xml.unix)= 139b246c832c52a706fe554f9d96e8c039fd775c
        [cstamas@Marvin NEXUS-3743]$ cp * ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/
        
        here I ran the "rebuild metadata" against Thirdparty repository on Nexus UI
        
        [cstamas@Marvin NEXUS-3743]$ l ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/
        total 120
        drwxr-xr-x  19 cstamas  staff   646 Ápr 12 14:09 .
        drwxr-xr-x  10 cstamas  staff   340 Ápr 12 14:08 ..
        drwxr-xr-x   3 cstamas  staff   102 Ápr 12 14:09 .meta
        drwxr-xr-x   3 cstamas  staff   102 Ápr 12 14:08 .nexus
        -rw-r--r--   1 cstamas  staff    25 Ápr 12 14:09 archetype-catalog.xml
        -rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 archetype-catalog.xml.md5
        -rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 archetype-catalog.xml.sha1
        -rw-r--r--@  1 cstamas  staff   970 Ápr 12 14:09 ivy.xml
        -rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 ivy.xml.md5
        -rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 ivy.xml.sha1
        -rw-r--r--   1 cstamas  staff   951 Ápr 12 14:09 ivy.xml.unix
        -rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 ivy.xml.unix.md5
        -rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 ivy.xml.unix.sha1
        -rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:09 test.ivy.xml
        -rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 test.ivy.xml.md5
        -rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 test.ivy.xml.sha1
        -rw-r--r--   1 cstamas  staff  1137 Ápr 12 14:09 test.ivy.xml.unix
        -rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 test.ivy.xml.unix.md5
        -rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 test.ivy.xml.unix.sha1
        [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.sha1 
        8cf3988f697168f860918bf48136228ec0d5bce8
        [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.unix.sha1 
        e779e3e74363e546fc22e1550b3655fd6684d4a3
        [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.sha1 
        7589a00c50f69777931a86ab26784b0bc674040b
        [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.unix.sha1 
        139b246c832c52a706fe554f9d96e8c039fd775c
        [cstamas@Marvin NEXUS-3743]$
        
        Show
        Tamás Cservenák added a comment - Ok, so to recap: I cannot reproduce this. Here is what I did: I downloaded now both attached files from this issue. First, I did confirm with Hex editor, they do have DOS line endings (verified sequence of 0x0D 0x0A ). Then I converted the two files (by creating two new files) to have Unix line endings. Then checksummed all the 4 files using openssl , and got checksums that do match all of those written in all comments. Then I put these files "under" Nexus (into thirdparty repository), and made Nexus to "Recreate (Maven) metadata". And finally, compared what Nexus did with hashes got from openssl . No difference, all are as they should. Here is the transcript: [cstamas@Marvin NEXUS-3743]$ l total 32 drwxr-xr-x 6 cstamas staff 204 Ápr 12 14:07 . drwxr-xr-x 40 cstamas staff 1360 Ápr 12 14:01 .. -rw-r--r--@ 1 cstamas staff 970 Ápr 11 14:17 ivy.xml -rw-r--r--@ 1 cstamas staff 1157 Ápr 12 14:01 test.ivy.xml [cstamas@Marvin NEXUS-3743]$ openssl sha1 * SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8 SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b [cstamas@Marvin NEXUS-3743]$ cat ivy.xml | tr -d '\r' > ivy.xml.unix [cstamas@Marvin NEXUS-3743]$ cat test.ivy.xml | tr -d '\r' > test.ivy.xml.unix [cstamas@Marvin NEXUS-3743]$ l total 32 drwxr-xr-x 6 cstamas staff 204 Ápr 12 14:07 . drwxr-xr-x 40 cstamas staff 1360 Ápr 12 14:01 .. -rw-r--r--@ 1 cstamas staff 970 Ápr 11 14:17 ivy.xml -rw-r--r-- 1 cstamas staff 951 Ápr 12 14:06 ivy.xml.unix -rw-r--r--@ 1 cstamas staff 1157 Ápr 12 14:01 test.ivy.xml -rw-r--r-- 1 cstamas staff 1137 Ápr 12 14:07 test.ivy.xml.unix [cstamas@Marvin NEXUS-3743]$ openssl sha1 * SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8 SHA1(ivy.xml.unix)= e779e3e74363e546fc22e1550b3655fd6684d4a3 SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b SHA1(test.ivy.xml.unix)= 139b246c832c52a706fe554f9d96e8c039fd775c [cstamas@Marvin NEXUS-3743]$ cp * ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ here I ran the "rebuild metadata" against Thirdparty repository on Nexus UI [cstamas@Marvin NEXUS-3743]$ l ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ total 120 drwxr-xr-x 19 cstamas staff 646 Ápr 12 14:09 . drwxr-xr-x 10 cstamas staff 340 Ápr 12 14:08 .. drwxr-xr-x 3 cstamas staff 102 Ápr 12 14:09 .meta drwxr-xr-x 3 cstamas staff 102 Ápr 12 14:08 .nexus -rw-r--r-- 1 cstamas staff 25 Ápr 12 14:09 archetype-catalog.xml -rw-r--r-- 1 cstamas staff 32 Ápr 12 14:09 archetype-catalog.xml.md5 -rw-r--r-- 1 cstamas staff 40 Ápr 12 14:09 archetype-catalog.xml.sha1 -rw-r--r--@ 1 cstamas staff 970 Ápr 12 14:09 ivy.xml -rw-r--r-- 1 cstamas staff 32 Ápr 12 14:09 ivy.xml.md5 -rw-r--r-- 1 cstamas staff 40 Ápr 12 14:09 ivy.xml.sha1 -rw-r--r-- 1 cstamas staff 951 Ápr 12 14:09 ivy.xml.unix -rw-r--r-- 1 cstamas staff 32 Ápr 12 14:09 ivy.xml.unix.md5 -rw-r--r-- 1 cstamas staff 40 Ápr 12 14:09 ivy.xml.unix.sha1 -rw-r--r--@ 1 cstamas staff 1157 Ápr 12 14:09 test.ivy.xml -rw-r--r-- 1 cstamas staff 32 Ápr 12 14:09 test.ivy.xml.md5 -rw-r--r-- 1 cstamas staff 40 Ápr 12 14:09 test.ivy.xml.sha1 -rw-r--r-- 1 cstamas staff 1137 Ápr 12 14:09 test.ivy.xml.unix -rw-r--r-- 1 cstamas staff 32 Ápr 12 14:09 test.ivy.xml.unix.md5 -rw-r--r-- 1 cstamas staff 40 Ápr 12 14:09 test.ivy.xml.unix.sha1 [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.sha1 8cf3988f697168f860918bf48136228ec0d5bce8 [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.unix.sha1 e779e3e74363e546fc22e1550b3655fd6684d4a3 [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.sha1 7589a00c50f69777931a86ab26784b0bc674040b [cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.unix.sha1 139b246c832c52a706fe554f9d96e8c039fd775c [cstamas@Marvin NEXUS-3743]$
        Hide
        Tamás Cservenák added a comment -

        @Christian "Do we have a dos2unix issue on the checksum calculation ?"

        Naturally, no, we do not parse XML or treat the file content in any other way than just "stream of bytes". Hence, we do not care about line endings either

        Show
        Tamás Cservenák added a comment - @Christian "Do we have a dos2unix issue on the checksum calculation ?" Naturally, no, we do not parse XML or treat the file content in any other way than just "stream of bytes". Hence, we do not care about line endings either
        Hide
        Tamás Cservenák added a comment -

        I am still at standpoint that "something else", other than Nexus either modifies the checksummed file (the ivy.xml), or the hashes, unsure. I simply cannot make my Nexus (on Mac OSX) do the same.

        Christian and Todd are on Solaris. You Randy? I do not have access to Solaris OS, but is it possible it has to do something with OS and/or env settings? Nothing else I can think of... we use JRE's DigestInputStream and MessageDigest to perform the hash calculations, and the fact they both are bad (Randy says SHA1 and MD5 are wrong) might me assume there is some problem on actual input?

        Show
        Tamás Cservenák added a comment - I am still at standpoint that "something else", other than Nexus either modifies the checksummed file (the ivy.xml), or the hashes, unsure. I simply cannot make my Nexus (on Mac OSX) do the same. Christian and Todd are on Solaris. You Randy? I do not have access to Solaris OS, but is it possible it has to do something with OS and/or env settings? Nothing else I can think of... we use JRE's DigestInputStream and MessageDigest to perform the hash calculations, and the fact they both are bad (Randy says SHA1 and MD5 are wrong) might me assume there is some problem on actual input?
        Hide
        Todd Merrill added a comment -

        Our default LANG setting is de.ISO8859-15

        Could it have something to do with an encoding config conflict somewhere? I looked over the Nexus sources and could see no special hash creation code, just streams opening and passing to the Java libs.

        The ivy.xml files are generated in place in the repo from existing POMs, not loaded through a deployment of any kind if that makes a difference.

        If you need specific environment info let us know.

        Show
        Todd Merrill added a comment - Our default LANG setting is de.ISO8859-15 Could it have something to do with an encoding config conflict somewhere? I looked over the Nexus sources and could see no special hash creation code, just streams opening and passing to the Java libs. The ivy.xml files are generated in place in the repo from existing POMs, not loaded through a deployment of any kind if that makes a difference. If you need specific environment info let us know.
        Hide
        Tamás Cservenák added a comment -

        Wait a moment... you gave me an idea: Nexus stores file attributes (simply, a "map" of key-vals). By definition, all files deployed (uploaded over HTTP) are being checksummed during transport, and the two checksum SHA1/MD5 are stored into attributes.

        When nexus "rebuilds" hashes, it only does read-calc-checksum as last resort, "worst scenario", that is needed usually for files that just "pops up" out of the blue on the disk, not deployed "normal way" (over HTTP), hence, have no attributes either.

        This is same thing I did above: I did a cp of the files in their place "under" Nexus, and Nexus just "got a knowledge about them" when I "touch" them (by any means, I did it by "Recreate Metadata" task that crawls repository, but same would happen if I'd ask the file over HTTP from Nexus). The file would be present on disk, but for Nexus, they came "out of blue", and on the fly re-checksums them, this was the "worse case" scenario.

        But, if the file in question has attributes, it would reach for them instead.

        So, please Todd, do a cat of the attribute file of the ivy.xml file in question, it has same path and name as "original" file but is a small XML file actually:

        sonatype-work/nexus/proxy/attributes/....repo path of the ivy.xml in question

        And look for keys "digest.md5" and "digest.sha1" in it. I have a beer to bet on values you find...

        Show
        Tamás Cservenák added a comment - Wait a moment... you gave me an idea: Nexus stores file attributes (simply, a "map" of key-vals). By definition, all files deployed (uploaded over HTTP) are being checksummed during transport , and the two checksum SHA1/MD5 are stored into attributes. When nexus "rebuilds" hashes, it only does read-calc-checksum as last resort , "worst scenario", that is needed usually for files that just "pops up" out of the blue on the disk, not deployed "normal way" (over HTTP), hence, have no attributes either. This is same thing I did above: I did a cp of the files in their place "under" Nexus, and Nexus just "got a knowledge about them" when I "touch" them (by any means, I did it by "Recreate Metadata" task that crawls repository, but same would happen if I'd ask the file over HTTP from Nexus). The file would be present on disk, but for Nexus, they came "out of blue", and on the fly re-checksums them, this was the "worse case" scenario. But, if the file in question has attributes , it would reach for them instead. So, please Todd, do a cat of the attribute file of the ivy.xml file in question, it has same path and name as "original" file but is a small XML file actually: sonatype-work/nexus/proxy/attributes/....repo path of the ivy.xml in question And look for keys "digest.md5" and "digest.sha1" in it. I have a beer to bet on values you find...
        Hide
        Todd Merrill added a comment -

        And the beer goes to...

        You're right! It's pulling the bad values from the attributes. But how did the bad attributes get there in the first place? Is the hash being created by Nexus before the file is finished being written on disk?

        Show
        Todd Merrill added a comment - And the beer goes to... You're right! It's pulling the bad values from the attributes. But how did the bad attributes get there in the first place? Is the hash being created by Nexus before the file is finished being written on disk?
        Hide
        Tamás Cservenák added a comment -

        Well, that in short means: the file was once "discovered" by Nexus, and back then the file did have those checksums. But later, that file content did change. Both events might happened long in past, only relation between them is that T-appeared < T-changed.

        Now, this "protection" I mentioned, the "auto attribute creation" – to create attributes for newly discovered files in storage – was actually a circumvention. Nexus storage folders should not be modified by anything else than Nexus itself. This circumvention was to ease "transitioning" for people still doing "ol'school" SCP deploys, and introducing Nexus gradually, and targeting complete switch to HTTP transport.

        This protection does it's job when file appears. But it does not when file is changed by process other than Nexus.

        You decide, is this a bug or not, and create an issue if you think it is. But, my reasoning is that nothing under sonatype-work/nexus/storage/... folders should be changed by anything else than Nexus. Nexus for it's basic operation (and many other features) does the bookkeeping of these "attributes" of files, but it assumes "he owns the (content) files", and he keeps attribute files in sync.

        Show
        Tamás Cservenák added a comment - Well, that in short means: the file was once "discovered" by Nexus, and back then the file did have those checksums. But later, that file content did change. Both events might happened long in past, only relation between them is that T-appeared < T-changed . Now, this "protection" I mentioned, the "auto attribute creation" – to create attributes for newly discovered files in storage – was actually a circumvention . Nexus storage folders should not be modified by anything else than Nexus itself. This circumvention was to ease "transitioning" for people still doing "ol'school" SCP deploys, and introducing Nexus gradually, and targeting complete switch to HTTP transport. This protection does it's job when file appears. But it does not when file is changed by process other than Nexus. You decide, is this a bug or not, and create an issue if you think it is. But, my reasoning is that nothing under sonatype-work/nexus/storage/... folders should be changed by anything else than Nexus. Nexus for it's basic operation (and many other features) does the bookkeeping of these "attributes" of files, but it assumes "he owns the (content) files", and he keeps attribute files in sync.
        Hide
        Randy Reddekopp added a comment -

        Good find Tamas! I'm also on the fence if this should be reported as a bug or not. I understand the logic, but to save on future user error it would be nice to at least get some notification from Nexus saying something like "File has been changed externally to Nexus since previously catalogued, checksum values will be out of sync".

        P.S. I was running Nexus on a Scientific (Redhat) 6.0 or 6.1 server.

        Show
        Randy Reddekopp added a comment - Good find Tamas! I'm also on the fence if this should be reported as a bug or not. I understand the logic, but to save on future user error it would be nice to at least get some notification from Nexus saying something like "File has been changed externally to Nexus since previously catalogued, checksum values will be out of sync". P.S. I was running Nexus on a Scientific (Redhat) 6.0 or 6.1 server.
        Hide
        Todd Merrill added a comment -

        OK, yes that makes sense. When a difference is found (frequently much later) between Maven's transitive dependency resolution and Ivy's, we usually make a change to the ivy file in place so both Ivy and Maven builds for a specific version function similarly. We regenerate the checksums ourselves and all works well until someone clicks "rebuild metadata".

        I deleted the attribute file and it regenerated correctly.

        Is it safe to delete attribute files at will if we have no custom attributes?

        Show
        Todd Merrill added a comment - OK, yes that makes sense. When a difference is found (frequently much later) between Maven's transitive dependency resolution and Ivy's, we usually make a change to the ivy file in place so both Ivy and Maven builds for a specific version function similarly. We regenerate the checksums ourselves and all works well until someone clicks "rebuild metadata". I deleted the attribute file and it regenerated correctly. Is it safe to delete attribute files at will if we have no custom attributes?
        Hide
        Tamás Cservenák added a comment -

        Yes, deleting the attribute along with content (or deleting the attribute along with modifying content) should "fix" this.

        "Fix", as in a moment you think about potential concurrency issues... (Nexus might be serving up the exact file you are modifying/deleting).

        Show
        Tamás Cservenák added a comment - Yes, deleting the attribute along with content (or deleting the attribute along with modifying content) should "fix" this. "Fix", as in a moment you think about potential concurrency issues... (Nexus might be serving up the exact file you are modifying/deleting).
        Hide
        Todd Merrill added a comment -

        Not really a problem for us, since if we're having to edit the file our inbox is already full to the brim with messages from angry developers

        Thanks very much for the help, Tamás!

        I realize this isn't how Nexus is normally used, but I doubt many Nexus users serve both ivy and maven descriptors to their teams either. Despite directly messing with the repository, Nexus is mostly stable.

        Show
        Todd Merrill added a comment - Not really a problem for us, since if we're having to edit the file our inbox is already full to the brim with messages from angry developers Thanks very much for the help, Tamás! I realize this isn't how Nexus is normally used, but I doubt many Nexus users serve both ivy and maven descriptors to their teams either. Despite directly messing with the repository, Nexus is mostly stable.
        Hide
        Tamás Cservenák added a comment -

        One thing you should consider: same stands for Nexus as for Maven (in broadest sense): I believe what you have there should be a Nexus plugin (same is said for Maven, Mojo mojo mojo...)

        You problem is perfect fit for "capabilities", similar to one that publishes Maven artifacts as OSGi bundles. Not really related, but you can read more about it here:
        https://docs.sonatype.org/display/Nexus/Nexus+OSGi+Experimental+Features+-+Bundle+Maker

        With "similar", I meant, that this plugin above does similar, as in it exposes OSGi bundles "on top" of a Maven2 repository... just like you do with Ivy on top of Maven2 repo...

        Show
        Tamás Cservenák added a comment - One thing you should consider: same stands for Nexus as for Maven (in broadest sense): I believe what you have there should be a Nexus plugin (same is said for Maven, Mojo mojo mojo...) You problem is perfect fit for "capabilities", similar to one that publishes Maven artifacts as OSGi bundles. Not really related, but you can read more about it here: https://docs.sonatype.org/display/Nexus/Nexus+OSGi+Experimental+Features+-+Bundle+Maker With "similar", I meant, that this plugin above does similar, as in it exposes OSGi bundles "on top" of a Maven2 repository... just like you do with Ivy on top of Maven2 repo...
        Hide
        Todd Merrill added a comment -

        Yes, ideally a more integrated solution would be better. Nexus should be controlling all the interactions with the repository. But as usual, it's hard to find the time...

        Show
        Todd Merrill added a comment - Yes, ideally a more integrated solution would be better. Nexus should be controlling all the interactions with the repository. But as usual, it's hard to find the time...
        Hide
        Peter Lynch added a comment -

        After the reading the long set of comments, this looks like Nexus is performing as designed and that this is acceptable.

        Show
        Peter Lynch added a comment - After the reading the long set of comments, this looks like Nexus is performing as designed and that this is acceptable.

          People

          • Assignee:
            Peter Lynch
            Reporter:
            Todd Merrill
          • Votes:
            4 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Date of First Response: