Bad SHA1 created for ivy.xml artifact

I have noticed the same issue with one minor difference, I have the version number within the ivy.xml file. For example "ivy-0.0.2.xml".

Nexus calculates the wrong sha1 AND md5 values. In turn this breaks our dependency management in our projects.

I'm facing the same problem with lots of POM files in my repository. My build log has lots of messages like this:

11:13:46  Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom
11:13:46  4K downloaded  (cpqd-commons-i18n-ejb-6.5.2.pom)
11:13:46  [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - RETRYING
11:13:46  Downloading: http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom
11:13:46  4K downloaded  (cpqd-commons-i18n-ejb-6.5.2.pom)
11:13:46  [WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = '1d1b1982b997b2c19903deb07b313594fea0bcd8'; remote = '6ba06799fbef9a11806c185569fcc0e5b01f0180' - IGNORING

I've confirmed that the http://nexus.cpqd.com.br:8081/nexus/content/groups/dsso/br/com/cpqd/commons/cpqd-commons-i18n-ejb/6.5.2/cpqd-commons-i18n-ejb-6.5.2.pom.sha1 file in Nexus has the wrong contents, i.e., the one refereed to as "remote" above. (The SHA1 calculated by sha1sum is the one referred to as "local" above.)

I've run manually a "Rebuild Maven Metadata Files" in this repository. The SHA1 files were regenerated but the newer ones have the same wrong contents.

I'm using Maven 2.2.1, Nexus 1.9.1.1 and JDK 1.6.0_23.

Todd, after downloading the attached ivy.xml the sha1 is: 8cf3988f697168f860918bf48136228ec0d5bce8
this doesn't match either of sha1's in the description

Randy, Gustavo, can you provide more details on where your files came from? Where they deployed to nexus, if so how? Is this from a proxy repository? How was the original sha1's calculated and uploaded?

Brian, I've just found NEXUS-4347 which is probably what's causing my problems. I'm using maven 2.2.1. I'm gonna try the solution of prefixing our Nexus URLs with "dav:". I'll let you know how that goes. (It will take a while since I depend on other to perform the test for me.)

After upgrading to 1.9.2.4, (mvn 2.2.1) I ran into the wrong ivy.xml.sha1 issue again.

What I do is:

go to the nexus gui, browse repository, select create metadata => ivy.xml.sha1 gets (re)created BUT IS WRONG.

I can fix this with an ant call that creates the correct checksum, but I have to remember to do it everytime, when recreate metadata was initiated.

-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
8639c9fd19e8b36e30164b993d05d88f35d9eb05

wrong value, after recreate metadata

fixed with ant call to:

nexus@server:/var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
139b246c832c52a706fe554f9d96e8c039fd775c

if recreate metadata is called again, the value is wrong again

using digest under solaris gives the same sha1 value as does ant, so nexus gui definitely produces the wrong sha1 value

/var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)digest -a sha1 ivy.xml
139b246c832c52a706fe554f9d96e8c039fd775c

@Todd: as Brian noted, the attached file is not the one you talk about, none of the checksums matches yours. I got same SHA1 as he did: 8cf3988f697168f860918bf48136228ec0d5bce8

@Randy, @Gustavo: you did not ping back did you fix client side checksumming problems or not.

@Christian: I am unable to reproduce any of these problems using attached ivy.xml deployed to "Third Party" repository as you have. Both, Nexus and openssl sha1 CLI tool creates same SHA1 for attached file. Are you sure it is not something external deploying this file? Can you give me a third checksum of same file (that once has 8639c9fd19e8b36e30164b993d05d88f35d9eb05 and once has 139b246c832c52a706fe554f9d96e8c039fd775c checksum) calculated by openssl sha1 (or alike, but not Nexus nor Ant)? Are you sure the file does not changes (the timestamp of it should be older or equal to timestamps of the SHA1/MD5 files)?

For example, if you do deploy this file, but you do not deploy it's checksums, Nexus will not recreate them unless asked for. Hence, after something deploys this file, the hashes of previous file might be wrong for you.

the ivy.xml file, where nexus creates wrong ivy.xml.sha1 file

just to be sure, I download test.ivy.xml from this page again, transferred it to my solaris machine and did the digest -a sha1 (sorry, no openssl on that machine) on the file, I get

139b246c832c52a706fe554f9d96e8c039fd775c

which is the correct value for ivy to process, and the same that ant produces.

When I ran rebuild metadata on my directory xp3_min/1.1.4c, the ivy.xml file does NOT change (time stamp is from last year). But all the md5 and sha1 files change - this is the expected behaviour. That way, I know that nexus created the md5 and sha1 files:

rw-rr- 1 nexus was 1137 Sep 22 2011 ivy.xml
rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.md5
rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-sources.jar.sha1
rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.pom.md5
rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.pom.sha1
rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c.jar.md5
rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c.jar.sha1
rw-rr- 1 nexus was 32 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.md5
rw-rr- 1 nexus was 40 Apr 11 13:41 xpp3_min-1.1.4c-javadoc.jar.sha1
rw-rr- 1 nexus was 32 Apr 11 13:41 ivy.xml.md5
rw-rr- 1 nexus was 40 Apr 11 13:41 ivy.xml.sha1

immediately after doing the recreate metadata for that directory, the ivy.xml.sha1 contents is wrong:

/var/nexus/sonatype-work/nexus/storage/thirdparty/xpp3/xpp3_min/1.1.4c)cat ivy.xml.sha1
8639c9fd19e8b36e30164b993d05d88f35d9eb05n

(and I go and fix it with my create-ivy-sha1 ant tool).

I think do to a transfer issue (dos \r\n versus just \n) the sha1sum under windows will show a different value for the files. If I look at the files on my PC, using cygwin, /usr/bin/sha1sum it will show a new, confusingly different value (7589a00c50f69777931a86ab26784b0bc674040b). But if I do a dos2unix (which replaces \r\n to \n) the sha1sum is:

139b246c832c52a706fe554f9d96e8c039fd775c

Do we have a dos2unix issue on the checksum calculation ?

typo, I meant to write: "I think due to a transfer issue..."

found openssl, same sha1 value on the unix system:
nexus@server:~)/opt/CollabNet_Subversion/bin/openssl sha1 test.ivy.xml
SHA1(test.ivy.xml)= 139b246c832c52a706fe554f9d96e8c039fd775c

@Tamas: Sorry, I have since changed employers and have not worked on the problem. From what I recall I used some kind of work around by deleting the files with the Nexus GUI, rebuilt the cache and metadata, then re-uploaded the ivy.xml. If I have time in the future I may look into debugging the issue, but time is quite tight at the moment.

Just to be clear on this:

Nexus does not create the checksum file during a deploy, this is computed and uploaded by the client. If you think about it, that is the way it has to be, the checksum file is used to verify the artifact wasn't corrupted during upload and storage.

Not sure exactly what is going wrong here, but it's not unknown for there to be client bugs that affect checksum calculation. See here for example:

http://jira.codehaus.org/browse/WAGON-277

Nexus will recompute the checksum if you run a "rebuild metadata" operation. This is not intended for normal use though, it's best thought of as a repair operation.

@Rich: This problem ONLY appears when "rebuild metadata" is called from the GUI on existing artifacts. No deploying is involved. If "rebuild metadata" is accidentally clicked by an admin, all our ivy builds begin crashing and we have to regenerate the .SHA1 files ourselves with a script.

@Tamás: The problem with my uploaded ivy.xml was because I had to use a Windows client to upload it, so the file endings are wrong. Sorry, but I actually have no direct internet access from a Unix-based client here. If you just run the dos2unix util that comes with most distribs, you should see the described results.

The original sha1sum for test.ivy.xml is 7589a00c50f69777931a86ab26784b0bc674040b

If I run dos2unix against the attached test.ivy.xml the sha1 is 139b246c832c52a706fe554f9d96e8c039fd775c

It looks to me like the content-type header is be getting set incorrectly when this file is uploaded, which is causing the line endings to be translated.

@Rich: My file is 'ivy.xml'. 'test.ivy.xml' is the one uploaded by Christian Höfig.

@Tamas: can I reach you by mail ? my mail is c.hoefig@ing-diba.de

Ok, so to recap: I cannot reproduce this. Here is what I did:

I downloaded now both attached files from this issue. First, I did confirm with Hex editor, they do have DOS line endings (verified sequence of 0x0D 0x0A). Then I converted the two files (by creating two new files) to have Unix line endings. Then checksummed all the 4 files using openssl, and got checksums that do match all of those written in all comments. Then I put these files "under" Nexus (into thirdparty repository), and made Nexus to "Recreate (Maven) metadata". And finally, compared what Nexus did with hashes got from openssl. No difference, all are as they should. Here is the transcript:

[cstamas@Marvin NEXUS-3743]$ l
total 32
drwxr-xr-x   6 cstamas  staff   204 Ápr 12 14:07 .
drwxr-xr-x  40 cstamas  staff  1360 Ápr 12 14:01 ..
-rw-r--r--@  1 cstamas  staff   970 Ápr 11 14:17 ivy.xml
-rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:01 test.ivy.xml
[cstamas@Marvin NEXUS-3743]$ openssl sha1 *
SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8
SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b
[cstamas@Marvin NEXUS-3743]$ cat ivy.xml | tr -d '\r' > ivy.xml.unix
[cstamas@Marvin NEXUS-3743]$ cat test.ivy.xml | tr -d '\r' > test.ivy.xml.unix
[cstamas@Marvin NEXUS-3743]$ l
total 32
drwxr-xr-x   6 cstamas  staff   204 Ápr 12 14:07 .
drwxr-xr-x  40 cstamas  staff  1360 Ápr 12 14:01 ..
-rw-r--r--@  1 cstamas  staff   970 Ápr 11 14:17 ivy.xml
-rw-r--r--   1 cstamas  staff   951 Ápr 12 14:06 ivy.xml.unix
-rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:01 test.ivy.xml
-rw-r--r--   1 cstamas  staff  1137 Ápr 12 14:07 test.ivy.xml.unix
[cstamas@Marvin NEXUS-3743]$ openssl sha1 *
SHA1(ivy.xml)= 8cf3988f697168f860918bf48136228ec0d5bce8
SHA1(ivy.xml.unix)= e779e3e74363e546fc22e1550b3655fd6684d4a3
SHA1(test.ivy.xml)= 7589a00c50f69777931a86ab26784b0bc674040b
SHA1(test.ivy.xml.unix)= 139b246c832c52a706fe554f9d96e8c039fd775c
[cstamas@Marvin NEXUS-3743]$ cp * ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/

here I ran the "rebuild metadata" against Thirdparty repository on Nexus UI

[cstamas@Marvin NEXUS-3743]$ l ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/
total 120
drwxr-xr-x  19 cstamas  staff   646 Ápr 12 14:09 .
drwxr-xr-x  10 cstamas  staff   340 Ápr 12 14:08 ..
drwxr-xr-x   3 cstamas  staff   102 Ápr 12 14:09 .meta
drwxr-xr-x   3 cstamas  staff   102 Ápr 12 14:08 .nexus
-rw-r--r--   1 cstamas  staff    25 Ápr 12 14:09 archetype-catalog.xml
-rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 archetype-catalog.xml.md5
-rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 archetype-catalog.xml.sha1
-rw-r--r--@  1 cstamas  staff   970 Ápr 12 14:09 ivy.xml
-rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 ivy.xml.md5
-rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 ivy.xml.sha1
-rw-r--r--   1 cstamas  staff   951 Ápr 12 14:09 ivy.xml.unix
-rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 ivy.xml.unix.md5
-rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 ivy.xml.unix.sha1
-rw-r--r--@  1 cstamas  staff  1157 Ápr 12 14:09 test.ivy.xml
-rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 test.ivy.xml.md5
-rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 test.ivy.xml.sha1
-rw-r--r--   1 cstamas  staff  1137 Ápr 12 14:09 test.ivy.xml.unix
-rw-r--r--   1 cstamas  staff    32 Ápr 12 14:09 test.ivy.xml.unix.md5
-rw-r--r--   1 cstamas  staff    40 Ápr 12 14:09 test.ivy.xml.unix.sha1
[cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.sha1 
8cf3988f697168f860918bf48136228ec0d5bce8
[cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/ivy.xml.unix.sha1 
e779e3e74363e546fc22e1550b3655fd6684d4a3
[cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.sha1 
7589a00c50f69777931a86ab26784b0bc674040b
[cstamas@Marvin NEXUS-3743]$ cat ~/tmp/nexus/nexus-oss-webapp/master/sonatype-work/nexus/storage/thirdparty/test.ivy.xml.unix.sha1 
139b246c832c52a706fe554f9d96e8c039fd775c
[cstamas@Marvin NEXUS-3743]$

@Christian "Do we have a dos2unix issue on the checksum calculation ?"

Naturally, no, we do not parse XML or treat the file content in any other way than just "stream of bytes". Hence, we do not care about line endings either

I am still at standpoint that "something else", other than Nexus either modifies the checksummed file (the ivy.xml), or the hashes, unsure. I simply cannot make my Nexus (on Mac OSX) do the same.

Christian and Todd are on Solaris. You Randy? I do not have access to Solaris OS, but is it possible it has to do something with OS and/or env settings? Nothing else I can think of... we use JRE's DigestInputStream and MessageDigest to perform the hash calculations, and the fact they both are bad (Randy says SHA1 and MD5 are wrong) might me assume there is some problem on actual input?

Our default LANG setting is de.ISO8859-15

Could it have something to do with an encoding config conflict somewhere? I looked over the Nexus sources and could see no special hash creation code, just streams opening and passing to the Java libs.

The ivy.xml files are generated in place in the repo from existing POMs, not loaded through a deployment of any kind if that makes a difference.

If you need specific environment info let us know.

Wait a moment... you gave me an idea: Nexus stores file attributes (simply, a "map" of key-vals). By definition, all files deployed (uploaded over HTTP) are being checksummed during transport, and the two checksum SHA1/MD5 are stored into attributes.

When nexus "rebuilds" hashes, it only does read-calc-checksum as last resort, "worst scenario", that is needed usually for files that just "pops up" out of the blue on the disk, not deployed "normal way" (over HTTP), hence, have no attributes either.

This is same thing I did above: I did a cp of the files in their place "under" Nexus, and Nexus just "got a knowledge about them" when I "touch" them (by any means, I did it by "Recreate Metadata" task that crawls repository, but same would happen if I'd ask the file over HTTP from Nexus). The file would be present on disk, but for Nexus, they came "out of blue", and on the fly re-checksums them, this was the "worse case" scenario.

But, if the file in question has attributes, it would reach for them instead.

So, please Todd, do a cat of the attribute file of the ivy.xml file in question, it has same path and name as "original" file but is a small XML file actually:

sonatype-work/nexus/proxy/attributes/....repo path of the ivy.xml in question

And look for keys "digest.md5" and "digest.sha1" in it. I have a beer to bet on values you find...

And the beer goes to...

You're right! It's pulling the bad values from the attributes. But how did the bad attributes get there in the first place? Is the hash being created by Nexus before the file is finished being written on disk?

Well, that in short means: the file was once "discovered" by Nexus, and back then the file did have those checksums. But later, that file content did change. Both events might happened long in past, only relation between them is that T-appeared < T-changed.

Now, this "protection" I mentioned, the "auto attribute creation" – to create attributes for newly discovered files in storage – was actually a circumvention. Nexus storage folders should not be modified by anything else than Nexus itself. This circumvention was to ease "transitioning" for people still doing "ol'school" SCP deploys, and introducing Nexus gradually, and targeting complete switch to HTTP transport.

This protection does it's job when file appears. But it does not when file is changed by process other than Nexus.

You decide, is this a bug or not, and create an issue if you think it is. But, my reasoning is that nothing under sonatype-work/nexus/storage/... folders should be changed by anything else than Nexus. Nexus for it's basic operation (and many other features) does the bookkeeping of these "attributes" of files, but it assumes "he owns the (content) files", and he keeps attribute files in sync.

Good find Tamas! I'm also on the fence if this should be reported as a bug or not. I understand the logic, but to save on future user error it would be nice to at least get some notification from Nexus saying something like "File has been changed externally to Nexus since previously catalogued, checksum values will be out of sync".

P.S. I was running Nexus on a Scientific (Redhat) 6.0 or 6.1 server.

OK, yes that makes sense. When a difference is found (frequently much later) between Maven's transitive dependency resolution and Ivy's, we usually make a change to the ivy file in place so both Ivy and Maven builds for a specific version function similarly. We regenerate the checksums ourselves and all works well until someone clicks "rebuild metadata".

I deleted the attribute file and it regenerated correctly.

Is it safe to delete attribute files at will if we have no custom attributes?

Yes, deleting the attribute along with content (or deleting the attribute along with modifying content) should "fix" this.

"Fix", as in a moment you think about potential concurrency issues... (Nexus might be serving up the exact file you are modifying/deleting).

Not really a problem for us, since if we're having to edit the file our inbox is already full to the brim with messages from angry developers

Thanks very much for the help, Tamás!

I realize this isn't how Nexus is normally used, but I doubt many Nexus users serve both ivy and maven descriptors to their teams either. Despite directly messing with the repository, Nexus is mostly stable.

One thing you should consider: same stands for Nexus as for Maven (in broadest sense): I believe what you have there should be a Nexus plugin (same is said for Maven, Mojo mojo mojo...)

You problem is perfect fit for "capabilities", similar to one that publishes Maven artifacts as OSGi bundles. Not really related, but you can read more about it here:
https://docs.sonatype.org/display/Nexus/Nexus+OSGi+Experimental+Features+-+Bundle+Maker

With "similar", I meant, that this plugin above does similar, as in it exposes OSGi bundles "on top" of a Maven2 repository... just like you do with Ivy on top of Maven2 repo...

Yes, ideally a more integrated solution would be better. Nexus should be controlling all the interactions with the repository. But as usual, it's hard to find the time...

After the reading the long set of comments, this looks like Nexus is performing as designed and that this is acceptable.