E130003: The XML response contains invalid XML - Follow-up

[NOCERA, ANDY]

Folks,

I used dump and load to debug the malformed node revision ID. Here are my steps and what learned. Looks like the revs' file text: entry has a zero instead of size. By just editing the size, verify worked. No other change was required. The question is can we correct this ourselves without a dump and load?

Thanks
Andy


# Notes from debugging E160062: Malformed node revision ID string after dump and load # rm -rf /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin create /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin dump -r0:3 wd2tva02 and /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin load /usr/tmp/xrepox
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.

/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 23.

/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify /usr/tmp/xrepox
* Verifying repository metadata ...
* Verified revision 2.

/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Error verifying revision 1.
svnadmin: E160062: Malformed node revision ID string

Compare dump and loaded repos revs 1-3

db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---

text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917

22c22
< _0.0.t0-4501db85ea8ce59fc8eed19225af3b1387e71cb10 add-dir false false /ccm06_prod_1_20387
---

_0.0.t0-0 add-dir false false /ccm06_prod_1_20387

25c25
< 137 260
---

137 261

db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
diff db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
26c26
< text: 3 74 134 0 a714b2fcbeb1639111a020a133ae1028
---

text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028

30c30
< _0.0.t2-4501db85ea8ce59fc8eed19225af3b1387e71cb12 add-dir false false /ccm06_n1_1_21751
---

_0.0.t2-2 add-dir false false /ccm06_n1_1_21751

33c33
< 221 346
---

221 348

db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
diff db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
22c22
< text: 2 76 92 0 83b62398481e74fcb4405d4739c25d5f
---

text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f

26c26
< _0.0.t1-4501db85ea8ce59fc8eed19225af3b1387e71cb11 add-dir false false /ccm06_prod_2_20387
---

_0.0.t1-1 add-dir false false /ccm06_prod_2_20387

29c29
< 181 305
---

181 306

IF I CHANGE - wd2tva02/db/revs/0/1 to include 48 48 like the loaded repo verify passes
'text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917'

The Problem moves to the next rev, which I can also edit

$ /opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Error verifying revision 2.
svnadmin: E160062: Malformed node revision ID string


Looks like size field is zero for this repo grep text wd2tva02/db/revs/0/*
wd2tva02/db/revs/0/0:text: 0 0 4 4 2d2977d1c96f487abe4a1e202dd03b4e
wd2tva02/db/revs/0/1:text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
wd2tva02/db/revs/0/10:text: 10 77 432 0 65cdd97c3c194413c12ddc41f5fd7437
wd2tva02/db/revs/0/11:text: 11 76 476 0 29907530feba5dbcb9105e61f29f8a2d
wd2tva02/db/revs/0/12:text: 12 78 522 0 ccd453b94ffecc6bac34afc1e62b977f
wd2tva02/db/revs/0/13:text: 13 78 568 0 97965d3ec3d1d91bbf6f6bdf9c176570
wd2tva02/db/revs/0/14:text: 14 76 612 0 ff2281abf3bea67c63df9c83199b9288
wd2tva02/db/revs/0/15:text: 15 76 656 0 01f577ad55e77d72d9833a0e6a864093
wd2tva02/db/revs/0/16:text: 16 78 702 0 1a4a8868f7a819c3a5a0b049f158d2a3
wd2tva02/db/revs/0/17:text: 17 78 748 0 3b8c4d8e3c95415a5098c12750547c22
wd2tva02/db/revs/0/18:text: 18 76 792 0 88533d0f4d89abb917a54aa40c67379c
wd2tva02/db/revs/0/19:text: 19 76 836 0 90e8b5563d2f92ccf804b6b57d42a962
wd2tva02/db/revs/0/2:text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
wd2tva02/db/revs/0/20:text: 20 78 882 0 4730fe1d4df3ca5f4e9d2be2d6b48273
wd2tva02/db/revs/0/21:text: 21 78 928 0 5a39be128a193e69edd603e5464029c0
wd2tva02/db/revs/0/22:text: 22 75 971 0 23986c9522e8ce73fa93e7d6e4070d5b
wd2tva02/db/revs/0/23:text: 23 75 1014 0 6b6b656e3a76f227bf0806162da70859
wd2tva02/db/revs/0/3:text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
wd2tva02/db/revs/0/4:text: 4 74 176 176 aa41a08702fc9281d222519cb8a7b01e
wd2tva02/db/revs/0/5:text: 5 75 219 0 d134abf460f591ffc0c4d996b376fdd9
wd2tva02/db/revs/0/6:text: 6 75 262 0 cf9070ac92de06910dbb38427ef61535
wd2tva02/db/revs/0/7:text: 7 73 303 0 9b9d5dd52df70c31bd9930d826dce5f6
wd2tva02/db/revs/0/8:text: 8 73 344 0 ce7870a25dc9904d281a056820f3e118
wd2tva02/db/revs/0/9:text: 9 75 387 0 dec2dc2a8c2438c86a6a087d7fb03b82

After making several edits

/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Verified revision 2.
* Verified revision 3.
* Verified revision 4.
* Error verifying revision 5.
svnadmin: E160062: Malformed node revision ID string


#
# Notes from debugging E160062: Malformed node revision ID string after dump and load # rm -rf /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin create /usr/tmp/xrepox /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin dump -r0:3 wd2tva02 and /opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin load /usr/tmp/xrepox
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.

/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 23.

/opt/app/scm/svn/binaries/svn_1.8/bin/svnadmin verify /usr/tmp/xrepox
* Verifying repository metadata ...
* Verified revision 2.

/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Error verifying revision 1.
svnadmin: E160062: Malformed node revision ID string

Compare dump and loaded repos revs 1-3

db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---

text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917

22c22
< _0.0.t0-4501db85ea8ce59fc8eed19225af3b1387e71cb10 add-dir false false /ccm06_prod_1_20387
---

_0.0.t0-0 add-dir false false /ccm06_prod_1_20387

25c25
< 137 260
---

137 261

db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
diff db/revs/0/3 /usr/tmp/xrepox/db/revs/0/3
26c26
< text: 3 74 134 0 a714b2fcbeb1639111a020a133ae1028
---

text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028

30c30
< _0.0.t2-4501db85ea8ce59fc8eed19225af3b1387e71cb12 add-dir false false /ccm06_n1_1_21751
---

_0.0.t2-2 add-dir false false /ccm06_n1_1_21751

33c33
< 221 346
---

221 348

db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
diff db/revs/0/2 /usr/tmp/xrepox/db/revs/0/2
22c22
< text: 2 76 92 0 83b62398481e74fcb4405d4739c25d5f
---

text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f

26c26
< _0.0.t1-4501db85ea8ce59fc8eed19225af3b1387e71cb11 add-dir false false /ccm06_prod_2_20387
---

_0.0.t1-1 add-dir false false /ccm06_prod_2_20387

29c29
< 181 305
---

181 306

IF I CHANGE - wd2tva02/db/revs/0/1 to include 48 48 like the loaded repo verify passes
'text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917'

The Problem moves to the next rev, which I can also edit

$ /opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Error verifying revision 2.
svnadmin: E160062: Malformed node revision ID string


Looks like size field is zero for this repo grep text wd2tva02/db/revs/0/*
wd2tva02/db/revs/0/0:text: 0 0 4 4 2d2977d1c96f487abe4a1e202dd03b4e
wd2tva02/db/revs/0/1:text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917
wd2tva02/db/revs/0/10:text: 10 77 432 0 65cdd97c3c194413c12ddc41f5fd7437
wd2tva02/db/revs/0/11:text: 11 76 476 0 29907530feba5dbcb9105e61f29f8a2d
wd2tva02/db/revs/0/12:text: 12 78 522 0 ccd453b94ffecc6bac34afc1e62b977f
wd2tva02/db/revs/0/13:text: 13 78 568 0 97965d3ec3d1d91bbf6f6bdf9c176570
wd2tva02/db/revs/0/14:text: 14 76 612 0 ff2281abf3bea67c63df9c83199b9288
wd2tva02/db/revs/0/15:text: 15 76 656 0 01f577ad55e77d72d9833a0e6a864093
wd2tva02/db/revs/0/16:text: 16 78 702 0 1a4a8868f7a819c3a5a0b049f158d2a3
wd2tva02/db/revs/0/17:text: 17 78 748 0 3b8c4d8e3c95415a5098c12750547c22
wd2tva02/db/revs/0/18:text: 18 76 792 0 88533d0f4d89abb917a54aa40c67379c
wd2tva02/db/revs/0/19:text: 19 76 836 0 90e8b5563d2f92ccf804b6b57d42a962
wd2tva02/db/revs/0/2:text: 2 76 92 92 83b62398481e74fcb4405d4739c25d5f
wd2tva02/db/revs/0/20:text: 20 78 882 0 4730fe1d4df3ca5f4e9d2be2d6b48273
wd2tva02/db/revs/0/21:text: 21 78 928 0 5a39be128a193e69edd603e5464029c0
wd2tva02/db/revs/0/22:text: 22 75 971 0 23986c9522e8ce73fa93e7d6e4070d5b
wd2tva02/db/revs/0/23:text: 23 75 1014 0 6b6b656e3a76f227bf0806162da70859
wd2tva02/db/revs/0/3:text: 3 74 134 134 a714b2fcbeb1639111a020a133ae1028
wd2tva02/db/revs/0/4:text: 4 74 176 176 aa41a08702fc9281d222519cb8a7b01e
wd2tva02/db/revs/0/5:text: 5 75 219 0 d134abf460f591ffc0c4d996b376fdd9
wd2tva02/db/revs/0/6:text: 6 75 262 0 cf9070ac92de06910dbb38427ef61535
wd2tva02/db/revs/0/7:text: 7 73 303 0 9b9d5dd52df70c31bd9930d826dce5f6
wd2tva02/db/revs/0/8:text: 8 73 344 0 ce7870a25dc9904d281a056820f3e118
wd2tva02/db/revs/0/9:text: 9 75 387 0 dec2dc2a8c2438c86a6a087d7fb03b82

After making several edits

/opt/app/scm/svn/binaries/svn_1.9.7/bin/svnadmin verify wd2tva02
* Verified revision 0.
* Verified revision 1.
* Verified revision 2.
* Verified revision 3.
* Verified revision 4.
* Error verifying revision 5.
svnadmin: E160062: Malformed node revision ID string


Can we edit the revs 'text:" entry instead of a dump a load?
-----Original Message-----
From: Johan Corveleyn [mailto:***@gmail.com]
Sent: Wednesday, March 07, 2018 6:27 AM
To: NOCERA, ANDY <***@att.com>
Cc: ***@subversion.apache.org
Subject: Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

John,
Good feedback.
Svnadmin 1.9 verify is showing the error also. I did a dump and load and it resolved that repo. Having reviewed my repos, it looks like around 30% have this issue. Not sure what we could have done wrong to cause it. A real simple repo is mine and has only a few commits shows the error. Is there a correction method quicker than a dump and load?

Nice, great that dump+load fixes it. I don't think there is a quicker method.

It might be worth investigating why this happened to begin with. But I don't really know where to start. One hypothesis is that this corruption is already lingering there for years (until 1.9 it wouldn't have been noticed) ... perhaps something outside of SVN manipulated the rev files years ago? Or perhaps there was a bug once in SVN that caused this to happen (but the corruption remained benign / unnoticed, until the stricter validation by 1.9). It's also possible that the stricter validation by 1.9 contains a bug, and is too strict for some cases (though in that case I would have expected more reports on this list).

Maybe you can make a more accurate hypothesis by investigating exactly what the "Malformed node revision ID string"s looks like. Actually, danielsh just improved that error message a few days ago, by adding the actual data to the error message:
https://urldefense.proofpoint.com/v2/url?u=http-3A__svn.apache.org_viewvc-3Fview-3Drevision-26revision-3D1825846&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lHImfy5Z3mUV3q_wXtPXpg&m=H1vU79y-LZUNg2o-MTOUnApkEdSPmq6uvVWkpF1n-ZM&s=VcsP7Qwl3TAy_zSkzRZaG5vEBgC4alj__qhGeBQr_54&e=

So if you can build svn from source you might be able to perform a build from the latest svn trunk, and run 'svnadmin verify' to get the more verbose error message (be careful not to use your trunk svn build on production data without creating a backup of course). Or alternatively you can try taking a look into the rev files yourself, to find the "malformed node

[Daniel Shahaf]

Post by NOCERA, ANDY
Folks,
I used dump and load to debug the malformed node revision ID. Here are
my steps and what learned. Looks like the revs' file text: entry has
a zero instead of size.

To be clear, you mean the fourth field. On line 562 'size' is the name
of the fourth field but on line 119, and in the representation_t struct,
'size' is the name of the third field.[1]

This is a recipe for confusion. Normally I think of the fields using
the size/expanded-size terminology so I propose to change 'structure' to
use that terminology in lieu of length/size (and leave a note behind
pointing out that old versions of the document used <size> differently).

WDYT?

Daniel

[1] All line numbers refer to subversion/libsvn_fs_fs/structure.

Post by NOCERA, ANDY
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---

text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917

22c22

[Philip Martin]

Post by NOCERA, ANDY
I used dump and load to debug the malformed node revision ID. Here
are my steps and what learned. Looks like the revs' file text: entry
has a zero instead of size. By just editing the size, verify worked.
No other change was required. The question is can we correct this
ourselves without a dump and load?
db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---

text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917

22c22

That looks like issue 4554

https://issues.apache.org/jira/projects/SVN/issues/SVN-4554

Editing the file is unlikely to work. Later revisons refer to data in
earlier revisions via a byte offset into the earlier file. When you
edit "0" to "48" you have changed the byte offset of all the data beyond
the edit and that breaks the references in all the later files. You
would need to identify all the affected offsets in later files and
modify them, which in turn may lead to more offset changes. You may
need to recompute some checksums as well.

[Daniel Shahaf]

Post by Philip Martin

Post by NOCERA, ANDY
I used dump and load to debug the malformed node revision ID. Here
are my steps and what learned. Looks like the revs' file text: entry
has a zero instead of size. By just editing the size, verify worked.
No other change was required. The question is can we correct this
ourselves without a dump and load?
db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
diff db/revs/0/1 /usr/tmp/xrepox/db/revs/0/1
18c18
< text: 1 76 48 0 ec69809945c46f2bb74e99a3ff7cd917
---

text: 1 76 48 48 ec69809945c46f2bb74e99a3ff7cd917

22c22

That looks like issue 4554
https://issues.apache.org/jira/projects/SVN/issues/SVN-4554
Editing the file is unlikely to work. Later revisons refer to data in
earlier revisions via a byte offset into the earlier file. When you
edit "0" to "48" you have changed the byte offset of all the data beyond
the edit and that breaks the references in all the later files.

Changing "0" to "48" would also have broken the <root-offset> and
<cp-offset> offsets in that revision file, so how come 'verify' worked after
that change?

Cheers,

Daniel
(The <foo-offset> terms are from the 'structure' file and describe
format 6; format 7 has <l2p offset> and <p2l offset> which are
analogous)

[Philip Martin]

Post by Daniel Shahaf
Changing "0" to "48" would also have broken the <root-offset> and
<cp-offset> offsets in that revision file, so how come 'verify' worked after
that change?

In the examples he gave it looks like the root node itself is being
edited. That works because the change is after <root-offset> but before
<cp-offset> and he shows <cp-offset> changing as well.

I guess you can get away with editing the last node in the file provided
you also change <cp-offset> in the same file.

[Johan Corveleyn]

Post by Philip Martin

Post by Daniel Shahaf
Changing "0" to "48" would also have broken the <root-offset> and
<cp-offset> offsets in that revision file, so how come 'verify' worked after
that change?

In the examples he gave it looks like the root node itself is being
edited. That works because the change is after <root-offset> but before
<cp-offset> and he shows <cp-offset> changing as well.
I guess you can get away with editing the last node in the file provided
you also change <cp-offset> in the same file.

Does that also explain why the OP could repair some repositories by
simply dumping and loading them? Or would dump+load also work for
corruption-instances that are not on the root node?

From the perspective of the recoveries done by the OP, this doesn't
seem like a "breaking corruption", since it can be recovered from
quite easily.

If that is not the case, and some unrecoverable instances remain (that
cannot be dumped+loaded), can we offer any other suggestions to
recover?