How to fix corrupt revision in repo?

[David Hopkins]

Greetings,

I have an SVN repo that is failing svnadmin verify on revision 192.

For some reason the verify output says:

[...various successful revisions...]
* Verified revision 191.
svnadmin: Can't read file 'E:\Repositories\Client_Name\db\revs\0\16
2': End of file found

I'm not sure why the verification command for revision 192 would throw
an error description for revision 162. Revision 192 affected a
completely different part of the repository to revision 162, so there is

no obvious relationship between them.

All the revisions from 193 to 332 (HEAD) are ok. It might be a one-off.

This looks like a FSFS-backed repository (I am very new to SVN and
inherited the server from someone else!). The server is VisualSVN 2.1.4,

which is based on SVN 1.6.13.

The clients are mostly TortoiseSVN 1.6.16, which uses SVN 1.6.17.

What steps should I take to fix the corrupted revision? Is there more
information that I should provide? (eg a copy of the rev 192 file?) This

problem is causing checkouts and updates to fail for files that were
last modified in that revision.

Regards,

David Hopkins
Serck Controls


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[Thorsten Schöning]

Guten Tag David Hopkins,

Post by David Hopkins
What steps should I take to fix the corrupted revision?

If you have a backup, replace the file wih the backed up one. If you
don't, really really bad and it's the next you should take care of. In
the worst case you have no other chance but to hope that one of you're
working copies has the most current version of every data, create a
new repository with the old, working revisions dumped into and make a
very large commit to get the most current source as a new version.

Mit freundlichen Grüßen,

Thorsten Schöning
--
Thorsten Schöning
AM-SoFT IT-Systeme - Hameln | Potsdam | Leipzig

Telefon: Potsdam: 0331-743881-0
E-Mail: ***@am-soft.de
Web: http://www.am-soft.de

AM-SoFT GmbH IT-Systeme, Konsumhof 1-5, 14482 Potsdam
Amtsgericht Potsdam HRB 21278 P, Geschäftsführer: Andreas Muchow

[Daniel Shahaf]

Post by David Hopkins
Greetings,
I have an SVN repo that is failing svnadmin verify on revision 192.
[...various successful revisions...]
* Verified revision 191.
svnadmin: Can't read file 'E:\Repositories\Client_Name\db\revs\0\16
2': End of file found
I'm not sure why the verification command for revision 192 would throw
an error description for revision 162. Revision 192 affected a
completely different part of the repository to revision 162, so there is
no obvious relationship between them.

Possibly due to rep-sharing? Does db/revs/0/192 contain the number
"162" in ASCII decimal delimited by whitespace? You can check that with
the following command: grep -a "^text:" db/revs/0/192

Does 'svnadmin dump -r 162 >NUL' work ?

To answer your question: yes, most definitely a copy of the r192 (and/or
r162) rev file would allow to pinpoint the problem, however you might
not want to share those files on a public list as they may contain
sensitive data (versioned file contents).

Post by David Hopkins
All the revisions from 193 to 332 (HEAD) are ok. It might be a one-off.
This looks like a FSFS-backed repository (I am very new to SVN and
inherited the server from someone else!). The server is VisualSVN 2.1.4,
which is based on SVN 1.6.13.
The clients are mostly TortoiseSVN 1.6.16, which uses SVN 1.6.17.
What steps should I take to fix the corrupted revision? Is there more
information that I should provide? (eg a copy of the rev 192 file?) This
problem is causing checkouts and updates to fail for files that were
last modified in that revision.
Regards,
David Hopkins
Serck Controls
===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[David Hopkins]

Thanks Daniel. Responses inline.

David Hopkins wrote on Thu, Sep 15, 2011 at 10:30:46 +0800: I'm not

sure

why the verification command for revision 192 would throw an error
description for revision 162. Revision 192 affected a completely
different part of the repository to revision 162, so there is no

obvious

relationship between them.

Possibly due to rep-sharing? Does db/revs/0/192 contain the number

"162"

in ASCII decimal delimited by whitespace? You can check that with the
following command: grep -a "^text:" db/revs/0/192

Yes it does.

Here's the context in the rev file:
id: y-31.0-32.r192/673830
type: file
pred: y-31.0.r31/264323
count: 1
text: 162 670867 6111 52486 5117fb0964ca1a78dd97447d23452e73
609f4745460d6e14860daff0803ee7024c54898c 191-5r/_m
cpath: [redacted]
copyroot: 32 [redacted]

Looking at the other nearby entries, they have "text: 192 [...]"
instead of "text: 162 [...]". Is that likely to be the problem?

Does 'svnadmin dump -r 162 >NUL' work ?

Yes it does.

To answer your question: yes, most definitely a copy of the r192 (and/or
r162) rev file would allow to pinpoint the problem, however you might
not want to share those files on a public list as they may contain
sensitive data (versioned file contents).

I'll find out if I can release the broken revisions in their entirety.

The corrupted revision doesn't actually contain anything particularly
important (almost all the modified files in it have since been replaced
by newer versions anyway). Can I fix the repository by dumping every
revision except 192, and then reloading the good revisions into a new
repo? Or will cause problems for the revisions after 192 since one of
the revisions no longer exists?

Regards,

David Hopkins
Serck Controls


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[Daniel Shahaf]

Post by David Hopkins
Thanks Daniel. Responses inline.

David Hopkins wrote on Thu, Sep 15, 2011 at 10:30:46 +0800: I'm not

sure

why the verification command for revision 192 would throw an error
description for revision 162. Revision 192 affected a completely
different part of the repository to revision 162, so there is no

obvious

relationship between them.

Possibly due to rep-sharing? Does db/revs/0/192 contain the number

"162"

in ASCII decimal delimited by whitespace? You can check that with the
following command: grep -a "^text:" db/revs/0/192

Yes it does.
id: y-31.0-32.r192/673830
type: file
pred: y-31.0.r31/264323
count: 1
text: 162 670867 6111 52486 5117fb0964ca1a78dd97447d23452e73
609f4745460d6e14860daff0803ee7024c54898c 191-5r/_m

That tells you that the 6111 bytes starting at offset 670867(bytes) into
the r162 rev file are a representation generating a file whose checksums
and uniquifier are given later. See subversion/libsvn_fs_fs/structure
for details --- basically, it's DELTA\n or PLAIN\n up through ENDREP\n.

Post by David Hopkins
cpath: [redacted]
copyroot: 32 [redacted]
Looking at the other nearby entries, they have "text: 192 [...]"
instead of "text: 162 [...]". Is that likely to be the problem?

It's normal for r192 to contain "text: 162" if rep-sharing is enabled or
if you did a copy-without-textmods from r162.

Post by David Hopkins

Does 'svnadmin dump -r 162 >NUL' work ?

Yes it does.

Hmmm.

Post by David Hopkins

To answer your question: yes, most definitely a copy of the r192

(and/or

r162) rev file would allow to pinpoint the problem, however you might
not want to share those files on a public list as they may contain
sensitive data (versioned file contents).

I'll find out if I can release the broken revisions in their entirety.

Perhaps someone would be willing to have a look at those two revision
files privately.

(In fact, I might be able to do this too. But I'm reluctant to make
a promise or commitment about this.)

Post by David Hopkins
The corrupted revision doesn't actually contain anything particularly
important (almost all the modified files in it have since been replaced
by newer versions anyway). Can I fix the repository by dumping every
revision except 192, and then reloading the good revisions into a new
repo? Or will cause problems for the revisions after 192 since one of
the revisions no longer exists?

That won't work if files after r192 are stored as deltas against the
fulltext of r192.

Post by David Hopkins
Regards,
David Hopkins
Serck Controls
===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[David Hopkins]

Post by Daniel Shahaf

Post by David Hopkins
id: y-31.0-32.r192/673830
type: file
pred: y-31.0.r31/264323
count: 1
text: 162 670867 6111 52486 5117fb0964ca1a78dd97447d23452e73
609f4745460d6e14860daff0803ee7024c54898c 191-5r/_m

That tells you that the 6111 bytes starting at offset 670867(bytes) into
the r162 rev file are a representation generating a file whose

checksums

Post by Daniel Shahaf
and uniquifier are given later. See subversion/libsvn_fs_fs/structure
for details --- basically, it's DELTA\n or PLAIN\n up through

ENDREP\n.

That's interesting. It certainly explains the "end of file" error
message
that is getting thrown, because rev 162 is only 1,506 bytes long.
Rev 162 was a deletion of a single file from a different folder in the
repo so I'd be surprised if it contained any file representations at
all.
Rev 192 is 683,471 bytes long, so it *is* long enough for a 670867 byte
offset to make sense.

Post by Daniel Shahaf

Post by David Hopkins
Looking at the other nearby entries, they have "text: 192 [...]"
instead of "text: 162 [...]". Is that likely to be the problem?

It's normal for r192 to contain "text: 162" if rep-sharing is enabled or
if you did a copy-without-textmods from r162.

Ok. I think rep-sharing is probably enabled because this server was
installed
using SVN 1.6, and we haven't altered the setting. (It's on by default,
yes?)

But, I can see from the CPATH which file in r192 is referencing r162
(EDGE.CSV), and that reference doesn't make sense.

The history of EDGE.CSV is as follows:
R31: EDGE.CSV added to repo

R32: one of the directories in EDGE.CSV's parent path was renamed

(R162: a single file in a completely different part of the repo was
deleted.
Literally the only part of their file path in common was the repo root
folder. EDGE.CSV and the deleted file have no shared history,
relationships,
or even data in common - one is a CSV file and the deleted file was a
binary
archive!)

R192: EDGE.CSV was modified, along with several other files in the same
folder.
I've now checked, and every single other text: field in R192 references
R192.
There are no other revisions referenced.

R335: EDGE.CSV was deleted. This is because that file wasn't very
important,
and all the other files which changed in r192 were updated in later
revisions
and apparently can be successfully checked out/updated.

Post by Daniel Shahaf

Post by David Hopkins

Post by Daniel Shahaf
To answer your question: yes, most definitely a copy of the r192

(and/or

Post by Daniel Shahaf
r162) rev file would allow to pinpoint the problem, however you might
not want to share those files on a public list as they may contain
sensitive data (versioned file contents).

I'll find out if I can release the broken revisions in their

entirety.

Post by Daniel Shahaf
Perhaps someone would be willing to have a look at those two revision
files privately.
(In fact, I might be able to do this too. But I'm reluctant to make
a promise or commitment about this.)

Post by David Hopkins
The corrupted revision doesn't actually contain anything

particularly

Post by Daniel Shahaf

Post by David Hopkins
important (almost all the modified files in it have since been replaced
by newer versions anyway). Can I fix the repository by dumping every
revision except 192, and then reloading the good revisions into a new
repo? Or will cause problems for the revisions after 192 since one of
the revisions no longer exists?

That won't work if files after r192 are stored as deltas against the
fulltext of r192.

Hmm, ok.

I'm thinking about making a copy of the repository folder, and seeing
what
happens if I replace "text: 162" with "text: 192" in revs\0\192, since
the
offsets appear to pass the "smell test" for file size. Is there _any_
chance
that that will work? Or are there other references I would also need to
patch
inside the revs\0\192 file?

I thought I'd try doing an svndump and then use svndumpfilter to exclude

EDGE.CSV, since it seems to be the only thing with an invalid rev
reference,
but the svnadmin dump operation fails when it gets to r192, since it
can't
process the reference to r162 either.

Regards,

David Hopkins
Serck Controls


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[Daniel Shahaf]

Quick reply, more verbose one might follow up later.

Your reply breaks the nested quoting levels, please try to avoid it, are
you sending mail as text/plain?

(more below)

Post by David Hopkins

Post by Daniel Shahaf
It's normal for r192 to contain "text: 162" if rep-sharing is enabled or
if you did a copy-without-textmods from r162.

Ok. I think rep-sharing is probably enabled because this server was
installed
using SVN 1.6, and we haven't altered the setting. (It's on by default,
yes?)

Yes. See $REPOS/db/rep-cache.db

Post by David Hopkins
But, I can see from the CPATH which file in r192 is referencing r162
(EDGE.CSV), and that reference doesn't make sense.

'cpath' is created path. It doesn't change in copies.

Post by David Hopkins

Post by Daniel Shahaf

Post by David Hopkins

Post by Daniel Shahaf
To answer your question: yes, most definitely a copy of the r192
(and/or r162) rev file would allow to pinpoint the problem,
however you might not want to share those files on a public list
as they may contain sensitive data (versioned file contents).

I'll find out if I can release the broken revisions in their

entirety.

Post by Daniel Shahaf
Perhaps someone would be willing to have a look at those two revision
files privately.
(In fact, I might be able to do this too. But I'm reluctant to make
a promise or commitment about this.)

Post by David Hopkins
The corrupted revision doesn't actually contain anything
particularly important (almost all the modified files in it have
since been replaced by newer versions anyway). Can I fix the
repository by dumping every revision except 192, and then
reloading the good revisions into a new repo? Or will cause
problems for the revisions after 192 since one of the revisions no
longer exists?

That won't work if files after r192 are stored as deltas against the
fulltext of r192.

Hmm, ok.
I'm thinking about making a copy of the repository folder, and seeing
what happens if I replace "text: 162" with "text: 192" in revs\0\192,
since the offsets appear to pass the "smell test" for file size. Is
there _any_ chance that that will work?

I don't think I've heard of precedents of this bug, but sure, try it.
If you want to test before doing it, use 'xxd -l 5 -s 670867 r162-rev-file',
it should say either "DELTA" or "PLAIN" (with no trailing end of line).

Also, standard practice, check your disk for hardware errors. (should
have said that earlier, sorry)

Post by David Hopkins
Or are there other references
I would also need to patch inside the revs\0\192 file?

If there are they would all appear as ASCII '162'. Revision numbers are
referred to in the text: node-revision header and sometimes in the header
of DELTA-style reps. See
https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure

[Daniel Shahaf]

One more thing. The fact that in r162 one file was deleted *and no
files were added or changed* implies that the only new representations
in r162 would be directory representations --- it wouldn't add any
*file* representations --- so the reference to r162 in the node-rev
header (the sequence of ASCII lines of which the "text:" line is part)
is almost certainly bogus.

I'm curious to hear whether the problem was indeed that the noderev
referred to r162 instead of r192.

Post by Daniel Shahaf
Quick reply, more verbose one might follow up later.
Your reply breaks the nested quoting levels, please try to avoid it, are
you sending mail as text/plain?
(more below)

Post by David Hopkins

Post by Daniel Shahaf
It's normal for r192 to contain "text: 162" if rep-sharing is enabled or
if you did a copy-without-textmods from r162.

Ok. I think rep-sharing is probably enabled because this server was
installed
using SVN 1.6, and we haven't altered the setting. (It's on by default,
yes?)

Yes. See $REPOS/db/rep-cache.db

Post by David Hopkins
But, I can see from the CPATH which file in r192 is referencing r162
(EDGE.CSV), and that reference doesn't make sense.

'cpath' is created path. It doesn't change in copies.

Post by David Hopkins

Post by Daniel Shahaf

Post by David Hopkins

Post by Daniel Shahaf
To answer your question: yes, most definitely a copy of the r192
(and/or r162) rev file would allow to pinpoint the problem,
however you might not want to share those files on a public list
as they may contain sensitive data (versioned file contents).

I'll find out if I can release the broken revisions in their

entirety.

Post by Daniel Shahaf
Perhaps someone would be willing to have a look at those two revision
files privately.
(In fact, I might be able to do this too. But I'm reluctant to make
a promise or commitment about this.)

Post by David Hopkins
The corrupted revision doesn't actually contain anything
particularly important (almost all the modified files in it have
since been replaced by newer versions anyway). Can I fix the
repository by dumping every revision except 192, and then
reloading the good revisions into a new repo? Or will cause
problems for the revisions after 192 since one of the revisions no
longer exists?

That won't work if files after r192 are stored as deltas against the
fulltext of r192.

Hmm, ok.
I'm thinking about making a copy of the repository folder, and seeing
what happens if I replace "text: 162" with "text: 192" in revs\0\192,
since the offsets appear to pass the "smell test" for file size. Is
there _any_ chance that that will work?

I don't think I've heard of precedents of this bug, but sure, try it.
If you want to test before doing it, use 'xxd -l 5 -s 670867 r162-rev-file',
it should say either "DELTA" or "PLAIN" (with no trailing end of line).
Also, standard practice, check your disk for hardware errors. (should
have said that earlier, sorry)

Post by David Hopkins
Or are there other references
I would also need to patch inside the revs\0\192 file?

If there are they would all appear as ASCII '162'. Revision numbers are
referred to in the text: node-revision header and sometimes in the header
of DELTA-style reps. See
https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure

[David Hopkins]

Post by Daniel Shahaf
One more thing. The fact that in r162 one file was deleted *and no
files were added or changed* implies that the only new representations
in r162 would be directory representations --- it wouldn't add any
*file* representations --- so the reference to r162 in the node-rev
header (the sequence of ASCII lines of which the "text:" line is part)
is almost certainly bogus.
I'm curious to hear whether the problem was indeed that the noderev
referred to r162 instead of r192.

Sadly, it wasn't. I've now experimented with that. The offset supposedly
within r162 is listed as 670867 bytes, which is well outside the total
length of r162 as we've already discussed. But it isn't a valid pointer
within r192 either; offset 670867 points to the middle of one of the
other rep blocks within the r192 file. I've had a look at the other
node-rev headers and it appears that all the rep blocks in the r192 file
are fully accounted for by the node-revs which have text: 192. (That is,
there are no representations in the r192 file which don't already have a
valid node-rev header).

I've had a look through all the revs between 162 and 192 which are at
least 600 KiB in size. But I can't find *any* rev files in the whole
repository history leading up to 192 where an offset of 670867 points to
the beginning of a DELTA or PLAIN representation.

So, I'm now assuming that both the reference to r162, and the offset of
670867 bytes, are bogus. But there aren't any obvious candidates for a
non-bogus representation of that particular file update.

Given that the file with the bogus node-rev is unimportant, and has
since been deleted from the repo, is there any way to patch the r192
rev-file so that the repository has enough internal consistency to
produce a valid dump file?

At the moment it looks like the "nuclear option" is to check out the
current version of everything and start a new repository with it. This
*should* work because the corrupted file isn't included in recent
revisions, so SVN won't need to de-reference the invalid reference in
r192 when performing the check out. But if I can purge the broken-ness
from the repo and keep the rest of the history, that would obviously be
better. I certainly don't want to keep using a repo that doesn't
validate and can't be dumped, though.

Post by Daniel Shahaf

Post by Daniel Shahaf
Quick reply, more verbose one might follow up later.
Your reply breaks the nested quoting levels, please try to avoid it,

are

Post by Daniel Shahaf
you sending mail as text/plain?

Sorry about breaking the nested quoting. I'm using Outlook which is
pretty mediocre as a plain-text email client. I was already using
text/plain, but Outlook's quoting style wasn't right, so I was trying to
manually fix the text-wrapping and quote marks. Clearly I wasn't getting
it right. I've now found a couple more Outlook settings which will
hopefully address the problem.

Unfortunately, it doesn't look like I'll be able to send you the actual
rev file(s), at least not without a lot of inconvenience that I don't
want to subject you to (ie an NDA, since we don't actually own the IP to
any of the code which may be included in the rev file). Sorry about
that.

Regards,

David Hopkins


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[Thorsten Schöning]

Guten Tag David Hopkins,

Post by David Hopkins
At the moment it looks like the "nuclear option" is to check out the
current version of everything and start a new repository with it.

You can dump the old repository until the last working rev and start
from that with your current working copies, so not loosing all of your
history.

Post by David Hopkins
But if I can purge the broken-ness
from the repo and keep the rest of the history, that would obviously be
better.

My impression on problems like yours, which I had too, is, that in
most cases trying to repair broken rev files was simply not
successful. Learn from it and take the time to design a proper
backup/sync mechanism.

Mit freundlichen Grüßen,

Thorsten Schöning
--
Thorsten Schöning
AM-SoFT IT-Systeme - Hameln | Potsdam | Leipzig

Telefon: Potsdam: 0331-743881-0
E-Mail: ***@am-soft.de
Web: http://www.am-soft.de

AM-SoFT GmbH IT-Systeme, Konsumhof 1-5, 14482 Potsdam
Amtsgericht Potsdam HRB 21278 P, Geschäftsführer: Andreas Muchow

[Nico Kadel-Garcia]

Post by Thorsten Schöning
Guten Tag David Hopkins,

Post by David Hopkins
At the moment it looks like the "nuclear option" is to check out the
current version of everything and start a new repository with it.

You can dump the old repository until the last working rev and start
from that with your current working copies, so not loosing all of your
history.

You *CAN*. You can also do that to a system without broken revisions.
You have to be very careful not to insert a more subtlely corrupted
revision, and you *should* change your uuid and check out new working
copies lest someone with the old revision in an old working copy wind
up severely horking themselves. Simply leaving out the broken
revision, renumbering, and keeping the old uuid will break things
very, very badly. (Did that once editing out a DVD image that someone
added, fortunately it was early in the project before the repository
was considered production grade.

Post by Thorsten Schöning

Post by David Hopkins
But if I can purge the broken-ness
from the repo and keep the rest of the history, that would obviously be
better.

My impression on problems like yours, which I had too, is, that in
most cases trying to repair broken rev files was simply not
successful. Learn from it and take the time to design a proper
backup/sync mechanism.

This is not as easy as it sounds. Any revision in a core repository
that is corrupted online and for which rsync overwrites the backup
server's copy is a real problem, and keeping numerous old copies lying
around on tape or separate disks is a support issue. And "rsync" based
backups have real problems with revisions in the process of being
committed on the primary server. The bulky and lengthy svnadmin "dump"
and "load" processes are resource intensive and awkward to manage, and
they take a *long* time on a big repository, leaving a significant
window of opportunity for corruption.

The revisions are also vulnerable to filesystem operations by the SVN
administrator account, or shared account, on the SVN server. This is
one of the reasons I *loathe* shared write permissions for file://
access, it's vulnerable to mistakes. . A backup system that merely
replicates those without reliable off-line or distinct disk space
storage is vulnerable to all sorts of surprises. My personal favorite
solution is a NetApp storage server with filesystem ".snapshot"
directories, and an off-site backup with svnsync running in case the
NetApp goes toes-up.

[Daniel Shahaf]

Post by David Hopkins

Post by Daniel Shahaf
One more thing. The fact that in r162 one file was deleted *and no
files were added or changed* implies that the only new representations
in r162 would be directory representations --- it wouldn't add any
*file* representations --- so the reference to r162 in the node-rev
header (the sequence of ASCII lines of which the "text:" line is part)
is almost certainly bogus.
I'm curious to hear whether the problem was indeed that the noderev
referred to r162 instead of r192.

Sadly, it wasn't. I've now experimented with that. The offset supposedly
within r162 is listed as 670867 bytes, which is well outside the total
length of r162 as we've already discussed. But it isn't a valid pointer
within r192 either; offset 670867 points to the middle of one of the
other rep blocks within the r192 file. I've had a look at the other
node-rev headers and it appears that all the rep blocks in the r192 file
are fully accounted for by the node-revs which have text: 192. (That is,
there are no representations in the r192 file which don't already have a
valid node-rev header).

So the question is whether the rep is lost or just misplaced.

Post by David Hopkins
I've had a look through all the revs between 162 and 192 which are at
least 600 KiB in size. But I can't find *any* rev files in the whole
repository history leading up to 192 where an offset of 670867 points to
the beginning of a DELTA or PLAIN representation.
So, I'm now assuming that both the reference to r162, and the offset of
670867 bytes, are bogus. But there aren't any obvious candidates for a
non-bogus representation of that particular file update.

Okay. FWIW, there is no easy way to find an orphaned/unreferenced rep
in the revision files. The rightest way would be to parse all node-ids
(which _is_ possible, think 'svn ls -R'), collect all the rep pointers
they contain, then see if there are any additional reps. And even then,
identifying their length would have to be done manually... (reps do not
contain their own length)

Post by David Hopkins
Given that the file with the bogus node-rev is unimportant, and has
since been deleted from the repo, is there any way to patch the r192
rev-file so that the repository has enough internal consistency to
produce a valid dump file?

I suppose you could do it in a number of ways, either by changing what
rep the node-rev refers to or by unlinking the node-rev from its parent.

Be careful about successors of the file --- whether new revision of the
file are expressed as deltas to the node-rev which is now corrupted.
You must know whether that's the case before deciding how to alter that
node-rev.

Also, you must cause no offsets in existing rev files to change (so, add
padding as needed). And probably a few other things that I forget right
now...

Post by David Hopkins
At the moment it looks like the "nuclear option" is to check out the
current version of everything and start a new repository with it. This
*should* work because the corrupted file isn't included in recent
revisions, so SVN won't need to de-reference the invalid reference in
r192 when performing the check out. But if I can purge the broken-ness
from the repo and keep the rest of the history, that would obviously be
better. I certainly don't want to keep using a repo that doesn't
validate and can't be dumped, though.

You ought to be able to keep the rest of the history even without fixing
the brokenness in r192. (as the file is deleted in HEAD, a checkout
should work; and you also have the option of dumping the history while
excluding the problematic file from it (via authz+svnsync/svnrdump or svndumpfilter).)

Post by David Hopkins

Post by Daniel Shahaf

Post by Daniel Shahaf
Quick reply, more verbose one might follow up later.
Your reply breaks the nested quoting levels, please try to avoid it,

are

Post by Daniel Shahaf
you sending mail as text/plain?

Sorry about breaking the nested quoting. I'm using Outlook which is
pretty mediocre as a plain-text email client. I was already using
text/plain, but Outlook's quoting style wasn't right, so I was trying to
manually fix the text-wrapping and quote marks. Clearly I wasn't getting
it right. I've now found a couple more Outlook settings which will
hopefully address the problem.

Thanks, much better now.

Post by David Hopkins
Unfortunately, it doesn't look like I'll be able to send you the actual
rev file(s), at least not without a lot of inconvenience that I don't
want to subject you to (ie an NDA, since we don't actually own the IP to
any of the code which may be included in the rev file). Sorry about
that.

No problems, and good luck.

If you need docs about the FS that 'structure' doesn't have, they would
be under ../libsvn_fs_base/.

Post by David Hopkins
Regards,
David Hopkins

[David Hopkins]

Post by Daniel Shahaf
You ought to be able to keep the rest of the history even without fixing
the brokenness in r192. (as the file is deleted in HEAD, a checkout
should work; and you also have the option of dumping the history while
excluding the problematic file from it (via authz+svnsync/svnrdump or svndumpfilter).)

I'll look into the authz+svnsync/svnrdump option. Svndumpfilter doesn't
work for me because the 'svnadmin dump' operation fails when it tries to
process 192 (before I get a chance to use svndumpfilter to eliminate the
bogus file). As far as I can tell svndumpfilter operates on dumpfiles
that already exist, and can't actually stop svnadmin from trying to
resolve the bogus node-rev header during the dump process. The
authz+svnsync solution will hopefully allow me to effectively do that
filtering at an earlier stage in the pipeline.

Thank you very much for all your help,

David Hopkins
Serck Controls


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[David Hopkins]

Post by Daniel Shahaf

Post by Daniel Shahaf
You ought to be able to keep the rest of the history even without

fixing

Post by Daniel Shahaf
the brokenness in r192. (as the file is deleted in HEAD, a checkout
should work; and you also have the option of dumping the history while
excluding the problematic file from it (via authz+svnsync/svnrdump

or

Post by Daniel Shahaf

Post by Daniel Shahaf
svndumpfilter).)

I'll look into the authz+svnsync/svnrdump option. Svndumpfilter

doesn't

Post by Daniel Shahaf
work for me because the 'svnadmin dump' operation fails when it tries

to

Post by Daniel Shahaf
process 192 (before I get a chance to use svndumpfilter to eliminate

the

Post by Daniel Shahaf
bogus file). As far as I can tell svndumpfilter operates on dumpfiles
that already exist, and can't actually stop svnadmin from trying to
resolve the bogus node-rev header during the dump process. The
authz+svnsync solution will hopefully allow me to effectively do that
filtering at an earlier stage in the pipeline.

For the benefit of anyone else who comes across this message thread in
the future, I thought I'd post a final follow-up message with my
results.

The authz+svnrdump solution *did* work for creating a dumpfile without
references to the corrupted file revision. I ended up setting up a
temporary server where I could set custom authz permissions, and
downloaded a beta SVN 1.7 client so that I could use svnrdump rather
than svnsync (which was much simpler to set up). I've successfully
loaded the purged dumpfile into a new repository which now works with
svnadmin verify, svnadmin dump, svnadmin hotcopy etc.

Thanks once again for all your help (especially the authz+svnrdump
suggestion).

Regards,

David Hopkins
Serck Controls


===== PRIVACY AND CONFIDENTIALITY NOTICE =====
The information contained in this message is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, you must not copy, distribute, take any action in reliance on it, or disclose any details of the message to any person, firm or corporation. If you have received this message in error, please notify the sender immediately by reply e-mail and delete all copies of this transmission together with any attachments.
The views or opinions expressed in this e-mail or any attachment are not necessarily those of Serck Controls Pty Ltd.
NOTE - You should carry out your own virus checks before opening any attachment.

[Neil Bird]

Around about 15/09/11 03:30, David Hopkins typed ...

Post by David Hopkins
I have an SVN repo that is failing svnadmin verify on revision 192.

Slightly OT, if I may: would a 'svnadmin hotcopy' also show up errors
that 'verify' would? We use hotcopy to pull our repos off the SVN server
onto a backup-up network drive, but we don't use verify regularly.

The network drive has incremental backups so if we see an error during
the copy we can still get older, valid, copies of revs. back.

Do we need to be running verify as well?

[Tony Sweeney]

Unlike 'svnadmin dump', hotcopy will happily back up a corrupt revision
and not tell you. It's really just a clever filesystem backup with a
very careful time ordering of certain key files in case there is a
transaction in progress when it runs. Having been bitten by this
myself[*], we now run svnadmin out of cron every night before we run our
hotcopy. We also keep a week's worth of hotcopies, and I check my cron
emails every morning.

Tony.
[*] fsfsverify.py is your friend

-----Original Message-----
From: Neil Bird [mailto:***@jibbyjobby.co.uk]
Sent: 16 September 2011 09:54
To: ***@subversion.apache.org
Subject: Re: How to fix corrupt revision in repo?

Around about 15/09/11 03:30, David Hopkins typed ...

Post by David Hopkins
I have an SVN repo that is failing svnadmin verify on revision 192.

Slightly OT, if I may: would a 'svnadmin hotcopy' also show up
errors that 'verify' would? We use hotcopy to pull our repos off the
SVN server onto a backup-up network drive, but we don't use verify
regularly.

The network drive has incremental backups so if we see an error
during the copy we can still get older, valid, copies of revs. back.

Do we need to be running verify as well?

--
[***@fnx ~]# rm -f .signature
[***@fnx ~]# ls -l .signature
ls: .signature: No such file or directory [***@fnx ~]# exit

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3899 - Release Date: 09/15/11

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________

[Neil Bird]

Around about 16/09/11 11:22, Tony Sweeney typed ...

Post by Tony Sweeney
Unlike 'svnadmin dump', hotcopy will happily back up a corrupt revision
and not tell you. It's really just a clever filesystem backup with a
very careful time ordering of certain key files in case there is a
transaction in progress when it runs. Having been bitten by this
myself[*], we now run svnadmin out of cron every night before we run our
hotcopy. We also keep a week's worth of hotcopies, and I check my cron
emails every morning.
[*] fsfsverify.py is your friend

OK, thanks: I'll update our backup script!

[edg]

Post by Tony Sweeney
Unlike 'svnadmin dump', hotcopy will happily back up a corrupt revision
and not tell you. It's really just a clever filesystem backup with a
very careful time ordering of certain key files in case there is a
transaction in progress when it runs. Having been bitten by this
myself[*], we now run svnadmin out of cron every night before we run our
hotcopy. We also keep a week's worth of hotcopies, and I check my cron
emails every morning.

I have been working under the assumption that the best way to "verify"
is by:

svnadmin dump REPO > NUL ( or to ' > /dev/null' for 'nix)

rather than

svnadmin verify REPO

The reason being that the former also checks revision property files.

See:

http://svn.haxx.se/users/archive-2009-10/0512.shtml

[Daniel Shahaf]

Post by edg

Post by Tony Sweeney
Unlike 'svnadmin dump', hotcopy will happily back up a corrupt revision
and not tell you. It's really just a clever filesystem backup with a
very careful time ordering of certain key files in case there is a
transaction in progress when it runs. Having been bitten by this
myself[*], we now run svnadmin out of cron every night before we run our
hotcopy. We also keep a week's worth of hotcopies, and I check my cron
emails every morning.

I have been working under the assumption that the best way to
svnadmin dump REPO > NUL ( or to ' > /dev/null' for 'nix)
rather than
svnadmin verify REPO
The reason being that the former also checks revision property files.

+0.5

Post by edg
http://svn.haxx.se/users/archive-2009-10/0512.shtml

See http://subversion.tigris.org/issues/show_bug.cgi?id=3441