Discussion:
merginfo corruption?
Peter van Hoof
2016-10-27 10:14:49 UTC
Permalink
Hi,

Recently we have been having multiple instances of a problem that doesn't seem
to be going away. The easiest way to see it is to reproduce the following steps.

1) check out a working copy of one of our development branches:

svn co svn://svn.nublado.org/cloudy/branches/backtrace

Changes from the trunk were last merged to this branch in r11144. This checkout
looks fine and I want to merge the more recent changes from the trunk. So cd
into it and proceed.

2) First take a look at which revisions are eligible:

svn mergeinfo --show-revs eligible ^/trunk .
r11156
r11159
r11160
r11165
r11166
r11167
[ snip... ]
r11337
r11338
r11339
r11340
r11341

This all looks very plausible. The first commit to the trunk after r11144 is
r11156 and the last is r11341. All numbers shown above are correct. I didn't
check all the numbers I cut out, but I assume they are correct as well.

3) All looks fine, so do the merge:

svn merge ^/trunk .
--- Merging r8669 into 'source':
C source/rt_continuum_shield_fcn.cpp
--- Recording mergeinfo for merge of r8667 through r8669 into '.':
U .
--- Recording mergeinfo for merge of r8667 through r8669 into 'source':
U source
Summary of conflicts:
Text conflicts: 1
Conflict discovered in file 'source/rt_continuum_shield_fcn.cpp'.
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
(mc) my side of conflict, (tc) their side of conflict,
(s) show all options: ^CSummary of conflicts:
Text conflicts: 1
svn: E155027: Unable to resolve conflicts on
'/home/pvh/noot/backtrace/source/rt_continuum_shield_fcn.cpp'

I hit ^C here. So all of a sudden svn wants to merge r8669, even though it was
not marked as eligible before (and was already merged a long time ago)...

So what is this? Corrupted information in the svn:mergeinfo propety? A bug in
svn? Or both?

And more importantly how do I fix this? I tried doing this brute-force approach

svn revert -R .

svn merge --record-only -r1:11144 ^/trunk .
--- Merging r2 through r11144 into '.':
G tsuite/auto
G tsuite
G .
--- Recording mergeinfo for merge of r2 through r11144 into '.':
U .
--- Recording mergeinfo for merge of r2 through r11144 into
'data/lamda/masterlist/Lamda.ini':
U data/lamda/masterlist/Lamda.ini
--- Recording mergeinfo for merge of r2 through r11144 into
'data/stout/zn/zn_19/zn_19.coll':
U data/stout/zn/zn_19/zn_19.coll
--- Eliding mergeinfo from 'data/stout/zn/zn_19/zn_19.coll':
U data/stout/zn/zn_19/zn_19.coll
--- Recording mergeinfo for merge of r2 through r11144 into
'data/stout/zn/zn_19/zn_19.nrg':
U data/stout/zn/zn_19/zn_19.nrg
--- Eliding mergeinfo from 'data/stout/zn/zn_19/zn_19.nrg':
U data/stout/zn/zn_19/zn_19.nrg
--- Recording mergeinfo for merge of r2 through r11144 into
'data/stout/zn/zn_19/zn_19.tp':
U data/stout/zn/zn_19/zn_19.tp
--- Eliding mergeinfo from 'data/stout/zn/zn_19/zn_19.tp':
U data/stout/zn/zn_19/zn_19.tp
--- Recording mergeinfo for merge of r2 through r11144 into 'source':
U source
--- Recording mergeinfo for merge of r2 through r11144 into 'tsuite':
U tsuite
--- Recording mergeinfo for merge of r2 through r11144 into 'tsuite/auto':
U tsuite/auto
--- Recording mergeinfo for merge of r2 through r11144 into
'tsuite/auto/chianti_all_cool.dat':
U tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r2 through r11144 into
'tsuite/auto/chianti_all_cool.in':
U tsuite/auto/chianti_all_cool.in

svn merge ^/trunk .
--- Merging r11145 through r11342 into '.':
U data/chianti/masterlist/CloudyChianti.ini
G data/lamda/masterlist/Lamda.ini
U data/stout/c/c_3/c_3.coll
[ snip... ]
C data/stout/zn/zn_19/zn_19.coll
C data/stout/zn/zn_19/zn_19.nrg
C data/stout/zn/zn_19/zn_19.tp
[ snip... ]
G tsuite
G .
--- Recording mergeinfo for merge of r11145 through r11342 into '.':
G .
--- Recording mergeinfo for merge of r11145 through r11342 into
'data/lamda/masterlist/Lamda.ini':
G data/lamda/masterlist/Lamda.ini
--- Recording mergeinfo for merge of r11145 through r11342 into 'source':
G source
--- Recording mergeinfo for merge of r11145 through r11342 into 'tsuite':
G tsuite
--- Recording mergeinfo for merge of r11145 through r11342 into 'tsuite/auto':
G tsuite/auto
--- Recording mergeinfo for merge of r11145 through r11342 into
'tsuite/auto/chianti_all_cool.dat':
G tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r11145 through r11342 into
'tsuite/auto/chianti_all_cool.in':
G tsuite/auto/chianti_all_cool.in
Summary of conflicts:
Property conflicts: 3
Conflict for property 'svn:mergeinfo' discovered on
'data/stout/zn/zn_19/zn_19.coll'.
local delete, incoming edit upon merge
Select: (p) postpone, (mf) my version, (tf) their version,
(dc) display conflict, (e) edit property, (q) quit resolution,
(h) help:

Here I hit ^C again. So this starts promising enough, but then I am faced with a
conflict in the svn:mergeinfo property and I have no idea how to resolve that.

This kind of problem seems to be spreading to more and more development
branches, so I am really keen on finding a definitive solution for this. This is
starting to interfere quite seriously with our workflow. So any help is appreciated!

The client I am using is version 1.9.4 (r1740329) from openSUSE Tumbleweed and
our server was recently upgraded to 1.9.4 from CollabNet. It was 1.8.5 from
CollabNet before.


Cheers,

Peter.
Johan Corveleyn
2016-10-28 14:16:48 UTC
Permalink
Post by Peter van Hoof
Hi,
Recently we have been having multiple instances of a problem that doesn't
seem to be going away. The easiest way to see it is to reproduce the
following steps.
svn co svn://svn.nublado.org/cloudy/branches/backtrace
Changes from the trunk were last merged to this branch in r11144. This
checkout looks fine and I want to merge the more recent changes from the
trunk. So cd into it and proceed.
svn mergeinfo --show-revs eligible ^/trunk .
r11156
r11159
r11160
r11165
r11166
r11167
[ snip... ]
r11337
r11338
r11339
r11340
r11341
This all looks very plausible. The first commit to the trunk after r11144 is
r11156 and the last is r11341. All numbers shown above are correct. I didn't
check all the numbers I cut out, but I assume they are correct as well.
svn merge ^/trunk .
C source/rt_continuum_shield_fcn.cpp
U .
U source
Text conflicts: 1
Conflict discovered in file 'source/rt_continuum_shield_fcn.cpp'.
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
(mc) my side of conflict, (tc) their side of conflict,
Text conflicts: 1
svn: E155027: Unable to resolve conflicts on
'/home/pvh/noot/backtrace/source/rt_continuum_shield_fcn.cpp'
I hit ^C here. So all of a sudden svn wants to merge r8669, even though it
was not marked as eligible before (and was already merged a long time
ago)...
So what is this? Corrupted information in the svn:mergeinfo propety? A bug
in svn? Or both?
Maybe your branch wasn't really fully synced with ^/trunk up to
r11144, or at least not all mergeinfo related to this sync was fully
committed. I would guess that svn is trying to merge r8669 into
'source', because 'source' has "subtree mergeinfo", and someone didn't
commit the mergeinfo-modification on 'source', way back when he merged
r8669.

Further down in your output below, I can see that 'source' actually
has subtree mergeinfo (because svn records it there). And as you
probably know: once you have subtree mergeinfo, it *always* has to be
kept up to date (further merges to the parent will automatically
modify any subtree mergeinfo below -- those property mods have to be
committed or mergeinfo will be wrong). So maybe someone forgot to
commit the property modification on 'source' when he merged r8669. You
can probably verify this by checking the exact svn:mergeinfo property
on 'source' (with 'svn propget svn:mergeinfo').
Post by Peter van Hoof
And more importantly how do I fix this? I tried doing this brute-force approach
Okay, seems like a good technique to fix this, but ...
Post by Peter van Hoof
svn revert -R .
svn merge --record-only -r1:11144 ^/trunk .
G tsuite/auto
G tsuite
G .
U .
--- Recording mergeinfo for merge of r2 through r11144 into
U data/lamda/masterlist/Lamda.ini
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.coll
U data/stout/zn/zn_19/zn_19.coll
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.nrg
U data/stout/zn/zn_19/zn_19.nrg
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.tp
U data/stout/zn/zn_19/zn_19.tp
U source
U tsuite
U tsuite/auto
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.in
Here you should probably perform a commit ("Fixing merginfo of
previous merges" or something like that). Note the lines "Eliding
mergeinfo from 'data/stout/zn/zn_19/zn_19.coll'" and other "Eliding"
lines. This means svn removes subtree mergeinfo there, because it's
redundant (it's the same as what the parent directory says).

But you leave those property modifications as local mods without
committing, which I think lead to the next problem.
Post by Peter van Hoof
svn merge ^/trunk .
U data/chianti/masterlist/CloudyChianti.ini
G data/lamda/masterlist/Lamda.ini
U data/stout/c/c_3/c_3.coll
[ snip... ]
C data/stout/zn/zn_19/zn_19.coll
C data/stout/zn/zn_19/zn_19.nrg
C data/stout/zn/zn_19/zn_19.tp
[ snip... ]
G tsuite
G .
G .
--- Recording mergeinfo for merge of r11145 through r11342 into
G data/lamda/masterlist/Lamda.ini
G source
G tsuite
G tsuite/auto
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.in
Property conflicts: 3
Conflict for property 'svn:mergeinfo' discovered on
'data/stout/zn/zn_19/zn_19.coll'.
local delete, incoming edit upon merge
Select: (p) postpone, (mf) my version, (tf) their version,
(dc) display conflict, (e) edit property, (q) quit resolution,
Here I hit ^C again. So this starts promising enough, but then I am faced
with a conflict in the svn:mergeinfo property and I have no idea how to
resolve that.
So, that's a property modification coming in on the node where your
previous step just removed (elided) the mergeinfo. I'm not 100% sure,
but I'd try to commit the previous step first, and then rerun that
last command. Hope that fixes it ...
Post by Peter van Hoof
This kind of problem seems to be spreading to more and more development
branches, so I am really keen on finding a definitive solution for this.
This is starting to interfere quite seriously with our workflow. So any help
is appreciated!
Maybe interesting for you:
https://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer

This hasn't been released as part of any official svn release yet, but
it seems to work quite well for some people, and could surely use
additional testing :-) (see [1] for another thread mentioning this
tool).

[1] https://svn.haxx.se/users/archive-2016-06/0062.shtml
--
Johan
Peter van Hoof
2016-11-03 11:49:26 UTC
Permalink
Hi Johan,
Post by Johan Corveleyn
Post by Peter van Hoof
Hi,
Recently we have been having multiple instances of a problem that doesn't
seem to be going away. The easiest way to see it is to reproduce the
following steps.
svn co svn://svn.nublado.org/cloudy/branches/backtrace
Changes from the trunk were last merged to this branch in r11144. This
checkout looks fine and I want to merge the more recent changes from the
trunk. So cd into it and proceed.
svn mergeinfo --show-revs eligible ^/trunk .
r11156
r11159
r11160
r11165
r11166
r11167
[ snip... ]
r11337
r11338
r11339
r11340
r11341
This all looks very plausible. The first commit to the trunk after r11144 is
r11156 and the last is r11341. All numbers shown above are correct. I didn't
check all the numbers I cut out, but I assume they are correct as well.
svn merge ^/trunk .
C source/rt_continuum_shield_fcn.cpp
U .
U source
Text conflicts: 1
Conflict discovered in file 'source/rt_continuum_shield_fcn.cpp'.
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
(mc) my side of conflict, (tc) their side of conflict,
Text conflicts: 1
svn: E155027: Unable to resolve conflicts on
'/home/pvh/noot/backtrace/source/rt_continuum_shield_fcn.cpp'
I hit ^C here. So all of a sudden svn wants to merge r8669, even though it
was not marked as eligible before (and was already merged a long time
ago)...
So what is this? Corrupted information in the svn:mergeinfo propety? A bug
in svn? Or both?
Maybe your branch wasn't really fully synced with ^/trunk up to
r11144, or at least not all mergeinfo related to this sync was fully
committed. I would guess that svn is trying to merge r8669 into
'source', because 'source' has "subtree mergeinfo", and someone didn't
commit the mergeinfo-modification on 'source', way back when he merged
r8669.
The branch was created in r8667 and the first commit to the branch was r8668.
The changes on the trunk (including r8669) were merged in r8689.

I am not even sure how you can commit changes without committing mergeinfo. As
far as I know that is all automatic. To my knowledge I never omitted mergeinfo
and I am pretty sure none of our other users did. I committed r8689 myself, so
the mergeinfo should have been committed too.

I double-checked this and checked out the branch at r8689. This is the result

% svn co svn://svn.nublado.org/cloudy/branches/***@8689
% cd backtrace/
% svn proplist --verbose source/
[...snip...]
/trunk/source:8667-8688

So the revision in question is definitely marked as merged there... This looks
OK, which is also consistent with the fact that subsequent merges from trunk
showed no problems (r8784, r8788, r8815, r11144).

I think that the fragmented mergeinfo is a result of cherry-picking commits on
other branches by other users.

Also, your suggestions do not explain why r8669 wasn't listed in the output of
the "svn mergeinfo --show-revs eligible ^/trunk ." command. If that revision
truly was missing from the mergeinfo, it should have been listed there.
Post by Johan Corveleyn
Further down in your output below, I can see that 'source' actually
has subtree mergeinfo (because svn records it there). And as you
probably know: once you have subtree mergeinfo, it *always* has to be
kept up to date (further merges to the parent will automatically
modify any subtree mergeinfo below -- those property mods have to be
committed or mergeinfo will be wrong). So maybe someone forgot to
commit the property modification on 'source' when he merged r8669. You
can probably verify this by checking the exact svn:mergeinfo property
on 'source' (with 'svn propget svn:mergeinfo').
Post by Peter van Hoof
And more importantly how do I fix this? I tried doing this brute-force approach
Okay, seems like a good technique to fix this, but ...
Post by Peter van Hoof
svn revert -R .
svn merge --record-only -r1:11144 ^/trunk .
G tsuite/auto
G tsuite
G .
U .
--- Recording mergeinfo for merge of r2 through r11144 into
U data/lamda/masterlist/Lamda.ini
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.coll
U data/stout/zn/zn_19/zn_19.coll
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.nrg
U data/stout/zn/zn_19/zn_19.nrg
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.tp
U data/stout/zn/zn_19/zn_19.tp
U source
U tsuite
U tsuite/auto
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.in
Here you should probably perform a commit ("Fixing merginfo of
previous merges" or something like that). Note the lines "Eliding
mergeinfo from 'data/stout/zn/zn_19/zn_19.coll'" and other "Eliding"
lines. This means svn removes subtree mergeinfo there, because it's
redundant (it's the same as what the parent directory says).
OK, this does seem to make a difference and the merge finishes OK, but I do not
understand why. The "svn merge ^/trunk ." command behaved differently before and
after the "svn merge --record-only -r1:11144 ^/trunk ." command. This makes it
clear that it is looking at the mergeinfo records in the local wc (this is the
only thing that changed). But it seems it is also looking at the mergeinfo
records on the server? Why is that? It sounds like a recipe for problems to me
if it is looking for mergeinfo in two different places...

But more importantly. What is the root cause of these problems? The mergeinfo
records look OK to me after r8689 as I showed above. So what convinced svn that
it needed to merge r8669 a second time?
Post by Johan Corveleyn
But you leave those property modifications as local mods without
committing, which I think lead to the next problem.
Post by Peter van Hoof
svn merge ^/trunk .
U data/chianti/masterlist/CloudyChianti.ini
G data/lamda/masterlist/Lamda.ini
U data/stout/c/c_3/c_3.coll
[ snip... ]
C data/stout/zn/zn_19/zn_19.coll
C data/stout/zn/zn_19/zn_19.nrg
C data/stout/zn/zn_19/zn_19.tp
[ snip... ]
G tsuite
G .
G .
--- Recording mergeinfo for merge of r11145 through r11342 into
G data/lamda/masterlist/Lamda.ini
G source
G tsuite
G tsuite/auto
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.in
Property conflicts: 3
Conflict for property 'svn:mergeinfo' discovered on
'data/stout/zn/zn_19/zn_19.coll'.
local delete, incoming edit upon merge
Select: (p) postpone, (mf) my version, (tf) their version,
(dc) display conflict, (e) edit property, (q) quit resolution,
Here I hit ^C again. So this starts promising enough, but then I am faced
with a conflict in the svn:mergeinfo property and I have no idea how to
resolve that.
So, that's a property modification coming in on the node where your
previous step just removed (elided) the mergeinfo. I'm not 100% sure,
but I'd try to commit the previous step first, and then rerun that
last command. Hope that fixes it ...
Post by Peter van Hoof
This kind of problem seems to be spreading to more and more development
branches, so I am really keen on finding a definitive solution for this.
This is starting to interfere quite seriously with our workflow. So any help
is appreciated!
https://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer
This hasn't been released as part of any official svn release yet, but
it seems to work quite well for some people, and could surely use
additional testing :-) (see [1] for another thread mentioning this
tool).
[1] https://svn.haxx.se/users/archive-2016-06/0062.shtml
I will have a look at this as well.


Thanks!

Peter.
Stefan Sperling
2016-11-03 12:32:19 UTC
Permalink
Post by Peter van Hoof
I am not even sure how you can commit changes without committing mergeinfo.
As far as I know that is all automatic.
The designers of SVN's merge-tracking decided to re-use the existing
versioned SVN properties to store mergeinfo, instead of iventing some new
meta-data storage specific to mergeinfo. This decision led to some
downsides for usability which users need to be aware of.

Say a change is merged from trunk and two existing svn:mergeinfo
properties are updated during this merge:

- on the root of the working copy (the merge target)
- and a subdirectory A

Further assume that the merge edits the file A/B/C/D/foo.txt.

Now, if the user runs a commit like this:

svn commit A/B/C/D/foo.txt

Then changes to foo.txt are committed, but the property changes on
the root directory and the subdirectory A remain as local changes
in the working copy. Which is *bad*. The whole working copy should
have been committed.

Say the user uses the TortoiseSVN client and right-clicks on the
file A/B/C/D/foo.txt to commit it. Again, the property changes may
not be committed along unless the user carefully checks the list
of paths in the commit dialog for additional changes which need
to be committed as well.
Post by Peter van Hoof
I double-checked this and checked out the branch at r8689. This is the result
% cd backtrace/
% svn proplist --verbose source/
[...snip...]
/trunk/source:8667-8688
So the revision in question is definitely marked as merged there... This
looks OK, which is also consistent with the fact that subsequent merges from
trunk showed no problems (r8784, r8788, r8815, r11144).
You're only looking at one path. Mergeinfo is only inherited to paths
which do not have an svn:mergeinfo property themselves.

In my example above, svn:mergeinfo on the root of the working copy
is irrelevant for any child (directory or file) within subtree A.
Only mergeinfo on subtree A is considered for those children (unless
of course where children have svn:mergeinfo themselves).
Post by Peter van Hoof
I think that the fragmented mergeinfo is a result of cherry-picking commits
on other branches by other users.
Quite likely. It mostly depends on which merge target people select.

If a subtree A has mergeinfo that means somebody once ran a command like:

svn merge -c 500 ^/trunk/A ./A

instead of

svn merge -c 500 ^/trunk .

In TortoiseSVN the merge target is selected by rhight-clicking on it and
selecting the 'Merge' command from the context menu. This should usually
be the working copy root (corresponding to a branch root) but users are
free to click on any other file or directory and often do just that.
Post by Peter van Hoof
Also, your suggestions do not explain why r8669 wasn't listed in the output
of the "svn mergeinfo --show-revs eligible ^/trunk ." command. If that
revision truly was missing from the mergeinfo, it should have been listed
there.
It seems the mergeinfo command is broken here because it did not consider
subtree mergeinfo on data/stout/zn/zn_19/zn_19.tp.

Evidently the file 'data/stout/zn/zn_19/zn_19.tp' had incomplete mergeinfo
Post by Peter van Hoof
Post by Johan Corveleyn
Post by Peter van Hoof
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.tp
U data/stout/zn/zn_19/zn_19.tp
Elision only happens when two svn:mergeinfo properties contain the exact
same lines of text, apart from the path-wise difference resulting from the
relative positions of these svn:mergeinfo properties in the directory tree.

So this merge changed mergeinfo on data/stout/zn/zn_19/zn_19.tp in such
a way that it became redundant with mergeinfo on the nearest parent.
Post by Peter van Hoof
Post by Johan Corveleyn
Post by Peter van Hoof
U source
U tsuite
U tsuite/auto
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.in
Here you should probably perform a commit ("Fixing merginfo of
previous merges" or something like that). Note the lines "Eliding
mergeinfo from 'data/stout/zn/zn_19/zn_19.coll'" and other "Eliding"
lines. This means svn removes subtree mergeinfo there, because it's
redundant (it's the same as what the parent directory says).
OK, this does seem to make a difference and the merge finishes OK, but I do
not understand why. The "svn merge ^/trunk ." command behaved differently
before and after the "svn merge --record-only -r1:11144 ^/trunk ." command.
A --record-only merge affects all files and directories with svn:mergeinfo.
Therefore it changed mergeinfo on data/stout/zn/zn_19/zn_19.tp.

A normal merge only changes svn:mergeinfo properties of files and directories
which were modified by the current merge. The normal merge used to behave like
the --record-only merge but that lead to complaints from users who found the
large amount of property changes after what could be a merge which affected
only a single file rather confusing.

If you had run a merge such as:

svn merge ^/trunk/data/stout/zn/zn_19/zn_19.tp data/stout/zn/zn_19/zn_19.tp

mergeinfo would also have been elided.

More details about mergeinfo can be found in this article:
https://www.open.collab.net/community/subversion/articles/merge-info.html
It's rather old but mostly still relevant.
The section "Mixed Revision Working Copies and Mergeinfo" for example is
now less relevant because merges into mixed-revision working copies have
since been disabled by default.
Peter van Hoof
2017-01-23 11:10:16 UTC
Permalink
Hi,

I have been sidetracked by other work for a while, but would like to get
back to this now...
Post by Stefan Sperling
Post by Peter van Hoof
I double-checked this and checked out the branch at r8689. This is the result
% cd backtrace/
% svn proplist --verbose source/
[...snip...]
/trunk/source:8667-8688
So the revision in question is definitely marked as merged there... This
looks OK, which is also consistent with the fact that subsequent merges from
trunk showed no problems (r8784, r8788, r8815, r11144).
You're only looking at one path. Mergeinfo is only inherited to paths
which do not have an svn:mergeinfo property themselves.
In my example above, svn:mergeinfo on the root of the working copy
is irrelevant for any child (directory or file) within subtree A.
Only mergeinfo on subtree A is considered for those children (unless
of course where children have svn:mergeinfo themselves).
The backtrace branch has only been maintained by me, and my routine has
always been to merge all changes from the root of the trunk to the root
of the branch and then commit from the root of the branch. So I would
expect the mergeinfo to be the same on subdirectories or individual
files. However, when I checked this, I found this not to be the case.

These paths have separate mergeinfo records at r8689:

data/lamda/masterlist/Lamda.ini
source
tsuite/auto/chianti_all_cool.dat
tsuite/auto/chianti_all_cool.in

Apart from "source" they are all files. The full mergeinfo record for
source is

svn:mergeinfo
/branches/HfsLines/source:6561-8024
/branches/SED/source:6190-7044
/branches/WangYe/source:5202-5690
/branches/block/source:6454-6465
/branches/cden/source:6478-6586
/branches/chianti/source:5345-5686,6680-6806
/branches/collapsed/source:7331-8615
/branches/convergence/source:8443-8533,8561-8633
/branches/convfix/source:7216-7234
/branches/depart/source:7256-7655
/branches/diatoms/source:4635-5051
/branches/fixct/source:5862-5945
/branches/gtest/source:7057-7198
/branches/h2star/source:5156
/branches/ionrec/source:4983-5281
/branches/linesInit/source:6627-6633
/branches/newmole/source:502-6043
/branches/nstout/source:8422-8435
/branches/optnl/source:6638-6757
/branches/ots2/source:6379,6393-6394,6429-6430,6448
/branches/parser/source:3753,3755,3763
/branches/parser2/source:7036-7070
/branches/romas3/source:6262-6786

/branches/stout/source:6885-6945,6976-7373,7388-7415,7417-7420,7668-7671,8202-8203,8378-8395
/branches/stout1/source:8135-8171
/branches/stout2/source:6109-6538,7581-7619,7694-7706
/branches/stout3/source:7626-7633,7635-7663
/branches/sup/source:7503-7533
/trunk/source:8667-8688

So it has the same information for /trunk as the record at the root. The
three files with individual mergeinfo records look different. One
example for tsuite/auto/chianti_all_cool.dat is

svn:mergeinfo
/branches/FixWvlng/tsuite/auto/chianti_all_cool.dat:7890-7912
/branches/collapsed/tsuite/auto/chianti_all_cool.dat:8609-8615

/branches/convergence/tsuite/auto/chianti_all_cool.dat:8443-8533,8561-8633
/branches/nstout/tsuite/auto/chianti_all_cool.dat:8422-8435
/branches/stout/tsuite/auto/chianti_all_cool.dat:8202-8203,8378-8395

So at revision 8689, the mergeinfo records at the root and the source
subdirectory both indicate that r8669 is already merged at that point.
Yet, when I do the merge now (at the head of this branch), it wants to
merge changes in source/rt_continuum_shield_fcn.cpp from r8669. For me
this points to corruption in the mergeinfo record or a bug in subversion
(or both of course...).

So I would like to consider the svn-mergeinfo-normalizer tool that was
mentioned earlier in this thread. But I need additional guidance on how
to compile it and how I should use it. Do I need to fix the trunk, or
the development branch, or both? How does this compare to the
mergeinfo-sanitizer.py script?


Cheers,

Peter.

Stefan
2016-11-08 22:41:16 UTC
Permalink
Hi Peter,
Post by Peter van Hoof
Hi Johan,
Post by Johan Corveleyn
Post by Peter van Hoof
Hi,
Recently we have been having multiple instances of a problem that doesn't
seem to be going away. The easiest way to see it is to reproduce the
following steps.
svn co svn://svn.nublado.org/cloudy/branches/backtrace
Changes from the trunk were last merged to this branch in r11144. This
checkout looks fine and I want to merge the more recent changes from the
trunk. So cd into it and proceed.
svn mergeinfo --show-revs eligible ^/trunk .
r11156
r11159
r11160
r11165
r11166
r11167
[ snip... ]
r11337
r11338
r11339
r11340
r11341
This all looks very plausible. The first commit to the trunk after r11144 is
r11156 and the last is r11341. All numbers shown above are correct. I didn't
check all the numbers I cut out, but I assume they are correct as well.
svn merge ^/trunk .
C source/rt_continuum_shield_fcn.cpp
U .
U source
Text conflicts: 1
Conflict discovered in file 'source/rt_continuum_shield_fcn.cpp'.
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
(mc) my side of conflict, (tc) their side of conflict,
Text conflicts: 1
svn: E155027: Unable to resolve conflicts on
'/home/pvh/noot/backtrace/source/rt_continuum_shield_fcn.cpp'
I hit ^C here. So all of a sudden svn wants to merge r8669, even though it
was not marked as eligible before (and was already merged a long time
ago)...
So what is this? Corrupted information in the svn:mergeinfo propety? A bug
in svn? Or both?
Maybe your branch wasn't really fully synced with ^/trunk up to
r11144, or at least not all mergeinfo related to this sync was fully
committed. I would guess that svn is trying to merge r8669 into
'source', because 'source' has "subtree mergeinfo", and someone didn't
commit the mergeinfo-modification on 'source', way back when he merged
r8669.
The branch was created in r8667 and the first commit to the branch was
r8668. The changes on the trunk (including r8669) were merged in r8689.
I am not even sure how you can commit changes without committing
mergeinfo. As far as I know that is all automatic. To my knowledge I
never omitted mergeinfo and I am pretty sure none of our other users
did. I committed r8689 myself, so the mergeinfo should have been
committed too.
I double-checked this and checked out the branch at r8689. This is the result
% cd backtrace/
% svn proplist --verbose source/
[...snip...]
/trunk/source:8667-8688
So the revision in question is definitely marked as merged there...
This looks OK, which is also consistent with the fact that subsequent
merges from trunk showed no problems (r8784, r8788, r8815, r11144).
I think that the fragmented mergeinfo is a result of cherry-picking
commits on other branches by other users.
Also, your suggestions do not explain why r8669 wasn't listed in the
output of the "svn mergeinfo --show-revs eligible ^/trunk ." command.
If that revision truly was missing from the mergeinfo, it should have
been listed there.
Post by Johan Corveleyn
Further down in your output below, I can see that 'source' actually
has subtree mergeinfo (because svn records it there). And as you
probably know: once you have subtree mergeinfo, it *always* has to be
kept up to date (further merges to the parent will automatically
modify any subtree mergeinfo below -- those property mods have to be
committed or mergeinfo will be wrong). So maybe someone forgot to
commit the property modification on 'source' when he merged r8669. You
can probably verify this by checking the exact svn:mergeinfo property
on 'source' (with 'svn propget svn:mergeinfo').
Post by Peter van Hoof
And more importantly how do I fix this? I tried doing this brute-force approach
Okay, seems like a good technique to fix this, but ...
Post by Peter van Hoof
svn revert -R .
svn merge --record-only -r1:11144 ^/trunk .
G tsuite/auto
G tsuite
G .
U .
--- Recording mergeinfo for merge of r2 through r11144 into
U data/lamda/masterlist/Lamda.ini
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.coll
U data/stout/zn/zn_19/zn_19.coll
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.nrg
U data/stout/zn/zn_19/zn_19.nrg
--- Recording mergeinfo for merge of r2 through r11144 into
U data/stout/zn/zn_19/zn_19.tp
U data/stout/zn/zn_19/zn_19.tp
U source
U tsuite
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r2 through r11144 into
U tsuite/auto/chianti_all_cool.in
Here you should probably perform a commit ("Fixing merginfo of
previous merges" or something like that). Note the lines "Eliding
mergeinfo from 'data/stout/zn/zn_19/zn_19.coll'" and other "Eliding"
lines. This means svn removes subtree mergeinfo there, because it's
redundant (it's the same as what the parent directory says).
OK, this does seem to make a difference and the merge finishes OK, but
I do not understand why. The "svn merge ^/trunk ." command behaved
differently before and after the "svn merge --record-only -r1:11144
^/trunk ." command. This makes it clear that it is looking at the
mergeinfo records in the local wc (this is the only thing that
changed). But it seems it is also looking at the mergeinfo records on
the server? Why is that? It sounds like a recipe for problems to me if
it is looking for mergeinfo in two different places...
But more importantly. What is the root cause of these problems? The
mergeinfo records look OK to me after r8689 as I showed above. So what
convinced svn that it needed to merge r8669 a second time?
Post by Johan Corveleyn
But you leave those property modifications as local mods without
committing, which I think lead to the next problem.
Post by Peter van Hoof
svn merge ^/trunk .
U data/chianti/masterlist/CloudyChianti.ini
G data/lamda/masterlist/Lamda.ini
U data/stout/c/c_3/c_3.coll
[ snip... ]
C data/stout/zn/zn_19/zn_19.coll
C data/stout/zn/zn_19/zn_19.nrg
C data/stout/zn/zn_19/zn_19.tp
[ snip... ]
G tsuite
G .
G .
--- Recording mergeinfo for merge of r11145 through r11342 into
G data/lamda/masterlist/Lamda.ini
G source
G tsuite
G tsuite/auto
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.dat
--- Recording mergeinfo for merge of r11145 through r11342 into
G tsuite/auto/chianti_all_cool.in
Property conflicts: 3
Conflict for property 'svn:mergeinfo' discovered on
'data/stout/zn/zn_19/zn_19.coll'.
local delete, incoming edit upon merge
Select: (p) postpone, (mf) my version, (tf) their version,
(dc) display conflict, (e) edit property, (q) quit resolution,
Here I hit ^C again. So this starts promising enough, but then I am faced
with a conflict in the svn:mergeinfo property and I have no idea how to
resolve that.
So, that's a property modification coming in on the node where your
previous step just removed (elided) the mergeinfo. I'm not 100% sure,
but I'd try to commit the previous step first, and then rerun that
last command. Hope that fixes it ...
Post by Peter van Hoof
This kind of problem seems to be spreading to more and more development
branches, so I am really keen on finding a definitive solution for this.
This is starting to interfere quite seriously with our workflow. So any help
is appreciated!
https://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer
This hasn't been released as part of any official svn release yet, but
it seems to work quite well for some people, and could surely use
additional testing :-) (see [1] for another thread mentioning this
tool).
[1] https://svn.haxx.se/users/archive-2016-06/0062.shtml
I will have a look at this as well.
Since I've been pointed to this thread by Daniel Shahaf, here's a few
things to point out the situation specific to your repository. Johan and
Stefan already provided you all the necessary general details, I take it.

Since you suggested you have the same issue in other branches, maybe the
svn normalizer tool pointed to by Johan might be of use to you (but bare
in mind it's quite experimental, since it hasn't been thoroughly tested
by a broad user base yet). If you have a Windows machine to play with, a
developer snapshot containing the precompiled tool is available here:
http://www.luke1410.de/typo3/index.php?id=97
"MaxSVN trunk-dev-r1743999-1 (Zip)" is the version you might wanna try (
http://www.luke1410.de/typo3/fileadmin/user_upload/maxsvn_trunk-dev-r1743999-1.zip
).



Back to your particular case:
So in your case you have multiple mergeinfo records (note: based on your
report here, I only checked backtrace, backtrace/source,
backtrace/data/stdout/zn/zn_19 --- there might be more).


List of relevant mergeinfo entries:
branch-related mergeinfo was omitted in the following list, since it's
not relevant for the case discussed here:

backtrace:
/trunk:8667-11143
/trunk/data/lamda/masterlist:7854-7904

backtrace/source:
*no mergeinfo record for trunk*

backtrace/data/stdout/zn/zn_19/zn_19.coll:
/trunk/data/lamda/masterlist/data/stout/zn/zn_19/zn_19.coll:7854-7904
/trunk/data/stout/zn/zn_19/zn_19.coll:8667-8814,9296-11143

backtrace/data/stdout/zn/zn_19/zn19.nrg:
/trunk/data/lamda/masterlist/data/stout/zn/zn_19/zn_19.nrg:7854-7904
/trunk/data/stout/zn/zn_19/zn_19.nrg:8667-8814,9296-11143

backtrace/data/stdout/zn/zn_19/zn19.tp:
/trunk/data/lamda/masterlist/data/stout/zn/zn_19/zn_19.tp:7854-7904
/trunk/data/stout/zn/zn_19/zn_19.tp:8667-8814,9296-11143



As guessed correctly by Johan, backtrace/source contains a list of
mergeinfos, but misses all the revisions from trunk. Therefore, svn
reads this as nothing from trunk/source was merged into backtrace/source
yet. The approach using --record-only is one way to correct this, as it
would add the missing mergeinfo.



Digging into the history of the repository shows that in revision 8026
on trunk the HfsLines branch was reintegrated. As part of this
reintegration the following merge info made it into the trunk root folder:
/trunk/data/lamda/masterlist:7854-7904
/branches/stout2/data/lamda/masterlist:7769-7853

Presumably something was merged incorrectly on the reintegrated HfsLines
branch. One possible explanation would be that someone wanted to merge
only part of revisions 7854-7905 and 7769-7853 but set the incorrect
destination path for the merge command as in:
svn merge [URL]/trunk/data/lamda/masterlist . (where . would have been
the trunk checkout) instead of svn merge
[URL]/trunk/data/lamda/masterlist data/lamda/masterlist

Another possible case could have been that it was intended to merge the
whole revisions, but by mistake the wrong path was specified.

The fallout was however, that these two "incorrect" mergeinfo entries
ended up in your trunk-mergeinfo and ever since were taken over from there.

I'd certainly correct these on trunk ASAP, if I were you (aka: determine
what was intended and either mark the full trunk as merged, or have the
record correctly on data/lamda/masterlist rather than at trunk).



Taking things from that revision further suggests that in revision 8689
the mergeinfo got correctly added to bracktace/source. This was all fine
until revision 11144. In this revision the trunk-mergeinfo got removed
from backtrace/source for some reason which is beyond what one can
determine by just looking at the revision log.

So all there is to do is to revert the incorrect revert of the mergeinfo
on backtrace/source in r11144 and the following "svn merge ^/trunk ."
should then work just fine.


As stated above: If I were you I'd certainly clean-up the mergeinfo a
bit and at least remove the known cases (aka: materlist-record on trunk
and file-merge hisotry on the zn_19 files).

Regards,
Stefan
Loading...