Discussion:
Is it safe to remove deleted and no longer used branches from the svn:mergeinfo property?
Alfred von Campe
2017-05-31 15:18:48 UTC
Permalink
After many years of development, we have over 100 entries in the svn:mergeinfo property of our trunk. I am thinking about manually editing the property to remove entries for all branches that have been deleted or are no longer used (and should be deleted). I am hoping that once I merge trunk into the other active branches, the cleanup of that property will also propagate to the other branches.

So my questions are:

Is this safe to do?
Will the changes to the properly be merged to all other branches over time as one would expect?

Thanks,
Alfred
Alfred von Campe
2017-06-02 11:15:11 UTC
Permalink
Post by Alfred von Campe
After many years of development, we have over 100 entries in the svn:mergeinfo property of our trunk. I am thinking about manually editing the property to remove entries for all branches that have been deleted or are no longer used (and should be deleted). I am hoping that once I merge trunk into the other active branches, the cleanup of that property will also propagate to the other branches.
Is this safe to do?
Will the changes to the properly be merged to all other branches over time as one would expect?
Has anyone ever done this? Are there any negative repercussions? Inquiring minds (well, maybe it’s just me) want to know


Alfred
Nico Kadel-Garcia
2017-06-02 12:25:40 UTC
Permalink
Post by Alfred von Campe
After many years of development, we have over 100 entries in the
svn:mergeinfo property of our trunk. I am thinking about manually editing
the property to remove entries for all branches that have been deleted or
are no longer used (and should be deleted). I am hoping that once I merge
trunk into the other active branches, the cleanup of that property will also
propagate to the other branches.
Is this safe to do?
Will the changes to the properly be merged to all other branches over time
as one would expect?
Has anyone ever done this? Are there any negative repercussions? Inquiring
minds (well, maybe it’s just me) want to know…
Alfred
I suspect you're going to hurt yourself if you try this sort of stunt.
Not that it might not work in theory for some cases, but it's not
tested and is against the basic "never, never, never, and did I
mention never alter history" approach to Subversion.

What I think you *can* do is an "svnadmin dump | svndumpfilter exclude
(obsolete branches) | svnadmin load" to another Subversion repo, and
switch to that repo to have a modified history. That's the only
officially supported way to flush obsolete content. There are some
alternatives that I've personally found effective, that are not so
well supported. Using "git-svn" to get the content to a git repo,
deleting content *there*, running a "git gc --aggressive" to clear
content, and then pushing the content back up to another Subversion
server has been effective for me. But it tends to lose svn:properties
settings, and it makes many Subversion admins twitch and shudder and
break out in hives when they realize you're using git to provide
features that violate the basic "never delete history" of Subversion
design philosophy.
Alfred von Campe
2017-06-02 12:40:15 UTC
Permalink
Post by Nico Kadel-Garcia
I suspect you're going to hurt yourself if you try this sort of stunt.
Not that it might not work in theory for some cases, but it's not
tested and is against the basic "never, never, never, and did I
mention never alter history" approach to Subversion.
Why? The svn:mergeinfo property just kept track of what revisions have
been merged from a given branch. If I delete the branch the information
is useless, but there is no automatic cleanup of that property.
Post by Nico Kadel-Garcia
What I think you *can* do is an "svnadmin dump | svndumpfilter exclude
(obsolete branches) | svnadmin load" to another Subversion repo, and
switch to that repo to have a modified history.
I don’t want to alter history. I want the older branches around so they
can be browsed and inspected for history. I just want to clean up the
svn:mergeinfo property so it’s easier to read.

I would love for one of the Subversion developers to chime in here...

Alfred
Cooke, Mark
2017-06-02 12:45:18 UTC
Permalink
-----Original Message-----
Sent: 02 June 2017 13:40
Post by Nico Kadel-Garcia
I suspect you're going to hurt yourself if you try this sort of stunt.
Not that it might not work in theory for some cases, but it's not
tested and is against the basic "never, never, never, and did I
mention never alter history" approach to Subversion.
Why? The svn:mergeinfo property just kept track of what revisions have
been merged from a given branch. If I delete the branch the information
is useless, but there is no automatic cleanup of that property.
How did you "delete the branch"? Unless you performed a dump/filter/load then they are all still there, in all those revisions referenced by the merge info property... Hence the information is still valid.

I seem to remember that new commands (or a script) are available to try to consolidate svn:mergeinfo, have you seen or tried any of those?
Post by Nico Kadel-Garcia
What I think you *can* do is an "svnadmin dump | svndumpfilter exclude
(obsolete branches) | svnadmin load" to another Subversion repo, and
switch to that repo to have a modified history.
I don’t want to alter history. I want the older branches around so they
can be browsed and inspected for history. I just want to clean up the
svn:mergeinfo property so it’s easier to read.
I would love for one of the Subversion developers to chime in here...
Alfred
(I'm not a
Mark Phippard
2017-06-02 12:46:29 UTC
Permalink
Have you seen this script?

http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/mergeinfo-sanitizer.py

I believe it does this though I have never tried it.

Mark
Post by Alfred von Campe
After many years of development, we have over 100 entries in the
svn:mergeinfo property of our trunk. I am thinking about manually editing
the property to remove entries for all branches that have been deleted or
are no longer used (and should be deleted). I am hoping that once I merge
trunk into the other active branches, the cleanup of that property will
also propagate to the other branches.
- Is this safe to do?
- Will the changes to the properly be merged to all other branches
over time as one would expect?
Thanks,
Alfred
--
Thanks

Mark Phippard
http://markphip.blogspot.com/
Mark Phippard
2017-06-02 12:48:38 UTC
Permalink
Actually, this is the newer tool I was thinking of:

http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer/

Mark
Post by Mark Phippard
Have you seen this script?
http://svn.apache.org/repos/asf/subversion/trunk/tools/
client-side/mergeinfo-sanitizer.py
I believe it does this though I have never tried it.
Mark
Post by Alfred von Campe
After many years of development, we have over 100 entries in the
svn:mergeinfo property of our trunk. I am thinking about manually editing
the property to remove entries for all branches that have been deleted or
are no longer used (and should be deleted). I am hoping that once I merge
trunk into the other active branches, the cleanup of that property will
also propagate to the other branches.
- Is this safe to do?
- Will the changes to the properly be merged to all other branches
over time as one would expect?
Thanks,
Alfred
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
--
Thanks

Mark Phippard
http://markphip.blogspot.com/
Stefan Hett
2017-06-02 18:03:39 UTC
Permalink
Hi Alfred,
Post by Mark Phippard
http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer/
Have you seen this script?
http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/mergeinfo-sanitizer.py
<http://svn.apache.org/repos/asf/subversion/trunk/tools/client-side/mergeinfo-sanitizer.py>
I believe it does this though I have never tried it.
On Wed, May 31, 2017 at 11:18 AM, Alfred von Campe
After many years of development, we have over 100 entries in
the svn:mergeinfo property of our trunk. I am thinking about
manually editing the property to remove entries for all
branches that have been deleted or are no longer used (and
should be deleted). I am hoping that once I merge trunk into
the other active branches, the cleanup of that property will
also propagate to the other branches.
* Is this safe to do?
* Will the changes to the properly be merged to all other
branches over time as one would expect?
jcorvel made me aware of your post here.

I'm regularly using the svn mergeinfo normalizer myself. It should suit
your requirements quite well, but you'd be aware that it hasn't been
tested thoroughly by a lot of people, since it's a new tool in the not
yet released 1.10 development branch.

To get some basic documentation about what the tool does, best start
with the integrated help (svn-mergeinfo-normalizer help).
In your case all you might actually need is to run
"svn-mergeinfo-normalizer normalize --remove-obsoletes" followed by
committing the changes. Carefully verify the changes before committing
them. As said: The tool hasn't gotten much test coverage by a broader
audience yet.

There are also other useful commands (--remove-redundant which tries to
elide unnecessary mergeinfo entries as well as --combine-ranges which
will combine mergeinfo ranges into shorter representations where possible).

Useful is also its "svn-mergeinfo-normalizer analyze" command, which
will print out the information about why certain mergeinfos could not be
elided (possibly providing you with the necessary information to correct
some invalid/incorrect mergeinfo entries which might have built up over
time).

As said above: I cannot emphasize enough to be careful with using that
tool atm. While I've personally been playing around with it for over two
years now, it might still contain undetected issues/bugs.

If you wanna give it a quick try and are running on Windows, there are
prebuilt binaries available for MaxSVN (disclaimer: that's a development
binary distribution of SVN I'm maintaining):
http://www.luke1410.de/typo3/index.php?id=97. Download MaxSVN
trunk-dev-r1771118-1 and run svn-mergeinfo-normalizer contained in the
package. I'm not aware of other prebuilt sources of the current SVN
development branch (otherwise I'd have listed them here as other examples).

On a more general note on your questions:
Is it safe to do that (i.e. remove the entries for obsolete/removed
branches)?

Kind of. If your working process means that you are not going to
reinstate the removed branch in a future revision again to merge
remaining revision from it to some other branch, I'd personally consider
it a safe habit to drop the then obsolete mergeinfos. If your work
process differs, you should not remove it though IMO, since then you
might cause conflicts on merges and also lose the information about what
was merged of the other branch (if it later is reinstated).

Will the changes to the properly be merged to all other branches over
time as one would expect?

That depends on how your merge process goes and how previous merges were
done. Simply speaking, I'd suggest to remove the obsolete mergeinfos
from any branch (not only trunk or specific development branches). Not
doing so can reinstate the removed obsolete mergeinfos again (due to
following merge operations). Another situation you would prevent by
completely removing obsolete branches from the entire repository are
mergeconflicts which can occur if you merge the revision where the
obsolete branch was removed from into another branch.
--
Regards,
Stefan Hett
Alfred von Campe
2017-06-05 14:21:31 UTC
Permalink
I'm regularly using the svn mergeinfo normalizer myself. It should suit your requirements quite well, but you'd be aware that it hasn't been tested thoroughly by a lot of people, since it's a new tool in the not yet released 1.10 development branch.
I will test out this tool. However, we are running on CentOS 6.9 and I may not be able to get all its dependencies installed.
To get some basic documentation about what the tool does, best start with the integrated help (svn-mergeinfo-normalizer help).
In your case all you might actually need is to run "svn-mergeinfo-normalizer normalize --remove-obsoletes" followed by committing the changes. Carefully verify the changes before committing them. As said: The tool hasn't gotten much test coverage by a broader audience yet.
Yes, I am very careful about inspecting all changes before committing a change.
If you wanna give it a quick try and are running on Windows, there are prebuilt binaries available for MaxSVN (disclaimer: that's a development binary distribution of SVN I'm maintaining):http://www.luke1410.de/typo3/index.php?id=97 <http://www.luke1410.de/typo3/index.php?id=97>. Download MaxSVN trunk-dev-r1771118-1 and run svn-mergeinfo-normalizer contained in the package. I'm not aware of other prebuilt sources of the current SVN development branch (otherwise I'd have listed them here as other examples).
If I can’t get the tool running on our Linux systems, I will definitely check this out.
Is it safe to do that (i.e. remove the entries for obsolete/removed branches)?
Kind of. If your working process means that you are not going to reinstate the removed branch in a future revision again to merge remaining revision from it to some other branch, I'd personally consider it a safe habit to drop the then obsolete mergeinfos. If your work process differs, you should not remove it though IMO, since then you might cause conflicts on merges and also lose the information about what was merged of the other branch (if it later is reinstated).
We don’t plan to ever reuse those branches again, and in fact most of them will be deleted.

Thanks again for all your comments. I’ll report back with my findings when I get a chance to run this tool sometime later this week.

Alfred
Daniel Shahaf
2017-06-05 18:14:35 UTC
Permalink
If you wanna give it a quick try and are running on Windows, there are prebuilt binaries available for MaxSVN (disclaimer: that's a development binary distribution of SVN I'm maintaining):http://www.luke1410.de/typo3/index.php?id=97 <http://www.luke1410.de/typo3/index.php?id=97>. Download MaxSVN trunk-dev-r1771118-1 and run svn-mergeinfo-normalizer contained in the package. I'm not aware of other prebuilt sources of the current SVN development branch (otherwise I'd have listed them here as other examples).
If I can’t get the tool running on our Linux systems, I will definitely check this out.
'make svn-mergeinfo-normalizer' will build the tool. It can be run
in-tree, or you can run 'make install-tools' to install it.

(I don't see this info in README/INSTALL, although it should be there...)
Alfred von Campe
2017-06-05 20:34:51 UTC
Permalink
Post by Daniel Shahaf
'make svn-mergeinfo-normalizer' will build the tool. It can be run
in-tree, or you can run 'make install-tools' to install it.
Hmm, I don’t see a Makefile in ^/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer
or its immediate parent directories, so I checked out the entire Subversion trunk, and then
had to do a “sh autopen.sh” followed by ./configure. Now it’s complaining about “no suitable
APR found”. I think I’m going down a bit of a rathole… I don’t need/want to build all of
Subversion; I really only want to check out the svn-mergeinfo-normalizer tool.

Alfred
Daniel Shahaf
2017-06-05 21:08:27 UTC
Permalink
Post by Alfred von Campe
Post by Daniel Shahaf
'make svn-mergeinfo-normalizer' will build the tool. It can be run
in-tree, or you can run 'make install-tools' to install it.
Hmm, I don’t see a Makefile in ^/subversion/trunk/tools/client-side/svn-mergeinfo-normalizer
or its immediate parent directories, so I checked out the entire Subversion trunk, and then
had to do a “sh autopen.sh” followed by ./configure. Now it’s complaining about “no suitable
APR found”. I think I’m going down a bit of a rathole… I don’t need/want to build all of
Subversion; I really only want to check out the svn-mergeinfo-normalizer tool.
What you need to run is:

./autogen.sh && ./configure && make svn-mergeinfo-normalizer

You do need APR libraries for that. (Install the 'apr-dev' package.)

The Makefile you were looking for would be this:

cc -o svn-mergeinfo-normalizer -I ${prefix}/include/apr-1.0/ -I ${prefix}/include/subversion-1/ -L /usr/lib -Wall -lsvn_{client,wc,ra,delta,diff,subr}-1 -lapr-1 *.c -I ${svnbuildtree}/subversion -I ${svnsrctree}/subversion/include/ && LD_LIBRARY_PATH=${prefix}/lib ./svn-mergeinfo-normalizer --help

... but this command requires ${svnbuildtree} to be to a directory you ran configure in.
Daniel Shahaf
2017-06-05 21:12:59 UTC
Permalink
[ Sorry, hit 'send' by accident. ]
Post by Daniel Shahaf
./autogen.sh && ./configure && make svn-mergeinfo-normalizer
You do need APR libraries for that. (Install the 'apr-dev' package.)
cc -o svn-mergeinfo-normalizer -I ${prefix}/include/apr-1.0/ -I ${prefix}/include/subversion-1/ -L /usr/lib -Wall -lsvn_{client,wc,ra,delta,diff,subr}-1 -lapr-1 *.c -I ${svnbuildtree}/subversion -I ${svnsrctree}/subversion/include/ && LD_LIBRARY_PATH=${prefix}/lib ./svn-mergeinfo-normalizer --help
... but this command requires ${svnbuildtree} to be to a directory you ran configure in.
s/to be to/to be pointed to/. That -I${svnbuildtree} directive is
required in order to find svn_config_private.h.

The recommended/supported way is the first one: to install the apr-dev packages and run configure.

Daniel
Alfred von Campe
2017-06-06 17:36:30 UTC
Permalink
Post by Daniel Shahaf
./autogen.sh && ./configure && make svn-mergeinfo-normalizer
You do need APR libraries for that. (Install the 'apr-dev' package.)
It took a while to get all dependencies sorted out on CentOS 6 (BTW, it’s
apr-devel on Red Hat/CentOS), but I finally was able to run the configure
and make. But now I am encountering the following issue at run time:

$ svn-mergeinfo-normalizer --help
svn: E200019: Version mismatch in 'svn_gnome_keyring' (expecting equality): found 1.9.4, expected 1.10.0-dev

I really don’t want/need to get everything on this system up to version
1.10; is there a way around this?

Alfred
Daniel Shahaf
2017-06-06 17:56:39 UTC
Permalink
Post by Alfred von Campe
Post by Daniel Shahaf
./autogen.sh && ./configure && make svn-mergeinfo-normalizer
You do need APR libraries for that. (Install the 'apr-dev' package.)
It took a while to get all dependencies sorted out on CentOS 6 (BTW, it’s
apr-devel on Red Hat/CentOS), but I finally was able to run the configure
$ svn-mergeinfo-normalizer --help
svn: E200019: Version mismatch in 'svn_gnome_keyring' (expecting equality): found 1.9.4, expected 1.10.0-dev
I really don’t want/need to get everything on this system up to version
1.10; is there a way around this?
The runtime linker finds libsvn_gnome_keyring.so in /usr/lib, rather
than the one in the build tree.

Your options include:

- Link statically. (--enable-all-static to configure)

- Set LD_LIBRARY_PATH as in my 'manual' example. (If running from the
build tree, I think you'll need to point it to subversion/libsvn_*/.libs/;
if running from an installed tree, point it to ${prefix}/lib.)

- Run the executable as 'libtool --mode=execute ./svn-mergeinfo-normalizer'.
(There'll be a 'libtool' script in the root of the build tree.)

- Pass --prefix to configure, then run 'make install-tools', and run the
binary from ${prefix}/bin/svn-tools/ rather than from the build tree.

- (included for completeness) Temporarily move /usr/lib/libsvn*.so out
of the way while testing.

Daniel
Alfred von Campe
2017-06-06 18:45:12 UTC
Permalink
Post by Daniel Shahaf
The runtime linker finds libsvn_gnome_keyring.so in /usr/lib, rather
than the one in the build tree.
OK, that makes sense (I think).
Post by Daniel Shahaf
- Link statically. (--enable-all-static to configure)
When I first tried this I was getting a link error because it couldn’t find
/usr/lib/libapr-1.a. But somehow now the problem went away, and even after
doing a “svn cleanup —-delete-ignored” and running autopen.sh and ./configure
again without the —-enable-all-static option) I am getting a statically
linked binary. Weird, but at the moment irrelevant to my next problem.
Post by Daniel Shahaf
- Set LD_LIBRARY_PATH as in my 'manual' example. (If running from the
build tree, I think you'll need to point it to subversion/libsvn_*/.libs/;
if running from an installed tree, point it to ${prefix}/lib.)
There was no keyring library in the output of “ldd” when I the binary was
linked dynamically.
Post by Daniel Shahaf
- Run the executable as 'libtool --mode=execute ./svn-mergeinfo-normalizer'.
(There'll be a 'libtool' script in the root of the build tree.)
That didn’t work for me.
Post by Daniel Shahaf
- Pass --prefix to configure, then run 'make install-tools', and run the
binary from ${prefix}/bin/svn-tools/ rather than from the build tree.
I did this, and I can run the svn-mergeinfo-normalizer tool now without
setting PATH or LD_LIBRARY_PATH to my prefix directory. However, when I
run the tool on an existing sandbox, I get the following error:

Scanning working copy /scratch/sandbox/XXX//trunk ...
svn: E170000: Unrecognized URL scheme for 'https://svn.xxx.com/hepd/XXX/trunk'

I thought perhaps I needed to check out a fresh sandbox with the newly
build 1.10 svn binaries, but I get the same error. I guess I need to
pass some additional options to ./configure?
Post by Daniel Shahaf
- (included for completeness) Temporarily move /usr/lib/libsvn*.so out
of the way while testing.
I didn’t want to try this.
Daniel Shahaf
2017-06-06 20:10:21 UTC
Permalink
Post by Alfred von Campe
Post by Daniel Shahaf
- Pass --prefix to configure, then run 'make install-tools', and run the
binary from ${prefix}/bin/svn-tools/ rather than from the build tree.
I did this, and I can run the svn-mergeinfo-normalizer tool now without
setting PATH or LD_LIBRARY_PATH to my prefix directory. However, when I
Scanning working copy /scratch/sandbox/XXX//trunk ...
svn: E170000: Unrecognized URL scheme for 'https://svn.xxx.com/hepd/XXX/trunk'
I thought perhaps I needed to check out a fresh sandbox with the newly
build 1.10 svn binaries, but I get the same error. I guess I need to
pass some additional options to ./configure?
Yes: --with-serf=/usr

You'll need to install a (sufficiently recent version of) serf-devel.

Daniel
Alfred von Campe
2017-06-07 00:57:29 UTC
Permalink
Post by Daniel Shahaf
Yes: --with-serf=/usr
You'll need to install a (sufficiently recent version of) serf-devel.
Yes, that worked even on CentOS 6 -- thanks! I was able to build the tool
and it cleaned up the svn:mergeinfo property exactly as I had expected.

I can also now build the Subversion binaries from source, which may come
in handy some day.

Thanks again for all your help,
Alfred

Thomas Wicklund
2017-06-03 01:57:35 UTC
Permalink
Post by Alfred von Campe
After many years of development, we have over 100 entries in the
svn:mergeinfo property of our trunk. I am thinking about manually
editing the property to remove entries for all branches that have been
deleted or are no longer used (and should be deleted). I am hoping
that once I merge trunk into the other active branches, the cleanup of
that property will also propagate to the other branches.
* Is this safe to do?
* Will the changes to the properly be merged to all other branches
over time as one would expect?
Thanks,
Alfred
I have done this in the past. mergeinfo got so large it was visibly
hurting performance. There may have also been svn command crashes but
it's been several years. I just deleted the mergeinfo property, starting
over.

If you are basically merging from branches back to a development trunk
the lack of mergeinfo shouldn't hurt. Do watch that the next time you
merge a branch into the trunk it doesn't merge the whole huge mergeinfo
back into the branch.

Personally I've never had much use for mergeinfo. Since the development
model I work with is to create a branch, make changes, then commit the
branch changes into the trunk, I generally revert any mergeinfo change
during the merge before committing the change to the trunk since the
fact something was done in a branch (which won't be reused) isn't useful
information, each change looks like it was developed and committed onto
the trunk.

Tom
Loading...