issue with relative externals after a rename

[Jonathan Schell]

I have a tree where one folder is referencing some other folders and documents in different parts of the tree. And those externals are pegged.

So, I did a rename of the top folder of the tree. Which of course breaks the externals.

So, I go fix the externals to be like they should, using the operative revision. This works for the files, but not for the folders.

Am I missing something or is this really a bug as it appears to be? Or is it maybe an issue with TortoiseSVN?

Jon

[Ryan Schmidt]

Post by Jonathan Schell
I have a tree where one folder is referencing some other folders and documents in different parts of the tree. And those externals are pegged.
So, I did a rename of the top folder of the tree. Which of course breaks the externals.
So, I go fix the externals to be like they should, using the operative revision. This works for the files, but not for the folders.

In what way does it not work? What did you do, what happened, what was supposed to happen instead?

Post by Jonathan Schell
Am I missing something or is this really a bug as it appears to be? Or is it maybe an issue with TortoiseSVN?

[Johan Corveleyn]

On Tue, Mar 27, 2018 at 10:46 AM, Ryan Schmidt

Post by Ryan Schmidt

Post by Jonathan Schell
I have a tree where one folder is referencing some other folders and documents in different parts of the tree. And those externals are pegged.
So, I did a rename of the top folder of the tree. Which of course breaks the externals.
So, I go fix the externals to be like they should, using the operative revision. This works for the files, but not for the folders.

In what way does it not work? What did you do, what happened, what was supposed to happen instead?

Also, to make sure we're on the same page:
Are you using the following syntax (relative and with peg revisions)
in the externals property?

../../***@1234 somedir
../../***@3214 somefile

Or are you using this syntax (with operative revisions instead of peg
revisions)?

-r1234 ../../somedir somedir
-r3214 ../../somefile somefile

Or perhaps a mix of both?

[Jonathan Schell]

I have a tree that looks like:

https://svn/repo/name1/externs/normal_files_here

The folder "externs" has two external properties:
../***@18 fold1
../fold2/***@18 file1.txt

These folders and file exist at rev 18. The commit to add the properties to "externs" is commit 32.

I then do a rename of "name1" to "name2".

The externs now break, as the item they were pointing to no longer exists. So I edit the properties to:
-r 18 ../fold1 fold1
-r 18 ../fold2/file1.txt file1.txt

And then did an update and the file got brought in, but there was still an error on the folder.

However, I just repeated the issue on a test repo and it worked fine for both the folder and the file, so maybe there's some other condition that's affecting it.

Jon

-----Original Message-----
From: Johan Corveleyn [mailto:***@gmail.com]
Sent: Tuesday, March 27, 2018 1:49 AM
To: Jonathan Schell <***@innoflight.com>
Cc: Subversion Users <***@subversion.apache.org>; Ryan Schmidt <subversion-***@ryandesign.com>
Subject: Re: issue with relative externals after a rename

Post by Ryan Schmidt

Post by Jonathan Schell
I have a tree where one folder is referencing some other folders and documents in different parts of the tree. And those externals are pegged.
So, I did a rename of the top folder of the tree. Which of course breaks the externals.
So, I go fix the externals to be like they should, using the operative revision. This works for the files, but not for the folders.

In what way does it not work? What did you do, what happened, what was supposed to happen instead?

Also, to make sure we're on the same page:
Are you using the following syntax (relative and with peg revisions) in the externals property?

../../***@1234 somedir
../../***@3214 somefile

Or are you using this syntax (with operative revisions instead of peg revisions)?

-r1234 ../../somedir somedir
-r3214 ../../somefile somefile

[Branko Čibej]

Post by Jonathan Schell
https://svn/repo/name1/externs/normal_files_here
These folders and file exist at rev 18. The commit to add the properties to "externs" is commit 32.
I then do a rename of "name1" to "name2".
The externs now break, as the item they were pointing to no longer exists.

Because the default operative revision is HEAD and there's no such
object in HEAD, yes.

Post by Jonathan Schell
-r 18 ../fold1 fold1
-r 18 ../fold2/file1.txt file1.txt
And then did an update and the file got brought in, but there was still an error on the folder.

You need both peg and operative revisions:

-r 18 ../fold1 ***@18
-r 18 ../fold2/***@18 file1.txt

The peg revision tells Subversion which object to look for and the
operative revision tells it which version of that object to use.

-- Brane

[Johan Corveleyn]

Post by Branko Čibej

Post by Jonathan Schell
https://svn/repo/name1/externs/normal_files_here
These folders and file exist at rev 18. The commit to add the properties to "externs" is commit 32.
I then do a rename of "name1" to "name2".
The externs now break, as the item they were pointing to no longer exists.

Because the default operative revision is HEAD and there's no such
object in HEAD, yes.

Are you sure? That's surprising to me, because for normal commands the
operative revision always defaults to the peg revision.

For example, in [1] I read "Notice that we didn't provide an operative
revision this time. That's because when no operative revision is
specified, Subversion assumes a default operative revision that's the
same as the peg revision."

Or is there a different rule for externals definitions?

[1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html

[Nathan Hartman]

Post by Branko Čibej
Because the default operative revision is HEAD and there's no such
object in HEAD, yes.

[snip]

Post by Branko Čibej
The peg revision tells Subversion which object to look for and the
operative revision tells it which version of that object to use.

And then...

Post by Branko Čibej
Are you sure? That's surprising to me, because for normal commands the
operative revision always defaults to the peg revision.
For example, in [1] I read "Notice that we didn't provide an operative
revision this time. That's because when no operative revision is
specified, Subversion assumes a default operative revision that's the
same as the peg revision."
Or is there a different rule for externals definitions?
[1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html

Oops, that makes me a bit nervous about how we've set up our
externals.

Our requirements are: at any time in the future, if someone checks
out code from the past, they should get exactly the same tree as what
existed in the past. I assume that this is probably THE #1 use case
and desired behavior for externals. Is that correct? If not, is there
a good reason to want otherwise?

My understanding is that:

(1) ^/SVN/path/to/project/tags/specific_tag

That will probably work most of the time but if someone commits
to that tag, future checkouts of past revisions will not look
exactly as they did in the past.

(2) -r [number] ^/SVN/path/to/project/tags/specific_tag

Subversion first goes to that path, then follows it back in time
to the specified revision.

So if someone commits to that tag, the future checkout of the
past revision will get the past revision.

But... if a tag is ever moved, removed, or renamed in the future,
(the project is declared defunct and removed, for example), what
Subversion finds at that path and tries to follow back in time
won't lead to the right place.

(3) ^/SVN/path/to/project/tags/specific_tag@[number]

Subversion first goes back in time to that revision, then goes to
that path.

This is where I'm a bit foggy: Once it goes back in time and goes
to that path, does it then follow it forward to HEAD? Or does it
stay in the past?

We've set up all of our externals like (3)... Do we need to go
through our repo and change all externals to specify both the peg
and operative revision?

Thanks,
Nathan Hartman

[Nico Kadel-Garcia]

On Wed, Mar 28, 2018 at 10:36 AM, Nathan Hartman

Post by Nathan Hartman
Our requirements are: at any time in the future, if someone checks
out code from the past, they should get exactly the same tree as what
existed in the past. I assume that this is probably THE #1 use case
and desired behavior for externals. Is that correct? If not, is there
a good reason to want otherwise?

Then make tags. Tags need to be locked down, with no commits permitted
on top of the tag.

Post by Nathan Hartman
(1) ^/SVN/path/to/project/tags/specific_tag
That will probably work most of the time but if someone commits
to that tag, future checkouts of past revisions will not look
exactly as they did in the past.

See above. "No commits permitted on top of the tag".

Post by Nathan Hartman
(2) -r [number] ^/SVN/path/to/project/tags/specific_tag
Subversion first goes to that path, then follows it back in time
to the specified revision.
So if someone commits to that tag, the future checkout of the
past revision will get the past revision.
But... if a tag is ever moved, removed, or renamed in the future,
(the project is declared defunct and removed, for example), what
Subversion finds at that path and tries to follow back in time
won't lead to the right place.
Subversion first goes back in time to that revision, then goes to
that path.
This is where I'm a bit foggy: Once it goes back in time and goes
to that path, does it then follow it forward to HEAD? Or does it
stay in the past?
We've set up all of our externals like (3)... Do we need to go
through our repo and change all externals to specify both the peg
and operative revision?
Thanks,
Nathan Hartman

If you're permitting commits on top of tags, probably so.

[Nathan Hartman]

Post by Nico Kadel-Garcia
On Wed, Mar 28, 2018 at 10:36 AM, Nathan Hartman

Post by Nathan Hartman
Our requirements are: at any time in the future, if someone checks
out code from the past, they should get exactly the same tree as what
existed in the past. I assume that this is probably THE #1 use case
and desired behavior for externals. Is that correct? If not, is there
a good reason to want otherwise?

Then make tags. Tags need to be locked down, with no commits permitted
on top of the tag.

[snip]

Post by Nico Kadel-Garcia

Post by Nathan Hartman
Subversion first goes back in time to that revision, then goes to
that path.
This is where I'm a bit foggy: Once it goes back in time and goes
to that path, does it then follow it forward to HEAD? Or does it
stay in the past?
We've set up all of our externals like (3)... Do we need to go
through our repo and change all externals to specify both the peg
and operative revision?

If you're permitting commits on top of tags, probably so.

Yup.

Right now tags are not locked down. Perhaps they should be. But we
will begin with specifying both the peg and operative revisions and
then go from there.

Thank you for your input. Thanks also to Branko and thanks to the
OP for asking the original question, without which we would not have
discovered this about our repo until after things broke later on. :-)

[Johan Corveleyn]

On Wed, Mar 28, 2018 at 4:36 PM, Nathan Hartman

Post by Nathan Hartman

Post by Branko Čibej
Because the default operative revision is HEAD and there's no such
object in HEAD, yes.

[snip]

Post by Branko Čibej
The peg revision tells Subversion which object to look for and the
operative revision tells it which version of that object to use.

And then...

Post by Branko Čibej
Are you sure? That's surprising to me, because for normal commands the
operative revision always defaults to the peg revision.
For example, in [1] I read "Notice that we didn't provide an operative
revision this time. That's because when no operative revision is
specified, Subversion assumes a default operative revision that's the
same as the peg revision."
Or is there a different rule for externals definitions?
[1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html

Oops, that makes me a bit nervous about how we've set up our
externals.
Our requirements are: at any time in the future, if someone checks
out code from the past, they should get exactly the same tree as what
existed in the past. I assume that this is probably THE #1 use case
and desired behavior for externals. Is that correct? If not, is there
a good reason to want otherwise?
(1) ^/SVN/path/to/project/tags/specific_tag
That will probably work most of the time but if someone commits
to that tag, future checkouts of past revisions will not look
exactly as they did in the past.
(2) -r [number] ^/SVN/path/to/project/tags/specific_tag
Subversion first goes to that path, then follows it back in time
to the specified revision.
So if someone commits to that tag, the future checkout of the
past revision will get the past revision.
But... if a tag is ever moved, removed, or renamed in the future,
(the project is declared defunct and removed, for example), what
Subversion finds at that path and tries to follow back in time
won't lead to the right place.
Subversion first goes back in time to that revision, then goes to
that path.
This is where I'm a bit foggy: Once it goes back in time and goes
to that path, does it then follow it forward to HEAD? Or does it
stay in the past?
We've set up all of our externals like (3)... Do we need to go
through our repo and change all externals to specify both the peg
and operative revision?

No, you do not need to change your externals. I believe your
understanding is correct, and I think Branko was incorrect. Indeed, in
the absence of an operative revision, the operative revision defaults
to the peg revision (so it does not follow it forward to HEAD), also
in externals definitions. The peg revision is sufficient to "pin" the
externals. I just tested this with a test repository with svn 1.9.4.

However, the problem Jonathan saw is, I think, because he renamed a
parent path that is outside of the scope of the relative path of the

Post by Nathan Hartman
https://svn/repo/name1/externs/normal_files_here
These folders and file exist at rev 18. The commit to add the properties to "externs" is commit 32.
I then do a rename of "name1" to "name2".
The externs now break, as the item they were pointing to no longer exists.

I think the externals processing sees "../***@18", and apparently it
first converts the relative path into an absolute path by looking at
HEAD ... so "../fold1" becomes "/name2/fold1". Then it sees the peg
revision, so it wants "/name2/***@18", which does not exist (at the
time of @18, the parent folder was name1). Adding an operative
revision "-r18" does not help.

I'm not sure if this behaviour is "as designed", or whether it should
be considered a bug. I'm inclined to think it's a bug. The "relative
url -> absolute url" conversion should probably also take the peg
revision into account.

Here below is a transcript of the tests I did with my 1.9.4 client,
confirming what I wrote above (with some comments prefixed with ###):

[[[
C:\svntest>svnadmin create repos

C:\svntest>svn co file:///C:/svntest/repos wc
Checked out revision 0.

C:\svntest>cd wc

C:\svntest\wc>mkdir name1\externs name1\fold1 name1\fold2

C:\svntest\wc>echo test > name1\externs\normal_file.txt

C:\svntest\wc>echo test > name1\fold1\file_in_fold1.txt

C:\svntest\wc>echo test > name1\fold2\file_in_fold2.txt

C:\svntest\wc>svn add name1
A name1
A name1\externs
A name1\externs\normal_file.txt
A name1\fold1
A name1\fold1\file_in_fold1.txt
A name1\fold2
A name1\fold2\file_in_fold2.txt

C:\svntest\wc>svn ci -m"initial commit"
Adding name1
Adding name1\externs
Adding name1\externs\normal_file.txt
Adding name1\fold1
Adding name1\fold1\file_in_fold1.txt
Adding name1\fold2
Adding name1\fold2\file_in_fold2.txt
Transmitting file data ...done
Committing transaction...
Committed revision 1.

C:\svntest\wc>cd name1\externs

C:\svntest\wc\name1\externs>svn pe svn:externals .
Set new value for property 'svn:externals' on '.'

### Contents of svn:externals:
### ../***@1 fold1
### ../fold2/***@1 file_in_fold2.txt

C:\svntest\wc\name1\externs>svn ci -m"add pegged externals definition"
Sending .
Committing transaction...
Committed revision 2.

C:\svntest\wc\name1\externs>svn up
Updating '.':

Fetching external item into 'fold1':
A fold1\file_in_fold1.txt
Updated external to revision 1.


Fetching external item into 'file_in_fold2.txt':
A file_in_fold2.txt
Updated external to revision 1.

At revision 2.

C:\svntest\wc\name1\externs>dir
Volume in drive C has no label.
Volume Serial Number is 889A-20D9

Directory of C:\svntest\wc\name1\externs

29/03/2018 01:01 <DIR> .
29/03/2018 01:01 <DIR> ..
29/03/2018 01:01 7 file_in_fold2.txt
29/03/2018 01:01 <DIR> fold1
29/03/2018 00:53 7 normal_file.txt
2 File(s) 14 bytes
3 Dir(s) 272,449,265,664 bytes free

C:\svntest\wc\name1\externs>cd ..\..

C:\svntest\wc>svn up name1
Updating 'name1':

Fetching external item into 'name1\externs\fold1':
External at revision 1.


Fetching external item into 'name1\externs\file_in_fold2.txt':
External at revision 1.

At revision 2.

C:\svntest\wc>svn mv name1 name2
A name2
D name1
D name1\externs
D name1\externs\normal_file.txt
D name1\fold1
D name1\fold1\file_in_fold1.txt
D name1\fold2
D name1\fold2\file_in_fold2.txt

C:\svntest\wc>svn ci -m"move name1 to name2"
Deleting name1
Adding name2
Committing transaction...
Committed revision 3.

### This is the problem Jonathan saw:

C:\svntest\wc>svn up
Updating '.':
svn: warning: W205011: Error handling externals definition for
'name2\externs\fold1':
svn: warning: W170000: URL 'file:///C:/svntest/repos/name2/fold1' at
revision 1 doesn't exist
svn: warning: W205011: Error handling externals definition for
'name2\externs\file_in_fold2.txt':
svn: warning: W170000: URL
'file:///C:/svntest/repos/name2/fold2/file_in_fold2.txt' at revision 1
doesn't exist
At revision 3.
svn: E205011: Failure occurred processing one or more externals definitions

C:\svntest\wc>cd name2\externs

C:\svntest\wc\name2\externs>svn pe svn:externals .
Set new value for property 'svn:externals' on '.'

### Contents of svn:externals, adding -r1 operative revisions:
### -r1 ../***@1 fold1
### -r1 ../fold2/***@1 file_in_fold2.txt

C:\svntest\wc\name2\externs>svn ci -m"add operative rev to externals"
Sending .
Committing transaction...
Committed revision 4.

### Still the same problem, operative rev doesn't help:

C:\svntest\wc\name2\externs>svn up
Updating '.':
svn: warning: W205011: Error handling externals definition for 'fold1':
svn: warning: W170000: URL 'file:///C:/svntest/repos/name2/fold1' at
revision 1 doesn't exist
svn: warning: W205011: Error handling externals definition for
'file_in_fold2.txt':
svn: warning: W170000: URL
'file:///C:/svntest/repos/name2/fold2/file_in_fold2.txt' at revision 1
doesn't exist
At revision 4.
svn: E205011: Failure occurred processing one or more externals definitions

C:\svntest\wc\name2\externs>svn pe svn:externals .
Set new value for property 'svn:externals' on '.'

### Contents of svn:externals, including name1 in the relative path:
### ../../name1/***@1 fold1
### ../../name1/fold2/***@1 file_in_fold2.txt

C:\svntest\wc\name2\externs>svn ci -m"no operative rev, but include
name1 in the path"
Sending .
Committing transaction...
Committed revision 5.

C:\svntest\wc\name2\externs>svn up
Updating '.':

Fetching external item into 'fold1':
External at revision 1.


Fetching external item into 'file_in_fold2.txt':
A file_in_fold2.txt
Updated external to revision 1.

At revision 5.

### Change content of file_in_fold2.txt, to see if it impacts the
pegged external

C:\svntest\wc\name2\externs>cd ..\fold2

C:\svntest\wc\name2\fold2>echo blabla>file_in_fold2.txt

C:\svntest\wc\name2\fold2>svn ci -m"change file_in_fold2.txt"
Sending file_in_fold2.txt
Transmitting file data .done
Committing transaction...
Committed revision 6.

C:\svntest\wc\name2\fold2>cd ..\..

C:\svntest\wc>svn up
Updating '.':

Fetching external item into 'name2\externs\fold1':
External at revision 1.


Fetching external item into 'name2\externs\file_in_fold2.txt':
External at revision 1.

At revision 6.

### The externals content is still at the content of the pegged revision.

C:\svntest\wc>type name2\externs\file_in_fold2.txt
test
]]]

[Nathan Hartman]

Post by Johan Corveleyn
No, you do not need to change your externals. I believe your
understanding is correct, and I think Branko was incorrect. Indeed, in
the absence of an operative revision, the operative revision defaults
to the peg revision (so it does not follow it forward to HEAD), also
in externals definitions. The peg revision is sufficient to "pin" the
externals. I just tested this with a test repository with svn 1.9.4.

Thank you. That is reassuring. We have not
changed anything yet.

However, the problem Jonathan saw is, I think, because he renamed a

Post by Johan Corveleyn
parent path that is outside of the scope of the relative path of the

Yes that makes sense.

Fortunately for us, all of our externals begin
with "^/SVN/" so we are unaffected by the
relative directory question. I think that when we
designed our monorepo we thought that we
may want to restructure in the future, e.g.
move or rename project directories, combine
projects, split projects, declare projects
defunct, etc. So we did not rely on the relative
placement of projects to each other, but we did
use the caret notation without a URL scheme
to ensure the repository itself is portable. So,
absolute paths within the repository with peg
revisions, relative path to the repository itself.

Post by Johan Corveleyn
first converts the relative path into an absolute path by looking at
HEAD ... so "../fold1" becomes "/name2/fold1". Then it sees the peg
revision "-r18" does not help.
I'm not sure if this behaviour is "as designed", or whether it should
be considered a bug. I'm inclined to think it's a bug. The "relative
url -> absolute url" conversion should probably also take the peg
revision into account.

It makes intuitive sense that ../***@18 should
Do The Right Thing. But if it takes "." and expands it to a full path, then
strips off the last
component, then applies the peg revision, that
will explain the source of the issue.

Part of the problem is trying to locate
something in space and time starting from a
leaf rather than the root. The leaf moves
around; the root stays put.

With the peg revision you usually know the full
path that existed at some point in the past, so
Subversion can pull up that revision and
traverse down the tree to the correct location.
But in this case you're starting with a leaf in
some revision so you have to take "." and
expand it to the full path in that revision, then
traverse its history back to the peg revision to
figure out what its full path was back then, and
then go on to strip the last component etc.

The problem is, what revision do you use to
expand "." to the full path? HEAD? BASE? The
way I see it, if it's HEAD and you check out an
old revision, and things have moved around
some more since then, I think you'll encounter
the same kind of breakage.

Nathan Hartman

[Bo Berglund]

Just curious since I am working on setting up our svn repositories migrated from CVS.How does one "lock down" tags to disallow further commits? In CVS that was built-in but svn works differently...
Bo B.

[David Chapman]

Post by Bo Berglund
Just curious since I am working on setting up our svn repositories
migrated from CVS.
How does one "lock down" tags to disallow further commits? In CVS that
was built-in but svn works differently...
Bo B.

Put a check for the commit path(s) in the pre-commit hook.  If a tag
directory is in the commit list, return an error.

http://svnbook.red-bean.com/nightly/en/svn.ref.reposhooks.pre-commit.html