Checkout/Update issues with Subversion 1.8 server and client via HTTP

Discussion:

Clay Porter

2015-05-12 16:27:25 UTC

We recently upgraded our Subversion server from 1.6 to 1.8 and since
then some of our users
our reporting problems checking out certain folders using a 1.8
client. The issues occur whether using TortoiseSVN 1.8.x or any 1.8
CLI client on Windows/Cygwin/Linux.

We have verified that these operations work with 1.4, 1.6 and 1.7 svn clients.

The errors we are seeing during a clean checkout from the client side
look like this depending
on the folder checked out:

svn: E175002: GET request failed: 400 Bad request
There is no corresponding error in the Apache HTTP logs.

svn: E200014: Checksum mismatch for <file>
expected: 4b4ef9e3432aa84aed190457b68c01ad
actual: 863b9f52f352a5cb20298ef0eecb9e97
In this case the server logs have this:
[Tue May 12 12:13:05 2015] [error] [client 153.65.184.225] Unable to
deliver content. [500, #0]
[Tue May 12 12:13:05 2015] [error] [client 153.65.184.225] Could not
write data to filter. [500, #175002]

The checksum mismatch error can be worked around by going to the root
of the WC and
doing:

svn update --set-depth empty
svn update --set-depth infinity

but it would be nice to have this fixed without having to jump through
these hoops.

Downgrading subversion to 1.6 fixes the issue, but without going into
too many details we will need to upgrade to 1.8 eventually.

I've done some searching and although I've seen these issues posted, I
haven't seen any solution that fixes our problem, including setting
http-chunked-requests = no.

Our environment looks like this:

OS: SLES 11 SP3
Subversion: 1.8.13
Server version: Apache/2.2.12 (Linux/SUSE)
APR 1.3.3, APR-Util 1.3.4

I would appreciate any feedback, even if it's something like "Hey
dumbass, did you try X?"

I can also provide abridged config files as well or any more
information need to fix the problem.

Thanks.

Clay Porter

Andreas Stieger

2015-05-12 21:21:57 UTC

Permalink

Hello,

Post by Clay Porter
svn: E175002: GET request failed: 400 Bad request
There is no corresponding error in the Apache HTTP logs.
svn: E200014: Checksum mismatch for <file>
expected: 4b4ef9e3432aa84aed190457b68c01ad
actual: 863b9f52f352a5cb20298ef0eecb9e97
[Tue May 12 12:13:05 2015] [error] [client 153.65.184.225] Unable to
deliver content. [500, #0]
[Tue May 12 12:13:05 2015] [error] [client 153.65.184.225] Could not
write data to filter. [500, #175002]

Does setting/changing SVNAllowBulkUpdates make any difference?
https://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default

Post by Clay Porter
OS: SLES 11 SP3
Subversion: 1.8.13
Server version: Apache/2.2.12 (Linux/SUSE)
APR 1.3.3, APR-Util 1.3.4

Are these the binaries from the OBS at devel:tools:scm:svn/subversion?

Andreas

Clay Porter

2015-05-13 19:00:28 UTC

Permalink

Hello,

Everything is working now (see inline below). I can't thank you enough
for your help with this.

Post by Andreas Stieger
Hello,

Does setting/changing SVNAllowBulkUpdates make any difference?
https://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default

Setting SVNAllowBulkUpdates to Prefer has fixed the problem for me.
I checked the behavior using SVN clients 1.5 through 1.8 on various
platforms and all worked.

Post by Andreas Stieger

Post by Clay Porter
OS: SLES 11 SP3
Subversion: 1.8.13
Server version: Apache/2.2.12 (Linux/SUSE)
APR 1.3.3, APR-Util 1.3.4

Are these the binaries from the OBS at devel:tools:scm:svn/subversion?

I don't know what the OBS is...

I added the repo from here:

http://software.opensuse.org/download.html?project=devel:tools:scm:svn&package=subversion

and used zypper to install the packages.

I used the SLE 11 SP3 section on that link.

Thanks again for your help!

Post by Andreas Stieger
Andreas

Clay

Johan Corveleyn

2015-05-14 17:47:57 UTC

Permalink

Post by Andreas Stieger
Hello,
Everything is working now (see inline below). I can't thank you enough
for your help with this.

Post by Andreas Stieger
Hello,

Does setting/changing SVNAllowBulkUpdates make any difference?
https://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default

Setting SVNAllowBulkUpdates to Prefer has fixed the problem for me.
I checked the behavior using SVN clients 1.5 through 1.8 on various
platforms and all worked.

I'm glad you got it fixed. However, it's not normal that serf's skelta
mode (the default for 1.8 clients, unless they get instructed by the
server with "SVNAllowBulkUpdates Prefer") doesn't work. If you have
the time it might be interesting to investigate this a little bit
further.

You should be able to reproduce the issue again, even when your server
is configured with "SVNAllowBulkUpdates Prefer", by using a client
with http-bulk-updates=no in its "servers" configuration file (see the
release notes snippet that Andreas pointed to). This will instruct
your client to never use bulk-updates mode (i.e. force skelta mode),
even though the server prefers it.

Some possible reasons for the error you were seeing include problems
with proxies or firewalls between client and server. Or other
components that interfere with the network communication, rewriting
packets or things like that (e.g. antivirus components, surf shield
stuff, ...). Skelta mode uses a lot more small requests/responses
(instead of one huge "update response" for the bulk-updates mode), so
maybe some security component on your network considers this an
insecure pattern, and decides to drop things ...

--
Johan

Branko Čibej

2015-05-14 18:08:01 UTC

Permalink

Post by Johan Corveleyn

Post by Andreas Stieger
Hello,
Everything is working now (see inline below). I can't thank you enough
for your help with this.

Post by Andreas Stieger
Hello,

Does setting/changing SVNAllowBulkUpdates make any difference?
https://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default

Setting SVNAllowBulkUpdates to Prefer has fixed the problem for me.
I checked the behavior using SVN clients 1.5 through 1.8 on various
platforms and all worked.

I'm glad you got it fixed. However, it's not normal that serf's skelta
mode (the default for 1.8 clients, unless they get instructed by the
server with "SVNAllowBulkUpdates Prefer") doesn't work. If you have
the time it might be interesting to investigate this a little bit
further.
You should be able to reproduce the issue again, even when your server
is configured with "SVNAllowBulkUpdates Prefer", by using a client
with http-bulk-updates=no in its "servers" configuration file (see the
release notes snippet that Andreas pointed to). This will instruct
your client to never use bulk-updates mode (i.e. force skelta mode),
even though the server prefers it.
Some possible reasons for the error you were seeing include problems
with proxies or firewalls between client and server. Or other
components that interfere with the network communication, rewriting
packets or things like that (e.g. antivirus components, surf shield
stuff, ...). Skelta mode uses a lot more small requests/responses
(instead of one huge "update response" for the bulk-updates mode), so
maybe some security component on your network considers this an
insecure pattern, and decides to drop things ...

We had a similar report a while ago ... which turned out to be a bug in
a Cisco ASA/IOS PIX, which had HTTP deep packet inspection enabled.
Turns out that once its internal queue of packets is full, it'll just
start dropping them. A browser user (almost) doesn't care, since she
just reloads the page and hopes for the best blaming "flaky internet
connection". The SVN client, on the other hand, isn't so forgiving. :)

-- Brane

Clay Porter

2015-05-14 20:44:18 UTC

Permalink

Post by Johan Corveleyn

Post by Andreas Stieger
Hello,
Everything is working now (see inline below). I can't thank you enough
for your help with this.

Post by Andreas Stieger
Hello,

Does setting/changing SVNAllowBulkUpdates make any difference?
https://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default

Setting SVNAllowBulkUpdates to Prefer has fixed the problem for me.
I checked the behavior using SVN clients 1.5 through 1.8 on various
platforms and all worked.

I have a test SVN server that I can use so I can reproduce the problem at
will but I don't know what else to do to debug the issue. There are no
errors in the web server log that indicate the problem and of course the
only errors I see on the client side are the ones I put in the original
post. I'd be glad to spend some time looking into the issue but I don't
know what else to look at.

Post by Johan Corveleyn
Some possible reasons for the error you were seeing include problems
with proxies or firewalls between client and server. Or other
components that interfere with the network communication, rewriting
packets or things like that (e.g. antivirus components, surf shield
stuff, ...). Skelta mode uses a lot more small requests/responses
(instead of one huge "update response" for the bulk-updates mode), so
maybe some security component on your network considers this an
insecure pattern, and decides to drop things ...

AFAIK, there are no proxy settings or firewalls between the client and
the server as this is all LAN traffic. There is antivirus and firewall
on the Windows clients but none of these exist on the Linux clients so
I don't see how it could be related to that.

I've thought about setting up another VM with a different OS and
SVN implementation, like those from Collabnet or WANDisco, but that
would require more time. I'm also open to other ways to proceed in
troubleshooting the problem.

Clay

Post by Johan Corveleyn
--
Johan