Subversion svn+ssh, sshd 100% CPU

[Zoran Petkovic]

In the past few days I have been doing extensive testing of Subversion with different clients, operating systems, client and server versions and have noticed very strange behaviour with windows clients connecting to Linux servers, hitting them with excessive CPU usage on the sshd process, where the Linux clients do not exhibit this behaviour.

A sample test setup is as follows:
Server Linux Ubuntu 16.04.3 LTS, OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016, Subversion version 1.9.3 (and 1.9.7).
Client TortoiseSVN 1.9.7

When checking out large repositories the linux server is hit on the sshd process, the process running with 100% cpu usage. This in effect slows down the performance and ultimately the speed at which the checkout runs. Linux clients connecting to the same server do not cause this load on the server.

This happens even when compressions is turned off and when encryption Cyphers are changed, as well as different versions of subversion. The behaviour is identical. I'm not sure who to address for this issue as this not only happens with TortoiseSVN but with SlikSVN as well. Any direction would be appreciated.

Regards,

[Matt Simmons]

Why does iostat show? Could it be that your underlying disk is io-saturated
and your CPU spike is due to iowait?

Post by Zoran Petkovic
In the past few days I have been doing extensive testing of Subversion
with different clients, operating systems, client and server versions and
have noticed very strange behaviour with windows clients connecting to
Linux servers, hitting them with excessive CPU usage on the sshd process,
where the Linux clients do not exhibit this behaviour.
Server Linux Ubuntu 16.04.3 LTS, OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL
1.0.2g 1 Mar 2016, Subversion version 1.9.3 (and 1.9.7).
Client TortoiseSVN 1.9.7
When checking out large repositories the linux server is hit on the sshd
process, the process running with 100% cpu usage. This in effect slows down
the performance and ultimately the speed at which the checkout runs. Linux
clients connecting to the same server do not cause this load on the server.
This happens even when compressions is turned off and when encryption
Cyphers are changed, as well as different versions of subversion. The
behaviour is identical. I'm not sure who to address for this issue as this
not only happens with TortoiseSVN but with SlikSVN as well. Any direction
would be appreciated.
Regards,
--
Zoran Petkovic

[Johan Corveleyn]

Post by Matt Simmons

Post by Zoran Petkovic
In the past few days I have been doing extensive testing of Subversion
with different clients, operating systems, client and server versions and
have noticed very strange behaviour with windows clients connecting to Linux
servers, hitting them with excessive CPU usage on the sshd process, where
the Linux clients do not exhibit this behaviour.
Server Linux Ubuntu 16.04.3 LTS, OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL
1.0.2g 1 Mar 2016, Subversion version 1.9.3 (and 1.9.7).
Client TortoiseSVN 1.9.7
When checking out large repositories the linux server is hit on the sshd
process, the process running with 100% cpu usage. This in effect slows down
the performance and ultimately the speed at which the checkout runs. Linux
clients connecting to the same server do not cause this load on the server.
This happens even when compressions is turned off and when encryption
Cyphers are changed, as well as different versions of subversion. The
behaviour is identical. I'm not sure who to address for this issue as this
not only happens with TortoiseSVN but with SlikSVN as well. Any direction
would be appreciated.

Why does iostat show? Could it be that your underlying disk is io-saturated
and your CPU spike is due to iowait?

One possible explanation would be that the windows clients perform the
checkout faster than the linux clients for some reason, thus being
able to put higher load on the server, saturating its cpu or disk I/O.

Other than that I have no idea.

The Linux client you're testing is also 1.9.7?

Perhaps the windows client and the linux client are performing the
checkout in a different way for some reason (I'm thinking of the
so-called "skelta mode" vs. "bulk mode", see
http://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default).

[Bert Huijben]

-----Original Message-----
Sent: donderdag 14 september 2017 22:10
Subject: Re: Subversion svn+ssh, sshd 100% CPU

On Tue, Sep 12, 2017 at 12:14 PM Zoran Petkovic

Perhaps the windows client and the linux client are performing the checkout
in a different way for some reason (I'm thinking of the so-called "skelta
mode" vs. "bulk mode", see http://subversion.apache.org/docs/release-
notes/1.8.html#serf-skelta-default).

'Skelta vs bulk' is specific to operations over http(s), so this wouldn't apply here. The relevant part here is probably which ssh client is used here. On most *nix systems this would be openssh, while on most Windows systems I would see some putty based tools.

Bert

--
Johan

[Johan Corveleyn]

Forwarding to the list (Zoran, please use reply all to keep the list in cc).


---------- Doorgestuurd bericht ----------
Van: "Zoran Petkovic" <***@ietcombustion.com>
Datum: 21 sep. 2017 19:01
Onderwerp: Re: Subversion svn+ssh, sshd 100% CPU
Aan: "Johan Corveleyn" <***@gmail.com>
Cc:

Correction, I meant to say KBytes/s.

--
Zoran Petkovic
Industrial Engineering Technology Pty Ltd
[phone] <tel:0413%20254%20315> | ***@ietcombustion.com


On 21/09/2017, 4:17 PM, "Zoran Petkovic" <***@ietcombustion.com> wrote:

My results show that Linux clients stress the server less and complete
the checkout quicker (5-10% on sshd and svnserve). Windows clients
connecting remotely start off fast 2000-3000kbit/s then slow down to
maximum 800kbit/s, the sshd process running at 100%, while the svnserve at
5%. So, I believe it is an issue with ssh rather than svn. Skelta mode
seems to refer to apache, however these clients connect over ssh.

--
Zoran Petkovic
Industrial Engineering Technology Pty Ltd

On Tue, Sep 12, 2017 at 12:14 PM Zoran Petkovic <

Post by Zoran Petkovic
In the past few days I have been doing extensive testing of

Subversion

Post by Zoran Petkovic
with different clients, operating systems, client and server

versions and

Post by Zoran Petkovic
have noticed very strange behaviour with windows clients

connecting to Linux

Post by Zoran Petkovic
servers, hitting them with excessive CPU usage on the sshd

process, where

Post by Zoran Petkovic
the Linux clients do not exhibit this behaviour.
Server Linux Ubuntu 16.04.3 LTS, OpenSSH_7.2p2

Ubuntu-4ubuntu2.2, OpenSSL

Post by Zoran Petkovic
1.0.2g 1 Mar 2016, Subversion version 1.9.3 (and 1.9.7).
Client TortoiseSVN 1.9.7
When checking out large repositories the linux server is hit on the sshd
process, the process running with 100% cpu usage. This in effect slows down
the performance and ultimately the speed at which the checkout runs. Linux
clients connecting to the same server do not cause this load on the server.
This happens even when compressions is turned off and when

encryption

Post by Zoran Petkovic
Cyphers are changed, as well as different versions of

subversion. The

Post by Zoran Petkovic
behaviour is identical. I'm not sure who to address for this

issue as this

Post by Zoran Petkovic
not only happens with TortoiseSVN but with SlikSVN as well. Any direction
would be appreciated.

Why does iostat show? Could it be that your underlying disk is

io-saturated

and your CPU spike is due to iowait?

One possible explanation would be that the windows clients perform
the
checkout faster than the linux clients for some reason, thus being
able to put higher load on the server, saturating its cpu or disk
I/O.

Other than that I have no idea.

The Linux client you're testing is also 1.9.7?

Perhaps the windows client and the linux client are performing the
checkout in a different way for some reason (I'm thinking of the
so-called "skelta mode" vs. "bulk mode", see
http://subversion.apache.org/docs/release-notes/1.8.html#
serf-skelta-default).

--
Johan