inodes usage of svn server

[Keva-Slient]

inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.


is there any way storing all version files into one file? or any other way to solve this problem.




------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>

[Nathan Hartman]

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
is there any way storing all version files into one file? or any other way to solve this problem.

See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.

[Keva-Slient]

thank you very much. that's what i want
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>







------------------ Original ------------------
From: "Nathan Hartman";<***@gmail.com>;
Date: Wed, Jan 3, 2018 11:08 AM
To: "users"<***@subversion.apache.org>;
Cc: "Keva-Slient"<***@qq.com>;
Subject: Re: inodes usage of svn server

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
is there any way storing all version files into one file? or any other way to solve this problem.

See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.

[Nico Kadel-Garcia]

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

Post by Keva-Slient
is there any way storing all version files into one file? or any other way
to solve this problem.
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>

[Keva-Slient]

thank you very much. that's exact what i want.
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>







------------------ Ô­ÊŒÓÊŒþ ------------------
·¢ŒþÈË: "Nico Kadel-Garcia";<***@gmail.com>;
·¢ËÍʱŒä: 2018Äê1ÔÂ3ÈÕ(ÐÇÆÚÈý) ÖÐÎç11:20
ÊÕŒþÈË: "Keva-Slient"<***@qq.com>;
³­ËÍ: "users"<***@subversion.apache.org>;
Ö÷Ìâ: Re: inodes usage of svn server

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

Post by Keva-Slient
is there any way storing all version files into one file? or any other way
to solve this problem.
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>

[Bo Berglund]

On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia

Post by Nico Kadel-Garcia

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

Is this a one-time cleanup operation or does it need to be executed
regularly? I.e. does running "svnadmin pack" on the repo consolidate
the many files into a smaller number of big files and set some
repository property such that it will work that way in the future?

This is what I found in svnbook 1.7 page 178:

"By concatenating all the files of a completed
shard into a single “pack” file and then removing the original
per-revision files, svnadmin pack reduces the file count within a
given shard down to just a single file."

and

"Repacking packed shards is legal, but will have no effect on the disk
usage of the repository."

What exactly is a "shard"?
Is it one of the numerical directories each containing exactly 1000
files?

If so the "shard" storage on my repos seem to only cost at most 4%
extra disk space compared to using a single file (comparing file sizes
against disk usage).

Or will compression of the "svnadmin pack" operation make the combined
size of the files even smaller? I.e. it does not only save unused file
allocation space but actually packs the content even better?

[Daniel Shahaf]

Post by Bo Berglund
Is this a one-time cleanup operation or does it need to be executed
regularly?

Regularly.

Post by Bo Berglund
I.e. does running "svnadmin pack" on the repo consolidate
the many files into a smaller number of big files and set some
repository property such that it will work that way in the future?

The former.

Post by Bo Berglund
"By concatenating all the files of a completed
shard into a single “pack” file and then removing the original
per-revision files, svnadmin pack reduces the file count within a
given shard down to just a single file."
and
"Repacking packed shards is legal, but will have no effect on the disk
usage of the repository."
What exactly is a "shard"?
Is it one of the numerical directories each containing exactly 1000
files?

Yes.

Post by Bo Berglund
If so the "shard" storage on my repos seem to only cost at most 4%
extra disk space compared to using a single file (comparing file sizes
against disk usage).

The concern was inode usage, not disk space usage. Packing reduces the
number of inodes by 99.8%.

Post by Bo Berglund
Or will compression of the "svnadmin pack" operation make the combined
size of the files even smaller? I.e. it does not only save unused file
allocation space but actually packs the content even better?

'svnadmin pack' neither recompresses nor redeltifies. It just
concatenates the files.

[Nico Kadel-Garcia]

Post by Bo Berglund
On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia

Post by Nico Kadel-Garcia

Post by Keva-Slient
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.

There is "svnadmin pack", which needs to run on the Subversion
repository server.

Is this a one-time cleanup operation or does it need to be executed
regularly? I.e. does running "svnadmin pack" on the repo consolidate
the many files into a smaller number of big files and set some
repository property such that it will work that way in the future?
"By concatenating all the files of a completed
shard into a single “pack” file and then removing the original
per-revision files, svnadmin pack reduces the file count within a
given shard down to just a single file."
and

Disk space != inodes. You asked about inodes. File systems have a
maximum number of inodes, the maximum number of files directories,
links, etc. that they know how to count for that filesystem at the
time the filesystem is built. Storing information about files takes
space on a filesystem, sot it's notrmally pre-allocated when making
the filesystem. From your own description, you've run out of inodes
and need to free some up, or rebuild your environment to better handle
structures that handle many small files.

Post by Bo Berglund
"Repacking packed shards is legal, but will have no effect on the disk
usage of the repository."
What exactly is a "shard"?
Is it one of the numerical directories each containing exactly 1000
files?

It's one of the files in those directories.

Post by Bo Berglund
If so the "shard" storage on my repos seem to only cost at most 4%
extra disk space compared to using a single file (comparing file sizes
against disk usage).
Or will compression of the "svnadmin pack" operation make the combined
size of the files even smaller? I.e. it does not only save unused file
allocation space but actually packs the content even better?

Please stop confusing "inode" with "disk space" It's more like you've
run out of boxes to put things in, i.e. inodes, than running out of
space to store store the boxes (i.e., disk space).

Post by Bo Berglund
--
Bo Berglund
Developer in Sweden

[Bo Berglund]

On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia

Post by Nico Kadel-Garcia
Please stop confusing "inode" with "disk space" It's more like you've
run out of boxes to put things in, i.e. inodes, than running out of
space to store store the boxes (i.e., disk space).

Please do not confuse me with the original poster "Keva-Slient"....
I merely stepped in to clarify if I might also have the problem, which
he apparently is having.
But I do not think so (I am on Windows Server 2016 with an NTFS file
system).

[Nico Kadel-Garcia]

Post by Bo Berglund
On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia

Post by Nico Kadel-Garcia
Please stop confusing "inode" with "disk space" It's more like you've
run out of boxes to put things in, i.e. inodes, than running out of
space to store store the boxes (i.e., disk space).

Please do not confuse me with the original poster "Keva-Slient"....
I merely stepped in to clarify if I might also have the problem, which
he apparently is having.
But I do not think so (I am on Windows Server 2016 with an NTFS file
system).

Sorry about the quotation confusion.

NTFS also has limits on the maximum number of files for a filesystem.
There is no complete escape from the underlying limitations of
trade-offs of easy processing versus performance versus capacity that
lead to these limitations. There are also notable performance limits
on having too many files in a directory. We could have a long
discussion about the fundamentals: suffice to say that organizing a
bunch of associated bits into a single file, rather than scattering
many such bits around a filesystem and overpopulating a directory or a
filesystem is not your friend.

Post by Bo Berglund
--
Bo Berglund
Developer in Sweden

[Dave Huang]

Post by Nico Kadel-Garcia
NTFS also has limits on the maximum number of files for a filesystem.

FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)

Post by Nico Kadel-Garcia
There are also notable performance limits on having too many files in
a directory.

I don't know enough about NTFS internals to say whether that's the case
with NTFS or not, but in the context of this discussion, the default SVN
shard size is 1000 revisions, which I don't think could be considered
"too many files in a directory".

While I don't have any actual numbers, my gut feeling is that packing
the repo isn't really needed on NTFS in terms of day to day performance
of clients using the repo, or in terms of filesystem limitations. That
said, I do it anyway because backing up the repo does go faster when
dealing with a couple dozen large files vs. tens of thousands of small
files.

[Branko Čibej]

Post by Dave Huang

Post by Nico Kadel-Garcia
NTFS also has limits on the maximum number of files for a filesystem.

FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)

Post by Nico Kadel-Garcia
There are also notable performance limits on having too many files in
a directory.

I don't know enough about NTFS internals to say whether that's the
case with NTFS or not, but in the context of this discussion, the
default SVN shard size is 1000 revisions, which I don't think could be
considered "too many files in a directory".
While I don't have any actual numbers, my gut feeling is that packing
the repo isn't really needed on NTFS in terms of day to day
performance of clients using the repo, or in terms of filesystem
limitations. That said, I do it anyway because backing up the repo
does go faster when dealing with a couple dozen large files vs. tens
of thousands of small files.

And there's your answer to the question of day to day performance of the
repository: Subversion also has to open many vs. just one file when its
reading historical revisions, so packing will definitely help to reduce
the number of directory lookups and file opens (the latter are
notoriously slow on Windows). Depending on usage patterns, the
performance boost may be significant.

-- Brane

[Bo Berglund]

Post by Branko Čibej
And there's your answer to the question of day to day performance of the
repository: Subversion also has to open many vs. just one file when its
reading historical revisions, so packing will definitely help to reduce
the number of directory lookups and file opens (the latter are
notoriously slow on Windows). Depending on usage patterns, the
performance boost may be significant.

THat seems to be a valid cause for using the function then!
I will look at my new converted repository in view of this info.
Thanks!

[Branko Čibej]

Post by Bo Berglund

Post by Branko Čibej
And there's your answer to the question of day to day performance of the
repository: Subversion also has to open many vs. just one file when its
reading historical revisions, so packing will definitely help to reduce
the number of directory lookups and file opens (the latter are
notoriously slow on Windows). Depending on usage patterns, the
performance boost may be significant.

THat seems to be a valid cause for using the function then!
I will look at my new converted repository in view of this info.
Thanks!

It's safe to create a post-commit hook that runs 'svnadmin pack'
asynchrouously, or schedule a task that does that (hourly or daily,
depending on your needs). The operation is cheap if it has nothing to do.

-- Brane

[Keva-Slient]

here is my config in dav_svn.conf:
<Location /svn>
DAV svn
SVNParentPath /wns/data/svn
SVNListParentPath On
AuthType Basic
AuthName "Subversion Repository"
AuthUserFile /wns/etc/apache2/dav_svn.passwd
Require valid-user
</Location> #CustomLog /var/log/apache2/svn_logfile "%{%Y-%m-%d %T}t %u@%h %>s repo:%{SVN-REPOS-NAME}e %{SVN-ACTION}e (%B Bytes in %T Sec)" env=SVN-ACTION
CustomLog /var/log/apache2/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION




all of the log settings are not working.
it only creates log file. there is not any content written when i commit , checkout or update.
why is this?






------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>