enhancement suggestion

[Christian Klie]

[Johan Corveleyn]

Hello,
currently I made a huge commit. In respect of the number of the files (see
attachment "working copy").
While the commit went down the transaction grew and grew and grew (see
attachment "transaction").
That all happend because of the files in the transaction's folder being very
small but huge in number so
those have a relativly huge overheat, because of the blocksize. So I am
suggesting to rethink the commit procedure but I am not
sure whether to report this or not.

So the directory 5-6.txn below db/transactions showed:
- Number of files: 244295 (0 folders)
- Size: 88.6 MB
- Size on disk: 29.8 GB
(Wow, that's huge! For only 88.6 MB of data in 244295 files, that's a
factor of more than 300 compared to the actual data size)

And your working copy showed:
- 118094 files, 5350 folders
- Size: 903 MB
- Size on disk: 1.01 GB

What SVN version is the server? What SVN version is the client?
What version of back-end filesystem on the server (BDB or FSFS, which version)?
-> If you have a 1.9 server, please send the output of "svnadmin info
REPOS_PATH". Otherwise, send the contents of REPOS_PATH/format and of
REPOS_PATH/db/format.

The server is on Windows apparently. Which version of Windows? NTFS I
presume? Anything special about that disk? Any special settings?

Oh and one more question: do those files in your working copy have
many svn properties?

[Branko Čibej]

Post by Johan Corveleyn

Hello,
currently I made a huge commit. In respect of the number of the files (see
attachment "working copy").
While the commit went down the transaction grew and grew and grew (see
attachment "transaction").
That all happend because of the files in the transaction's folder being very
small but huge in number so
those have a relativly huge overheat, because of the blocksize. So I am
suggesting to rethink the commit procedure but I am not
sure whether to report this or not.

- Number of files: 244295 (0 folders)
- Size: 88.6 MB
- Size on disk: 29.8 GB
(Wow, that's huge! For only 88.6 MB of data in 244295 files, that's a
factor of more than 300 compared to the actual data size)

344 exactly. The filesystem cluster is 128 KiB and the average file size
is 380 bytes. The cluster (allocation unit) size is quite large, which
suggests it may not be NTFS, or it is a really humongously huge volume. See:

https://support.microsoft.com/en-gb/help/140365/default-cluster-size-for-ntfs-fat-and-exfat

Post by Johan Corveleyn
- 118094 files, 5350 folders
- Size: 903 MB
- Size on disk: 1.01 GB
What SVN version is the server? What SVN version is the client?
What version of back-end filesystem on the server (BDB or FSFS, which version)?
-> If you have a 1.9 server, please send the output of "svnadmin info
REPOS_PATH". Otherwise, send the contents of REPOS_PATH/format and of
REPOS_PATH/db/format.
The server is on Windows apparently. Which version of Windows? NTFS I
presume? Anything special about that disk? Any special settings?
Oh and one more question: do those files in your working copy have
many svn properties?

You're missing the point, Johan ... the "enhancement suggestion" was to
"rethink the commit procedure". Having rethunked it, I suggest the
obvious cheapest solution would be to fix the underlying filesystem. :)

-- Brane

[Johan Corveleyn]

[ Hi Christian, please keep the mailinglist in cc, i.e. use Reply All.
Also, we prefer plain-text emails on this list, of possible. More
below ...]

Post by Johan Corveleyn
What SVN version is the server?
What SVN version is the client?

Both are version 1.9.7 (r1800392).

Post by Johan Corveleyn
What version of back-end filesystem on the server (BDB or FSFS, which version)?

Path: .
UUID: 6baea231-e241-2848-b57a-28e303214589
Repository Format: 5
Compatible With Version: 1.9.0
Repository Capability: mergeinfo
Filesystem Type: fsfs
Filesystem Format: 7
FSFS Sharded: yes
FSFS Shard Size: 1000
FSFS Shards Packed: 0/0
FSFS Logical Addressing: yes
Configuration File: db\fsfs.conf

Post by Johan Corveleyn
Which version of Windows?

Windows 10

Post by Johan Corveleyn
Anything special about that disk?

Format is exFAT

Okay, that probably explains your problem.

As Branko also said on this thread (but you might have missed it, he
...

Post by Johan Corveleyn
- Number of files: 244295 (0 folders)
- Size: 88.6 MB
- Size on disk: 29.8 GB
(Wow, that's huge! For only 88.6 MB of data in 244295 files, that's a
factor of more than 300 compared to the actual data size)

344 exactly. The filesystem cluster is 128 KiB and the average file size
is 380 bytes. The cluster (allocation unit) size is quite large, which
https://support.microsoft.com/en-gb/help/140365/default-cluster-size-for-ntfs-fat-and-exfat

If you go to the page linked above, you'll see at the bottom that the
default cluster size for exFAT is 128 KB if your volume is 32 GB or
larger. So this extremely large "size on disk" for so little actual
data is expected, and is caused by the configuration of your
filesystem. As Branko said, the easiest fix is to fix your filesystem.
Either switch to NTFS or some other filesystem, or if you really
*must* use exFAT, try to format the drive with a smaller cluster size
(I suppose 128 KB is the default, but you can still use a smaller
setting if you deviate from the default -- that's just a guess, I
don't know exFAT).