Error E140001: Sum of subblock sizes larger than total block content length

[Ronald Taneza]

Hi,

I got the error below while running "svnadmin load -M 0" to load a dump
file created by "svnrdump dump".

svnadmin: E140001: Sum of subblock sizes larger than total block content
length

This error was reported when "svnadmin load" was loading a big file (around
2 GB) from a revision in the dump file.
I checked the dump file produced by svnrdump, and noticed that the
Content-length for the 2GB file is a negative value! (see below)

I finally got it working by first creating a local dump file using
"svnadmin dump", then loading this dump file using the same "svnadmin load"
command used above.

More details below:
We are migrating our subversion server from Ubuntu Linux to Window Server
2012.
* Current server (Linux): svn, version 1.9.3 (r1718519)
* New server (Windows): svn, version 1.8.19 (r1800620)

We are actually using an older svn version in the new Windows server,
because we are using CollabNet Subversion Edge, which still uses the svn
1.8 series.

To migrate data, we have a script running in the new server that does:
* svnrdump dump https://current-server/repo --incremental > dumpfile
* svnadmin load -M 0 local-path\repo < dumpfile

The "svnadmin load" command failed with the error message above, when
loading a revision containing a big file (around 2 GB).
Checking the dump file produced by svnrdump (svn version 1.8.19), I noticed
that the Content-length for the 2GB file is a negative value!
The expected Content-length value is (prop-content length +
text-content-length) = 2238208388.
But the actual value is -2056758908, which is what you get when you try to
interpret 2238208388 as a signed 32-bit integer (max 2147483647).

SVN-fs-dump-format-version: 3

Node-path: TheBigFile
Node-kind: file
Node-action: add
Prop-delta: true
Prop-content-length: 59
Text-delta: true
Text-content-length: 2238208329
Text-content-md5: d2f79377abb0db99b37460c8156727fb
Content-length: -2056758908

When the same revision is dumped using "svnadmin dump" in the Linux server
(svn version 1.9.3), the content-length is a positive value, as expected:

SVN-fs-dump-format-version: 2

Node-path: TheBigFile
Node-kind: file
Node-action: add
Text-content-md5: d2f79377abb0db99b37460c8156727fb
Text-content-sha1: 5f5c06f734f394697f90cfab26dee5d820216ee5
Prop-content-length: 59
Text-content-length: 2237924210
Content-length: 2237924269

Lastly, I tried "svnrdump dump" in the Linux server (svn version 1.9.3),
and the content-length is also correct:

SVN-fs-dump-format-version: 3

Node-path: TheBigFile
Node-kind: file
Node-action: add
Prop-delta: true
Text-delta: true
Text-content-md5: d2f79377abb0db99b37460c8156727fb
Prop-content-length: 59
Text-content-length: 2238208329
Content-length: 2238208388

So this seems to be a problem with svnrdump version 1.8.x?

Of course, we would prefer to use the latest svn 1.9.x version, but we find
the CollabNet Subversion Edge package quite useful (apache2 and svn in one
installer package; provides web-based management console).

Regards,
Ronald

[Daniel Shahaf]

Post by Ronald Taneza
Checking the dump file produced by svnrdump (svn version 1.8.19), I noticed
that the Content-length for the 2GB file is a negative value!
The expected Content-length value is (prop-content length +
text-content-length) = 2238208388.
But the actual value is -2056758908, which is what you get when you try to
interpret 2238208388 as a signed 32-bit integer (max 2147483647).

Subversion stores file sizes as svn_filesize_t which is an alias to
apr_int64_t which ought not to wrap until 2**63-1. Are apr_int64_t and
APR_INT64_T_FMT correct on your systems?

It's possible that we're casting 64bit->32bit somewhere --- that'd be a
bug --- but let's first rule out the above (heeding Occam's razor).

Cheers,

Daniel

[Julian Foad]

*Sent:* dinsdag 21 november 2017 15:44
I got the error below while running "svnadmin load -M 0" to load a dump
file created by "svnrdump dump".
  svnadmin: E140001: Sum of subblock sizes larger than total block
content length
This error was reported when "svnadmin load" was loading a big file
(around 2 GB) from a revision in the dump file.

[...]

Checking the dump file produced by svnrdump (svn version 1.8.19), I
noticed that the Content-length for the 2GB file is a negative value!

[...]

  SVN-fs-dump-format-version: 3
  Node-path: TheBigFile
  Node-kind: file
  Node-action: add
  Prop-delta: true
  Prop-content-length: 59
  Text-delta: true
  Text-content-length: 2238208329
  Text-content-md5: d2f79377abb0db99b37460c8156727fb
  Content-length: -2056758908

Thank you for finding this!

I can see this bug existed in svnrdump up to 1.8.19. (For 1.9 I refactored this to use common code shared with 'svnadmin dump' which does not have this bug.)

In 1.8.19, subversion/svnrdump/svnrdump.c:close_file() contains:

if (fb->dump_text)
...
SVN_ERR(svn_stream_printf(eb->stream, pool,
SVN_REPOS_DUMPFILE_TEXT_CONTENT_LENGTH
": %lu\n",
(unsigned long)info->size));
...
if (fb->dump_props)
SVN_ERR(svn_stream_printf(eb->stream, pool,
SVN_REPOS_DUMPFILE_CONTENT_LENGTH
": %ld\n\n",
(unsigned long)info->size +
propstring->len));
else if (fb->dump_text)
SVN_ERR(svn_stream_printf(eb->stream, pool,
SVN_REPOS_DUMPFILE_CONTENT_LENGTH
": %ld\n\n",
(unsigned long)info->size));
...


info->size is apr_off_t ... probably 64 bits on most systems.
propstring->len is apr_size_t ... probably 64 bits on most systems.

It uses "%lu" for the text content, which thus work OK up to 4 GB, and "%ld" for the overall content length which thus only works up to 2 GB.

Earlier in this file, the property content length is printed correctly:

buf = apr_psprintf(pool, SVN_REPOS_DUMPFILE_CONTENT_LENGTH
": %" APR_SIZE_T_FMT "\n", len);

The attached patch should fix it; not yet tested.

- Julian