Bug 471102 (lfs-tracker) – [TRACKER] handling of Large File System (LFS) in packages (typically 32bit/glibc userland)

• Product: Gentoo Linux
• Status: CONFIRMED ---

[Description SpanKY 2013-05-23 19:23:43 UTC]

tracker for packages which still use 32bit filesystem interfaces (stat/read/lseek/etc... instead of stat64/read64/lseek64/etc...).  feel free to ask general questions here, but specific bug reports should be in dedicated bugs (and listing them as blockers to this).

this has always been a problem.  typically though people only notice when they try to do with large files (like working with files over 2GiB in size).  but the stat() issue is going to be affecting people even more as filesystems (like NFS and XFS) start to deploy 64bit inodes.  now if you stat() a file on disk that happens to have a 64bit inode, you'll get an error (EOVERFLOW).

for many apps, the fix is simple: build with -D_FILE_OFFSET_BITS=64.  this will redefine funcs/types/structs from the 32bit version to the 64bit version (e.g. off_t will automatically turn into off64_t, stat() will become stat64(), etc...).

if you're building a library that exports off_t as part of its ABI (in a struct or func return/arg), this change will most likely change things.  in that case, you should consider defining:
 -D_LARGEFILE_SOURCE
 -D_LARGEFILE64_SOURCE

then you can internally change the code to call stat64() and use off64_t without changing the sizes of the 32bit types (e.g. off_t is still off_t).

this will require more work though to make things portable (like checking if stat64() even exists) as not all C libraries/operating systems have taken this approach.

some random background links:
http://sourceware.org/ml/libc-alpha/2013-02/msg00575.html
http://blog.fmeh.org/2013/05/11/does-the-world-need-32-bit-inodes/

[Comment 1 Michael Weber (RETIRED) 2014-01-08 15:04:11 UTC]

I hit this with ~amd64 current sys-devel/gcc:4.7 and :4.8 (running glibc-2.17)

[Comment 2 SpanKY 2014-01-08 19:40:16 UTC]

(In reply to Michael Weber from comment #1)

i have no idea what this means.  you really need to provide actual details.

[Comment 3 SpanKY 2014-03-13 22:35:04 UTC]

it's come up again:
https://sourceware.org/ml/libc-alpha/2014-03/msg00092.html

this time i've posted a patch.  i'll probably include it in glibc-2.20 when it comes out to see what goes boom ...

[Comment 4 LW 2014-03-21 07:20:57 UTC]

Is it possible that the glusterfs network file system is always returning
inode numbers larger than 2**32?

Trying to rebuild gcc where /var/tmp/portage is a glusterfs share
from an XFS file system on the file server:

  configure: updating cache ./config.cache
   * ../../sandbox-2.6/libsandbox/libsandbox.c:check_syscall():879: failure   (Value too large for defined data type):
   * ISE:
        abs_path: (null)
        res_path: /var/tmp/portage/sys-devel/gcc-4.7.3-r1/work/build/x86_64-pc-linux-gnu/32/libgomp/conftest.val
  configure: error: unsupported system, cannot find sizeof (omp_lock_t)
  make[2]: *** [configure-stage1-target-libgomp] Error 1

Seems to be similar to the forum discussion
    https://forums.gentoo.org/viewtopic-t-960128-start-0.html

This smells like a stat problem, since the server side logs
many of these:
    [server-rpc-fops.c:1739:server_stat_cbk] 0-porttmp-server: 2676750: STAT
    (null) (adbcef83-56c6-4caf-b974-7b14796bde8c) ==> 
        (No such file or directory)

-------

Something that might also be related is that
    emerge --buildpkg y <some-package> fails, e.g. sys-libs/tmezone-data
fails with a glusterfs file system for /var/tmp/portage, but
    emerge --buildpkg n <some-package> does not.

    >>> Completed installing timezone-data-2013d into /var/tmp/portage/sys-libs
    /timezone-data-2013d/image/

    strip: x86_64-pc-linux-gnu-strip --strip-unneeded -R .comment -R
       .GCC.command.line -R .note.gnu.gold-version
       usr/sbin/zic
       usr/sbin/zdump
    ecompressdir: bzip2 -9 /usr/share/man
    tar: ./usr/share/zoneinfo/posix: file changed as we read it
    tar: ./usr/share/zoneinfo/right/Asia: file changed as we read it
    tar: ./usr: file changed as we read it

This also generates STAT (null) errors on the server side

[Comment 5 Ryan Hill (RETIRED) 2014-04-06 22:07:00 UTC]

The first bit looks similar to bug #502202.

[Comment 6 SpanKY 2015-05-26 06:24:05 UTC]

if you're using autotools, you can use one of these macros in your configure.ac:
  AC_SYS_LARGEFILE
  AC_USE_SYSTEM_EXTENSIONS

the latter is a super set of the former.  which is appropriate depends entirely on the package in question.  the manual has more details:
https://www.gnu.org/software/autoconf/manual/autoconf.html

[Comment 7 SpanKY 2015-05-26 06:35:18 UTC]

(In reply to SpanKY from comment #6)

to clarify, the latter will enable _GNU_SOURCE which implicitly enables LFS support.  however, it does not turn on transparent LFS, so packages still have to use alternative function names like fseeko.

if you use AC_SYS_LARGEFILE, then the build will automatically enable transparent LFS which means you don't have to do anything else.

if you wish to use both, make sure you call AC_SYS_LARGEFILE after AC_USE_SYSTEM_EXTENSIONS otherwise you'll get warnings when you run autotools.

[Comment 8 Mario Klebsch 2021-04-20 18:46:14 UTC]

Ok, it seems that in glibc, EOVERFLOW can be returned by stat(), although the lstat64() call was successful.

Unfortunately, I cannot see any way, to work around this problem.

[Comment 9 SpanKY 2021-04-20 21:09:07 UTC]

(In reply to Mario Klebsch from comment #8)

the fix is to build the app in question with LFS flags.  it's not a bug in glibc that this happens since the struct the app is using is too small.

[addCollapseLink(10, 1, 'Toggle comment display'); Comment 10 Larry the Git Cow 2021-05-17 21:00:28 UTC Comment hidden (obsolete)]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b68f75ced2dd0ea6843ad4ac2f012bdcac698295

commit b68f75ced2dd0ea6843ad4ac2f012bdcac698295
Author:     François Degros <[e-mail]>
AuthorDate: 2021-04-28 00:21:41 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2021-05-17 21:00:12 +0000

    dev-libs/libzip: Add large file support (LFS)
    
    This is needed to handle big ZIPs (4+GB) on 32-bit devices.
    
    Bug: https://bugs.gentoo.org/471102
    Signed-off-by: François Degros <[e-mail]>
    Closes: https://github.com/gentoo/gentoo/pull/20564
    Signed-off-by: Sam James <[e-mail]>

 dev-libs/libzip/{libzip-1.7.3.ebuild => libzip-1.7.3-r1.ebuild} | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)