Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 471102 (lfs-tracker) - [TRACKER] handling of Large File System (LFS) in packages (typically 32bit/glibc userland)
Summary: [TRACKER] handling of Large File System (LFS) in packages (typically 32bit/gl...
Status: CONFIRMED
Alias: lfs-tracker
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords: Tracker
Depends on: 549092 550482 583282 471024 550438 550502 550508 550708 559572 576396 629704 681866 687548
Blocks:
  Show dependency tree
 
Reported: 2013-05-23 19:23 UTC by SpanKY
Modified: 2019-06-07 00:59 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description SpanKY gentoo-dev 2013-05-23 19:23:43 UTC
tracker for packages which still use 32bit filesystem interfaces (stat/read/lseek/etc... instead of stat64/read64/lseek64/etc...).  feel free to ask general questions here, but specific bug reports should be in dedicated bugs (and listing them as blockers to this).

this has always been a problem.  typically though people only notice when they try to do with large files (like working with files over 2GiB in size).  but the stat() issue is going to be affecting people even more as filesystems (like NFS and XFS) start to deploy 64bit inodes.  now if you stat() a file on disk that happens to have a 64bit inode, you'll get an error (EOVERFLOW).

for many apps, the fix is simple: build with -D_FILE_OFFSET_BITS=64.  this will redefine funcs/types/structs from the 32bit version to the 64bit version (e.g. off_t will automatically turn into off64_t, stat() will become stat64(), etc...).

if you're building a library that exports off_t as part of its ABI (in a struct or func return/arg), this change will most likely change things.  in that case, you should consider defining:
 -D_LARGEFILE_SOURCE
 -D_LARGEFILE64_SOURCE

then you can internally change the code to call stat64() and use off64_t without changing the sizes of the 32bit types (e.g. off_t is still off_t).

this will require more work though to make things portable (like checking if stat64() even exists) as not all C libraries/operating systems have taken this approach.

some random background links:
http://sourceware.org/ml/libc-alpha/2013-02/msg00575.html
http://blog.fmeh.org/2013/05/11/does-the-world-need-32-bit-inodes/
Comment 1 Michael Weber (RETIRED) gentoo-dev 2014-01-08 15:04:11 UTC
I hit this with ~amd64 current sys-devel/gcc:4.7 and :4.8 (running glibc-2.17)
Comment 2 SpanKY gentoo-dev 2014-01-08 19:40:16 UTC
(In reply to Michael Weber from comment #1)

i have no idea what this means.  you really need to provide actual details.
Comment 3 SpanKY gentoo-dev 2014-03-13 22:35:04 UTC
it's come up again:
https://sourceware.org/ml/libc-alpha/2014-03/msg00092.html

this time i've posted a patch.  i'll probably include it in glibc-2.20 when it comes out to see what goes boom ...
Comment 4 LW 2014-03-21 07:20:57 UTC
Is it possible that the glusterfs network file system is always returning
inode numbers larger than 2**32?

Trying to rebuild gcc where /var/tmp/portage is a glusterfs share
from an XFS file system on the file server:

  configure: updating cache ./config.cache
   * ../../sandbox-2.6/libsandbox/libsandbox.c:check_syscall():879: failure   (Value too large for defined data type):
   * ISE:
        abs_path: (null)
        res_path: /var/tmp/portage/sys-devel/gcc-4.7.3-r1/work/build/x86_64-pc-linux-gnu/32/libgomp/conftest.val
  configure: error: unsupported system, cannot find sizeof (omp_lock_t)
  make[2]: *** [configure-stage1-target-libgomp] Error 1

Seems to be similar to the forum discussion
    https://forums.gentoo.org/viewtopic-t-960128-start-0.html

This smells like a stat problem, since the server side logs
many of these:
    [server-rpc-fops.c:1739:server_stat_cbk] 0-porttmp-server: 2676750: STAT
    (null) (adbcef83-56c6-4caf-b974-7b14796bde8c) ==> 
        (No such file or directory)

-------

Something that might also be related is that
    emerge --buildpkg y <some-package> fails, e.g. sys-libs/tmezone-data
fails with a glusterfs file system for /var/tmp/portage, but
    emerge --buildpkg n <some-package> does not.

    >>> Completed installing timezone-data-2013d into /var/tmp/portage/sys-libs
    /timezone-data-2013d/image/

    strip: x86_64-pc-linux-gnu-strip --strip-unneeded -R .comment -R
       .GCC.command.line -R .note.gnu.gold-version
       usr/sbin/zic
       usr/sbin/zdump
    ecompressdir: bzip2 -9 /usr/share/man
    tar: ./usr/share/zoneinfo/posix: file changed as we read it
    tar: ./usr/share/zoneinfo/right/Asia: file changed as we read it
    tar: ./usr: file changed as we read it

This also generates STAT (null) errors on the server side
Comment 5 Ryan Hill (RETIRED) gentoo-dev 2014-04-06 22:07:00 UTC
The first bit looks similar to bug #502202.
Comment 6 SpanKY gentoo-dev 2015-05-26 06:24:05 UTC
if you're using autotools, you can use one of these macros in your configure.ac:
  AC_SYS_LARGEFILE
  AC_USE_SYSTEM_EXTENSIONS

the latter is a super set of the former.  which is appropriate depends entirely on the package in question.  the manual has more details:
https://www.gnu.org/software/autoconf/manual/autoconf.html
Comment 7 SpanKY gentoo-dev 2015-05-26 06:35:18 UTC
(In reply to SpanKY from comment #6)

to clarify, the latter will enable _GNU_SOURCE which implicitly enables LFS support.  however, it does not turn on transparent LFS, so packages still have to use alternative function names like fseeko.

if you use AC_SYS_LARGEFILE, then the build will automatically enable transparent LFS which means you don't have to do anything else.

if you wish to use both, make sure you call AC_SYS_LARGEFILE after AC_USE_SYSTEM_EXTENSIONS otherwise you'll get warnings when you run autotools.