Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 14606 - Gentoo needs coherent handling of LARGEFILE support
Summary: Gentoo needs coherent handling of LARGEFILE support
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Unspecified (show other bugs)
Hardware: All All
: High critical (vote)
Assignee: Gentoo's Team for Core System packages
: 14353 (view as bug list)
Depends on:
Blocks: 27922 30985
  Show dependency tree
Reported: 2003-01-26 23:17 UTC by Bruce A. Locke (RETIRED)
Modified: 2006-10-25 10:29 UTC (History)
11 users (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Bruce A. Locke (RETIRED) gentoo-dev 2003-01-26 23:17:03 UTC
LARGEFILE support in Gentoo currently is rather haphazard.  The 'core' packages
such as fileutils and such compile with LARGEFILE support by default while other
packages (such as many GNOME and KDE packages, etc) do not.  The downsides of
having LARGEFILE support are nonexistant compared to the downsides of not having
it enabled.

2GB files are common these days and LARGEFILE should be enabled for any and all
executables on platforms that fully support it.  The easiest solution may be to
automatically add "-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE" to CFLAGS and
CXXFLAGS for all packages within portage itself.

Comment 1 Bruce A. Locke (RETIRED) gentoo-dev 2003-01-26 23:19:21 UTC
*** Bug 14353 has been marked as a duplicate of this bug. ***
Comment 2 SpanKY gentoo-dev 2003-01-27 14:29:04 UTC
i just did `emerge -e system` on a 1.4 and 1.2 system and both compiled w/out errors ... 
i guess ill know if there are any runtimes bugs in the next few days ;) 
but this is a good sign i think 
Comment 3 Andrew Johnson 2003-01-28 02:22:53 UTC
I also believe that Gentoo needs to take a look at supporting largefiles better. 

I've noticed that some ebuilds (such as apache) do switch on the appropriate
compiler flags, while others (such as less and sysklogd) do not. It seems to me
that Gentoo, one of the more cutting-edge linux distributions (IMHO, at least)
should offer better support for them. I would certainly be willing to contribute
patches, etc. 

There was actually a decent article concerning largefile support posted on
freshmeat recently:
I feel that an issue brought up in the above freshmeat article should be
addressed before working on this issue:
As a result, however, it is highly dangerous to use off_t in header files in
largefile-sensitive systems. Most software writers do not expect that an
integral type like off_t can change its size; it is just used in the exported
interface, as in making a new call "off_t my_lseek(int,off_t,int)".

How much of a concern is this for us?
Comment 4 SpanKY gentoo-dev 2003-01-28 02:41:10 UTC
this bug is a continuation of -core discussion that originated around that freshmeat article 
Comment 5 SpanKY gentoo-dev 2003-01-30 15:11:27 UTC
on glibc-2.2.5 systems, distcc has a problem ...

gcc -march=i686 -O3 -pipe -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -W -Wall -Wshadow -Wpointer-arith -Wcast-align -DHAVE_CONFIG_H -I./src  -c -o src/exec.o src/exec.c
In file included from src/io.c:66:
/usr/include/sys/sendfile.h:26: #error "<sys/sendfile.h> cannot be used with _FILE_OFFSET_BITS=64"

so i guess any package on a glibc-2.2.5 system will fail when they goto utilize this header ... glibc-2.3.x has updated this header file to support 64 bit fopen's
Comment 6 SpanKY gentoo-dev 2003-02-18 02:23:44 UTC
the new strace (4.4.93) fails to emerge with -D_FILE_OFFSET_BITS=64 
ill e-mail the author of strace about the bug ... or just file something upstream ... 
this happens on both gcc-2.x and gcc-3.x, glibc-2.2.x and glibc-2.3.x ... 
source='mem.c' object='mem.o' libtool=no \ 
depfile='.deps/mem.Po' tmpdepfile='.deps/mem.TPo' \ 
depmode=gcc /bin/sh ./depcomp \ 
gcc -DHAVE_CONFIG_H -I. -I. -I. -Ilinux/i386 -I./linux/i386 -Ilinux -I./linux   -Wall 
-D_GNU_SOURCE -march=i686 -O3 -pipe -D_FILE_OFFSET_BITS=64 
-D_LARGEFILE_SOURCE -c `test -f 'mem.c' || echo './'`mem.c 
io.c: In function `sys_sendfile': 
io.c:279: warning: long unsigned int format, different type arg (arg 2) 
io.c: At top level: 
io.c:309: redefinition of `sys_pread' 
io.c:236: `sys_pread' previously defined here 
io.c:327: redefinition of `sys_pwrite' 
io.c:254: `sys_pwrite' previously defined here 
Comment 7 SpanKY gentoo-dev 2003-03-06 08:33:07 UTC
latest xmms fails with latest glibc (2.3.2) ...
the file /usr/include/fts.h cannot be used when enabling large file support

In file included from util.c:26:
/usr/include/fts.h:41:3: #error "<fts.h> cannot be used with -D_FILE_OFFSET_BITS==64"
make[4]: *** [util.o] Error 1
make[4]: Leaving directory `/var/tmp/portage/xmms-1.2.7-r19/work/xmms-1.2.7/xmms'
Comment 8 SpanKY gentoo-dev 2003-06-18 07:23:51 UTC
i added filter-flags to xmms ebuild (all glibc/gcc vers) and the distcc ebuild
(gcc-2.x ver)

i patched strace to fix the prototyping ... turned out that the prototype was
defined with 'off_t' while the func used 'long int' ...
Comment 9 Jesse Adelman 2004-10-01 16:27:59 UTC
Hmm... Any new developments on this front? Personally, I'm interested in having this option apply to Apache (via a "largefile" USE flag, perhaps), but obviously the problem is broader than just web servers... Thanks.
Comment 10 Jesse Adelman 2004-10-01 16:29:52 UTC
Here's a narrow case forum message detailing what I could gather was the current state of this issue (apologies if it is mischaracterized).

(I'm NightMonkey)
Comment 11 Jesse Adelman 2004-10-27 12:46:50 UTC
Any movement on this? Thanks!
Comment 12 Jesse Adelman 2004-10-31 21:28:25 UTC
Can this be re-classified out of "Unclassified (old)"? This is still a present problem.
Comment 13 Jesse Adelman 2004-11-15 13:43:33 UTC
OK, well, since there's no recent response from the development team on this bug, perhaps it should be closed as CLOSED:WONTFIX?
Comment 14 Stefan Schweizer (RETIRED) gentoo-dev 2004-11-15 13:50:56 UTC
How do you want to solve it? Is it enough to add the D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE flag to the example make.conf files?
Comment 15 Jesse Adelman 2004-11-15 13:58:50 UTC
I believe that currently there is no coherent handling of this across ebuilds. Some ebuilds add the gcc flag, some do not. Some actively strip the LFS flags from CFLAGS - which, at least in >=apache-2.0.49 and >=mod_php-2.3.9, is no longer necessary. What I am not sure about is the stability impact of having the LFS flags added to CFLAGS in /etc/make.conf and used for the entire build history of a system (I have only used it for Apache and mod_php). So, in order to allow users to choose and make an informed decision, I think that a "lfs" or "largefile" USE flag would be key to making the choice in a "Gentoo" way. Then, each ebuild can be configured to add the LFS flags, or, in the case of known incompatibilities,  issue warnings.
Comment 16 Stefan Schweizer (RETIRED) gentoo-dev 2004-11-15 14:01:59 UTC
I think the user can make the descicion in /etc/make.conf, so no ebuild needs to be changed for this, just the defaults. We can filter the flag if it does not work.
Comment 17 Chris Gianelloni (RETIRED) gentoo-dev 2004-11-15 14:06:31 UTC
No... he is saying we should add them cursory to what the user supplies in CFLAGS...

I think the problem is that
 1) we generally think that forcing anything is a bad idea
 2) the wrong team has the bug
 3) it really is a bug in the package which should be sent upstream
Comment 18 Chris Gianelloni (RETIRED) gentoo-dev 2004-11-15 14:07:57 UTC
Also, I have built *all* of my systems with these flags in my CFLAGS and they work fine
Comment 19 Andrew Johnson 2004-11-15 14:10:07 UTC
So, the proper thing to do is to file individual bugs for every package that doesn't currently support largefile?

Also, other than lack of testing, is there any reason NOT to have largefile support in every package?
Comment 20 Chris Gianelloni (RETIRED) gentoo-dev 2004-11-15 14:13:15 UTC
Actually, as I said, it really should be enabled upstream, so definitely do not file individual bugs.

I see no problem with enabling it by default.  Perhaps we should reassign this to the portage team rather than base-system.
Comment 21 SpanKY gentoo-dev 2004-11-15 14:34:46 UTC
re Comment 18

i build all my system with these flags to make sure they're ok ... that means ive built amd64/arm/mips/hppa/ppc/sparc/x86 systems and they all work just fine for me

re ebuilds pruning LFS flags

that is unavoidable in some cases (glibc-2.2.x's sendfile.h is NOT LFS safe for example)
Comment 22 Chris Gianelloni (RETIRED) gentoo-dev 2004-11-16 07:21:29 UTC
Removing release@ as this really has nothing to do with the releases themselves.  It is really starting to seem instead like this should be implemented as a FEATURE perhaps?  FEATURES="largefile" which auto-appends the LFS stuff to CLFAGS?  That way ebuilds could still filter them, if they break the ebuild.
Comment 23 Jesse Adelman 2004-12-05 17:06:36 UTC
As a FEATURE or USE, it would be the same to me, from a user perspective. However, what I'm not sure about is how much awareness of FEATURES vis a vis USE is prevalent among Gentoo application packagers. My concern is centered around whether or not application packagers think as much about the impact of differening FEATURES settings vs. the impact of "USE" flags, and how much they might test the effects of USE flag changes vs. FEATURES changes. Am I making sense?
Comment 24 Jesse Adelman 2005-02-08 16:29:44 UTC
Over two years and no resolution on this... And the forum posts continue (just search on "largefile", or "file size exceeded"). I think Chris' idea of implementing this as a FEATURE looks great. But, whether the resolution is FEATURES or a new USE flag, something should be implemented! >2GB files are not that uncommon these days...
Comment 25 Micheal Marineau (RETIRED) gentoo-dev 2005-02-09 09:39:30 UTC
Sad that this has never gone anywhere, sounds easy to impement though.  It seems to me a USE flag for it would make more sense from the users point of view as FEATURES are for how portage behaves, not what actually gets compiled into the package.  Is there a dev watching this bug who knows portage who could submit a patch for this?  If not I'll take a look once I have time, I just don't know anything about the portage code.
Comment 26 Jesse Adelman 2005-03-09 14:20:09 UTC
Is there any way for this to be re-classified out of "Unclassified (old)", please? And can we get a decision on this from the Uber-Devs, yea or nay? The fracturing among packages and their handling of LFS (and its associated flags) appears to be getting worse and is curently unmanaged and unpredictable. Thank you.
Comment 27 Jesse Adelman 2005-05-10 14:05:40 UTC
A decision to reverse and de-support existing LFS support in Apache is being executed now, due to problems not with Apache, but with other masked packages, apr and apr-utils. Were an LFS USE flag in place, this decsion would be made by the end-user, rather than having to ham-handedly remove functionality from a software package.

How do we get greater attention to this glaring issue? Open and unresolved for over 2 years now?
Comment 28 Jesse Adelman 2005-05-12 01:57:32 UTC
Can someone PLEASE move this bug to the appropriate component group? Thanks.
Comment 29 Jesse Adelman 2005-09-01 01:35:05 UTC
Bug marked "NEW" for over 2 1/2 years? Some very cogent solutions have been
proposed in this bug...
Comment 30 Bradley Mcalister 2005-09-10 00:33:46 UTC
I've recently come to realize how annoying this is...didn't realize it has been
going on for such an extended period with virtually no change.
Comment 31 Jesse Adelman 2005-09-10 01:20:42 UTC
At this point, I think that the FEATURES="largefile" option is not the best, as
the flexibility of USE="lfs" has become more apparent. Especially in light of
the new abilities (well, new since this bug was really actively discussed) of
Portage and setting individual USE flag options in /etc/portage/package.use, a
USE flag would be the best, in my opinion. In addition, if there are pacakges
that are broken upstream for LFS, then the package maintainers could just not
support the LFS USE flag at all in their package, and add functionality to new
package versions when upstream has released a fixed version (AKA Apache).

Of course, for me personally, Apache and PHP LFS support was a primary (though
not the only) reason for my wanting this added, but the Apache team has removed
LFS support for now from Apache proper due to an upstream problem with apr. But,
if someone out there _does not_ want LFS support on a package that supports it,
for whatever reason, well, that should be their choice. This is Gentoo, after
all. :)
Comment 32 Jesse Adelman 2005-09-10 22:57:08 UTC
OK, I think I've come around to a different conclusion, after some very
enlightening discussions and work with vericgar in #gentoo-apache. It appears
that the handling of the implementation of LFS support is moving more and more
into upstream development, rather than optional CFLAGS. For instance, he
discovered that the developmental apr-1.x ./configure has these options now:

<vericgar> from ./configure --help for apr:  --disable-lfs           Disable
large file support on 32-bit platforms

And that gives these lines in the configure script:

<vericgar> Check for compiler flags...
<vericgar> checking whether to enable -D_LARGEFILE64_SOURCE... yes
<vericgar>   adding "-D_LARGEFILE64_SOURCE" to CPPFLAGS
<vericgar> that's from running the ./configure
<vericgar> so it will be taken care of automaticly

So, this development would make LFS USE flags and FEATURES="lfs" irrelevant, for
the most part, unless there are a ton of other packages that need explicit CFLAG
settings, which doesn't appear to be true. About 5-6 packages have hard-coded
the -D_LARGEFILE_SOURCE and related CFLAGS into their pacakges, but,
significantly, none of these are multimedia apps, which, if upstream weren't
handling it, would already have filled bugzilla with complaints.

So, I guess that this bug can probably be closed, at least from my point of
view. Anyone else see flaws in this logic? I'll miss this bug... I'll be lonely
without it. ;)
Comment 33 Avuton Olrich 2005-10-15 00:19:47 UTC
ecasound is another package that needs --enable-largefile for >2gb files. 
Comment 34 Stefan Schweizer (RETIRED) gentoo-dev 2006-09-01 08:41:30 UTC
I think this should be closed, can someone from base-system please do so?
Comment 35 Jakub Moc (RETIRED) gentoo-dev 2006-10-25 10:29:16 UTC
I fail to see what's requested here... This can't be done "coherently", nor base-system can solve it, nor there's any reason for lfs use flag or whatever. Either is works and then it should be enabled by default, or it doesn't... E.g., <apache-2.2 will never work with LFS.

File bugs upstream or to ebuild maintainers for individual packages if they are missing LFS support. Closing this one.