Bugzilla Bug 30985

When attempting to download a file of over two gigabytes, wget dies, and I see:
zsh: file size limit exceeded wget --ignore-length
http://not.the.real.url/bigfile.tar.
Also, wget without --ignore-length thinks that a 2.5 gig file has a negative
length, and dies at zero bytes.  This seems to be a known bug in wget;
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=137989 mentions it and has a
patch (which I've not tried).

Reproducible: Always
Steps to Reproduce:
1. Try to download a file over 2 gigabytes in size with wget.
2. Try again with --ignore-length as it will have downloaded 0 bytes, claiming a
negative length.
3. At 2 gigabytes, it will stop downloading and you will recieve a "file size
limit exceeded" message.
Actual Results:  
Wget refuses to download more than zero bytes of the file without
--ignore-length, as it thinks that the Content-Length header is a negative
number.  With --ignore-length, it downloads two gigabytes and stops.

Expected Results:  
Wget should correctly deal with large files and long Content-Length headers.

I can create files larger than this using dd, which do not appear to cause any
problems; there are no limits/ulimits/quotas/etc which prevent me from having
files > 2 gigabytes large.

Portage 2.0.49-r10 (default-x86-1.4, gcc-3.2.3, glibc-2.3.2-r1, 2.4.22)
=================================================================
System uname: 2.4.22 i686 AMD Athlon(TM) XP 2500+
Gentoo Base System version 1.4.3.10p1
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-mcpu=athlon-xp -O2 -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /var/qmail/control /usr/kde/2/share/config
/usr/kde/3/share/config /usr/X11R6/lib/X11/xkb /usr/kde/3.1/share/config
/usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/
/usr/share/config"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
CXXFLAGS="-mcpu=athlon-xp -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="sandbox ccache autoaddcvs"
GENTOO_MIRRORS="http://www.mirror.ac.uk/sites/www.ibiblio.org/gentoo/
ftp://gentoo.inode.at/source/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 apm avi gif jpeg libg++ mad mikmod mpeg ncurses pdflib png xml2 xmms
zlib gdbm berkdb slang readline tetex bonobo X sdl gpm tcpd pam libwww ssl perl
imlib oggvorbis gtk qt mozilla 3dnow acl alsa -arts canna cdr cjk crypt csope
cups dvd encode -esd foomaticdb freewnn gatos -gnome guile ipv6 java jikes junit
-kde -motif nls opengl -oss python quicktime ruby scanner -svga spell tcltk
truetype usb videos xinerama xv video_cards_radeon"

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=181634

We need a hacker :/

wget-1.8.2-r3 in portage should solve this.

Still have this problem with wget 1.9.1.  Download died when it got to 2GB,
with error: "File size limit exceeded"

reopening for new wget

We need someone to wiggle the patch in, doesn't apply clean.

Created an attachment (id=20969) [edit]
Patch to fix 2G limit

Patch for the new wget (1.9.1) that resolves the 2G download limit.

Created an attachment (id=20970) [edit]
patch to fix 2GB download limit

Cleaned up patch.

seemant / katerina can you please test this new patch? Thanks.

Created an attachment (id=31870) [edit]
wget-cvs_LFS.diff (for wget-cvs version)

Alvaro Lopez <alvaro@gnu.org> hacked wget-cvs to support LFS (Large File
Support)

Patch attached, and you can find it at
http://www.alobbs.com/cherokee/wget-cvs_LFS.diff

Why isn't the 1.9.1 patch in portage yet ?  Doesn't it work ?

I've got some problems downloading >2GB files.
After googeling a bit, I've tried a CVS version of wget from here:

http://software.lpetrov.net/wget-LFS/ 

and it works for mit. But it's no solution at all.

So I noticed the patch there and I tried it, but it doesn't work.
A friend of me and I modified it do work with the 1.9.1 in portage.
The patch can be find here:

http://polynomial-c.homelinux.net/pub/gentoo/portage/net-misc/wget/files/wget-LFS-20040909-gentoo.patch.bz2

But I haven't testet it yet.

It might work or not. 

Hi,

there's also an (at least for x86 working) ebuild in http://polynomial-c.homelinux.net/pub/gentoo/portage/net-misc/wget/

The patch Rene mentioned applys against unpatched wget-1.9.1

Testing would be highly appreciated...

Poly

*** Bug 75596 has been marked as a duplicate of this bug. ***

Hmm, that doesn't work for me and it seems to work with these additional
cflags:

-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE 

but I think  -D_FILE_OFFSET_BITS=64  should be enough.

*** Bug 86569 has been marked as a duplicate of this bug. ***

Created an attachment (id=54647) [edit]
LFS patch against wget-1.9.1 from Fedora Core 2

The folks at Redhat are using this, and I've had no problems with it.

The patch from #16 works fine.

ouch, I haven't been monitoring my bugs.  I added wget-1.9.1-r4 into portage
yesterday with a set of Mandrake patches, the last one being largefilesupport. 
Can you all please test and report?

closing due to lack of response

*** Bug 92525 has been marked as a duplicate of this bug. ***

Sorry for the lack of of response, it works great. I just finished downloading
the SUSE 9.2 DVD iso, no problems.

Bug is still present, please reopen.


# wget -c ftp://ftp.freenet.de/pub/ftp.suse.com/pub/suse/i386/10.0/iso/SUSE-10.0-EvalDVD-i386-GM.
iso
--10:17:13--  ftp://ftp.freenet.de/pub/ftp.suse.com/pub/suse/i386/10.0/iso/SUSE-10.0-EvalDVD-i386-
GM.iso
           => `SUSE-10.0-EvalDVD-i386-GM.iso'
Resolving ftp.freenet.de... 194.97.2.68, 194.97.2.67, 194.97.2.70, ...
Connecting to ftp.freenet.de|194.97.2.68|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /pub/ftp.suse.com/pub/suse/i386/10.0/iso ... done.
==> SIZE SUSE-10.0-EvalDVD-i386-GM.iso ... done.
==> PASV ... done.    ==> REST 2147483647 ... done.
==> RETR SUSE-10.0-EvalDVD-i386-GM.iso ... done.
Length: 3,621,957,632 (3.4G), 1,474,473,985 (1.4G) remaining

59% [++++++++++++++++++++++++++++++++++++++++++++++                                 ] 2,147,483,647  
 --.--K/s

nothing happens.


# emerge info
Portage 2.0.51.22-r3 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r2, 2.6.11-hardened-r15 i686)
=================================================================
System uname: 2.6.11-hardened-r15 i686 Intel(R) Xeon(TM) CPU 3.00GHz
Gentoo Base System version 1.6.13
ccache version 2.3 [disabled]
dev-lang/python:     2.3.5-r2, 2.4.2
sys-apps/sandbox:    1.2.12
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils:  2.15.92.0.2-r10
sys-devel/libtool:   1.5.20
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/
3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -march=pentium4 -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks sandbox sfperms strict"
GENTOO_MIRRORS="ftp://gentoo-mirror/gentoo ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo"
LANG="C"
LINGUAS="de"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://gentoo-mirror/gentoo-portage"
USE="x86 X acpi acpi4linux apache2 avi bash-completion berkdb bigger-fonts bitmap-fonts 
browserplugin bzlib crypt cups curldevfs26 dnd dvdread eds emboss fam font-server foomaticdb 
fortran freetype ftp gd gdbm gif gimp gimpprint gstreamer gtk gtk2 hardened hardenedphp iconv 
imagemagick imap imlib innodb ithreads java jpeg jpeg2k junit kde libg++ libgd libwww mailbox maildir 
mbox md5sum mime mmx mmx2 mozdevelop mozilla mozsvg mozxmlterm mp3 ncurses network nls 
nocd nsplugin ogg oss pam pam_console pdf pdflib perl png posix python qt quicktime readline rtc 
samba scanner silverxp sse sse2 ssl svg tcpd tga tiff truetype truetype-fonts type1 type1-fonts 
uudeview v4l v4l2 vorbis xml2 xprint xvid zlib linguas_de userland_GNU kernel_linux elibc_glibc"
Unset:  ASFLAGS, CTARGET, LC_ALL, LDFLAGS, PORTDIR_OVERLAY


# emerge -pv wget

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild   R   ] net-misc/wget-1.10.2  -build -debug -ipv6 +nls -socks5 +ssl -static 4 kB

Your problem is not wget's fault, rather it's the mirror you are using.  Try a
different mirror,
ftp://suse.mirrors.tds.net/pub/suse/i386/10.0/iso/SUSE-10.0-EvalDVD-i386-GM.iso
works well.