| Summary: | app-shells/bash will `read` data forever if no newline is found | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Gordon Malm (RETIRED) <gengor> |
| Component: | New packages | Assignee: | Gentoo's Team for Core System packages <base-system> |
| Status: | RESOLVED INVALID | ||
| Severity: | normal | CC: | jamesb.fe80, kfm, zorry |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
Appears similar to: https://lists.ubuntu.com/archives/hardy-changes/2008-May/011496.html However, it was supposedly fixed in bash-3.2.38, so don't know what to make of that. > foo.txt:
> dd if=/dev/zero of=/dev/null bs=1M count=10
> (or pick your favorite way to make this file)
I apologize, that should've obviously read:
foo.txt:
dd if=/dev/zero of=foo.txt bs=1M count=10
(or pick your favorite way to make a 10MB or larger file)
I've tinkered with this a little on my Kubuntu laptop, and mistakenly concluded it was fixed there. After repetitive tinkering, I have concluded that (for mine) a limit setting of less than 43476 produces the following output. Which may or may not be related.
>foo2.sh: xrealloc: ../../bash/builtins/../../bash/builtins/read.def:444: cannot reallocate 8388720 bytes (0 bytes allocated)
>FAIL
stuff
I built bash 4.0.0 from source on my laptop, and It exhibits an additional behaviors, with modified version of foo.sh values of 43472 and over pass. values of 10143 and under segfault. values between usually fail producing output like: >bash: xrealloc: ../bash/general.c:888: cannot reallocate 262144 bytes (0 bytes allocated) sometimes I got other results such as the following three. >foo2.sh: xrealloc: ../../bash/builtins/../../bash/builtins/read.def:444: cannot reallocate 524400 bytes (0 bytes allocated) >bash: error while loading shared libraries: libdl.so.2: failed to map segment from shared object: Cannot allocate memory >bash: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory Ignore the previous comment, it was valid if bash4 was running the outer script and bash3 the inner. the following is slightly more relevant junk. 4.0_p17-r1 (w/ 'examples net nls' on my Hardened Funtoo): FAIL To succeed that version needs 14900 in the ulimit line. I don't suppose bash could be mmap'ing the file and then using lot's of overhead to run the scripts? I'd think that a better test would be to run a test w/ multiple discrete parts each requiring a set amount of memory, and see it the suite require significantly more vm than the individual tests. /me exposes ignorance and stupidity. Tested more recent version on x86-32 compiled with gcc-3.4.6-r2 and gcc-4.3.4. app-shells/bash-4.0_p28: FAIL i dont think this is a bug. you're giving bash a large stream of characters without any newlines. `read` requires a newline to stop its reading. as such, bash continues to read in from your "endless" stream until whatever arbitrary limit you set is hit.
simpler example:
cat /dev/zero | { read v ; }
then in another shell, do:
while sleep 1s; do clear; ps auxOR | grep bash | tail -n1 ; done
|
I've run across what I believe to be a possible memory leak in app-shells/bash. Here are the scripts to reproduce: --- begin foo.sh --- result=FAIL ulimit -v 10240 bash foo2.sh if [ "$?" -eq 0 ]; then result=PASS fi echo ${result} --- end foo.sh --- --- begin foo2.sh --- cat foo.txt | { while read -r LINE; do FOO="bar" done } --- end foo2.sh --- foo.txt: dd if=/dev/zero of=/dev/null bs=1M count=10 (or pick your favorite way to make this file) Here are my results for various bash versions and sh-alike shells: bash-3.1.17: FAIL bash-3.2.33: FAIL bash-3.2.39: FAIL bash-3.2.48: FAIL busybox 1.12.2-r1 sh: FAIL busybox 1.13.2 sh: PASS dash 0.5.4.11: PASS Here's my emerge --info, though I tested on amd64, non-hardened, gcc-4, etc. as well: Portage 2.1.6.7 (hardened/x86/2.6, gcc-3.4.6, glibc-2.8_p20080602-r1, 2.6.28-hardened-r1-rc1-2009030801 i686) ================================================================= System uname: Linux-2.6.28-hardened-r1-rc1-2009030801-i686-Intel-R-_Core-TM-2_Quad_CPU_Q9550_@_2.83GHz-with-glibc2.3.2 Timestamp of tree: Wed, 11 Mar 2009 21:45:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] app-shells/bash: 3.2_p48-r1 dev-java/java-config: 2.1.7 dev-lang/python: 2.5.2-r7 dev-python/pycrypto: 2.0.1-r8 dev-util/cmake: 2.6.2-r1 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.63 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.27-r2 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=nocona -O2 -fweb -fomit-frame-pointer -fno-ident -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-march=nocona -O2 -fweb -fomit-frame-pointer -fno-ident -pipe" DISTDIR="/var/cache/portage/distfiles" FEATURES="distclean distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch userpriv usersandbox usersync" GENTOO_MIRRORS="http://gentoo.mirrors.tds.net/gentoo/ http://gentoo.mirrors.easynews.com/linux/gentoo/ http://mirror.datapipe.net/gentoo/ http://gentoo.chem.wisc.edu/gentoo/ http://gentoo.netnitco.net/ http://gentoo.mirrors.pair.com/ http://open-systems.ufl.edu/mirrors/gentoo/" LDFLAGS="-Wl,-O1,--hash-style=gnu" LINGUAS="en en_US en_GB" MAKEOPTS="-j5" PKGDIR="/var/cache/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--progress --exclude-from=/etc/portage/rsync_excludes" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/layman/sunrise /usr/local/portage" SYNC="Con't do this, using read-only shared portage over NFS!" USE="X Xaw3d a52 aac aalib acl acpi adns alsa ao apache2 audiofile audit bash-completion bcmath berkdb bidi blas bluetooth bzip2 cairo calendar canna caps cdb cddb cdinstall cdparanoia cdr cjk clamav cracklib crypt cscope css ctype cups curl curlwrappers cvs cxx dbm dbus dbx dedicated directfb djvu doc dri dts dv dvb dvd dvdr dvdread encode enscript evo exif expat fam fbcon ffmpeg fftw firefox flac flash flatfile fltk fontconfig foomaticdb fortran freetds freewnn ftp gcj gd gdbm geoip ggi gif gimp ginac glut gmp gnome gnuplot gnutls gphoto2 gpm gps graphviz gsl gstreamer gtk gtkhtml guile hal hardened hddtemp hdf5 iconv idn ieee1394 imagemagick imap imlib inifile ipod ipv6 jack java java5 java6 javascript jbig joystick jpeg jpeg2k kde kerberos ladspa lame lapack lash latex lcms ldap libcaca libgda libnotify libsamplerate libwww lirc lm_sensors logrotate lua lzo mad maildir mailwrapper matroska mbox memlimit mhash midi mikmod mime mmap mmx mmxext mng modplug mono motif mp3 mpeg mpi mplayer mtp musepack musicbrainz nas ncurses neXt netboot netcdf nis nls nntp nocd nptl nptlonly nsplugin ocaml ocamlopt offensive ofx ogg openal openexr opengl osc pam pcntl pcre pda pdf perl php pic plotutils png portaudio posix ppds pulseaudio python qdbm qt3 qt4 quicktime radius rdesktop readline recode rss ruby samba sasl scanner sdl seamonkey session sftp sharedext sharedmem shorten simplexml skey slang slp smartcard smp sndfile snmp soap sockets socks5 sox speex spell spl sqlite sqlite3 sse sse2 sse3 ssl startup-notification subversion svg svga syslog sysvipc szip taglib tcl tcpd theora threads tidy tiff timidity tk tokenizer truetype unicode urandom usb v4l v4l2 vcd vhosts videos vim-syntax vnc vorbis wavpack wddx wifi win32codecs wma wmf wxwindows x264 x86 xattr xcomposite xface xine xinerama xinetd xml xmlrpc xorg xosd xpm xprint xscreensaver xsl xv xvid xvmc yaz zlib" ALSA_CARDS="au8820 ca0106 emu10k1 hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse joystick" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US en_GB" USERLAND="GNU" VIDEO_CARDS="apm ati dummy fbdev fglrx nv radeonhd v4l vesa vga" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS