Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 37976

Summary: Sed 4.1.2 pattern recognition for "\n" characters does not work properly
Product: Gentoo Linux Reporter: Jeremy Warren <jwarren>
Component: [OLD] Core systemAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED INVALID    
Severity: normal CC: mholzer
Priority: High    
Version: 1.4   
Hardware: x86   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Jeremy Warren 2004-01-12 07:23:57 UTC
Given a file of the following example type (named "test"):
------------------------------------------
foo

bar

foo

bar
------------------------------------------

sed -e 's/\n//' test

Should return:

foobarfoobar

Instead, it returns the original file, when, as a control test, 

perl -p -e 's/\n//' test

works as expected.
Comment 1 Martin Holzer (RETIRED) gentoo-dev 2004-01-12 09:00:17 UTC
please try 4.0.8 and/or 4.0.9
Comment 2 Jeremy Warren 2004-01-12 09:18:46 UTC
FYI, I've tested this on two different machines:  a P4 laptop and a Tbird Athlon machine.

Sed 4.0.8 does not work either.  Neither does 4.0.9.
Comment 3 Martin Schlemmer (RETIRED) gentoo-dev 2004-01-12 09:26:05 UTC
Hmm, he is right.  Please give 'emerge info'.
Comment 4 Jeremy Warren 2004-01-12 09:32:08 UTC
One additional tidbit:

I have discovered that if you do this:

sed -e 'N;s/\n//' test

You will get the following output:

foo
bar
foo
bar

Very strange.  This all goes back to me trying to find a way around the "Portage Loop of Death" that occurs right now when you try to do a emerge -e world with portage-2.49-r21.  So I wrote these two scripts to do the job, which is where I noticed the issue:

--------------------------------------------------------------
a sed script called "parseworld"
--------------------------------------------------------------
#!/bin/sed -f
s/sys-apps\/portage//
s/\[ebuild  N    \] //
s/\[ebuild   R   \] //
s/\[ebuild    D  \] //
s/\[ebuild     U \] //
s/-[0-9].*//
s/These.*//
s/Calculating.*//
s/*** Please update portage to the above version before proceeding.//
s/    Failure to do so may result in failed or improper merges.//
s/    A simple 'emerge -u portage' is sufficient.//
s/\n/ /
/./,/^$/!d
------------------------------------------------------------------





------------------------------------------------------------------
the script that calls parseworld to run emerge
(with the emerge options being passed on the command line)
------------------------------------------------------------------
#!/bin/bash
FILESTOEMERGE=`emerge -uUDpe world | ./parseworld | perl -p -e 's/\n/ /'`
nice emerge $* ${FILESTOEMERGE}
------------------------------------------------------------------




Here's the funny bit with the whole "sed -e 'N;s/\n/ /'" bit:  If the output from the parseworld script is filtered with the above sed command instead of the perl line, what you get is a two-column formatted list, which means that expression matching for "\n" appears to work, but only under certain circumstances.

-------------------------------------------------------------------------------
EMERGE INFO (for the P4 Machine):

Portage 2.0.49-r20 (default-x86-1.4, gcc-3.2.3, glibc-2.3.2-r3, 2.6.0-mm1)
=================================================================
System uname: 2.6.0-mm1 i686 Mobile Intel(R) Pentium(R) 4 - M CPU 1.70GHz
Gentoo Base System version 1.4.3.10
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-Os -march=pentium4 -mmmx -msse -msse2 -mfpmath=sse -momit-leaf-frame-pointer -maccumulate-outgoing-args -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.1/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
CXXFLAGS="-Os -march=pentium4 -mmmx -msse -msse2 -mfpmath=sse -momit-leaf-frame-pointer -maccumulate-outgoing-args -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache sandbox"
GENTOO_MIRRORS="ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo http://128.213.5.34/gentoo/ http://mirrors.tds.net/gentoo ftp://mirrors.tds.net/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="S3TC X aalib acpi alsa autofs avi berkdb bluetooth bonobo crypt cups dga directfb dvd encode esd evo fam fbcon ffmpeg foomaticdb freetype fs gdbm gif gnome gpm gtk gtk2 gtkhtml icc icc_pgo ifc imap imlib irmc java jikes jpeg kde ldap libg++ libwww mad mikmod mmx motif mozilla mpeg ncurses nls oggvorbis opengl opie oss pam pcmcia pda pdflib perl pic png pnp python qt quicktime readline scanner sdl slang spell sse ssl svga tcltk tcpd tiff truetype wmf x86 xml xml2 xmms xv zlib"

-------------------------------------------------------------------------------
EMERGE INFO (For the Athlon Machine)

Portage 2.0.49-r20 (default-x86-1.4, gcc-3.2.3, glibc-2.3.2-r3, 2.4.22-gentoo-r4)
=================================================================
System uname: 2.4.22-gentoo-r4 i686 AMD Athlon(tm) Processor
Gentoo Base System version 1.4.3.10
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-Os -march=athlon-tbird -mmmx -m3dnow -fomit-frame-pointer -falign-loops -falign-jumps -funroll-loops -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.1/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
CXXFLAGS="-Os -march=athlon-tbird -mmmx -m3dnow -fomit-frame-pointer -falign-loops -falign-jumps -funroll-loops -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache sandbox"
GENTOO_MIRRORS="ftp://ftp.ussg.iu.edu/pub/linux/gentoo ftp://csociety-ftp.ecn.purdue.edu/pub/gentoo/ ftp://gentoo.noved.org/ http://gentoo.noved.org/"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="3dnow X acpi alsa apm arts avi berkdb bonobo canna cjk crypt cups dga directfb encode fbcon ffmpeg fftw foomaticdb freetype gdbm gif gnome gpm gtk gtk2 imap imlib java javascript jikes jpeg kde ldap libg++ libwww mad mikmod mmx motif mozilla mpeg mysql ncurses nls oggvorbis opengl oss pam pdflib perl pic png python qt quicktime readline samba sasl sdl sis slang spell ssl svga tcpd tiff truetype x86 xine xml2 xmms xv zlib"
Comment 5 Martin Schlemmer (RETIRED) gentoo-dev 2004-01-12 11:28:41 UTC
Reverting to 4.0.5 do not work. A 'make check' does not fail though :/
Comment 6 SpanKY gentoo-dev 2004-01-12 15:23:47 UTC
odd that \r works ...
given the test file and running unix2dos on it then using
sed -i 's:\r::' test
gives a file w/out \r in it ...
Comment 7 SpanKY gentoo-dev 2004-10-03 00:46:26 UTC
e-mailed upstream to bonzini@gnu.org per the sed manpage
Comment 8 SpanKY gentoo-dev 2004-10-12 13:55:50 UTC
afaict, this is not a bug :/

http://www.opengroup.org/onlinepubs/009695399/utilities/sed.html

this line in particular:
In default operation, sed cyclically shall append a line of input, less its terminating <newline>, into the pattern space.