Bugzilla Bug 2679

ebuild.sh should be able to use colorgcc if you emerged it.
Here is a very trivial patch (don't know if it is the best way to do it):
--- ebuild.sh.orig      Sun May 12 20:00:30 2002
+++ ebuild.sh   Sun May 12 20:05:15 2002
@@ -17,6 +17,8 @@
 shopt -s expand_aliases
 source /etc/profile.env > /dev/null 2>&1
 export PATH="/sbin:/usr/sbin:/usr/lib/portage/bin:/bin:/usr/bin:${ROOTPATH}"
+# Use colorgcc if we have it
+[ -d /usr/bin/wrappers ] && PATH=/usr/bin/wrappers:${PATH}
 if [ -e /etc/init.d/functions.sh ]
 then
        source /etc/init.d/functions.sh > /dev/null 2>&1

Daniel,

I tend to think that /usr/bin/wrappers should be in the ebuild.sh path
only if "wrappers" is in FEATURES, but I'll leave that decision up to you.

Just as a sidenote, to have support for both colorgcc and ccache, you need to   
change colorgcc to use to the ccache in the colorgcc script itself. Just add 
ccache before the call to the $compiler: 

if (-x '/usr/bin/ccache') { 
        $compiler_pid = open3('<&STDIN', \*GCCOUT, \*GCCOUT, 
"/bin/bash -c '/usr/bin/ccache $compiler @ARGV'"); 
} else { 
        $compiler_pid = open3('<&STDIN', \*GCCOUT, \*GCCOUT, 
"/bin/bash -c '$compiler @ARGV'"); 
} 

That will call ccache with the correct path to the executable. Then, you need 
to add support for the other compilers that ccache supports but colorgcc 
doesn't out of the box, like cpp and i686-linux-pc-gcc, by adding them in the 
colorgcc script and adding the links. 

Works like a charm ;)  

As a second aside, I think that ebuild should log the output of the building in
a file, with each line prefixed by the timestamp and an 'E' if it was on stderr.

Then, the user can choose to see the compile output, with the following cool
possibilities:
- No output, only errors, all
- compile times
- see output in a different window, so you have the emerge just telling you
  what is being built
- a very rough percentage complete indicator based on previously collected
  number-of-lines for the ebuild

Just dumping my thoughts here ;)

*** Bug 3820 has been marked as a duplicate of this bug. ***

Created an attachment (id=8098) [details]
Add support for colorgcc-1.3.2-r3 to ebuild.sh

In bug 15387 I provide an updated ebuild for colorgcc that improves how it
determines what program to execute next (it uses the PATH-prefix-stripping
idiom that I designed for the upcoming distcc ebuild, not hard-wired paths). 
This makes it is trivially easy to add colorgcc support to ebuild.sh, and the
attached patch does this (the patch is based on portage-2.0.46-r12).

Created an attachment (id=8213) [details]
Patch colorgcc support into ebuild.sh.

While my previous patch succeeded in getting colorgcc called from an ebuild,
it failed to get any color to output because colorgcc checks if STDOUT is a
tty, and since it is not, it bypasses the colorization of the output.  To
work around this, the latest ebuild that I attached to bug 15387 has support
for turning the tty check off.	This new ebuild.sh patch just adds one extra
line compared to the last one:

export COLORGCC_REQUIRE_TTY=no

With this in place, I've gotten colorized output running on my system.

NOTE:  anyone who has a pre-existing .colorgccrc file in their home dir,
be sure to comment out all the gcc/cc/g++/c++ command paths in the file
when using this new setup (so that colorgcc doesn't bypass other wrappers
on the path, such as ccache and distcc).

Created an attachment (id=8218) [details]
Switching to a patch for emake.

I've been playing with the colorgcc setup from ebuild.sh, and it interferes
with configure, so I've backed-out the last patch.  I'm now testing a patch
for emake that causes colorgcc to only be used once the make actually begins.

However, I'm thinking that a much safer thing to do would be to just filter
the output of the ebuild process as it goes to the screen, so we might have
to come up with a more intricate solution -- one that does what colorgcc does
but not at the gcc level, at a higher ebuild level.  It may be possible to
cajole the colorgcc to do this itself, but I haven't investigated this yet.

Created an attachment (id=8747) [details]
Patch portage.py to colorize the errors/warnings in the output

OK, I've come up with what I think is the best way to colorize the compiler
output of emerge and "ebuild ... compile" -- have portage.py optionally run
the ebuild.sh script using colorgcc as a wrapper.  This solution is simple,
it works for all packages (unlike the emake patch), and it doesn't rely on
any ill-conceived kludges to disregard the tty-ness of stdout.	I've been
using this for several days now, and I think it ready to go.

To try this out for yourself, you must have installed my latest colorgcc
ebuild (see bug 15387) because it handles the ebuild.sh path forwarding
and it avoids coloring too much of the ebuild output (with slightly
improved regex matching).  Then, just apply this portage.py patch to
the library you'll find in /usr/lib/python2.2/site-packages and set the
"colorgcc" option in your FEATURES (in /etc/make.conf) and you should be
ready to go.

Created an attachment (id=8810) [details]
Updated portage.py patch for portage-2.0.47-r7.

Still todo:  need to copy root's .colorgccrc file into the portage/homedir if
userpriv is on in the FEATURES.

Created an attachment (id=8862) [details]
New colorgcc portage.py patch with .colorgccrc change and 1 bugfix

This version of the patch (still for portage-2.0.47-r7) copies root's version
of the .colorgccrc file into portage/homedir if the userpriv FEATURE is set and

the source files exists in /root.  I also added a chown() call for the homedir
since it was owned by root (which causes some problems if a KDE-using ebuild
tries to create $HOME/.kde, for instance).

One thing I thought of recently that the ebuild doesn't yet do:

We should output a warning that tells users that they should modify any
existing .colorgccrc files to remove hard-coded compiler paths (since the
script now figures out who to call on its own).  We should probably also
mention to the user that the "masquerade dir" (where the symlinks are
placed) has changed.

Please ignore the prior comment (#11) -- I thought I was commenting on bug
15387.

This touches too many files... And I will not include support into portage
for anything that preempts any of the internal scripts of files. If you
reduce it to a complexity similar to distcc or ccache I'll probably add
it. The only required change can be an addition to ebuild.sh.

The problem with adding colorgcc support to ebuild.sh is in how it does per-
ebuild logging.  If logging is disabled, then a trivial patch such as the
second ebuild.sh one above (but without the export of COLORGCC_REQUIRE_TTY)
would do the trick.  With logging enabled, the output handle of the commands
that ebuild.sh runs is no longer a tty, and thus the colorgcc check that
figures out if it should do coloration fails (this check is needed to avoid
confusing configure, for instance).

So, one possible fix would be to change the way ebuild.sh does per-ebuild
logging to one that uses ptys (so the is-a-tty check still succeeds).  Does
anyone know of a logging program (similar to "script") that will work on
arbitrary shell commands ("script" only appears to work on an interactive
shell process).

Failing this, does someone know of a better "output is going to the user"
check for colorgcc to use besides "-t STDOUT" (I can't think of one).

The final option (that I can think of now) is to extend colorgcc to allow
it to filter stdin to stdout and insert a colorgcc-filter command into the
per-ebuild logging line (right before the "| tee ..." bit).

Nicholas:  any preferences here?

Created an attachment (id=11928) [details]
Add colorgcc support to ebuild.sh

I've just heard back from the maintainer of the util-linux package, and he has
accepted my patch for the "script" command to allow it to run an arbitrary
command rather than just an interactive shell.	The patch is very simple, and
you can see it here (it applies to either 2.11y or 2.11z):

    http://www.blorf.net/script.patch

What this means is that the ebuild.sh script can change over to using the new
"script -c COMMAND OUTPUT-FILE" command to do its logging instead of using the
"tee" command.	Doing this has the advantage that all the build commands that
get run by ebuild.sh think that they're outputting to a tty, which makes the
colorgcc command "just work".  So, the simple patch that is in this attachment
is now sufficient to add colorgcc support to the ebuild.sh script (so long as
your "script" command supports the new -c option).  Also keep in mind that the
colorgcc-wrapper path in this patch is the new one that (I have been assured)
will be committed soon.  If you're using an ebuild with an older path, you can
tweak the patch very easily to have it refer to /usr/bin/wrapper.

Note also that you need to add "colorgdc" to your FEATURES to turn this on.

This patch is relative to portage-2.0.48_pre6 because 2.0.47-r10 has a bug that

prevents per-ebuild logging from working.  Note that if someone wants to try
this with 2.0.47-r10 wants to try this out, I have a patched ebuild.sh script
that fixes the logging code and includes this colorgcc support patch (remember
that you must have a patched "script" command to use it!):

    http://www.blorf.net/ebuild.sh

I'll submit a revised util-linux ebuild that includes this patch next, just so
we can get a jump on the next util-linux release (if desired).

Lemme know when this becomes part of out util linux ebuilds then
so I can depend by version.

As requested, I'm letting you know that the enhancement to the "script" command
is now in the sys-apps/util-linux-2.11z-r4 ebuild, which is now unmasked (on
x86 and amd64). The updated colorgcc ebuild is not yet committed, so that part
of my ebuild.sh patch is not yet ready to go (alas). The logging change can be
made now, if you so desire, and I recommend adding the -q option to my
previously suggested logging change so that the screen doesn't get cluttered
with extra start/stop messages:

script -q -c "$0 $*" -a "${PORT_LOGDIR}/${LOG_COUNTER}-${PF}.log"

This has been working great for me since mid-May when I first switched over.

Created an attachment (id=17817) [details]
Updated ebuild.sh patch for colorgcc support

The latest version of my patch has the aforementioned -q option on the
script line and is updated to apply to more recent portage versions
(including portage-2.0.49-r3 and portage-2.0.49-r4).

*** Bug 20356 has been marked as a duplicate of this bug. ***

*** Bug 30261 has been marked as a duplicate of this bug. ***

It's been 90 days (or more) since any changes were made to this bug.

I know I'm being annoying, but what's the status of things? It's no doing all this work and then forgetting about it.

*** Bug 39859 has been marked as a duplicate of this bug. ***

Created an attachment (id=28563) [details]
patch to add colorgcc to ebuild.sh for portage 2.0.50-r1

This is based on the previous (2003-09-16 11:12 PST) patch. I just took the
changes out of it, manually added them to ebuild.sh. This may have taken out
some updated code - I had to remove a line from about 9 lines into the file,
which looked to be a more comprehensive version of the code which was removed
by the old patch (probably from a portage 2.0.49/48 system) - however, seems to
work fine.

Please update this patch or add to it if you want, I just wanted to get
colorgcc working for 2.0.50-r1 and it'd been badgering me to upgrade for months
from my patched version of 2.0.50.

(From update of attachment 28563 [details])
27,28c27
<			{ $0 $* || rm -f "${T}/successful" ; } 2>&1 \
<			  | tee -i -a "${PORT_LOGDIR}/${LOG_COUNTER}-${PF}.log"
---
>                       script -q -c "$0 $*" -a "${PORT_LOGDIR}/${LOG_COUNTER}-${PF}.log"
1267a1267,1273
>
>     if has colorgcc ${FEATURES} &>/dev/null; then
>           if [ -d /usr/lib/colorgcc/bin ]; then
>             export PATH="/usr/lib/colorgcc/bin:${PATH}"
>         fi
>     fi
>

Technically an enhancement, so marking severity as such...

You can colorgcc support in Portage with the following:

a) create "/etc/env.d/04colorgcc" with the contents:

PATH="/usr/lib/colorgcc/bin"
ROOTPATH="/usr/lib/colorgcc/bin"
PREROOTPATH="/usr/lib/colorgcc/bin"

b) the just do "env-update"

The important bit for Portage is the var "PREROOTPATH".  Perhaps this env file should be added to colorgcc, so no changes would be made to Portage ...

Wouldn't there be a conflict between ccache and colorgcc?

why doesnt colorgcc install an env.d entry?

because wrappers interfere with normal operation and usually you'd want to be
able to opt in or out of them easily

what happend to this bug? colourgcc support would be really cool! 
 

re-open when colorgcc is actually stable

search bugzilla, you'll find a bunch of very obscure bugs which were caused by
the colorgcc wrappers

(In reply to comment #31)
> re-open when colorgcc is actually stable
> 
> search bugzilla, you'll find a bunch of very obscure bugs which were caused by
> the colorgcc wrappers

Just stumbled across this one again. My bugzilla search for colorgcc related
bugs returned nill, so perhaps it's time to reconsider this feature?

Bug toolchain people to add it to gcc-config/eselect-compiler, thats where this
(as well as ccache, distcc and all the other cc wrappers) belong.

(In reply to comment #33)
> Bug toolchain people to add it to gcc-config/eselect-compiler, thats where this
> (as well as ccache, distcc and all the other cc wrappers) belong.

toolchain herd, any comments? :)

i used to use colorgcc myself ... i stopped when it started causing
random/weird build errors

If portage do not mess with PATH order, I do not see why we should stick all
weird obscure stuff in our wrappers - the user can just setup his PATH properly
if he want this.

Azarah - could you comment on bug #128810, where we're suggesting doing
ccache/distcc in the wrappers instead of in Portage, if you feel it's not a
good idea.

*** Bug 166428 has been marked as a duplicate of this bug. ***

Recent ebuild doesn't support colorgcc since the colorgcc detect the stdout is
not a tty. Maybe something look like below should be used instead?

# ebuild.sh
if hasq colorgcc $FEATURES ; then
        qa_call src_compile | colorgcc
    else
        qa_call src_compile
fi

please read the full history of this bug ... you'll see that we have no plans
to include colorgcc support at all

either it works or it doesnt, no point in trying to support half way hacks
since it'll just annoy us devs ;)

simple enough to setup your own PATH wrapper for colorgcc ...

>SpanKY
>>simple enough to setup your own PATH wrapper for colorgcc ...

The PATH wrapper doesn't actually work, since colorgcc detects the stdout in
ebuilds is not a tty. Anyway, I don't care whether you'll include colorgcc
support or not.