Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 327945

Summary: Stop suggesting ccache to all users
Product: [OLD] Docs on www.gentoo.org Reporter: Diego Elio Pettenò (RETIRED) <flameeyes>
Component: Installation HandbookAssignee: nm (RETIRED) <nightmorph>
Status: RESOLVED FIXED    
Severity: normal CC: bruno, comrel, council, dev-portage, docs-team, jacobgodserv, jcallen, kevinlyles, polynomial-c, qa, scen
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3#doc_chap3
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 326291    
Bug Blocks: 328099    

Description Diego Elio Pettenò (RETIRED) gentoo-dev 2010-07-12 15:33:31 UTC
ccache is a fast compiler cache. When you compile a program, it will cache intermediate results so that, whenever you recompile the same program, the compilation time is greatly reduced. In common compilations this can result in 5 to 10 times faster compilation times.

_Where_ is the "5 to 10 times faster" figure coming from? It's definitely not the case for _normal_ Gentoo operation. It might have such effect on stage generation, but on a normal user system it actually _slows_ down the builds.

I've been repeating the "ccache is not good for you" mantra for over two years since http://blog.flameeyes.eu/2008/06/21/debuking-ccache-myths ... then there are Robin's instruction on how to debug ccache-related build problems...

Can we simply drop it out of the documentation, please?
Comment 1 Sebastian Luther (few) 2010-07-12 15:42:20 UTC
+1 from me
Comment 2 Samuli Suominen (RETIRED) gentoo-dev 2010-07-12 16:04:36 UTC
+1, I'm tired of dealing with issues like bug 327871 and bug 79519
Comment 3 Markos Chandras (RETIRED) gentoo-dev 2010-07-12 16:48:52 UTC
yes yes yes

I already opened another ccache bug about broken compilations 

bug #326291
Comment 4 nm (RETIRED) gentoo-dev 2010-07-12 18:48:20 UTC
Remove it from Portage, and Portage's documentation, and we'll consider it. In the mean time, it's a FEATURE that needs to be documented.

Closing for now; do reopen when folks are ready to kick it out of Portage. Then we can do it in the docs so that we're all on the same page.
Comment 5 Jacob Godserv 2010-07-12 18:53:21 UTC
(In reply to comment #4)
> Closing for now; do reopen when folks are ready to kick it out of Portage. Then
> we can do it in the docs so that we're all on the same page.

In the meantime, can we make clear that ccache is intended primarily for developer use, and that if users use it they are on their own?
Comment 6 Diego Elio Pettenò (RETIRED) gentoo-dev 2010-07-12 18:55:43 UTC
Can I call this bullshit? The page does not document split-debug, just to name one and is throwing idiotic statements at users such as the already noted "5 to 10 times faster".

I'll ask this nicely once again, next step I'll bring it up the council: please drop this section from the documentation, users can find how to set up ccache in a number of other ways, including the man pages, but this should *not* be suggested for a standard install.

Comment 7 Tomáš Chvátal (RETIRED) gentoo-dev 2010-07-12 18:57:24 UTC
We have ton of feautres that are not documented.
But if you read the features text above that says it is best and shiny way, then everyone will enable it.
It can be documented, heck it even can stay in portage. But the current text seems like we encourage it to users. Everyone listens to _5 to 10 times faster_ text.

Imagine we would put the same text about strict. Hell even base system should not compile with strict today. So should we remove it from portage? Nope we just dont propagate it so much in docs.
Comment 8 Luca Barbato gentoo-dev 2010-07-12 19:59:41 UTC
since the documentation and the feature are completely off would be nice fixing the documentation: the first run of ccache could slow everything down a lot and you want to use it ONLY if you are building many times the same app.
Comment 9 nm (RETIRED) gentoo-dev 2010-07-12 22:46:52 UTC
Diego, watch your tone. If your hostility continues I'll take this to devrel. In the meantime I'll CC them just so they can keep an eye on it.

Everyone, I have no particular objection to modifying something in our documentation. However, I don't like it when people get hostile simply because I asked them to do *your* part in straightening something out.

If something is obviously broken in your own words, and you flat-out demand that the docs team remove it (without ever bothering to learn the history of why some text is in there, either), and then you insist that you're not going to do anything to actually fix said brokenness elsewhere . . . well, that's going to make the GDP reluctant to cooperate in a timely manner.

Not that we won't do it, but yelling at us, throwing up ridiculously childish threats like "getting the council involved if we don't do it" when I'd already said that we'll do the change, and then telling us you won't lift a finger to get a real solution . . . is simply NOT a good way to work with your fellow developers.

Please keep that in mind, thanks. Restricting to developers only, just in case someone flies off the handle again.
Comment 10 Diego Elio Pettenò (RETIRED) gentoo-dev 2010-07-12 23:00:55 UTC
*Where else* does it say it has a 5 to 10 times increase in speed?

I'm not asking to get rid of ccache, I'm asking to get rid of ccache suggestion to _all_ users. As Jacob said it's enough to say that it's a feature for developers mostly. I'm not asking the moon and what is your response? Closing the bug asking for getting rid of the ccache package and feature at once.

You're being totally unreasonable, and it's not the first time. I've asked nicely, twice. Half of QA already weighed their vote in for the change.

CCing council and un-restricting the bug because there is *nothing* here that shouldn't be public. If devrel feels otherwise they can take it up to me, but the *docs* bug here is valid _and_ should be left public.
Comment 11 nm (RETIRED) gentoo-dev 2010-07-12 23:16:11 UTC
(In reply to comment #10)
> *Where else* does it say it has a 5 to 10 times increase in speed?
> I'm not asking to get rid of ccache, I'm asking to get rid of ccache suggestion
> to _all_ users. As Jacob said it's enough to say that it's a feature for
> developers mostly. I'm not asking the moon and what is your response? Closing
> the bug asking for getting rid of the ccache package and feature at once.
> You're being totally unreasonable, and it's not the first time. I've asked
> nicely, twice.

I asked for it to be fixed in Portage as well: fixing the Portage manpages and whatnot so there won't be a mismatch between the handbook and man portage. And this, to you, is completely unreasonable, given that you've been demanding that we do something right this second, calling us idiots, and resorting to swearing and threatening to run to the council.

I've already stated my desire to get a similar fix in Portage to match the handbooks (and not just a one-liner note about possible breakage), and now I'll add to it the suggestion that maybe folks should consider dropping support for ccache entirely. After all, confcache was useful, but also had issues where it broke compilation, so it was entirely removed, not just redocumented. Perhaps it'd be worth considering this for ccache. If that happpens, we can remove it completely from the docs.

What you want is to leave it documented but state: "You should probably never do this since its only useful in very few circumstances and it'll likely break your box." That's extremely poor practice. There's no point in mentioning an available feature, then immediately telling folks, "Don't actually do this." It'll just confuse them.
Comment 12 Jorge Manuel B. S. Vicetto (RETIRED) Gentoo Infrastructure gentoo-dev 2010-07-12 23:24:26 UTC
Josh,

I can say from my experience as a member of the KDE team that we've got through the years quite a few unexplainable bugs that turned out to disappear after the users disabled ccache. Therefore, I too would appreciate any change that helps reduce its use and agree with warning users about it.

Diego,

there's no need to go over the documentation team or to threat going to the council. Let the documentation team address this bug.

@docs-team:

what can any interested Gentoo Developer do to help fixing this bug? The request here is to stop "promoting" the use of ccache and warn users about its consequences - removing the "5 to 10 times faster" note would be a start. If you want to talk with the Portage team about their docs, that's understandable, but I don't think that should stop you from addressing this bug.
Comment 13 Diego Elio Pettenò (RETIRED) gentoo-dev 2010-07-12 23:25:47 UTC
Given *I* was the one deleting confcache from the whole system, I'd be surprised if I didn't know how to handle that. ccache is *not* the same situation.

confcache was broken by design (by design clash with autotools if you want to say so), while ccache has its uses, it should very well be kept documented in the portage man pages, but it should *not* be documented in the handbook the newly-coming users will be looking at, and *especially* it should *not* outright lie regarding a figure that has no place to be there (the already cited four times "5 to 10 times faster").

> Remove it from Portage, and Portage's documentation, and we'll consider it.
> In the mean time, it's a FEATURE that needs to be documented.

This leaves to intend that you have *no* intention in either hiding a feature that users shouldn't be documenting, nor fix it so that it actually spells the truth, nor do anything about it as long as it's in portage and you wish for us to nuke it from there. Good luck with your black and white world because we don't live there. I asked you to stop suggesting it to *all* users; you can write proper documentation for it if you don't want to delete the section, something that tells the users they _are_ going to experience slower builds, that they should *only* be using this when rebuilding the same package for whatever reason.

But as it is, your documentation is portraying bullshit (5 to 10 times faster) as truth, which *is not the case*. And if we want to speak about tone, instead of asking me to suggest something to replace the current documentation with, you suggested me to go out on a quest to get rid of a feature that (albeit not always) is useful, before you "consider" doing something about it. Are you kidding me? I even asked please, twice.
Comment 14 Tomáš Chvátal (RETIRED) gentoo-dev 2010-07-12 23:26:07 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > *Where else* does it say it has a 5 to 10 times increase in speed?
> > I'm not asking to get rid of ccache, I'm asking to get rid of ccache suggestion
> > to _all_ users. As Jacob said it's enough to say that it's a feature for
> > developers mostly. I'm not asking the moon and what is your response? Closing
> > the bug asking for getting rid of the ccache package and feature at once.
> > You're being totally unreasonable, and it's not the first time. I've asked
> > nicely, twice.
> 
> I asked for it to be fixed in Portage as well: fixing the Portage manpages and
> whatnot so there won't be a mismatch between the handbook and man portage. And
> this, to you, is completely unreasonable, given that you've been demanding that
> we do something right this second, calling us idiots, and resorting to swearing
> and threatening to run to the council.
> 
> I've already stated my desire to get a similar fix in Portage to match the
> handbooks (and not just a one-liner note about possible breakage), and now I'll
> add to it the suggestion that maybe folks should consider dropping support for
> ccache entirely. After all, confcache was useful, but also had issues where it
> broke compilation, so it was entirely removed, not just redocumented. Perhaps
> it'd be worth considering this for ccache. If that happpens, we can remove it
> completely from the docs.
> 
> What you want is to leave it documented but state: "You should probably never
> do this since its only useful in very few circumstances and it'll likely break
> your box." That's extremely poor practice. There's no point in mentioning an
> available feature, then immediately telling folks, "Don't actually do this."
> It'll just confuse them.
> 
make.conf manpage:
              ccache Enable portage support for the ccache package.  If the ccache dir is not present in the user's environment,  then  portage  will default to ${PORTAGE_TMPDIR}/ccache.

So it does not say much anything, and most people would not even consider enabling it.
It has its use but only in rare cases. So why documenting it and not lets say splitdebug that is much more usefull to upstreams... We as qa simply ask to remove the misleading text:
-When you compile a program, it will cache intermediate results so that, whenever you recompile the same program, the compilation time is greatly reduced. In common compilations this can result in 5 to 10 times faster compilation times.
+When you compile a program, it will cache intermediate results so that, whenever you recompile the same program multiple times (testing purposes, live ebuilds) the compilation time is reduced. Up to 20% of compilation time in average.

But prefferably the part should be goner as is, because there are much more usefull features, that are not documented in handbook at all so why should be this one.
</qa></council>
Comment 15 Tomáš Chvátal (RETIRED) gentoo-dev 2010-07-12 23:27:57 UTC
(In reply to comment #14)
> But prefferably the part should be goner as is, because there are much more
> usefull features, that are not documented in handbook at all so why should be
> this one.
By "the part" i mean section 3.c
Comment 16 nm (RETIRED) gentoo-dev 2010-07-12 23:29:18 UTC
(In reply to comment #12)
> what can any interested Gentoo Developer do to help fixing this bug? The
> request here is to stop "promoting" the use of ccache and warn users about its
> consequences - removing the "5 to 10 times faster" note would be a start. If
> you want to talk with the Portage team about their docs, that's understandable,
> but I don't think that should stop you from addressing this bug.

No, indeed, it shouldn't. I've already expressed willingness to fix the docs. In fact, since the original reporter didn't do his homework properly, I've also been going through our other documentation and removing references to ccache from the Quickinstall Guide.

It'll just take more time to get the fixes into the handbook, since I want to back up the "why you might not want to do this" with info from bug #326291. Diego's attitude doesn't help, but I can ignore 'im for the time being while I do the handbooks.

Also, I would appreciate it if the Portage folks would add some of the same text to the manpage; zmedico's one-liner fix was a start, but more explanation to help our users is always a good idea.
Comment 17 nm (RETIRED) gentoo-dev 2010-07-12 23:32:27 UTC
(In reply to comment #14)
> It has its use but only in rare cases. So why documenting it and not lets say
> splitdebug that is much more usefull to upstreams...
The reason why ccache was documented and not splitdebug was that ccache has been around longer. And no one's asked us to document splitdebug. I agree that splitdebug is more useful, so we'd like to get it documented.

If someone could open a separate bug for that and provide a patch or just plaintext with a description of the FEATURE, we'd appreciate it!
Comment 18 Diego Elio Pettenò (RETIRED) gentoo-dev 2010-07-12 23:34:57 UTC
Mirror climbing is an art is it now?
Comment 19 nm (RETIRED) gentoo-dev 2010-07-13 00:29:50 UTC
Fixed in CVS. 

From the commit log:

"redo the ccache section so it's not recommended for everyone. make it clear that it's for folks doing development work, and that ccache is known to cause numerous compile failures. also added a suggestion to disable ccache and recompile before reporting any bugs."

Portage team, can I get you to put some of the handbook text into the ccache section of the manpage? something a bit more explanatory than the initial one-liner would be useful. Thanks!
Comment 20 Petteri Räty (RETIRED) gentoo-dev 2010-07-13 12:42:53 UTC
(In reply to comment #19)
> 
> Portage team, can I get you to put some of the handbook text into the ccache
> section of the manpage? something a bit more explanatory than the initial
> one-liner would be useful. Thanks!
> 

Filed as bug 328099 so it actually can get done.
Comment 21 Jorge Manuel B. S. Vicetto (RETIRED) Gentoo Infrastructure gentoo-dev 2010-07-14 01:03:28 UTC
I was warned that my comment in this bug was read by some developers as being an official action from Developer Relations.
To be very clear, I was pointed to the bug by one of the parties, but my involvement in this bug was that of an interested developer only. Furthermore, this bug was solved without any need for an official process from devrel.
I didn't use neither my devrel nor my council hat in this bug.
Comment 22 Ryan Hill (RETIRED) gentoo-dev 2010-08-14 02:48:15 UTC
I really hate to revive this party, but I believe ccache-3 renders most of your misgivings about ccache moot.  Direct mode highly reduces the mentioned "slow down" caused by running the preprocessor twice.  Compressed cache files means data stays cached longer, increasing the chances of hits (though admittedly increasing processing time).  And finally CCACHE_BASEDIR means that cache data can now be reused across version bumps (where previously every package bump rendered the cached data invalid due to the working directory changing), meaning it's useful for everyone.

I don't understand why you're disputing the "5 to 10 times" figure.  Looking at my logs that's the average time savings for recompilations.  I maintain one package in particular (ppl) which takes 2 1/2 hours to compile the first time and 10 minutes from ccache.  Any user who is changing a USE flag benefits from ccache.  I don't know about you, but I consider that normal Gentoo operation.