Steps to reproduce: gcc-config i686-pc-linux-gnu-3.4.3 && source /etc/profile emerge cdrtools --> cache misses emerge cdrtools --> cache hits emerge cdrtools --> cache hits gcc-config i686-pc-linux-gnu-3.4.3 && source /etc/profile emerge cdrtools --> cache misses emerge cdrtools --> cache hits emerge cdrtools --> cache hits It seems all compilations done before gcc-config won't generate any cache hits afterwards. In the case one uses [vmware-friendly] gcc3.3 to compile a kernel and gcc3.4 to compile everything else, the cache will be killed every time we switch to gcc3.3 for a few minutes to compile a kernel. Since the cache never empties, I guess this would be a bug..
Created attachment 43594 [details] Output of emerge info
anyone feel like researching how ccache works ? gcc-config updates mtimes of things like /usr/bin/gcc everytime `gcc-config` is run ... perhaps that's the problem ?
I think that behavior is okay - different compiler versions generate different binaries so when changing compilers, the ccache's cache needs to be cleared. Otherwise you would have gcc 3.4.3 for example but having your package partly compiled by 3.4.3, partly by 3.3 and partly with 3.4 (whichever was compiling the respective part first) causing random crashes (binaries generated by various versions of gcc may be incompatible).
Well, I agree that no output data shall be kept from one compiler version to another. But what if the user uses gcc 3.3 for compiling kernels, and gcc 3.4 for everything else ? For example: - Compile many programs with gcc 3.4 - Compile kernel with gcc 3.3 - Compile many programs with gcc 3.4 The third step cannot re-use any of the past gcc-3.4 compilations from the cache, because another gcc version was used inbetween. Well it would work if the gcc versioning info were hashed together with the input data. Perhaps it is already implemented in ccache, but the mtime of gcc symlinks make the hashes mitmatch ? I should really have a look at the code :-)
gcc-config doesnt use symlinks it compiles a small binary wrapper C and stores it at /usr/lib/gcc-config/wrapper ... then gcc-config just copies it to /usr/bin/gcc and stuff
i think you hit it on the head with comment #2. i don't know how portage integrates ccache, but the standard way w/o portage is to put ccache in the same directory as the links to the compilers make sure it gets called before (eg.) gcc does. this is done by exporting CC="ccache gcc", or in Gentoo by instructing the user to put /usr/lib/ccache/bin/ into their PATHS, which contains a group of symlinks to /usr/bin/ccache. these are the things that ccache takes into account when matching hashes: "The basic idea is to detect when you are compiling exactly the same code a 2nd time and use the previously compiled output. You detect that it is the same code by forming a hash of: * the pre-processor output from running the compiler with -E * the command line options * the real compilers size and modification time * any stderr output generated by the compiler These are hashed using md4 (a strong hash) and a cache file is formed based on that hash result. When the same compilation is done a second time ccache is able to supply the correct compiler output (including all warnings etc) from the cache."[1] since gcc-config changes the "real" compilers modification size ("real" in this is determined by "which gcc"[2], all previous cache data is useless. just a comment on some of the other comments - ccache is not going to include 3.4 code in 3.3 compiled programs. it's not that dumb. that's the point of it hashing in the version, modtime, and command line used. hell, changing CFLAGS can set you back to scratch sometimes ;P #4 raises a good situation. in a non-gentoo world, and you had both installers installed, you could use them back and forth without problem. it's definitely the gcc-config modtimes that are what's doing it. [1]http://ccache.samba.org/ccache/ccache-man.html [2]http://www-106.ibm.com/developerworks/linux/library/l-ccache.html?Open&ca=daw-ad
er.. two compilers installed i meant. ;P sorry, too early in the morning..
perhaps this could help: CCACHE_PATH You can optionally set CCACHE_PATH to a colon separated path where ccache will look for the real compilers. If you don't do this then ccache will look for the first executable matching the compiler name in the normal PATH that isn't a symbolic link to ccache itself. ;D
ok, i have a (semi) workaround, but i don't know if it goes against policy. setting a CCACHE_PATH variable in each file in /etc/env.d/gcc equal to the PATH variable in said file (see below) definitely solves the issue. but some might (rightly) argue that CCACHE settings have no business being in these files, but it was the only way i could figure out how to keep the variables in sync as a user changes profiles. the colon delimited list mentioned works, but the order the directories appear in would have to dynamically change, and there's no way (ok there is, but it's a pain) to know exactly what versions every user has slotted on their system and generate the path from that. i tested it out and it just didn't work for me. # cat /etc/env.d/gcc/i686-pc-linux-gnu-3.3.4 PATH="/usr/i686-pc-linux-gnu/gcc-bin/3.3" ROOTPATH="/usr/i686-pc-linux-gnu/gcc-bin/3.3" CCACHE_PATH="/usr/i686-pc-linux-gnu/gcc-bin/3.3" # cat /etc/env.d/gcc/i686-pc-linux-gnu-3.4.3 PATH="/usr/i686-pc-linux-gnu/gcc-bin/3.4.3" ROOTPATH="/usr/i686-pc-linux-gnu/gcc-bin/3.4.3" CCACHE_PATH="/usr/i686-pc-linux-gnu/gcc-bin/3.4.3" i used patch and nano as the test packages. i reset the cache, compiled each with gcc 3.4.3 three times and got the expected results. the did a "gcc-config i686-pc-linux-gnu-3.4.3 && env-update && source /etc/profile" and compiled again. the packages continued to use the ccached data. :D i repeated this several times and the cache never missed. to take it a level further i switched to 3.3.4 and repeated the tests as above with identical results. i confirmed that 3.3.4 was being used with a gcc -v -Q on an empty testcase. satisfied, i switched back to 3.4.3, and ran the original tests again without another miss, so it even works between switching profiles. :D now the downside. i can't compile cdrtools using this. i'm guessing it's weirdass Schily makefile system does not agree at all with this workaround. it fails like so: Trying to find cc Found cc Creating empty 'incs/Dcc.x86-linux' ==> MAKING DIRECTORY "incs/x86-linux-cc/Inull" ==> CONFIGURING RULES "incs/x86-linux-cc/rules.cnf" creating cache ./config.cache checking host system type... i686-pc-linux-gnu checking for Cygwin environment... no checking for mingw32 environment... no checking for EMX/OS2 environment... no checking for executable suffix... configure: error: installation or configuration problem: compiler cannot create executables. make: *** [incs/x86-linux-cc/rules.cnf] Error 1 everything else i've tried hasn't exploded on me yet, but i haven't done much. if this fix is considered a possible one, let me know and i'll start testing. i need to do an emerge -e system anyways. ;D otherwise i'll see what else i can do. cheers, --de.
(*forehead slap*) cd /usr/i686-pc-linux-gnu/gcc-bin/<version> ln -s gcc cc working fine on everything tested now. i can't believe i missed that. :P
I'll take a look at this as it has some implications for multilib compilation as well... getenv("ABI") needs to be considered in the hash (see bug #75420).
rather than doing some kind of PATH manipulation, what if we have gcc-config run `touch -r <GCC BINARY> <GCC WRAPPER>` ?
good idea. based on some very basic tests i just did, it looks like that will do it. i used patch as a testcase again since it's fast and it only uses i686-pc-linux-gnu-gcc to compile. here's the rundown. root ~ # ccache -Czs Cleared cache Statistics cleared cache directory /var/cache/ccache cache hit 0 cache miss 0 files in cache 0 cache size 0 Kbytes max cache size 1024.0 Mbytes root ~ # gcc-config 2 * Switching to i686-pc-linux-gnu-3.4.3 compiler... root ~ # env-update && source /etc/profile root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Jan 2 04:07 /usr/bin/i686-pc-linux-gnu-gcc root ~ # touch -r /usr/i686-pc-linux-gnu/gcc-bin/3.4.3/i686-pc-linux-gnu-gcc /usr/bin/i686-pc-linux-gnu-gcc root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Dec 25 10:05 /usr/bin/i686-pc-linux-gnu-gcc root ~ # emerge patch root ~ # ccache -s cache hit 7 cache miss 68 root ~ # emerge patch root ~ # ccache -s cache hit 82 cache miss 68 root ~ # emerge patch root ~ # ccache -s cache hit 157 cache miss 68 root ~ # gcc-config 2 * Switching to i686-pc-linux-gnu-3.4.3 compiler... root ~ # env-update && source /etc/profile root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Jan 2 04:17 /usr/bin/i686-pc-linux-gnu-gcc root ~ # touch -r /usr/i686-pc-linux-gnu/gcc-bin/3.4.3/i686-pc-linux-gnu-gcc /usr/bin/i686-pc-linux-gnu-gcc root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Dec 25 10:05 /usr/bin/i686-pc-linux-gnu-gcc root ~ # emerge patch root ~ # ccache -s cache hit 232 # this is the point where it would normally miss, so touch does work cache miss 68 # now let's test changing compilers root ~ # gcc-config 1 * Switching to i686-pc-linux-gnu-3.3.4 compiler... root ~ # env-update && source /etc/profile root ~ # emerge patch root ~ # ccache -s cache hit 239 cache miss 136 root ~ # gcc-config 2 * Switching to i686-pc-linux-gnu-3.4.3 compiler... root ~ # env-update && source /etc/profile root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Jan 2 04:25 /usr/bin/i686-pc-linux-gnu-gcc root ~ # touch -r /usr/i686-pc-linux-gnu/gcc-bin/3.4.3/i686-pc-linux-gnu-gcc /usr/bin/i686-pc-linux-gnu-gcc root ~ # ll /usr/bin/i686-pc-linux-gnu-gcc -rwxr-xr-x 1 root root 7.6K Dec 25 10:05 /usr/bin/i686-pc-linux-gnu-gcc root ~ # emerge patch root ~ # ccache -s cache hit 314 # looking good cache miss 136
Created attachment 47592 [details] gcc-config-1.3.8 test this for me please
wfm. (x86)
thanks, gcc-config-1.3.8-r4 now has this
wfm too -- that's great, thanks a lot !!
