During a recent system update, I attempted to emerge x11-libs/cairo-1.0.2, as it is now a dependency of x11-libs/gtk+-2.8 and x11-libs/pango-1.10. Cairo successfully builds (so I do not believe this is related to bug #110180), but the build fails during the `make test` phase. It seems that the cairo developers specifically mark and ignore those tests expected to fail, so this may indicate a more significant problem.
If not, it may be appropriate to tag this ebuild with RESTRICT=maketest.
The build I am attempting:
Calculating dependencies ...done!
[ebuild N ] x11-libs/cairo-1.0.2 +X +doc -glitz +png 0 kB
The current state of my machine:
Gentoo Base System version 1.6.13
Portage 2.0.53 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r2, 2.6.14-gentoo-r5 i686)
System uname: 2.6.14-gentoo-r5 i686 AMD Athlon(tm) processor
sys-devel/autoconf: 2.13, 2.59-r6
sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
CFLAGS="-march=athlon-tbird -Os -pipe"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/bind /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-march=athlon-tbird -Os -pipe"
FEATURES="autoconfig buildpkg distlocks fixpackages maketest sandbox sfperms strict test"
GENTOO_MIRRORS="http://gentoo.mirrors.easynews.com/linux/gentoo/ http://mirror.espri.arizona.edu/gentoo/ http://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://mirror.datapipe.net/gentoo"
USE="x86 3dnow X aalib acl alsa bash-completion berkdb cjk crypt dga directfb doc esd fbcon flac gdbm gif gnome gpm gtk gtk2 ipv6 java jpeg ldap mad mbox mmx mpeg mysql ncurses nls offensive ogg oggvorbis opengl oss pam perl png python readline sdl skey slang ssl svga tcltk tcpd tetex tiff truetype unicode xv zlib userland_GNU kernel_linux elibc_glibc"
Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS
Error below pruned for sake of brevity; I will attach the full moby in a moment.
>>> Test phase [enabled]: x11-libs/cairo-1.0.2
>>> Test phase [check]: x11-libs/cairo-1.0.2
make: Entering directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'
22 of 60 tests failed
Please report to http://bugs.freedesktop.org/enter_bug.cgi?product=cairo
make: *** [check-TESTS] Error 1
make: Leaving directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'
make: *** [check-am] Error 2
make: Leaving directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'
make: *** [check-recursive] Error 1
Created attachment 76402 [details]
The full log of the failed `make check`.
This is also pruned a bit. Let me know if the full configure and make log might be helpful.
just to add a comment, the gnome policy in the past has been this:
*** Bug 118112 has been marked as a duplicate of this bug. ***
On closer examination of the error, I noticed that the only tests which failed were the *-xlib tests. This is the ebuild which satisfies the virtual/xft RDEPEND on my system:
[ebuild R ] x11-base/xorg-x11-6.8.2-r6 -3dfx +3dnow +bitmap-fonts +cjk -debug -dlloader -dmx +doc +font-server -insecure-drivers +ipv6 -minimal +mmx +nls -nocxx +opengl +pam +sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint +xv 45,146 kB
allanonjl: In bug #117505, ferdy mentioned that he had trouble with the tests while building gtk+, but you mentioned that you didn't have any such problems. Which version of X are you using?
When cairo is compiled, you can also run some automated tests of
NOTE: Some versions of X servers will cause the -xlib tests to
report failures in make check even when cairo is working just
fine. If you see failures in nothing but -xlib tests, please
examine the corresponding -xlib-out.png images and compare them to
the -ref.png reference images (the -xlib-diff.png images might also
be useful). If the results seem "close enough" please do not report
a bug against cairo as the "failures" you are seeing are just due
to subtle variations in X server implementations.
Looking at the log, it is exactly the -xlibs test that fail, so they are most probably just false positives. Since upstream says their test-suite is overly strict and there's not an easy way to fix it, best idea might be to restrict it.
*** Bug 130016 has been marked as a duplicate of this bug. ***
I'm also having similar problems merging cairo-1.0.4. In my case there is one test failing (aside from the expected fails, which I don't take into account). Taking a closer look I see the failing test is an -xlib one, so I guess it'll be safe for now to restrict test and do with it.
However, I don't think restricting tests is a good practice, since tests are there for a particular purpose. Maybe we could just black out tests with a tendency for false positives but let the others to be run. I am not a dev and I don't pretty know whether this would mean much work or even whether it would feasible, but I think it'd be a good idea.
I resynced yesterday, updated everything but cairo (emerge --resume --skipfirst). Lastly I tried to update cairo to 1.0.4 and it worked!
(In reply to comment #8)
> I resynced yesterday, updated everything but cairo (emerge --resume
> --skipfirst). Lastly I tried to update cairo to 1.0.4 and it worked!
It seems you were luckier than me, because I proceeded exactly like you and it didn't work... Still having the same problem.
after checking with Doug, I put RESTRICT=test into the latest stable (testing already had it), so this is fixed now :)
restrict=test is plain pointless. If you decide to run tests for your build, then disabling tests in the ebuild is just hiding the problem from the users who chose to use it.
Sure, some tests in cairo at this point will always fail, but that is what you want to know when setting FEATURES=test.
This is NOT fixed in any way, it is only made worse.
Foser, take your rhetoric else where. Tests are expected to fail. You're suppose to manually run tests and then compare the PNGs visually by hand. Ask Carl.
(In reply to comment #12)
> Foser, take your rhetoric else where.
This sounds quite rude for me!
Anyway I agree to foser - althought I doubt that this is of much interest for some devs...
When I build with tests on it is because I want to know that the package was built correctly (ie was not broken by compiler optimizations or bad memory). Tests that are known to fail, and are not disabled are annoying and useless from this point of view. In fact, they are so widespread that they make this feature virtually useless for me.
This test would need much of work to work correctly - compare the images, and decide wether the difference is "small enough" or not.
If might be possible to disable the tests that are known to fail with a patch, and still perform some tests on the parts of the library that are expected to do exact rendering.
Know that the results of these tests differ per platform and backends used. So now we should disable tests because they fail for you, but what about the ones that went ok for you and failed for another user ?
I'm not all that concerned with users who think gentoo is about compiling as much code as possible, I'm concerned with people who want to run tests for valid reasons and are handicapped by these kind of 'tricks'. It's not just cairo, cairo is just an example.
foser, if you look at comment #5 there is a citation (supposedly from some cairo docs) that says failures in xlib tests are expected.
If the tests failed only for me it would be my problem and not a gentoo bug.
However, these tests do not fail only for me, they fail for a lot of people. They even fail so often they bothered to document the fact. It is a bug in the tests and should be resolved in some way. Disabling all tests is a bit crude but easy. If you want it fixed properly, why haven't you fixed the tests?
And yes, gentoo is about compiling stuff. It is even listed somewhere as its advantage. You are supposed to get packages with the options you want that way.
If running tests to make sure my package was built properly is not a valid reason what is?
> So now we should disable tests because they fail for you, but what
> about the ones that went ok for you and failed for another user ?
Yes, disable those tests, too. Or mask the package on the "broken" platforms. If the packages cannot build because of some --enable-wombats feature, then either that feature is disabled or the package must be masked. If a package cannot build because of flaky tests, then those tests need to be disabled.
This instance in particular is a wonderful illustration of just how dangerous it is to insist that any time a package has tests that Portage must run them. As several other people have pointed out, the script can test for identical images, but the final verification needs to be done interactively by the user's highly calibrated eyeball.
If you leave broken tests in a package, they mean nothing, whether they pass or fail. The purpose of RESTRICT=test is to prevent broken tests from breaking Portage. Use it.