Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 118106

Summary: trouble with `make test` phase of x11-libs/cairo-1.0.2
Product: Gentoo Linux Reporter: Drake Wyrm <lilwyrm>
Component: New packagesAssignee: Doug Goldstein (RETIRED) <cardoe>
Severity: normal CC: abraham, as.gentoo, bugs+gentoo, cems, gnome, hramrach, sascha-gentoo-bugzilla
Priority: High    
Version: 2005.0   
Hardware: All   
OS: Linux   
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 117505    
Attachments: The full log of the failed `make check`.

Description Drake Wyrm 2006-01-06 14:56:18 UTC
During a recent system update, I attempted to emerge x11-libs/cairo-1.0.2, as it is now a dependency of x11-libs/gtk+-2.8 and x11-libs/pango-1.10. Cairo successfully builds (so I do not believe this is related to bug #110180), but the build fails during the `make test` phase. It seems that the cairo developers specifically mark and ignore those tests expected to fail, so this may indicate a more significant problem.

If not, it may be appropriate to tag this ebuild with RESTRICT=maketest.

The build I am attempting:

Calculating dependencies    ...done!
[ebuild  N    ] x11-libs/cairo-1.0.2  +X +doc -glitz +png 0 kB 

The current state of my machine:

Gentoo Base System version 1.6.13
Portage 2.0.53 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r2, 2.6.14-gentoo-r5 i686)
System uname: 2.6.14-gentoo-r5 i686 AMD Athlon(tm) processor
dev-lang/python:     2.4.2
sys-apps/sandbox:    1.2.12
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils:  2.16.1
sys-devel/libtool:   1.5.20
virtual/os-headers:  2.6.11-r2
CFLAGS="-march=athlon-tbird -Os -pipe"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/bind /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-march=athlon-tbird -Os -pipe"
FEATURES="autoconfig buildpkg distlocks fixpackages maketest sandbox sfperms strict test"
MAKEOPTS="-j2 -w"
PORTDIR_OVERLAY="/var/cache/portage/boxed-main /usr/local/portage"
USE="x86 3dnow X aalib acl alsa bash-completion berkdb cjk crypt dga directfb doc esd fbcon flac gdbm gif gnome gpm gtk gtk2 ipv6 java jpeg ldap mad mbox mmx mpeg mysql ncurses nls offensive ogg oggvorbis opengl oss pam perl png python readline sdl skey slang ssl svga tcltk tcpd tetex tiff truetype unicode xv zlib userland_GNU kernel_linux elibc_glibc"

Error below pruned for sake of brevity; I will attach the full moby in a moment.

>>> Test phase [enabled]: x11-libs/cairo-1.0.2
>>> Test phase [check]: x11-libs/cairo-1.0.2


make  check-TESTS
make[2]: Entering directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'


22 of 60 tests failed
Please report to
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/var/tmp/portage/cairo-1.0.2/work/cairo-1.0.2/test'
make: *** [check-recursive] Error 1
Comment 1 Drake Wyrm 2006-01-06 14:59:09 UTC
Created attachment 76402 [details]
The full log of the failed `make check`.

This is also pruned a bit. Let me know if the full configure and make log might be helpful.
Comment 2 John N. Laliberte (RETIRED) gentoo-dev 2006-01-06 15:25:54 UTC
just to add a comment, the gnome policy in the past has been this:
Comment 3 Jakub Moc (RETIRED) gentoo-dev 2006-01-06 15:44:51 UTC
*** Bug 118112 has been marked as a duplicate of this bug. ***
Comment 4 Drake Wyrm 2006-01-06 15:48:57 UTC
On closer examination of the error, I noticed that the only tests which failed were the *-xlib tests. This is the ebuild which satisfies the virtual/xft RDEPEND on my system:

[ebuild   R   ] x11-base/xorg-x11-6.8.2-r6  -3dfx +3dnow +bitmap-fonts +cjk -debug -dlloader -dmx +doc +font-server -insecure-drivers +ipv6 -minimal +mmx +nls -nocxx +opengl +pam +sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint +xv 45,146 kB 

allanonjl: In bug #117505, ferdy mentioned that he had trouble with the tests while building gtk+, but you mentioned that you didn't have any such problems. Which version of X are you using?
Comment 5 Simon Stelling (RETIRED) gentoo-dev 2006-04-01 01:01:39 UTC
   When cairo is compiled, you can also run some automated tests of
   cairo with:

        make check

   NOTE: Some versions of X servers will cause the -xlib tests to
   report failures in make check even when cairo is working just
   fine. If you see failures in nothing but -xlib tests, please
   examine the corresponding -xlib-out.png images and compare them to
   the -ref.png reference images (the -xlib-diff.png images might also
   be useful). If the results seem "close enough" please do not report
   a bug against cairo as the "failures" you are seeing are just due
   to subtle variations in X server implementations.

Looking at the log, it is exactly the -xlibs test that fail, so they are most probably just false positives. Since upstream says their test-suite is overly strict and there's not an easy way to fix it, best idea might be to restrict it. 
Comment 6 Jakub Moc (RETIRED) gentoo-dev 2006-04-15 03:50:08 UTC
*** Bug 130016 has been marked as a duplicate of this bug. ***
Comment 7 Abraham Marin Perez 2006-04-20 08:04:46 UTC
I'm also having similar problems merging cairo-1.0.4. In my case there is one test failing (aside from the expected fails, which I don't take into account). Taking a closer look I see the failing test is an -xlib one, so I guess it'll be safe for now to restrict test and do with it.

However, I don't think restricting tests is a good practice, since tests are there for a particular purpose. Maybe we could just black out tests with a tendency for false positives but let the others to be run. I am not a dev and I don't pretty know whether this would mean much work or even whether it would feasible, but I think it'd be a good idea.
Comment 8 Carlos Eduardo Santos 2006-04-22 11:32:43 UTC
I resynced yesterday, updated everything but cairo (emerge --resume --skipfirst). Lastly I tried to update cairo to 1.0.4 and it worked!
Comment 9 Abraham Marin Perez 2006-04-23 08:45:08 UTC
(In reply to comment #8)
> I resynced yesterday, updated everything but cairo (emerge --resume
> --skipfirst). Lastly I tried to update cairo to 1.0.4 and it worked!

It seems you were luckier than me, because I proceeded exactly like you and it didn't work... Still having the same problem.
Comment 10 Simon Stelling (RETIRED) gentoo-dev 2006-07-14 08:22:59 UTC
after checking with Doug, I put RESTRICT=test into the latest stable (testing already had it), so this is fixed now :)
Comment 11 foser (RETIRED) gentoo-dev 2006-07-15 03:15:23 UTC
restrict=test is plain pointless. If you decide to run tests for your build, then disabling tests in the ebuild is just hiding the problem from the users who chose to use it.

Sure, some tests in cairo at this point will always fail, but that is what you want to know when setting FEATURES=test.

This is NOT fixed in any way, it is only made worse.
Comment 12 Doug Goldstein (RETIRED) gentoo-dev 2006-07-15 07:47:01 UTC
Foser, take your rhetoric else where. Tests are expected to fail. You're suppose to manually run tests and then compare the PNGs visually by hand. Ask Carl.
Comment 13 Attila Stehr 2006-07-15 09:37:02 UTC
(In reply to comment #12)
> Foser, take your rhetoric else where.
This sounds quite rude for me! 
Anyway I agree to foser - althought I doubt that this is of much interest for some devs...
Comment 14 Michal Suchanek 2006-07-17 04:55:15 UTC

When I build with tests on it is because I want to know that the package was built correctly (ie was not broken by compiler optimizations or bad memory). Tests that are known to fail, and are not disabled are annoying and useless from this point of view. In fact, they are so widespread that they make this feature virtually useless for me.

This test would need much of work to work correctly - compare the images, and decide wether the difference is "small enough" or not.

If might be possible to disable the tests that are known to fail with a patch, and still perform some tests on the parts of the library that are expected to do exact rendering.
Comment 15 foser (RETIRED) gentoo-dev 2006-07-17 15:29:13 UTC
Know that the results of these tests differ per platform and backends used. So now we should disable tests because they fail for you, but what about the ones that went ok for you and failed for another user ?

I'm not all that concerned with users who think gentoo is about compiling as much code as possible, I'm concerned with people who want to run tests for valid reasons and are handicapped by these kind of 'tricks'. It's not just cairo, cairo is just an example.
Comment 16 Michal Suchanek 2006-07-18 04:51:26 UTC
foser, if you look at comment #5 there is a citation (supposedly from some cairo docs) that says failures in xlib tests are expected. 

If the tests failed only for me it would be my problem and not a gentoo bug.
However, these tests do not fail only for me, they fail for a lot of people. They even fail so often they bothered to document the fact. It is a bug in the tests and should be resolved in some way. Disabling all tests is a bit crude but easy. If you want it fixed properly, why haven't you fixed the tests?

And yes, gentoo is about compiling stuff. It is even listed somewhere as its advantage. You are supposed to get packages with the options you want that way.

If running tests to make sure my package was built properly is not a valid reason what is?
Comment 17 Drake Wyrm 2006-07-18 10:01:47 UTC
> So now we should disable tests because they fail for you, but what
> about the ones that went ok for you and failed for another user ?

Yes, disable those tests, too. Or mask the package on the "broken" platforms. If the packages cannot build because of some --enable-wombats feature, then either that feature is disabled or the package must be masked. If a package cannot build because of flaky tests, then those tests need to be disabled.

This instance in particular is a wonderful illustration of just how dangerous it is to insist that any time a package has tests that Portage must run them. As several other people have pointed out, the script can test for identical images, but the final verification needs to be done interactively by the user's highly calibrated eyeball.

If you leave broken tests in a package, they mean nothing, whether they pass or fail. The purpose of RESTRICT=test is to prevent broken tests from breaking Portage. Use it.