first, thanks for the the deblob USE flag. really a great feature for us trying to use Gentoo libre/FLOSS. i am having some problems with the deblob USE flag enabled on the latest vanilla Gentoo and RT kernels: sys-kernel/gentoo-sources-2.6.32-r7 (from portage) sys-kernel/librert-sources-2.6.31.6-r19 (from NJW overlay) sys-kernel/rt-sources-2.6.33.3-r19 (from pro-audio overlay) sys-kernel/rt-sources-2.6.33.5-r23 (from pro-audio overlay) i tried the deblob keyword the above kernels from portage and their respective overlays, and while the non-libre (i.e. -deblob) versions build fine, the libre (i.e. deblob) versions i have yet to compile. i get errors like these below: LD drivers/bluetooth/built-in.o CC [M] drivers/bluetooth/hci_vhci.o CC [M] drivers/bluetooth/btmrvl_main.o CC [M] drivers/bluetooth/btmrvl_debugfs.o CC [M] drivers/bluetooth/hci_ldisc.o CC [M] drivers/bluetooth/hci_h4.o CC [M] drivers/bluetooth/hci_bcsp.o CC [M] drivers/bluetooth/hci_ll.o LD [M] drivers/bluetooth/hci_uart.o CC [M] drivers/bluetooth/bpa10x.o CC [M] drivers/bluetooth/btusb.o CC [M] drivers/bluetooth/btsdio.o LD [M] drivers/bluetooth/btmrvl.o make[2]: *** No rule to make target `drivers/bluetooth/btmrvl_sdio.c', needed by `drivers/bluetooth/btmrvl_sdio.o'. Stop. make[1]: *** [drivers/bluetooth] Error 2 make: *** [drivers] Error 2 CC [M] drivers/media/video/cx25840/cx25840-core.o CC [M] drivers/media/video/cx25840/cx25840-audio.o make[4]: *** No rule to make target `drivers/media/video/cx25840/cx25840-firmware.o', needed by `drivers/media/video/cx25840/cx25840.o'. Stop. make[3]: *** [drivers/media/video/cx25840] Error 2 make[2]: *** [drivers/media/video] Error 2 make[1]: *** [drivers/media] Error 2 make: *** [drivers] Error 2 LD drivers/net/built-in.o CC [M] drivers/net/mii.o CC [M] drivers/net/ipg.o CC [M] drivers/net/jme.o CC [M] drivers/net/sis190.o CC [M] drivers/net/yellowfin.o CC [M] drivers/net/ns83820.o make[2]: *** No rule to make target `drivers/net/bnx2.c', needed by `drivers/net/bnx2.o'. Stop. make[1]: *** [drivers/net] Error 2 make: *** [drivers] Error 2 in the case of the Bluetooth error, disabling Bluetooth in .config solves that for now. i have been trying to track down the various config options to disable, but hunting via "menu makeconfig", i have not been able to disable enough options to let the kernel build yet. i am also fairly sure that my hardware is libre compatible, since it works with the latest libre kernels for Trisquel, as well as the original non deblob version of sys-kernel/librert-sources-2.6.31.6-r19, which is already deblob'd. i read the log for the deblob'd kernels install (via elogv), and i could not find where to post bugs/errors exactly relating to the deblob script. a quick google search also turned up nothing on the errors. thanks in advance. peace # emerge --info Portage 2.1.8.3 (default/linux/amd64/10.0, gcc-4.4.3, glibc-2.10.1-r1, 2.6.33.3-rt19 x86_64) ================================================================= System uname: Linux-2.6.33.3-rt19-x86_64-Intel-R-_Core-TM-2_Duo_CPU_P8700_@_2.53GHz-with-gentoo-1.12.13 Timestamp of tree: Thu, 17 Jun 2010 14:15:02 +0000 ccache version 2.4 [enabled] app-shells/bash: 4.0_p37 dev-java/java-config: 2.1.10 dev-lang/python: 2.6.5-r2, 3.1.2-r3 dev-util/ccache: 2.4-r7 dev-util/cmake: 2.6.4-r3 sys-apps/baselayout: 1.12.13 sys-apps/sandbox: 1.6-r2 sys-devel/autoconf: 2.13, 2.65 sys-devel/automake: 1.6.3, 1.8.5-r3, 1.9.6-r2, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.3-r2 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6b virtual/os-headers: 2.6.30-r1 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=core2 -mmmx -msse -msse2 -msse3 -msse4.1 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=core2 -mmmx -msse -msse2 -msse3 -msse4.1 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests ccache distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch" GENTOO_MIRRORS="http://gentoo.wheel.sk/" LANG="it_IT.UTF-8" LDFLAGS="-Wl,-O1" LINGUAS="en it fr es" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/pro-audio /var/lib/layman/njw /usr/local/portage" SYNC="rsync://rsync.fr.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi aim alsa amd64 audiofile avahi bash-completion berkdb bluetooth bzip2 caps cdda cddb cdparanoia cdr cli cracklib crypt css cups cxx dbus deblob dirac dri dssi dts dv dvd dvdr eds emacs encode evo exif ffmpeg fftw flac fontconfig ftp gdbm gif gimp gmp gnome gnome-keyring gnutls gpg gphoto2 gpm gsm gstreamer gtk hal hddtemp iconv icu ieee1394 imap iptc ipv6 jabber jack jackmidi jbig jpeg jpeg2k ladspa lash lcms libnotify libsamplerate lirc lm_sensors lv2 lzo mad matroska midi mime mmx mng modules mp3 mp4 mpeg mtp mudflap multilib musicbrainz nautilus ncurses networkmanager nls normalize nptl nptlonly nsplugin offensive ogg opengl openmp osc oscar pam pcre pdf perl png policykit pppd pulseaudio python quicktime raw readline reflection samba scanner schroedinger session skey smp sndfile speex spell spl sse sse2 sse3 sse4_1 ssl startup-notification svg sysfs taglib tcpd theora threads tiff tracker truetype unicode usb v4l2 vcd vnc vorbis wav wavpack wifi wmf x264 xattr xcb xcomposite xface xft xinerama xml xmp xorg xpm xv xvid yahoo zeroconf zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="canon" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev synpatics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en it fr es" LIRC_DEVICES="devinput" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS Reproducible: Always Steps to Reproduce: 1. USE="$USE deblob" emerge -av rt-sources 2. make Actual Results: various compile errors when kernels are emerged with deblob USE flag. note that so far, the same errors occur on all kernel versions listed above. for example: LD drivers/bluetooth/built-in.o CC [M] drivers/bluetooth/hci_vhci.o CC [M] drivers/bluetooth/btmrvl_main.o CC [M] drivers/bluetooth/btmrvl_debugfs.o CC [M] drivers/bluetooth/hci_ldisc.o CC [M] drivers/bluetooth/hci_h4.o CC [M] drivers/bluetooth/hci_bcsp.o CC [M] drivers/bluetooth/hci_ll.o LD [M] drivers/bluetooth/hci_uart.o CC [M] drivers/bluetooth/bpa10x.o CC [M] drivers/bluetooth/btusb.o CC [M] drivers/bluetooth/btsdio.o LD [M] drivers/bluetooth/btmrvl.o make[2]: *** No rule to make target `drivers/bluetooth/btmrvl_sdio.c', needed by `drivers/bluetooth/btmrvl_sdio.o'. Stop. make[1]: *** [drivers/bluetooth] Error 2 make: *** [drivers] Error 2 CC [M] drivers/media/video/cx25840/cx25840-core.o CC [M] drivers/media/video/cx25840/cx25840-audio.o make[4]: *** No rule to make target `drivers/media/video/cx25840/cx25840-firmware.o', needed by `drivers/media/video/cx25840/cx25840.o'. Stop. make[3]: *** [drivers/media/video/cx25840] Error 2 make[2]: *** [drivers/media/video] Error 2 make[1]: *** [drivers/media] Error 2 make: *** [drivers] Error 2 LD drivers/net/built-in.o CC [M] drivers/net/mii.o CC [M] drivers/net/ipg.o CC [M] drivers/net/jme.o CC [M] drivers/net/sis190.o CC [M] drivers/net/yellowfin.o CC [M] drivers/net/ns83820.o make[2]: *** No rule to make target `drivers/net/bnx2.c', needed by `drivers/net/bnx2.o'. Stop. make[1]: *** [drivers/net] Error 2 make: *** [drivers] Error 2 Expected Results: kernel should be compiled and linked. deblob is a new USE flag, discussed in this bug report: http://bugs.gentoo.org/show_bug.cgi?id=266157
Can you please attach your .config?
Created attachment 235791 [details] linux-2.6.33.5-rt23 .config file for real time deblob'd kernel linux-2.6.33.5-rt23 .config file for real time deblob'd kernel
(In reply to comment #1) > Can you please attach your .config? > OK, attached. i apologize in advance for missing something simple, and i am still trying to eliminate options from the config file that are causing the dependencies. i am definitely a kernel building newbie. however, if i understand the deblob script correctly, it should remove any CONFIG options that are not valid on a libre kernel, no? thanks again.
Yup, deblob should remove those options. Did you modify that config before posting it? CONFIG_BT is off, yet you showed me output of the bluetooth directory compiling. Try setting the following on the config that originally failed and see if that helps you: CONFIG_BNX2=n CONFIG_BT_MRVL=n CONFIG_BT_MRVL_SDIO=n
I haven't tried the overlay kernels, but I tested gentoo-sources-2.6.32-r7 with your posted config, and it compiled fine with the following change: CONFIG_CNIC=n Hope you manage to get it working!
(In reply to comment #4) > Yup, deblob should remove those options. > sorry for slow reply, limited Internet access in spurts. > Did you modify that config before posting it? > CONFIG_BT is off, yet you showed me output of the bluetooth directory > compiling. yes, i am sorry. i have been trying various config files and options and picked the wrong one. i included the original Bluetooth errors since that was the first CONFIG error i encountered. > > Try setting the following on the config that originally failed and see if that > helps you: > CONFIG_BNX2=n > CONFIG_BT_MRVL=n > CONFIG_BT_MRVL_SDIO=n > setting those along with CONFIG_CNIC=n allowed both the Gentoo and RT kernels to compile without problems. thanks for the quick help. the new kernels seems to be working well. now should i mark this bug as fixed? i assume there is a problem with the deblob script still, since it leaves in these options. please let me know how to proceed, first Gentoo bug report ;) peace
The eclass should add deblob-check to SRC_URI and copy it along with the main deblob script. For several config options, including this one, if deblob-check is not present, required files are removed, while if deblob-check is present, the files are patched to only remove some parts of it.
Harald, should we file a new bug covering the situation with missing deblob-check? Actually, the absence of this script makes a deblobbed kernel somewhat unusable.
My comment was a suggested fix for this bug, so I don't see a need to open a new one.
Well, I just wanted to emphasize that a quick fix implying turning off some features is not a fix at all. A lot of hardware should work without blobs, but will not work without deblob-check. Say, it includes Realtek 816* ethernet cards and even Nouveau AFAICS! This makes people believe that deblobbing makes most of their hardware completely useless, which is not a good thing (and not true either).
(In reply to comment #7) > The eclass should add deblob-check to SRC_URI and copy it along with the main > deblob script. For several config options, including this one, if deblob-check > is not present, required files are removed, while if deblob-check is present, > the files are patched to only remove some parts of it. (In reply to comment #10) > Well, I just wanted to emphasize that a quick fix implying turning off some > features is not a fix at all. A lot of hardware should work without blobs, but > will not work without deblob-check. Say, it includes Realtek 816* ethernet > cards and even Nouveau AFAICS! > > This makes people believe that deblobbing makes most of their hardware > completely useless, which is not a good thing (and not true either). Just as background, the deblob-check script strips out non-free sections of code files, which takes more time, but ensures free code in the same files as non-free code isn't stripped out. The alternative, which the deblob option uses currently, is to just remove any source file containing non-free code. This is much faster, but can leave things broken. The reason I didn't add the deblob-check script to the ebuild initially was that it takes around 30 minutes to run on my machine (as opposed to 30 seconds without), and I was under the (apparently probably mistaken) impression that it wouldn't make a difference for most setups. I proposed in bug 266157 (comment 83) the possibility of a deblob-thorough flag, or we could just turn on deblob-check functionality permanently. What do people think? Either option would be pretty easy, I'd be happy to rewrite the eclass to do this (indeed I did when testing the possibility in the past), once we decide what's the preferred behaviour.
I think deblob-check must be used by default with the "deblob" USE flag. Deblobbing without deblob-check may be turned on using, say, "deblob-shallow" USE flag, but its usage should be discouraged, with explanation. Users have to realize that a lot of hardware (judging by deblob-check's size) will not work with quick deblobbing.
(In reply to comment #12) > I think deblob-check must be used by default with the "deblob" USE flag. > Deblobbing without deblob-check may be turned on using, say, "deblob-shallow" > USE flag, but its usage should be discouraged, with explanation. > > Users have to realize that a lot of hardware (judging by deblob-check's size) > will not work with quick deblobbing. Yes, I think it would be very useful to have both options, to accommodate users who might have otherwise been forced into non-free firmware to get their hardware working. Using deblob-check by default would also (I assume) eliminate these possible compilation errors. The name 'deblob-shallow' might be a bit misleading though, as it seems to imply that it would remove less proprietary code than the full deblob. That's the impression I get anyway. Maybe 'deblob-fast' would be better? I don't know if there's a character limit on descriptions, but how about something like this: deblob: Remove proprietary firmware blobs from the kernel sources. deblob-fast: Remove any kernel source files that contain proprietary firmware blobs. For better hardware support, use 'deblob' which removes blobs without destroying whole files. With these names, I think it's obvious that the full deblob is slower.
(In reply to comment #13) > (In reply to comment #12) > > I think deblob-check must be used by default with the "deblob" USE flag. > > Deblobbing without deblob-check may be turned on using, say, "deblob-shallow" > > USE flag, but its usage should be discouraged, with explanation. > > > > Users have to realize that a lot of hardware (judging by deblob-check's size) > > will not work with quick deblobbing. > > Yes, I think it would be very useful to have both options, to accommodate users > who might have otherwise been forced into non-free firmware to get their > hardware working. Using deblob-check by default would also (I assume) eliminate > these possible compilation errors. > > The name 'deblob-shallow' might be a bit misleading though, as it seems to > imply that it would remove less proprietary code than the full deblob. That's > the impression I get anyway. Maybe 'deblob-fast' would be better? > > I don't know if there's a character limit on descriptions, but how about > something like this: > > deblob: Remove proprietary firmware blobs from the kernel sources. > deblob-fast: Remove any kernel source files that contain proprietary firmware > blobs. For better hardware support, use 'deblob' which removes blobs without > destroying whole files. > > With these names, I think it's obvious that the full deblob is slower. > i think the above descriptions/etc. make sense as well. with a warning that the deblob-check will be slower, i think it is better to have the default "deblob" USE flag run the full deblob-check, to avoid the problems already discussed. the "deblob-fast" option can be another USE flag, however, if it only works for a limited number of cases, it could present a bad impression of the libre/deblob'd kernel to people trying to make Gentoo more libre (as discussed above). thanks again for everyone's help.
(In reply to comment #14) > The "deblob-fast" option can be another USE flag, however, if it > only works for a limited number of cases, it could present a bad impression of > the libre/deblob'd kernel to people trying to make Gentoo more libre (as > discussed above). Exactly. I suggest the following USE flags: deblob: Remove binary blobs from kernel sources to provide libre license compliance. deblob-crude: Crude but fast version of "deblob", use it only if you know what you are doing. I think it's not necessary to explain that kernel compilation may fail and some hardware will be disabled without a strong reason. This would be too long for the flag description. If someone is going to build a kernel, I believe he/she would better wait 30 minutes more to make everything right :-) I'm sure that majority of users *will not* be satisfied with fast deblobbing.
*** Bug 327849 has been marked as a duplicate of this bug. ***
(In reply to comment #15) > (In reply to comment #14) > > The "deblob-fast" option can be another USE flag, however, if it > > only works for a limited number of cases, it could present a bad impression of > > the libre/deblob'd kernel to people trying to make Gentoo more libre (as > > discussed above). > > Exactly. I suggest the following USE flags: > > deblob: Remove binary blobs from kernel sources to provide libre license > compliance. > deblob-crude: Crude but fast version of "deblob", use it only if you know what > you are doing. > > I think it's not necessary to explain that kernel compilation may fail and some > hardware will be disabled without a strong reason. This would be too long for > the flag description. > > If someone is going to build a kernel, I believe he/she would better wait 30 > minutes more to make everything right :-) I'm sure that majority of users *will > not* be satisfied with fast deblobbing. > any news on the decision made on how the deblob use flag will work (see above for info on deblob and deblob-crude) in the future and when the changes will be made? peace/thanks
(In reply to comment #17) > any news on the decision made on how the deblob use flag will work (see above > for info on deblob and deblob-crude) in the future and when the changes will be > made? > > peace/thanks I've been meaning to create a patch for the eclass for a few weeks, but haven't got around to it yet. I'll give it a go tonight. As for the naming of different deblob flags, I'll leave that to others. Though I'm starting to feel that maybe we should just always use deblob-check - two separate flags seems like more places for bugs and confusion.
(In reply to comment #18) > (In reply to comment #17) > > any news on the decision made on how the deblob use flag will work (see above > > for info on deblob and deblob-crude) in the future and when the changes will be > > made? > > > > peace/thanks > > I've been meaning to create a patch for the eclass for a few weeks, but haven't > got around to it yet. I'll give it a go tonight. no worries. whenever you can get the time. just wondering if there was some process for deciding how it would work and where that process was. > > As for the naming of different deblob flags, I'll leave that to others. Though > I'm starting to feel that maybe we should just always use deblob-check - two > separate flags seems like more places for bugs and confusion. i personally agree, since we always want the libre kernels to work with as much hardware as possible. just need a big warning for people during the emerge if possible. peace >
Created attachment 238509 [details] kernel-2.eclass modified to use deblob-check Here we go. This eclass does what we want. I'll attach just the patch in a minute. I'm using the if EAPI=2 use '->' for deblob-check, so that later versions of the script (if changed) are unlikely to be kept around. Other than that it should be nice and straightforward.
Created attachment 238511 [details, diff] kernel-2.eclass modified to use deblob-check [diff] As above, only a diff of the current kernel-2.eclass in the tree. So it's easy to see the changes.
(In reply to comment #21) > kernel-2.eclass modified to use deblob-check [diff] > > As above, only a diff of the current kernel-2.eclass in the tree. So it's easy > to see the changes. If different kernel versions with EAPI<2 are in the tree, then it will not be possible to generate a correct Manifest file, because the different deblob-check versions all download to the same file. Meaning the way that's done in your suggestion won't work. Some alternatives: the eclass could keep the current behaviour for EAPI<2, the eclass could require EAPI>=2, or the eclass could specify mirror://gentoo/ URIs for EAPI<2. The cp and chmod calls should have || die added (preferably even the one you didn't write); outdated metadata cache because of eclass updates can cause files such as deblob-check not to appear in $DISTDIR, and without || die, the installation will seem to finish successfully. That said, the patch seems like the right direction to take, to me.
Thanks alot for the comments Harald. As for choosing the best path for EAPI, I'll leave that to the folks more intimately aquainted with the best ways for Gentoo. The only thing I'd say is that I wouldn't trust the deblob-check script to be the same between kernel versions, so it doesn't seem wise just to stick with calling it deblob-check (as, if I'm not mistaken, portage won't re-download the script if an old one exists, and the integrity check would fail.)
*** Bug 325203 has been marked as a duplicate of this bug. ***
Created attachment 241059 [details, diff] kernel-2.eclass modified to use deblob-check [diff] [2nd try] Does this look good to people? It now uses mirror://gentoo as deblob-check uri if EAPI < 2, and adds appropriate die functionality. If not, do speak up, as the current version is causing problems for quite a few people.
Thanks for the update. One small comment: EAPI needn't be numeric and there needn't be any easy ordering. Replacing if [[ $EAPI -ge 2 ]] ; then with if ! has "${EAPI:-0}" 0 1 ; then it looks good to me though. Kernel folks, any objections from you? I'd like to see this fixed too, I'll be happy to give it an extra check and commit this if you want.
Testing it out now, to go with the 2.6.35 deblob update. Will probably commit in the morning.
Committed to kernel-2.eclass and all required kernel-package (ck,gentoo,hardened,mips,tuxonice,vanilla,xen,zen) Manifests updated.
