Bugzilla Bug 32737

Openning a new meta-bug for dealing with gentoo's handling of external modules
compiling against 2.6 kernels. In particular, the suggested method in makefiles
of doing
make -C /usr/src/linux SUBDIRS=/some/path/to/external/modules/ modules

fails to succeed since it ends up need to write to /usr/src/linux at some point,
and sandbox kills it.

For now, any ebuilds which need to do this can just have sandbox disabled, but a
real fix needs to be worked on. Elementary support for outputing to a different
directory via KBUILD_OUTPUT and O options to the make line seem to be the right
way to go, but that still has issues.

Ebuilds so far that i know have issues:
hostap (old)/ hostap-driver

Please use this ebuild to discuess any particular ebuilds that suffer from this,
or more generally a fix that will work for all ebuilds that need to add external
modules (that use the above make method)

Reproducible: Always
Steps to Reproduce:
1.
2.
3.

could we perhaps start pushing these kind of ebuilds through an eclass?
am I misunderstood or is kernel-mod.eclass designed for this?
it will of course need work for 2.6 no doubt, an my appologies for not looking
at the eclass properly before posting this.

however regardless, I do feel this would be the more correct solution.

Yeah, I was definitely thinking an appropriate eclass would be the best
solution
for this. Just need to find out what foo we need to exert on the kbuild stuff
from 2.6 to get around these sandbox errors.

xfree-drm, ati-drivers, ati-gatos are candidates. I'm not sure about
nvidia-kernel.

you could look at the script that the cisco-vpnclient-3des client uses

Created an attachment (id=23927) [edit]
patch kernel-mod.eclass

It's not a solution, but until we find something good, i think an addition to
kernel-mod.eclass like this patch would be handy. Adds the "check_module_safe"
function which will die with verbose errors if someone is attemtping to install
kernel modules with sandbox or usersandbox enabled.

Anybody using this in ebuilds will have to be *very* sure that the ebuild only
touches /usr/src/linux, cause we *really* don't want to tell users to disable
sandbox to emerge something then have it do Bad Things (tm).

What do people think of this to at least inform users of what's going on?

UPDATE: Okay, after pounding my head against this issue for at least a month, i
think i've finally found a viable, moderately sexy solution. I'm gonna post the
general concept here now, and patch to a few eclasses and ebuilds to
demonstrate will be forthcoming to both the gentoo-dev ML and this bug.

PROBLEM: sandbox issues!

FURTHER PROBLEM: setting O=/some/tmp/path just makes the kbuild stuff complain
that you need to clean out your /usr/src/linux by doing "make mrproper:

REAL PROBLEM: fighting the kbuild stuff instead of "embracing and extending"

PROPOSED SOLUTION: the new 2.6 stuff has two different variables used in
deciding where output files end up, KBUILD_OUTPUT and the O that has been
dicsussed before. O is used for changing the output on the command line, etc.
KBUILD_OUTPUT can be defined in /usr/src/linux/Makefile and sets the default
output for *all* files built. that means .config, include/linux/version.h, etc.

I'm suggesting that we add to johnm's new kernel-2.eclass to add some sedding
to set KBUILD_OUTPUT=/some/to/be/decided/standard/path for all new 2.6 kernels.

REPURCUSIONS:

1) We need to decide on where things would go (i was thinking
/var/lib/kernel-output/${KV} maybe?)

2) DOCS!!!!  This is a pretty major change, so we'll need to keep users
informed of what's changed, why it's used, etc.

3) check_KV would need fixing cause it keeps thinking include/linux/version.h
should be in /usr/src/linux/ (it's generated, so it ends up in KBUILD_OUTPUT)

4) Driver ebuilds will need to use the "make -C /usr/src/linux
SUBDIRS=some/path O=some/tmp/dir modules" (most use -C, and SUBDIRS already,
but not necessarily the O= stuff). Some also need some makefile editing to look
for .config, etc in a different location (this may be the hard part) I'm hoping
these types of changes can be generalized and sent upstream so we'll stop
having to worry about lots of patching, etc. I've got hostap-driver working by
hand with some makefile changes, i'll be getting my local ebuild using it
jiving soon.

5) Anybody can compile a kernel from the same code base! joeuser can just do
"make O=/home/joeuser/kernel-output/ menuconfig", configure his kernel, then do
"make O=/home/joeuser/kernel-output/" and he's compiled a kernel.

6) Unmerging a kernel actually would remove the whole directory (and you'd be
able to safely zap one directory to clean things out in KBUILD_OUTPUT, etc.


Okay, if you've made it this far... Comments? It's a big change, but i think
we're going to have to embrace this stuff or it's just going to get uglier and
uglier trying to get things working. Please post your ideas here, i'll have
example patches, ebuilds, etc coming shortly.

p.s. *phew*

I like the idea, but I'd suggest /var/tmp or /var/cache based on FHS.

/var/lib:
This hierarchy holds state information pertaining to an application or the system. State information is data that programs modify while they run, and that pertains to one specific host. Users must never need to modify files in /var/lib to configure a package's operation.

/var/tmp:
The /var/tmp directory is made available for programs that require temporary files or directories that are preserved between system reboots.

/var/cache:
/var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system.

Good job tracking down a solution on this latexer. I'd suggest we send an
emissary to LKML to get some input on paths and everything. There may be some
standard already in use for other distros that makes sense to follow (although
I've not heard of anyone else using this in standard practice). It would be
nice if we could be the initators of this standard rather than having to follow
some corporate lead set in a boardroom.

donnie: okay, yeah /var/tmp or /var/cache does sound better. /var/cache sounds
good except for maybe the "The data must remain valid between invocations of
the application and rebooting the system" part which isn't necessarily true
even for a single boot as changing something like SMP causes pretty all the
object files to be recreated. /var/tmp seems more in line with gentoo's current
setup as /var/tmp/portage is our standard location. Personally i think i prefer
/var/tmp/<something> (i think either /var/tmp/kernel-output/${KV} or
/var/tmp/kernel-objs/${KV})

iggy: I'm not subscribed to LKML, but i can send an email briefly detailing
things if you want, asking for feedback.

Created an attachment (id=24094) [edit]
Patch to get KV set using /usr/src/linux/Makefile

This is a patch to /usr/lib/portage/pym/portage.py which makes portage use
/usr/src/linux/Makefile to set KV, and not
/usr/src/linux/include/linux/version.h

This is needed since version.h is generated (and ends up in KBUILD_OUTPUT)

This is 2.4 backwards compatible, since the makefiles use the same layout for
the first 4 lines (thank god/thor/someone for that)

Created an attachment (id=24095) [edit]
Patch to johnm's kernel-2.eclass to set KBUILD_OUTPUT in
/usr/src/linux/Makefile

This is the small bit that makes things get output to
/var/tmp/kernel-output/${KV}

It tells the user what it's doing, makes the directory, etc. Not much here, but
this ends up making the whole shebang interesting/messy/great/horrible

Patch then emerge mm-sources-2.6.1-mm4 to get a taste if you dare.
You can run "make menuconfig" as usual, but all the things output end up in
/var/tmp/kernel-output/${KV}

Created an attachment (id=24096) [edit]
new hostap-driver ebuild that uses O= foo to compile against clean 2.6 kernel
source

Here's a revised hostap-driver ebuild that now will work if you have a clean
2.6 kernel tree.

I've got linux-wlan-ng-0.2.1_pre16 *almost* working with this same method.

Created an attachment (id=24131) [edit]
pcmcia-cs-3.2.7 ebuild that has a fix for finding .config

Here's an ebuild for pcmcia-cs-3.2.7 that has a fix for finding a users .config


The patch is up at http://dev.gentoo.org/~latexer/files/patches/

It's unintrusive and general enough that i may send the patch upstream once i
make it a little prettier.

Created an attachment (id=24646) [edit]
More robust ExtractKernelVersion

Here's a more robust patch to portage.py which doesn't traceback if somebody
messes with the variables and doesn't have spaces between the "=" sign. (First
version was a biproduct of my naive familiarity with python)

Created an attachment (id=24647) [edit]
kernel-mod.eclass changes to make ebuilds easier with new system

This is a patch against kernel-mod.eclass which does a few things:

1) Set some new variable at peoples disposal -
   KV_OUTPUT - location of kernel output, can be used for both 2.4 and 2.6
kernels as it will be set to /usr/src/linux if KBUILD_OUTPUT isn't set in the
toplevel kernel Makefile.
   KV_OBJ - the extension for kernel objects. If the packages installation make
foo is too annoying, modules can be installed by hand using doinst
foo.${KV_OJB} so you don't have to worry about .o vs .ko

2) Add a kernel-mod_src_unpack for a new better default. If an ebuild needs to
do any patching, makefile editing, etc, they should still call this function in
order to set needed variables, check for a sane environment, etc. 

3) Along with the kernel-mod_src_unpack, a new variable KERNEL_MOD_SOURCES can
be set to all the sources that should be unpacked. This may be different from
${A} if you have patches, or conditionally unpacked things.


Comments? Additions? I'll be adding new versions of the hostap-driver and
pcmcia-cs ebuilds, as well as my working linux-wlan-ng ebuilds soon that use
the items in this new eclass

Created an attachment (id=24649) [edit]
Path to linux-wlan-ng ebuild (see details!)

Here is a patch to the 0.2.1_pre16 in portage that uses the new koutput stuff
with the latest revisions in kernel-mod.eclass.

NOTE: This will only work with recent -mm* series kernels. There was/is a bug
with includes in EXTRA_CFLAGS. Sam Ravnborg posted a patch to LKML a while ago
with the patch but it has not made its way into the vanilla kernel yet. That
patch is found here: http://lkml.org/lkml/2004/1/16/247

Created an attachment (id=24650) [edit]
Cleaner hostap-driver ebuild using new kernel-mod.eclass

Updated cleaner hostap-driver ebuild using the new features added to the
kernel-mod.eclass

Created an attachment (id=24651) [edit]
Cleaner updated pcmcia-cs ebuild

Same cleanups using new kernel-mod.eclass

Okay, I'm going to stop spamming the attachments with fixed ebuilds. Both the
linux-wlan-ng and hostap-driver ebuilds along with the patches they pull should
be suffient for people to look at to get a sense of what needs to be done on
the ebuild end of things.

I have ebuilds for *new* drivers prism54 and at76c503a up at:
http://dev.gentoo.org/~latexer/files/ebuilds/

On the documentation side, I've started work on a general 2.6 kernel guide that
will hopefully sufficiently cover the KBUILD stuff so that we can refer users
there when the change occurs.

If there is sufficient desire, I will also make up a guide for *developers*
detailing how the new system affects them, and how they can write ebuilds to
effectively use this new system. Would that be appreciated?

If I understand correctly you want to add a hole in sandbox to a
/var/??/kernel-output directory. Why is this any better than making that hole
/usr/src/linux? We do need a shared kernel tree in somewhere in any case. Or am
I missing something?

No no. this is actually a solution so we exactly *don't* have to do something
like that.

What is going on in a nutshell (i'll be writing an extensive doc aimed at
developers this weekend detailing all the pieces involved):

Kernel compiles output their stuff to a different directory
(/var/tmp/kernel-output/${KV})

External Modules use the kernel source in /usr/src/linux (and sometimes the
.config in the output directory) to compile but *OUTPUT* to a subdirectory of
their temp directory (this is done with the O=$(PWD)/tmp stuff in the
makefiles, see the patches downloaded from any of these ebuilds i've posted)
This allows us to build against a clean source tree, and not have sandbox
errors. Currently we get this error since it's trying to build using the kernel
sources AND the files already output to /usr/src/linux, it thinks it needs to
update things (make is a blessing and a curse) and BOOM! sandbox error. Having
a clean sources directory, and having external modules output to their own
location is the *key* to all this working.

All of the extra work is making sure packages that need to find .config,
version.h, etc don't die from these things being in a new location.

The document will hopefully be more detailed, but does this make sense for now?

ahh that's great.. didnt realise you had two separate output directories... But
I'm still worried about the un-vanillaness of the solution.. but I guess we
dont really have a choice..

If anybody has a more "vanilla" solution that gets things done cleanly, i'm all
for it. As this is just taking advantage of a new kernel build feature, i don't
see this as being *that* "unvanilla." If users feel like building to the same
directory, and disabling sandbox for their modules, all they need to do is edit
one line in a Makefile (i'll have that clearly described as an option in the
doc i've got going) and voila. The kernel-mod class stuff is flexible enough
that this is all it takes.

Added a dep on a new bug i opened specifically for the portage.py changes and
assigned to dev-portage. see bug #40099

Correct me if I'm wrong, but as I understand this solution now, for it to work,
essentially we have to insure that all new kernels are built using
O=/var/tmp/kernel-output/${KV}. That way the kernel source remains clean so
ebuilds containing kernel modules can also then output to their own tmp
directory inside the sandbox.

Do we plan on modifying the ebuilds for all the sys-kernel/*-sources to patch
the Makefile to make that the default output directory? Or modify genkernel?
How are we going to handle all the legacy kernel builds where people have
manually built their kernels.  Just add checks inside the ebuild and notify the
user of the proper Gentoo way of building a linux kernel?

Don't get me wrong, I think this is the best long term solution, but I want to
make sure it doesn't create too much of a hassle for users who built their own
kernels.

anti: All 2.6 kernels (there's not too many yet, fortunately) would need to use
johnm's new kernel-2.eclass (some/most already do). The changed eclass adds a
define for KBUILD_OUTPUT which is a variable for use in the Makefile, which can
be overriden by the O variable. Those ebuilds would not need to be changed in
any way. I was already planning on adding a check in the kernel-mod eclass
changes to spit out warnings if a 2.6 kernel is found that is built in the old
method. In theory we could do lots of extra work to *not* add the O= stuff in
the ebuild and give the user an option to disable sandbox to merge, but since
this is not guaranteed to work plus is a security problem i don't see any
strong reason for doing this.

To make this less of a hassle, i think strong docs and even an announcement on
www.gentoo.org about changes will help ease (at least somewhat) the transition
once we're ready.

Opened bug #40933 for documentation on this change. Already attached a
developer oriented document so hopefully i can get some more testing from
people. I'll be posting some revised patches soon for a few things, including
small changes to kernel-mod.eclass and a small change to the new pcmcia.eclass
and drivers that use it to make things even easier.

Created an attachment (id=25235) [edit]
Change pcmcia.eclass to use kernel-mod.eclass stuff

Patch to pcmcia.eclass to enabled using koutput by using variables from the
kernel-mod.eclass

Versions of linux-wlan-ng and hostap-driver using this eclass and updated
kernel-mod.eclass to follow soon.

Created an attachment (id=25237) [edit]
Newest version of kernel-mod.eclass patch

Added some checks to see if on 2.6 kernel but not using koutput stuff. Also
added use of variable KERNEL_MOD_KOUTPUT_PATCH which can be set if someone
needs just to patch the sources for koutput compatibility. (Potentially letting
someone not have to write their own src_unpack(), just set
KERNEL_MOD_KOUTPUT_PATCH and KERNEL_MOD_SOURCES)

Created an attachment (id=25238) [edit]
New linux-wlan-ng ebuild using latest {pcmcia,kernel-mod}.eclass changes

New version of the linux-wlan-ng ebuild using latest eclass changes. (I swear,
i'm almost done spamming this bug for the night)

Created an attachment (id=25239) [edit]
Patch to hostap-0.1.3.ebuild to use latest eclasses

Last one for the night. Just keeping these two up-to-date so people have
something to test with the latest work (*nudge* *nudge*)

Hi, I recently submitted an ebuild for prism54 that builds fine as an external
module in a sandbox. I use a build directory with contents symlinked to the
current kernel sources:

src_unpack() {
        check_KV
        unpack ${A}
        # Make our own build directory symlinked to the kernel sources
        mkdir ${WORKDIR}/build
        cd ${WORKDIR}/build
        SRC=/lib/modules/${KV}/build
        for f in ${SRC}/.config ${SRC}/.version ${SRC}/*; do ln -s $f .; done
}


src_compile() {
        cd ${S}
        make clean
        # hmm portage exports ARCH as x86?
        emake KDIR=${WORKDIR}/build ARCH=i386 V=1 modules || die
}

I expect this will work for other external modules as well. 

Interesting, i'd noticed this behavior in the ebuild, but hadn't had time to
investigate it fully. This might end up being easier in the long wrong (as much
as i hate to admit it after all this work i've done)

If this turns out to be the easy way out, do we want to do this, or use this
new method which gives us lots of other goodies but more complexity changing
the tree and educating users?

Okay, after taking a shower and thinking on it, this is *great*

Why? We can offer *both* so that users have maximum flexibility. Consider the koutput method a feature! kernel-mod.eclass can be relatively trivially modified to use this newly discovered method (if it turns out to work with other packages) if it sees ${KV_OUTPUT}=/usr/src/linux when on a 2.6 kernel. This would allow seemless use by people using the old method, and anyone looking for advanced stuff like:
/usr mounted readonly
Building by arbitrary users into their $HOME
Multiarch building with ease

CAN do that, just by using KBUILD_OUTPUT in their kernel makefile. I think i will also create some variable a user can set (mabye just use "O") so they can on the fly emerge things against a kernel config that is in an arbitrary directory they happened to build their kernel too.

If people don't wanna do it, they just use the old method with this symlinks magic (i'm not convinced it's magic yet, it still kinda "hacks" its way around the kbuild stuff)!

What do people think of *this*?

Created an attachment (id=25279) [edit]
Patch to kernel-mod.eclass dubbed "Have My Cake and Eat it Too"

Okay, after thinking that we can make *everyone* happy by allowing any of the
three options, 2.4, 2.6 normally, or 2.6 with files output elsewhere, i've
hacked up a new version of the kernel-mod.eclass patch.

*simpler* ebuild for prism54 and updated ebuild for hostap-0.1.3-r3 to follow!
linux-wlan-ng is mostly working, but when using the "make symlinks" version,
some funkiness happens with .tmp_versions

Created an attachment (id=25280) [edit]
hostap-driver-0.1.3-r1 ebuild that works with latest no matter what.

Using latest "Cake and eat it too" kernel-mod.eclass changes.

Created an attachment (id=25281) [edit]
New prism54 ebuild using "cake and eat it too" kernel-mod.eclass

Simpler prism54 using new kernel-mod.eclass stuff

Created an attachment (id=25282) [edit]
"Cake and eat it too" fixed to add a symlinks for .tmp_versions

Okay, only mildly modified to include a symlink for .tmp_versions when using
the symlinks style building.

linux-wlan-ng ebuild to follow.

Created an attachment (id=25283) [edit]
Latest linux-wlan-ng

Oh yeah.

Created an attachment (id=25363) [edit]
Updated 3.2.7-r1 ebuild to fix patch for monitor mode!

Just a small fix for the monitor mode patch

Created an attachment (id=25453) [edit]
"Cake and eat it too 2" - 2nd version of the "Cake and Eat it Too" style
kernel-mod.eclass patch

Okay, Here's an updated patch to kernel-mod.eclass. Below is s summary of *new*
changes (beyond what was added/altered in last patch)

1) KV_BUILD now puts the symlinks fake silliness in ${T}/kernel-build instead
of ${S}/kernel-build. Thankfully the abstraction of that layer means all
ebuilds from before should still work like a charm. This was done for packages
like nvidia-kernel which must do some unpacking by hand (they have an annoying
bash script) since ${S} won't be around. ${T} is really a better spot for this
temporary fake kernel build tree anyway.

2) Moved some of the stuff out of kernel-mod_src_unpack into seperate
functions. Ebuilds like nvidia-kernel which need to do more by hand now can use
these and not duplicate the work in the ebuilds themselves.

3) Fixed kernel-mod_getmakefilevar() so it removes anything returned with a
":="  in it. This is needed so we get "" when KBUILD_OUTPUT isn't actually
defined in the 2.6 makefiles.

4) Can now have KERNEL_MOD_SOURCES="none" if you don't want *anything* unpacked
by kernel-mod_src_unpack


In other news:
I searched for which ebuilds where using the old kernel-mod so I can see what
might break with these changes. So far looks like the few other old functions
that were lying around should be uneffected. Found a few ebuilds like
nvidia-kernel which were doing things like looking for /proc/mtrr (tsk tsk!)
but otherwise things should be mostly hunky dory on that front.

nvidia-kernel ebuild using latest stuff to follow

For those looking for an easy way to test things, here a quick and dirty
rundown:
Apply latest kernel-mod.eclass patch
Apply pcmcia.eclass patch
Download any of recent driver ebuilds or patches, or try on
http://dev.gentoo.org/~latexer/files/ebuilds for one or two not here, and
digest and try building!

Testing on all combinations, 2.4, 2.6 without any kernel output mucking, and
2.6 with kernel output location changed is all appreciated!

To change the kernel output by hand, just add
"KBUILD_OUTPUT=/var/tmp/kernel-output/<kernel version" in your toplevel
Makefile, then make sure you create that directory.

I'm more than willing to help via email or irc in #gentoo-laptop on freenode if
people want help or info on testing this!

Created an attachment (id=25454) [edit]
nvidia-kernel ebuild using latest foo!

Yes, it does work for drivers other than wireless ones! (:

Created an attachment (id=25497) [edit]
ati-drivers ebuild using new foo

Chalk up ati-drivers as another one working. Next is the beast that is
xfree-drm... this might get ugly.

Pete,
Keep in mind you'll have to essentially do it three times for xfree-drm: once with any video card but mach64, once with mach64, and once with USE=gatos and video cards radeon or r128. Watch the patches to see where the Makefile patch is re-used, if it is.

I applied the kernel-mod.eclass patch and tried the new prism54 ebuild. Firstly
the ebuild fails because the firmware doesn't exist on prism54.org anymore :-(
After adding it to distfiles manually I get:

>>> Unpacking prism54-cvs20040208.tar.bz2 to /var/tmp/portage/prism54-20040208-r1/work
 * Creating temporary build directory:
 *      /var/tmp/portage/prism54-20040208-r1/temp/kernel-build
 * using files in:
 *      /usr/src/linux
>>> Source unpacked.
make -C ksrc/ modules
make[1]: Entering directory
`/var/tmp/portage/prism54-20040208-r1/work/prism54-cvs20040208/ksrc'
make -C /var/tmp/portage/prism54-20040208-r1/temp/kernel-build
SUBDIRS=/var/tmp/portage/prism54-20040208-r1/work/prism54-cvs20040208/ksrc V=1
modules
make[2]: Entering directory
`/var/tmp/portage/prism54-20040208-r1/temp/kernel-build'
make -f scripts/Makefile.build obj=scripts
  gcc -Wp,-MD,scripts/.empty.o.d -nostdinc -iwithprefix include -D__KERNEL__
-Iinclude  -D__KERNEL__ -Iinclude  -Wall -Wstrict-prototypes -Wno-trigraphs
-fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2
-march=pentium3 -Iinclude/asm-i386/mach-default -O2 -fomit-frame-pointer
-Wdeclaration-after-statement    -DKBUILD_BASENAME=empty -DKBUILD_MODNAME=empty
-c -o scripts/empty.o scripts/empty.c
cc1: Permission denied: opening dependency file scripts/.empty.o.d
Assembler messages:
FATAL: can't create scripts/empty.o: Permission denied
make[3]: *** [scripts/empty.o] Error 1
make[2]: *** [scripts] Error 2
make[2]: Leaving directory
`/var/tmp/portage/prism54-20040208-r1/temp/kernel-build'
make[1]: *** [modules] Error 2
make[1]: Leaving directory
`/var/tmp/portage/prism54-20040208-r1/work/prism54-cvs20040208/ksrc'
make: *** [modules-all] Error 2

I have userpriv and usersandbox set. I also see a message "By not using the
kernel's ability to output to an alternative directory, some external module
builds may fail."

Hrm. odd. Check in /usr/src/linux/scripts, and see if you have all the needed
scripts. If those aren't built, this might be the issue. Is this with a
compiled kernel tree?

I fixed that problem with 'make all' in the kernel src, which seemed to want to
rebuild postmod (no idea why, this is from a freshly built kernel). Now I get:

make[1]: Entering directory
`/var/tmp/portage/prism54-20040208-r1/work/prism54-cvs20040208/ksrc'
make -C /var/tmp/portage/prism54-20040208-r1/temp/kernel-build
SUBDIRS=/var/tmp/portage/prism54-20040208-r1/work/prism54-cvs20040208/ksrc V=1
modules
make[2]: Entering directory
`/var/tmp/portage/prism54-20040208-r1/temp/kernel-build'
make -f scripts/Makefile.build obj=scripts
*** Warning: Overriding SUBDIRS on the command line can cause
***          inconsistencies
mkdir -p .tmp_versions
make -f scripts/Makefile.build obj=arch/i386/kernel
arch/i386/kernel/asm-offsets.s
make[3]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/asm-i386/asm_offsets.h
/bin/sh: line 1: include/asm-i386/asm_offsets.h.tmp: Permission denied
  UPD     include/asm-i386/asm_offsets.h
mv: cannot stat `include/asm-i386/asm_offsets.h.tmp': No such file or directory
make[2]: *** [include/asm-i386/asm_offsets.h] Error 1

This is with development-sources-2.6.3. From the kernel Makefile it looks as
though it does:

       echo '  CHK     $@';                    \
        mkdir -p $(dir $@);                     \
        $(filechk_$(1)) < $< > $@.tmp;          \
        if [ -r $@ ] && cmp -s $@ $@.tmp; then  \

which outputs stuff to a .tmp file. It may be necessary to create a more
complete directory structure in the kernel build directory and only symlink the
actual files (allowing the CHK to run in the build dir but using the original
kernel source files)

Pete: If you could add omnibook-20040120 to the list as well, that'd be great.
Thanks in advance.

Created an attachment (id=26588) [edit]
omnibook ebuild

Your wish is my command. (:


In other news:
After talking to johnm about a few things, it seems things will be behaving a
little differently.
1) Using the "symlink" method ultimately is no better than doing "addwrite
${ROOT}/usr/src/linux". However, this is a security no-no. I *really* don't
like doing this whatsoever. BUT, we need to have a relatively smooth upgrade
path for users (sigh, the users). We can't just one day have a user suddenly
unable to emerge any of the kernel module ebuilds they know and love. As a
temporary solution til we can get people using koutput, using kernel-mod.eclass
to uniformly do this addwrite, warn users about it, and most importantly FORCE
THEM to accept it initially is a Good Thing(tm). This can be done via a
"ALLOW_LINUX_WRITABLE" or somesuch variable found in /etc/env.d/20kernel (or
somewhere in env.d). To facilitate this, and a few other things, we'll come to
point 2.

2) kernel-config tool. *-config tools seem to be the latest rage. Why not have
on for some kernel configuration stuff. One use would be the above, e.g
# kernel-config allow-writable yes
Or somesuch. 
Other things would include configuring the prefix for koutput stuff (i'll be
submitting a new kernel-2.eclass patch which shows how this would be used.
All the info on what i've been mulling around with john about is found here:
http://dev.gentoo.org/~latexer/kernel-config-proposal

3) I'm really excited for this. No change here though.

Looks good, great!

I also agree that the kernel-config would help everyone, as copying your .config and also changing symlinks when doing testing gets very tiring. Regarding the security issue.

However, I think the best transparent solution would be to have a "pass-through" sandbox, which logs all write events in big bold red and writes straight to /usr/src/linux if it's needed but doesn't prohibit the write action: i.e. the user knows what the ebuild touched while at the same time the user's ebuild that he loved still works.

Pete, join the latest craze and make config-* tools. That way people can
actually find them with tab completion.

Created an attachment (id=26635) [edit]
New patch implementing things last discussed

Okay, here's a new patch for my latest work on the kernel-mod.eclass stuff.
This behaves pretty differently in a few areas, so old ebuilds posted here
probably won't work. I've got the changes for those mostly ready. From now on,
i'll be posting the latest ebuilds, patches, etc at:
http://dev.gentoo.org/~latexer/files/koutput/
so i don't keep spamming this bug with tons of ebuilds. Things that are
different in this version.

1) No longer does the symlink stuff. For simplicity's sake, and since it's a
mess no matter how we do it, this version adds kernel-mod_make_linux_writable()
which is used to intelligently make /usr/src/linux-${KV} writable. Forces the
variable LINUX_PORTAGE_WRITABLE=yes in order to do it. For now i still like the
user being made aware of this *once* before we do it automatically for them.

2) added is_koutput() command to more easily tell if we're on the new setup.
This is prefered over forcing ebuilds to check what KV_OUTPUT equals, etc.

3) Added "kernel-mod_getconfigvar() FOO" which prints the config value for
CONFIG_FOO from the current kernel config when called. 

4) Syntax fixes from spyderous. he's the man

5) Probably more i'm forgetting.

I'll be posting a new version of my kernel-2.eclass changes here, then putting
all the new ebuilds up at the above URL. I've got acx100 working now too. (:

Created an attachment (id=26643) [edit]
New patch to kernel-2.eclass

Okay, here's a new patch to kernel-2.eclass. This now checks for the
KBUILD_OUTPUT_PREFIX variable to decide whether or not to set KBUILD_OUTPUT in
the newly installed kernel's toplevel Makefile. This variable would be one of
the one's controlled by a new kernel-config/config-kernel tool.

I just committed your acx100 2.6.x out-of-kernel build patch (just when I
intended to migrate to 2.6 and do this work myself :).
Thanks a LOT for this very useful work!

I tested the latest patches with prism54; it seems to work fine! I get a
warning about the name of my source dir (I have linux-beta symlinked to
linux-2.6.3) which should probably not happen:

* The kernel Makefile says that this is a 2.6.3 kernel
 * but the source is in a directory for a beta kernel.
 *
 * This goes against the recommended Gentoo naming convention.
 * Please rename your source directory to 'linux-2.6.3'

For anyone else interested this was my list of things that I had to do to test:

patch /usr/portage/eclass/kernel-mod.eclass kernel-mod-KV_OUTPUT-v6.diff
patch /usr/portage/eclass/kernel-2.eclass kernel-2-koutput-v2.diff
cd /usr/src/linux
make mrproper
edit Makefile, add KBUILD_OUTPUT=/var/tmp/kernel-output/2.6.3 to top line
mv .config /var/tmp/kernel-output/2.6.3
make all modules_install
emerge the new kernel module

Just a thought, would it be possible to use this method to also build vmware
modules?

Excellent! glad it's working for other people. One thing, you should really put
KBUILD_OUTPUT *below* the first four lines of the toplevel Makefile (after the
EXTRAVERSION line) because new portage uses those first four lines to determine
KV

A few other updates:
1) I may redo this as a new seperate eclass kmod-2.eclass, in order to aboid
having to retain some of the cruft from the original kernel-mod.eclass

2) I've started work on the new config-kernel (decided on config-kernel instead
of kernel-config) in python. I've got most of the framework, and some of the
functions implemented, as soon as it's in quasi-usable state i'll be adding it
to the "gentoo" cvs module.

As for vmware, i'll have to check it out, that's one i hadn't looked at yet.
Thanks for testing!

The prism54 firmware is back at http://prism54.org/~mcgrof/firmware/. The file
1.0.4.3.arm is the same as the old isl3890 (in fact, it needs renaming to this
before it will work).

This is a great piece of work you're doing here and I don't want to add any new
tasks but will it work with lirc?

Okay, so here's the latest update:

1) DOCS - the developer oriented doc is much better, still at bug #40933, I've just submitted a user oriented doc at bug #44807 which i will hopefully be fleshing out more soon

2) UTILITY - I added config-kernel into CVS today, so that people can take a look at it. I'm not a python coder, so anybody who wants to submit patches to clean/improve that, please do. All of the functionality for that is working though. Waiting on johnm or i to add the corresponding support to kernel-2.eclass (not mission critical)

3) EBUILD/ECLASSES - I've decided to take over the kmod.eclass name, instead of patching the kernel-mod.eclass. First, because the name is nicer, and second, so i don't accidentally break anything currently using kernel-mod.eclass. ALL the latest ebuilds to reflect this, along with eclasses, etc is up at: http://dev.gentoo.org/~latexer/files/koutput/
so get your test on.

Where do we go now?
I need eyes on the user oriented doc, we really can't commit any of this til that is ready. config-kernel needs some pounding on beyond what i've done. I'll hopefully have an ebuild and current tarball up at the above location soon for people. I need to convert more ebuilds, like lufs, xfree-drm (battousai and i mostly have this done), and a few others.

WORKING:
hostap-driver, linux-wlan-ng, ndiswrapper, prism54, at76c503a, nvidia-kernel, ati-drivers, qc-usb (with CVS snapshot), omnibook, ipw2100 (new Intell 2100 drivers), acx100

NEED FIXING:
lufs, xfree-drm, mtx-drivers-pro, kyro-kernel, lirc (THIS is going to hurt), slmodem, madwifi (i've got this working with a LOT of makefile changes, i've submitted them upstream, hopefully they will merge those), esm (some random thing max wanted me to do)

What this boils down to. I WANT TO GET THIS MERGED!!!!! i'm dying to have this come to fruition. My goal is to clean up a few things in config-kernel, expand that user doc a bit more, then ROLL WITH THIS. hopefully wednesday and thursday will be merge days.

Anyone who wants to speak out against this, do so now! If people want, i will send a final email to gentoo-dev requesting eyes/testing/info.

Created an attachment (id=27538) [edit]
New version of kernel-2 eclass patch to make --set-extraversion useful

Okay, to make life easier, I'm going to not be posting any more of the ebuilds
here, instead, they will be available in an overlay tarball at:
http://dev.gentoo.org/~latexer/files/koutput

This should make testing easier, as it includes ebuilds in the right spot,
eclasses, etc, along with a README.

config-kernel in CVS (i forgot before, its in gentoo/src/kernel/config-kernel)
now has the added --set-extraversion as discusson gentoo-dev. An ebuild for
installing a snapshot of config-kernel is available in the kmod-overlay tarball
as well. this should let people more easily test things.

Attached is the latest patch to kernel-2.eclass. This implements setting
KBUILD_OUTPUT as dicussed in that same thread. Now uses
${KBUILD_OUTPUT_PREFIX}, but then sets it such that the makefile itself
determines the rest. This is the bit that makes the --set-extraversion so
handy. (: Still waiting on johnm for the auto symlink and auto config code in
the eclass, but that's more icing than core functionality.

Hopefully the steps requied to test this are clear/easy enough that this can
get wider attention. Thanks all who've given me feedback, specially Georgi for
poking me today to make this even more flexible. Nice to get these things
ironned out *now* and not later when we will have to worry about breaking the
previous setup. Any other wrinkles people want to point out? (: