Attachment 25276 Details for Bug 40933 – Reviewed 2.6-koutput.xml document

Reviewed 2.6-koutput.xml document

2.6-koutput.xml (text/xml), 14.92 KB, created by Sven Vermeulen (RETIRED) on 2004-02-09 10:58:04 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Sven Vermeulen (RETIRED)

Created: 2004-02-09 10:58:04 UTC

Size: 14.92 KB

patch

obsolete

><?xml version='1.0' encoding="UTF-8"?>
>
><!DOCTYPE guide SYSTEM "/dtd/guide.dtd">
>
><guide link="/doc/en/2.6-koutput.xml">
>
><title>2.6 Kernel Output</title>
><author title="Developer">
>    <mail link="latexer@gentoo.org">Peter Johanson</mail>
></author>
>
><license/>
>
><abstract>
>This guide is aimed at developers and covers the details of the new kernel 
>output changes in Gentoo. 
></abstract>
>
><version>1.1</version>
><date>February 9, 2004</date>
>
><chapter>
><title>Preface</title>
><section>
><title>The New System</title>
><body>
>
><p>
>Among the myriad of other improvement in the 2.6 series Linux kernel, a new 
>infrastructure, "kbuild",  has been developed to create a highly configurable 
>and versatile build system. The full scope of this change is not the focus of 
>this document. Of importance is the new recommended method for compiling 
>external kernel modules against a 2.6 kernel tree. From 
><path>Documentation/kbuild/modules.txt</path>:
></p>
>
><pre caption="Documentation/kbuild/modules.txt">
>Compiling modules outside the official kernel
>---------------------------------------------
>Often modules are developed outside the official kernel.
>To keep up with changes in the build system the most portable way
>to compile a module outside the kernel is to use the following command-line:
>
>make -C path/to/kernel/src SUBDIRS=$PWD modules
></pre>
>
><p>
>By overriding the SUBDIRS variable, the latest kbuild infrastructure can easily 
>be used to compile external modules. This in itself is a <e>great</e> boon, but
>the particulars are where things become problematic.
></p>
>
></body>
></section>
><section>
><title>The Problem</title>
><body>
>
><p>
>Having configured and compiled a 2.6 series kernel residing in 
><path>/usr/src/linux</path>, it's not unreasonable to want to compile some 
>external modules against this kernel source tree. Beyond that, one would hope 
>that this could be done by an unprivileged user. Trying to do so shows that 
>this is <e>not</e> the case. Such an attempt fails as <c>make</c> attempt to 
>update files in <path>/usr/src/linux</path>, which an unprivileged user has no 
>access to write to.
></p>
>
><p>
>Modules being built within the portage system suffer from this exact same 
>problem, dying with sandbox violations when such updates are attempted. The 
>added features of kbuild have also added complexity, of a sort which is very 
>problematic for those wishing to follow safe practices of least privilege.
></p>
>
><p>
>Searching Gentoo's bugzilla will find many bugs opened because of this very 
>problem and many forums posts have arisen suggesting sandbox disabling as the 
>only solution. Indeed, with the current state of affairs, it seemed inevitable 
>that some writing or updating to <path>/usr/src/linux</path> would be 
>inevitable for modules built against 2.6 kernels.
></p>
>
></body>
></section>
><section>
><title>Embracing Change</title>
><body>
>
><p>
>The solution has arisen out of a realization that fighting the new features of 
>kbuild would only result and further and further problems in the future. In 
>fact, kbuild's new ability to send all output files to a separate directory 
>would prove to be the solution. Keeping the kernel source tree completely 
>clean and allowing external modules to build against a clean source tree 
>while they themselves sent their output files to a temporary directory is the 
>key.
></p>
>
><p>
>The details of how these elements interplay is somewhat complex. Hopefully the 
>following section will break the concepts into more easily digestable pieces, 
>such that developers and users wishing to enhance or use this new system will 
>have the knowledge needed to do so.
></p>
>
></body>
></section>
></chapter>
>
><chapter>
><title>Kernel Output</title>
><section>
><title>KBUILD_OUTPUT and O variables</title>
><body>
>
><p>
>kbuild provides two variables for dictating where the kernel should output its 
>files. 
></p>
>
><table>
><tr>
>  <th>Variable</th>
>  <th>Usage</th>
></tr>
><tr>
>  <ti>KBUILD_OUTPUT</ti>
>  <ti>
>    This variables can be defined in the top-level kernel <path>Makefile</path>.
>    This can be used if you want to set a default output other than in the 
>    source tree itself.
>  </ti>
></tr>
><tr>
>  <ti>O</ti>
>  <ti>
>    This variable is to be used on the command line to override any other 
>    settings and indicate where the output for the current command should be 
>    placed.
>  </ti>
></tr>
></table>
>
><p>
>The combination of these two variables is the key to successfully using kbuild 
>and portage to install kernel modules.
></p>
>
></body>
></section>
><section>
><title>Kernel Installation Changes</title>
><body>
>
><p>
>To make use of the kernel output features, the new <path>kernel-2.eclass</path> written by 
><mail link="johnm@gentoo.org">John Mylchreest</mail> has been patched to add a 
>default <c>KBUILD_OUTPUT</c> path in the top level kernel makefile 
>(<path>/usr/src/linux/Makefile</path>). By default, this path will be set to 
><path>/var/tmp/kernel-output/${KV}</path> as <path>/var/tmp</path> is a 
>suitable place for temporary files that need to remain between reboots.
></p>
>
><p>
>Once this variable is set, all make commands issued in the kernel source tree 
>send their outputs to this new directory. No further work is required on the 
>users part and the change is most (see the next section for exceptions) 
>seamless. Once a kernel is installed, a user can still just do:
></p>
>
><pre caption="Configuring the kernel">
># <i>make menuconfig</i>
></pre>
>
><p>
>and they will be able to configure their kernel and then build it.
></p>
>
></body>
></section>
><section>
><title>Important File Location Changes</title>
><body>
>
><p>
>As all generated files are now placed in a separate directory, a few key files 
>will end up in a place unexpected by the user. In particular, a users 
><path>.config</path> and <path>bzImage</path> will no longer be in 
><path>/usr/src/linux</path> as they were before. <mail 
>link="latexer@gentoo.org">Peter Johanson</mail> has written a new 2.6 kernel 
>guide oriented at end users which emphasises the kbuild changes and in 
>particular the new locations of these files.
></p>
>
>
>
></body>
></section>
></chapter>
>
><chapter>
><title>External Module Ebuilds</title>
><section>
><title>General Approach</title>
><body>
>
><p>
>Now that kernels are outputting their files to a different location, writing 
>ebuilds that adhere to the new system is the next step. Updating ebuilds to 
>look for certain generated files and headers in the proper location and 
>getting them using the correct <path>Makefile</path> magic is key.
></p>
>
><p>
>The general idea is to patch <path>Makefiles</path> in packages so that when 
>compiling against 2.6 kernels, they use the O variable to output to a 
>temporary directory within our sandboxed environment. Adding a few small 
>pieces to make this all succeed and we've got another successfully merging 
>external module package.
></p>
>
><note>
>The actual usability and functionality of modules after they are built and 
>installed is <e>not</e> the goal of this document. That is a whole separate 
>issue that upstream driver authors should be handling.
></note>
>
><p>
>Changes have been added to the <path>kernel-mod.eclass</path> to include some 
>variables and tools that ebuild authors may use to more easily affect these 
>changes on the ebuild level.
></p>
>
></body>
></section>
><section>
><title>Makefile requirements</title>
><body>
>
><p>
>The new koutput setup adds certain necessities to the <path>Makefile</path>s of 
>kernel module packages. In particular, they shall need to use the kbuild 
>variable <c>O</c> to output the built files into some subdirectory of 
><c>WORKDIR</c>. 
>Usually this will be something like <path>${S}/tmp</path>. When using this 
>system, the output directory must have the kernel's <path>.config</path> file 
>present in that directory. The copying of the config can be done in either the 
>makefile or in the ebuild, but doing so in the makefile in a sensible way might 
>lead to an upstream patch aleviating Gentoo from doing so.
></p>
>
></body>
></section>
>
><section>
><title>kernel-mod.eclass Features</title>
><body>
>
><p>
>All kernel module ebuilds should now use the <path>kernel-mod.eclass</path>. 
>Here's a quick rundown of its use. The kernel-mod_src_unpack() function handles 
>unpacking things, as well as setting a group of extremely useful variables 
>which can be used elsewhere. Two variables can be set by the ebuild which 
>affect how kernel-mod_src-unpack() functions.
></p>
>
><table>
><tr>
>  <th>Variable</th>
>  <th>Usage</th>
></tr>
><tr>
>  <ti>KERNEL_MOD_SOURCES</ti>
>  <ti>
>    If set, these files (tarballs usually) will be unpacked into WORKDIR. 
>    Otherwise ${A} will be unpacked.
>  </ti>
></tr>
><tr>
>  <ti>KERNEL_MOD_KOUTPUT_PATCH</ti>
>  <ti>
>    If set, this patch will be applied using epatch in ${S}. This could be 
>    done by hand of course, but this might be useful to avoid needing to 
>    define a src_unpack() at all in an ebuild.
>  </ti>
></tr>
></table>
>
><p>
>After kernel-mod_src_unpack() has been called, there are a plethora of variables
>available for use elsewhere in the ebuild.
></p>
>
><table>
><tr>
>  <th>Variable</th>
>  <th>Meaning</th>
></tr>
><tr>
>  <ti>KV_OUTPUT</ti>
>  <ti>
>    The full path where the kernel is outputting its files. For 2.4 kernels 
>    this will always be <path>/usr/src/linux</path>. For 2.6 this will 
>    hopefully be a different directory (otherwise things will fail during 
>    module compilation).
>  </ti>
></tr>
><tr>
>  <ti>KV_OJB</ti>
>  <ti>
>    The extension for kernel modules for this kernel version. Either "ko" or 
>    "o" depending.
>  </ti>
></tr>
><tr>
>  <ti>KV_VERSION_FULL</ti>
>  <ti>The full kernel version.</ti>
></tr>
><tr>
>  <ti>KV_MAJOR</ti>
>  <ti>The major number of the kernel.</ti>
></tr>
><tr>
>  <ti>KV_MINOR</ti>
>  <ti>The minor number of the kernel.</ti>
></tr>
><tr>
>  <ti>KV_PATCH</ti>
>  <ti>The patch number of the kernel.</ti>
></tr>
><tr>
>  <ti>KV_TYPE</ti>
>  <ti>The type of kernel (e.g. "-gentoo-r1" in "2.4.23-gentoo-r1")</ti>
></tr>
></table>
>
></body>
></section>
><section>
><title>Example ebuild and Makefile fix</title>
><body>
>
><p>
>The following is a real world example of how the hostap-driver package was 
>fixed to create a fully functioning ebuild for both 2.4 and 2.6 series kernels. 
>It uses a patch to the hostap-driver's top level <path>Makefile</path> (which is
>generalized enough to be sent upstream and allow future ebuilds to be 
>simplified) as well as some changes to the ebuild to account for some file 
>location changes.
></p>
>
><p>
>Below are some excerpts from the original <path>Makefile</path> and some new and
>modified sections which enable proper building. First, we need to fix a line 
>which included the <path>.config</path> expected to be in the path with the 
>kernel sources.
></p>
>
><note>
>The KERNEL_OUTPUT_PATH variable is added earlier in the build to complement the 
>KERNEL_PATH variable in the original.
></note>
>
><pre caption="Original include line">
>include $(KERNEL_PATH)/.config
></pre>
>
><p>
>Below is the fixed version which lets KERNEL_OUTPUT_PATH be set if not already 
>(for 2.4 backwards compatibility) and looks for the <path>.config</path> in 
>that location.
></p>
>
><pre caption="Fixed include line">
>ifndef KERNEL_OUTPUT_PATH
>KERNEL_OUTPUT_PATH=$(KERNEL_PATH)
>endif
>
>include $(KERNEL_OUTPUT_PATH)/.config
></pre>
>
><p>
>Since we now have a KERNEL_OUTPUT_PATH variable at our disposal, fixing 
>the following variable declaration involving the generated 
><path>version.h</path> is simple. The original:
></p>
>
><pre caption="Original VERFILE">
>VERFILE := $(KERNEL_PATH)/include/linux/version.h
></pre>
>
><p>
>And now the edited fixed version:
></p>
>
><pre caption="Edited VERFILE">
>VERFILE := $(KERNEL_OUTPUT_PATH)/include/linux/version.h
></pre>
>
><p>
>Finally, our patch fixes the line which invokes the 2.6 kbuild system to 
>include the use of the O variable, setting the system to output to a directory 
><path>tmp/</path> in the current directory.
></p>
>
><warn>
>kbuild expects to find a valid <path>.config</path> in the output directory 
>specified by <c>O=foo</c>. Either the ebuild or the makefile should be edited 
>to copy the <path>.config</path> in ${KV_OUTPUT} (set by kernel-mod.eclass) 
>into the output directory.
></warn>
>
><p>
>Here is the original:
></p>
>
><pre caption="Original output directory">
>$(MAKE) -C $(KERNEL_PATH) SUBDIRS=$(PWD)/driver/modules \
>	MODVERDIR=$(PWD)/driver/modules modules
></pre>
>
><p>
>And here is the edited version:
></p>
>
><pre caption="Edited output directory">
>mkdir -p $(PWD)/tmp
>-cp $(KERNEL_OUTPUT_PATH)/.config $(PWD)/tmp
>$(MAKE) -C $(KERNEL_PATH) O=$(PWD)/tmp \
>	SUBDIRS=$(PWD)/driver/modules \
>	MODVERDIR=$(PWD)/driver/modules  modules
></pre>
>
><p>
>The changes to the ebuild are relatively straightforward. Things are made 
>particularly easy by the kernel-mod eclass. Here are the important pieces of 
>the src_unpack section and some variables we set in the ebuild:
></p>
>
><pre caption="src_unpack() function">
>KERNEL_MOD_SOURCES="${P}.tar.gz"
>KERNEL_MOD_KOUTPUT_PATCH="${PN}-koutput.diff.gz"
>
>src_unpack() {
>    # Unpack and set some variables
>    kernel-mod_src_unpack
>	
>    ## unpack the pcmcia-cs sources if needed
>    pcmcia_src_unpack
>
>    epatch "${FILESDIR}/${P}.firmware.diff.bz2"
>    
>    # If on 2.5 or 2.6, set the kernel output stuff
>    if [ "${KV_MINOR}" -gt "4" ]
>    then
>        sed -i -e "s:^# KERNEL_OUTPUT_PATH=.*:KERNEL_OUTPUT_PATH=${KV_OUTPUT}:" \
>            ${S}/Makefile
>    fi
>}
></pre>
>
><p>
>Notice the use of the two variables, <c>KERNEL_MOD_SOURCES</c> and 
><c>KERNEL_MOD_KOUTPUT_PATCH</c>. These can be set as needed to specify which 
>sources should be unpacked by kernel-mod_src_unpack() and a file to patch if 
>needed to fix the kernel output in the package's <path>Makefile</path>s. Here is
>a trimmed down version of the src_compile() function:
></p>
>
><pre caption="src_compile() function">
>src_compile() {
>    # Configure the pcmcia-cs sources as needed.
>    pcmcia_configure
>
>    einfo "Building hostap-driver for kernel version: ${KV}"
>    case ${KV_MINOR} in
>        [34])
>            local mydrivers
>
>            use pcmcia &amp;&amp; mydrivers="${mydrivers} pccard"
>            use hostap-nopci || mydrivers="${mydrivers} pci"
>            use hostap-noplx || mydrivers="${mydrivers} plx"
>
>            einfo "Building the following drivers: ${mydrivers}"
>            emake EXTRA_CFLAGS="-DPRISM2_DOWNLOAD_SUPPORT" ${mydrivers} || die "make failed"
>            ;;
>        [56])
>            unset ARCH
>            emake EXTRA_CFLAGS="-DPRISM2_DOWNLOAD_SUPPORT" all || die "make failed"
>            ;;
>        *)
>            eerror "Unsupported kernel version: ${KV}"
>	    die
>	    ;;
>    esac
>}
></pre>
>
><p>
>Notice in particular that the <c>ARCH</c> variable needs to be unset. The new 
>kernel makefiles use <c>ARCH</c> to determine what to build for and use 
>different syntax for i386, etc.
></p>
>
><p>
>This ebuild's src_install() function will not be shown, as it basically just 
>installs all the modules into <path>/lib/modules/${KV}/</path>. Of note 
>it handles the change in kernel module extensions from <path>.o</path> to 
><path>.ko</path> in the 2.6 kernel. kernel-mod.eclass sets the variable 
><c>KV_OBJ</c> to either <path>o</path> or <path>ko</path> as appropriate to 
>make things easier on you.
></p>
>
></body>
></section>
></chapter>
></guide>

Actions: View

Attachments on bug 40933: 25229 | 25276 | 25314 | 25339 | 25457 | 25461 | 25496 | 27437 | 28025 | 28081 | 28194