Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 367291 - waf-utils.eclass should support patching the waflib module
Summary: waf-utils.eclass should support patching the waflib module
Status: RESOLVED WONTFIX
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All Linux
: Normal enhancement (vote)
Assignee: No maintainer - Look at https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers if you want to take care of it
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 367293 373273 373275 378989
  Show dependency tree
 
Reported: 2011-05-14 19:49 UTC by Nathan Phillip Brink (binki) (RETIRED)
Modified: 2020-12-20 23:09 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
waf-utils.eclass-waflib-patchable.patch (waf-utils.eclass-waflib-patchable.patch,2.19 KB, patch)
2011-05-14 19:49 UTC, Nathan Phillip Brink (binki) (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Nathan Phillip Brink (binki) (RETIRED) gentoo-dev 2011-05-14 19:49:27 UTC
Created attachment 273223 [details, diff]
waf-utils.eclass-waflib-patchable.patch

With autotools.eclass, there are mechanisms for patching/replacing the buildsystem. If one needs to apply a patch to the buildscripts, it is simply applied and eautoreconf() is called. Or, if a package ships barebones (such as from a github tagged tarball or a repository checkout), eautoreconf() can replace the lack of a buildsystem with a buildsystem. I don't understand how the current waf.eclass (nonexistent) or waf-utils.eclass provide for patching a package's waflib.

I did, though, figure out how to apply a patch to waflib and I would like this workflow to be made a part of waf-utils.eclass somehow.

waf_binary=./waf
waflibdir=$(python -c "import imp, sys; sys.argv[0] = '${waf_binary}'; waflite = imp.load_source('waflite', '${waf_binary}'); print(waflite.find_lib());")

pushd "${waflibdir}" || die "Unable to patch waflib"
epatch "${FILESDIR}"/${P}-multilib.patch
popd

It is necessary to listen to waflite.find_lib()'s output because it returns a directory name which is about as predictable as the tarball subdirectories from github-generated tarballs. This directory is normally extracted when ./waf is first called, but waf will not reextract waflib if it finds this directory to already exist. Thus, getting waf to half-load in this manner is necessary to extract the waflib and works because waf doesn't overwrite any patches applied to waflib between the waflite.find_lib() call and subsequent calls to ${WAF_BINARY}.
Comment 1 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-05-14 20:27:51 UTC
Now, that you are a dev. I get to tell you to "always assign your own bug reports" =D
Comment 2 Gilles Dartiguelongue (RETIRED) gentoo-dev 2011-05-14 22:04:12 UTC
While this may sound like a reasonable request, it looks rather doubtful it would be any practical considering how waf upstream states that distros should not use a waf release over the version shipped in the package using it itself.
Comment 3 Nathan Phillip Brink (binki) (RETIRED) gentoo-dev 2011-05-15 00:35:33 UTC
(In reply to comment #2)

I understand that Gentoo should not package or replace the waflib shipped with individual packages. This patch is just an attempt at making it possible to patch whatever copy of waflib is shipped with a given package. I think that this hack will be useful at some point if a package is released with a defective version of waflib which needs minor patching.

It is more likely that the bugs in a wafed package will be found in wscript rather than in waf itself. But I did hit a patchable portage-multilib-specific bug in waf itself (bug 367293) today and I spent a fun time figuring out how to apply a patch to the shipped waflib ;-).
Comment 4 Gilles Dartiguelongue (RETIRED) gentoo-dev 2011-05-15 16:25:41 UTC
What I wanted to say is: if upstream doesn't want us to use a packaged waf, why would it allow us to patch waflib either ? I'm pretty sure one or both may have buggy release as I already encountered supposedly python3 ready copies that didn't work at all with python3 for example.

I'm actually worried that waf upstream would deny us any kind of support just because it thinks we are doing it wrong.
Comment 5 Mike Gilbert gentoo-dev 2011-12-24 19:19:26 UTC
(In reply to comment #4)
> I'm actually worried that waf upstream would deny us any kind of support just
> because it thinks we are doing it wrong.

So we have to live with broken packages because we might not get "support"? I don't follow.

If we were an upstream package provider, we could just cut a new release with an updated waf script. As a distro, we don't have that option.
Comment 6 Steven Newbury 2013-06-10 05:55:25 UTC
I've been using this patch for the last couple of years, without problem until python3.3.  Changes to the loader system has resulted in imp.load_source() (in Python 3.3 it's provided as a compatiblity wrapper in imp.py rather than built-in) rejecting the "WAF_BINARY" due to the embedded archive containing "null" characters.  I've been hitting my head against the wall the last couple of days trying to understand how the python loader system works, and how to work around this.

Otherwise, if this can be made to work again, can we please get this into the portage tree?  It makes no sense to have it impossible for package maintainers to be unable to fix-up the build system for Gentoo integration, the argument that we'll get no support from upstream could equally apply to any package in portage which carries (non-upstreamed) patches.
Comment 7 Nathan Phillip Brink (binki) (RETIRED) gentoo-dev 2013-07-26 11:27:58 UTC
(In reply to Steven Newbury from comment #6)
> I've been using this patch for the last couple of years, without problem
> until python3.3.  Changes to the loader system has resulted in
> imp.load_source() (in Python 3.3 it's provided as a compatiblity wrapper in
> imp.py rather than built-in) rejecting the "WAF_BINARY" due to the embedded
> archive containing "null" characters.  I've been hitting my head against the
> wall the last couple of days trying to understand how the python loader
> system works, and how to work around this.

I do not have time at this moment to update the patch, but the following code can be used to get python-3.3 to work with the same style hack. Should be backwards-compatible, but I have only tested it with python-3.3.2-r1.

	local waf_binary=${WAF_BINARY} waf_binary_loader
	[[ -n ${1} ]] && waf_binary=${1}
	waf_binary_loader=${waf_binary}-loader.py

	# Starting with python3.3, imp.load_source() refuses to read files
	# with NUL bytes. Thus, extract the loader itself and only ask
	# python itself to load that:
	sed -ne '1,/^#==>$/p' "${waf_binary}" > "${waf_binary_loader}" || die

	python -c "import imp, sys; sys.argv[0] = '${waf_binary}'; waflite = imp.load_source('waflite', '${waf_binary_loader}'); print(waflite.find_lib());" \
		|| die "Unable to locate or unpack waflib module from the waf script at ${waf_binary}"


It works by nabbing the python-only part of the archive and, with sys.argv[0], still pointing to the ./waf but only asking the python loader to load the python-only file in ./waf-loader.py.