| Summary: | app-admin/webapp-config is broken if dev-python/configparser is installed | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Tobias Klausmann (RETIRED) <klausman> |
| Component: | Current packages | Assignee: | Devan Franchini (RETIRED) <twitch153> |
| Status: | RESOLVED FIXED | ||
| Severity: | critical | CC: | harold, leho, qingxianhao, tb, web-apps |
| Priority: | Normal | Keywords: | InVCS |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | Ebuild based on the experimental branch | ||
|
Description
Tobias Klausmann (RETIRED)
2014-11-09 16:34:12 UTC
(In reply to Tobias Klausmann from comment #0) > # equery f tt-rss > * Searching for tt-rss ... > * Contents of www-apps/tt-rss-1.13: > /.keep_www-apps_tt-rss-1.13 > /conf > /hooks > /htdocs > /htdocs/LICENSE > /htdocs/README.md Yeah this is bad. twitch153 has mostly been taking care of this stuff, but I don't think he did anything to trigger this. Do you know if its isolated ot tt-rss or does it affect other webapp-config stuff? Also, what version of webapp-config are you using? Any hints how I could trigger that behaviour? I've never seen it on any of my tt-rss installations? (In reply to Thomas Kahle from comment #2) > Any hints how I could trigger that behaviour? I've never seen it on any of > my tt-rss installations? This looks like vhost_root was set to "/" in /etc/vhosts/webapp-config. Ideally it should be something like "/var/www/${vhost_hostname}" If you changed your webapp-config from the default, please post it. Also, what version of webapp-config are you using? My webapp-config is below, I don't remember ever changing it.
$ grep -v ^# /etc/vhosts/webapp-config|sort -u
allow_absolute="no"
package_manager="portage"
vhost_config_default_dirs="default-owned"
vhost_config_dir="${vhost_root}/conf"
vhost_config_virtual_files="virtual"
vhost_default_gid="root"
vhost_default_uid="root"
vhost_hostname="localhost"
vhost_htdocs_insecure="htdocs"
vhost_htdocs_secure="htdocs-secure"
vhost_perms_configowned_dir="0755"
vhost_perms_configowned_file="0644"
vhost_perms_defaultowned_dir="0755"
vhost_perms_installdir="0755"
vhost_perms_serverowned_dir="0775"
vhost_perms_serverowned_file="0664"
vhost_perms_virtualowned_file="o-w"
vhost_root="/var/www/${vhost_hostname}"
vhost_server="apache"
WA_CONF_VERSION="7"
I'm using app-admin/webapp-config-1.53, installed 11:36:05 AM 09/07/2014. I'm not sure when this broke, since tt-rss is the only webapp-config'd package I use and I only noticed it when the upgrade to 1.14 happeneds. Normally, I'd suspect the eclass to have broken, but there are no changes there, either. As a result, I suspect some rule about variable expansion changed, maybe in Portage. Note that the docs to webapp.eclass claim that MY_HTDOCS is exported, but I see no code to that effect in it. Some time ago there was another hiccup with config.php not being protected by webapp-config: bug #493520. We work around this now, but I find these webapp-config related issues impossible to debug :( So I tried the same thing on one of my other machines: no problems at all. Things that I suspected were at fault (spoiler: none of them are): - Recent glibc update after which I hadn't rebooted. - Python versions - webapp-config somehow got corrupted on one machine[0] - Portage versions - Portage tree version/sync time - Different FEATURES - Different USE-Flags on webapp-config, portage, python. The global USE flags _are_ different since the two machines have very different purposes. [0] Compared MD5sums on both machines, they are the same aside from some of the pyo/pyc files. The machines have different processors, so... I'll keep digging, but I suspect emerge -1 --emptytree is going to be my only recourse. And it may not even help. (In reply to Tobias Klausmann from comment #7) > So I tried the same thing on one of my other machines: no problems at all. I can't reproduce either. > > Things that I suspected were at fault (spoiler: none of them are): > > - Recent glibc update after which I hadn't rebooted. > - Python versions > - webapp-config somehow got corrupted on one machine[0] > - Portage versions > - Portage tree version/sync time > - Different FEATURES > - Different USE-Flags on webapp-config, portage, python. The global USE > flags _are_ different since the two machines have very different purposes. > > [0] Compared MD5sums on both machines, they are the same aside from some of > the pyo/pyc files. The machines have different processors, so... > > I'll keep digging, but I suspect emerge -1 --emptytree is going to be my > only recourse. And it may not even help. My guess is that its some variable not being exported somewhere. You could try to print out all the variables after config.parseparams() in main of webapp-config using config.get_config(<var>) and see if the variables are different on your two machines. (Exit right aftre that if you want to not actually go through and install the webapp.) Some more endless digging. Most of it pointless.
I added set -x in the eclass (at line 60) to see what falls out of the call to webapp-config --query. On the working machine, I get (amongst other stuff):
VHOST_CONFIG_DIR="/var/www/localhost/conf"
whereas the broken machine gives me:
VHOST_CONFIG_DIR="${vhost_root}/conf"
Note how the ${} is not interpolated. So I set out to understand the inner workings of webapp-config, specifically its configuration loading. It's in Python and I am quite well versed in that language.
But boy is w-c a pile of excrement. I've put three hours into it, trying to understand how it works, including trying to make sense of its completely broken debug logging (yay, another hour wasted, I should have gone with print-debugging in the first place).
Anyway, I still don't know what broke interpolation. I am amazed it ever worked.
I'll keep digging.
I found it. It's a bug in webapp-config: it uses ConfigParser as shipped with dev-lang/python. However, it makes the mistake of assuming that the superclass calls _interpolate_some() during parsing. This is wrong since functions prefixed with _ are, by convention, private and should not be relied on. The add-on package dev-python/config-parser, which is a backport of the Python-3 stock versions does not call this method and thus installation of webapps fails in the manner described here. So at the very least w-c needs to have a !dep for d-p/configparser. Ideally, its use of ConfigParser should be fixed. (In reply to Tobias Klausmann from comment #10) > I found it. > > It's a bug in webapp-config: it uses ConfigParser as shipped with > dev-lang/python. However, it makes the mistake of assuming that the > superclass calls _interpolate_some() during parsing. This is wrong since > functions prefixed with _ are, by convention, private and should not be > relied on. The add-on package dev-python/config-parser, which is a backport > of the Python-3 stock versions does not call this method and thus > installation of webapps fails in the manner described here. > > So at the very least w-c needs to have a !dep for d-p/configparser. Ideally, > its use of ConfigParser should be fixed. Nice work. For the time being I'm going to block against configparser until this is properly fixed. I can confirm that this is only relevant with Python2.7. I'll be working on removing the internal interpolate_some() function over the next few days so we can remove the blocker. Thanks again for finding this weirdness. After reviewing the code, I realized that the internal _interpolate_some() function did not need to be removed at all but instead what needed to be fixed was the automagic importing of the dev-python/configparser class. To fix this I forced webapp-config to import ConfigParser if the python version is less than 3.0 as opposed to attempting to import configparser (the py3.x variant) and then if this failed to fall back to the ConfigParser class. This makes the importing more explicit and leaves no room for error when having the backport package dev-python/configparser installed on the system. The change has been made and pushed the webapp-config's experimental branch and after review of the commits residing in that branch we'll merge them to master and release a new version of webapp-config. *** Bug 508434 has been marked as a duplicate of this bug. *** Could you please post a link to the experimental branch so I can test/use it? (In reply to Harold Naparst from comment #15) > Could you please post a link to the experimental branch so I can test/use it? To get the experimental webapp-config code run these commands: 1.) git clone git://git.overlays.gentoo.org/proj/webapp-config.git 2.) cd webapp-config 3.) git checkout experimental Warning: This is an experimental branch for a reason. Some of the stuff on here might not work and stability is not guaranteed. However, we try our best to make sure webapp-config is fully functional. Also be warned that because this is a git repo a new commit could come at any time and your version of webapp-config will not be up to date with the current experimental HEAD unless you git sync. So any bugs in here shouldn't be reported via bugs.gentoo.org. Good luck :) Created attachment 390884 [details] Ebuild based on the experimental branch This ebuild tracks the experimental branch of webapp-config. I gave it the revision 1.54_alpha because it is intended that the next release from the experimental branch may be called 1.54. This ebuild is not a proposal for portage. It is an ebuild for users who are currently blocked in their work by bug 528752, and have to have configparser and webapp-config both installed. This ebuild allows this. Please note that the syntax of webapp-config as a command has changed. The new syntax is given in the post install notes when this ebuild is run. If you want to run the attached ebuild, you can do so from my overlay: layman -a hnaparst This is fixed in webapp-config-1.54 which is currently on the tree. Marking as resolved. |