| Summary: | portage only outputs brief error message when python imports fail, causing secondary failures later | ||
|---|---|---|---|
| Product: | Gentoo/Alt | Reporter: | Henning Schild <henning> |
| Component: | FreeBSD | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | henning, lavajoe |
| Priority: | High | Keywords: | InVCS |
| Version: | unspecified | ||
| Hardware: | x86 | ||
| OS: | FreeBSD | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 193766, 192469, 194041 | ||
| Attachments: |
make emerge bail out and allow override via FEATURES="ignore-missing-freebsd-module"
implement chflags() and lchflags() by spawning the chflags command |
||
|
Description
Henning Schild
2007-09-12 18:57:04 UTC
these files also had this flag set /usr/lib/libthr.so.2 /lib/libcrypt.so.3 /lib/libc.so.6 /lib/libpthread.so.2 since i dont have freebsd-lib in my emerge log i guess this flags came from the stage3-image i used on Jun 23 this year. bug 192354 describes the same problem for another package I think i found where the flags came from. In stage3-x86-freebsd-6.2-r1.tar.bz2 the following files are flagged system immutable: /usr/bin/chpass /usr/bin/chfn /usr/bin/chsh /usr/bin/login /usr/bin/opieinfo /usr/bin/opiepasswd /usr/bin/passwd /usr/bin/su /usr/lib/libc_r.so.6 /usr/lib/libthr.so.2 /usr/sbin/sliplogin /lib/libcrypt.so.3 /lib/libc.so.6 /lib/libpthread.so.2 /libexec/ld-elf.so.1 /sbin/init this means that systems based on this archive will fail with updates of: sys-freebsd/freebsd-usbin sys-freebsd/freebsd-sbin sys-freebsd/freebsd-lib sys-freebsd/freebsd-libexec *** Bug 192354 has been marked as a duplicate of this bug. *** This is a bit of a mystery... I (as well as UberLord) also see the schg flag set on these files, but neither of us has had emerge quit due to it. I would think at least this would be consistent. One potential wildcard is the security level. Can you do a "sysctl kern.securelevel"? If it is other than -1, let us know. However, if you had the security level set, you'd have other issues, like not even being able to chflags the files... Still, it is in fact strange that I can even re-emerge freebsd-lib at all, since the schg flag should prohibit this (as you are seeing)..... Anyway, still investigating. localhost ~ # sysctl kern.securelevel kern.securelevel: -1 The following is not really what it was before. I could update because i removed the flags manually. localhost ~ # emerge --info Portage 2.1.3.9 (default-bsd/fbsd/6.2/x86, gcc-4.1.2, freebsd-lib-6.2-r2, 6.2-RELEASE i386) ================================================================= System uname: 6.2-RELEASE i386 i386 Timestamp of tree: Thu, 13 Sep 2007 14:59:01 +0000 app-shells/bash: 3.2_p17-r1 dev-lang/python: 2.4.4-r4, 2.5.1-r2 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 2.0.0_rc4-r1 sys-devel/autoconf: 2.61-r1 sys-devel/automake: 1.10 sys-devel/binutils: 2.17-r1 sys-devel/gcc-config: 1.4.0-r2 sys-devel/libtool: 1.5.24 virtual/os-headers: 6.2-r2 ACCEPT_KEYWORDS="x86-fbsd ~x86-fbsd" CBUILD="i686-gentoo-freebsd6.2" CFLAGS="-O2 -mtune=i686 -pipe" CHOST="i686-gentoo-freebsd6.2" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/terminfo" CXXFLAGS="-O2 -mtune=i686 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="collision-protect distlocks metadata-transfer parallel-fetch sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ " PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="berkdb cli cracklib crypt cups dri gdbm iconv java5 midi ncurses nls oss pam pcre perl ppds python readline reflection session slang spl ssl tcpd unicode x86-fbsd xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="FreeBSD" INPUT_DEVICES="keyboard mouse" KERNEL="FreeBSD" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="BSD" VIDEO_CARDS="apm ark chips cirrus cyrix dummy i128 i810 mach64 mga neomagic nv r128 radeon rendition s3 s3virge savage siliconmotion sis tga trident tseng vga via vmware" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, MAKEOPTS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY This continues to be odd, since I just re-emerged freebsd-lib *with* the schg flags intact on libthr.so.2 and libc_r.so.6, and it succeeded. Not only that, but I verified this by looking at the file mod times; they had been updated. So I do not know why your systems fail. More importantly, I do not know why my systems *do not* fail! :) Unless portage is taking care of the flag for us somehow (which your experience seems to deny), I do not see how it's *working* here..... I spoke too soon. Just looked at the portage source, and it indeed does use chflags to take care of this (and then it looks like it sets the flags back after). So now the question is why yours does not do this correctly... Here's some code from portage: bsd_chflags = None if os.uname()[0] in ["FreeBSD"]: try: import freebsd as bsd_chflags except ImportError: pass So it keys off uname. Is there a possibility that your uname is not returning "FreeBSD"? That seems unlikely... You name it! Its exaclty the code you quoted:
File "/usr/lib/portage/pym/portage.py", line 48, in <module>
import freebsd as bsd_chflags
ImportError: No module named freebsd
I did not run python-updater after my last python update. So all this is my fault because i did not yet configure elog mails on this machine.
And there is not much portage could do to handle that problem. Maybe one could replace the "pass" with or "raise" for every merge but one that was triggerd by python-updater or depends on python.
However i think this bug is no real bug.
I have changed the summary and reopened this as an issue in portage. Since it can be non-obvious to users what action to take in the event of portage errors (and evidently some of those errors are easy to miss), I was wondering if there is a way to let them know what the real problem is. In this particular case, not running python-updater after upgrading python caused a failure to import the "freebsd" module: ImportError: No module named freebsd And this resulted in a seemingly unrelated issue of not being able to "chflags" some files, causing portage to bomb when installing to root. This bug was originally filed against freebsd-lib, but we eventually tracked it back to the python/portage issue. So my suggestions are: 1) Cause portage to bomb out on such failures to import (Note that I am assuming that portage did *not* bomb at this point due to the fact that it proceeded to the secondary error) 2) Give a suggestion to the user of what to do (or if this is not possible, perhaps just say it could be related to not running python-updater) Created attachment 130873 [details, diff]
make emerge bail out and allow override via FEATURES="ignore-missing-freebsd-module"
With this patch, emerge will bail out when appropriate with the following error message:
* An error occurred while attempting to import the freebsd python module.
* This usually means that python has just been upgraded and the py-freebsd
* package has not yet been rebuilt by python-updater. The freebsd python
* module is required for proper operation. Please install or rebuild py-
* freebsd as soon as possible. In order to bypass this error, add "ignore-
* missing-freebsd-module" to FEATURES. Do NOT forget to remove it from
* FEATURES as soon as the freebsd python module has been properly
* installed.
If the freebsd python module imports properly but FEATURES="ignore-missing-freebsd-module" is still enabled, emerge will display the following warning message:
* Do NOT forget to remove "ignore-missing-freebsd-module" from FEATURES as
* soon as the freebsd python module has been properly installed.
Great! I just created a new bug (192469) on python-updater suggesting they add the passing of this new feature - this should eliminate the need for the user to set his/her FEATURES variable, and portage could then suggest to the user to first try running python-updater... Do we really need to guard this with a FEATURES flag? Rather than guard if with FEATURES="ignore-missing-freebsd-module", we could simply add a time delay and/or bell. The way that it comes up at the very beginning, I doubt there would be any problem with people missing it or ignoring it. But you do need to be able to switch the behavior on and off. If only a bell/warning were to be issued when the the missing module is detected, the user would likely miss it, and a confusing error would ensue. We had several users get really confused by the missing module (they missed the error message and got a seemlingly unrelated condition later). (In reply to comment #15) > (they missed the error message > and got a seemlingly unrelated condition later). What error message? There currently is no such message afaik. We're suggesting that a prominent message be show as soon as emerge starts up (like the one that the patch shows). I think we can assume that someone who misses an ewarn after a python update is likely to miss ewarns at the beginning of other merges. And since passing that ImportError in general can really damage the setup i think comments 11 and 12 are the right way to handle that. (In reply to comment #12) > I just created a new bug (192469) on python-updater suggesting they add the > passing of this new feature - this should eliminate the need for the user to > set his/her FEATURES variable, and portage could then suggest to the user to > first try running python-updater... The thing is, we'd rather not couple portage too tightly with other packages. If we change the message to something different then portage becomes dependent on python-updater interacting with it in a certain way. I'd really prefer to avoid dependencies like that. (In reply to comment #17) > I think we can assume that someone who misses an ewarn after a python update is > likely to miss ewarns at the beginning of other merges. We'll, I'd say that a message somewhere in the middle of a bunch of merges or at the end even is a lot more likely to be ignored that one at the very beginning. Anyway, considering the potential for system damage, I think guarding it with FEATURES="ignore-missing-freebsd-module" as implemented in the patch is a reasonable solution. But like I said, I'd prefer that the portage messages don't rely on python-updater behavior in any way. (In reply to comment #0) > Steps to Reproduce: > (0. chflags schg /usr/lib/libc_r.so.6) > i am sure i never did that by hand > 1. emerge -av1 =sys-freebsd/freebsd-lib-6.2-r2 > > Actual Results: > "Operation not permitted" when trying to overwrite /usr/lib/libc_r.so.6 How exactly does it behave after the "Operation not permitted" error? Does emerge exit with a failure immediately or does it continue as if nothing happened? If it just continues then we need to do something about that. For clarification: I'm not opposed to make this conditional on something if that's the better solution, but I don't like to abuse the FEATURES variable for working around a cornercase problem on a minority platform. (In reply to comment #19) > How exactly does it behave after the "Operation not permitted" error? Does > emerge exit with a failure immediately or does it continue as if nothing > happened? If it just continues then we need to do something about that. This happens in one of the final steps when files are moved to the live filesystem. And portage exits at the file protected by a flag. And that causes half updated packages. Another idea: The import is needed to detect/set/unset flags with python. So one could use "chflags" as fallback when the import fails simply using extern binaries instead of the python module. (In reply to comment #21) > The import is needed to detect/set/unset flags with python. So one could use > "chflags" as fallback when the import fails simply using extern binaries > instead of the python module. That seems like an ideal solution. Then we wouldn't have to provide some interface that changes portage behavior to work around the problem. I'm not familiar with the commands that we'd need to use to do that. I suppose we might be able to blindly clear immutable flags without restoring them later, but that would be sloppy. Hey, if it works, that sounds fine. I can help you with the right commands for fbsd. I'd rather we restore the flags; flags can be read with "ls -lo". Actually, if system immutable is the only one to worry about, it'd be as simple as checking for "schg" from the ls -lo output. To remove the flag, do "chflags noschg <file>", and to restore, do "chflags schg <file>". Apparently python stat result objects have a st_flags attribute that our freebsd code is already utilizing. That means we won't have to parse any `ls -lo` output. We only have to convert the st_flags attribute into suitable command line options for the chflags command. According to the chflags(1) man page, it will accept the flags as an octal number, so this could be really easy. Issue 1490190: add os.chflags() and os.lchflags() where available http://bugs.python.org/issue1490190 If that patch gets included in python-2.6 then we won't even need the standalone freebsd module anymore. I think the code should be written to detect and use os.chflags when it is available. That way when the 2.5 -> 2.6 upgrade comes, this bug will not be an issue anymore. Created attachment 131918 [details, diff]
implement chflags() and lchflags() by spawning the chflags command
This patch implements chflags() and lchflags() by spawning the chflags command so that we don't have to rely on the external python module at all. The functions are only called when merging/unmerging files that actually have flags set so the performance difference should be negligible.
Tested patch - works great. I tested removing & replacing files and dirs with the immutable flag set. This has been released in 3.1.3.10. |
