I just tried to upgrade eix from 0.15.2 to 0.15.3, but that died with a sandbox violation. python was trying to remove the docutils/utils.pyc and the sandbox stopped that. The cause seemed to be, that emerge "forgot" to rebuild the .pyc and .pyo files from dev-python/docutils when I upgraded that package from 0.4-r3 to 0.5 Reproducible: Always Steps to Reproduce: I think the following was happening: * docutils-0.4 source gets packaged -> Timestamp A * docutils-0.5 source gets packaged -> Timestamp B * I emerge docutils-0.4-r3 -> .py has Timestamp A; .pyc/.pyo get Timestamp C with A<B<C * I upgrade to docutils-0.5 * emerge installs the .py that still have the original Timestamp B * emerge tries to build the .pyc/.pyo, but because B<C they will not be rebuild * I try to upgrade eix -> wants to use docutils/utils.py -> sandbox violation After manually removing all .pyc/.pyo from /usr/lib64/python2.5/site-packages/docutils and all subdirectories and reemerging dev-python/docutils-0.5 the emerge of eix-0.15.3 worked. (Just remerging docutils-0.5 without manually removing these files did not work) Actual Results: --------------------------- ACCESS VIOLATION SUMMARY --------------------------- LOG FILE "/var/log/sandbox/sandbox-24636.log" VERSION 1.0 FORMAT: F - Function called FORMAT: S - Access Status FORMAT: P - Path as passed to function FORMAT: A - Absolute Path (not canonical) FORMAT: R - Canonical Path FORMAT: C - Command Line F: unlink S: deny P: /usr/lib64/python2.5/site-packages/docutils/utils.pyc A: /usr/lib64/python2.5/site-packages/docutils/utils.pyc R: /usr/lib64/python2.5/site-packages/docutils/utils.pyc C: python /usr/bin/rst2html.py format.txt --embed-stylesheet --stylesheet-path=stylesheet.css -------------------------------------------------------------------------------- Expected Results: Upgrade of eix. From /usr/lib64/python2.5/site-packages/docutils: after the eix failure: -rw-r--r-- 1 root root 21280 Apr 3 2007 utils.py -rw-r--r-- 1 root root 21973 Aug 24 2007 utils.pyc -rw-r--r-- 1 root root 21870 Aug 24 2007 utils.pyo after just remerging docutils-0.5: -rw-r--r-- 1 root root 21280 Apr 3 2007 utils.py -rw-r--r-- 1 root root 21973 Aug 24 2007 utils.pyc -rw-r--r-- 1 root root 21870 Aug 24 2007 utils.pyo after removing all .pyc/.pco and remerging docutils-0.5: -rw-r--r-- 1 root root 21280 Apr 3 2007 utils.py -rw-r--r-- 1 root root 22441 Jan 30 16:40 utils.pyc -rw-r--r-- 1 root root 22338 Jan 30 16:40 utils.pyo (August 24 2007 matches the last emerge of docutils-0.4) I'm currently using sys-apps/portage-2.2_rc23. In August, when I initially upgraded to docutils-0.5 I was using sys-apps/portage-2.2_rc8.
Please post your `emerge --info' too.
emerge --info: Portage 2.2_rc23 (default/linux/amd64/2008.0, gcc-4.3.3, glibc-2.9_p20081201-r1, 2.6.29-rc1 x86_64) ================================================================= System uname: Linux-2.6.29-rc1-x86_64-Dual-Core_AMD_Opteron-tm-_Processor_2218-with-glibc2.2.5 Timestamp of tree: Fri, 30 Jan 2009 14:00:01 +0000 ccache version 2.4 [disabled] app-shells/bash: 3.2_p48 dev-java/java-config: 1.3.7-r1, 2.1.7 dev-lang/python: 2.5.4-r2 dev-util/ccache: 2.4-r8 dev-util/cmake: 2.6.2-r1 sys-apps/baselayout: 2.0.0 sys-apps/openrc: 0.4.2 sys-apps/sandbox: 1.3.3 sys-devel/autoconf: 2.13, 2.63 sys-devel/automake: 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2 sys-devel/binutils: 2.19 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6a virtual/os-headers: 2.6.28-r1 ACCEPT_KEYWORDS="amd64 ~amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-pipe -march=opteron -O3 -fomit-frame-pointer -fweb -frename-registers -fsee -ftracer -ftree-loop-im -funswitch-loops -fivopts" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/kde/4.1/env /usr/kde/4.1/share/config /usr/kde/4.1/shutdown /usr/kde/4.2/env /usr/kde/4.2/share/config /usr/kde/4.2/shutdown /usr/share/config /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-pipe -march=opteron -O3 -fomit-frame-pointer -fweb -frename-registers -fsee -ftracer -ftree-loop-im -funswitch-loops -fivopts" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps y --jobs=4 --load-average=5 --keep-going" FEATURES="buildpkg distlocks fixpackages parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-orphans userfetch userpriv usersandbox" GENTOO_MIRRORS="http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ http://mirror.ovh.net/gentoo-distfiles/ http://ds.thn.htu.se/linux/gentoo http://gentoo.inf.elte.hu/ http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/" LDFLAGS="-Wl,-O1" LINGUAS="de en" MAKEOPTS="-j5" PKGDIR="/var/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage-local/layman/sunrise /usr/portage-local/layman/nx /usr/portage-local/layman/secondlife /usr/portage-local/layman/x11 /usr/portage-local/layman/desktop-effects /usr/portage-local/layman/gentoojp /root/ebuilds" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow 3dnowext 7zip X a52 aac accessibility acct ace acpi aim aio akode alsa altenburgcards amarok amd64 amr amrnb amrr amrwb animgif apache2 arts async audacious audiofile bash-completion beagle berkdb bigpatch binfilter bittorrent bl blas blender-game boost bzip2 cairo ccache cdda cddb cdparanoia cdr cgi chroot cli clucene compress connectionstatus cpio cracklib crypt css cups curl cvs cvsgraph dbus dga dhcp directfb divx dmi dmx doc domainkeys dri dts dv dvb dvd dvdr dvdread ecc editor embedded encode esd exif fam fastbuild fat fbcon festival ffmpeg flac font-server fontconfig fortran gd gd-external gdbm gif gimp glitz glut gmp gnutls gpgme gpm gstreamer gtk gzip hal hddtemp html http httpd hvm iceweasel iconv icq id3 idea idn ieee1394 image imagemagick innodb ipsec irc isdnlog ithreads jabber jack java java5 java6 javascript jce jfs jit john jpeg jpeg2k kde kdecards kdehiddenvisibility kdepim kdeprefix kdrive kexi lame latex libnotify libsamplerate lirc lm_sensors logrotate lzo mad mailwrapper matroska mbox mbrola midi mikmod mime mixer mjpeg mmap mmx mng mod_python motif mozdevelop mozdom mozilla mp3 mp3rtp mp4 mp4live mpd mpeg mpeg2 mpi mplayer mudflap multilib music mysql mysqli ncurses net network network-cron nforce2 nfs nls nntp nodrm nptl nptlonly nsplugin ntfs ntlm nvidia odbc offensive ogg ole opengl openmp openssl oscar pam pango pascal paste64 pcap pcre pdf perl php player pmu png posix postfix povray pppd python qt3support qt4 quicktime randr12 rar rc5 rdesktop readline realmedia reflection remix restrict-javascript rpm rss rtsp samba scanner screen sdl sdl-image sdl-sound sdlaudio seamonkey sensord server session sharedmem shorten shout simplexml skins slang smime smp sndfile sockets socks5 soundex sox speech speex spell spl sql sqlite sqlite3 sse sse2 ssl stream subtitles subversion suhosin svg svgz sylpheed sysfs syslog szip t1lib tcl tcpd tcpwrapper test tetex tga theora thesaurus threads threadsafe thunderbird tidy tiff tk transcode truetype type1 ucs2 unicode usb utempter uuencode v4l v4l2 valgrind vcd vdr vlm vnc vncviewer voice vorbis wav wavpack wifi wireshark wma wmf wmp wordperfect wxwindows x264 xanim xcb xcomposite xen xext xface xforms xfs xine xinerama xinetd xml xorg xosd xpm xrandr xscreensaver xskatcards xterm xv xvid xvmc xvnc yahoo yv12 zip zlib zrtp" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de en" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l vesa vga radeon" Unset: CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS Trying to upgrade to kde-4.2 I had the same problem with stale .pyc/.pyo file with kde-base/pykde4. Removing the stale files and re-emerging pykde4 fixed this and I was able to sucessfully complete the build of kde-meta-4.2.
I guess that at some point, something went wrong and was later fixed in python.eclass.
I think its still broken. When I was just reemerging dev-python/docutils without manually removing the .pyc/.pyo files, this did not fix the problem. So even a current portage / python.eclass does not overwrite .pyc/.pyo files that are newer then the newly installed .py. Looking at python.eclass and /usr/lib64/python2.5/compileall.py I would except, that the fault lies more on the site of python. From compileall.py (in a loop over all .py files): if (ctime > ftime) and not force: continue Hmm... is there a "-f" missing in the call to compileall.py in python.eclass? (I'm just guessing, I do not have deeper knowlegde of python or the eclasses...)
Created attachment 180696 [details, diff] Use max(mtime, ctime) of source (In reply to comment #4) > From compileall.py (in a loop over all .py files): > if (ctime > ftime) and not force: continue Good reference! Notice, however, that "ctime" is not really the creation time of any inode, but instead the modification time of the compiled file, whereas ftime is the modification time of the corresponding source file. When packages are installed, the modification times are pretty much arbitrary. On the other hand, as portage unlinks files befor merging new versions, the creation time of the inode will reflect the time when the package was last merged. This leads to the attached improvement for compileall.py, which always takes the later of mtime and ctime in an attempt to more reliably identify the last change to the file represented by a given path. I'll try to take this patch upstream as well, as I believe it would make sense in all environments, not only for Gentoo. See http://bugs.python.org/issue5128
Created attachment 180702 [details] Script to remerge affected packages This script helps you to remerge all affected packages once your compileall.py has been corrected. Maybe running this script should be suggested in an elog of a python revbump that includes the patch to compileall.py?
(In reply to comment #5) > I'll try to take this patch upstream as well, as I believe it would make sense > in all environments, not only for Gentoo. See http://bugs.python.org/issue5128 Pretty active so far, looks promising. Have a look at the latest development there so I don't have to keep this tracker here in sync.
Package masked sys-apps/sandbox-1.3.3 for now.
*** Bug 257669 has been marked as a duplicate of this bug. ***
*** Bug 257488 has been marked as a duplicate of this bug. ***
*** Bug 257116 has been marked as a duplicate of this bug. ***
*** Bug 257701 has been marked as a duplicate of this bug. ***
for the record, I've been playing with this bug for the last week in a chroot on a clean install. Using paludis instead of portage, I could experiment with things otherwise impossible. I removed *both* python *and* portage, and everything else python related, there were no .pyc files anywhere to be found. then I , using paludis, reinstalled python, and reinstalled 'cracklib' ( which was the package that had my sandbox violation ) and it instantly tried to unlink the very same .pyc files that I had just installed to the system, while performing the version check ( tried to unlink site.pyc ) cracking open a copy of python every time I had a voilation and running import <violated> ; was all that was needed to get python to rebuild its own file outside the sandbox, and then eventually cracklib would install. This pattern is 100% repeatable under sandbox 1.3.3,
*** Bug 257205 has been marked as a duplicate of this bug. ***
*** Bug 257105 has been marked as a duplicate of this bug. ***
*** Bug 256834 has been marked as a duplicate of this bug. ***
ive added a .py[co] check to sandbox-1.3.4 so people should stop hitting this bug for now. once python itself can get sorted out, that check can be dropped. thanks for pursing that Martin.
*** Bug 256845 has been marked as a duplicate of this bug. ***
# Serkan Kaba <serkan@gentoo.org> (04 Feb 2009) # Masked until bug #256953 is resolved >=sys-apps/sandbox-1.3.3 Shouldn't >=sandbox-1.3.4 be the (temporary) fix for this bug? As I understand the problem is not the sandbox, but a bug in the python compileall.py script, as acknowledged in the upstream python bug. The sandbox just correctly catches accesses to stale .pyc/.pyo that compleall.py forgot to update. This fix from comment #14 works, because when running python manually it is not sandboxed and can regenerate the .pyc/.pyo.
it shouldnt have been masked in the first place. fixed now.
(In reply to comment #20) > it shouldnt have been masked in the first place. fixed now. > Sorry I didn't notice that.
Created attachment 181517 [details, diff] Backport of upstream r69481 Upstream accepted a patch for 2.7 and 3.1 but is not going to backport it. Attached is a backport of the core modification from upstream r69481 to the release25-maint branch. See http://svn.python.org/view?view=rev&rev=69481 for the full original changeset and http://bugs.python.org/issue5128 for previous patches against 2.5.
Created attachment 181526 [details] Script to remerge affected packages And here is an updated script to identify and remerge affected packages. In terms of efficiency, it might make more sense to recompile the affected files instead of remerging the whole packages. I'm a bit worried about making too many assumptions about what the ebuild intends its files to be. I'm also not happy with the fact that it seems I'd have to fork a new python process in order to compile both *.pyc and *.pyo, as the debug flag cannot be assigned to. On the whole, this script should play it safe, by remerging the offending packages and trusting that this will never be wrong. I guess the portage guys could hack together a version making use of the database of installed files through some nice portage api, instead of my clumsy parsing of CONTENTS files. Feel free to improve, but I didn't feel like learning the portage api for this. I'm not sure if features like support for different ROOT directories would be required in such a script, but there might be need for improvement in that kind of direction.
Created attachment 181535 [details, diff] Backport of upstream r69481 without "with" This is a variation of comment 22, replacing the "with ... as" with a "try ... catch". It should therefore not only apply to python 2.4 as well, but even work for it as expected. I don't have a 2.4 system around, though, so someone with such a system please test this, see if it works.
*** Bug 260763 has been marked as a duplicate of this bug. ***
Ping? Patch available for over a month now. Similar patch accepted upstream. Bug causes degraded performance for several python packages. What are you waiting for?
*** Bug 264308 has been marked as a duplicate of this bug. ***
Ping again? Python 2.6 has been unmasked, issue still not fixed.
Fixed in Python 2.7 and 3.1.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/sandbox.git/commit/?id=60cff8d682fe7816ca0656d4da27e630855287e7 commit 60cff8d682fe7816ca0656d4da27e630855287e7 Author: Mike Frysinger <vapier@gentoo.org> AuthorDate: 2021-10-22 04:18:15 +0000 Commit: Mike Frysinger <vapier@gentoo.org> CommitDate: 2021-10-22 04:19:44 +0000 libsandbox: drop old *.py[co] hack #775416 With our eclasses & python frameworks responsible for generating these files now, we should be able to reject write attempts to these again. Lets turn it back on and see what blows up. Bug: http://bugs.gentoo.org/256953 Closes: https://bugs.gentoo.org/775416 Signed-off-by: Mike Frysinger <vapier@gentoo.org> libsandbox/libsandbox.c | 14 -------------- 1 file changed, 14 deletions(-)
*** Bug 554252 has been marked as a duplicate of this bug. ***
