Bugzilla Bug 256953

I just tried to upgrade eix from 0.15.2 to 0.15.3, but that died with a sandbox
violation. python was trying to remove the docutils/utils.pyc and the sandbox
stopped that.

The cause seemed to be, that emerge "forgot" to rebuild the .pyc and .pyo files
from dev-python/docutils when I upgraded that package from 0.4-r3 to 0.5




Reproducible: Always

Steps to Reproduce:
I think the following was happening:
* docutils-0.4 source gets packaged -> Timestamp A
* docutils-0.5 source gets packaged -> Timestamp B
* I emerge docutils-0.4-r3 -> .py has Timestamp A; .pyc/.pyo get Timestamp C
with A<B<C
* I upgrade to docutils-0.5
  * emerge installs the .py that still have the original Timestamp B
  * emerge tries to build the .pyc/.pyo, but because B<C they will not be
rebuild
* I try to upgrade eix -> wants to use docutils/utils.py -> sandbox violation

After manually removing all .pyc/.pyo from
/usr/lib64/python2.5/site-packages/docutils and all subdirectories and
reemerging dev-python/docutils-0.5 the emerge of eix-0.15.3 worked.
(Just remerging docutils-0.5 without manually removing these files did not
work)
Actual Results:  
--------------------------- ACCESS VIOLATION SUMMARY
---------------------------
LOG FILE "/var/log/sandbox/sandbox-24636.log"

VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: unlink
S: deny
P: /usr/lib64/python2.5/site-packages/docutils/utils.pyc
A: /usr/lib64/python2.5/site-packages/docutils/utils.pyc
R: /usr/lib64/python2.5/site-packages/docutils/utils.pyc
C: python /usr/bin/rst2html.py format.txt --embed-stylesheet
--stylesheet-path=stylesheet.css
--------------------------------------------------------------------------------


Expected Results:  
Upgrade of eix.

From /usr/lib64/python2.5/site-packages/docutils:
after the eix failure:
-rw-r--r-- 1 root root 21280 Apr  3  2007 utils.py
-rw-r--r-- 1 root root 21973 Aug 24  2007 utils.pyc
-rw-r--r-- 1 root root 21870 Aug 24  2007 utils.pyo
after just remerging docutils-0.5:
-rw-r--r-- 1 root root 21280 Apr  3  2007 utils.py
-rw-r--r-- 1 root root 21973 Aug 24  2007 utils.pyc
-rw-r--r-- 1 root root 21870 Aug 24  2007 utils.pyo
after removing all .pyc/.pco and remerging docutils-0.5:
-rw-r--r-- 1 root root 21280 Apr  3  2007 utils.py
-rw-r--r-- 1 root root 22441 Jan 30 16:40 utils.pyc
-rw-r--r-- 1 root root 22338 Jan 30 16:40 utils.pyo

(August 24 2007 matches the last emerge of docutils-0.4)

I'm currently using sys-apps/portage-2.2_rc23. In August, when I initially
upgraded to docutils-0.5 I was using sys-apps/portage-2.2_rc8.

Please post your `emerge --info' too.

emerge --info:
Portage 2.2_rc23 (default/linux/amd64/2008.0, gcc-4.3.3,
glibc-2.9_p20081201-r1, 2.6.29-rc1 x86_64)
=================================================================
System uname:
Linux-2.6.29-rc1-x86_64-Dual-Core_AMD_Opteron-tm-_Processor_2218-with-glibc2.2.5
Timestamp of tree: Fri, 30 Jan 2009 14:00:01 +0000
ccache version 2.4 [disabled]
app-shells/bash:     3.2_p48
dev-java/java-config: 1.3.7-r1, 2.1.7
dev-lang/python:     2.5.4-r2
dev-util/ccache:     2.4-r8
dev-util/cmake:      2.6.2-r1
sys-apps/baselayout: 2.0.0
sys-apps/openrc:     0.4.2
sys-apps/sandbox:    1.3.3
sys-devel/autoconf:  2.13, 2.63
sys-devel/automake:  1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2
sys-devel/binutils:  2.19
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6a
virtual/os-headers:  2.6.28-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-pipe -march=opteron -O3 -fomit-frame-pointer -fweb -frename-registers
-fsee -ftracer -ftree-loop-im -funswitch-loops -fivopts"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/kde/4.1/env /usr/kde/4.1/share/config
/usr/kde/4.1/shutdown /usr/kde/4.2/env /usr/kde/4.2/share/config
/usr/kde/4.2/shutdown /usr/share/config /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/
/etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release
/etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/
/etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo
/etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-pipe -march=opteron -O3 -fomit-frame-pointer -fweb
-frename-registers -fsee -ftracer -ftree-loop-im -funswitch-loops -fivopts"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps y --jobs=4 --load-average=5 --keep-going"
FEATURES="buildpkg distlocks fixpackages parallel-fetch protect-owned sandbox
sfperms splitdebug strict unmerge-orphans userfetch userpriv usersandbox"
GENTOO_MIRRORS="http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/
http://mirror.ovh.net/gentoo-distfiles/ http://ds.thn.htu.se/linux/gentoo
http://gentoo.inf.elte.hu/
http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/"
LDFLAGS="-Wl,-O1"
LINGUAS="de en"
MAKEOPTS="-j5"
PKGDIR="/var/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --stats --timeout=180 --exclude=/distfiles
--exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/portage-local/layman/sunrise /usr/portage-local/layman/nx
/usr/portage-local/layman/secondlife /usr/portage-local/layman/x11
/usr/portage-local/layman/desktop-effects /usr/portage-local/layman/gentoojp
/root/ebuilds"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext 7zip X a52 aac accessibility acct ace acpi aim aio akode
alsa altenburgcards amarok amd64 amr amrnb amrr amrwb animgif apache2 arts
async audacious audiofile bash-completion beagle berkdb bigpatch binfilter
bittorrent bl blas blender-game boost bzip2 cairo ccache cdda cddb cdparanoia
cdr cgi chroot cli clucene compress connectionstatus cpio cracklib crypt css
cups curl cvs cvsgraph dbus dga dhcp directfb divx dmi dmx doc domainkeys dri
dts dv dvb dvd dvdr dvdread ecc editor embedded encode esd exif fam fastbuild
fat fbcon festival ffmpeg flac font-server fontconfig fortran gd gd-external
gdbm gif gimp glitz glut gmp gnutls gpgme gpm gstreamer gtk gzip hal hddtemp
html http httpd hvm iceweasel iconv icq id3 idea idn ieee1394 image imagemagick
innodb ipsec irc isdnlog ithreads jabber jack java java5 java6 javascript jce
jfs jit john jpeg jpeg2k kde kdecards kdehiddenvisibility kdepim kdeprefix
kdrive kexi lame latex libnotify libsamplerate lirc lm_sensors logrotate lzo
mad mailwrapper matroska mbox mbrola midi mikmod mime mixer mjpeg mmap mmx mng
mod_python motif mozdevelop mozdom mozilla mp3 mp3rtp mp4 mp4live mpd mpeg
mpeg2 mpi mplayer mudflap multilib music mysql mysqli ncurses net network
network-cron nforce2 nfs nls nntp nodrm nptl nptlonly nsplugin ntfs ntlm nvidia
odbc offensive ogg ole opengl openmp openssl oscar pam pango pascal paste64
pcap pcre pdf perl php player pmu png posix postfix povray pppd python
qt3support qt4 quicktime randr12 rar rc5 rdesktop readline realmedia reflection
remix restrict-javascript rpm rss rtsp samba scanner screen sdl sdl-image
sdl-sound sdlaudio seamonkey sensord server session sharedmem shorten shout
simplexml skins slang smime smp sndfile sockets socks5 soundex sox speech speex
spell spl sql sqlite sqlite3 sse sse2 ssl stream subtitles subversion suhosin
svg svgz sylpheed sysfs syslog szip t1lib tcl tcpd tcpwrapper test tetex tga
theora thesaurus threads threadsafe thunderbird tidy tiff tk transcode truetype
type1 ucs2 unicode usb utempter uuencode v4l v4l2 valgrind vcd vdr vlm vnc
vncviewer voice vorbis wav wavpack wifi wireshark wma wmf wmp wordperfect
wxwindows x264 xanim xcb xcomposite xen xext xface xforms xfs xine xinerama
xinetd xml xorg xosd xpm xrandr xscreensaver xskatcards xterm xv xvid xvmc xvnc
yahoo yv12 zip zlib zrtp" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x
ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0
intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci"
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file
hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug
rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic
auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm
authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache
dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter
file_cache filter headers ident imagemap include info log_config logio
mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer
proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir
usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="mouse evdev" KERNEL="linux"
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses
text" LINGUAS="de en" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l vesa vga
radeon"
Unset:  CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LANG, LC_ALL,
PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


Trying to upgrade to kde-4.2 I had the same problem with stale .pyc/.pyo file
with kde-base/pykde4. Removing the stale files and re-emerging pykde4 fixed
this and I was able to sucessfully complete the build of kde-meta-4.2.

I guess that at some point, something went wrong and was later fixed in
python.eclass.

I think its still broken.

When I was just reemerging dev-python/docutils without manually removing the
.pyc/.pyo files, this did not fix the problem.
So even a current portage / python.eclass does not overwrite .pyc/.pyo files
that are newer then the newly installed .py.

Looking at python.eclass and /usr/lib64/python2.5/compileall.py I would except,
that the fault lies more on the site of python.
From compileall.py (in a loop over all .py files):
                if (ctime > ftime) and not force: continue

Hmm... is there a "-f" missing in the call to compileall.py in python.eclass?
(I'm just guessing, I do not have deeper knowlegde of python or the
eclasses...)

Created an attachment (id=180696) [details]
Use max(mtime, ctime) of source

(In reply to comment #4)
> From compileall.py (in a loop over all .py files):
>                 if (ctime > ftime) and not force: continue

Good reference! Notice, however, that "ctime" is not really the creation time
of any inode, but instead the modification time of the compiled file, whereas
ftime is the modification time of the corresponding source file.

When packages are installed, the modification times are pretty much arbitrary.
On the other hand, as portage unlinks files befor merging new versions, the
creation time of the inode will reflect the time when the package was last
merged.

This leads to the attached improvement for compileall.py, which always takes
the later of mtime and ctime in an attempt to more reliably identify the last
change to the file represented by a given path.

I'll try to take this patch upstream as well, as I believe it would make sense
in all environments, not only for Gentoo. See http://bugs.python.org/issue5128

Created an attachment (id=180702) [details]
Script to remerge affected packages

This script helps you to remerge all affected packages once your compileall.py
has been corrected. Maybe running this script should be suggested in an elog of
a python revbump that includes the patch to compileall.py?

(In reply to comment #5)
> I'll try to take this patch upstream as well, as I believe it would make sense
> in all environments, not only for Gentoo. See http://bugs.python.org/issue5128

Pretty active so far, looks promising. Have a look at the latest development
there so I don't have to keep this tracker here in sync.

Package masked sys-apps/sandbox-1.3.3 for now.

*** Bug 257669 has been marked as a duplicate of this bug. ***

*** Bug 257488 has been marked as a duplicate of this bug. ***

*** Bug 257116 has been marked as a duplicate of this bug. ***

*** Bug 257701 has been marked as a duplicate of this bug. ***

for the record, I've been playing with this bug for the last week in a chroot
on a clean install. Using paludis instead of portage, I could experiment with
things otherwise impossible. 

I removed *both* python *and* portage,  and everything else python related,
there were no .pyc files anywhere to be found. 

then I , using paludis, reinstalled python, and reinstalled 'cracklib' ( which
was the package that had my sandbox violation ) and it instantly tried to
unlink the very same .pyc files that I had just installed to the system, while
performing  the version check ( tried to unlink site.pyc ) 

cracking open a copy of python every time I had a voilation and running 

import <violated> ; 

was all that was needed to get python to rebuild its own file outside the
sandbox, and then eventually cracklib would install.

This pattern is 100% repeatable under sandbox 1.3.3, 

*** Bug 257205 has been marked as a duplicate of this bug. ***

*** Bug 257105 has been marked as a duplicate of this bug. ***

*** Bug 256834 has been marked as a duplicate of this bug. ***

ive added a .py[co] check to sandbox-1.3.4 so people should stop hitting this
bug for now.  once python itself can get sorted out, that check can be dropped.
 thanks for pursing that Martin.

*** Bug 256845 has been marked as a duplicate of this bug. ***

# Serkan Kaba <serkan@gentoo.org> (04 Feb 2009)
# Masked until bug #256953 is resolved
>=sys-apps/sandbox-1.3.3

Shouldn't >=sandbox-1.3.4 be the (temporary) fix for this bug?

As I understand the problem is not the sandbox, but a bug in the python
compileall.py script, as acknowledged in the upstream python bug.

The sandbox just correctly catches accesses to stale .pyc/.pyo that
compleall.py forgot to update.

This fix from comment #14 works, because when running python manually it is not
sandboxed and can regenerate the .pyc/.pyo.

it shouldnt have been masked in the first place.  fixed now.

(In reply to comment #20)
> it shouldnt have been masked in the first place.  fixed now.
> 

Sorry I didn't notice that.

Created an attachment (id=181517) [details]
Backport of upstream r69481

Upstream accepted a patch for 2.7 and 3.1 but is not going to backport it.
Attached is a backport of the core modification from upstream r69481 to the
release25-maint branch. See http://svn.python.org/view?view=rev&rev=69481 for
the full original changeset and http://bugs.python.org/issue5128 for previous
patches against 2.5.

Created an attachment (id=181526) [details]
Script to remerge affected packages

And here is an updated script to identify and remerge affected packages.

In terms of efficiency, it might make more sense to recompile the affected
files instead of remerging the whole packages. I'm a bit worried about making
too many assumptions about what the ebuild intends its files to be. I'm also
not happy with the fact that it seems I'd have to fork a new python process in
order to compile both *.pyc and *.pyo, as the debug flag cannot be assigned to.
On the whole, this script should play it safe, by remerging the offending
packages and trusting that this will never be wrong.

I guess the portage guys could hack together a version making use of the
database of installed files through some nice portage api, instead of my clumsy
parsing of CONTENTS files. Feel free to improve, but I didn't feel like
learning the portage api for this. I'm not sure if features like support for
different ROOT directories would be required in such a script, but there might
be need for improvement in that kind of direction.

Created an attachment (id=181535) [details]
Backport of upstream r69481 without "with"

This is a variation of comment 22, replacing the "with ... as" with a "try ...
catch". It should therefore not only apply to python 2.4 as well, but even work
for it as expected. I don't have a 2.4 system around, though, so someone with
such a system please test this, see if it works.

*** Bug 260763 has been marked as a duplicate of this bug. ***

Ping?

Patch available for over a month now.
Similar patch accepted upstream.
Bug causes degraded performance for several python packages.

What are you waiting for?

*** Bug 264308 has been marked as a duplicate of this bug. ***

Ping again? Python 2.6 has been unmasked, issue still not fixed.