Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 100479 - KeyError: '.' in dblink.unmerge() when CONTENTS contains a missing mtime
Summary: KeyError: '.' in dblink.unmerge() when CONTENTS contains a missing mtime
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 87147 123827 (view as bug list)
Depends on:
Blocks: 115839
  Show dependency tree
 
Reported: 2005-07-27 09:06 UTC by Zarick Lau
Modified: 2006-02-26 13:58 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
The offending CONTENTS file (CONTENTS,3.75 KB, text/plain)
2005-07-27 09:08 UTC, Zarick Lau
Details
portage-2.0.51.22-dblink.unmerge-pkgfiles-key.patch (portage-2.0.51.22-dblink.unmerge-pkgfiles-key.patch,3.09 KB, patch)
2005-07-28 16:57 UTC, Zac Medico
Details | Diff
My patch against dblink.getcontents (portage-2.0.51.22-r2-getcontents.patch,565 bytes, patch)
2005-07-29 10:37 UTC, Zarick Lau
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zarick Lau 2005-07-27 09:06:37 UTC
When I just upgraded from screen-4.0.1 -> screen-4.0.2-r4
screen 4.0.2-r4 merged successfully.
but unmerging of screen 4.0.1 does complete nicely.
When running emerge --unmerge =screen-4.0.1
it will give me this stackstrace
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2954, in ?
    if 1==unmerge(myaction, myfiles):
  File "/usr/bin/emerge", line 2251, in unmerge
   
retval=portage.unmerge(mysplit[0],mysplit[1],portage.root,mysettings,unmerge_action
not in ["clean","prune"])
  File "/usr/lib/portage/pym/portage.py", line 2901, in unmerge
    mylink.unmerge(trimworld=mytrimworld,cleanup=1)
  File "/usr/lib/portage/pym/portage.py", line 6176, in unmerge
    if (pkgfiles[obj][0] not in ("dir","fif","dev","sym")) and (lmtime !=
pkgfiles[obj][1]):
KeyError: '.'


I have try to trace down the problem and turns out that, it seems that 
the CONTENTS (of screen 4.0.1) has a weired line
obj /var/run/screen/.keep d41d8cd98f00b204e9800998ecf8427e

The problem is, the timestamp field is absent.

In dblink.getcontents() it always assume that, there is 4 fields in CONTENTS
file.

I am not familiar with portage, I also don't know if this is a bug of portage or
caused by a broken ebuild. But I think, dblink.getcontents should be able to
handle this trivial exception case.

I'll attach my broken CONTENTS

Reproducible: Always
Steps to Reproduce:
1.
2.
3.
Comment 1 Zarick Lau 2005-07-27 09:08:05 UTC
Created attachment 64445 [details]
The offending CONTENTS file

Notice the '/var/run/screen/.keep' line which has missed the timestamp
Comment 2 Zac Medico gentoo-dev 2005-07-28 16:57:54 UTC
Created attachment 64584 [details, diff]
portage-2.0.51.22-dblink.unmerge-pkgfiles-key.patch

The problem here is that we are iterating through pkgfiles.keys() and inside
the iteration loop we modify the obj variable so subsequent calls to
pkgfiles[obj] may fail with the reported error.

for obj in mykeys:
	obj=os.path.normpath(objkey)
	if obj[:2]=="//":
		obj=obj[1:]

The obvious solution is to use a separate variable for the pkgfiles key.  My
patch introduces a new variable called objkey and replaces all pkgfiles[obj]
calls with pkgfiles[objkey].
Comment 3 Zarick Lau 2005-07-29 10:09:29 UTC
Right, I forget to provide the `emerge info`
Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.3.5-20050130,
glibc-2.3.5-r0, 2.6.11-hardened-r15-idesirq-nat-v4l-allmod i686)
=================================================================
System uname: 2.6.11-hardened-r15-idesirq-nat-v4l-allmod i686 AMD Athlon(tm) XP
1800+
Gentoo Base System version 1.6.13
distcc 2.13 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632)
[disabled]dev-lang/python:     2.3.5
sys-apps/sandbox:    1.2.11
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9, 1.8.5-r3, 1.9.5
sys-devel/binutils:  2.15.92.0.2-r10
sys-devel/libtool:   1.5.18-r1
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=athlon-xp -pipe -ffast-math -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config
/var/bind /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/init.d /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -march=athlon-xp -pipe -ffast-math -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig buildpkg distlocks sandbox sfperms strict"
GENTOO_MIRRORS="http://gentoo.oregonstate.edu
http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="en_US.UTF-8"
LINGUAS="en zh_HK zh_TW zh_CN ja kr"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 3dnow X a52 aac aalib alsa apache2 apm avi berkdb bitmap-fonts cdr cjk
crypt cscope cups curl directfb divx4linux dvd dvdread eds emboss encode esd evo
fam fame fbcon flac foomaticdb fortran gcj gd gdbm gif gnome gpm gstreamer gtk
gtk2 guile hal imagemagick imap imlib ipv6 java jpeg junit libg++ libwww lzo mad
mbox mcal mikmod mjpeg mmx motif mozilla mp3 mpeg mysql ncurses nls ogg
oggvorbis opengl oss pam pdflib perl png ppds python qt quicktime readline samba
sasl scanner sdl slang speex spell sqlite sse ssl svga tcltk tcpd tiff truetype
truetype-fonts type1-fonts usb v4l v4l2 vorbis xine xml xml2 xrandr xv xvid zlib
linguas_en linguas_zh_HK linguas_zh_TW linguas_zh_CN linguas_ja linguas_kr
userland_GNU kernel_linux elibc_glibc"
Unset:  ASFLAGS, CTARGET, LC_ALL, LDFLAGS, MAKEOPTS, PORTDIR_OVERLAY
Comment 4 Zarick Lau 2005-07-29 10:33:46 UTC
Hi Zac, Gentoo users / dev,

I can confirm that, your patch make the unmerge completed successfully.
And, I also confirm that, the problem is caused by malformed CONTENTS in pkg db.

However, when I studied the problem again, I found the following point:

When everything is okay and correct, the flow should be:
1. In the CONTENTS file, there is a line:
   obj /var/run/screen/.keep d41d8cd98f00b204e9800998ecf8427e 1122656930
2. During emerge --unmerge, dblink.getcontents() will parse the line and
   create a pkgfiles mapping. It use the following lines of code to handle
   'obj' type line:

   if mydat[0]=="obj":
       #format: type, mtime, md5sum
       pkgfiles[string.join(mydat[1:-2]," ")]=[mydat[0], mydat[-1], mydat[-2]]

   The resulting record created will be like this:
   '/var/run/screen/.keep': ['obj', 
                             '1122656930', 
                             'd41d8cd98f00b204e9800998ecf8427e'] 

3. The pkgfiles mapping created in getcontents() is processed by  
   dblink.unmerge(), it will extract all keys (ie. path for all files 
   belongs to a package) and compare the mtime, md5sum, etc. for obj
   type entry.

In the current situation, the flow should be:
1. First the offending line (in CONTENTS) is (NOTE: no timestamp field):
   obj /var/run/screen/.keep d41d8cd98f00b204e9800998ecf8427e

2. During dblink.getcontents(), the logic to parse the line for 'obj'
   type will create a wrong resulting record:
   
   '': ['obj', 'd41d8cd98f00b204e9800998ecf8427e', '/var/run/screen/.keep']

   Note that, the key of the mapping should be the filepath. And list item
   is wrong also. In correct case, it should be [type, timestamp, md5sum]

   If the problem is the following line, I like:

   pkgfiles[string.join(mydat[1:-2]," ")]=[mydat[0], mydat[-1], mydat[-2]]

   Note the use of "mydat[1:-2]", if the line has only three field (my case)
   len(mydat) is only 3, so the slice will resulting a empty list. which 
   result a empty string as the key of the record.

I think, the patch Zac is good (I already test your patch :) by unmerge 
my stall screen-4.0.1. But, the problem is not how unmerge() iterate 
the keys, the problem is rooted at, getcontents always assume 
len(mydat) >=4. I believe, the use of mydat[1:-2] instead of mydat[1] 
is catering filename with 'space'. (mydat[1] is only part of the filename)

I have another solution, which is to simply check the len(mydat) if type
is 'obj'.
Comment 5 Zarick Lau 2005-07-29 10:37:17 UTC
Created attachment 64645 [details, diff]
My patch against dblink.getcontents

The patch target dblink.getcontents() directly.
If the current parsing line is of type 'obj', check the number
of field. If the len is <4, by pass that line.

As record in CONTENTS for type obj must be >=4.
Comment 6 Zac Medico gentoo-dev 2005-07-29 11:54:57 UTC
Hi Zarick, I agree that the contents parsing is broken, in fact, *severely*
broken (see Bug 14983).  We need a new and improved file format.

Do you have any idea how you got the malformed line in your contents file?  We
should address that too.  Mine has the timestamp:

obj /var/run/screen/.keep d41d8cd98f00b204e9800998ecf8427e 1118274194
Comment 7 Zac Medico gentoo-dev 2005-07-31 19:40:58 UTC
This seems to be a duplicate of Bug 87147.
Comment 8 Zarick Lau 2005-08-03 04:55:37 UTC
Hi Zac,

Yes agree, this seems a duplicate of 87147.

For the "broken" CONTENTS file, I have no idea why the file is broken.
The timestamp is missed. May be due to system crash.

Though, my system haven't crashed for months.

On the other hand, I do think that, portage should have able to 
handle broken CONTENTS file.
At least, omit the borken line in the files or tell user that the file
is broken and the unmerge can't be continued unless user manually
repair the file.
Comment 9 Zac Medico gentoo-dev 2005-08-03 13:57:05 UTC
(In reply to comment #8)
> For the "broken" CONTENTS file, I have no idea why the file is broken.
> The timestamp is missed. May be due to system crash.
> 

A system crash will not delete 1 mtime from a CONTENTS file.  There must be a
problem when the mtime is written in dblink.mergeme() inside portage.py.

It do not recommend changing the contents parsing code in order to workaround a
bug in dblink.mergeme().  That would clutter the contents parsing code and make
the root problem less visible.  If necessary, it is trivial to write a script to
fix your broken CONTENTS files.
Comment 10 Alec Warner (RETIRED) archtester gentoo-dev Security 2005-08-07 16:51:24 UTC
*** Bug 87147 has been marked as a duplicate of this bug. ***
Comment 11 Zac Medico gentoo-dev 2006-02-09 23:31:17 UTC
A fix for the KeyError: '.' in dblink.unmerge() is in svn for 2.1_pre5.
Comment 12 Zac Medico gentoo-dev 2006-02-23 09:15:28 UTC
*** Bug 123827 has been marked as a duplicate of this bug. ***
Comment 13 Zac Medico gentoo-dev 2006-02-26 13:58:37 UTC
Released in 2.1_pre5.