bash-2.05b$ cat test EOL after test/pkg test/pkg with space test/pkg-1.2 with version bash-2.05b$ python -c 'import portage; portage.fixdbentries("test/pkg","test/done",".")' bash-2.05b$ cat test EOL after test/donetest/donewith space test/done.2 with version ...and with less... EOL after test/done^Atest/done^Awith space test/done^A.2 with version It also seems to miss some files but I have been unable to manually reproduce it thus far.
Created attachment 28274 [details, diff] Fixes the corruption Very simple. The regexps were using a single '\' and python was escaping it. Still not sure on why files are being missed though. Will investigate further.
Created attachment 28275 [details, diff] Fixes the corruption and missing files I'll go through my changes. - mycontent = re.sub(old_value+"$", new_value, mycontent) - mycontent = re.sub(old_value+"(\s)", new_value+"\1", mycontent) + mycontent = re.sub(old_value+"(\\s)", new_value+"\\1", mycontent) \s already catches newlines so the $ test isn't needed. - mycontent = re.sub(old_value+"(-[^a-zA-Z])", new_value+"\1", mycontent) - mycontent = re.sub(old_value+"([^a-zA-Z0-9-])", new_value+"\1", mycontent) + mycontent = re.sub(old_value+"(-[0-9])", new_value+"\\1", mycontent) The originals here were not catching ">=some/pkg-1.2". I'm not sure exactly what they are meant to do, but my change will ensure that any replacements are done without affecting package names that include the moved package as a substring. eg. "net-p2p/gift-openft" won't be affected by a change to "net-p2p/gift"
*** Bug 46466 has been marked as a duplicate of this bug. ***
Upgrading severity to blocker as it's preventing uninstallation of affected packages.
I don't mind if it's changed because I'm not sure what the latter two regex's were meant to do but this really needs to go into the next release. If I am able to do it myself, let me know. I don't feel like experimenting that much. :)
Okay, I can see what the original second pair is meant to do and it seems to be correct. I'll investigate why it wasn't working and fix this up.
Created attachment 28929 [details, diff] Reverted to previous latter pair (against -r4) The previous fix would not catch "foo/bar)" which is what the second pair of regexps addressed in the originals. The cause for them not catching previously seems to be due to the earlier corruption. Anyway, this one works.
jstubbs: The patch seems good. But why did you remove this line? mycontent = re.sub(old_value+"$", new_value, mycontent) If the line doesn't exists, just "cat/pkg-ver" would not match.
Because the strings include newlines, re.sub seems to automatically switch to multi-line mode. This causes \s to match newline characters as well. I haven't yet found documentation to back this up, but my testing shows that it is not required.
There is a file with no newline character sometimes. (only 1 line or in last line) And if we don't remove the line, there would not be bug. So, I think we should not remove the line.
Sorry for not getting earlier to his, I'm a bit distracted atm.
Created attachment 29014 [details] Script to fix /var/db/pkg Doesn't consider $ROOT or binary packages. Will add support for those next.
Created attachment 29020 [details, diff] Patch to portage to fix corruption automatically Adds a "corrupted" key to mtimedb["updates"] and performs the same function as the previous script if the key is not defined. Small bug in the previous script. - if "\1" not in orig: + if "\1" not in "".join(orig): I *will* get out of the habit of making last minute changes without retesting.
By the way, I didn't modify the script/patch to fix *DEPEND and xxx.ebuild and fixed in the xpak of tbz2s or in ${ROOT}var/db/pkg because do_update doesn't do it either. Is this a problem?
forgot to add this so that updates get reprocessed after fixing corruption: if not mtimedb["updates"].has_key("corrupted"): + mtimedb["updates"] = {} for cat in os.listdir("/"+VDB_PATH):
So am I correct to assume that Bug# 48545 is a duplicate to this bug? If so to get things running again - should I just run the "Script to fix /var/db/pkg" above? or the patch? Should I be recording things when running the script for latter analysis? What other information can I give to help sort this out?
As my Bug# 48545 was closed - I tried running the above "Script to fix /var/db/pkg". This didn't change the symptoms of my problem as shown below: Does anyone have any suggestions? >>> sys-kernel/linux-headers-2.4.22 merged. >>> clean: No packages selected for removal. >>> Auto-cleaning packages ... net-firewall/shorewall selected: 1.4.10b protected: 2.0.1 omitted: none >>> 'Selected' packages are slated for removal. >>> 'Protected' and 'omitted' packages will not be removed. >>> Waiting 5 seconds before starting... >>> (Control-C to abort)... >>> Unmerging in: 5 4 3 2 1 >>> Unmerging net-firewall/shorewall-1.4.10b... No package files given... Grabbing a set. /usr/sbin/ebuild.sh: /var/db/pkg/net-firewall/shorewall-1.4.10b/shorewall-1.4.10b.ebuild: line 39: unexpected EOF while looking for matching `"' /usr/sbin/ebuild.sh: /var/db/pkg/net-firewall/shorewall-1.4.10b/shorewall-1.4.10b.ebuild: line 41: syntax error: unexpected end of file !!! ERROR: net-firewall/shorewall-1.4.10b failed. !!! Function , Line 1291, Exitcode 1 !!! error sourcing ebuild !!! FAILED prerm: 1 Thanks for your time.
After hunting through everything I know; I found that the ebuild in /var/db/pkg/net-firewall/shorewall-1.4.10b/ had a ^A character instead of " after the DEPEND. I changed it back to a " and the unemerge problem was resolved. I've just run a badblocks and have picked up nothing - so I'm at a loss as to how that ^A character came to be. Therefore, I appologise for the noise.
The "Script to fix /var/db/pkg" should have fixed the shorewall ebuild. Finding and fixing the ^A is exactly what it is designed to do. You said that the final quote had been replaced by a ^A, though. I think I missed out that case. Will do some more testing and fix. :)
*** Bug 49872 has been marked as a duplicate of this bug. ***
*** Bug 53213 has been marked as a duplicate of this bug. ***
After saving the previously attached script and running it, I get the following error and am still unable to cleanly purge shorewall without manual modification of /var/db/pkg/net-firewall/shorewall-1.4.10b/shorewall-1.4.10b.ebuild # ./fix Traceback (most recent call last): File "./fix", line 69, in ? fix_entries(vardb+cat+"/"+pkg) File "./fix", line 10, in fix_entries ebuildfile = open(path+ebuild) IOError: [Errno 2] No such file or directory: '/var/db/pkg/media-sound/madplay-0.14.2b-r2/madplay-0.14.2b-r2.ebuild' It seems it's expecting the file to be named differently than it is: # ls /var/db/pkg/media-sound/madplay-0.14.2b-r2/mad* mad-0.14.2b-r2.ebuild Should the script be modified to ignore such "No such file" errors?
Hum, I overlooked the bug in the script. - if "\1" not in orig: + if "\1" not in "".join(orig) After making that change, the script executes successfully. # ./fix broken ebuild mailx-8.1.2.20021129-r2.ebuild # However, shorewall still does not purge successfully, failing with the same error.
The "Script to fix /var/db/pkg" does not work on my system because there is no file "environment.bz2" but a file "environment".
After bzip2'ing the environment file manually, I get the same output as in Comment #23: broken ebuild mailx-8.1.2.20021129-r2.ebuild Since it's bailing out, the ebuild does not get repaired.
On another one of my hosts, there's a package installed that has neither "environment" nor "environment.bz2": [normal system update] mail-client/mailx selected: 8.1.2.20021129-r2 protected: 8.1.2.20021129-r3 omitted: none >>> 'Selected' packages are slated for removal. >>> 'Protected' and 'omitted' packages will not be removed. >>> Unmerging mail-client/mailx-8.1.2.20021129-r2... No package files given... Grabbing a set. /usr/sbin/ebuild.sh: /var/db/pkg/mail-client/mailx-8.1.2.20021129-r2/mailx-8.1.2.20021129-r2.ebuild: line 40: unexpected EOF while looking for matching `"' /usr/sbin/ebuild.sh: /var/db/pkg/mail-client/mailx-8.1.2.20021129-r2/mailx-8.1.2.20021129-r2.ebuild: line 67: syntax error: unexpected end of file !!! ERROR: mail-client/mailx-8.1.2.20021129-r2 failed. !!! Function , Line 1294, Exitcode 1 !!! error sourcing ebuild !!! FAILED prerm: 1 root@hybrid:~# ./2004-07-23-fixvardbentries bunzip2: Can't open input file /var/db/pkg/app-admin/logrotate-3.6.5-r1/environment.bz2: No such file or directory. Traceback (most recent call last): File "./2004-07-23-fixvardbentries", line 70, in ? fix_entries(vardb+cat+"/"+pkg) File "./2004-07-23-fixvardbentries", line 17, in fix_entries envfile = open(path+"environment") IOError: [Errno 2] No such file or directory: '/var/db/pkg/app-admin/logrotate-3.6.5-r1/environment' root@hybrid:~# ls /var/db/pkg/app-admin/logrotate-3.6.5-r1/ CATEGORY CC CFLAGS CONTENTS CXX DEPEND PDEPEND PROVIDE SLOT logrotate-3.6.5-r1.ebuild CBUILD CDEPEND CHOST COUNTER CXXFLAGS LICENSE PF RDEPEND USE root@hybrid:~# emerge info Portage 2.0.50-r9 (default-1.0, gcc-2.95.3, glibc-2.2.5-r4,2.2.5-r9, 2.4.25-hybrid-1) ================================================================= System uname: 2.4.25-hybrid-1 i586 AMD-K6(tm) 3D processor Gentoo Base System version 1.4.16 distcc 2.13 i586-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] Autoconf: sys-devel/autoconf-2.59-r3 Automake: sys-devel/automake-1.8.3 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-mcpu=k6" CHOST="i586-pc-linux-gnu" COMPILER="" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/afs/C /etc/afs/afsws /etc/afs/modload /etc/gconf /etc/make.globals /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -mcpu=i686 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://192.168.1.2/gentoo http://mirrors.sec.informatik.tu-darmstadt.de/gentoo http://ftp.easynet.nl/mirror/gentoo/ ftp://ftp.easynet.nl/mirror/gentoo/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://hybrid.sascha.silbe.org/gentoo-portage" USE="3dnow X acl afs berkdb cdr crypt gdbm imlib ipv6 j-noaim j-nomsn j-noyahoo jpeg krb4 libg++ libwww mikmod mmx mpeg nas ncurses nls oggvorbis oss pam pdflib perl png python readline scanner slang spell ssl tetex tiff truetype x86 xml2" root@hybrid:~#
The escaping in fixdbentries isn't fixed in 2.0.50-r9. Nick, can you add the following in and push out another release? The longer this waits, the more corruption occurs... - mycontent = re.sub(old_value+"(\s)", new_value+"\1", mycontent) - mycontent = re.sub(old_value+"(-[^a-zA-Z])", new_value+"\1", mycontent) - mycontent = re.sub(old_value+"([^a-zA-Z0-9-])", new_value+"\1", mycontent) + mycontent = re.sub(old_value+"(\\s)", new_value+"\\1", mycontent) + mycontent = re.sub(old_value+"(-[^a-zA-Z])", new_value+"\\1", mycontent) + mycontent = re.sub(old_value+"([^a-zA-Z0-9-])", new_value+"\\1", mycontent) As for the corruption that's already out there, I'll see if I can fix that script.
This is a difficult bug to search for :-) I just found and fixed 4 /var/db/pkg/XXX.ebuild's with corruption in them. I found the first because of this error when emerge was cleaning old sox after upgrade today: >>> Unmerging media-sound/sox-12.17.4-r1... No package files given... Grabbing a set. /usr/sbin/ebuild.sh: /var/db/pkg/media-sound/sox-12.17.4-r1/sox-12.17.4-r1.ebuild: line 27: unexpected EOF while looking for matching `"' /usr/sbin/ebuild.sh: /var/db/pkg/media-sound/sox-12.17.4-r1/sox-12.17.4-r1.ebuild: line 45: syntax error: unexpected end of file I found an a char 0x01 (hex 01) in this file. Searching /var/db/pkg/*.ebuild found 3 more of this bogus char in ebuilds. Only 1 of the four ebuilds, sox, was actually missing an " in the DEPEND line maybe overwritten by the bad char? Looking at the past entries in this bug makes me wonder if this is related to this error: Invalid package name: xfree-4.3.0-r6/xfree-4.3.0-r6 when trying to emerge --usepkg xfree The full bug I was about to enter: I tried to switch to xorg so I made quickpkg's of xfree. After trying xorg and wanting to switch back to xfree I got this error and had to compile the whole xfree package again. 1. I can ftp this package to another box and emerge it with no problems. It must be specific to this one box. 2. This box can emerge other packages with the sampling I've tested, just not xfree or xorg. 3. I have emerged portage and gentoolkit again on this box. It seems to me that after unpacking the binary emerge is called with a bad package category. Can someone point me to in the right direction? The error message comes from /usr/lib/portage/pym/portage.py lines 3577-3582 =================== elif operator in [">", ">=", "<", "<="]: for x in candidate_list: try: result = pkgcmp(pkgsplit(x), [cat+"/"+pkg,ver,rev]) except: writemsg("\nInvalid package name: %s\n" % x) ================= The whole failed event----- gate1 root # quickpkg xfree * Building package for xfree-4.3.0-r6... [ ok ] * Packages now in /usr/portage/packages: * xfree-4.3.0-r6: 55M gate1 root # emerge --usepkg xfree Calculating dependencies ...done! >>> emerge (1 of 1) x11-base/xfree-4.3.0-r6 to / >>> extracting info >>> extracting xfree-4.3.0-r6 >>> Merging x11-base/xfree-4.3.0-r6 to / Invalid package name: xfree-4.3.0-r6/xfree-4.3.0-r6 gate1 root # emerge info Portage 2.0.50-r9 (default-x86-2004.2, gcc-3.3.3, glibc-2.3.3.20040420-r0, 2.6.6-rc2-love2) ================================================================= System uname: 2.6.6-rc2-love2 i686 Pentium II (Deschutes) Gentoo Base System version 1.4.16 distcc 2.13 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] ccache version 2.3 [enabled] Autoconf: sys-devel/autoconf-2.59-r4 Automake: sys-devel/automake-1.8.3 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-march=pentium2 -O2 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.2/share/config /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /usr/X11R6/bin/startx /etc/env.d" CXXFLAGS="-march=pentium2 -O2 -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://gentoo.osuosl.org http://www.gtlib.cc.gatech.edu/pub/gentoo" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X alsa apache1 apm arts avi berkdb bonobo cdr crypt cups directfb dvd dvdr encode foomaticdb gdbm gif gphoto2 imlib java jpeg kde libwww mikmod mmx mozilla mpeg mysql ncurses oggvorbis opengl oss pam perl png python qt quicktime readline samba scanner sdl slang spell ssl svga tcpd truetype usb x86 xml2 xmms xv xvid zlib"
*** Bug 58884 has been marked as a duplicate of this bug. ***
*** Bug 58804 has been marked as a duplicate of this bug. ***
Created attachment 36556 [details] Script to fix /var/db/pkg Scans for the ebuild instead of assuming the filename matches the directory. Checks and fixes unbalanced quotes caused by \001.
Even the new version of the script is unable to fix sox-12.17.4-r1 and it certainly does not like being called several times: broken ebuild sox-12.17.4-r1.ebuild Traceback (most recent call last): File "./2004-07-23-fixvardbentries", line 108, in ? del portage.mtimedb["updates"] KeyError: 'updates'
*** Bug 59157 has been marked as a duplicate of this bug. ***
*** Bug 58801 has been marked as a duplicate of this bug. ***
*** Bug 52289 has been marked as a duplicate of this bug. ***
*** Bug 59426 has been marked as a duplicate of this bug. ***
*** Bug 59415 has been marked as a duplicate of this bug. ***
*** Bug 58468 has been marked as a duplicate of this bug. ***
*** Bug 59532 has been marked as a duplicate of this bug. ***
*** Bug 45912 has been marked as a duplicate of this bug. ***
*** Bug 59466 has been marked as a duplicate of this bug. ***
*** Bug 59577 has been marked as a duplicate of this bug. ***
./fixvardbentries Traceback (most recent call last): File "./fixvardbentries", line 105, in ? fix_entries(vardb+cat+"/"+pkg) File "./fixvardbentries", line 86, in fix_entries quotes += orig[x].count('"') IndexError: list index out of range
Created attachment 37209 [details] Script to fix /var/db/pkg Please try this one
*** Bug 60230 has been marked as a duplicate of this bug. ***
*** Bug 60433 has been marked as a duplicate of this bug. ***
Output of the new script (comment #45): Unfixable ebuild sox-12.17.4-r1.ebuild Please attach it to bug 46906 I'll attach the ebuild to this bug (#46096).
Created attachment 37526 [details] /var/db/pkg/media-sound/sox-12.17.4-r1/sox-12.17.4-r1.ebuild
Created attachment 37536 [details, diff] Patch against the script from comment #45 to fix the quote fixing There are 2 errors in the script from comment #45 (and probably in the ones before, too): 1. Improper check for even number of quotes quotecount/2 always is the same as int(quotecount/2). Use (quotecount % 2) instead. 2. orig[x] is a string and as such cannot be modified Use orig[x] = orig[x][:y] + '"' + orig[x][y+1:] instead of orig[x][y] = '"' The attached patch fixes both bugs.
Created attachment 37537 [details] The fixed script The script with the above patch applied.
Created attachment 37541 [details] New version of the script with support for unpacked "environment" This version will compress an "environment" file if found. Fixes the "no environment.bz2 found => package not fixed" problem.
The current script (from comment #52) fixes the broken ebuilds on all but one of my hosts. On the remaining host the "environment[.bz2]" file is missing, maybe by accident.
Created attachment 37591 [details] Script to fix /var/db/pkg Thanks for the work Sascha. I guess I was confusing languages again... Incorporated your fixes and added friendly output at each of the failure points. Still won't be able to fix all ebuilds though, but will usually be able to do enough to be able to unmerge.
*** Bug 60073 has been marked as a duplicate of this bug. ***
*** Bug 61104 has been marked as a duplicate of this bug. ***
*** Bug 61667 has been marked as a duplicate of this bug. ***
*** Bug 62052 has been marked as a duplicate of this bug. ***
in 50-r11
I've now found another (previously untouched) host where the environment file is missing: snake root # ls /var/db/pkg/mail-client/mailx-8.1.2.20021129-r2 CATEGORY CC CFLAGS CONTENTS CXX DEPEND PDEPEND PROVIDE SLOT mailx-8.1.2.20021129-r2.ebuild CBUILD CDEPEND CHOST COUNTER CXXFLAGS LICENSE PF RDEPEND USE snake root # emerge info Portage 2.0.50-r11 (default-x86-1.4, gcc-3.3.4, glibc-2.3.3.20040420-r1, 2.4.22-gentoo-r2) ================================================================= System uname: 2.4.22-gentoo-r2 i686 AMD Athlon(tm) XP 1600+ Gentoo Base System version 1.4.16 Autoconf: sys-devel/autoconf-2.59-r4 Automake: sys-devel/automake-1.8.5-r1 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-mcpu=athlon-xp -O3 -pipe" CHOST="i686-pc-linux-gnu" COMPILER="" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.1/share/config /usr/kde/3.2/share/config /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/make.globals /etc/terminfo /etc/env.d" CXXFLAGS="-mcpu=athlon-xp -O3 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache fixpackages sandbox" GENTOO_MIRRORS="ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dfx X aalib avi berkdb bitmap-fonts cdr crypt dvd encode esd gdbm gif gnome gphoto2 gtk gtk2 guile imlib ipv6 java jpeg kde libg++ libwww mad maildir mikmod motif mozilla mpeg ncurses nls oggvorbis opengl oss pam pdflib png quicktime readline sdl slang spell ssl svga tcpd truetype usb x86 xml2 xprint xv zlib" snake root #
*** Bug 64915 has been marked as a duplicate of this bug. ***
Bug has been fixed and released in stable portages on or before 2.0.51-r2
*** Bug 68639 has been marked as a duplicate of this bug. ***
*** Bug 91889 has been marked as a duplicate of this bug. ***
